Orionnnn/hub-stats huggingface dataset:
Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient
  Video Latent Generation
Authors: name: Chenyu Wang, name: Shuo Yan, name: Yixuan Chen, name: Yujiang Wang, name: Mingzhi Dong, name: Xiaochen Yang, name: Dongsheng Li, name: Robert P. Dick, name: Qin Lv, name: Fan Yang, name: Tun Lu, name: Ning Gu, name: Li Shang
Published: 2024-09-19 07:50:34

Video generation using diffusion-based models is constrained by high
computational costs due to the frame-wise iterative diffusion process. This
work presents a Diffusion Reuse MOtion (Dr. Mo) network to accelerate latent
video generation. Our key discovery is that coarse-grained noises in earlier
denoising steps have demonstrated high motion consistency across consecutive
video frames. Following this observation, Dr. Mo propagates those
coarse-grained noises onto the next frame by incorporating carefully designed,
lightweight inter-frame motions, eliminating massive computational redundancy
in frame-wise diffusion models. The more sensitive and fine-grained noises are
still acquired via later denoising steps, which can be essential to retain
visual qualities. As such, deciding which intermediate steps should switch from
motion-based propagations to denoising can be a crucial problem and a key
tradeoff between efficiency and quality. Dr. Mo employs a meta-network named
Denoising Step Selector (DSS) to dynamically determine desirable intermediate
steps across video frames. Extensive evaluations on video generation and
editing tasks have shown that Dr. Mo can substantially accelerate diffusion
models in video tasks with improved visual qualities.
MURI: High-Quality Instruction Tuning Datasets for Low-Resource
  Languages via Reverse Instructions
Authors: name: Abdullatif Köksal, name: Marion Thaler, name: Ayyoob Imani, name: Ahmet Üstün, name: Anna Korhonen, name: Hinrich Schütze
Published: 2024-09-19 17:59:20

Instruction tuning enhances large language models (LLMs) by aligning them
with human preferences across diverse tasks. Traditional approaches to create
instruction tuning datasets face serious challenges for low-resource languages
due to their dependence on data annotation. This work introduces a novel
method, Multilingual Reverse Instructions (MURI), which generates high-quality
instruction tuning datasets for low-resource languages without requiring human
annotators or pre-existing multilingual models. Utilizing reverse instructions
and a translation pipeline, MURI produces instruction-output pairs from
existing human-written texts in low-resource languages. This method ensures
cultural relevance and diversity by sourcing texts from different native
domains and applying filters to eliminate inappropriate content. Our dataset,
MURI-IT, includes more than 2 million instruction-output pairs across 200
languages. Evaluation by native speakers and fine-tuning experiments with mT5
models demonstrate the approach's effectiveness for both NLU and open-ended
generation. We publicly release datasets and models at
https://github.com/akoksal/muri.
RoMath: A Mathematical Reasoning Benchmark in Romanian
Authors: name: Adrian Cosma, name: Ana-Maria Bucur, name: Emilian Radoi
Published: 2024-09-17 11:03:46

Mathematics has long been conveyed through natural language, primarily for
human understanding. With the rise of mechanized mathematics and proof
assistants, there is a growing need to understand informal mathematical text,
yet most existing benchmarks focus solely on English, overlooking other
languages. This paper introduces RoMath, a Romanian mathematical reasoning
benchmark suite comprising three datasets: RoMath-Baccalaureate,
RoMath-Competitions and RoMath-Synthetic, which cover a range of mathematical
domains and difficulty levels, aiming to improve non-English language models
and promote multilingual AI development. By focusing on Romanian, a
low-resource language with unique linguistic features, RoMath addresses the
limitations of Anglo-centric models and emphasizes the need for dedicated
resources beyond simple automatic translation. We benchmark several open-weight
language models, highlighting the importance of creating resources for
underrepresented languages. We make the code and dataset available.
CORE-Bench: Fostering the Credibility of Published Research Through a
  Computational Reproducibility Agent Benchmark
Authors: name: Zachary S. Siegel, name: Sayash Kapoor, name: Nitya Nagdir, name: Benedikt Stroebl, name: Arvind Narayanan
Published: 2024-09-17 17:13:19

AI agents have the potential to aid users on a variety of consequential
tasks, including conducting scientific research. To spur the development of
useful agents, we need benchmarks that are challenging, but more crucially,
directly correspond to real-world tasks of interest. This paper introduces such
a benchmark, designed to measure the accuracy of AI agents in tackling a
crucial yet surprisingly challenging aspect of scientific research:
computational reproducibility. This task, fundamental to the scientific
process, involves reproducing the results of a study using the provided code
and data. We introduce CORE-Bench (Computational Reproducibility Agent
Benchmark), a benchmark consisting of 270 tasks based on 90 scientific papers
across three disciplines (computer science, social science, and medicine).
Tasks in CORE-Bench consist of three difficulty levels and include both
language-only and vision-language tasks. We provide an evaluation system to
measure the accuracy of agents in a fast and parallelizable way, saving days of
evaluation time for each run compared to a sequential implementation. We
evaluated two baseline agents: the general-purpose AutoGPT and a task-specific
agent called CORE-Agent. We tested both variants using two underlying language
models: GPT-4o and GPT-4o-mini. The best agent achieved an accuracy of 21% on
the hardest task, showing the vast scope for improvement in automating routine
scientific tasks. Having agents that can reproduce existing work is a necessary
step towards building agents that can conduct novel research and could verify
and improve the performance of other research agents. We hope that CORE-Bench
can improve the state of reproducibility and spur the development of future
research agents.
fMRI-3D: A Comprehensive Dataset for Enhancing fMRI-based 3D
  Reconstruction
Authors: name: Jianxiong Gao, name: Yuqian Fu, name: Yun Wang, name: Xuelin Qian, name: Jianfeng Feng, name: Yanwei Fu
Published: 2024-09-17 16:13:59

Reconstructing 3D visuals from functional Magnetic Resonance Imaging (fMRI)
data, introduced as Recon3DMind in our conference work, is of significant
interest to both cognitive neuroscience and computer vision. To advance this
task, we present the fMRI-3D dataset, which includes data from 15 participants
and showcases a total of 4768 3D objects. The dataset comprises two components:
fMRI-Shape, previously introduced and accessible at
https://huggingface.co/datasets/Fudan-fMRI/fMRI-Shape, and fMRI-Objaverse,
proposed in this paper and available at
https://huggingface.co/datasets/Fudan-fMRI/fMRI-Objaverse. fMRI-Objaverse
includes data from 5 subjects, 4 of whom are also part of the Core set in
fMRI-Shape, with each subject viewing 3142 3D objects across 117 categories,
all accompanied by text captions. This significantly enhances the diversity and
potential applications of the dataset. Additionally, we propose MinD-3D, a
novel framework designed to decode 3D visual information from fMRI signals. The
framework first extracts and aggregates features from fMRI data using a
neuro-fusion encoder, then employs a feature-bridge diffusion model to generate
visual features, and finally reconstructs the 3D object using a generative
transformer decoder. We establish new benchmarks by designing metrics at both
semantic and structural levels to evaluate model performance. Furthermore, we
assess our model's effectiveness in an Out-of-Distribution setting and analyze
the attribution of the extracted features and the visual ROIs in fMRI signals.
Our experiments demonstrate that MinD-3D not only reconstructs 3D objects with
high semantic and spatial accuracy but also deepens our understanding of how
human brain processes 3D visual information. Project page at:
https://jianxgao.github.io/MinD-3D.
Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning
Authors: name: Claude Formanek, name: Louise Beyers, name: Callum Rhys Tilbury, name: Jonathan P. Shock, name: Arnu Pretorius
Published: 2024-09-18 14:13:24

Offline multi-agent reinforcement learning (MARL) is an exciting direction of
research that uses static datasets to find optimal control policies for
multi-agent systems. Though the field is by definition data-driven, efforts
have thus far neglected data in their drive to achieve state-of-the-art
results. We first substantiate this claim by surveying the literature, showing
how the majority of works generate their own datasets without consistent
methodology and provide sparse information about the characteristics of these
datasets. We then show why neglecting the nature of the data is problematic,
through salient examples of how tightly algorithmic performance is coupled to
the dataset used, necessitating a common foundation for experiments in the
field. In response, we take a big step towards improving data usage and data
awareness in offline MARL, with three key contributions: (1) a clear guideline
for generating novel datasets; (2) a standardisation of over 80 existing
datasets, hosted in a publicly available repository, using a consistent storage
format and easy-to-use API; and (3) a suite of analysis tools that allow us to
understand these datasets better, aiding further development.
BERT-VBD: Vietnamese Multi-Document Summarization Framework
Authors: name: Tuan-Cuong Vuong, name: Trang Mai Xuan, name: Thien Van Luong
Published: 2024-09-18 16:56:06

In tackling the challenge of Multi-Document Summarization (MDS), numerous
methods have been proposed, spanning both extractive and abstractive
summarization techniques. However, each approach has its own limitations,
making it less effective to rely solely on either one. An emerging and
promising strategy involves a synergistic fusion of extractive and abstractive
summarization methods. Despite the plethora of studies in this domain, research
on the combined methodology remains scarce, particularly in the context of
Vietnamese language processing. This paper presents a novel Vietnamese MDS
framework leveraging a two-component pipeline architecture that integrates
extractive and abstractive techniques. The first component employs an
extractive approach to identify key sentences within each document. This is
achieved by a modification of the pre-trained BERT network, which derives
semantically meaningful phrase embeddings using siamese and triplet network
structures. The second component utilizes the VBD-LLaMA2-7B-50b model for
abstractive summarization, ultimately generating the final summary document.
Our proposed framework demonstrates a positive performance, attaining ROUGE-2
scores of 39.6% on the VN-MDS dataset and outperforming the state-of-the-art
baselines.
Measuring Human and AI Values based on Generative Psychometrics with
  Large Language Models
Authors: name: Haoran Ye, name: Yuhang Xie, name: Yuanyi Ren, name: Hanjun Fang, name: Xin Zhang, name: Guojie Song
Published: 2024-09-18 16:26:22

Human values and their measurement are long-standing interdisciplinary
inquiry. Recent advances in AI have sparked renewed interest in this area, with
large language models (LLMs) emerging as both tools and subjects of value
measurement. This work introduces Generative Psychometrics for Values (GPV), an
LLM-based, data-driven value measurement paradigm, theoretically grounded in
text-revealed selective perceptions. We begin by fine-tuning an LLM for
accurate perception-level value measurement and verifying the capability of
LLMs to parse texts into perceptions, forming the core of the GPV pipeline.
Applying GPV to human-authored blogs, we demonstrate its stability, validity,
and superiority over prior psychological tools. Then, extending GPV to LLM
value measurement, we advance the current art with 1) a psychometric
methodology that measures LLM values based on their scalable and free-form
outputs, enabling context-specific measurement; 2) a comparative analysis of
measurement paradigms, indicating response biases of prior methods; and 3) an
attempt to bridge LLM values and their safety, revealing the predictive power
of different value systems and the impacts of various values on LLM safety.
Through interdisciplinary efforts, we aim to leverage AI for next-generation
psychometrics and psychometrics for value-aligned AI.
SoloAudio: Target Sound Extraction with Language-oriented Audio
  Diffusion Transformer
Authors: name: Helin Wang, name: Jiarui Hai, name: Yen-Ju Lu, name: Karan Thakkar, name: Mounya Elhilali, name: Najim Dehak
Published: 2024-09-12 23:12:25

In this paper, we introduce SoloAudio, a novel diffusion-based generative
model for target sound extraction (TSE). Our approach trains latent diffusion
models on audio, replacing the previous U-Net backbone with a skip-connected
Transformer that operates on latent features. SoloAudio supports both
audio-oriented and language-oriented TSE by utilizing a CLAP model as the
feature extractor for target sounds. Furthermore, SoloAudio leverages synthetic
audio generated by state-of-the-art text-to-audio models for training,
demonstrating strong generalization to out-of-domain data and unseen sound
events. We evaluate this approach on the FSD Kaggle 2018 mixture dataset and
real data from AudioSet, where SoloAudio achieves the state-of-the-art results
on both in-domain and out-of-domain data, and exhibits impressive zero-shot and
few-shot capabilities. Source code and demos are released.
Vista3D: Unravel the 3D Darkside of a Single Image
Authors: name: Qiuhong Shen, name: Xingyi Yang, name: Michael Bi Mi, name: Xinchao Wang
Published: 2024-09-18 17:59:44

We embark on the age-old quest: unveiling the hidden dimensions of objects
from mere glimpses of their visible parts. To address this, we present Vista3D,
a framework that realizes swift and consistent 3D generation within a mere 5
minutes. At the heart of Vista3D lies a two-phase approach: the coarse phase
and the fine phase. In the coarse phase, we rapidly generate initial geometry
with Gaussian Splatting from a single image. In the fine phase, we extract a
Signed Distance Function (SDF) directly from learned Gaussian Splatting,
optimizing it with a differentiable isosurface representation. Furthermore, it
elevates the quality of generation by using a disentangled representation with
two independent implicit functions to capture both visible and obscured aspects
of objects. Additionally, it harmonizes gradients from 2D diffusion prior with
3D-aware diffusion priors by angular diffusion prior composition. Through
extensive evaluation, we demonstrate that Vista3D effectively sustains a
balance between the consistency and diversity of the generated 3D objects.
Demos and code will be available at https://github.com/florinshen/Vista3D.
Towards Diverse and Efficient Audio Captioning via Diffusion Models
Authors: name: Manjie Xu, name: Chenxing Li, name: Xinyi Tu, name: Yong Ren, name: Ruibo Fu, name: Wei Liang, name: Dong Yu
Published: 2024-09-14 10:23:35

We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive
diffusion model tailored for diverse and efficient audio captioning. Although
existing captioning models relying on language backbones have achieved
remarkable success in various captioning tasks, their insufficient performance
in terms of generation speed and diversity impede progress in audio
understanding and multimedia applications. Our diffusion-based framework offers
unique advantages stemming from its inherent stochasticity and holistic context
modeling in captioning. Through rigorous evaluation, we demonstrate that DAC
not only achieves SOTA performance levels compared to existing benchmarks in
the caption quality, but also significantly outperforms them in terms of
generation speed and diversity. The success of DAC illustrates that text
generation can also be seamlessly integrated with audio and visual generation
tasks using a diffusion backbone, paving the way for a unified, audio-related
generative model across different modalities.
A Controlled Study on Long Context Extension and Generalization in LLMs
Authors: name: Yi Lu, name: Jing Nathan Yan, name: Songlin Yang, name: Justin T. Chiu, name: Siyu Ren, name: Fei Yuan, name: Wenting Zhao, name: Zhiyong Wu, name: Alexander M. Rush
Published: 2024-09-18 17:53:17

Broad textual understanding and in-context learning require language models
that utilize full document contexts. Due to the implementation challenges
associated with directly training long-context models, many methods have been
proposed for extending models to handle long contexts. However, owing to
differences in data and model classes, it has been challenging to compare these
approaches, leading to uncertainty as to how to evaluate long-context
performance and whether it differs from standard evaluation. We implement a
controlled protocol for extension methods with a standardized evaluation,
utilizing consistent base models and extension data. Our study yields several
insights into long-context behavior. First, we reaffirm the critical role of
perplexity as a general-purpose performance indicator even in longer-context
tasks. Second, we find that current approximate attention methods
systematically underperform across long-context tasks. Finally, we confirm that
exact fine-tuning based methods are generally effective within the range of
their extension, whereas extrapolation remains challenging. All codebases,
models, and checkpoints will be made available open-source, promoting
transparency and facilitating further research in this critical area of AI
development.
Preference Tuning with Human Feedback on Language, Speech, and Vision
  Tasks: A Survey
Authors: name: Genta Indra Winata, name: Hanyang Zhao, name: Anirban Das, name: Wenpin Tang, name: David D. Yao, name: Shi-Xiong Zhang, name: Sambit Sahu
Published: 2024-09-17 21:28:51

Preference tuning is a crucial process for aligning deep generative models
with human preferences. This survey offers a thorough overview of recent
advancements in preference tuning and the integration of human feedback. The
paper is organized into three main sections: 1) introduction and preliminaries:
an introduction to reinforcement learning frameworks, preference tuning tasks,
models, and datasets across various modalities: language, speech, and vision,
as well as different policy approaches, 2) in-depth examination of each
preference tuning approach: a detailed analysis of the methods used in
preference tuning, and 3) applications, discussion, and future directions: an
exploration of the applications of preference tuning in downstream tasks,
including evaluation methods for different modalities, and an outlook on future
research directions. Our objective is to present the latest methodologies in
preference tuning and model alignment, enhancing the understanding of this
field for researchers and practitioners. We hope to encourage further
engagement and innovation in this area.
Qwen2.5-Coder Technical Report
Authors: name: Binyuan Hui, name: Jian Yang, name: Zeyu Cui, name: Jiaxi Yang, name: Dayiheng Liu, name: Lei Zhang, name: Tianyu Liu, name: Jiajun Zhang, name: Bowen Yu, name: Kai Dang, name: An Yang, name: Rui Men, name: Fei Huang, name: Xingzhang Ren, name: Xuancheng Ren, name: Jingren Zhou, name: Junyang Lin
Published: 2024-09-18 17:57:57

In this report, we introduce the Qwen2.5-Coder series, a significant upgrade
from its predecessor, CodeQwen1.5. This series includes two models:
Qwen2.5-Coder-1.5B and Qwen2.5-Coder-7B. As a code-specific model,
Qwen2.5-Coder is built upon the Qwen2.5 architecture and continues pretrained
on a vast corpus of over 5.5 trillion tokens. Through meticulous data cleaning,
scalable synthetic data generation, and balanced data mixing, Qwen2.5-Coder
demonstrates impressive code generation capabilities while retaining general
versatility. The model has been evaluated on a wide range of code-related
tasks, achieving state-of-the-art (SOTA) performance across more than 10
benchmarks, including code generation, completion, reasoning, and repair,
consistently outperforming larger models of the same model size. We believe
that the release of the Qwen2.5-Coder series will not only push the boundaries
of research in code intelligence but also, through its permissive licensing,
encourage broader adoption by developers in real-world applications.
GRIN: GRadient-INformed MoE
Authors: name: Liyuan Liu, name: Young Jin Kim, name: Shuohang Wang, name: Chen Liang, name: Yelong Shen, name: Hao Cheng, name: Xiaodong Liu, name: Masahiro Tanaka, name: Xiaoxia Wu, name: Wenxiang Hu, name: Vishrav Chaudhary, name: Zeqi Lin, name: Chenruidong Zhang, name: Jilong Xue, name: Hany Awadalla, name: Jianfeng Gao, name: Weizhu Chen
Published: 2024-09-18 17:00:20

Mixture-of-Experts (MoE) models scale more effectively than dense models due
to sparse computation through expert routing, selectively activating only a
small subset of expert modules. However, sparse computation challenges
traditional training practices, as discrete expert routing hinders standard
backpropagation and thus gradient-based optimization, which are the cornerstone
of deep learning. To better pursue the scaling power of MoE, we introduce GRIN
(GRadient-INformed MoE training), which incorporates sparse gradient estimation
for expert routing and configures model parallelism to avoid token dropping.
Applying GRIN to autoregressive language modeling, we develop a top-2
16times3.8B MoE model. Our model, with only 6.6B activated parameters,
outperforms a 7B dense model and matches the performance of a 14B dense model
trained on the same data. Extensive evaluations across diverse tasks
demonstrate the potential of GRIN to significantly enhance MoE efficacy,
achieving 79.4 on MMLU, 83.7 on HellaSwag, 74.4 on HumanEval, and 58.9 on MATH.
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic
  reasoning
Authors: name: Zayne Sprague, name: Fangcong Yin, name: Juan Diego Rodriguez, name: Dongwei Jiang, name: Manya Wadhwa, name: Prasann Singhal, name: Xinyu Zhao, name: Xi Ye, name: Kyle Mahowald, name: Greg Durrett
Published: 2024-09-18 17:55:00

Chain-of-thought (CoT) via prompting is the de facto method for eliciting
reasoning capabilities from large language models (LLMs). But for what kinds of
tasks is this extra ``thinking'' really helpful? To analyze this, we conducted
a quantitative meta-analysis covering over 100 papers using CoT and ran our own
evaluations of 20 datasets across 14 models. Our results show that CoT gives
strong performance benefits primarily on tasks involving math or logic, with
much smaller gains on other types of tasks. On MMLU, directly generating the
answer without CoT leads to almost identical accuracy as CoT unless the
question or model's response contains an equals sign, indicating symbolic
operations and reasoning. Following this finding, we analyze the behavior of
CoT on these problems by separating planning and execution and comparing
against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic
execution, but it underperforms relative to using a symbolic solver. Our
results indicate that CoT can be applied selectively, maintaining performance
while saving inference costs. Furthermore, they suggest a need to move beyond
prompt-based CoT to new paradigms that better leverage intermediate computation
across the whole range of LLM applications.