Orionnnn/hub-stats huggingface dataset:
UniMuMo: Unified Text, Music and Motion Generation
Authors: name: Han Yang, name: Kun Su, name: Yutong Zhang, name: Jiaben Chen, name: Kaizhi Qian, name: Gaowen Liu, name: Chuang Gan
Published: 2024-10-06 16:04:05

We introduce UniMuMo, a unified multimodal model capable of taking arbitrary
text, music, and motion data as input conditions to generate outputs across all
three modalities. To address the lack of time-synchronized data, we align
unpaired music and motion data based on rhythmic patterns to leverage existing
large-scale music-only and motion-only datasets. By converting music, motion,
and text into token-based representation, our model bridges these modalities
through a unified encoder-decoder transformer architecture. To support multiple
generation tasks within a single framework, we introduce several architectural
improvements. We propose encoding motion with a music codebook, mapping motion
into the same feature space as music. We introduce a music-motion parallel
generation scheme that unifies all music and motion generation tasks into a
single transformer decoder architecture with a single training task of
music-motion joint generation. Moreover, the model is designed by fine-tuning
existing pre-trained single-modality models, significantly reducing
computational demands. Extensive experiments demonstrate that UniMuMo achieves
competitive results on all unidirectional generation benchmarks across music,
motion, and text modalities. Quantitative results are available in the
https://hanyangclarence.github.io/unimumo_demo/{project page}.
Differential Transformer
Authors: name: Tianzhu Ye, name: Li Dong, name: Yuqing Xia, name: Yutao Sun, name: Yi Zhu, name: Gao Huang, name: Furu Wei
Published: 2024-10-07 17:57:38

Transformer tends to overallocate attention to irrelevant context. In this
work, we introduce Diff Transformer, which amplifies attention to the relevant
context while canceling noise. Specifically, the differential attention
mechanism calculates attention scores as the difference between two separate
softmax attention maps. The subtraction cancels noise, promoting the emergence
of sparse attention patterns. Experimental results on language modeling show
that Diff Transformer outperforms Transformer in various settings of scaling up
model size and training tokens. More intriguingly, it offers notable advantages
in practical applications, such as long-context modeling, key information
retrieval, hallucination mitigation, in-context learning, and reduction of
activation outliers. By being less distracted by irrelevant context, Diff
Transformer can mitigate hallucination in question answering and text
summarization. For in-context learning, Diff Transformer not only enhances
accuracy but is also more robust to order permutation, which was considered as
a chronic robustness issue. The results position Diff Transformer as a highly
effective and promising architecture to advance large language models.
TLDR: Token-Level Detective Reward Model for Large Vision Language
  Models
Authors: name: Deqing Fu, name: Tong Xiao, name: Rui Wang, name: Wang Zhu, name: Pengchuan Zhang, name: Guan Pang, name: Robin Jia, name: Lawrence Chen
Published: 2024-10-07 04:00:22

Although reward models have been successful in improving multimodal large
language models, the reward models themselves remain brutal and contain minimal
information. Notably, existing reward models only mimic human annotations by
assigning only one binary feedback to any text, no matter how long the text is.
In the realm of multimodal language models, where models are required to
process both images and texts, a naive reward model may learn implicit biases
toward texts and become less grounded in images. In this paper, we propose a
Token-Level Detective Reward Model
(TLDR) to provide fine-grained annotations to each text token. We
first introduce a perturbation-based method to generate synthetic hard
negatives and their token-level labels to train TLDR models. Then we show the
rich usefulness of TLDR models both in assisting off-the-shelf models to
self-correct their generations, and in serving as a hallucination evaluation
tool. Finally, we show that TLDR models can significantly speed up human
annotation by 3 times to acquire a broader range of high-quality vision
language data.
VideoGuide: Improving Video Diffusion Models without Training Through a
  Teacher's Guide
Authors: name: Dohun Lee, name: Bryan S Kim, name: Geon Yeong Park, name: Jong Chul Ye
Published: 2024-10-06 05:46:17

Text-to-image (T2I) diffusion models have revolutionized visual content
creation, but extending these capabilities to text-to-video (T2V) generation
remains a challenge, particularly in preserving temporal consistency. Existing
methods that aim to improve consistency often cause trade-offs such as reduced
imaging quality and impractical computational time. To address these issues we
introduce VideoGuide, a novel framework that enhances the temporal consistency
of pretrained T2V models without the need for additional training or
fine-tuning. Instead, VideoGuide leverages any pretrained video diffusion model
(VDM) or itself as a guide during the early stages of inference, improving
temporal quality by interpolating the guiding model's denoised samples into the
sampling model's denoising process. The proposed method brings about
significant improvement in temporal consistency and image fidelity, providing a
cost-effective and practical solution that synergizes the strengths of various
video diffusion models. Furthermore, we demonstrate prior distillation,
revealing that base models can achieve enhanced text coherence by utilizing the
superior data prior of the guiding model through the proposed method. Project
Page: http://videoguide2025.github.io/
Presto! Distilling Steps and Layers for Accelerating Music Generation
Authors: name: Zachary Novack, name: Ge Zhu, name: Jonah Casebeer, name: Julian McAuley, name: Taylor Berg-Kirkpatrick, name: Nicholas J. Bryan
Published: 2024-10-07 16:24:18

Despite advances in diffusion-based text-to-music (TTM) methods, efficient,
high-quality generation remains a challenge. We introduce Presto!, an approach
to inference acceleration for score-based diffusion transformers via reducing
both sampling steps and cost per step. To reduce steps, we develop a new
score-based distribution matching distillation (DMD) method for the EDM-family
of diffusion models, the first GAN-based distillation method for TTM. To reduce
the cost per step, we develop a simple, but powerful improvement to a recent
layer distillation method that improves learning via better preserving hidden
state variance. Finally, we combine our step and layer distillation methods
together for a dual-faceted approach. We evaluate our step and layer
distillation methods independently and show each yield best-in-class
performance. Our combined distillation method can generate high-quality outputs
with improved diversity, accelerating our base model by 10-18x (230/435ms
latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) --
the fastest high-quality TTM to our knowledge. Sound examples can be found at
https://presto-music.github.io/web/.
TurtleBench: Evaluating Top Language Models via Real-World Yes/No
  Puzzles
Authors: name: Qingchen Yu, name: Shichao Song, name: Ke Fang, name: Yunfeng Shi, name: Zifan Zheng, name: Hanyu Wang, name: Simin Niu, name: Zhiyu Li
Published: 2024-10-07 17:58:47

As the application of Large Language Models (LLMs) expands, the demand for
reliable evaluations increases. Existing LLM evaluation benchmarks primarily
rely on static datasets, making it challenging to assess model performance in
dynamic interactions with users. Moreover, these benchmarks often depend on
specific background knowledge, complicating the measurement of a model's
logical reasoning capabilities. Other dynamic evaluation methods based on
strong models or manual efforts may introduce biases and incur high costs and
time demands, hindering large-scale application. To address these issues, we
propose TurtleBench. TurtleBench collects real user guesses from our online
Turtle Soup Puzzle platform that we developed. This approach allows for the
relatively dynamic generation of evaluation datasets, mitigating the risk of
model cheating while aligning assessments more closely with genuine user needs
for reasoning capabilities, thus enhancing the reliability of evaluations.
TurtleBench includes 1,532 user guesses along with the correctness of guesses
after annotation. Using this dataset, we thoroughly evaluated nine of the most
advanced LLMs available today. Notably, the OpenAI o1 series models did not
achieve leading results in these evaluations. We propose several hypotheses for
further research, such as "the latent reasoning of o1 utilizes trivial
Chain-of-Thought (CoT) techniques" and "increasing CoT length not only provides
reasoning benefits but also incurs noise costs."
Grounding Language in Multi-Perspective Referential Communication
Authors: name: Zineng Tang, name: Lingjun Mao, name: Alane Suhr
Published: 2024-10-04 22:42:30

We introduce a task and dataset for referring expression generation and
comprehension in multi-agent embodied environments. In this task, two agents in
a shared scene must take into account one another's visual perspective, which
may be different from their own, to both produce and understand references to
objects in a scene and the spatial relations between them. We collect a dataset
of 2,970 human-written referring expressions, each paired with human
comprehension judgments, and evaluate the performance of automated models as
speakers and listeners paired with human partners, finding that model
performance in both reference generation and comprehension lags behind that of
pairs of human agents. Finally, we experiment training an open-weight speaker
model with evidence of communicative success when paired with a listener,
resulting in an improvement from 58.9 to 69.3% in communicative success and
even outperforming the strongest proprietary model.
FAN: Fourier Analysis Networks
Authors: name: Yihong Dong, name: Ge Li, name: Yongding Tao, name: Xue Jiang, name: Kechi Zhang, name: Jia Li, name: Jing Su, name: Jun Zhang, name: Jingjing Xu
Published: 2024-10-03 17:02:21

Despite the remarkable success achieved by neural networks, particularly
those represented by MLP and Transformer, we reveal that they exhibit potential
flaws in the modeling and reasoning of periodicity, i.e., they tend to memorize
the periodic data rather than genuinely understanding the underlying principles
of periodicity. However, periodicity is a crucial trait in various forms of
reasoning and generalization, underpinning predictability across natural and
engineered systems through recurring patterns in observations. In this paper,
we propose FAN, a novel network architecture based on Fourier Analysis, which
empowers the ability to efficiently model and reason about periodic phenomena.
By introducing Fourier Series, the periodicity is naturally integrated into the
structure and computational processes of the neural network, thus achieving a
more accurate expression and prediction of periodic patterns. As a promising
substitute to multi-layer perceptron (MLP), FAN can seamlessly replace MLP in
various models with fewer parameters and FLOPs. Through extensive experiments,
we demonstrate the effectiveness of FAN in modeling and reasoning about
periodic functions, and the superiority and generalizability of FAN across a
range of real-world tasks, including symbolic formula representation, time
series forecasting, and language modeling.
MLP-KAN: Unifying Deep Representation and Function Learning
Authors: name: Yunhong He, name: Yifeng Xie, name: Zhengqing Yuan, name: Lichao Sun
Published: 2024-10-03 22:22:43

Recent advancements in both representation learning and function learning
have demonstrated substantial promise across diverse domains of artificial
intelligence. However, the effective integration of these paradigms poses a
significant challenge, particularly in cases where users must manually decide
whether to apply a representation learning or function learning model based on
dataset characteristics. To address this issue, we introduce MLP-KAN, a unified
method designed to eliminate the need for manual model selection. By
integrating Multi-Layer Perceptrons (MLPs) for representation learning and
Kolmogorov-Arnold Networks (KANs) for function learning within a
Mixture-of-Experts (MoE) architecture, MLP-KAN dynamically adapts to the
specific characteristics of the task at hand, ensuring optimal performance.
Embedded within a transformer-based framework, our work achieves remarkable
results on four widely-used datasets across diverse domains. Extensive
experimental evaluation demonstrates its superior versatility, delivering
competitive performance across both deep representation and function learning
tasks. These findings highlight the potential of MLP-KAN to simplify the model
selection process, offering a comprehensive, adaptable solution across various
domains. Our code and weights are available at
https://github.com/DLYuanGod/MLP-KAN.
AuroraCap: Efficient, Performant Video Detailed Captioning and a New
  Benchmark
Authors: name: Wenhao Chai, name: Enxin Song, name: Yilun Du, name: Chenlin Meng, name: Vashisht Madhavan, name: Omer Bar-Tal, name: Jeng-Neng Hwang, name: Saining Xie, name: Christopher D. Manning
Published: 2024-10-04 00:13:54

Video detailed captioning is a key task which aims to generate comprehensive
and coherent textual descriptions of video content, benefiting both video
understanding and generation. In this paper, we propose AuroraCap, a video
captioner based on a large multimodal model. We follow the simplest
architecture design without additional parameters for temporal modeling. To
address the overhead caused by lengthy video sequences, we implement the token
merging strategy, reducing the number of input visual tokens. Surprisingly, we
found that this strategy results in little performance loss. AuroraCap shows
superior performance on various video and image captioning benchmarks, for
example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and
Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include
simple descriptions, consisting of a few dozen words, which limits research in
this field. Therefore, we develop VDC, a video detailed captioning benchmark
with over one thousand carefully annotated structured captions. In addition, we
propose a new LLM-assisted metric VDCscore for bettering evaluation, which
adopts a divide-and-conquer strategy to transform long caption evaluation into
multiple short question-answer pairs. With the help of human Elo ranking, our
experiments show that this benchmark better correlates with human judgments of
video detailed captioning quality.
CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding
  Capabilities of CodeLLMs
Authors: name: Dung Nguyen Manh, name: Thang Phan Chau, name: Nam Le Hai, name: Thong T. Doan, name: Nam V. Nguyen, name: Quang Pham, name: Nghi D. Q. Bui
Published: 2024-10-02 20:04:02

Recent advancements in Code Large Language Models (CodeLLMs) have
predominantly focused on open-ended code generation tasks, often neglecting the
critical aspect of code understanding and comprehension. To bridge this gap, we
present CodeMMLU, a comprehensive multiple-choice question-answer benchmark
designed to evaluate the depth of software and code understanding in LLMs.
CodeMMLU includes over 10,000 questions sourced from diverse domains,
encompassing tasks such as code analysis, defect detection, and software
engineering principles across multiple programming languages. Unlike
traditional benchmarks, CodeMMLU assesses models's ability to reason about code
rather than merely generate it, providing deeper insights into their grasp of
complex software concepts and systems. Our extensive evaluation reveals that
even state-of-the-art models face significant challenges with CodeMMLU,
highlighting deficiencies in comprehension beyond code generation. By
underscoring the crucial relationship between code understanding and effective
generation, CodeMMLU serves as a vital resource for advancing AI-assisted
software development, ultimately aiming to create more reliable and capable
coding assistants.
Horizon-Length Prediction: Advancing Fill-in-the-Middle Capabilities for
  Code Generation with Lookahead Planning
Authors: name: Yifeng Ding, name: Hantian Ding, name: Shiqi Wang, name: Qing Sun, name: Varun Kumar, name: Zijian Wang
Published: 2024-10-04 02:53:52

Fill-in-the-Middle (FIM) has become integral to code language models,
enabling generation of missing code given both left and right contexts.
However, the current FIM training paradigm, which reorders original training
sequences and then performs regular next-token prediction (NTP), often leads to
models struggling to generate content that aligns smoothly with the surrounding
context. Crucially, while existing works rely on rule-based post-processing to
circumvent this weakness, such methods are not practically usable in
open-domain code completion tasks as they depend on restrictive,
dataset-specific assumptions (e.g., generating the same number of lines as in
the ground truth). Moreover, model performance on FIM tasks deteriorates
significantly without these unrealistic assumptions.
  We hypothesize that NTP alone is insufficient for models to learn effective
planning conditioned on the distant right context, a critical factor for
successful code infilling. To overcome this, we propose Horizon-Length
Prediction (HLP), a novel training objective that teaches models to predict the
number of remaining middle tokens (i.e., horizon length) at each step. HLP
advances FIM with lookahead planning, enabling models to inherently learn
infilling boundaries for arbitrary left and right contexts without relying on
dataset-specific post-processing. Our evaluation across different models and
sizes shows that HLP significantly improves FIM performance by up to 24%
relatively on diverse benchmarks, across file-level and repository-level, and
without resorting to unrealistic post-processing methods. Furthermore, the
enhanced planning capability gained through HLP boosts model performance on
code reasoning. Importantly, HLP only incurs negligible training overhead and
no additional inference cost, ensuring its practicality for real-world
scenarios.
Accelerating Auto-regressive Text-to-Image Generation with Training-free
  Speculative Jacobi Decoding
Authors: name: Yao Teng, name: Han Shi, name: Xian Liu, name: Xuefei Ning, name: Guohao Dai, name: Yu Wang, name: Zhenguo Li, name: Xihui Liu
Published: 2024-10-02 16:05:27

The current large auto-regressive models can generate high-quality,
high-resolution images, but these models require hundreds or even thousands of
steps of next-token prediction during inference, resulting in substantial time
consumption. In existing studies, Jacobi decoding, an iterative parallel
decoding algorithm, has been used to accelerate the auto-regressive generation
and can be executed without training. However, the Jacobi decoding relies on a
deterministic criterion to determine the convergence of iterations. Thus, it
works for greedy decoding but is incompatible with sampling-based decoding
which is crucial for visual quality and diversity in the current
auto-regressive text-to-image generation. In this paper, we propose a
training-free probabilistic parallel decoding algorithm, Speculative Jacobi
Decoding (SJD), to accelerate auto-regressive text-to-image generation. By
introducing a probabilistic convergence criterion, our SJD accelerates the
inference of auto-regressive text-to-image generation while maintaining the
randomness in sampling-based token decoding and allowing the model to generate
diverse images. Specifically, SJD facilitates the model to predict multiple
tokens at each step and accepts tokens based on the probabilistic criterion,
enabling the model to generate images with fewer steps than the conventional
next-token-prediction paradigm. We also investigate the token initialization
strategies that leverage the spatial locality of visual data to further improve
the acceleration ratio under specific scenarios. We conduct experiments for our
proposed SJD on multiple auto-regressive text-to-image generation models,
showing the effectiveness of model acceleration without sacrificing the visual
quality.
GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning
  LLMs
Authors: name: Pu Hua, name: Minghuan Liu, name: Annabella Macaluso, name: Yunfeng Lin, name: Weinan Zhang, name: Huazhe Xu, name: Lirui Wang
Published: 2024-10-04 17:51:33

Robotic simulation today remains challenging to scale up due to the human
efforts required to create diverse simulation tasks and scenes.
Simulation-trained policies also face scalability issues as many sim-to-real
methods focus on a single task. To address these challenges, this work proposes
GenSim2, a scalable framework that leverages coding LLMs with multi-modal and
reasoning capabilities for complex and realistic simulation task creation,
including long-horizon tasks with articulated objects. To automatically
generate demonstration data for these tasks at scale, we propose planning and
RL solvers that generalize within object categories. The pipeline can generate
data for up to 100 articulated tasks with 200 objects and reduce the required
human efforts. To utilize such data, we propose an effective multi-task
language-conditioned policy architecture, dubbed proprioceptive point-cloud
transformer (PPT), that learns from the generated demonstrations and exhibits
strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the
policy architecture, we show a promising usage of GenSim2 that the generated
data can be used for zero-shot transfer or co-train with real-world collected
data, which enhances the policy performance by 20% compared with training
exclusively on limited real data.
NRGBoost: Energy-Based Generative Boosted Trees
Authors: name: João Bravo
Published: 2024-10-04 15:54:02

Despite the rise to dominance of deep learning in unstructured data domains,
tree-based methods such as Random Forests (RF) and Gradient Boosted Decision
Trees (GBDT) are still the workhorses for handling discriminative tasks on
tabular data. We explore generative extensions of these popular algorithms with
a focus on explicitly modeling the data density (up to a normalization
constant), thus enabling other applications besides sampling. As our main
contribution we propose an energy-based generative boosting algorithm that is
analogous to the second order boosting implemented in popular packages like
XGBoost. We show that, despite producing a generative model capable of handling
inference tasks over any input variable, our proposed algorithm can achieve
similar discriminative performance to GBDT on a number of real world tabular
datasets, outperforming alternative generative approaches. At the same time, we
show that it is also competitive with neural network based models for sampling.