Orionnnn/hub-stats huggingface dataset:
MuCodec: Ultra Low-Bitrate Music Codec
Authors: name: Yaoxun Xu, name: Hangting Chen, name: Jianwei Yu, name: Wei Tan, name: Rongzhi Gu, name: Shun Lei, name: Zhiwei Lin, name: Zhiyong Wu
Published: 2024-09-20 05:06:49

Music codecs are a vital aspect of audio codec research, and ultra
low-bitrate compression holds significant importance for music transmission and
generation. Due to the complexity of music backgrounds and the richness of
vocals, solely relying on modeling semantic or acoustic information cannot
effectively reconstruct music with both vocals and backgrounds. To address this
issue, we propose MuCodec, specifically targeting music compression and
reconstruction tasks at ultra low bitrates. MuCodec employs MuEncoder to
extract both acoustic and semantic features, discretizes them with RVQ, and
obtains Mel-VAE features via flow-matching. The music is then reconstructed
using a pre-trained MEL-VAE decoder and HiFi-GAN. MuCodec can reconstruct
high-fidelity music at ultra low (0.35kbps) or high bitrates (1.35kbps),
achieving the best results to date in both subjective and objective metrics.
Code and Demo: https://xuyaoxun.github.io/MuCodec_demo/.
CLAIR-A: Leveraging Large Language Models to Judge Audio Captions
Authors: name: Tsung-Han Wu, name: Joseph E. Gonzalez, name: Trevor Darrell, name: David M. Chan
Published: 2024-09-19 17:59:52

The Automated Audio Captioning (AAC) task asks models to generate natural
language descriptions of an audio input. Evaluating these machine-generated
audio captions is a complex task that requires considering diverse factors,
among them, auditory scene understanding, sound-object inference, temporal
coherence, and the environmental context of the scene. While current methods
focus on specific aspects, they often fail to provide an overall score that
aligns well with human judgment. In this work, we propose CLAIR-A, a simple and
flexible method that leverages the zero-shot capabilities of large language
models (LLMs) to evaluate candidate audio captions by directly asking LLMs for
a semantic distance score. In our evaluations, CLAIR-A better predicts human
judgements of quality compared to traditional metrics, with a 5.8% relative
accuracy improvement compared to the domain-specific FENSE metric and up to 11%
over the best general-purpose measure on the Clotho-Eval dataset. Moreover,
CLAIR-A offers more transparency by allowing the language model to explain the
reasoning behind its scores, with these explanations rated up to 30% better by
human evaluators than those provided by baseline methods. CLAIR-A is made
publicly available at https://github.com/DavidMChan/clair-a.
B4: Towards Optimal Assessment of Plausible Code Solutions with
  Plausible Tests
Authors: name: Mouxiang Chen, name: Zhongxin Liu, name: He Tao, name: Yusu Hong, name: David Lo, name: Xin Xia, name: Jianling Sun
Published: 2024-09-13 10:22:08

Selecting the best code solution from multiple generated ones is an essential
task in code generation, which can be achieved by using some reliable
validators (e.g., developer-written test cases) for assistance. Since reliable
test cases are not always available and can be expensive to build in practice,
researchers propose to automatically generate test cases to assess code
solutions. However, when both code solutions and test cases are plausible and
not reliable, selecting the best solution becomes challenging. Although some
heuristic strategies have been proposed to tackle this problem, they lack a
strong theoretical guarantee and it is still an open question whether an
optimal selection strategy exists. Our work contributes in two ways. First, we
show that within a Bayesian framework, the optimal selection strategy can be
defined based on the posterior probability of the observed passing states
between solutions and tests. The problem of identifying the best solution is
then framed as an integer programming problem. Second, we propose an efficient
approach for approximating this optimal (yet uncomputable) strategy, where the
approximation error is bounded by the correctness of prior knowledge. We then
incorporate effective prior knowledge to tailor code generation tasks. Both
theoretical and empirical studies confirm that existing heuristics are limited
in selecting the best solutions with plausible test cases. Our proposed
approximated optimal strategy B4 significantly surpasses existing heuristics in
selecting code solutions generated by large language models (LLMs) with
LLM-generated tests, achieving a relative performance improvement by up to 50%
over the strongest heuristic and 246% over the random selection in the most
challenging scenarios. Our code is publicly available at
https://github.com/ZJU-CTAG/B4.
Training Language Models to Self-Correct via Reinforcement Learning
Authors: name: Aviral Kumar, name: Vincent Zhuang, name: Rishabh Agarwal, name: Yi Su, name: John D Co-Reyes, name: Avi Singh, name: Kate Baumli, name: Shariq Iqbal, name: Colton Bishop, name: Rebecca Roelofs, name: Lei M Zhang, name: Kay McKinney, name: Disha Shrivastava, name: Cosmin Paduraru, name: George Tucker, name: Doina Precup, name: Feryal Behbahani, name: Aleksandra Faust
Published: 2024-09-19 17:16:21

Self-correction is a highly desirable capability of large language models
(LLMs), yet it has consistently been found to be largely ineffective in modern
LLMs. Existing approaches for training self-correction either require multiple
models or rely on a more capable model or other forms of supervision. To this
end, we develop a multi-turn online reinforcement learning (RL) approach,
SCoRe, that significantly improves an LLM's self-correction ability using
entirely self-generated data. To build SCoRe, we first show that variants of
supervised fine-tuning (SFT) on offline model-generated correction traces are
insufficient for instilling self-correction behavior. In particular, we observe
that training via SFT either suffers from a distribution mismatch between the
training data and the model's own responses or implicitly prefers only a
certain mode of correction behavior that is often not effective at test time.
SCoRe addresses these challenges by training under the model's own distribution
of self-generated correction traces and using appropriate regularization to
steer the learning process into learning a self-correction strategy that is
effective at test time as opposed to simply fitting high-reward responses for a
given prompt. This regularization prescribes running a first phase of RL on a
base model to generate a policy initialization that is less susceptible to
collapse and then using a reward bonus to amplify self-correction during
training. When applied to Gemini 1.0 Pro and 1.5 Flash models, we find that
SCoRe achieves state-of-the-art self-correction performance, improving the base
models' self-correction by 15.6% and 9.1% respectively on the MATH and
HumanEval benchmarks.
MMSearch: Benchmarking the Potential of Large Models as Multi-modal
  Search Engines
Authors: name: Dongzhi Jiang, name: Renrui Zhang, name: Ziyu Guo, name: Yanmin Wu, name: Jiayi Lei, name: Pengshuo Qiu, name: Pan Lu, name: Zehui Chen, name: Guanglu Song, name: Peng Gao, name: Yu Liu, name: Chunyuan Li, name: Hongsheng Li
Published: 2024-09-19 17:59:45

The advent of Large Language Models (LLMs) has paved the way for AI search
engines, e.g., SearchGPT, showcasing a new paradigm in human-internet
interaction. However, most current AI search engines are limited to text-only
settings, neglecting the multimodal user queries and the text-image interleaved
nature of website information. Recently, Large Multimodal Models (LMMs) have
made impressive strides. Yet, whether they can function as AI search engines
remains under-explored, leaving the potential of LMMs in multimodal search an
open question. To this end, we first design a delicate pipeline,
MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On
top of this, we introduce MMSearch, a comprehensive evaluation benchmark to
assess the multimodal search performance of LMMs. The curated dataset contains
300 manually collected instances spanning 14 subfields, which involves no
overlap with the current LMMs' training data, ensuring the correct answer can
only be obtained within searching. By using MMSearch-Engine, the LMMs are
evaluated by performing three individual tasks (requery, rerank, and
summarization), and one challenging end-to-end task with a complete searching
process. We conduct extensive experiments on closed-source and open-source
LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best
results, which surpasses the commercial product, Perplexity Pro, in the
end-to-end task, demonstrating the effectiveness of our proposed pipeline. We
further present error analysis to unveil current LMMs still struggle to fully
grasp the multimodal search tasks, and conduct ablation study to indicate the
potential of scaling test-time computation for AI search engine. We hope
MMSearch may provide unique insights to guide the future development of
multimodal AI search engine. Project Page: https://mmsearch.github.io
Language Models Learn to Mislead Humans via RLHF
Authors: name: Jiaxin Wen, name: Ruiqi Zhong, name: Akbir Khan, name: Ethan Perez, name: Jacob Steinhardt, name: Minlie Huang, name: Samuel R. Boman, name: He He, name: Shi Feng
Published: 2024-09-19 14:50:34

Language models (LMs) can produce errors that are hard to detect for humans,
especially when the task is complex. RLHF, the most popular post-training
method, may exacerbate this problem: to achieve higher rewards, LMs might get
better at convincing humans that they are right even when they are wrong. We
study this phenomenon under a standard RLHF pipeline, calling it "U-SOPHISTRY"
since it is Unintended by model developers. Specifically, we ask
time-constrained (e.g., 3-10 minutes) human subjects to evaluate the
correctness of model outputs and calculate humans' accuracy against gold
labels. On a question-answering task (QuALITY) and programming task (APPS),
RLHF makes LMs better at convincing our subjects but not at completing the task
correctly. RLHF also makes the model harder to evaluate: our subjects' false
positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show
that probing, a state-of-the-art approach for detecting Intended Sophistry
(e.g. backdoored LMs), does not generalize to U-SOPHISTRY. Our results
highlight an important failure mode of RLHF and call for more research in
assisting humans to align them.
Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary
  Resolution
Authors: name: Zuyan Liu, name: Yuhao Dong, name: Ziwei Liu, name: Winston Hu, name: Jiwen Lu, name: Yongming Rao
Published: 2024-09-19 17:59:51

Visual data comes in various forms, ranging from small icons of just a few
pixels to long videos spanning hours. Existing multi-modal LLMs usually
standardize these diverse visual inputs to a fixed resolution for visual
encoders and yield similar numbers of tokens for LLMs. This approach is
non-optimal for multimodal understanding and inefficient for processing inputs
with long and short visual contents. To solve the problem, we propose Oryx, a
unified multimodal architecture for the spatial-temporal understanding of
images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to
seamlessly and efficiently process visual inputs with arbitrary spatial sizes
and temporal lengths through two core innovations: 1) a pre-trained OryxViT
model that can encode images at any resolution into LLM-friendly visual
representations; 2) a dynamic compressor module that supports 1x to 16x
compression on visual tokens by request. These design features enable Oryx to
accommodate extremely long visual contexts, such as videos, with lower
resolution and high compression while maintaining high recognition precision
for tasks like document understanding with native resolution and no
compression. Beyond the architectural improvements, enhanced data curation and
specialized training on long-context retrieval and spatial-aware data help Oryx
achieve strong capabilities in image, video, and 3D multimodal understanding
simultaneously. Our work is open-sourced at https://github.com/Oryx-mllm/Oryx.
Scaling Smart: Accelerating Large Language Model Pre-training with Small
  Model Initialization
Authors: name: Mohammad Samragh, name: Iman Mirzadeh, name: Keivan Alizadeh Vahid, name: Fartash Faghri, name: Minsik Cho, name: Moin Nabi, name: Devang Naik, name: Mehrdad Farajtabar
Published: 2024-09-19 16:50:26

The pre-training phase of language models often begins with randomly
initialized parameters. With the current trends in scaling models, training
their large number of parameters can be extremely slow and costly. In contrast,
small language models are less expensive to train, but they often cannot
achieve the accuracy of large models. In this paper, we explore an intriguing
idea to connect these two different regimes: Can we develop a method to
initialize large language models using smaller pre-trained models? Will such
initialization bring any benefits in terms of training time and final accuracy?
In this paper, we introduce HyperCloning, a method that can expand the
parameters of a pre-trained language model to those of a larger model with
increased hidden dimensions. Our method ensures that the larger model retains
the functionality of the smaller model. As a result, the larger model already
inherits the predictive power and accuracy of the smaller model before the
training starts. We demonstrate that training such an initialized model results
in significant savings in terms of GPU hours required for pre-training large
language models.
LVCD: Reference-based Lineart Video Colorization with Diffusion Models
Authors: name: Zhitong Huang, name: Mohan Zhang, name: Jing Liao
Published: 2024-09-19 17:59:48

We propose the first video diffusion framework for reference-based lineart
video colorization. Unlike previous works that rely solely on image generative
models to colorize lineart frame by frame, our approach leverages a large-scale
pretrained video diffusion model to generate colorized animation videos. This
approach leads to more temporally consistent results and is better equipped to
handle large motions. Firstly, we introduce Sketch-guided ControlNet which
provides additional control to finetune an image-to-video diffusion model for
controllable video synthesis, enabling the generation of animation videos
conditioned on lineart. We then propose Reference Attention to facilitate the
transfer of colors from the reference frame to other frames containing fast and
expansive motions. Finally, we present a novel scheme for sequential sampling,
incorporating the Overlapped Blending Module and Prev-Reference Attention, to
extend the video diffusion model beyond its original fixed-length limitation
for long video colorization. Both qualitative and quantitative results
demonstrate that our method significantly outperforms state-of-the-art
techniques in terms of frame and video quality, as well as temporal
consistency. Moreover, our method is capable of generating high-quality, long
temporal-consistent animation videos with large motions, which is not
achievable in previous works. Our code and model are available at
https://luckyhzt.github.io/lvcd.
3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive
  Diffusion
Authors: name: Zhaoxi Chen, name: Jiaxiang Tang, name: Yuhao Dong, name: Ziang Cao, name: Fangzhou Hong, name: Yushi Lan, name: Tengfei Wang, name: Haozhe Xie, name: Tong Wu, name: Shunsuke Saito, name: Liang Pan, name: Dahua Lin, name: Ziwei Liu
Published: 2024-09-19 17:59:06

The increasing demand for high-quality 3D assets across various industries
necessitates efficient and automated 3D content creation. Despite recent
advancements in 3D generative models, existing methods still face challenges
with optimization speed, geometric fidelity, and the lack of assets for
physically based rendering (PBR). In this paper, we introduce 3DTopia-XL, a
scalable native 3D generative model designed to overcome these limitations.
3DTopia-XL leverages a novel primitive-based 3D representation, PrimX, which
encodes detailed shape, albedo, and material field into a compact tensorial
format, facilitating the modeling of high-resolution geometry with PBR assets.
On top of the novel representation, we propose a generative framework based on
Diffusion Transformer (DiT), which comprises 1) Primitive Patch Compression, 2)
and Latent Primitive Diffusion. 3DTopia-XL learns to generate high-quality 3D
assets from textual or visual inputs. We conduct extensive qualitative and
quantitative experiments to demonstrate that 3DTopia-XL significantly
outperforms existing methods in generating high-quality 3D assets with
fine-grained textures and materials, efficiently bridging the quality gap
between generative models and real-world applications.
3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-Marquardt
Authors: name: Lukas Höllein, name: Aljaž Božič, name: Michael Zollhöfer, name: Matthias Nießner
Published: 2024-09-19 16:31:44

We present 3DGS-LM, a new method that accelerates the reconstruction of 3D
Gaussian Splatting (3DGS) by replacing its ADAM optimizer with a tailored
Levenberg-Marquardt (LM). Existing methods reduce the optimization time by
decreasing the number of Gaussians or by improving the implementation of the
differentiable rasterizer. However, they still rely on the ADAM optimizer to
fit Gaussian parameters of a scene in thousands of iterations, which can take
up to an hour. To this end, we change the optimizer to LM that runs in
conjunction with the 3DGS differentiable rasterizer. For efficient GPU
parallization, we propose a caching data structure for intermediate gradients
that allows us to efficiently calculate Jacobian-vector products in custom CUDA
kernels. In every LM iteration, we calculate update directions from multiple
image subsets using these kernels and combine them in a weighted mean. Overall,
our method is 30% faster than the original 3DGS while obtaining the same
reconstruction quality. Our optimization is also agnostic to other methods that
acclerate 3DGS, thus enabling even faster speedups compared to vanilla 3DGS.
StoryMaker: Towards Holistic Consistent Characters in Text-to-image
  Generation
Authors: name: Zhengguang Zhou, name: Jing Li, name: Huaxia Li, name: Nemo Chen, name: Xu Tang
Published: 2024-09-19 08:53:06

Tuning-free personalized image generation methods have achieved significant
success in maintaining facial consistency, i.e., identities, even with multiple
characters. However, the lack of holistic consistency in scenes with multiple
characters hampers these methods' ability to create a cohesive narrative. In
this paper, we introduce StoryMaker, a personalization solution that preserves
not only facial consistency but also clothing, hairstyles, and body
consistency, thus facilitating the creation of a story through a series of
images. StoryMaker incorporates conditions based on face identities and cropped
character images, which include clothing, hairstyles, and bodies. Specifically,
we integrate the facial identity information with the cropped character images
using the Positional-aware Perceiver Resampler (PPR) to obtain distinct
character features. To prevent intermingling of multiple characters and the
background, we separately constrain the cross-attention impact regions of
different characters and the background using MSE loss with segmentation masks.
Additionally, we train the generation network conditioned on poses to promote
decoupling from poses. A LoRA is also employed to enhance fidelity and quality.
Experiments underscore the effectiveness of our approach. StoryMaker supports
numerous applications and is compatible with other societal plug-ins. Our
source codes and model weights are available at
https://github.com/RedAIGC/StoryMaker.
InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced
  Mathematical Reasoning
Authors: name: Xiaotian Han, name: Yiren Jian, name: Xuefeng Hu, name: Haogeng Liu, name: Yiqi Wang, name: Qihang Fan, name: Yuang Ai, name: Huaibo Huang, name: Ran He, name: Zhenheng Yang, name: Quanzeng You
Published: 2024-09-19 08:41:21

Pre-training on large-scale, high-quality datasets is crucial for enhancing
the reasoning capabilities of Large Language Models (LLMs), especially in
specialized domains such as mathematics. Despite the recognized importance, the
Multimodal LLMs (MLLMs) field currently lacks a comprehensive open-source
pre-training dataset specifically designed for mathematical reasoning. To
address this gap, we introduce InfiMM-WebMath-40B, a high-quality dataset of
interleaved image-text documents. It comprises 24 million web pages, 85 million
associated image URLs, and 40 billion text tokens, all meticulously extracted
and filtered from CommonCrawl. We provide a detailed overview of our data
collection and processing pipeline. To demonstrate the robustness of
InfiMM-WebMath-40B, we conducted evaluations in both text-only and multimodal
settings. Our evaluations on text-only benchmarks show that, despite utilizing
only 40 billion tokens, our dataset significantly enhances the performance of
our 1.3B model, delivering results comparable to DeepSeekMath-1.3B, which uses
120 billion tokens for the same model size. Nevertheless, with the introduction
of our multi-modal math pre-training dataset, our models set a new
state-of-the-art among open-source models on multi-modal math benchmarks such
as MathVerse and We-Math. We release our data at
https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.
FlexiTex: Enhancing Texture Generation with Visual Guidance
Authors: name: DaDong Jiang, name: Xianghui Yang, name: Zibo Zhao, name: Sheng Zhang, name: Jiaao Yu, name: Zeqiang Lai, name: Shaoxiong Yang, name: Chunchao Guo, name: Xiaobo Zhou, name: Zhihui Ke
Published: 2024-09-19 03:24:22

Recent texture generation methods achieve impressive results due to the
powerful generative prior they leverage from large-scale text-to-image
diffusion models. However, abstract textual prompts are limited in providing
global textural or shape information, which results in the texture generation
methods producing blurry or inconsistent patterns. To tackle this, we present
FlexiTex, embedding rich information via visual guidance to generate a
high-quality texture. The core of FlexiTex is the Visual Guidance Enhancement
module, which incorporates more specific information from visual guidance to
reduce ambiguity in the text prompt and preserve high-frequency details. To
further enhance the visual guidance, we introduce a Direction-Aware Adaptation
module that automatically designs direction prompts based on different camera
poses, avoiding the Janus problem and maintaining semantically global
consistency. Benefiting from the visual guidance, FlexiTex produces
quantitatively and qualitatively sound results, demonstrating its potential to
advance texture generation for real-world applications.