Orionnnn/hub-stats huggingface dataset:
Scaling Laws for Reward Model Overoptimization in Direct Alignment
  Algorithms
Authors: name: Rafael Rafailov, name: Yaswanth Chittepu, name: Ryan Park, name: Harshit Sikchi, name: Joey Hejna, name: Bradley Knox, name: Chelsea Finn, name: Scott Niekum
Published: 2024-06-05 03:41:37

Reinforcement Learning from Human Feedback (RLHF) has been crucial to the
recent success of Large Language Models (LLMs), however, it is often a complex
and brittle process. In the classical RLHF framework, a reward model is first
trained to represent human preferences, which is in turn used by an online
reinforcement learning (RL) algorithm to optimize the LLM. A prominent issue
with such methods is reward over-optimization or reward hacking,
where performance as measured by the learned proxy reward model increases, but
true quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs)
like Direct Preference Optimization have emerged as alternatives to the
classical RLHF pipeline by circumventing the reward modeling phase. However,
although DAAs do not use a separate proxy reward model, they still commonly
deteriorate from over-optimization. While the so-called reward hacking
phenomenon is not well-defined for DAAs, we still uncover similar trends: at
higher KL budgets, DAA algorithms exhibit similar degradation patterns to their
classic RLHF counterparts. In particular, we find that DAA methods deteriorate
not only across a wide range of KL budgets but also often before even a single
epoch of the dataset is completed. Through extensive empirical experimentation,
this work formulates and formalizes the reward over-optimization or hacking
problem for DAAs and explores its consequences across objectives, training
regimes, and model scales.
LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive
  Modeling of Audio Discrete Codes
Authors: name: Trung Dang, name: David Aponte, name: Dung Tran, name: Kazuhito Koishida
Published: 2024-06-05 03:36:11

Prior works have demonstrated zero-shot text-to-speech by using a generative
language model on audio tokens obtained via a neural audio codec. It is still
challenging, however, to adapt them to low-latency scenarios. In this paper, we
present LiveSpeech - a fully autoregressive language model-based approach for
zero-shot text-to-speech, enabling low-latency streaming of the output audio.
To allow multiple token prediction within a single decoding step, we propose
(1) using adaptive codebook loss weights that consider codebook contribution in
each frame and focus on hard instances, and (2) grouping codebooks and
processing groups in parallel. Experiments show our proposed models achieve
competitive results to state-of-the-art baselines in terms of content accuracy,
speaker similarity, audio quality, and inference speed while being suitable for
low-latency streaming applications.
PLaD: Preference-based Large Language Model Distillation with
  Pseudo-Preference Pairs
Authors: name: Rongzhi Zhang, name: Jiaming Shen, name: Tianqi Liu, name: Haorui Wang, name: Zhen Qin, name: Feng Han, name: Jialu Liu, name: Simon Baumgartner, name: Michael Bendersky, name: Chao Zhang
Published: 2024-06-05 03:08:25

Large Language Models (LLMs) have exhibited impressive capabilities in
various tasks, yet their vast parameter sizes restrict their applicability in
resource-constrained settings. Knowledge distillation (KD) offers a viable
solution by transferring expertise from large teacher models to compact student
models. However, traditional KD techniques face specific challenges when
applied to LLMs, including restricted access to LLM outputs, significant
teacher-student capacity gaps, and the inherited mis-calibration issue. In this
work, we present PLaD, a novel preference-based LLM distillation framework.
PLaD exploits the teacher-student capacity discrepancy to generate
pseudo-preference pairs where teacher outputs are preferred over student
outputs. Then, PLaD leverages a ranking loss to re-calibrate student's
estimation of sequence likelihood, which steers the student's focus towards
understanding the relative quality of outputs instead of simply imitating the
teacher. PLaD bypasses the need for access to teacher LLM's internal states,
tackles the student's expressivity limitations, and mitigates the student
mis-calibration issue. Through extensive experiments on two sequence generation
tasks and with various LLMs, we demonstrate the effectiveness of our proposed
PLaD framework.
PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with
  LLM
Authors: name: Tao Yang, name: Yingmin Luo, name: Zhongang Qi, name: Yang Wu, name: Ying Shan, name: Chang Wen Chen
Published: 2024-06-05 03:05:52

Layout generation is the keystone in achieving automated graphic design,
requiring arranging the position and size of various multi-modal design
elements in a visually pleasing and constraint-following manner. Previous
approaches are either inefficient for large-scale applications or lack
flexibility for varying design requirements. Our research introduces a unified
framework for automated graphic layout generation, leveraging the multi-modal
large language model (MLLM) to accommodate diverse design tasks. In contrast,
our data-driven method employs structured text (JSON format) and visual
instruction tuning to generate layouts under specific visual and textual
constraints, including user-defined natural language specifications. We
conducted extensive experiments and achieved state-of-the-art (SOTA)
performance on public multi-modal layout generation benchmarks, demonstrating
the effectiveness of our method. Moreover, recognizing existing datasets'
limitations in capturing the complexity of real-world graphic designs, we
propose two new datasets for much more challenging tasks (user-constrained
generation and complicated poster), further validating our model's utility in
real-life settings. Marking by its superior accessibility and adaptability,
this approach further automates large-scale graphic design tasks. The code and
datasets will be publicly available on
https://github.com/posterllava/PosterLLaVA.
Xmodel-LM Technical Report
Authors: name: Yichuan Wang, name: Yang Liu, name: Yu Yan, name: Xucheng Huang, name: Ling Jiang
Published: 2024-06-05 02:12:06

We introduce Xmodel-LM, a compact and efficient 1.1B language model
pre-trained on over 2 trillion tokens. Trained on our self-built dataset
(Xdata), which balances Chinese and English corpora based on downstream task
optimization, Xmodel-LM exhibits remarkable performance despite its smaller
size. It notably surpasses existing open-source language models of similar
scale. Our model checkpoints and code are publicly accessible on GitHub at
https://github.com/XiaoduoAILab/XmodelLM.
Item-Language Model for Conversational Recommendation
Authors: name: Li Yang, name: Anushya Subbiah, name: Hardik Patel, name: Judith Yue Li, name: Yanwei Song, name: Reza Mirghaderi, name: Vikram Aggarwal
Published: 2024-06-05 01:35:50

Large-language Models (LLMs) have been extremely successful at tasks like
complex dialogue understanding, reasoning and coding due to their emergent
abilities. These emergent abilities have been extended with multi-modality to
include image, audio, and video capabilities. Recommender systems, on the other
hand, have been critical for information seeking and item discovery needs.
Recently, there have been attempts to apply LLMs for recommendations. One
difficulty of current attempts is that the underlying LLM is usually not
trained on the recommender system data, which largely contains user interaction
signals and is often not publicly available. Another difficulty is user
interaction signals often have a different pattern from natural language text,
and it is currently unclear if the LLM training setup can learn more
non-trivial knowledge from interaction signals compared with traditional
recommender system methods. Finally, it is difficult to train multiple LLMs for
different use-cases, and to retain the original language and reasoning
abilities when learning from recommender system data. To address these three
limitations, we propose an Item-Language Model (ILM), which is composed of an
item encoder to produce text-aligned item representations that encode user
interaction signals, and a frozen LLM that can understand those item
representations with preserved pretrained knowledge. We conduct extensive
experiments which demonstrate both the importance of the language-alignment and
of user interaction knowledge in the item encoder.
Block Transformer: Global-to-Local Language Modeling for Fast Inference
Authors: name: Namgyu Ho, name: Sangmin Bae, name: Taehyeon Kim, name: Hyunjik Jo, name: Yireun Kim, name: Tal Schuster, name: Adam Fisch, name: James Thorne, name: Se-Young Yun
Published: 2024-06-04 17:45:26

This paper presents the Block Transformer architecture which adopts
hierarchical global-to-local modeling to autoregressive transformers to
mitigate the inference bottlenecks of self-attention. To apply self-attention,
the key-value (KV) cache of all previous sequences must be retrieved from
memory at every decoding step. Thereby, this KV cache IO becomes a significant
bottleneck in batch inference. We notice that these costs stem from applying
self-attention on the global context, therefore we isolate the expensive
bottlenecks of global modeling to lower layers and apply fast local modeling in
upper layers. To mitigate the remaining costs in the lower layers, we aggregate
input tokens into fixed size blocks and then apply self-attention at this
coarse level. Context information is aggregated into a single embedding to
enable upper layers to decode the next block of tokens, without global
attention. Free of global attention bottlenecks, the upper layers can fully
utilize the compute hardware to maximize inference throughput. By leveraging
global and local modules, the Block Transformer architecture demonstrates
10-20x gains in inference throughput compared to vanilla transformers with
equivalent perplexity. Our work introduces a new approach to optimize language
model inference through novel application of global-to-local modeling. Code is
available at https://github.com/itsnamgyu/block-transformer.
Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
Authors: name: Hao Wen, name: Zehuan Huang, name: Yaohui Wang, name: Xinyuan Chen, name: Yu Qiao, name: Lu Sheng
Published: 2024-06-05 12:15:22

Existing single image-to-3D creation methods typically involve a two-stage
process, first generating multi-view images, and then using these images for 3D
reconstruction. However, training these two stages separately leads to
significant data bias in the inference phase, thus affecting the quality of
reconstructed results. We introduce a unified 3D generation framework, named
Ouroboros3D, which integrates diffusion-based multi-view image generation and
3D reconstruction into a recursive diffusion process. In our framework, these
two modules are jointly trained through a self-conditioning mechanism, allowing
them to adapt to each other's characteristics for robust inference. During the
multi-view denoising process, the multi-view diffusion model uses the 3D-aware
maps rendered by the reconstruction module at the previous timestep as
additional conditions. The recursive diffusion framework with 3D-aware feedback
unites the entire process and improves geometric consistency.Experiments show
that our framework outperforms separation of these two stages and existing
methods that combine them at the inference phase. Project page:
https://costwen.github.io/Ouroboros3D/
Parrot: Multilingual Visual Instruction Tuning
Authors: name: Hai-Long Sun, name: Da-Wei Zhou, name: Yang Li, name: Shiyin Lu, name: Chao Yi, name: Qing-Guo Chen, name: Zhao Xu, name: Weihua Luo, name: Kaifu Zhang, name: De-Chuan Zhan, name: Han-Jia Ye
Published: 2024-06-04 17:56:28

The rapid development of Multimodal Large Language Models (MLLMs) like GPT-4V
has marked a significant step towards artificial general intelligence. Existing
methods mainly focus on aligning vision encoders with LLMs through supervised
fine-tuning (SFT) to endow LLMs with multimodal abilities, making MLLMs'
inherent ability to react to multiple languages progressively deteriorate as
the training process evolves. We empirically find that the imbalanced SFT
datasets, primarily composed of English-centric image-text pairs, lead to
significantly reduced performance in non-English languages. This is due to the
failure of aligning the vision encoder and LLM with multilingual tokens during
the SFT process. In this paper, we introduce Parrot, a novel method that
utilizes textual guidance to drive visual token alignment at the language
level. Parrot makes the visual tokens condition on diverse language inputs and
uses Mixture-of-Experts (MoE) to promote the alignment of multilingual tokens.
Specifically, to enhance non-English visual tokens alignment, we compute the
cross-attention using the initial visual features and textual embeddings, the
result of which is then fed into the MoE router to select the most relevant
experts. The selected experts subsequently convert the initial visual tokens
into language-specific visual tokens. Moreover, considering the current lack of
benchmarks for evaluating multilingual capabilities within the field, we
collect and make available a Massive Multilingual Multimodal Benchmark which
includes 6 languages, 15 categories, and 12,000 questions, named as MMMB. Our
method not only demonstrates state-of-the-art performance on multilingual
MMBench and MMMB, but also excels across a broad range of multimodal tasks.
Both the source code and the training dataset of Parrot will be made publicly
available.
Self-Improving Robust Preference Optimization
Authors: name: Eugene Choi, name: Arash Ahmadian, name: Matthieu Geist, name: Oilvier Pietquin, name: Mohammad Gheshlaghi Azar
Published: 2024-06-03 17:53:25

Both online and offline RLHF methods such as PPO and DPO have been extremely
successful in aligning AI with human preferences. Despite their success, the
existing methods suffer from a fundamental problem that their optimal solution
is highly task-dependent (i.e., not robust to out-of-distribution (OOD) tasks).
Here we address this challenge by proposing Self-Improving Robust Preference
Optimization SRPO, a practical and mathematically principled offline RLHF
framework that is completely robust to the changes in the task. The key idea of
SRPO is to cast the problem of learning from human preferences as a
self-improvement process, which can be mathematically expressed in terms of a
min-max objective that aims at joint optimization of self-improvement policy
and the generative policy in an adversarial fashion. The solution for this
optimization problem is independent of the training task and thus it is robust
to its changes. We then show that this objective can be re-expressed in the
form of a non-adversarial offline loss which can be optimized using standard
supervised optimization techniques at scale without any need for reward model
and online inference. We show the effectiveness of SRPO in terms of AI Win-Rate
(WR) against human (GOLD) completions. In particular, when SRPO is evaluated on
the OOD XSUM dataset, it outperforms the celebrated DPO by a clear margin of
15% after 5 self-revisions, achieving WR of 90%.
Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
Authors: name: Philip Anastassiou, name: Jiawei Chen, name: Jitong Chen, name: Yuanzhe Chen, name: Zhuo Chen, name: Ziyi Chen, name: Jian Cong, name: Lelai Deng, name: Chuang Ding, name: Lu Gao, name: Mingqing Gong, name: Peisong Huang, name: Qingqing Huang, name: Zhiying Huang, name: Yuanyuan Huo, name: Dongya Jia, name: Chumin Li, name: Feiya Li, name: Hui Li, name: Jiaxin Li, name: Xiaoyang Li, name: Xingxing Li, name: Lin Liu, name: Shouda Liu, name: Sichao Liu, name: Xudong Liu, name: Yuchen Liu, name: Zhengxi Liu, name: Lu Lu, name: Junjie Pan, name: Xin Wang, name: Yuping Wang, name: Yuxuan Wang, name: Zhen Wei, name: Jian Wu, name: Chao Yao, name: Yifeng Yang, name: Yuanhao Yi, name: Junteng Zhang, name: Qidi Zhang, name: Shuo Zhang, name: Wenjie Zhang, name: Yang Zhang, name: Zilin Zhao, name: Dejian Zhong, name: Xiaobin Zhuang
Published: 2024-06-04 15:48:29

We introduce Seed-TTS, a family of large-scale autoregressive text-to-speech
(TTS) models capable of generating speech that is virtually indistinguishable
from human speech. Seed-TTS serves as a foundation model for speech generation
and excels in speech in-context learning, achieving performance in speaker
similarity and naturalness that matches ground truth human speech in both
objective and subjective evaluations. With fine-tuning, we achieve even higher
subjective scores across these metrics. Seed-TTS offers superior
controllability over various speech attributes such as emotion and is capable
of generating highly expressive and diverse speech for speakers in the wild.
Furthermore, we propose a self-distillation method for speech factorization, as
well as a reinforcement learning approach to enhance model robustness, speaker
similarity, and controllability. We additionally present a non-autoregressive
(NAR) variant of the Seed-TTS model, named Seed-TTS_DiT, which
utilizes a fully diffusion-based architecture. Unlike previous NAR-based TTS
systems, Seed-TTS_DiT does not depend on pre-estimated phoneme
durations and performs speech generation through end-to-end processing. We
demonstrate that this variant achieves comparable performance to the language
model-based variant and showcase its effectiveness in speech editing. We
encourage readers to listen to demos at
https://bytedancespeech.github.io/seedtts_tech_report.
To Believe or Not to Believe Your LLM
Authors: name: Yasin Abbasi Yadkori, name: Ilja Kuzborskij, name: András György, name: Csaba Szepesvári
Published: 2024-06-04 17:58:18

We explore uncertainty quantification in large language models (LLMs), with
the goal to identify when uncertainty in responses given a query is large. We
simultaneously consider both epistemic and aleatoric uncertainties, where the
former comes from the lack of knowledge about the ground truth (such as about
facts or the language), and the latter comes from irreducible randomness (such
as multiple possible answers). In particular, we derive an
information-theoretic metric that allows to reliably detect when only epistemic
uncertainty is large, in which case the output of the model is unreliable. This
condition can be computed based solely on the output of the model obtained
simply by some special iterative prompting based on the previous responses.
Such quantification, for instance, allows to detect hallucinations (cases when
epistemic uncertainty is high) in both single- and multi-answer responses. This
is in contrast to many standard uncertainty quantification strategies (such as
thresholding the log-likelihood of a response) where hallucinations in the
multi-answer case cannot be detected. We conduct a series of experiments which
demonstrate the advantage of our formulation. Further, our investigations shed
some light on how the probabilities assigned to a given output by an LLM can be
amplified by iterative prompting, which might be of independent interest.
Guiding a Diffusion Model with a Bad Version of Itself
Authors: name: Tero Karras, name: Miika Aittala, name: Tuomas Kynkäänniemi, name: Jaakko Lehtinen, name: Timo Aila, name: Samuli Laine
Published: 2024-06-04 17:25:59

The primary axes of interest in image-generating diffusion models are image
quality, the amount of variation in the results, and how well the results align
with a given condition, e.g., a class label or a text prompt. The popular
classifier-free guidance approach uses an unconditional model to guide a
conditional model, leading to simultaneously better prompt alignment and
higher-quality images at the cost of reduced variation. These effects seem
inherently entangled, and thus hard to control. We make the surprising
observation that it is possible to obtain disentangled control over image
quality without compromising the amount of variation by guiding generation
using a smaller, less-trained version of the model itself rather than an
unconditional model. This leads to significant improvements in ImageNet
generation, setting record FIDs of 1.01 for 64x64 and 1.25 for 512x512, using
publicly available networks. Furthermore, the method is also applicable to
unconditional diffusion models, drastically improving their quality.
RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
Authors: name: Soroush Nasiriany, name: Abhiram Maddukuri, name: Lance Zhang, name: Adeet Parikh, name: Aaron Lo, name: Abhishek Joshi, name: Ajay Mandlekar, name: Yuke Zhu
Published: 2024-06-04 17:41:31

Recent advancements in Artificial Intelligence (AI) have largely been
propelled by scaling. In Robotics, scaling is hindered by the lack of access to
massive robot datasets. We advocate using realistic physical simulation as a
means to scale environments, tasks, and datasets for robot learning methods. We
present RoboCasa, a large-scale simulation framework for training generalist
robots in everyday environments. RoboCasa features realistic and diverse scenes
focusing on kitchen environments. We provide thousands of 3D assets across over
150 object categories and dozens of interactable furniture and appliances. We
enrich the realism and diversity of our simulation with generative AI tools,
such as object assets from text-to-3D models and environment textures from
text-to-image models. We design a set of 100 tasks for systematic evaluation,
including composite tasks generated by the guidance of large language models.
To facilitate learning, we provide high-quality human demonstrations and
integrate automated trajectory generation methods to substantially enlarge our
datasets with minimal human burden. Our experiments show a clear scaling trend
in using synthetically generated robot data for large-scale imitation learning
and show great promise in harnessing simulation data in real-world tasks.
Videos and open-source code are available at https://robocasa.ai/
V-Express: Conditional Dropout for Progressive Training of Portrait
  Video Generation
Authors: name: Cong Wang, name: Kuan Tian, name: Jun Zhang, name: Yonghang Guan, name: Feng Luo, name: Fei Shen, name: Zhiwei Jiang, name: Qing Gu, name: Xiao Han, name: Wei Yang
Published: 2024-06-04 17:32:52

In the field of portrait video generation, the use of single images to
generate portrait videos has become increasingly prevalent. A common approach
involves leveraging generative models to enhance adapters for controlled
generation. However, control signals (e.g., text, audio, reference image, pose,
depth map, etc.) can vary in strength. Among these, weaker conditions often
struggle to be effective due to interference from stronger conditions, posing a
challenge in balancing these conditions. In our work on portrait video
generation, we identified audio signals as particularly weak, often
overshadowed by stronger signals such as facial pose and reference image.
However, direct training with weak signals often leads to difficulties in
convergence. To address this, we propose V-Express, a simple method that
balances different control signals through the progressive training and the
conditional dropout operation. Our method gradually enables effective control
by weak conditions, thereby achieving generation capabilities that
simultaneously take into account the facial pose, reference image, and audio.
The experimental results demonstrate that our method can effectively generate
portrait videos controlled by audio. Furthermore, a potential solution is
provided for the simultaneous and effective use of conditions of varying
strengths.