Orionnnn/hub-stats huggingface dataset:
Teaching Large Language Models to Reason with Reinforcement Learning
Authors: name: Alex Havrilla, name: Yuqing Du, name: Sharath Chandra Raparthy, name: Christoforos Nalmpantis, name: Jane Dwivedi-Yu, name: Maksym Zhuravinskyi, name: Eric Hambro, name: Sainbayar Sukhbaatar, name: Roberta Raileanu
Published: 2024-03-07 16:36:29

Reinforcement Learning from Human Feedback (RLHF) has emerged as a
dominant approach for aligning LLM outputs with human preferences. Inspired by
the success of RLHF, we study the performance of multiple algorithms that learn
from feedback (Expert Iteration, Proximal Policy Optimization (PPO),
Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate
both sparse and dense rewards provided to the LLM both heuristically and via a
learned reward model. We additionally start from multiple model sizes and
initializations both with and without supervised fine-tuning (SFT)
data. Overall, we find all algorithms perform comparably, with Expert Iteration
performing best in most cases. Surprisingly, we find the sample complexity of
Expert Iteration is similar to that of PPO, requiring at most on the order of
10^6 samples to converge from a pretrained checkpoint. We investigate why
this is the case, concluding that during RL training models fail to explore
significantly beyond solutions already produced by SFT models. Additionally, we
discuss a trade off between maj@1 and pass@96 metric performance during SFT
training and how conversely RL training improves both simultaneously. We then
conclude by discussing the implications of our findings for RLHF and the future
role of RL in LLM fine-tuning.
Yi: Open Foundation Models by 01.AI
Authors: name: 01. AI, name: Alex Young, name: Bei Chen, name: Chao Li, name: Chengen Huang, name: Ge Zhang, name: Guanwei Zhang, name: Heng Li, name: Jiangcheng Zhu, name: Jianqun Chen, name: Jing Chang, name: Kaidong Yu, name: Peng Liu, name: Qiang Liu, name: Shawn Yue, name: Senbin Yang, name: Shiming Yang, name: Tao Yu, name: Wen Xie, name: Wenhao Huang, name: Xiaohui Hu, name: Xiaoyi Ren, name: Xinyao Niu, name: Pengcheng Nie, name: Yuchi Xu, name: Yudong Liu, name: Yue Wang, name: Yuxuan Cai, name: Zhenyu Gu, name: Zhiyuan Liu, name: Zonghong Dai
Published: 2024-03-07 16:52:49

We introduce the Yi model family, a series of language and multimodal models
that demonstrate strong multi-dimensional capabilities. The Yi model family is
based on 6B and 34B pretrained language models, then we extend them to chat
models, 200K long context models, depth-upscaled models, and vision-language
models. Our base models achieve strong performance on a wide range of
benchmarks like MMLU, and our finetuned chat models deliver strong human
preference rate on major evaluation platforms like AlpacaEval and Chatbot
Arena. Building upon our scalable super-computing infrastructure and the
classical transformer architecture, we attribute the performance of Yi models
primarily to its data quality resulting from our data-engineering efforts. For
pretraining, we construct 3.1 trillion tokens of English and Chinese corpora
using a cascaded data deduplication and quality filtering pipeline. For
finetuning, we polish a small scale (less than 10K) instruction dataset over
multiple iterations such that every single instance has been verified directly
by our machine learning engineers. For vision-language, we combine the chat
language model with a vision transformer encoder and train the model to align
visual representations to the semantic space of the language model. We further
extend the context length to 200K through lightweight continual pretraining and
demonstrate strong needle-in-a-haystack retrieval performance. We show that
extending the depth of the pretrained checkpoint through continual pretraining
further improves performance. We believe that given our current results,
continuing to scale up model parameters using thoroughly optimized data will
lead to even stronger frontier models.
LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
Authors: name: Boshi Wang, name: Hao Fang, name: Jason Eisner, name: Benjamin Van Durme, name: Yu Su
Published: 2024-03-07 18:50:51

Tools are essential for large language models (LLMs) to acquire up-to-date
information and take consequential actions in external environments. Existing
work on tool-augmented LLMs primarily focuses on the broad coverage of tools
and the flexibility of adding new tools. However, a critical aspect that has
surprisingly been understudied is simply how accurately an LLM uses tools for
which it has been trained. We find that existing LLMs, including GPT-4 and
open-source LLMs specifically fine-tuned for tool use, only reach a correctness
rate in the range of 30% to 60%, far from reliable use in practice. We propose
a biologically inspired method for tool-augmented LLMs, simulated trial and
error (STE), that orchestrates three key mechanisms for successful tool use
behaviors in the biological system: trial and error, imagination, and memory.
Specifically, STE leverages an LLM's 'imagination' to simulate plausible
scenarios for using a tool, after which the LLM interacts with the tool to
learn from its execution feedback. Both short-term and long-term memory are
employed to improve the depth and breadth of the exploration, respectively.
Comprehensive experiments on ToolBench show that STE substantially improves
tool learning for LLMs under both in-context learning and fine-tuning settings,
bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform
GPT-4. We also show effective continual learning of tools via a simple
experience replay strategy.
How Far Are We from Intelligent Visual Deductive Reasoning?
Authors: name: Yizhe Zhang, name: He Bai, name: Ruixiang Zhang, name: Jiatao Gu, name: Shuangfei Zhai, name: Josh Susskind, name: Navdeep Jaitly
Published: 2024-03-07 18:35:54

Vision-Language Models (VLMs) such as GPT-4V have recently demonstrated
incredible strides on diverse vision language tasks. We dig into vision-based
deductive reasoning, a more sophisticated but less explored realm, and find
previously unexposed blindspots in the current SOTA VLMs. Specifically, we
leverage Raven's Progressive Matrices (RPMs), to assess VLMs' abilities to
perform multi-hop relational and deductive reasoning relying solely on visual
clues. We perform comprehensive evaluations of several popular VLMs employing
standard strategies such as in-context learning, self-consistency, and
Chain-of-thoughts (CoT) on three diverse datasets, including the Mensa IQ test,
IntelligenceTest, and RAVEN. The results reveal that despite the impressive
capabilities of LLMs in text-based reasoning, we are still far from achieving
comparable proficiency in visual deductive reasoning. We found that certain
standard strategies that are effective when applied to LLMs do not seamlessly
translate to the challenges presented by visual reasoning tasks. Moreover, a
detailed analysis reveals that VLMs struggle to solve these tasks mainly
because they are unable to perceive and comprehend multiple, confounding
abstract patterns in RPM examples.
Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
Authors: name: Yuanhao Cai, name: Yixun Liang, name: Jiahao Wang, name: Angtian Wang, name: Yulun Zhang, name: Xiaokang Yang, name: Zongwei Zhou, name: Alan Yuille
Published: 2024-03-07 00:12:08

X-ray is widely applied for transmission imaging due to its stronger
penetration than natural light. When rendering novel view X-ray projections,
existing methods mainly based on NeRF suffer from long training time and slow
inference speed. In this paper, we propose a 3D Gaussian splatting-based
framework, namely X-Gaussian, for X-ray novel view synthesis. Firstly, we
redesign a radiative Gaussian point cloud model inspired by the isotropic
nature of X-ray imaging. Our model excludes the influence of view direction
when learning to predict the radiation intensity of 3D points. Based on this
model, we develop a Differentiable Radiative Rasterization (DRR) with CUDA
implementation. Secondly, we customize an Angle-pose Cuboid Uniform
Initialization (ACUI) strategy that directly uses the parameters of the X-ray
scanner to compute the camera information and then uniformly samples point
positions within a cuboid enclosing the scanned object. Experiments show that
our X-Gaussian outperforms state-of-the-art methods by 6.5 dB while enjoying
less than 15% training time and over 73x inference speed. The application on
sparse-view CT reconstruction also reveals the practical values of our method.
Code and models will be publicly available at
https://github.com/caiyuanhao1998/X-Gaussian . A video demo of the training
process visualization is at https://www.youtube.com/watch?v=gDVf_Ngeghg .
PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K
  Text-to-Image Generation
Authors: name: Junsong Chen, name: Chongjian Ge, name: Enze Xie, name: Yue Wu, name: Lewei Yao, name: Xiaozhe Ren, name: Zhongdao Wang, name: Ping Luo, name: Huchuan Lu, name: Zhenguo Li
Published: 2024-03-07 17:41:37

In this paper, we introduce PixArt-\Sigma, a Diffusion Transformer
model~(DiT) capable of directly generating images at 4K resolution.
PixArt-\Sigma represents a significant advancement over its predecessor,
PixArt-\alpha, offering images of markedly higher fidelity and improved
alignment with text prompts. A key feature of PixArt-\Sigma is its training
efficiency. Leveraging the foundational pre-training of PixArt-\alpha, it
evolves from the `weaker' baseline to a `stronger' model via incorporating
higher quality data, a process we term "weak-to-strong training". The
advancements in PixArt-\Sigma are twofold: (1) High-Quality Training Data:
PixArt-\Sigma incorporates superior-quality image data, paired with more
precise and detailed image captions. (2) Efficient Token Compression: we
propose a novel attention module within the DiT framework that compresses both
keys and values, significantly improving efficiency and facilitating
ultra-high-resolution image generation. Thanks to these improvements,
PixArt-\Sigma achieves superior image quality and user prompt adherence
capabilities with significantly smaller model size (0.6B parameters) than
existing text-to-image diffusion models, such as SDXL (2.6B parameters) and SD
Cascade (5.1B parameters). Moreover, PixArt-\Sigma's capability to generate 4K
images supports the creation of high-resolution posters and wallpapers,
efficiently bolstering the production of high-quality visual content in
industries such as film and gaming.
Backtracing: Retrieving the Cause of the Query
Authors: name: Rose E. Wang, name: Pawan Wirawarn, name: Omar Khattab, name: Noah Goodman, name: Dorottya Demszky
Published: 2024-03-06 18:59:02

Many online content portals allow users to ask questions to supplement their
understanding (e.g., of lectures). While information retrieval (IR) systems may
provide answers for such user queries, they do not directly assist content
creators -- such as lecturers who want to improve their content -- identify
segments that _caused_ a user to ask those questions. We introduce the task of
backtracing, in which systems retrieve the text segment that most likely caused
a user query. We formalize three real-world domains for which backtracing is
important in improving content delivery and communication: understanding the
cause of (a) student confusion in the Lecture domain, (b) reader curiosity in
the News Article domain, and (c) user emotion in the Conversation domain. We
evaluate the zero-shot performance of popular information retrieval methods and
language modeling methods, including bi-encoder, re-ranking and
likelihood-based methods and ChatGPT. While traditional IR systems retrieve
semantically relevant information (e.g., details on "projection matrices" for a
query "does projecting multiple times still lead to the same point?"), they
often miss the causally relevant context (e.g., the lecturer states "projecting
twice gets me the same answer as one projection"). Our results show that there
is room for improvement on backtracing and it requires new retrieval
approaches. We hope our benchmark serves to improve future retrieval systems
for backtracing, spawning systems that refine content generation and identify
linguistic triggers influencing user queries. Our code and data are
open-sourced: https://github.com/rosewang2008/backtracing.
Stop Regressing: Training Value Functions via Classification for
  Scalable Deep RL
Authors: name: Jesse Farebrother, name: Jordi Orbay, name: Quan Vuong, name: Adrien Ali Taïga, name: Yevgen Chebotar, name: Ted Xiao, name: Alex Irpan, name: Sergey Levine, name: Pablo Samuel Castro, name: Aleksandra Faust, name: Aviral Kumar, name: Rishabh Agarwal
Published: 2024-03-06 18:55:47

Value functions are a central component of deep reinforcement learning (RL).
These functions, parameterized by neural networks, are trained using a mean
squared error regression objective to match bootstrapped target values.
However, scaling value-based RL methods that use regression to large networks,
such as high-capacity Transformers, has proven challenging. This difficulty is
in stark contrast to supervised learning: by leveraging a cross-entropy
classification loss, supervised methods have scaled reliably to massive
networks. Observing this discrepancy, in this paper, we investigate whether the
scalability of deep RL can also be improved simply by using classification in
place of regression for training value functions. We demonstrate that value
functions trained with categorical cross-entropy significantly improves
performance and scalability in a variety of domains. These include: single-task
RL on Atari 2600 games with SoftMoEs, multi-task RL on Atari with large-scale
ResNets, robotic manipulation with Q-transformers, playing Chess without
search, and a language-agent Wordle task with high-capacity Transformers,
achieving state-of-the-art results on these domains. Through careful analysis,
we show that the benefits of categorical cross-entropy primarily stem from its
ability to mitigate issues inherent to value-based RL, such as noisy targets
and non-stationarity. Overall, we argue that a simple shift to training value
functions with categorical cross-entropy can yield substantial improvements in
the scalability of deep RL at little-to-no cost.
3D Diffusion Policy
Authors: name: Yanjie Ze, name: Gu Zhang, name: Kangning Zhang, name: Chenyuan Hu, name: Muhan Wang, name: Huazhe Xu
Published: 2024-03-06 18:58:49

Imitation learning provides an efficient way to teach robots dexterous
skills; however, learning complex skills robustly and generalizablely usually
consumes large amounts of human demonstrations. To tackle this challenging
problem, we present 3D Diffusion Policy (DP3), a novel visual imitation
learning approach that incorporates the power of 3D visual representations into
diffusion policies, a class of conditional action generative models. The core
design of DP3 is the utilization of a compact 3D visual representation,
extracted from sparse point clouds with an efficient point encoder. In our
experiments involving 72 simulation tasks, DP3 successfully handles most tasks
with just 10 demonstrations and surpasses baselines with a 55.3% relative
improvement. In 4 real robot tasks, DP3 demonstrates precise control with a
high success rate of 85%, given only 40 demonstrations of each task, and shows
excellent generalization abilities in diverse aspects, including space,
viewpoint, appearance, and instance. Interestingly, in real robot experiments,
DP3 rarely violates safety requirements, in contrast to baseline methods which
frequently do, necessitating human intervention. Our extensive evaluation
highlights the critical importance of 3D representations in real-world robot
learning. Videos, code, and data are available on
https://3d-diffusion-policy.github.io .
SaulLM-7B: A pioneering Large Language Model for Law
Authors: name: Pierre Colombo, name: Telmo Pessoa Pires, name: Malik Boudiaf, name: Dominic Culver, name: Rui Melo, name: Caio Corro, name: Andre F. T. Martins, name: Fabrizio Esposito, name: Vera Lúcia Raposo, name: Sofia Morgado, name: Michael Desa
Published: 2024-03-06 17:42:16

In this paper, we introduce SaulLM-7B, a large language model (LLM) tailored
for the legal domain. With 7 billion parameters, SaulLM-7B is the first LLM
designed explicitly for legal text comprehension and generation. Leveraging the
Mistral 7B architecture as its foundation, SaulLM-7B is trained on an English
legal corpus of over 30 billion tokens. SaulLM-7B exhibits state-of-the-art
proficiency in understanding and processing legal documents. Additionally, we
present a novel instructional fine-tuning method that leverages legal datasets
to further enhance SaulLM-7B's performance in legal tasks. SaulLM-7B is
released under the CC-BY-SA-4.0 License.
Learning to Decode Collaboratively with Multiple Language Models
Authors: name: Shannon Zejiang Shen, name: Hunter Lang, name: Bailin Wang, name: Yoon Kim, name: David Sontag
Published: 2024-03-06 17:23:28

We propose a method to teach multiple large language models (LLM) to
collaborate by interleaving their generations at the token level. We model the
decision of which LLM generates the next token as a latent variable. By
optimizing the marginal likelihood of a training set under our latent variable
model, the base LLM automatically learns when to generate itself and when to
call on one of the ``assistant'' language models to generate, all without
direct supervision. Token-level collaboration during decoding allows for a
fusion of each model's expertise in a manner tailored to the specific task at
hand. Our collaborative decoding is especially useful in cross-domain settings
where a generalist base LLM learns to invoke domain expert models. On
instruction-following, domain-specific QA, and reasoning tasks, we show that
the performance of the joint system exceeds that of the individual models.
Through qualitative analysis of the learned latent decisions, we show models
trained with our method exhibit several interesting collaboration patterns,
e.g., template-filling. Our code is available at
https://github.com/clinicalml/co-llm.
ShortGPT: Layers in Large Language Models are More Redundant Than You
  Expect
Authors: name: Xin Men, name: Mingyu Xu, name: Qingyu Zhang, name: Bingning Wang, name: Hongyu Lin, name: Yaojie Lu, name: Xianpei Han, name: Weipeng Chen
Published: 2024-03-06 17:04:18

As Large Language Models (LLMs) continue to advance in performance, their
size has escalated significantly, with current LLMs containing billions or even
trillions of parameters. However, in this study, we discovered that many layers
of LLMs exhibit high similarity, and some layers play a negligible role in
network functionality. Based on this observation, we define a metric called
Block Influence (BI) to gauge the significance of each layer in LLMs. We then
propose a straightforward pruning approach: layer removal, in which we directly
delete the redundant layers in LLMs based on their BI scores. Experiments
demonstrate that our method, which we call ShortGPT, significantly outperforms
previous state-of-the-art (SOTA) methods in model pruning. Moreover, ShortGPT
is orthogonal to quantization-like methods, enabling further reduction in
parameters and computation. The ability to achieve better results through
simple layer removal, as opposed to more complex pruning techniques, suggests a
high degree of redundancy in the model architecture.
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Authors: name: Jiawei Zhao, name: Zhenyu Zhang, name: Beidi Chen, name: Zhangyang Wang, name: Anima Anandkumar, name: Yuandong Tian
Published: 2024-03-06 07:29:57

Training Large Language Models (LLMs) presents significant memory challenges,
predominantly due to the growing size of weights and optimizer states. Common
memory-reduction approaches, such as low-rank adaptation (LoRA), add a
trainable low-rank matrix to the frozen pre-trained weight in each layer,
reducing trainable parameters and optimizer states. However, such approaches
typically underperform training with full-rank weights in both pre-training and
fine-tuning stages since they limit the parameter search to a low-rank subspace
and alter the training dynamics, and further, may require full-rank warm start.
In this work, we propose Gradient Low-Rank Projection (GaLore), a training
strategy that allows full-parameter learning but is more memory-efficient than
common low-rank adaptation methods such as LoRA. Our approach reduces memory
usage by up to 65.5% in optimizer states while maintaining both efficiency and
performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset
with up to 19.7B tokens, and on fine-tuning RoBERTa on GLUE tasks. Our 8-bit
GaLore further reduces optimizer memory by up to 82.5% and total training
memory by 63.3%, compared to a BF16 baseline. Notably, we demonstrate, for the
first time, the feasibility of pre-training a 7B model on consumer GPUs with
24GB memory (e.g., NVIDIA RTX 4090) without model parallel, checkpointing, or
offloading strategies.
Enhancing Vision-Language Pre-training with Rich Supervisions
Authors: name: Yuan Gao, name: Kunyu Shi, name: Pengkai Zhu, name: Edouard Belval, name: Oren Nuriel, name: Srikar Appalaraju, name: Shabnam Ghadar, name: Vijay Mahadevan, name: Zhuowen Tu, name: Stefano Soatto
Published: 2024-03-05 22:14:58

We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel
pre-training paradigm for Vision-Language Models using data from large-scale
web screenshot rendering. Using web screenshots unlocks a treasure trove of
visual and textual cues that are not present in using image-text pairs. In S4,
we leverage the inherent tree-structured hierarchy of HTML elements and the
spatial localization to carefully design 10 pre-training tasks with large scale
annotated data. These tasks resemble downstream tasks across different domains
and the annotations are cheap to obtain. We demonstrate that, compared to
current screenshot pre-training objectives, our innovative pre-training method
significantly enhances performance of image-to-text model in nine varied and
popular downstream tasks - up to 76.1% improvements on Table Detection, and at
least 1% on Widget Captioning.
Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
Authors: name: Yair Schiff, name: Chia-Hsiang Kao, name: Aaron Gokaslan, name: Tri Dao, name: Albert Gu, name: Volodymyr Kuleshov
Published: 2024-03-05 01:42:51

Large-scale sequence modeling has sparked rapid advances that now extend into
biology and genomics. However, modeling genomic sequences introduces challenges
such as the need to model long-range token interactions, the effects of
upstream and downstream regions of the genome, and the reverse complementarity
(RC) of DNA. Here, we propose an architecture motivated by these challenges
that builds off the long-range Mamba block, and extends it to a BiMamba
component that supports bi-directionality, and to a MambaDNA block that
additionally supports RC equivariance. We use MambaDNA as the basis of
Caduceus, the first family of RC equivariant bi-directional long-range DNA
language models, and we introduce pre-training and fine-tuning strategies that
yield Caduceus DNA foundation models. Caduceus outperforms previous long-range
models on downstream benchmarks; on a challenging long-range variant effect
prediction task, Caduceus exceeds the performance of 10x larger models that do
not leverage bi-directionality or equivariance.