Orionnnn/hub-stats huggingface dataset:
Pre-training Small Base LMs with Fewer Tokens
Authors: name: Sunny Sanyal, name: Sujay Sanghavi, name: Alexandros G. Dimakis
Published: 2024-04-12 17:53:34

We study the effectiveness of a simple approach to develop a small base
language model (LM) starting from an existing large base LM: first inherit a
few transformer blocks from the larger LM, and then train this smaller model on
a very small subset (0.1\%) of the raw pretraining data of the larger model. We
call our simple recipe Inheritune and first demonstrate it for building a small
base LM with 1.5B parameters using 1B tokens (and a starting few layers of
larger LM of 3B parameters); we do this using a single A6000 GPU for less than
half a day. Across 9 diverse evaluation datasets as well as the MMLU benchmark,
the resulting model compares favorably to publicly available base models of
1B-2B size, some of which have been trained using 50-1000 times more tokens.
  We investigate Inheritune in a slightly different setting where we train
small LMs utilizing larger LMs and their full pre-training dataset. Here we
show that smaller LMs trained utilizing some of the layers of GPT2-medium
(355M) and GPT-2-large (770M) can effectively match the val loss of their
bigger counterparts when trained from scratch for the same number of training
steps on OpenWebText dataset with 9B tokens. We analyze our recipe with
extensive experiments and demonstrate it efficacy on diverse settings. Our code
is available at https://github.com/sanyalsunny111/LLM-Inheritune.
MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based
  Monocular Guidance
Authors: name: Yuqun Wu, name: Jae Yong Lee, name: Chuhang Zou, name: Shenlong Wang, name: Derek Hoiem
Published: 2024-04-12 05:43:10

The latest regularized Neural Radiance Field (NeRF) approaches produce poor
geometry and view extrapolation for multiview stereo (MVS) benchmarks such as
ETH3D. In this paper, we aim to create 3D models that provide accurate geometry
and view synthesis, partially closing the large geometric performance gap
between NeRF and traditional MVS methods. We propose a patch-based approach
that effectively leverages monocular surface normal and relative depth
predictions. The patch-based ray sampling also enables the appearance
regularization of normalized cross-correlation (NCC) and structural similarity
(SSIM) between randomly sampled virtual and training views. We further show
that "density restrictions" based on sparse structure-from-motion points can
help greatly improve geometric accuracy with a slight drop in novel view
synthesis metrics. Our experiments show 4x the performance of RegNeRF and 8x
that of FreeNeRF on average F1@2cm for ETH3D MVS benchmark, suggesting a
fruitful research direction to improve the geometric accuracy of NeRF-based
models, and sheds light on a potential future approach to enable NeRF-based
optimization to eventually outperform traditional MVS.
WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents
Authors: name: Michael Lutz, name: Arth Bohra, name: Manvel Saroyan, name: Artem Harutyunyan, name: Giovanni Campagna
Published: 2024-04-08 23:10:47

In the realm of web agent research, achieving both generalization and
accuracy remains a challenging problem. Due to high variance in website
structure, existing approaches often fail. Moreover, existing fine-tuning and
in-context learning techniques fail to generalize across multiple websites. We
introduce Wilbur, an approach that uses a differentiable ranking model and a
novel instruction synthesis technique to optimally populate a black-box large
language model's prompt with task demonstrations from previous runs. To
maximize end-to-end success rates, we also propose an intelligent backtracking
mechanism that learns and recovers from its mistakes. Finally, we show that our
ranking model can be trained on data from a generative auto-curriculum which
samples representative goals from an LLM, runs the agent, and automatically
evaluates it, with no manual annotation. Wilbur achieves state-of-the-art
results on the WebVoyager benchmark, beating text-only models by 8% overall,
and up to 36% on certain websites. On the same benchmark, Wilbur is within 5%
of a strong multi-modal model despite only receiving textual inputs, and
further analysis reveals a substantial number of failures are due to
engineering challenges of operating the web.
HGRN2: Gated Linear RNNs with State Expansion
Authors: name: Zhen Qin, name: Songlin Yang, name: Weixuan Sun, name: Xuyang Shen, name: Dong Li, name: Weigao Sun, name: Yiran Zhong
Published: 2024-04-11 16:43:03

Hierarchically gated linear RNN (HGRN,Qin et al. 2023) has demonstrated
competitive training speed and performance in language modeling, while offering
efficient inference. However, the recurrent state size of HGRN remains
relatively small, which limits its expressiveness.To address this issue,
inspired by linear attention, we introduce a simple outer-product-based state
expansion mechanism so that the recurrent state size can be significantly
enlarged without introducing any additional parameters. The linear attention
form also allows for hardware-efficient training.Our extensive experiments
verify the advantage of HGRN2 over HGRN1 in language modeling, image
classification, and Long Range Arena.Our largest 3B HGRN2 model slightly
outperforms Mamba and LLaMa Architecture Transformer for language modeling in a
controlled experiment setting; and performs competitively with many open-source
3B models in downstream evaluation while using much fewer total training
tokens.
Best Practices and Lessons Learned on Synthetic Data for Language Models
Authors: name: Ruibo Liu, name: Jerry Wei, name: Fangyu Liu, name: Chenglei Si, name: Yanzhe Zhang, name: Jinmeng Rao, name: Steven Zheng, name: Daiyi Peng, name: Diyi Yang, name: Denny Zhou, name: Andrew M. Dai
Published: 2024-04-11 06:34:17

The success of AI models relies on the availability of large, diverse, and
high-quality datasets, which can be challenging to obtain due to data scarcity,
privacy concerns, and high costs. Synthetic data has emerged as a promising
solution by generating artificial data that mimics real-world patterns. This
paper provides an overview of synthetic data research, discussing its
applications, challenges, and future directions. We present empirical evidence
from prior art to demonstrate its effectiveness and highlight the importance of
ensuring its factuality, fidelity, and unbiasedness. We emphasize the need for
responsible use of synthetic data to build more powerful, inclusive, and
trustworthy language models.
Sparse Laneformer
Authors: name: Ji Liu, name: Zifeng Zhang, name: Mingjie Lu, name: Hongyang Wei, name: Dong Li, name: Yile Xie, name: Jinzhang Peng, name: Lu Tian, name: Ashish Sirasao, name: Emad Barsoum
Published: 2024-04-11 15:00:55

Lane detection is a fundamental task in autonomous driving, and has achieved
great progress as deep learning emerges. Previous anchor-based methods often
design dense anchors, which highly depend on the training dataset and remain
fixed during inference. We analyze that dense anchors are not necessary for
lane detection, and propose a transformer-based lane detection framework based
on a sparse anchor mechanism. To this end, we generate sparse anchors with
position-aware lane queries and angle queries instead of traditional explicit
anchors. We adopt Horizontal Perceptual Attention (HPA) to aggregate the lane
features along the horizontal direction, and adopt Lane-Angle Cross Attention
(LACA) to perform interactions between lane queries and angle queries. We also
propose Lane Perceptual Attention (LPA) based on deformable cross attention to
further refine the lane predictions. Our method, named Sparse Laneformer, is
easy-to-implement and end-to-end trainable. Extensive experiments demonstrate
that Sparse Laneformer performs favorably against the state-of-the-art methods,
e.g., surpassing Laneformer by 3.0% F1 score and O2SFormer by 0.7% F1 score
with fewer MACs on CULane with the same ResNet-34 backbone.
Applying Guidance in a Limited Interval Improves Sample and Distribution
  Quality in Diffusion Models
Authors: name: Tuomas Kynkäänniemi, name: Miika Aittala, name: Tero Karras, name: Samuli Laine, name: Timo Aila, name: Jaakko Lehtinen
Published: 2024-04-11 13:16:47

Guidance is a crucial technique for extracting the best performance out of
image-generating diffusion models. Traditionally, a constant guidance weight
has been applied throughout the sampling chain of an image. We show that
guidance is clearly harmful toward the beginning of the chain (high noise
levels), largely unnecessary toward the end (low noise levels), and only
beneficial in the middle. We thus restrict it to a specific range of noise
levels, improving both the inference speed and result quality. This limited
guidance interval improves the record FID in ImageNet-512 significantly, from
1.81 to 1.40. We show that it is quantitatively and qualitatively beneficial
across different sampler parameters, network architectures, and datasets,
including the large-scale setting of Stable Diffusion XL. We thus suggest
exposing the guidance interval as a hyperparameter in all diffusion models that
use guidance.
ControlNet++: Improving Conditional Controls with Efficient Consistency
  Feedback
Authors: name: Ming Li, name: Taojiannan Yang, name: Huafeng Kuang, name: Jie Wu, name: Zhaoning Wang, name: Xuefeng Xiao, name: Chen Chen
Published: 2024-04-11 17:59:09

To enhance the controllability of text-to-image diffusion models, existing
efforts like ControlNet incorporated image-based conditional controls. In this
paper, we reveal that existing methods still face significant challenges in
generating images that align with the image conditional controls. To this end,
we propose ControlNet++, a novel approach that improves controllable generation
by explicitly optimizing pixel-level cycle consistency between generated images
and conditional controls. Specifically, for an input conditional control, we
use a pre-trained discriminative reward model to extract the corresponding
condition of the generated images, and then optimize the consistency loss
between the input conditional control and extracted condition. A
straightforward implementation would be generating images from random noises
and then calculating the consistency loss, but such an approach requires
storing gradients for multiple sampling timesteps, leading to considerable time
and memory costs. To address this, we introduce an efficient reward strategy
that deliberately disturbs the input images by adding noise, and then uses the
single-step denoised images for reward fine-tuning. This avoids the extensive
costs associated with image sampling, allowing for more efficient reward
fine-tuning. Extensive experiments show that ControlNet++ significantly
improves controllability under various conditional controls. For example, it
achieves improvements over ControlNet by 7.9% mIoU, 13.4% SSIM, and 7.6% RMSE,
respectively, for segmentation mask, line-art edge, and depth conditions.
LLoCO: Learning Long Contexts Offline
Authors: name: Sijun Tan, name: Xiuyu Li, name: Shishir Patil, name: Ziyang Wu, name: Tianjun Zhang, name: Kurt Keutzer, name: Joseph E. Gonzalez, name: Raluca Ada Popa
Published: 2024-04-11 17:57:22

Processing long contexts remains a challenge for large language models (LLMs)
due to the quadratic computational and memory overhead of the self-attention
mechanism and the substantial KV cache sizes during generation. We propose a
novel approach to address this problem by learning contexts offline through
context compression and in-domain parameter-efficient finetuning. Our method
enables an LLM to create a concise representation of the original context and
efficiently retrieve relevant information to answer questions accurately. We
introduce LLoCO, a technique that combines context compression, retrieval, and
parameter-efficient finetuning using LoRA. Our approach extends the effective
context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We
evaluate our approach on several long-context question-answering datasets,
demonstrating that LLoCO significantly outperforms in-context learning while
using 30times fewer tokens during inference. LLoCO achieves up to
7.62times speed-up and substantially reduces the cost of long document
question answering, making it a promising solution for efficient long context
processing. Our code is publicly available at
https://github.com/jeffreysijuntan/lloco.
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real
  Computer Environments
Authors: name: Tianbao Xie, name: Danyang Zhang, name: Jixuan Chen, name: Xiaochuan Li, name: Siheng Zhao, name: Ruisheng Cao, name: Toh Jing Hua, name: Zhoujun Cheng, name: Dongchan Shin, name: Fangyu Lei, name: Yitao Liu, name: Yiheng Xu, name: Shuyan Zhou, name: Silvio Savarese, name: Caiming Xiong, name: Victor Zhong, name: Tao Yu
Published: 2024-04-11 17:56:05

Autonomous agents that accomplish complex computer tasks with minimal human
interventions have the potential to transform human-computer interaction,
significantly enhancing accessibility and productivity. However, existing
benchmarks either lack an interactive environment or are limited to
environments specific to certain applications or domains, failing to reflect
the diverse and complex nature of real-world computer use, thereby limiting the
scope of tasks and agent scalability. To address this issue, we introduce
OSWorld, the first-of-its-kind scalable, real computer environment for
multimodal agents, supporting task setup, execution-based evaluation, and
interactive learning across various operating systems such as Ubuntu, Windows,
and macOS. OSWorld can serve as a unified, integrated computer environment for
assessing open-ended computer tasks that involve arbitrary applications.
Building upon OSWorld, we create a benchmark of 369 computer tasks involving
real web and desktop apps in open domains, OS file I/O, and workflows spanning
multiple applications. Each task example is derived from real-world computer
use cases and includes a detailed initial state setup configuration and a
custom execution-based evaluation script for reliable, reproducible evaluation.
Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld
reveals significant deficiencies in their ability to serve as computer
assistants. While humans can accomplish over 72.36% of the tasks, the best
model achieves only 12.24% success, primarily struggling with GUI grounding and
operational knowledge. Comprehensive analysis using OSWorld provides valuable
insights for developing multimodal generalist agents that were not possible
with previous benchmarks. Our code, environment, baseline models, and data are
publicly available at https://os-world.github.io.
Ferret-v2: An Improved Baseline for Referring and Grounding with Large
  Language Models
Authors: name: Haotian Zhang, name: Haoxuan You, name: Philipp Dufter, name: Bowen Zhang, name: Chen Chen, name: Hong-You Chen, name: Tsu-Jui Fu, name: William Yang Wang, name: Shih-Fu Chang, name: Zhe Gan, name: Yinfei Yang
Published: 2024-04-11 17:56:05

While Ferret seamlessly integrates regional understanding into the Large
Language Model (LLM) to facilitate its referring and grounding capability, it
poses certain limitations: constrained by the pre-trained fixed visual encoder
and failed to perform well on broader tasks. In this work, we unveil Ferret-v2,
a significant upgrade to Ferret, with three key designs. (1) Any resolution
grounding and referring: A flexible approach that effortlessly handles higher
image resolution, improving the model's ability to process and understand
images in greater detail. (2) Multi-granularity visual encoding: By integrating
the additional DINOv2 encoder, the model learns better and diverse underlying
contexts for global and fine-grained visual information. (3) A three-stage
training paradigm: Besides image-caption alignment, an additional stage is
proposed for high-resolution dense alignment before the final instruction
tuning. Experiments show that Ferret-v2 provides substantial improvements over
Ferret and other state-of-the-art methods, thanks to its high-resolution
scaling and fine-grained visual processing.
Rho-1: Not All Tokens Are What You Need
Authors: name: Zhenghao Lin, name: Zhibin Gou, name: Yeyun Gong, name: Xiao Liu, name: Yelong Shen, name: Ruochen Xu, name: Chen Lin, name: Yujiu Yang, name: Jian Jiao, name: Nan Duan, name: Weizhu Chen
Published: 2024-04-11 17:52:01

Previous language model pre-training methods have uniformly applied a
next-token prediction loss to all training tokens. Challenging this norm, we
posit that "Not all tokens in a corpus are equally important for language model
training". Our initial analysis delves into token-level training dynamics of
language model, revealing distinct loss patterns for different tokens.
Leveraging these insights, we introduce a new language model called Rho-1.
Unlike traditional LMs that learn to predict every next token in a corpus,
Rho-1 employs Selective Language Modeling (SLM), which selectively trains on
useful tokens that aligned with the desired distribution. This approach
involves scoring pretraining tokens using a reference model, and then training
the language model with a focused loss on tokens with higher excess loss. When
continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute
improvement in few-shot accuracy of up to 30% in 9 math tasks. After
fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and
51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the
pretraining tokens. Furthermore, when pretraining on 80B general tokens, Rho-1
achieves 6.8% average enhancement across 15 diverse tasks, increasing both
efficiency and performance of the language model pre-training.
RecurrentGemma: Moving Past Transformers for Efficient Open Language
  Models
Authors: name: Aleksandar Botev, name: Soham De, name: Samuel L Smith, name: Anushan Fernando, name: George-Cristian Muraru, name: Ruba Haroun, name: Leonard Berrada, name: Razvan Pascanu, name: Pier Giuseppe Sessa, name: Robert Dadashi, name: Léonard Hussenot, name: Johan Ferret, name: Sertan Girgin, name: Olivier Bachem, name: Alek Andreev, name: Kathleen Kenealy, name: Thomas Mesnard, name: Cassidy Hardin, name: Surya Bhupatiraju, name: Shreya Pathak, name: Laurent Sifre, name: Morgane Rivière, name: Mihir Sanjay Kale, name: Juliette Love, name: Pouya Tafti, name: Armand Joulin, name: Noah Fiedel, name: Evan Senter, name: Yutian Chen, name: Srivatsan Srinivasan, name: Guillaume Desjardins, name: David Budden, name: Arnaud Doucet, name: Sharad Vikram, name: Adam Paszke, name: Trevor Gale, name: Sebastian Borgeaud, name: Charlie Chen, name: Andy Brock, name: Antonia Paterson, name: Jenny Brennan, name: Meg Risdal, name: Raj Gundluru, name: Nesh Devanathan, name: Paul Mooney, name: Nilay Chauhan, name: Phil Culliton, name: Luiz GUStavo Martins, name: Elisa Bandy, name: David Huntsperger, name: Glenn Cameron, name: Arthur Zucker, name: Tris Warkentin, name: Ludovic Peran, name: Minh Giang, name: Zoubin Ghahramani, name: Clément Farabet, name: Koray Kavukcuoglu, name: Demis Hassabis, name: Raia Hadsell, name: Yee Whye Teh, name: Nando de Frietas
Published: 2024-04-11 15:27:22

We introduce RecurrentGemma, an open language model which uses Google's novel
Griffin architecture. Griffin combines linear recurrences with local attention
to achieve excellent performance on language. It has a fixed-sized state, which
reduces memory use and enables efficient inference on long sequences. We
provide a pre-trained model with 2B non-embedding parameters, and an
instruction tuned variant. Both models achieve comparable performance to
Gemma-2B despite being trained on fewer tokens.
Audio Dialogues: Dialogues dataset for audio and music understanding
Authors: name: Arushi Goel, name: Zhifeng Kong, name: Rafael Valle, name: Bryan Catanzaro
Published: 2024-04-11 10:08:34

Existing datasets for audio understanding primarily focus on single-turn
interactions (i.e. audio captioning, audio question answering) for describing
audio in natural language, thus limiting understanding audio via interactive
dialogue. To address this gap, we introduce Audio Dialogues: a multi-turn
dialogue dataset containing 163.8k samples for general audio sounds and music.
In addition to dialogues, Audio Dialogues also has question-answer pairs to
understand and compare multiple input audios together. Audio Dialogues
leverages a prompting-based approach and caption annotations from existing
datasets to generate multi-turn dialogues using a Large Language Model (LLM).
We evaluate existing audio-augmented large language models on our proposed
dataset to demonstrate the complexity and applicability of Audio Dialogues. Our
code for generating the dataset will be made publicly available. Detailed
prompts and generated dialogues can be found on the demo website
https://audiodialogues.github.io/.
Transferable and Principled Efficiency for Open-Vocabulary Segmentation
Authors: name: Jingxuan Xu, name: Wuyang Chen, name: Yao Zhao, name: Yunchao Wei
Published: 2024-04-11 03:08:53

Recent success of pre-trained foundation vision-language models makes
Open-Vocabulary Segmentation (OVS) possible. Despite the promising performance,
this approach introduces heavy computational overheads for two challenges: 1)
large model sizes of the backbone; 2) expensive costs during the fine-tuning.
These challenges hinder this OVS strategy from being widely applicable and
affordable in real-world scenarios. Although traditional methods such as model
compression and efficient fine-tuning can address these challenges, they often
rely on heuristics. This means that their solutions cannot be easily
transferred and necessitate re-training on different models, which comes at a
cost. In the context of efficient OVS, we target achieving performance that is
comparable to or even better than prior OVS works based on large
vision-language foundation models, by utilizing smaller models that incur lower
training costs. The core strategy is to make our efficiency principled and thus
seamlessly transferable from one OVS framework to others without further
customization. Comprehensive experiments on diverse OVS benchmarks demonstrate
our superior trade-off between segmentation accuracy and computation costs over
previous works. Our code is available on https://github.com/Xujxyang/OpenTrans
JetMoE: Reaching Llama2 Performance with 0.1M Dollars
Authors: name: Yikang Shen, name: Zhen Guo, name: Tianle Cai, name: Zengyi Qin
Published: 2024-04-11 00:52:39

Large Language Models (LLMs) have achieved remarkable results, but their
increasing resource demand has become a major obstacle to the development of
powerful and accessible super-human intelligence. This report introduces
JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens
from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its
low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B
outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the
Llama2-13B-Chat model. These results suggest that LLM training can be much more
cost-effective than generally thought. JetMoE-8B is based on an efficient
Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention
and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B
to have 8B parameters while only activating 2B for each input token, reducing
inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B
is highly open and academia-friendly, using only public datasets and training
code. All training parameters and data mixtures have been detailed in this
report to facilitate future efforts in the development of open foundation
models. This transparency aims to encourage collaboration and further
advancements in the field of accessible and efficient LLMs. The model weights
are publicly available at https://github.com/myshell-ai/JetMoE.