Orionnnn/hub-stats huggingface dataset:
Dolma: an Open Corpus of Three Trillion Tokens for Language Model
  Pretraining Research
Authors: name: Luca Soldaini, name: Rodney Kinney, name: Akshita Bhagia, name: Dustin Schwenk, name: David Atkinson, name: Russell Authur, name: Ben Bogin, name: Khyathi Chandu, name: Jennifer Dumas, name: Yanai Elazar, name: Valentin Hofmann, name: Ananya Harsh Jha, name: Sachin Kumar, name: Li Lucy, name: Xinxi Lyu, name: Nathan Lambert, name: Ian Magnusson, name: Jacob Morrison, name: Niklas Muennighoff, name: Aakanksha Naik, name: Crystal Nam, name: Matthew E. Peters, name: Abhilasha Ravichander, name: Kyle Richardson, name: Zejiang Shen, name: Emma Strubell, name: Nishant Subramani, name: Oyvind Tafjord, name: Pete Walsh, name: Luke Zettlemoyer, name: Noah A. Smith, name: Hannaneh Hajishirzi, name: Iz Beltagy, name: Dirk Groeneveld, name: Jesse Dodge, name: Kyle Lo
Published: 2024-01-31 20:29:50

Language models have become a critical technology to tackling a wide range of
natural language processing tasks, yet many details about how the
best-performing language models were developed are not reported. In particular,
information about their pretraining corpora is seldom discussed: commercial
language models rarely provide any information about their data; even open
models rarely release datasets they are trained on, or an exact recipe to
reproduce them. As a result, it is challenging to conduct certain threads of
language modeling research, such as understanding how training data impacts
model capabilities and shapes their limitations. To facilitate open research on
language model pretraining, we release Dolma, a three trillion tokens English
corpus, built from a diverse mixture of web content, scientific papers, code,
public-domain books, social media, and encyclopedic materials. In addition, we
open source our data curation toolkit to enable further experimentation and
reproduction of our work. In this report, we document Dolma, including its
design principles, details about its construction, and a summary of its
contents. We interleave this report with analyses and experimental results from
training language models on intermediate states of Dolma to share what we have
learned about important data curation practices, including the role of content
or quality filters, deduplication, and multi-source mixing. Dolma has been used
to train OLMo, a state-of-the-art, open language model and framework designed
to build and study the science of language modeling.
AnimateLCM: Accelerating the Animation of Personalized Diffusion Models
  and Adapters with Decoupled Consistency Learning
Authors: name: Fu-Yun Wang, name: Zhaoyang Huang, name: Xiaoyu Shi, name: Weikang Bian, name: Guanglu Song, name: Yu Liu, name: Hongsheng Li
Published: 2024-02-01 16:58:11

Video diffusion models has been gaining increasing attention for its ability
to produce videos that are both coherent and of high fidelity. However, the
iterative denoising process makes it computationally intensive and
time-consuming, thus limiting its applications. Inspired by the Consistency
Model (CM) that distills pretrained image diffusion models to accelerate the
sampling with minimal steps and its successful extension Latent Consistency
Model (LCM) on conditional image generation, we propose AnimateLCM, allowing
for high-fidelity video generation within minimal steps. Instead of directly
conducting consistency learning on the raw video dataset, we propose a
decoupled consistency learning strategy that decouples the distillation of
image generation priors and motion generation priors, which improves the
training efficiency and enhance the generation visual quality. Additionally, to
enable the combination of plug-and-play adapters in stable diffusion community
to achieve various functions (e.g., ControlNet for controllable generation). we
propose an efficient strategy to adapt existing adapters to our distilled
text-conditioned video consistency model or train adapters from scratch without
harming the sampling speed. We validate the proposed strategy in
image-conditioned video generation and layout-conditioned video generation, all
achieving top-performing results. Experimental results validate the
effectiveness of our proposed method. Code and weights will be made public.
More details are available at https://github.com/G-U-N/AnimateLCM.
Advances in 3D Generation: A Survey
Authors: name: Xiaoyu Li, name: Qi Zhang, name: Di Kang, name: Weihao Cheng, name: Yiming Gao, name: Jingbo Zhang, name: Zhihao Liang, name: Jing Liao, name: Yan-Pei Cao, name: Ying Shan
Published: 2024-01-31 13:06:48

Generating 3D models lies at the core of computer graphics and has been the
focus of decades of research. With the emergence of advanced neural
representations and generative models, the field of 3D content generation is
developing rapidly, enabling the creation of increasingly high-quality and
diverse 3D models. The rapid growth of this field makes it difficult to stay
abreast of all recent developments. In this survey, we aim to introduce the
fundamental methodologies of 3D generation methods and establish a structured
roadmap, encompassing 3D representation, generation methods, datasets, and
corresponding applications. Specifically, we introduce the 3D representations
that serve as the backbone for 3D generation. Furthermore, we provide a
comprehensive overview of the rapidly growing literature on generation methods,
categorized by the type of algorithmic paradigms, including feedforward
generation, optimization-based generation, procedural generation, and
generative novel view synthesis. Lastly, we discuss available datasets,
applications, and open challenges. We hope this survey will help readers
explore this exciting topic and foster further advancements in the field of 3D
content generation.
ReplaceAnything3D:Text-Guided 3D Scene Editing with Compositional Neural
  Radiance Fields
Authors: name: Edward Bartrum, name: Thu Nguyen-Phuoc, name: Chris Xie, name: Zhengqin Li, name: Numair Khan, name: Armen Avetisyan, name: Douglas Lanman, name: Lei Xiao
Published: 2024-01-31 15:02:26

We introduce ReplaceAnything3D model (RAM3D), a novel text-guided 3D scene
editing method that enables the replacement of specific objects within a scene.
Given multi-view images of a scene, a text prompt describing the object to
replace, and a text prompt describing the new object, our Erase-and-Replace
approach can effectively swap objects in the scene with newly generated content
while maintaining 3D consistency across multiple viewpoints. We demonstrate the
versatility of ReplaceAnything3D by applying it to various realistic 3D scenes,
showcasing results of modified foreground objects that are well-integrated with
the rest of the scene without affecting its overall integrity.
Anything in Any Scene: Photorealistic Video Object Insertion
Authors: name: Chen Bai, name: Zeman Shao, name: Guoxiang Zhang, name: Di Liang, name: Jie Yang, name: Zhuorui Zhang, name: Yujian Guo, name: Chengzhang Zhong, name: Yiqiao Qiu, name: Zhendong Wang, name: Yichen Guan, name: Xiaoyin Zheng, name: Tao Wang, name: Cheng Lu
Published: 2024-01-30 23:54:43

Realistic video simulation has shown significant potential across diverse
applications, from virtual reality to film production. This is particularly
true for scenarios where capturing videos in real-world settings is either
impractical or expensive. Existing approaches in video simulation often fail to
accurately model the lighting environment, represent the object geometry, or
achieve high levels of photorealism. In this paper, we propose Anything in Any
Scene, a novel and generic framework for realistic video simulation that
seamlessly inserts any object into an existing dynamic video with a strong
emphasis on physical realism. Our proposed general framework encompasses three
key processes: 1) integrating a realistic object into a given scene video with
proper placement to ensure geometric realism; 2) estimating the sky and
environmental lighting distribution and simulating realistic shadows to enhance
the light realism; 3) employing a style transfer network that refines the final
video output to maximize photorealism. We experimentally demonstrate that
Anything in Any Scene framework produces simulated videos of great geometric
realism, lighting realism, and photorealism. By significantly mitigating the
challenges associated with video data generation, our framework offers an
efficient and cost-effective solution for acquiring high-quality videos.
Furthermore, its applications extend well beyond video data augmentation,
showing promising potential in virtual reality, video editing, and various
other video-centric applications. Please check our project website
https://anythinginanyscene.github.io for access to our project code and more
high-resolution video results.
CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting
Authors: name: Jiezhi Yang, name: Khushi Desai, name: Charles Packer, name: Harshil Bhatia, name: Nicholas Rhinehart, name: Rowan McAllister, name: Joseph Gonzalez
Published: 2024-01-31 18:56:09

We propose CARFF: Conditional Auto-encoded Radiance Field for 3D Scene
Forecasting, a method for predicting future 3D scenes given past observations,
such as 2D ego-centric images. Our method maps an image to a distribution over
plausible 3D latent scene configurations using a probabilistic encoder, and
predicts the evolution of the hypothesized scenes through time. Our latent
scene representation conditions a global Neural Radiance Field (NeRF) to
represent a 3D scene model, which enables explainable predictions and
straightforward downstream applications. This approach extends beyond previous
neural rendering work by considering complex scenarios of uncertainty in
environmental states and dynamics. We employ a two-stage training of
Pose-Conditional-VAE and NeRF to learn 3D representations. Additionally, we
auto-regressively predict latent scene representations as a partially
observable Markov decision process, utilizing a mixture density network. We
demonstrate the utility of our method in realistic scenarios using the CARLA
driving simulator, where CARFF can be used to enable efficient trajectory and
contingency planning in complex multi-agent autonomous driving scenarios
involving visual occlusions.
LongAlign: A Recipe for Long Context Alignment of Large Language Models
Authors: name: Yushi Bai, name: Xin Lv, name: Jiajie Zhang, name: Yuze He, name: Ji Qi, name: Lei Hou, name: Jie Tang, name: Yuxiao Dong, name: Juanzi Li
Published: 2024-01-31 18:29:39

Extending large language models to effectively handle long contexts requires
instruction fine-tuning on input sequences of similar length. To address this,
we present LongAlign -- a recipe of the instruction data, training, and
evaluation for long context alignment. First, we construct a long
instruction-following dataset using Self-Instruct. To ensure the data
diversity, it covers a broad range of tasks from various long context sources.
Second, we adopt the packing and sorted batching strategies to speed up
supervised fine-tuning on data with varied length distributions. Additionally,
we develop a loss weighting method to balance the contribution to the loss
across different sequences during packing training. Third, we introduce the
LongBench-Chat benchmark for evaluating instruction-following capabilities on
queries of 10k-100k in length. Experiments show that LongAlign outperforms
existing recipes for LLMs in long context tasks by up to 30\%, while also
maintaining their proficiency in handling short, generic tasks. The code, data,
and long-aligned models are open-sourced at https://github.com/THUDM/LongAlign.
Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion
Authors: name: Tairan He, name: Chong Zhang, name: Wenli Xiao, name: Guanqi He, name: Changliu Liu, name: Guanya Shi
Published: 2024-01-31 03:58:28

Legged robots navigating cluttered environments must be jointly agile for
efficient task execution and safe to avoid collisions with obstacles or humans.
Existing studies either develop conservative controllers (< 1.0 m/s) to ensure
safety, or focus on agility without considering potentially fatal collisions.
This paper introduces Agile But Safe (ABS), a learning-based control framework
that enables agile and collision-free locomotion for quadrupedal robots. ABS
involves an agile policy to execute agile motor skills amidst obstacles and a
recovery policy to prevent failures, collaboratively achieving high-speed and
collision-free navigation. The policy switch in ABS is governed by a learned
control-theoretic reach-avoid value network, which also guides the recovery
policy as an objective function, thereby safeguarding the robot in a closed
loop. The training process involves the learning of the agile policy, the
reach-avoid value network, the recovery policy, and an exteroception
representation network, all in simulation. These trained modules can be
directly deployed in the real world with onboard sensing and computation,
leading to high-speed and collision-free navigation in confined indoor and
outdoor spaces with both static and dynamic obstacles.
Efficient Tool Use with Chain-of-Abstraction Reasoning
Authors: name: Silin Gao, name: Jane Dwivedi-Yu, name: Ping Yu, name: Xiaoqing Ellen Tan, name: Ramakanth Pasunuru, name: Olga Golovneva, name: Koustuv Sinha, name: Asli Celikyilmaz, name: Antoine Bosselut, name: Tianlu Wang
Published: 2024-01-30 21:53:30

To achieve faithful reasoning that aligns with human expectations, large
language models (LLMs) need to ground their reasoning to real-world knowledge
(e.g., web facts, math and physical rules). Tools help LLMs access this
external knowledge, but there remains challenges for fine-tuning LLM agents
(e.g., Toolformer) to invoke tools in multi-step reasoning problems, where
inter-connected tool calls require holistic and efficient tool usage planning.
  In this work, we propose a new method for LLMs to better leverage tools in
multi-step reasoning. Our method, Chain-of-Abstraction (CoA), trains LLMs to
first decode reasoning chains with abstract placeholders, and then call domain
tools to reify each reasoning chain by filling in specific knowledge. This
planning with abstract chains enables LLMs to learn more general reasoning
strategies, which are robust to shifts of domain knowledge (e.g., math results)
relevant to different reasoning questions. It also allows LLMs to perform
decoding and calling of external tools in parallel, which avoids the inference
delay caused by waiting for tool responses. In mathematical reasoning and Wiki
QA domains, we show that our method consistently outperforms previous
chain-of-thought and tool-augmented baselines on both in-distribution and
out-of-distribution test sets, with an average ~6% absolute QA accuracy
improvement. LLM agents trained with our method also show more efficient tool
use, with inference speed being on average ~1.4x faster than baseline
tool-augmented LLMs.
Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion
  Tokens
Authors: name: Jiacheng Liu, name: Sewon Min, name: Luke Zettlemoyer, name: Yejin Choi, name: Hannaneh Hajishirzi
Published: 2024-01-30 19:03:49

Are n-gram language models still relevant in this era of neural large
language models (LLMs)? Our answer is yes, and we show their values in both
text analysis and improving neural LLMs. Yet this necessitates modernizing
n-gram models in two aspects. First, we train them at the same data scale as
neural LLMs -- 1.4 trillion tokens. This is the largest n-gram model ever
built. Second, existing n-gram models use small n which hinders their
performance; we instead allow n to be arbitrarily large, by introducing a new
infty-gram LM with backoff. Instead of pre-computing n-gram count tables
(which would be very expensive), we develop an engine named infini-gram --
powered by suffix arrays -- that can compute infty-gram (as well as n-gram
with arbitrary n) probabilities with millisecond-level latency. The
infty-gram framework and infini-gram engine enable us to conduct many novel
and interesting analyses of human-written and machine-generated text: we find
that the infty-gram LM has fairly high accuracy for next-token prediction
(47%), and can complement neural LLMs to greatly reduce their language modeling
perplexities. When analyzing machine-generated text, we also observe
irregularities in the machine--infty-gram agreement level with respect to
the suffix length, which indicates deficiencies in neural LLM pretraining and
the positional embeddings of Transformers. We open-source our infini-gram
engine in the hopes of enabling more study on how to best use verbatim
information retrieved from large text corpora.
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Authors: name: Parth Sarthi, name: Salman Abdullah, name: Aditi Tuli, name: Shubh Khanna, name: Anna Goldie, name: Christopher D. Manning
Published: 2024-01-31 18:30:21

Retrieval-augmented language models can better adapt to changes in world
state and incorporate long-tail knowledge. However, most existing methods
retrieve only short contiguous chunks from a retrieval corpus, limiting
holistic understanding of the overall document context. We introduce the novel
approach of recursively embedding, clustering, and summarizing chunks of text,
constructing a tree with differing levels of summarization from the bottom up.
At inference time, our RAPTOR model retrieves from this tree, integrating
information across lengthy documents at different levels of abstraction.
Controlled experiments show that retrieval with recursive summaries offers
significant improvements over traditional retrieval-augmented LMs on several
tasks. On question-answering tasks that involve complex, multi-step reasoning,
we show state-of-the-art results; for example, by coupling RAPTOR retrieval
with the use of GPT-4, we can improve the best performance on the QuALITY
benchmark by 20% in absolute accuracy.
Scavenging Hyena: Distilling Transformers into Long Convolution Models
Authors: name: Tokiniaina Raharison Ralambomihanta, name: Shahrad Mohammadzadeh, name: Mohammad Sami Nur Islam, name: Wassim Jabbour, name: Laurence Liang
Published: 2024-01-31 03:39:07

The rapid evolution of Large Language Models (LLMs), epitomized by
architectures like GPT-4, has reshaped the landscape of natural language
processing. This paper introduces a pioneering approach to address the
efficiency concerns associated with LLM pre-training, proposing the use of
knowledge distillation for cross-architecture transfer. Leveraging insights
from the efficient Hyena mechanism, our method replaces attention heads in
transformer models by Hyena, offering a cost-effective alternative to
traditional pre-training while confronting the challenge of processing long
contextual information, inherent in quadratic attention mechanisms. Unlike
conventional compression-focused methods, our technique not only enhances
inference speed but also surpasses pre-training in terms of both accuracy and
efficiency. In the era of evolving LLMs, our work contributes to the pursuit of
sustainable AI solutions, striking a balance between computational power and
environmental impact.
StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis
Authors: name: Zecheng Tang, name: Chenfei Wu, name: Zekai Zhang, name: Mingheng Ni, name: Shengming Yin, name: Yu Liu, name: Zhengyuan Yang, name: Lijuan Wang, name: Zicheng Liu, name: Juntao Li, name: Nan Duan
Published: 2024-01-30 15:20:26

To leverage LLMs for visual synthesis, traditional methods convert raster
image information into discrete grid tokens through specialized visual modules,
while disrupting the model's ability to capture the true semantic
representation of visual scenes. This paper posits that an alternative
representation of images, vector graphics, can effectively surmount this
limitation by enabling a more natural and semantically coherent segmentation of
the image information. Thus, we introduce StrokeNUWA, a pioneering work
exploring a better visual representation ''stroke tokens'' on vector graphics,
which is inherently visual semantics rich, naturally compatible with LLMs, and
highly compressed. Equipped with stroke tokens, StrokeNUWA can significantly
surpass traditional LLM-based and optimization-based methods across various
metrics in the vector graphic generation task. Besides, StrokeNUWA achieves up
to a 94x speedup in inference over the speed of prior methods with an
exceptional SVG code compression ratio of 6.9%.
Proactive Detection of Voice Cloning with Localized Watermarking
Authors: name: Robin San Roman, name: Pierre Fernandez, name: Alexandre Défossez, name: Teddy Furon, name: Tuan Tran, name: Hady Elsahar
Published: 2024-01-30 18:56:22

In the rapidly evolving field of speech generative models, there is a
pressing need to ensure audio authenticity against the risks of voice cloning.
We present AudioSeal, the first audio watermarking technique designed
specifically for localized detection of AI-generated speech. AudioSeal employs
a generator/detector architecture trained jointly with a localization loss to
enable localized watermark detection up to the sample level, and a novel
perceptual loss inspired by auditory masking, that enables AudioSeal to achieve
better imperceptibility. AudioSeal achieves state-of-the-art performance in
terms of robustness to real life audio manipulations and imperceptibility based
on automatic and human evaluation metrics. Additionally, AudioSeal is designed
with a fast, single-pass detector, that significantly surpasses existing models
in speed - achieving detection up to two orders of magnitude faster, making it
ideal for large-scale and real-time applications.
High-Quality Image Restoration Following Human Instructions
Authors: name: Marcos V. Conde, name: Gregor Geigle, name: Radu Timofte
Published: 2024-01-29 18:53:33

Image restoration is a fundamental problem that involves recovering a
high-quality clean image from its degraded observation. All-In-One image
restoration models can effectively restore images from various types and levels
of degradation using degradation-specific information as prompts to guide the
restoration model. In this work, we present the first approach that uses
human-written instructions to guide the image restoration model. Given natural
language prompts, our model can recover high-quality images from their degraded
counterparts, considering multiple degradation types. Our method, InstructIR,
achieves state-of-the-art results on several restoration tasks including image
denoising, deraining, deblurring, dehazing, and (low-light) image enhancement.
InstructIR improves +1dB over previous all-in-one restoration methods.
Moreover, our dataset and results represent a novel benchmark for new research
on text-guided image restoration and enhancement. Our code, datasets and models
are available at: https://github.com/mv-lab/InstructIR