Orionnnn/hub-stats huggingface dataset:
Programming Every Example: Lifting Pre-training Data Quality like
  Experts at Scale
Authors: name: Fan Zhou, name: Zengzhi Wang, name: Qian Liu, name: Junlong Li, name: Pengfei Liu
Published: 2024-09-25 17:28:13

Large language model pre-training has traditionally relied on human experts
to craft heuristics for improving the corpora quality, resulting in numerous
rules developed to date. However, these rules lack the flexibility to address
the unique characteristics of individual example effectively. Meanwhile,
applying tailored rules to every example is impractical for human experts. In
this paper, we demonstrate that even small language models, with as few as 0.3B
parameters, can exhibit substantial data refining capabilities comparable to
those of human experts. We introduce Programming Every Example (ProX), a novel
framework that treats data refinement as a programming task, enabling models to
refine corpora by generating and executing fine-grained operations, such as
string normalization, for each individual example at scale. Experimental
results show that models pre-trained on ProX-curated data outperform either
original data or data filtered by other selection methods by more than 2%
across various downstream benchmarks. Its effectiveness spans various model
sizes and pre-training corpora, including C4, RedPajama-V2, and FineWeb.
Furthermore, ProX exhibits significant potential in domain-specific continual
pre-training: without domain specific design, models trained on OpenWebMath
refined by ProX outperform human-crafted rule-based methods, improving average
accuracy by 7.6% over Mistral-7B, with 14.6% for Llama-2-7B and 20.3% for
CodeLlama-7B, all within 10B tokens to be comparable to models like Llemma-7B
trained on 200B tokens. Further analysis highlights that ProX significantly
saves training FLOPs, offering a promising path for efficient LLM
pre-training.We are open-sourcing ProX with >100B corpus, models, and sharing
all training and implementation details for reproducible research and future
innovation. Code: https://github.com/GAIR-NLP/ProX
TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans
Authors: name: Aggelina Chatziagapi, name: Bindita Chaudhuri, name: Amit Kumar, name: Rakesh Ranjan, name: Dimitris Samaras, name: Nikolaos Sarafianos
Published: 2024-09-25 06:51:57

We introduce a novel framework that learns a dynamic neural radiance field
(NeRF) for full-body talking humans from monocular videos. Prior work
represents only the body pose or the face. However, humans communicate with
their full body, combining body pose, hand gestures, as well as facial
expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network
that represents the holistic 4D human motion. Given a monocular video of a
subject, we learn corresponding modules for the body, face, and hands, that are
combined together to generate the final result. To capture complex finger
articulation, we learn an additional deformation field for the hands. Our
multi-identity representation enables simultaneous training for multiple
subjects, as well as robust animation under completely unseen poses. It can
also generalize to novel identities, given only a short video as input. We
demonstrate state-of-the-art performance for animating full-body talking
humans, with fine-grained hand articulation and facial expressions.
Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing
Authors: name: Pei Xu, name: Ruocheng Wang
Published: 2024-09-25 05:16:53

We present a novel approach to synthesize dexterous motions for physically
simulated hands in tasks that require coordination between the control of two
hands with high temporal precision. Instead of directly learning a joint policy
to control two hands, our approach performs bimanual control through
cooperative learning where each hand is treated as an individual agent. The
individual policies for each hand are first trained separately, and then
synchronized through latent space manipulation in a centralized environment to
serve as a joint policy for two-hand control. By doing so, we avoid directly
performing policy learning in the joint state-action space of two hands with
higher dimensions, greatly improving the overall training efficiency. We
demonstrate the effectiveness of our proposed approach in the challenging
guitar-playing task. The virtual guitarist trained by our approach can
synthesize motions from unstructured reference data of general guitar-playing
practice motions, and accurately play diverse rhythms with complex chord
pressing and string picking patterns based on the input guitar tabs that do not
exist in the references. Along with this paper, we provide the motion capture
data that we collected as the reference for policy training. Code is available
at: https://pei-xu.github.io/guitar.
Attention Prompting on Image for Large Vision-Language Models
Authors: name: Runpeng Yu, name: Weihao Yu, name: Xinchao Wang
Published: 2024-09-25 17:59:13

Compared with Large Language Models (LLMs), Large Vision-Language Models
(LVLMs) can also accept images as input, thus showcasing more interesting
emergent capabilities and demonstrating impressive performance on various
vision-language tasks. Motivated by text prompting in LLMs, visual prompting
has been explored to enhance LVLMs' capabilities of perceiving visual
information. However, previous visual prompting techniques solely process
visual inputs without considering text queries, limiting the models' ability to
follow text instructions to complete tasks. To fill this gap, in this work, we
propose a new prompting technique named Attention Prompting on Image, which
just simply overlays a text-query-guided attention heatmap on the original
input image and effectively enhances LVLM on various tasks. Specifically, we
generate an attention heatmap for the input image dependent on the text query
with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel
values of the original image to obtain the actual input image for the LVLM.
Extensive experiments on various vison-language benchmarks verify the
effectiveness of our technique. For example, Attention Prompting on Image
improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks,
respectively.
RRM: Robust Reward Model Training Mitigates Reward Hacking
Authors: name: Tianqi Liu, name: Wei Xiong, name: Jie Ren, name: Lichang Chen, name: Junru Wu, name: Rishabh Joshi, name: Yang Gao, name: Jiaming Shen, name: Zhen Qin, name: Tianhe Yu, name: Daniel Sohn, name: Anastasiia Makarova, name: Jeremiah Liu, name: Yuan Liu, name: Bilal Piot, name: Abe Ittycheriah, name: Aviral Kumar, name: Mohammad Saleh
Published: 2024-09-20 01:46:07

Reward models (RMs) play a pivotal role in aligning large language models
(LLMs) with human preferences. However, traditional RM training, which relies
on response pairs tied to specific prompts, struggles to disentangle
prompt-driven preferences from prompt-independent artifacts, such as response
length and format. In this work, we expose a fundamental limitation of current
RM training methods, where RMs fail to effectively distinguish between
contextual signals and irrelevant artifacts when determining preferences. To
address this, we introduce a causal framework that learns preferences
independent of these artifacts and propose a novel data augmentation technique
designed to eliminate them. Extensive experiments show that our approach
successfully filters out undesirable artifacts, yielding a more robust reward
model (RRM). Our RRM improves the performance of a pairwise reward model
trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to
84.15%. Additionally, we train two DPO policies using both the RM and RRM,
demonstrating that the RRM significantly enhances DPO-aligned policies,
improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in
AlpacaEval-2 from 33.46% to 52.49%.
Improvements to SDXL in NovelAI Diffusion V3
Authors: name: Juan Ossa, name: Eren Doğan, name: Alex Birch, name: F. Johnson
Published: 2024-09-24 11:57:12

In this technical report, we document the changes we made to SDXL in the
process of training NovelAI Diffusion V3, our state of the art anime image
generation model.
OmniBench: Towards The Future of Universal Omni-Language Models
Authors: name: Yizhi Li, name: Ge Zhang, name: Yinghao Ma, name: Ruibin Yuan, name: Kang Zhu, name: Hangyu Guo, name: Yiming Liang, name: Jiaheng Liu, name: Jian Yang, name: Siwei Wu, name: Xingwei Qu, name: Jinjie Shi, name: Xinyue Zhang, name: Zhenzhu Yang, name: Xiangzhou Wang, name: Zhaoxiang Zhang, name: Zachary Liu, name: Emmanouil Benetos, name: Wenhao Huang, name: Chenghua Lin
Published: 2024-09-23 17:59:05

Recent advancements in multimodal large language models (MLLMs) have aimed to
integrate and interpret data across diverse modalities. However, the capacity
of these models to concurrently process and reason about multiple modalities
remains inadequately explored, partly due to the lack of comprehensive
modality-wise benchmarks. We introduce OmniBench, a novel benchmark designed to
rigorously evaluate models' ability to recognize, interpret, and reason across
visual, acoustic, and textual inputs simultaneously. We define models capable
of such tri-modal processing as omni-language models (OLMs). OmniBench is
distinguished by high-quality human annotations, ensuring that accurate
responses require integrated understanding and reasoning across all three
modalities. Our main findings reveal that: i) open-source OLMs exhibit critical
limitations in instruction-following and reasoning capabilities within
tri-modal contexts; and ii) the baseline models perform poorly (below 50%
accuracy) even when provided with alternative textual representations of images
and audio. These results suggest that the ability to construct a consistent
context from text, image, and audio is often overlooked in existing MLLM
training paradigms. We advocate for future research to focus on developing more
robust tri-modal integration techniques and training strategies to enhance OLM
performance across diverse modalities. The codes and live leaderboard could be
found at https://m-a-p.ai/OmniBench.
Tabular Data Generation using Binary Diffusion
Authors: name: Vitaliy Kinakh, name: Slava Voloshynovskiy
Published: 2024-09-20 20:22:28

Generating synthetic tabular data is critical in machine learning, especially
when real data is limited or sensitive. Traditional generative models often
face challenges due to the unique characteristics of tabular data, such as
mixed data types and varied distributions, and require complex preprocessing or
large pretrained models. In this paper, we introduce a novel, lossless binary
transformation method that converts any tabular data into fixed-size binary
representations, and a corresponding new generative model called Binary
Diffusion, specifically designed for binary data. Binary Diffusion leverages
the simplicity of XOR operations for noise addition and removal and employs
binary cross-entropy loss for training. Our approach eliminates the need for
extensive preprocessing, complex noise parameter tuning, and pretraining on
large datasets. We evaluate our model on several popular tabular benchmark
datasets, demonstrating that Binary Diffusion outperforms existing
state-of-the-art models on Travel, Adult Income, and Diabetes datasets while
being significantly smaller in size.
Present and Future Generalization of Synthetic Image Detectors
Authors: name: Pablo Bernabeu-Perez, name: Enrique Lopez-Cuena, name: Dario Garcia-Gasulla
Published: 2024-09-21 12:46:17

The continued release of new and better image generation models increases the
demand for synthetic image detectors. In such a dynamic field, detectors need
to be able to generalize widely and be robust to uncontrolled alterations. The
present work is motivated by this setting, when looking at the role of time,
image transformations and data sources, for detector generalization. In these
experiments, none of the evaluated detectors is found universal, but results
indicate an ensemble could be. Experiments on data collected in the wild show
this task to be more challenging than the one defined by large-scale datasets,
pointing to a gap between experimentation and actual practice. Finally, we
observe a race equilibrium effect, where better generators lead to better
detectors, and vice versa. We hypothesize this pushes the field towards a
perpetually close race between generators and detectors.
SLIMER-IT: Zero-Shot NER on Italian Language
Authors: name: Andrew Zamai, name: Leonardo Rigutini, name: Marco Maggini, name: Andrea Zugarini
Published: 2024-09-24 09:57:25

Traditional approaches to Named Entity Recognition (NER) frame the task into
a BIO sequence labeling problem. Although these systems often excel in the
downstream task at hand, they require extensive annotated data and struggle to
generalize to out-of-distribution input domains and unseen entity types. On the
contrary, Large Language Models (LLMs) have demonstrated strong zero-shot
capabilities. While several works address Zero-Shot NER in English, little has
been done in other languages. In this paper, we define an evaluation framework
for Zero-Shot NER, applying it to the Italian language. Furthermore, we
introduce SLIMER-IT, the Italian version of SLIMER, an instruction-tuning
approach for zero-shot NER leveraging prompts enriched with definition and
guidelines. Comparisons with other state-of-the-art models, demonstrate the
superiority of SLIMER-IT on never-seen-before entity tags.
Seeing Faces in Things: A Model and Dataset for Pareidolia
Authors: name: Mark Hamilton, name: Simon Stent, name: Vasha DuTell, name: Anne Harrington, name: Jennifer Corbett, name: Ruth Rosenholtz, name: William T. Freeman
Published: 2024-09-24 14:50:21

The human visual system is well-tuned to detect faces of all shapes and
sizes. While this brings obvious survival advantages, such as a better chance
of spotting unknown predators in the bush, it also leads to spurious face
detections. ``Face pareidolia'' describes the perception of face-like structure
among otherwise random stimuli: seeing faces in coffee stains or clouds in the
sky. In this paper, we study face pareidolia from a computer vision
perspective. We present an image dataset of ``Faces in Things'', consisting of
five thousand web images with human-annotated pareidolic faces. Using this
dataset, we examine the extent to which a state-of-the-art human face detector
exhibits pareidolia, and find a significant behavioral gap between humans and
machines. We find that the evolutionary need for humans to detect animal faces,
as well as human faces, may explain some of this gap. Finally, we propose a
simple statistical model of pareidolia in images. Through studies on human
subjects and our pareidolic face detectors we confirm a key prediction of our
model regarding what image conditions are most likely to induce pareidolia.
Dataset and Website: https://aka.ms/faces-in-things
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of
  Experts
Authors: name: Xiaoming Shi, name: Shiyu Wang, name: Yuqi Nie, name: Dianqi Li, name: Zhou Ye, name: Qingsong Wen, name: Ming Jin
Published: 2024-09-24 12:42:18

Deep learning for time series forecasting has seen significant advancements
over the past decades. However, despite the success of large-scale pre-training
in language and vision domains, pre-trained time series models remain limited
in scale and operate at a high cost, hindering the development of larger
capable forecasting models in real-world applications. In response, we
introduce Time-MoE, a scalable and unified architecture designed to pre-train
larger, more capable forecasting foundation models while reducing inference
costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE
enhances computational efficiency by activating only a subset of networks for
each prediction, reducing computational load while maintaining high model
capacity. This allows Time-MoE to scale effectively without a corresponding
increase in inference costs. Time-MoE comprises a family of decoder-only
transformer models that operate in an auto-regressive manner and support
flexible forecasting horizons with varying input context lengths. We
pre-trained these models on our newly introduced large-scale data Time-300B,
which spans over 9 domains and encompassing over 300 billion time points. For
the first time, we scaled a time series foundation model up to 2.4 billion
parameters, achieving significantly improved forecasting precision. Our results
validate the applicability of scaling laws for training tokens and model size
in the context of time series forecasting. Compared to dense models with the
same number of activated parameters or equivalent computation budgets, our
models consistently outperform them by large margin. These advancements
position Time-MoE as a state-of-the-art solution for tackling real-world time
series forecasting challenges with superior capability, efficiency, and
flexibility.
EuroLLM: Multilingual Language Models for Europe
Authors: name: Pedro Henrique Martins, name: Patrick Fernandes, name: João Alves, name: Nuno M. Guerreiro, name: Ricardo Rei, name: Duarte M. Alves, name: José Pombal, name: Amin Farajian, name: Manuel Faysse, name: Mateusz Klimaszewski, name: Pierre Colombo, name: Barry Haddow, name: José G. C. de Souza, name: Alexandra Birch, name: André F. T. Martins
Published: 2024-09-24 16:51:36

The quality of open-weight LLMs has seen significant improvement, yet they
remain predominantly focused on English. In this paper, we introduce the
EuroLLM project, aimed at developing a suite of open-weight multilingual LLMs
capable of understanding and generating text in all official European Union
languages, as well as several additional relevant languages. We outline the
progress made to date, detailing our data collection and filtering process, the
development of scaling laws, the creation of our multilingual tokenizer, and
the data mix and modeling configurations. Additionally, we release our initial
models: EuroLLM-1.7B and EuroLLM-1.7B-Instruct and report their performance on
multilingual general benchmarks and machine translation.
Reward-Robust RLHF in LLMs
Authors: name: Yuzi Yan, name: Xingzhou Lou, name: Jialian Li, name: Yiping Zhang, name: Jian Xie, name: Chao Yu, name: Yu Wang, name: Dong Yan, name: Yuan Shen
Published: 2024-09-18 02:35:41

As Large Language Models (LLMs) continue to progress toward more advanced
forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is
increasingly seen as a key pathway toward achieving Artificial General
Intelligence (AGI). However, the reliance on reward-model-based (RM-based)
alignment methods introduces significant challenges due to the inherent
instability and imperfections of Reward Models (RMs), which can lead to
critical issues such as reward hacking and misalignment with human intentions.
In this paper, we introduce a reward-robust RLHF framework aimed at addressing
these fundamental challenges, paving the way for more reliable and resilient
learning in LLMs. Our approach introduces a novel optimization objective that
carefully balances performance and robustness by incorporating Bayesian Reward
Model Ensembles (BRME) to model the uncertainty set of reward functions. This
allows the framework to integrate both nominal performance and minimum reward
signals, ensuring more stable learning even with imperfect reward models.
Empirical results demonstrate that our framework consistently outperforms
traditional RLHF across diverse benchmarks, showing improved accuracy and
long-term stability. We also provide a theoretical analysis, demonstrating that
reward-robust RLHF approaches the stability of constant reward settings, which
proves to be effective in a stochastic-case analysis. Together, these
contributions highlight the framework potential to enhance both the performance
and stability of LLM alignment with RLHF.
MaskBit: Embedding-free Image Generation via Bit Tokens
Authors: name: Mark Weber, name: Lijun Yu, name: Qihang Yu, name: Xueqing Deng, name: Xiaohui Shen, name: Daniel Cremers, name: Liang-Chieh Chen
Published: 2024-09-24 16:12:12

Masked transformer models for class-conditional image generation have become
a compelling alternative to diffusion models. Typically comprising two stages -
an initial VQGAN model for transitioning between latent space and image space,
and a subsequent Transformer model for image generation within latent space -
these frameworks offer promising avenues for image synthesis. In this study, we
present two primary contributions: Firstly, an empirical and systematic
examination of VQGANs, leading to a modernized VQGAN. Secondly, a novel
embedding-free generation network operating directly on bit tokens - a binary
quantized representation of tokens with rich semantics. The first contribution
furnishes a transparent, reproducible, and high-performing VQGAN model,
enhancing accessibility and matching the performance of current
state-of-the-art methods while revealing previously undisclosed details. The
second contribution demonstrates that embedding-free image generation using bit
tokens achieves a new state-of-the-art FID of 1.52 on the ImageNet 256x256
benchmark, with a compact generator model of mere 305M parameters.
MIMO: Controllable Character Video Synthesis with Spatial Decomposed
  Modeling
Authors: name: Yifang Men, name: Yuan Yao, name: Miaomiao Cui, name: Liefeng Bo
Published: 2024-09-24 15:00:07

Character video synthesis aims to produce realistic videos of animatable
characters within lifelike scenes. As a fundamental problem in the computer
vision and graphics community, 3D works typically require multi-view captures
for per-case training, which severely limits their applicability of modeling
arbitrary characters in a short time. Recent 2D methods break this limitation
via pre-trained diffusion models, but they struggle for pose generality and
scene interaction. To this end, we propose MIMO, a novel framework which can
not only synthesize character videos with controllable attributes (i.e.,
character, motion and scene) provided by simple user inputs, but also
simultaneously achieve advanced scalability to arbitrary characters, generality
to novel 3D motions, and applicability to interactive real-world scenes in a
unified framework. The core idea is to encode the 2D video to compact spatial
codes, considering the inherent 3D nature of video occurrence. Concretely, we
lift the 2D frame pixels into 3D using monocular depth estimators, and
decompose the video clip to three spatial components (i.e., main human,
underlying scene, and floating occlusion) in hierarchical layers based on the
3D depth. These components are further encoded to canonical identity code,
structured motion code and full scene code, which are utilized as control
signals of synthesis process. The design of spatial decomposed modeling enables
flexible user control, complex motion expression, as well as 3D-aware synthesis
for scene interactions. Experimental results demonstrate effectiveness and
robustness of the proposed method.