Orionnnn/hub-stats huggingface dataset:
CS-Bench: A Comprehensive Benchmark for Large Language Models towards
  Computer Science Mastery
Authors: name: Xiaoshuai Song, name: Muxi Diao, name: Guanting Dong, name: Zhengyang Wang, name: Yujia Fu, name: Runqi Qiao, name: Zhexu Wang, name: Dayuan Fu, name: Huangxuan Wu, name: Bin Liang, name: Weihao Zeng, name: Yejie Wang, name: Zhuoma GongQue, name: Jianing Yu, name: Qiuna Tan, name: Weiran Xu
Published: 2024-06-12 18:47:28

Computer Science (CS) stands as a testament to the intricacies of human
intelligence, profoundly advancing the development of artificial intelligence
and modern society. However, the current community of large language models
(LLMs) overly focuses on benchmarks for analyzing specific foundational skills
(e.g. mathematics and code generation), neglecting an all-round evaluation of
the computer science field. To bridge this gap, we introduce CS-Bench, the
first bilingual (Chinese-English) benchmark dedicated to evaluating the
performance of LLMs in computer science. CS-Bench comprises approximately 5K
meticulously curated test samples, covering 26 subfields across 4 key areas of
computer science, encompassing various task forms and divisions of knowledge
and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of
over 30 mainstream LLMs, revealing the relationship between CS performance and
model scales. We also quantitatively analyze the reasons for failures in
existing LLMs and highlight directions for improvements, including knowledge
supplementation and CS-specific reasoning. Further cross-capability experiments
show a high correlation between LLMs' capabilities in computer science and
their abilities in mathematics and coding. Moreover, expert LLMs specialized in
mathematics and coding also demonstrate strong performances in several CS
subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM
applications in the CS field and paving new avenues in assessing LLMs' diverse
reasoning capabilities. The CS-Bench data and evaluation code are available at
https://github.com/csbench/csbench.
Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
Authors: name: Bahare Fatemi, name: Mehran Kazemi, name: Anton Tsitsulin, name: Karishma Malkan, name: Jinyeong Yim, name: John Palowitch, name: Sungyong Seo, name: Jonathan Halcrow, name: Bryan Perozzi
Published: 2024-06-13 14:31:19

Large language models (LLMs) have showcased remarkable reasoning
capabilities, yet they remain susceptible to errors, particularly in temporal
reasoning tasks involving complex temporal logic. Existing research has
explored LLM performance on temporal reasoning using diverse datasets and
benchmarks. However, these studies often rely on real-world data that LLMs may
have encountered during pre-training or employ anonymization techniques that
can inadvertently introduce factual inconsistencies. In this work, we address
these limitations by introducing novel synthetic datasets specifically designed
to assess LLM temporal reasoning abilities in various scenarios. The diversity
of question types across these datasets enables systematic investigation into
the impact of the problem structure, size, question type, fact order, and other
factors on LLM performance. Our findings provide valuable insights into the
strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster
further research in this area, we are open-sourcing the datasets and evaluation
framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.
Interpreting the Weight Space of Customized Diffusion Models
Authors: name: Amil Dravid, name: Yossi Gandelsman, name: Kuan-Chieh Wang, name: Rameen Abdal, name: Gordon Wetzstein, name: Alexei A. Efros, name: Kfir Aberman
Published: 2024-06-13 17:59:56

We investigate the space of weights spanned by a large collection of
customized diffusion models. We populate this space by creating a dataset of
over 60,000 models, each of which is a base model fine-tuned to insert a
different person's visual identity. We model the underlying manifold of these
weights as a subspace, which we term weights2weights. We demonstrate three
immediate applications of this space -- sampling, editing, and inversion.
First, as each point in the space corresponds to an identity, sampling a set of
weights from it results in a model encoding a novel identity. Next, we find
linear directions in this space corresponding to semantic edits of the identity
(e.g., adding a beard). These edits persist in appearance across generated
samples. Finally, we show that inverting a single image into this space
reconstructs a realistic identity, even if the input image is out of
distribution (e.g., a painting). Our results indicate that the weight space of
fine-tuned diffusion models behaves as an interpretable latent space of
identities.
Explore the Limits of Omni-modal Pretraining at Scale
Authors: name: Yiyuan Zhang, name: Handong Li, name: Jing Liu, name: Xiangyu Yue
Published: 2024-06-13 17:59:53

We propose to build omni-modal intelligence, which is capable of
understanding any modality and learning universal representations. In specific,
we propose a scalable pretraining paradigm, named Multimodal Context (MiCo),
which can scale up the numbers of modalities and amount of data, together with
the model parameters, in the pretraining process. With MiCo, the pretrained
models show significant emergent abilities in multimodal learning, which are
evaluated on the following tasks: i) single-modality perception benchmarks of
10 different modalities, ii) 25 cross-modality understanding tasks of
retrieval, question-answering, captioning, and iii) 18 multimodal large
language model benchmarks. Our models establish 37 new records for
state-of-the-art performance. We hope that our research could contribute to the
development of omni-modal intelligence. Code and Models are at
https://github.com/invictus717/MiCo
Transformers meet Neural Algorithmic Reasoners
Authors: name: Wilfried Bounsi, name: Borja Ibarz, name: Andrew Dudzik, name: Jessica B. Hamrick, name: Larisa Markeeva, name: Alex Vitvitskyi, name: Razvan Pascanu, name: Petar Veličković
Published: 2024-06-13 16:42:06

Transformers have revolutionized machine learning with their simple yet
effective architecture. Pre-training Transformers on massive text datasets from
the Internet has led to unmatched generalization for natural language
understanding (NLU) tasks. However, such language models remain fragile when
tasked with algorithmic forms of reasoning, where computations must be precise
and robust. To address this limitation, we propose a novel approach that
combines the Transformer's language understanding with the robustness of graph
neural network (GNN)-based neural algorithmic reasoners (NARs). Such NARs
proved effective as generic solvers for algorithmic tasks, when specified in
graph form. To make their embeddings accessible to a Transformer, we propose a
hybrid architecture with a two-phase training procedure, allowing the tokens in
the language model to cross-attend to the node embeddings from the NAR. We
evaluate our resulting TransNAR model on CLRS-Text, the text-based version of
the CLRS-30 benchmark, and demonstrate significant gains over Transformer-only
models for algorithmic reasoning, both in and out of distribution.
OpenVLA: An Open-Source Vision-Language-Action Model
Authors: name: Moo Jin Kim, name: Karl Pertsch, name: Siddharth Karamcheti, name: Ted Xiao, name: Ashwin Balakrishna, name: Suraj Nair, name: Rafael Rafailov, name: Ethan Foster, name: Grace Lam, name: Pannag Sanketi, name: Quan Vuong, name: Thomas Kollar, name: Benjamin Burchfiel, name: Russ Tedrake, name: Dorsa Sadigh, name: Sergey Levine, name: Percy Liang, name: Chelsea Finn
Published: 2024-06-13 15:46:55

Large policies pretrained on a combination of Internet-scale vision-language
data and diverse robot demonstrations have the potential to change how we teach
robots new skills: rather than training new behaviors from scratch, we can
fine-tune such vision-language-action (VLA) models to obtain robust,
generalizable policies for visuomotor control. Yet, widespread adoption of VLAs
for robotics has been challenging as 1) existing VLAs are largely closed and
inaccessible to the public, and 2) prior work fails to explore methods for
efficiently fine-tuning VLAs for new tasks, a key component for adoption.
Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source
VLA trained on a diverse collection of 970k real-world robot demonstrations.
OpenVLA builds on a Llama 2 language model combined with a visual encoder that
fuses pretrained features from DINOv2 and SigLIP. As a product of the added
data diversity and new model components, OpenVLA demonstrates strong results
for generalist manipulation, outperforming closed models such as RT-2-X (55B)
by 16.5% in absolute task success rate across 29 tasks and multiple robot
embodiments, with 7x fewer parameters. We further show that we can effectively
fine-tune OpenVLA for new settings, with especially strong generalization
results in multi-task environments involving multiple objects and strong
language grounding abilities, and outperform expressive from-scratch imitation
learning methods such as Diffusion Policy by 20.4%. We also explore compute
efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned
on consumer GPUs via modern low-rank adaptation methods and served efficiently
via quantization without a hit to downstream success rate. Finally, we release
model checkpoints, fine-tuning notebooks, and our PyTorch codebase with
built-in support for training VLAs at scale on Open X-Embodiment datasets.
An Image is Worth More Than 16x16 Patches: Exploring Transformers on
  Individual Pixels
Authors: name: Duy-Kien Nguyen, name: Mahmoud Assran, name: Unnat Jain, name: Martin R. Oswald, name: Cees G. M. Snoek, name: Xinlei Chen
Published: 2024-06-13 17:59:58

This work does not introduce a new method. Instead, we present an interesting
finding that questions the necessity of the inductive bias -- locality in
modern computer vision architectures. Concretely, we find that vanilla
Transformers can operate by directly treating each individual pixel as a token
and achieve highly performant results. This is substantially different from the
popular design in Vision Transformer, which maintains the inductive bias from
ConvNets towards local neighborhoods (e.g. by treating each 16x16 patch as a
token). We mainly showcase the effectiveness of pixels-as-tokens across three
well-studied tasks in computer vision: supervised learning for object
classification, self-supervised learning via masked autoencoding, and image
generation with diffusion models. Although directly operating on individual
pixels is less computationally practical, we believe the community must be
aware of this surprising piece of knowledge when devising the next generation
of neural architectures for computer vision.
Cognitively Inspired Energy-Based World Models
Authors: name: Alexi Gladstone, name: Ganesh Nanduru, name: Md Mofijul Islam, name: Aman Chadha, name: Jundong Li, name: Tariq Iqbal
Published: 2024-06-13 06:54:37

One of the predominant methods for training world models is autoregressive
prediction in the output space of the next element of a sequence. In Natural
Language Processing (NLP), this takes the form of Large Language Models (LLMs)
predicting the next token; in Computer Vision (CV), this takes the form of
autoregressive models predicting the next frame/token/pixel. However, this
approach differs from human cognition in several respects. First, human
predictions about the future actively influence internal cognitive processes.
Second, humans naturally evaluate the plausibility of predictions regarding
future states. Based on this capability, and third, by assessing when
predictions are sufficient, humans allocate a dynamic amount of time to make a
prediction. This adaptive process is analogous to System 2 thinking in
psychology. All these capabilities are fundamental to the success of humans at
high-level reasoning and planning. Therefore, to address the limitations of
traditional autoregressive models lacking these human-like capabilities, we
introduce Energy-Based World Models (EBWM). EBWM involves training an
Energy-Based Model (EBM) to predict the compatibility of a given context and a
predicted future state. In doing so, EBWM enables models to achieve all three
facets of human cognition described. Moreover, we developed a variant of the
traditional autoregressive transformer tailored for Energy-Based models, termed
the Energy-Based Transformer (EBT). Our results demonstrate that EBWM scales
better with data and GPU Hours than traditional autoregressive transformers in
CV, and that EBWM offers promising early scaling in NLP. Consequently, this
approach offers an exciting path toward training future models capable of
System 2 thinking and intelligently searching across state spaces.
HelpSteer2: Open-source dataset for training top-performing reward
  models
Authors: name: Zhilin Wang, name: Yi Dong, name: Olivier Delalleau, name: Jiaqi Zeng, name: Gerald Shen, name: Daniel Egert, name: Jimmy J. Zhang, name: Makesh Narsimhan Sreedhar, name: Oleksii Kuchaiev
Published: 2024-06-12 22:28:08

High-quality preference datasets are essential for training reward models
that can effectively guide large language models (LLMs) in generating
high-quality responses aligned with human preferences. As LLMs become stronger
and better aligned, permissively licensed preference datasets, such as Open
Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for
reward modeling. Methods that distil preference data from proprietary LLMs such
as GPT-4 have restrictions on commercial usage imposed by model providers. To
improve upon both generated responses and attribute labeling quality, we
release HelpSteer2, a permissively licensed preference dataset (CC-BY-4.0).
Using a powerful internal base model trained on HelpSteer2, we are able to
achieve the SOTA score (92.0%) on Reward-Bench's primary dataset, outperforming
currently listed open and proprietary models, as of June 12th, 2024. Notably,
HelpSteer2 consists of only ten thousand response pairs, an order of magnitude
fewer than existing preference datasets (e.g., HH-RLHF), which makes it highly
efficient for training reward models. Our extensive experiments demonstrate
that reward models trained with HelpSteer2 are effective in aligning LLMs. In
particular, we propose SteerLM 2.0, a model alignment approach that can
effectively make use of the rich multi-attribute score predicted by our reward
models. HelpSteer2 is available at
https://huggingface.co/datasets/nvidia/HelpSteer2 and code is available at
https://github.com/NVIDIA/NeMo-Aligner
Mistral-C2F: Coarse to Fine Actor for Analytical and Reasoning
  Enhancement in RLHF and Effective-Merged LLMs
Authors: name: Chen Zheng, name: Ke Sun, name: Xun Zhou
Published: 2024-06-12 21:42:13

Despite the advances in Large Language Models (LLMs), exemplified by models
like GPT-4 and Claude, smaller-scale LLMs such as Llama and Mistral often
struggle with generating in-depth and coherent dialogues. This paper presents a
novel two-step Coarse-to-Fine Actor model to address the inherent limitations
in conversational and analytical capabilities of small-sized LLMs. Our approach
begins with the Policy-based Coarse Actor, employing a technique we term
"Continuous Maximization". The Coarse Actor establishes an enhanced,
knowledge-rich pool adept at aligning with human preference styles in analysis
and reasoning. Through the RLHF process, it employs Continuous Maximization, a
strategy that dynamically and adaptively extends the output length limit,
enabling the generation of more detailed and analytical content. Subsequently,
the Fine Actor refines this analytical content, addressing the generation of
excessively redundant information from the Coarse Actor. We introduce a
"Knowledge Residue Merger" approach, refining the content from the Coarse Actor
and merging it with an existing Instruction model to improve quality,
correctness, and reduce redundancies. We applied our methodology to the popular
Mistral model, creating Mistral-C2F, which has demonstrated exceptional
performance across 11 general language tasks and the MT-Bench Dialogue task,
outperforming similar-scale models and even larger models with 13B and 30B
parameters. Our model has significantly improved conversational and analytical
reasoning abilities.
Depth Anything V2
Authors: name: Lihe Yang, name: Bingyi Kang, name: Zilong Huang, name: Zhen Zhao, name: Xiaogang Xu, name: Jiashi Feng, name: Hengshuang Zhao
Published: 2024-06-13 17:59:56

This work presents Depth Anything V2. Without pursuing fancy techniques, we
aim to reveal crucial findings to pave the way towards building a powerful
monocular depth estimation model. Notably, compared with V1, this version
produces much finer and more robust depth predictions through three key
practices: 1) replacing all labeled real images with synthetic images, 2)
scaling up the capacity of our teacher model, and 3) teaching student models
via the bridge of large-scale pseudo-labeled real images. Compared with the
latest models built on Stable Diffusion, our models are significantly more
efficient (more than 10x faster) and more accurate. We offer models of
different scales (ranging from 25M to 1.3B params) to support extensive
scenarios. Benefiting from their strong generalization capability, we fine-tune
them with metric depth labels to obtain our metric depth models. In addition to
our models, considering the limited diversity and frequent noise in current
test sets, we construct a versatile evaluation benchmark with precise
annotations and diverse scenes to facilitate future research.
DiTFastAttn: Attention Compression for Diffusion Transformer Models
Authors: name: Zhihang Yuan, name: Pu Lu, name: Hanling Zhang, name: Xuefei Ning, name: Linfeng Zhang, name: Tianchen Zhao, name: Shengen Yan, name: Guohao Dai, name: Yu Wang
Published: 2024-06-12 18:00:08

Diffusion Transformers (DiT) excel at image and video generation but face
computational challenges due to self-attention's quadratic complexity. We
propose DiTFastAttn, a novel post-training compression method to alleviate
DiT's computational bottleneck. We identify three key redundancies in the
attention computation during DiT inference: 1. spatial redundancy, where many
attention heads focus on local information; 2. temporal redundancy, with high
similarity between neighboring steps' attention outputs; 3. conditional
redundancy, where conditional and unconditional inferences exhibit significant
similarity. To tackle these redundancies, we propose three techniques: 1.
Window Attention with Residual Caching to reduce spatial redundancy; 2.
Temporal Similarity Reduction to exploit the similarity between steps; 3.
Conditional Redundancy Elimination to skip redundant computations during
conditional generation. To demonstrate the effectiveness of DiTFastAttn, we
apply it to DiT, PixArt-Sigma for image generation tasks, and OpenSora for
video generation tasks. Evaluation results show that for image generation, our
method reduces up to 88\% of the FLOPs and achieves up to 1.6x speedup at high
resolution generation.
VCR: Visual Caption Restoration
Authors: name: Tianyu Zhang, name: Suyuchen Wang, name: Lu Li, name: Ge Zhang, name: Perouz Taslakian, name: Sai Rajeswar, name: Jie Fu, name: Bang Liu, name: Yoshua Bengio
Published: 2024-06-10 16:58:48

We introduce Visual Caption Restoration (VCR), a novel vision-language task
that challenges models to accurately restore partially obscured texts using
pixel-level hints within images. This task stems from the observation that text
embedded in images is intrinsically different from common visual elements and
natural language due to the need to align the modalities of vision, text, and
text embedded in images. While numerous works have integrated text embedded in
images into visual question-answering tasks, approaches to these tasks
generally rely on optical character recognition or masked language modeling,
thus reducing the task to mainly text-based processing. However, text-based
processing becomes ineffective in VCR as accurate text restoration depends on
the combined information from provided images, context, and subtle cues from
the tiny exposed areas of masked texts. We develop a pipeline to generate
synthetic images for the VCR task using image-caption pairs, with adjustable
caption visibility to control the task difficulty. With this pipeline, we
construct a dataset for VCR called VCR-Wiki using images with captions from
Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and
hard split variants. Our results reveal that current vision language models
significantly lag behind human performance in the VCR task, and merely
fine-tuning the models on our dataset does not lead to notable improvements. We
release VCR-Wiki and the data construction code to facilitate future research.
Simplified and Generalized Masked Diffusion for Discrete Data
Authors: name: Jiaxin Shi, name: Kehang Han, name: Zhe Wang, name: Arnaud Doucet, name: Michalis K. Titsias
Published: 2024-06-06 17:59:10

Masked (or absorbing) diffusion is actively explored as an alternative to
autoregressive models for generative modeling of discrete data. However,
existing work in this area has been hindered by unnecessarily complex model
formulations and unclear relationships between different perspectives, leading
to suboptimal parameterization, training objectives, and ad hoc adjustments to
counteract these issues. In this work, we aim to provide a simple and general
framework that unlocks the full potential of masked diffusion models. We show
that the continuous-time variational objective of masked diffusion models is a
simple weighted integral of cross-entropy losses. Our framework also enables
training generalized masked diffusion models with state-dependent masking
schedules. When evaluated by perplexity, our models trained on OpenWebText
surpass prior diffusion language models at GPT-2 scale and demonstrate superior
performance on 4 out of 5 zero-shot language modeling tasks. Furthermore, our
models vastly outperform previous discrete diffusion models on pixel-level
image modeling, achieving 2.78~(CIFAR-10) and 3.42 (ImageNet 64times64) bits
per dimension that are comparable or better than autoregressive models of
similar sizes.