Orionnnn/hub-stats huggingface dataset:
OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for
  Superintelligent AI
Authors: name: Zhen Huang, name: Zengzhi Wang, name: Shijie Xia, name: Xuefeng Li, name: Haoyang Zou, name: Ruijie Xu, name: Run-Ze Fan, name: Lyumanshan Ye, name: Ethan Chern, name: Yixin Ye, name: Yikai Zhang, name: Yuqing Yang, name: Ting Wu, name: Binjie Wang, name: Shichao Sun, name: Yang Xiao, name: Yiyuan Li, name: Fan Zhou, name: Steffi Chern, name: Yiwei Qin, name: Yan Ma, name: Jiadi Su, name: Yixiu Liu, name: Yuxiang Zheng, name: Shaoting Zhang, name: Dahua Lin, name: Yu Qiao, name: Pengfei Liu
Published: 2024-06-18 16:20:53

The evolution of Artificial Intelligence (AI) has been significantly
accelerated by advancements in Large Language Models (LLMs) and Large
Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning
abilities in problem-solving and scientific discovery (i.e., AI4Science) once
exclusive to human intellect. To comprehensively evaluate current models'
performance in cognitive reasoning abilities, we introduce OlympicArena, which
includes 11,163 bilingual problems across both text-only and interleaved
text-image modalities. These challenges encompass a wide range of disciplines
spanning seven fields and 62 international Olympic competitions, rigorously
examined for data leakage. We argue that the challenges in Olympic competition
problems are ideal for evaluating AI's cognitive reasoning due to their
complexity and interdisciplinary nature, which are essential for tackling
complex scientific challenges and facilitating discoveries. Beyond evaluating
performance across various disciplines using answer-only criteria, we conduct
detailed experiments and analyses from multiple perspectives. We delve into the
models' cognitive reasoning abilities, their performance across different
modalities, and their outcomes in process-level evaluations, which are vital
for tasks requiring complex reasoning with lengthy solutions. Our extensive
evaluations reveal that even advanced models like GPT-4o only achieve a 39.97%
overall accuracy, illustrating current AI limitations in complex reasoning and
multimodal integration. Through the OlympicArena, we aim to advance AI towards
superintelligence, equipping it to address more complex challenges in science
and beyond. We also provide a comprehensive set of resources to support AI
research, including a benchmark dataset, an open-source annotation platform, a
detailed evaluation tool, and a leaderboard with automatic submission features.
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code
  Intelligence
Authors: name: DeepSeek-AI, name: Qihao Zhu, name: Daya Guo, name: Zhihong Shao, name: Dejian Yang, name: Peiyi Wang, name: Runxin Xu, name: Y. Wu, name: Yukun Li, name: Huazuo Gao, name: Shirong Ma, name: Wangding Zeng, name: Xiao Bi, name: Zihui Gu, name: Hanwei Xu, name: Damai Dai, name: Kai Dong, name: Liyue Zhang, name: Yishi Piao, name: Zhibin Gou, name: Zhenda Xie, name: Zhewen Hao, name: Bingxuan Wang, name: Junxiao Song, name: Deli Chen, name: Xin Xie, name: Kang Guan, name: Yuxiang You, name: Aixin Liu, name: Qiushi Du, name: Wenjun Gao, name: Xuan Lu, name: Qinyu Chen, name: Yaohui Wang, name: Chengqi Deng, name: Jiashi Li, name: Chenggang Zhao, name: Chong Ruan, name: Fuli Luo, name: Wenfeng Liang
Published: 2024-06-17 13:51:35

We present DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code
language model that achieves performance comparable to GPT4-Turbo in
code-specific tasks. Specifically, DeepSeek-Coder-V2 is further pre-trained
from an intermediate checkpoint of DeepSeek-V2 with additional 6 trillion
tokens. Through this continued pre-training, DeepSeek-Coder-V2 substantially
enhances the coding and mathematical reasoning capabilities of DeepSeek-V2,
while maintaining comparable performance in general language tasks. Compared to
DeepSeek-Coder-33B, DeepSeek-Coder-V2 demonstrates significant advancements in
various aspects of code-related tasks, as well as reasoning and general
capabilities. Additionally, DeepSeek-Coder-V2 expands its support for
programming languages from 86 to 338, while extending the context length from
16K to 128K. In standard benchmark evaluations, DeepSeek-Coder-V2 achieves
superior performance compared to closed-source models such as GPT4-Turbo,
Claude 3 Opus, and Gemini 1.5 Pro in coding and math benchmarks.
TroL: Traversal of Layers for Large Language and Vision Models
Authors: name: Byung-Kwan Lee, name: Sangyun Chung, name: Chae Won Kim, name: Beomchan Park, name: Yong Man Ro
Published: 2024-06-18 03:42:00

Large language and vision models (LLVMs) have been driven by the
generalization power of large language models (LLMs) and the advent of visual
instruction tuning. Along with scaling them up directly, these models enable
LLVMs to showcase powerful vision language (VL) performances by covering
diverse tasks via natural language instructions. However, existing open-source
LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often
considered too large (e.g., 26B, 34B, and 110B parameters), having a larger
number of layers. These large models demand costly, high-end resources for both
training and inference. To address this issue, we present a new efficient LLVM
family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL),
which enables the reuse of layers in a token-wise manner. This layer traversing
technique simulates the effect of looking back and retracing the answering
stream while increasing the number of forward propagation layers without
physically adding more layers. We demonstrate that TroL employs a simple layer
traversing approach yet efficiently outperforms the open-source LLVMs with
larger model sizes and rivals the performances of the closed-source LLVMs with
substantial sizes.
Language Models are Surprisingly Fragile to Drug Names in Biomedical
  Benchmarks
Authors: name: Jack Gallifant, name: Shan Chen, name: Pedro Moreira, name: Nikolaj Munch, name: Mingye Gao, name: Jackson Pond, name: Leo Anthony Celi, name: Hugo Aerts, name: Thomas Hartvigsen, name: Danielle Bitterman
Published: 2024-06-17 20:09:24

Medical knowledge is context-dependent and requires consistent reasoning
across various natural language expressions of semantically equivalent phrases.
This is particularly crucial for drug names, where patients often use brand
names like Advil or Tylenol instead of their generic equivalents. To study
this, we create a new robustness dataset, RABBITS, to evaluate performance
differences on medical benchmarks after swapping brand and generic drug names
using physician expert annotations.
  We assess both open-source and API-based LLMs on MedQA and MedMCQA, revealing
a consistent performance drop ranging from 1-10\%. Furthermore, we identify a
potential source of this fragility as the contamination of test data in widely
used pre-training datasets. All code is accessible at
https://github.com/BittermanLab/RABBITS, and a HuggingFace leaderboard is
available at https://huggingface.co/spaces/AIM-Harvard/rabbits-leaderboard.
RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen
  Reference Content
Authors: name: Joao Monteiro, name: Pierre-Andre Noel, name: Etienne Marcotte, name: Sai Rajeswar, name: Valentina Zantedeschi, name: David Vazquez, name: Nicolas Chapados, name: Christopher Pal, name: Perouz Taslakian
Published: 2024-06-17 17:52:54

Large Language Models (LLMs) are trained on vast amounts of data, most of
which is automatically scraped from the internet. This data includes
encyclopedic documents that harbor a vast amount of general knowledge (e.g.,
Wikipedia) but also potentially overlap with benchmark datasets used for
evaluating LLMs. Consequently, evaluating models on test splits that might have
leaked into the training set is prone to misleading conclusions. To foster
sound evaluation of language models, we introduce a new test dataset named
RepLiQA, suited for question-answering and topic retrieval tasks. RepLiQA is a
collection of five splits of test sets, four of which have not been released to
the internet or exposed to LLM APIs prior to this publication. Each sample in
RepLiQA comprises (1) a reference document crafted by a human annotator and
depicting an imaginary scenario (e.g., a news article) absent from the
internet; (2) a question about the document's topic; (3) a ground-truth answer
derived directly from the information in the document; and (4) the paragraph
extracted from the reference document containing the answer. As such, accurate
answers can only be generated if a model can find relevant content within the
provided document. We run a large-scale benchmark comprising several
state-of-the-art LLMs to uncover differences in performance across models of
various types and sizes in a context-conditional language modeling setting.
Released splits of RepLiQA can be found here:
https://huggingface.co/datasets/ServiceNow/repliqa.
Deep Bayesian Active Learning for Preference Modeling in Large Language
  Models
Authors: name: Luckeciano C. Melo, name: Panagiotis Tigas, name: Alessandro Abate, name: Yarin Gal
Published: 2024-06-14 13:32:43

Leveraging human preferences for steering the behavior of Large Language
Models (LLMs) has demonstrated notable success in recent years. Nonetheless,
data selection and labeling are still a bottleneck for these systems,
particularly at large scale. Hence, selecting the most informative points for
acquiring human feedback may considerably reduce the cost of preference
labeling and unleash the further development of LLMs. Bayesian Active Learning
provides a principled framework for addressing this challenge and has
demonstrated remarkable success in diverse settings. However, previous attempts
to employ it for Preference Modeling did not meet such expectations. In this
work, we identify that naive epistemic uncertainty estimation leads to the
acquisition of redundant samples. We address this by proposing the Bayesian
Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition
policy that not only targets points of high epistemic uncertainty according to
the preference model but also seeks to maximize the entropy of the acquired
prompt distribution in the feature space spanned by the employed LLM. Notably,
our experiments demonstrate that BAL-PM requires 33% to 68% fewer preference
labels in two popular human preference datasets and exceeds previous stochastic
Bayesian acquisition policies.
Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for
  Cartoon Captioning
Authors: name: Jifan Zhang, name: Lalit Jain, name: Yang Guo, name: Jiayi Chen, name: Kuan Lok Zhou, name: Siddharth Suresh, name: Andrew Wagenmaker, name: Scott Sievert, name: Timothy Rogers, name: Kevin Jamieson, name: Robert Mankoff, name: Robert Nowak
Published: 2024-06-15 06:26:25

We present a novel multimodal preference dataset for creative tasks,
consisting of over 250 million human ratings on more than 2.2 million captions,
collected through crowdsourcing rating data for The New Yorker's weekly cartoon
caption contest over the past eight years. This unique dataset supports the
development and evaluation of multimodal large language models and
preference-based fine-tuning algorithms for humorous caption generation. We
propose novel benchmarks for judging the quality of model-generated captions,
utilizing both GPT4 and human judgments to establish ranking-based evaluation
strategies. Our experimental results highlight the limitations of current
fine-tuning methods, such as RLHF and DPO, when applied to creative tasks.
Furthermore, we demonstrate that even state-of-the-art models like GPT4 and
Claude currently underperform top human contestants in generating humorous
captions. As we conclude this extensive data collection effort, we release the
entire preference dataset to the research community, fostering further
advancements in AI humor generation and evaluation.
Unifying Multimodal Retrieval via Document Screenshot Embedding
Authors: name: Xueguang Ma, name: Sheng-Chieh Lin, name: Minghan Li, name: Wenhu Chen, name: Jimmy Lin
Published: 2024-06-17 06:27:35

In the real world, documents are organized in different formats and varied
modalities. Traditional retrieval pipelines require tailored document parsing
techniques and content extraction modules to prepare input for indexing. This
process is tedious, prone to errors, and has information loss. To this end, we
propose Document Screenshot Embedding} (DSE), a novel retrieval paradigm that
regards document screenshots as a unified input format, which does not require
any content extraction preprocess and preserves all the information in a
document (e.g., text, image and layout). DSE leverages a large vision-language
model to directly encode document screenshots into dense representations for
retrieval. To evaluate our method, we first craft the dataset of Wiki-SS, a
1.3M Wikipedia web page screenshots as the corpus to answer the questions from
the Natural Questions dataset. In such a text-intensive document retrieval
setting, DSE shows competitive effectiveness compared to other text retrieval
methods relying on parsing. For example, DSE outperforms BM25 by 17 points in
top-1 retrieval accuracy. Additionally, in a mixed-modality task of slide
retrieval, DSE significantly outperforms OCR text retrieval methods by over 15
points in nDCG@10. These experiments show that DSE is an effective document
retrieval paradigm for diverse types of documents. Model checkpoints, code, and
Wiki-SS collection will be released.
A Simple and Effective L_2 Norm-Based Strategy for KV Cache
  Compression
Authors: name: Alessio Devoto, name: Yu Zhao, name: Simone Scardapane, name: Pasquale Minervini
Published: 2024-06-17 11:35:16

The deployment of large language models (LLMs) is often hindered by the
extensive memory requirements of the Key-Value (KV) cache, especially as
context lengths increase. Existing approaches to reduce the KV cache size
involve either fine-tuning the model to learn a compression strategy or
leveraging attention scores to reduce the sequence length. We analyse the
attention distributions in decoder-only Transformers-based models and observe
that attention allocation patterns stay consistent across most layers.
Surprisingly, we find a clear correlation between the L_2 and the attention
scores over cached KV pairs, where a low L_2 of a key embedding usually leads
to a high attention score during decoding. This finding indicates that the
influence of a KV pair is potentially determined by the key embedding itself
before being queried. Based on this observation, we compress the KV cache based
on the L_2 of key embeddings. Our experimental results show that this simple
strategy can reduce the KV cache size by 50% on language modelling and
needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing
accuracy.
Pandora: Towards General World Model with Natural Language Actions and
  Video States
Authors: name: Jiannan Xiang, name: Guangyi Liu, name: Yi Gu, name: Qiyue Gao, name: Yuting Ning, name: Yuheng Zha, name: Zeyu Feng, name: Tianhua Tao, name: Shibo Hao, name: Yemin Shi, name: Zhengzhong Liu, name: Eric P. Xing, name: Zhiting Hu
Published: 2024-06-12 18:55:51

World models simulate future states of the world in response to different
actions. They facilitate interactive content creation and provides a foundation
for grounded, long-horizon reasoning. Current foundation models do not fully
meet the capabilities of general world models: large language models (LLMs) are
constrained by their reliance on language modality and their limited
understanding of the physical world, while video models lack interactive action
control over the world simulations. This paper makes a step towards building a
general world model by introducing Pandora, a hybrid autoregressive-diffusion
model that simulates world states by generating videos and allows real-time
control with free-text actions. Pandora achieves domain generality, video
consistency, and controllability through large-scale pretraining and
instruction tuning. Crucially, Pandora bypasses the cost of
training-from-scratch by integrating a pretrained LLM (7B) and a pretrained
video model, requiring only additional lightweight finetuning. We illustrate
extensive outputs by Pandora across diverse domains (indoor/outdoor,
natural/urban, human/robot, 2D/3D, etc.). The results indicate great potential
of building stronger general world models with larger-scale training.
HiddenTables & PyQTax: A Cooperative Game and Dataset For TableQA to
  Ensure Scale and Data Privacy Across a Myriad of Taxonomies
Authors: name: William Watson, name: Nicole Cho, name: Tucker Balch, name: Manuela Veloso
Published: 2024-06-16 04:53:29

A myriad of different Large Language Models (LLMs) face a common challenge in
contextually analyzing table question-answering tasks. These challenges are
engendered from (1) finite context windows for large tables, (2) multi-faceted
discrepancies amongst tokenization patterns against cell boundaries, and (3)
various limitations stemming from data confidentiality in the process of using
external models such as gpt-3.5-turbo. We propose a cooperative game dubbed
"HiddenTables" as a potential resolution to this challenge. In essence,
"HiddenTables" is played between the code-generating LLM "Solver" and the
"Oracle" which evaluates the ability of the LLM agents to solve Table QA tasks.
This game is based on natural language schemas and importantly, ensures the
security of the underlying data. We provide evidential experiments on a diverse
set of tables that demonstrate an LLM's collective inability to generalize and
perform on complex queries, handle compositional dependencies, and align
natural language to programmatic commands when concrete table schemas are
provided. Unlike encoder-based models, we have pushed the boundaries of
"HiddenTables" to not be limited by the number of rows - therefore we exhibit
improved efficiency in prompt and completion tokens. Our infrastructure has
spawned a new dataset "PyQTax" that spans across 116,671 question-table-answer
triplets and provides additional fine-grained breakdowns & labels for varying
question taxonomies. Therefore, in tandem with our academic contributions
regarding LLMs' deficiency in TableQA tasks, "HiddenTables" is a tactile
manifestation of how LLMs can interact with massive datasets while ensuring
data security and minimizing generation costs.
In-Context Editing: Learning Knowledge from Self-Induced Distributions
Authors: name: Siyuan Qi, name: Bangcheng Yang, name: Kailin Jiang, name: Xiaobo Wang, name: Jiaqi Li, name: Yifan Zhong, name: Yaodong Yang, name: Zilong Zheng
Published: 2024-06-17 04:00:04

The existing fine-tuning paradigm for language models is brittle in knowledge
editing scenarios, where the model must incorporate new information without
extensive retraining. This brittleness often results in overfitting, reduced
performance, and unnatural language generation. To address this, we propose
Consistent In-Context Editing (ICE), a novel approach that leverages the
model's in-context learning capability to tune toward a contextual distribution
rather than a one-hot target. ICE introduces a straightforward optimization
framework that includes both a target and a procedure, enhancing the robustness
and effectiveness of gradient-based tuning methods. We provide analytical
insights into ICE across four critical aspects of knowledge editing: accuracy,
locality, generalization, and linguistic quality, showing its advantages.
Experimental results across four datasets confirm the effectiveness of ICE and
demonstrate its potential for continual editing, ensuring that updated
information is incorporated while preserving the integrity of the model.
Consistency^2: Consistent and Fast 3D Painting with Latent Consistency
  Models
Authors: name: Tianfu Wang, name: Anton Obukhov, name: Konrad Schindler
Published: 2024-06-17 04:40:07

Generative 3D Painting is among the top productivity boosters in
high-resolution 3D asset management and recycling. Ever since text-to-image
models became accessible for inference on consumer hardware, the performance of
3D Painting methods has consistently improved and is currently close to
plateauing. At the core of most such models lies denoising diffusion in the
latent space, an inherently time-consuming iterative process. Multiple
techniques have been developed recently to accelerate generation and reduce
sampling iterations by orders of magnitude. Designed for 2D generative imaging,
these techniques do not come with recipes for lifting them into 3D. In this
paper, we address this shortcoming by proposing a Latent Consistency Model
(LCM) adaptation for the task at hand. We analyze the strengths and weaknesses
of the proposed model and evaluate it quantitatively and qualitatively. Based
on the Objaverse dataset samples study, our 3D painting method attains strong
preference in all evaluations. Source code is available at
https://github.com/kongdai123/consistency2.
Just How Flexible are Neural Networks in Practice?
Authors: name: Ravid Shwartz-Ziv, name: Micah Goldblum, name: Arpit Bansal, name: C. Bayan Bruss, name: Yann LeCun, name: Andrew Gordon Wilson
Published: 2024-06-17 12:24:45

It is widely believed that a neural network can fit a training set containing
at least as many samples as it has parameters, underpinning notions of
overparameterized and underparameterized models. In practice, however, we only
find solutions accessible via our training procedure, including the optimizer
and regularizers, limiting flexibility. Moreover, the exact parameterization of
the function class, built into an architecture, shapes its loss surface and
impacts the minima we find. In this work, we examine the ability of neural
networks to fit data in practice. Our findings indicate that: (1) standard
optimizers find minima where the model can only fit training sets with
significantly fewer samples than it has parameters; (2) convolutional networks
are more parameter-efficient than MLPs and ViTs, even on randomly labeled data;
(3) while stochastic training is thought to have a regularizing effect, SGD
actually finds minima that fit more training data than full-batch gradient
descent; (4) the difference in capacity to fit correctly labeled and
incorrectly labeled samples can be predictive of generalization; (5) ReLU
activation functions result in finding minima that fit more data despite being
designed to avoid vanishing and exploding gradients in deep architectures.