Orionnnn/hub-stats huggingface dataset:
Repulsive Score Distillation for Diverse Sampling of Diffusion Models
Authors: name: Nicolas Zilberstein, name: Morteza Mardani, name: Santiago Segarra
Published: 2024-06-24 14:43:02

Score distillation sampling has been pivotal for integrating diffusion models
into generation of complex visuals. Despite impressive results it suffers from
mode collapse and lack of diversity. To cope with this challenge, we leverage
the gradient flow interpretation of score distillation to propose Repulsive
Score Distillation (RSD). In particular, we propose a variational framework
based on repulsion of an ensemble of particles that promotes diversity. Using a
variational approximation that incorporates a coupling among particles, the
repulsion appears as a simple regularization that allows interaction of
particles based on their relative pairwise similarity, measured e.g., via
radial basis kernels. We design RSD for both unconstrained and constrained
sampling scenarios. For constrained sampling we focus on inverse problems in
the latent space that leads to an augmented variational formulation, that
strikes a good balance between compute, quality and diversity. Our extensive
experiments for text-to-image generation, and inverse problems demonstrate that
RSD achieves a superior trade-off between diversity and quality compared with
state-of-the-art alternatives.
Found in the Middle: Calibrating Positional Attention Bias Improves Long
  Context Utilization
Authors: name: Cheng-Yu Hsieh, name: Yung-Sung Chuang, name: Chun-Liang Li, name: Zifeng Wang, name: Long T. Le, name: Abhishek Kumar, name: James Glass, name: Alexander Ratner, name: Chen-Yu Lee, name: Ranjay Krishna, name: Tomas Pfister
Published: 2024-06-23 04:35:42

Large language models (LLMs), even when specifically trained to process long
input contexts, struggle to capture relevant information located in the middle
of their input. This phenomenon has been known as the lost-in-the-middle
problem. In this work, we make three contributions. First, we set out to
understand the factors that cause this phenomenon. In doing so, we establish a
connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs
exhibit a U-shaped attention bias where the tokens at the beginning and at the
end of its input receive higher attention, regardless of their relevance.
Second, we mitigate this positional bias through a calibration mechanism,
found-in-the-middle, that allows the model to attend to contexts faithfully
according to their relevance, even though when they are in the middle. Third,
we show found-in-the-middle not only achieves better performance in locating
relevant information within a long context, but also eventually leads to
improved retrieval-augmented generation (RAG) performance across various tasks,
outperforming existing methods by up to 15 percentage points. These findings
open up future directions in understanding LLM attention bias and its potential
consequences.
Can Few-shot Work in Long-Context? Recycling the Context to Generate
  Demonstrations
Authors: name: Arie Cattan, name: Alon Jacovi, name: Alex Fabrikant, name: Jonathan Herzig, name: Roee Aharoni, name: Hannah Rashkin, name: Dror Marcus, name: Avinatan Hassidim, name: Yossi Matias, name: Idan Szpektor, name: Avi Caciularu
Published: 2024-06-19 15:28:29

Despite recent advancements in Large Language Models (LLMs), their
performance on tasks involving long contexts remains sub-optimal. In-Context
Learning (ICL) with few-shot examples may be an appealing solution to enhance
LLM performance in this scenario; However, naively adding ICL examples with
long context introduces challenges, including substantial token overhead added
for each few-shot example and context mismatch between the demonstrations and
the target query. In this work, we propose to automatically generate few-shot
examples for long context QA tasks by recycling contexts. Specifically, given a
long input context (1-3k tokens) and a query, we generate additional
query-output pairs from the given context as few-shot examples, while
introducing the context only once. This ensures that the demonstrations are
leveraging the same context as the target query while only adding a small
number of tokens to the prompt. We further enhance each demonstration by
instructing the model to explicitly identify the relevant paragraphs before the
answer, which improves performance while providing fine-grained attribution to
the answer source. We apply our method on multiple LLMs and obtain substantial
improvements (+23\% on average across models) on various QA datasets with long
context, especially when the answer lies within the middle of the context.
Surprisingly, despite introducing only single-hop ICL examples, LLMs also
successfully generalize to multi-hop long-context QA using our approach.
Confidence Regulation Neurons in Language Models
Authors: name: Alessandro Stolfo, name: Ben Wu, name: Wes Gurnee, name: Yonatan Belinkov, name: Xingyi Song, name: Mrinmaya Sachan, name: Neel Nanda
Published: 2024-06-24 01:31:03

Despite their widespread use, the mechanisms by which large language models
(LLMs) represent and regulate uncertainty in next-token predictions remain
largely unexplored. This study investigates two critical components believed to
influence this uncertainty: the recently discovered entropy neurons and a new
set of components that we term token frequency neurons. Entropy neurons are
characterized by an unusually high weight norm and influence the final layer
normalization (LayerNorm) scale to effectively scale down the logits. Our work
shows that entropy neurons operate by writing onto an unembedding null space,
allowing them to impact the residual stream norm with minimal direct effect on
the logits themselves. We observe the presence of entropy neurons across a
range of models, up to 7 billion parameters. On the other hand, token frequency
neurons, which we discover and describe here for the first time, boost or
suppress each token's logit proportionally to its log frequency, thereby
shifting the output distribution towards or away from the unigram distribution.
Finally, we present a detailed case study where entropy neurons actively manage
confidence in the setting of induction, i.e. detecting and continuing repeated
subsequences.
Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex
  Models
Authors: name: Xinrong Zhang, name: Yingfa Chen, name: Shengding Hu, name: Xu Han, name: Zihang Xu, name: Yuanwei Xu, name: Weilin Zhao, name: Maosong Sun, name: Zhiyuan Liu
Published: 2024-06-22 03:20:10

As large language models (LLMs) increasingly permeate daily lives, there is a
growing demand for real-time interactions that mirror human conversations.
Traditional turn-based chat systems driven by LLMs prevent users from verbally
interacting with the system while it is generating responses. To overcome these
limitations, we adapt existing LLMs to duplex models so that these
LLMs can listen for users while generating output and dynamically adjust
themselves to provide users with instant feedback. % such as in response to
interruptions. Specifically, we divide the queries and responses of
conversations into several time slices and then adopt a
time-division-multiplexing (TDM) encoding-decoding strategy to
pseudo-simultaneously process these slices. Furthermore, to make LLMs
proficient enough to handle real-time conversations, we build a fine-tuning
dataset consisting of alternating time slices of queries and responses as well
as covering typical feedback types in instantaneous interactions. Our
experiments show that although the queries and responses of conversations are
segmented into incomplete slices for processing, LLMs can preserve their
original performance on standard benchmarks with a few fine-tuning steps on our
dataset. Automatic and human evaluation indicate that duplex models make
user-AI interactions more natural and human-like, and greatly improve user
satisfaction compared to vanilla LLMs. Our duplex model and dataset will be
released.
Sparser is Faster and Less is More: Efficient Sparse Attention for
  Long-Range Transformers
Authors: name: Chao Lou, name: Zixia Jia, name: Zilong Zheng, name: Kewei Tu
Published: 2024-06-24 15:55:59

Accommodating long sequences efficiently in autoregressive Transformers,
especially within an extended context window, poses significant challenges due
to the quadratic computational complexity and substantial KV memory
requirements inherent in self-attention mechanisms. In this work, we introduce
SPARSEK Attention, a novel sparse attention mechanism designed to overcome
these computational and memory obstacles while maintaining performance. Our
approach integrates a scoring network and a differentiable top-k mask operator,
SPARSEK, to select a constant number of KV pairs for each query, thereby
enabling gradient-based optimization. As a result, SPARSEK Attention offers
linear time complexity and constant memory footprint during generation.
Experimental results reveal that SPARSEK Attention outperforms previous sparse
attention methods and provides significant speed improvements during both
training and inference, particularly in language modeling and downstream tasks.
Furthermore, our method can be seamlessly integrated into pre-trained Large
Language Models (LLMs) with minimal fine-tuning, offering a practical solution
for effectively managing long-range dependencies in diverse applications.
How Many Parameters Does it Take to Change a Light Bulb? Evaluating
  Performance in Self-Play of Conversational Games as a Function of Model
  Characteristics
Authors: name: Nidhir Bhavsar, name: Jonathan Jordan, name: Sherzod Hakimov, name: David Schlangen
Published: 2024-06-20 07:17:09

What makes a good Large Language Model (LLM)? That it performs well on the
relevant benchmarks -- which hopefully measure, with some validity, the
presence of capabilities that are also challenged in real application. But what
makes the model perform well? What gives a model its abilities? We take a
recently introduced type of benchmark that is meant to challenge capabilities
in a goal-directed, agentive context through self-play of conversational games,
and analyse how performance develops as a function of model characteristics
like number of parameters, or type of training. We find that while there is a
clear relationship between number of parameters and performance, there is still
a wide spread of performance points within a given size bracket, which is to be
accounted for by training parameters such as fine-tuning data quality and
method. From a more practical angle, we also find a certain degree of
unpredictability about performance across access methods, possible due to
unexposed sampling parameters, and a, very welcome, performance stability
against at least moderate weight quantisation during inference.
AutoDetect: Towards a Unified Framework for Automated Weakness Detection
  in Large Language Models
Authors: name: Jiale Cheng, name: Yida Lu, name: Xiaotao Gu, name: Pei Ke, name: Xiao Liu, name: Yuxiao Dong, name: Hongning Wang, name: Jie Tang, name: Minlie Huang
Published: 2024-06-24 15:16:45

Although Large Language Models (LLMs) are becoming increasingly powerful,
they still exhibit significant but subtle weaknesses, such as mistakes in
instruction-following or coding tasks. As these unexpected errors could lead to
severe consequences in practical deployments, it is crucial to investigate the
limitations within LLMs systematically. Traditional benchmarking approaches
cannot thoroughly pinpoint specific model deficiencies, while manual
inspections are costly and not scalable. In this paper, we introduce a unified
framework, AutoDetect, to automatically expose weaknesses in LLMs across
various tasks. Inspired by the educational assessment process that measures
students' learning outcomes, AutoDetect consists of three LLM-powered agents:
Examiner, Questioner, and Assessor. The collaboration among these three agents
is designed to realize comprehensive and in-depth weakness identification. Our
framework demonstrates significant success in uncovering flaws, with an
identification success rate exceeding 30% in prominent models such as ChatGPT
and Claude. More importantly, these identified weaknesses can guide specific
model improvements, proving more effective than untargeted data augmentation
methods like Self-Instruct. Our approach has led to substantial enhancements in
popular LLMs, including the Llama series and Mistral-7b, boosting their
performance by over 10% across several benchmarks. Code and data are publicly
available at https://github.com/thu-coai/AutoDetect.
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls
  and Complex Instructions
Authors: name: Terry Yue Zhuo, name: Minh Chien Vu, name: Jenny Chim, name: Han Hu, name: Wenhao Yu, name: Ratnadira Widyasari, name: Imam Nur Bani Yusuf, name: Haolan Zhan, name: Junda He, name: Indraneil Paul, name: Simon Brunner, name: Chen Gong, name: Thong Hoang, name: Armel Randy Zebaze, name: Xiaoheng Hong, name: Wen-Ding Li, name: Jean Kaddour, name: Ming Xu, name: Zhihan Zhang, name: Prateek Yadav, name: Naman Jain, name: Alex Gu, name: Zhoujun Cheng, name: Jiawei Liu, name: Qian Liu, name: Zijian Wang, name: David Lo, name: Binyuan Hui, name: Niklas Muennighoff, name: Daniel Fried, name: Xiaoning Du, name: Harm de Vries, name: Leandro Von Werra
Published: 2024-06-22 15:52:04

Automated software engineering has been greatly empowered by the recent
advances in Large Language Models (LLMs) for programming. While current
benchmarks have shown that LLMs can perform various software engineering tasks
like human developers, the majority of their evaluations are limited to short
and self-contained algorithmic tasks. Solving challenging and practical
programming tasks requires the capability of utilizing diverse function calls
as tools to efficiently implement functionalities like data analysis and web
development. In addition, using multiple tools to solve a task needs
compositional reasoning by accurately understanding complex instructions.
Fulfilling both of these characteristics can pose a great challenge for LLMs.
To assess how well LLMs can solve challenging and practical programming tasks,
we introduce Bench, a benchmark that challenges LLMs to invoke multiple
function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained
programming tasks. To evaluate LLMs rigorously, each programming task
encompasses 5.6 test cases with an average branch coverage of 99%. In addition,
we propose a natural-language-oriented variant of Bench, Benchi, that
automatically transforms the original docstrings into short instructions only
with essential information. Our extensive evaluation of 60 LLMs shows that LLMs
are not yet capable of following complex instructions to use function calls
precisely, with scores up to 60%, significantly lower than the human
performance of 97%. The results underscore the need for further advancements in
this area.
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
Authors: name: Shengbang Tong, name: Ellis Brown, name: Penghao Wu, name: Sanghyun Woo, name: Manoj Middepogu, name: Sai Charitha Akula, name: Jihan Yang, name: Shusheng Yang, name: Adithya Iyer, name: Xichen Pan, name: Austin Wang, name: Rob Fergus, name: Yann LeCun, name: Saining Xie
Published: 2024-06-24 17:59:42

We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a
vision-centric approach. While stronger language models can enhance multimodal
capabilities, the design choices for vision components are often insufficiently
explored and disconnected from visual representation learning research. This
gap hinders accurate sensory grounding in real-world scenarios. Our study uses
LLMs and visual instruction tuning as an interface to evaluate various visual
representations, offering new insights into different models and architectures
-- self-supervised, strongly supervised, or combinations thereof -- based on
experiments with over 20 vision encoders. We critically examine existing MLLM
benchmarks, addressing the difficulties involved in consolidating and
interpreting results from various tasks, and introduce a new vision-centric
benchmark, CV-Bench. To further improve visual grounding, we propose the
Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that
integrates high-resolution vision features with LLMs while reducing the number
of tokens. Additionally, we discuss the curation of high-quality visual
instruction-tuning data from publicly available sources, emphasizing the
importance of data source balancing and distribution ratio. Collectively,
Cambrian-1 not only achieves state-of-the-art performance but also serves as a
comprehensive, open cookbook for instruction-tuned MLLMs. We provide model
weights, code, supporting tools, datasets, and detailed instruction-tuning and
evaluation recipes. We hope our release will inspire and accelerate
advancements in multimodal systems and visual representation learning.
OlympicArena Medal Ranks: Who Is the Most Intelligent AI So Far?
Authors: name: Zhen Huang, name: Zengzhi Wang, name: Shijie Xia, name: Pengfei Liu
Published: 2024-06-24 16:31:12

In this report, we pose the following question: Who is the most intelligent
AI model to date, as measured by the OlympicArena (an Olympic-level,
multi-discipline, multi-modal benchmark for superintelligent AI)? We
specifically focus on the most recently released models: Claude-3.5-Sonnet,
Gemini-1.5-Pro, and GPT-4o. For the first time, we propose using an Olympic
medal Table approach to rank AI models based on their comprehensive performance
across various disciplines. Empirical results reveal: (1) Claude-3.5-Sonnet
shows highly competitive overall performance over GPT-4o, even surpassing
GPT-4o on a few subjects (i.e., Physics, Chemistry, and Biology). (2)
Gemini-1.5-Pro and GPT-4V are ranked consecutively just behind GPT-4o and
Claude-3.5-Sonnet, but with a clear performance gap between them. (3) The
performance of AI models from the open-source community significantly lags
behind these proprietary models. (4) The performance of these models on this
benchmark has been less than satisfactory, indicating that we still have a long
way to go before achieving superintelligence. We remain committed to
continuously tracking and evaluating the performance of the latest powerful
models on this benchmark (available at
https://github.com/GAIR-NLP/OlympicArena).
Evaluating D-MERIT of Partial-annotation on Information Retrieval
Authors: name: Royi Rassin, name: Yaron Fairstein, name: Oren Kalinsky, name: Guy Kushilevitz, name: Nachshon Cohen, name: Alexander Libov, name: Yoav Goldberg
Published: 2024-06-23 08:24:08

Retrieval models are often evaluated on partially-annotated datasets. Each
query is mapped to a few relevant texts and the remaining corpus is assumed to
be irrelevant. As a result, models that successfully retrieve false negatives
are punished in evaluation. Unfortunately, completely annotating all texts for
every query is not resource efficient. In this work, we show that using
partially-annotated datasets in evaluation can paint a distorted picture. We
curate D-MERIT, a passage retrieval evaluation set from Wikipedia, aspiring to
contain all relevant passages for each query. Queries describe a group (e.g.,
``journals about linguistics'') and relevant passages are evidence that
entities belong to the group (e.g., a passage indicating that Language is a
journal about linguistics). We show that evaluating on a dataset containing
annotations for only a subset of the relevant passages might result in
misleading ranking of the retrieval systems and that as more relevant texts are
included in the evaluation set, the rankings converge. We propose our dataset
as a resource for evaluation and our study as a recommendation for balance
between resource-efficiency and reliable evaluation when annotating evaluation
sets for text retrieval.
Semantic Entropy Probes: Robust and Cheap Hallucination Detection in
  LLMs
Authors: name: Jannik Kossen, name: Jiatong Han, name: Muhammed Razzak, name: Lisa Schut, name: Shreshth Malik, name: Yarin Gal
Published: 2024-06-22 19:46:06

We propose semantic entropy probes (SEPs), a cheap and reliable method for
uncertainty quantification in Large Language Models (LLMs). Hallucinations,
which are plausible-sounding but factually incorrect and arbitrary model
generations, present a major challenge to the practical adoption of LLMs.
Recent work by Farquhar et al. (2024) proposes semantic entropy (SE), which can
detect hallucinations by estimating uncertainty in the space semantic meaning
for a set of model generations. However, the 5-to-10-fold increase in
computation cost associated with SE computation hinders practical adoption. To
address this, we propose SEPs, which directly approximate SE from the hidden
states of a single generation. SEPs are simple to train and do not require
sampling multiple model generations at test time, reducing the overhead of
semantic uncertainty quantification to almost zero. We show that SEPs retain
high performance for hallucination detection and generalize better to
out-of-distribution data than previous probing methods that directly predict
model accuracy. Our results across models and tasks suggest that model hidden
states capture SE, and our ablation studies give further insights into the
token positions and model layers for which this is the case.
video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models
Authors: name: Guangzhi Sun, name: Wenyi Yu, name: Changli Tang, name: Xianzhao Chen, name: Tian Tan, name: Wei Li, name: Lu Lu, name: Zejun Ma, name: Yuxuan Wang, name: Chao Zhang
Published: 2024-06-22 01:36:11

Speech understanding as an element of the more generic video understanding
using audio-visual large language models (av-LLMs) is a crucial yet
understudied aspect. This paper proposes video-SALMONN, a single end-to-end
av-LLM for video processing, which can understand not only visual frame
sequences, audio events and music, but speech as well. To obtain fine-grained
temporal information required by speech understanding, while keeping efficient
for other video elements, this paper proposes a novel multi-resolution causal
Q-Former (MRC Q-Former) structure to connect pre-trained audio-visual encoders
and the backbone large language model. Moreover, dedicated training approaches
including the diversity loss and the unpaired audio-visual mixed training
scheme are proposed to avoid frames or modality dominance. On the introduced
speech-audio-visual evaluation benchmark, video-SALMONN achieves more than 25\%
absolute accuracy improvements on the video-QA task and over 30\% absolute
accuracy improvements on audio-visual QA tasks with human speech. In addition,
video-SALMONN demonstrates remarkable video comprehension and reasoning
abilities on tasks that are unprecedented by other av-LLMs. Our training code
and model checkpoints are available at
\url{https://github.com/bytedance/SALMONN/}.
IRASim: Learning Interactive Real-Robot Action Simulators
Authors: name: Fangqi Zhu, name: Hongtao Wu, name: Song Guo, name: Yuxiao Liu, name: Chilam Cheang, name: Tao Kong
Published: 2024-06-20 17:50:16

Scalable robot learning in the real world is limited by the cost and safety
issues of real robots. In addition, rolling out robot trajectories in the real
world can be time-consuming and labor-intensive. In this paper, we propose to
learn an interactive real-robot action simulator as an alternative. We
introduce a novel method, IRASim, which leverages the power of generative
models to generate extremely realistic videos of a robot arm that executes a
given action trajectory, starting from an initial given frame. To validate the
effectiveness of our method, we create a new benchmark, IRASim Benchmark, based
on three real-robot datasets and perform extensive experiments on the
benchmark. Results show that IRASim outperforms all the baseline methods and is
more preferable in human evaluations. We hope that IRASim can serve as an
effective and scalable approach to enhance robot learning in the real world. To
promote research for generative real-robot action simulators, we open-source
code, benchmark, and checkpoints at https: //gen-irasim.github.io.