Orionnnn/hub-stats huggingface dataset:
Linear Transformers with Learnable Kernel Functions are Better
  In-Context Models
Authors: name: Yaroslav Aksenov, name: Nikita Balagansky, name: Sofia Maria Lo Cicero Vaina, name: Boris Shaposhnikov, name: Alexey Gorbatovski, name: Daniil Gavrilov
Published: 2024-02-16 12:44:15

Advancing the frontier of subquadratic architectures for Language Models
(LMs) is crucial in the rapidly evolving field of natural language processing.
Current innovations, including State Space Models, were initially celebrated
for surpassing Transformer performance on language modeling tasks. However,
these models have revealed deficiencies in essential In-Context Learning
capabilities - a domain where the Transformer traditionally shines. The Based
model emerged as a hybrid solution, blending a Linear Transformer with a kernel
inspired by the Taylor expansion of exponential functions, augmented by
convolutional networks. Mirroring the Transformer's in-context adeptness, it
became a strong contender in the field. In our work, we present a singular,
elegant alteration to the Based kernel that amplifies its In-Context Learning
abilities evaluated with the Multi-Query Associative Recall task and overall
language modeling process, as demonstrated on the Pile dataset.
SPAR: Personalized Content-Based Recommendation via Long Engagement
  Attention
Authors: name: Chiyu Zhang, name: Yifei Sun, name: Jun Chen, name: Jie Lei, name: Muhammad Abdul-Mageed, name: Sinong Wang, name: Rong Jin, name: Sem Park, name: Ning Yao, name: Bo Long
Published: 2024-02-16 10:36:38

Leveraging users' long engagement histories is essential for personalized
content recommendations. The success of pretrained language models (PLMs) in
NLP has led to their use in encoding user histories and candidate items,
framing content recommendations as textual semantic matching tasks. However,
existing works still struggle with processing very long user historical text
and insufficient user-item interaction. In this paper, we introduce a
content-based recommendation framework, SPAR, which effectively tackles the
challenges of holistic user interest extraction from the long user engagement
history. It achieves so by leveraging PLM, poly-attention layers and attention
sparsity mechanisms to encode user's history in a session-based manner. The
user and item side features are sufficiently fused for engagement prediction
while maintaining standalone representations for both sides, which is efficient
for practical model deployment. Moreover, we enhance user profiling by
exploiting large language model (LLM) to extract global interests from user
engagement history. Extensive experiments on two benchmark datasets demonstrate
that our framework outperforms existing state-of-the-art (SoTA) methods.
LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large
  Language Models
Authors: name: Minsuk Kahng, name: Ian Tenney, name: Mahima Pushkarna, name: Michael Xieyang Liu, name: James Wexler, name: Emily Reif, name: Krystal Kallarackal, name: Minsuk Chang, name: Michael Terry, name: Lucas Dixon
Published: 2024-02-16 09:14:49

Automatic side-by-side evaluation has emerged as a promising approach to
evaluating the quality of responses from large language models (LLMs). However,
analyzing the results from this evaluation approach raises scalability and
interpretability challenges. In this paper, we present LLM Comparator, a novel
visual analytics tool for interactively analyzing results from automatic
side-by-side evaluation. The tool supports interactive workflows for users to
understand when and why a model performs better or worse than a baseline model,
and how the responses from two models are qualitatively different. We
iteratively designed and developed the tool by closely working with researchers
and engineers at a large technology company. This paper details the user
challenges we identified, the design and development of the tool, and an
observational study with participants who regularly evaluate their models.
DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM
  Workflows
Authors: name: Ajay Patel, name: Colin Raffel, name: Chris Callison-Burch
Published: 2024-02-16 00:10:26

Large language models (LLMs) have become a dominant and important tool for
NLP researchers in a wide range of tasks. Today, many researchers use LLMs in
synthetic data generation, task evaluation, fine-tuning, distillation, and
other model-in-the-loop research workflows. However, challenges arise when
using these models that stem from their scale, their closed source nature, and
the lack of standardized tooling for these new and emerging workflows. The
rapid rise to prominence of these models and these unique challenges has had
immediate adverse impacts on open science and on the reproducibility of work
that uses them. In this paper, we introduce DataDreamer, an open source Python
library that allows researchers to write simple code to implement powerful LLM
workflows. DataDreamer also helps researchers adhere to best practices that we
propose to encourage open science and reproducibility. The library and
documentation are available at https://github.com/datadreamer-dev/DataDreamer .
LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video
  Editing
Authors: name: Bryan Wang, name: Yuliang Li, name: Zhaoyang Lv, name: Haijun Xia, name: Yan Xu, name: Raj Sodhi
Published: 2024-02-15 19:53:11

Video creation has become increasingly popular, yet the expertise and effort
required for editing often pose barriers to beginners. In this paper, we
explore the integration of large language models (LLMs) into the video editing
workflow to reduce these barriers. Our design vision is embodied in LAVE, a
novel system that provides LLM-powered agent assistance and language-augmented
editing features. LAVE automatically generates language descriptions for the
user's footage, serving as the foundation for enabling the LLM to process
videos and assist in editing tasks. When the user provides editing objectives,
the agent plans and executes relevant actions to fulfill them. Moreover, LAVE
allows users to edit videos through either the agent or direct UI manipulation,
providing flexibility and enabling manual refinement of agent actions. Our user
study, which included eight participants ranging from novices to proficient
editors, demonstrated LAVE's effectiveness. The results also shed light on user
perceptions of the proposed LLM-assisted editing paradigm and its impact on
users' creativity and sense of co-creation. Based on these findings, we propose
design implications to inform the future development of agent-assisted content
editing.
RLVF: Learning from Verbal Feedback without Overgeneralization
Authors: name: Moritz Stephan, name: Alexander Khazatsky, name: Eric Mitchell, name: Annie S Chen, name: Sheryl Hsu, name: Archit Sharma, name: Chelsea Finn
Published: 2024-02-16 18:50:24

The diversity of contexts in which large language models (LLMs) are deployed
requires the ability to modify or customize default model behaviors to
incorporate nuanced requirements and preferences. A convenient interface to
specify such model adjustments is high-level verbal feedback, such as "Don't
use emojis when drafting emails to my boss." However, while writing high-level
feedback is far simpler than collecting annotations for reinforcement learning
from human feedback (RLHF), we find that simply prompting a model with such
feedback leads to overgeneralization of the feedback to contexts where it is
not relevant. We study the problem of incorporating verbal feedback without
such overgeneralization, inspiring a new method Contextualized Critiques with
Constrained Preference Optimization (C3PO). C3PO uses a piece of high-level
feedback to generate a small synthetic preference dataset specifying how the
feedback should (and should not) be applied. It then fine-tunes the model in
accordance with the synthetic preference data while minimizing the divergence
from the original model for prompts where the feedback does not apply. Our
experimental results indicate that our approach effectively applies verbal
feedback to relevant scenarios while preserving existing behaviors for other
contexts. For both human- and GPT-4-generated high-level feedback, C3PO
effectively adheres to the given feedback comparably to in-context baselines
while reducing overgeneralization by 30%.
In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs
  Miss
Authors: name: Yuri Kuratov, name: Aydar Bulatov, name: Petr Anokhin, name: Dmitry Sorokin, name: Artyom Sorokin, name: Mikhail Burtsev
Published: 2024-02-16 16:15:01

This paper addresses the challenge of processing long documents using
generative transformer models. To evaluate different approaches, we introduce
BABILong, a new benchmark designed to assess model capabilities in extracting
and processing distributed facts within extensive texts. Our evaluation, which
includes benchmarks for GPT-4 and RAG, reveals that common methods are
effective only for sequences up to 10^4 elements. In contrast, fine-tuning
GPT-2 with recurrent memory augmentations enables it to handle tasks involving
up to 10^7 elements. This achievement marks a substantial leap, as it is by
far the longest input processed by any open neural network model to date,
demonstrating a significant improvement in the processing capabilities for long
sequences.
Large Language Models as Zero-shot Dialogue State Tracker through
  Function Calling
Authors: name: Zekun Li, name: Zhiyu Zoey Chen, name: Mike Ross, name: Patrick Huber, name: Seungwhan Moon, name: Zhaojiang Lin, name: Xin Luna Dong, name: Adithya Sagar, name: Xifeng Yan, name: Paul A. Crook
Published: 2024-02-16 06:13:18

Large language models (LLMs) are increasingly prevalent in conversational
systems due to their advanced understanding and generative capabilities in
general contexts. However, their effectiveness in task-oriented dialogues
(TOD), which requires not only response generation but also effective dialogue
state tracking (DST) within specific tasks and domains, remains less
satisfying. In this work, we propose a novel approach FnCTOD for solving DST
with LLMs through function calling. This method improves zero-shot DST,
allowing adaptation to diverse domains without extensive data collection or
model tuning. Our experimental results demonstrate that our approach achieves
exceptional performance with both modestly sized open-source and also
proprietary LLMs: with in-context prompting it enables various 7B or 13B
parameter models to surpass the previous state-of-the-art (SOTA) achieved by
ChatGPT, and improves ChatGPT's performance beating the SOTA by 5.6% Avg. JGA.
Individual model results for GPT-3.5 and GPT-4 are boosted by 4.8% and 14%,
respectively. We also show that by fine-tuning on a small collection of diverse
task-oriented dialogues, we can equip modestly sized models, specifically a 13B
parameter LLaMA2-Chat model, with function-calling capabilities and DST
performance comparable to ChatGPT while maintaining their chat capabilities. We
plan to open-source experimental code and model.
Make a Cheap Scaling: A Self-Cascade Diffusion Model for
  Higher-Resolution Adaptation
Authors: name: Lanqing Guo, name: Yingqing He, name: Haoxin Chen, name: Menghan Xia, name: Xiaodong Cun, name: Yufei Wang, name: Siyu Huang, name: Yong Zhang, name: Xintao Wang, name: Qifeng Chen, name: Ying Shan, name: Bihan Wen
Published: 2024-02-16 07:48:35

Diffusion models have proven to be highly effective in image and video
generation; however, they still face composition challenges when generating
images of varying sizes due to single-scale training data. Adapting large
pre-trained diffusion models for higher resolution demands substantial
computational and optimization resources, yet achieving a generation capability
comparable to low-resolution models remains elusive. This paper proposes a
novel self-cascade diffusion model that leverages the rich knowledge gained
from a well-trained low-resolution model for rapid adaptation to
higher-resolution image and video generation, employing either tuning-free or
cheap upsampler tuning paradigms. Integrating a sequence of multi-scale
upsampler modules, the self-cascade diffusion model can efficiently adapt to a
higher resolution, preserving the original composition and generation
capabilities. We further propose a pivot-guided noise re-schedule strategy to
speed up the inference process and improve local structural details. Compared
to full fine-tuning, our approach achieves a 5X training speed-up and requires
only an additional 0.002M tuning parameters. Extensive experiments demonstrate
that our approach can quickly adapt to higher resolution image and video
synthesis by fine-tuning for just 10k steps, with virtually no additional
inference time.
Hierarchical State Space Models for Continuous Sequence-to-Sequence
  Modeling
Authors: name: Raunaq Bhirangi, name: Chenyu Wang, name: Venkatesh Pattabiraman, name: Carmel Majidi, name: Abhinav Gupta, name: Tess Hellebrekers, name: Lerrel Pinto
Published: 2024-02-15 18:59:43

Reasoning from sequences of raw sensory data is a ubiquitous problem across
fields ranging from medical devices to robotics. These problems often involve
using long sequences of raw sensor data (e.g. magnetometers, piezoresistors) to
predict sequences of desirable physical quantities (e.g. force, inertial
measurements). While classical approaches are powerful for locally-linear
prediction problems, they often fall short when using real-world sensors. These
sensors are typically non-linear, are affected by extraneous variables (e.g.
vibration), and exhibit data-dependent drift. For many problems, the prediction
task is exacerbated by small labeled datasets since obtaining ground-truth
labels requires expensive equipment. In this work, we present Hierarchical
State-Space Models (HiSS), a conceptually simple, new technique for continuous
sequential prediction. HiSS stacks structured state-space models on top of each
other to create a temporal hierarchy. Across six real-world sensor datasets,
from tactile-based state prediction to accelerometer-based inertial
measurement, HiSS outperforms state-of-the-art sequence models such as causal
Transformers, LSTMs, S4, and Mamba by at least 23% on MSE. Our experiments
further indicate that HiSS demonstrates efficient scaling to smaller datasets
and is compatible with existing data-filtering techniques. Code, datasets and
videos can be found on https://hiss-csp.github.io.
Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversion
Authors: name: Hila Manor, name: Tomer Michaeli
Published: 2024-02-15 15:17:26

Editing signals using large pre-trained models, in a zero-shot manner, has
recently seen rapid advancements in the image domain. However, this wave has
yet to reach the audio domain. In this paper, we explore two zero-shot editing
techniques for audio signals, which use DDPM inversion on pre-trained diffusion
models. The first, adopted from the image domain, allows text-based editing.
The second, is a novel approach for discovering semantically meaningful editing
directions without supervision. When applied to music signals, this method
exposes a range of musically interesting modifications, from controlling the
participation of specific instruments to improvisations on the melody. Samples
can be found on our examples page in https://hilamanor.github.io/AudioEditing/
and code can be found in https://github.com/hilamanor/AudioEditing/ .
DreamMatcher: Appearance Matching Self-Attention for
  Semantically-Consistent Text-to-Image Personalization
Authors: name: Jisu Nam, name: Heesu Kim, name: DongJae Lee, name: Siyoon Jin, name: Seungryong Kim, name: Seunggyu Chang
Published: 2024-02-15 09:21:16

The objective of text-to-image (T2I) personalization is to customize a
diffusion model to a user-provided reference concept, generating diverse images
of the concept aligned with the target prompts. Conventional methods
representing the reference concepts using unique text embeddings often fail to
accurately mimic the appearance of the reference. To address this, one solution
may be explicitly conditioning the reference images into the target denoising
process, known as key-value replacement. However, prior works are constrained
to local editing since they disrupt the structure path of the pre-trained T2I
model. To overcome this, we propose a novel plug-in method, called
DreamMatcher, which reformulates T2I personalization as semantic matching.
Specifically, DreamMatcher replaces the target values with reference values
aligned by semantic matching, while leaving the structure path unchanged to
preserve the versatile capability of pre-trained T2I models for generating
diverse structures. We also introduce a semantic-consistent masking strategy to
isolate the personalized concept from irrelevant regions introduced by the
target prompts. Compatible with existing T2I models, DreamMatcher shows
significant improvements in complex scenarios. Intensive analyses demonstrate
the effectiveness of our approach.
GES: Generalized Exponential Splatting for Efficient Radiance Field
  Rendering
Authors: name: Abdullah Hamdi, name: Luke Melas-Kyriazi, name: Guocheng Qian, name: Jinjie Mai, name: Ruoshi Liu, name: Carl Vondrick, name: Bernard Ghanem, name: Andrea Vedaldi
Published: 2024-02-15 17:32:50

Advancements in 3D Gaussian Splatting have significantly accelerated 3D
reconstruction and generation. However, it may require a large number of
Gaussians, which creates a substantial memory footprint. This paper introduces
GES (Generalized Exponential Splatting), a novel representation that employs
Generalized Exponential Function (GEF) to model 3D scenes, requiring far fewer
particles to represent a scene and thus significantly outperforming Gaussian
Splatting methods in efficiency with a plug-and-play replacement ability for
Gaussian-based utilities. GES is validated theoretically and empirically in
both principled 1D setup and realistic 3D scenes.
  It is shown to represent signals with sharp edges more accurately, which are
typically challenging for Gaussians due to their inherent low-pass
characteristics. Our empirical analysis demonstrates that GEF outperforms
Gaussians in fitting natural-occurring signals (e.g. squares, triangles, and
parabolic signals), thereby reducing the need for extensive splitting
operations that increase the memory footprint of Gaussian Splatting. With the
aid of a frequency-modulated loss, GES achieves competitive performance in
novel-view synthesis benchmarks while requiring less than half the memory
storage of Gaussian Splatting and increasing the rendering speed by up to 39%.
The code is available on the project website https://abdullahamdi.com/ges .
Generative Representational Instruction Tuning
Authors: name: Niklas Muennighoff, name: Hongjin Su, name: Liang Wang, name: Nan Yang, name: Furu Wei, name: Tao Yu, name: Amanpreet Singh, name: Douwe Kiela
Published: 2024-02-15 12:12:19

All text-based language problems can be reduced to either generation or
embedding. Current models only perform well at one or the other. We introduce
generative representational instruction tuning (GRIT) whereby a large language
model is trained to handle both generative and embedding tasks by
distinguishing between them through instructions. Compared to other open
models, our resulting GritLM 7B sets a new state of the art on the Massive Text
Embedding Benchmark (MTEB) and outperforms all models up to its size on a range
of generative tasks. By scaling up further, GritLM 8x7B outperforms all open
generative language models that we tried while still being among the best
embedding models. Notably, we find that GRIT matches training on only
generative or embedding data, thus we can unify both at no performance loss.
Among other benefits, the unification via GRIT speeds up Retrieval-Augmented
Generation (RAG) by > 60% for long documents, by no longer requiring separate
retrieval and generation models. Models, code, etc. are freely available at
https://github.com/ContextualAI/gritlm.
Chain-of-Thought Reasoning Without Prompting
Authors: name: Xuezhi Wang, name: Denny Zhou
Published: 2024-02-15 18:55:41

In enhancing the reasoning capabilities of large language models (LLMs),
prior research primarily focuses on specific prompting techniques such as
few-shot or zero-shot chain-of-thought (CoT) prompting. These methods, while
effective, often involve manually intensive prompt engineering. Our study takes
a novel approach by asking: Can LLMs reason effectively without prompting? Our
findings reveal that, intriguingly, CoT reasoning paths can be elicited from
pre-trained LLMs by simply altering the decoding process. Rather than
conventional greedy decoding, we investigate the top-k alternative tokens,
uncovering that CoT paths are frequently inherent in these sequences. This
approach not only bypasses the confounders of prompting but also allows us to
assess the LLMs' intrinsic reasoning abilities. Moreover, we observe
that the presence of a CoT in the decoding path correlates with a higher
confidence in the model's decoded answer. This confidence metric effectively
differentiates between CoT and non-CoT paths. Extensive empirical studies on
various reasoning benchmarks show that the proposed CoT-decoding substantially
outperforms the standard greedy decoding.
BitDelta: Your Fine-Tune May Only Be Worth One Bit
Authors: name: James Liu, name: Guangxuan Xiao, name: Kai Li, name: Jason D. Lee, name: Song Han, name: Tri Dao, name: Tianle Cai
Published: 2024-02-15 18:50:06

Large Language Models (LLMs) are typically trained in two phases:
pre-training on large internet-scale datasets, and fine-tuning for downstream
tasks. Given the higher computational demand of pre-training, it's intuitive to
assume that fine-tuning adds less new information to the model, and is thus
more compressible. We explore this assumption by decomposing the weights of
fine-tuned models into their pre-trained components and an additional delta. We
introduce a simple method, BitDelta, which successfully quantizes this delta
down to 1 bit without compromising performance. This interesting finding not
only highlights the potential redundancy of information added during
fine-tuning, but also has significant implications for the multi-tenant serving
and multi-tenant storage of fine-tuned models. By enabling the use of a single
high-precision base model accompanied by multiple 1-bit deltas, BitDelta
dramatically reduces GPU memory requirements by more than 10x, which can also
be translated to enhanced generation latency in multi-tenant settings. We
validate BitDelta through experiments across Llama-2 and Mistral model
families, and on models up to 70B parameters, showcasing minimal performance
degradation over all tested settings.
How to Train Data-Efficient LLMs
Authors: name: Noveen Sachdeva, name: Benjamin Coleman, name: Wang-Cheng Kang, name: Jianmo Ni, name: Lichan Hong, name: Ed H. Chi, name: James Caverlee, name: Julian McAuley, name: Derek Zhiyuan Cheng
Published: 2024-02-15 02:27:57

The training of large language models (LLMs) is expensive. In this paper, we
study data-efficient approaches for pre-training LLMs, i.e., techniques that
aim to optimize the Pareto frontier of model quality and training resource/data
consumption. We seek to understand the tradeoffs associated with data selection
routines based on (i) expensive-to-compute data-quality estimates, and (ii)
maximization of coverage and diversity-based measures in the feature space. Our
first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of
instruction-tuned LLMs to directly assess the quality of a training example. To
target coverage, we propose Density sampling, which models the data
distribution to select a diverse sample. In our comparison of 19 samplers,
involving hundreds of evaluation tasks and pre-training runs, we find that
Ask-LLM and Density are the best methods in their respective categories.
Coverage sampling can recover the performance of the full data, while models
trained on Ask-LLM data consistently outperform full-data training -- even when
we reject 90% of the original dataset, while converging up to 70% faster.