Orionnnn/hub-stats huggingface dataset:
SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers
Authors: name: Shraman Pramanick, name: Rama Chellappa, name: Subhashini Venugopalan
Published: 2024-07-12 16:37:59

Seeking answers to questions within long scientific research articles is a
crucial area of study that aids readers in quickly addressing their inquiries.
However, existing question-answering (QA) datasets based on scientific papers
are limited in scale and focus solely on textual content. To address this
limitation, we introduce SPIQA (Scientific Paper Image Question Answering), the
first large-scale QA dataset specifically designed to interpret complex figures
and tables within the context of scientific research articles across various
domains of computer science. Leveraging the breadth of expertise and ability of
multimodal large language models (MLLMs) to understand figures, we employ
automatic and manual curation to create the dataset. We craft an
information-seeking task involving multiple images that cover a wide variety of
plots, charts, tables, schematic diagrams, and result visualizations. SPIQA
comprises 270K questions divided into training, validation, and three different
evaluation splits. Through extensive experiments with 12 prominent foundational
models, we evaluate the ability of current multimodal systems to comprehend the
nuanced aspects of research articles. Additionally, we propose a
Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that
allows fine-grained, step-by-step assessment and improves model performance. We
further explore the upper bounds of performance enhancement with additional
textual information, highlighting its promising potential for future research
and the dataset's impact on revolutionizing how we interact with scientific
literature.
New Desiderata for Direct Preference Optimization
Authors: name: Xiangkun Hu, name: Tong He, name: David Wipf
Published: 2024-07-12 07:52:32

Large language models in the past have typically relied on some form of
reinforcement learning with human feedback (RLHF) to better align model
responses with human preferences. However, because of oft-observed
instabilities when implementing these RLHF pipelines, various
reparameterization techniques have recently been introduced to sidestep the
need for separately learning an RL reward model. Instead, directly fine-tuning
for human preferences is achieved via the minimization of a single closed-form
training objective, a process originally referred to as direct preference
optimization (DPO) and followed by several notable descendants. Although
effective in certain real-world settings, we introduce new evaluation criteria
that serve to highlight unresolved shortcomings in the ability of existing DPO
methods to interpolate between a pre-trained reference model and empirical
measures of human preferences, as well as unavoidable trade-offs in how low-
and high-quality responses are regularized and constraints are handled. Our
insights then motivate an alternative DPO-like loss that provably mitigates
these limitations. Empirical results serve to corroborate notable aspects of
our analyses.
Characterizing Prompt Compression Methods for Long Context Inference
Authors: name: Siddharth Jha, name: Lutfi Eren Erdogan, name: Sehoon Kim, name: Kurt Keutzer, name: Amir Gholami
Published: 2024-07-11 23:34:32

Long context inference presents challenges at the system level with increased
compute and memory requirements, as well as from an accuracy perspective in
being able to reason over long contexts. Recently, several methods have been
proposed to compress the prompt to reduce the context length. However, there
has been little work on comparing the different proposed methods across
different tasks through a standardized analysis. This has led to conflicting
results. To address this, here we perform a comprehensive characterization and
evaluation of different prompt compression methods. In particular, we analyze
extractive compression, summarization-based abstractive compression, and token
pruning methods. Surprisingly, we find that extractive compression often
outperforms all the other approaches, and enables up to 10x compression with
minimal accuracy degradation. Interestingly, we also find that despite several
recent claims, token pruning methods often lag behind extractive compression.
We only found marginal improvements on summarization tasks.
Human-like Episodic Memory for Infinite Context LLMs
Authors: name: Zafeirios Fountas, name: Martin A Benfeghoul, name: Adnan Oomerjee, name: Fenia Christopoulou, name: Gerasimos Lampouras, name: Haitham Bou-Ammar, name: Jun Wang
Published: 2024-07-12 17:34:03

Large language models (LLMs) have shown remarkable capabilities, but still
struggle with processing extensive contexts, limiting their ability to maintain
coherence and accuracy over long sequences. In contrast, the human brain excels
at organising and retrieving episodic experiences across vast temporal scales,
spanning a lifetime. In this work, we introduce EM-LLM, a novel approach that
integrates key aspects of human episodic memory and event cognition into LLMs,
enabling them to effectively handle practically infinite context lengths while
maintaining computational efficiency. EM-LLM organises sequences of tokens into
coherent episodic events using a combination of Bayesian surprise and
graph-theoretic boundary refinement in an on-line fashion. When needed, these
events are retrieved through a two-stage memory process, combining
similarity-based and temporally contiguous retrieval for efficient and
human-like access to relevant information. Experiments on the LongBench dataset
demonstrate EM-LLM's superior performance, outperforming the state-of-the-art
InfLLM model with an overall relative improvement of 4.3% across various tasks,
including a 33% improvement on the PassageRetrieval task. Furthermore, our
analysis reveals strong correlations between EM-LLM's event segmentation and
human-perceived events, suggesting a bridge between this artificial system and
its biological counterpart. This work not only advances LLM capabilities in
processing extended contexts but also provides a computational framework for
exploring human memory mechanisms, opening new avenues for interdisciplinary
research in AI and cognitive science.
MUSCLE: A Model Update Strategy for Compatible LLM Evolution
Authors: name: Jessica Echterhoff, name: Fartash Faghri, name: Raviteja Vemulapalli, name: Ting-Yao Hu, name: Chun-Liang Li, name: Oncel Tuzel, name: Hadi Pouransari
Published: 2024-07-12 17:12:48

Large Language Models (LLMs) are frequently updated due to data or
architecture changes to improve their performance. When updating models,
developers often focus on increasing overall performance metrics with less
emphasis on being compatible with previous model versions. However, users often
build a mental model of the functionality and capabilities of a particular
machine learning model they are interacting with. They have to adapt their
mental model with every update -- a draining task that can lead to user
dissatisfaction. In practice, fine-tuned downstream task adapters rely on
pretrained LLM base models. When these base models are updated, these
user-facing downstream task models experience instance regression or negative
flips -- previously correct instances are now predicted incorrectly. This
happens even when the downstream task training procedures remain identical. Our
work aims to provide seamless model updates to a user in two ways. First, we
provide evaluation metrics for a notion of compatibility to prior model
versions, specifically for generative tasks but also applicable for
discriminative tasks. We observe regression and inconsistencies between
different model versions on a diverse set of tasks and model updates. Second,
we propose a training strategy to minimize the number of inconsistencies in
model updates, involving training of a compatibility model that can enhance
task fine-tuned language models. We reduce negative flips -- instances where a
prior model version was correct, but a new model incorrect -- by up to 40% from
Llama 1 to Llama 2.
GAVEL: Generating Games Via Evolution and Language Models
Authors: name: Graham Todd, name: Alexander Padula, name: Matthew Stephenson, name: Éric Piette, name: Dennis J. N. J. Soemers, name: Julian Togelius
Published: 2024-07-12 16:08:44

Automatically generating novel and interesting games is a complex task.
Challenges include representing game rules in a computationally workable form,
searching through the large space of potential games under most such
representations, and accurately evaluating the originality and quality of
previously unseen games. Prior work in automated game generation has largely
focused on relatively restricted rule representations and relied on
domain-specific heuristics. In this work, we explore the generation of novel
games in the comparatively expansive Ludii game description language, which
encodes the rules of over 1000 board games in a variety of styles and modes of
play. We draw inspiration from recent advances in large language models and
evolutionary computation in order to train a model that intelligently mutates
and recombines games and mechanics expressed as code. We demonstrate both
quantitatively and qualitatively that our approach is capable of generating new
and interesting games, including in regions of the potential rules space not
covered by existing games in the Ludii dataset. A sample of the generated games
are available to play online through the Ludii portal.
Transformer Layers as Painters
Authors: name: Qi Sun, name: Marc Pickett, name: Aakash Kumar Nain, name: Llion Jones
Published: 2024-07-12 14:31:05

Despite their nearly universal adoption for large language models, the
internal workings of transformers are not well understood. We aim to better
understand the impact of removing or reorganizing information throughout the
layers of a pretrained transformer. Such an understanding could both yield
better usage of existing models as well as to make architectural improvements
to produce new variants. We present a series of empirical studies on frozen
models that show that the lower and final layers of pretrained transformers
differ from middle layers, but that middle layers have a surprising amount of
uniformity. We further show that some classes of problems have robustness to
skipping layers, running the layers in an order different from how they were
trained, or running the layers in parallel. Our observations suggest that even
frozen pretrained models may gracefully trade accuracy for latency by skipping
layers or running layers in parallel.
H2O-Danube3 Technical Report
Authors: name: Pascal Pfeiffer, name: Philipp Singer, name: Yauhen Babakhin, name: Gabor Fodor, name: Nischay Dhankhar, name: Sri Satish Ambati
Published: 2024-07-12 14:09:40

We present H2O-Danube3, a series of small language models consisting of
H2O-Danube3-4B, trained on 6T tokens and H2O-Danube3-500M, trained on 4T
tokens. Our models are pre-trained on high quality Web data consisting of
primarily English tokens in three stages with different data mixes before final
supervised tuning for chat version. The models exhibit highly competitive
metrics across a multitude of academic, chat, and fine-tuning benchmarks.
Thanks to its compact architecture, H2O-Danube3 can be efficiently run on a
modern smartphone, enabling local inference and rapid processing capabilities
even on mobile devices. We make all models openly available under Apache 2.0
license further democratizing LLMs to a wider audience economically.
SpreadsheetLLM: Encoding Spreadsheets for Large Language Models
Authors: name: Yuzhang Tian, name: Jianbo Zhao, name: Haoyu Dong, name: Junyu Xiong, name: Shiyu Xia, name: Mengyu Zhou, name: Yun Lin, name: José Cambronero, name: Yeye He, name: Shi Han, name: Dongmei Zhang
Published: 2024-07-12 06:34:21

Spreadsheets, with their extensive two-dimensional grids, various layouts,
and diverse formatting options, present notable challenges for large language
models (LLMs). In response, we introduce SpreadsheetLLM, pioneering an
efficient encoding method designed to unleash and optimize LLMs' powerful
understanding and reasoning capability on spreadsheets. Initially, we propose a
vanilla serialization approach that incorporates cell addresses, values, and
formats. However, this approach was limited by LLMs' token constraints, making
it impractical for most applications. To tackle this challenge, we develop
SheetCompressor, an innovative encoding framework that compresses spreadsheets
effectively for LLMs. It comprises three modules: structural-anchor-based
compression, inverse index translation, and data-format-aware aggregation. It
significantly improves performance in spreadsheet table detection task,
outperforming the vanilla approach by 25.6% in GPT4's in-context learning
setting. Moreover, fine-tuned LLM with SheetCompressor has an average
compression ratio of 25 times, but achieves a state-of-the-art 78.9% F1 score,
surpassing the best existing models by 12.3%. Finally, we propose Chain of
Spreadsheet for downstream tasks of spreadsheet understanding and validate in a
new and demanding spreadsheet QA task. We methodically leverage the inherent
layout and structure of spreadsheets, demonstrating that SpreadsheetLLM is
highly effective across a variety of spreadsheet tasks.
Understanding Retrieval Robustness for Retrieval-Augmented Image
  Captioning
Authors: name: Wenyan Li, name: Jiaang Li, name: Rita Ramos, name: Raphael Tang, name: Desmond Elliott
Published: 2024-06-04 12:41:54

Recent advances in retrieval-augmented models for image captioning highlight
the benefit of retrieving related captions for efficient, lightweight models
with strong domain-transfer capabilities. While these models demonstrate the
success of retrieval augmentation, retrieval models are still far from perfect
in practice: the retrieved information can sometimes mislead the model,
resulting in incorrect generation and worse performance. In this paper, we
analyze the robustness of a retrieval-augmented captioning model SmallCap. Our
analysis shows that the model is sensitive to tokens that appear in the
majority of the retrieved captions, and the input attribution shows that those
tokens are likely copied into the generated output. Given these findings, we
propose to train the model by sampling retrieved captions from more diverse
sets. This decreases the chance that the model learns to copy majority tokens,
and improves both in-domain and cross-domain performance.
StyleSplat: 3D Object Style Transfer with Gaussian Splatting
Authors: name: Sahil Jain, name: Avik Kuthiala, name: Prabhdeep Singh Sethi, name: Prakanshul Saxena
Published: 2024-07-12 17:55:08

Recent advancements in radiance fields have opened new avenues for creating
high-quality 3D assets and scenes. Style transfer can enhance these 3D assets
with diverse artistic styles, transforming creative expression. However,
existing techniques are often slow or unable to localize style transfer to
specific objects. We introduce StyleSplat, a lightweight method for stylizing
3D objects in scenes represented by 3D Gaussians from reference style images.
Our approach first learns a photorealistic representation of the scene using 3D
Gaussian splatting while jointly segmenting individual 3D objects. We then use
a nearest-neighbor feature matching loss to finetune the Gaussians of the
selected objects, aligning their spherical harmonic coefficients with the style
image to ensure consistency and visual appeal. StyleSplat allows for quick,
customizable style transfer and localized stylization of multiple objects
within a scene, each with a different style. We demonstrate its effectiveness
across various 3D scenes and styles, showcasing enhanced control and
customization in 3D creation.
TCAN: Animating Human Images with Temporally Consistent Pose Guidance
  using Diffusion Models
Authors: name: Jeongho Kim, name: Min-Jung Kim, name: Junsoo Lee, name: Jaegul Choo
Published: 2024-07-12 06:02:13

Pose-driven human-image animation diffusion models have shown remarkable
capabilities in realistic human video synthesis. Despite the promising results
achieved by previous approaches, challenges persist in achieving temporally
consistent animation and ensuring robustness with off-the-shelf pose detectors.
In this paper, we present TCAN, a pose-driven human image animation method that
is robust to erroneous poses and consistent over time. In contrast to previous
methods, we utilize the pre-trained ControlNet without fine-tuning to leverage
its extensive pre-acquired knowledge from numerous pose-image-caption pairs. To
keep the ControlNet frozen, we adapt LoRA to the UNet layers, enabling the
network to align the latent space between the pose and appearance features.
Additionally, by introducing an additional temporal layer to the ControlNet, we
enhance robustness against outliers of the pose detector. Through the analysis
of attention maps over the temporal axis, we also designed a novel temperature
map leveraging pose information, allowing for a more static background.
Extensive experiments demonstrate that the proposed method can achieve
promising results in video synthesis tasks encompassing various poses, like
chibi. Project Page: https://eccv2024tcan.github.io/
Self-Recognition in Language Models
Authors: name: Tim R. Davidson, name: Viacheslav Surkov, name: Veniamin Veselovsky, name: Giuseppe Russo, name: Robert West, name: Caglar Gulcehre
Published: 2024-07-09 15:23:28

A rapidly growing number of applications rely on a small set of closed-source
language models (LMs). This dependency might introduce novel security risks if
LMs develop self-recognition capabilities. Inspired by human identity
verification methods, we propose a novel approach for assessing
self-recognition in LMs using model-generated "security questions". Our test
can be externally administered to keep track of frontier models as it does not
require access to internal model parameters or output probabilities. We use our
test to examine self-recognition in ten of the most capable open- and
closed-source LMs currently publicly available. Our extensive experiments found
no empirical evidence of general or consistent self-recognition in any examined
LM. Instead, our results suggest that given a set of alternatives, LMs seek to
pick the "best" answer, regardless of its origin. Moreover, we find indications
that preferences about which models produce the best answers are consistent
across LMs. We additionally uncover novel insights on position bias
considerations for LMs in multiple-choice settings.
Scaling Up Personalized Aesthetic Assessment via Task Vector
  Customization
Authors: name: Jooyeol Yun, name: Jaegul Choo
Published: 2024-07-09 18:42:41

The task of personalized image aesthetic assessment seeks to tailor aesthetic
score prediction models to match individual preferences with just a few
user-provided inputs. However, the scalability and generalization capabilities
of current approaches are considerably restricted by their reliance on an
expensive curated database. To overcome this long-standing scalability
challenge, we present a unique approach that leverages readily available
databases for general image aesthetic assessment and image quality assessment.
Specifically, we view each database as a distinct image score regression task
that exhibits varying degrees of personalization potential. By determining
optimal combinations of task vectors, known to represent specific traits of
each database, we successfully create personalized models for individuals. This
approach of integrating multiple models allows us to harness a substantial
amount of data. Our extensive experiments demonstrate the effectiveness of our
approach in generalizing to previously unseen domains-a challenge previous
approaches have struggled to achieve-making it highly applicable to real-world
scenarios. Our novel approach significantly advances the field by offering
scalable solutions for personalized aesthetic assessment and establishing high
standards for future research.
https://yeolj00.github.io/personal-projects/personalized-aesthetics/
Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning
  Instruction Using Language Model
Authors: name: Wenqi Zhang, name: Zhenglin Cheng, name: Yuanyu He, name: Mengna Wang, name: Yongliang Shen, name: Zeqi Tan, name: Guiyang Hou, name: Mingqian He, name: Yanna Ma, name: Weiming Lu, name: Yueting Zhuang
Published: 2024-07-09 17:18:27

Although most current large multimodal models (LMMs) can already understand
photos of natural scenes and portraits, their understanding of abstract images,
e.g., charts, maps, or layouts, and visual reasoning capabilities remains quite
rudimentary. They often struggle with simple daily tasks, such as reading time
from a clock, understanding a flowchart, or planning a route using a road map.
In light of this, we design a multi-modal self-instruct, utilizing large
language models and their code capabilities to synthesize massive abstract
images and visual reasoning instructions across daily scenarios. Our strategy
effortlessly creates a multimodal benchmark with 11,193 instructions for eight
visual scenarios: charts, tables, simulated maps, dashboards, flowcharts,
relation graphs, floor plans, and visual puzzles. This benchmark,
constructed with simple lines and geometric elements, exposes the shortcomings
of most advanced LMMs like Claude-3.5-Sonnet and GPT-4o in abstract image
understanding, spatial relations reasoning, and visual element induction.
Besides, to verify the quality of our synthetic data, we fine-tune an LMM using
62,476 synthetic chart, table and road map instructions. The results
demonstrate improved chart understanding and map navigation performance, and
also demonstrate potential benefits for other visual reasoning tasks. Our code
is available at: https://github.com/zwq2018/Multi-modal-Self-instruct.
Is Your Model Really A Good Math Reasoner? Evaluating Mathematical
  Reasoning with Checklist
Authors: name: Zihao Zhou, name: Shudong Liu, name: Maizhen Ning, name: Wei Liu, name: Jindong Wang, name: Derek F. Wong, name: Xiaowei Huang, name: Qiufeng Wang, name: Kaizhu Huang
Published: 2024-07-11 17:58:58

Exceptional mathematical reasoning ability is one of the key features that
demonstrate the power of large language models (LLMs). How to comprehensively
define and evaluate the mathematical abilities of LLMs, and even reflect the
user experience in real-world scenarios, has emerged as a critical issue.
Current benchmarks predominantly concentrate on problem-solving capabilities,
which presents a substantial risk of model overfitting and fails to accurately
represent genuine mathematical reasoning abilities. In this paper, we argue
that if a model really understands a problem, it should be robustly and readily
applied across a diverse array of tasks. Motivated by this, we introduce
MATHCHECK, a well-designed checklist for testing task generalization and
reasoning robustness, as well as an automatic tool to generate checklists
efficiently. MATHCHECK includes multiple mathematical reasoning tasks and
robustness test types to facilitate a comprehensive evaluation of both
mathematical reasoning ability and behavior testing. Utilizing MATHCHECK, we
develop MATHCHECK-GSM and MATHCHECK-GEO to assess mathematical textual
reasoning and multi-modal reasoning capabilities, respectively, serving as
upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K.
We adopt MATHCHECK-GSM and MATHCHECK-GEO to evaluate over 20 LLMs and 11 MLLMs,
assessing their comprehensive mathematical reasoning abilities. Our results
demonstrate that while frontier LLMs like GPT-4o continue to excel in various
abilities on the checklist, many other model families exhibit a significant
decline. Further experiments indicate that, compared to traditional math
benchmarks, MATHCHECK better reflects true mathematical abilities and
represents mathematical intelligence more linearly, thereby supporting our
design. On our MATHCHECK, we can easily conduct detailed behavior analysis to
deeply investigate models.