Orionnnn/hub-stats huggingface dataset:
SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization
  of Scientific Topics
Authors: name: Zhiwen You, name: Kanyao Han, name: Haotian Zhu, name: Bertram Ludäscher, name: Jana Diesner
Published: 2024-10-02 18:45:04

Prompt-based fine-tuning has become an essential method for eliciting
information encoded in pre-trained language models for a variety of tasks,
including text classification. For multi-class classification tasks,
prompt-based fine-tuning under low-resource scenarios has resulted in
performance levels comparable to those of fully fine-tuning methods. Previous
studies have used crafted prompt templates and verbalizers, mapping from the
label terms space to the class space, to solve the classification problem as a
masked language modeling task. However, cross-domain and fine-grained
prompt-based fine-tuning with an automatically enriched verbalizer remains
unexplored, mainly due to the difficulty and costs of manually selecting domain
label terms for the verbalizer, which requires humans with domain expertise. To
address this challenge, we introduce SciPrompt, a framework designed to
automatically retrieve scientific topic-related terms for low-resource text
classification tasks. To this end, we select semantically correlated and
domain-specific label terms within the context of scientific literature for
verbalizer augmentation. Furthermore, we propose a new verbalization strategy
that uses correlation scores as additional weights to enhance the prediction
performance of the language model during model tuning. Our method outperforms
state-of-the-art, prompt-based fine-tuning methods on scientific text
classification tasks under few and zero-shot settings, especially in
classifying fine-grained and emerging scientific topics.
Intelligence at the Edge of Chaos
Authors: name: Shiyang Zhang, name: Aakash Patel, name: Syed A Rizvi, name: Nianchen Liu, name: Sizhuang He, name: Amin Karbasi, name: Emanuele Zappala, name: David van Dijk
Published: 2024-10-03 14:42:34

We explore the emergence of intelligent behavior in artificial systems by
investigating how the complexity of rule-based systems influences the
capabilities of models trained to predict these rules. Our study focuses on
elementary cellular automata (ECA), simple yet powerful one-dimensional systems
that generate behaviors ranging from trivial to highly complex. By training
distinct Large Language Models (LLMs) on different ECAs, we evaluated the
relationship between the complexity of the rules' behavior and the intelligence
exhibited by the LLMs, as reflected in their performance on downstream tasks.
Our findings reveal that rules with higher complexity lead to models exhibiting
greater intelligence, as demonstrated by their performance on reasoning and
chess move prediction tasks. Both uniform and periodic systems, and often also
highly chaotic systems, resulted in poorer downstream performance, highlighting
a sweet spot of complexity conducive to intelligence. We conjecture that
intelligence arises from the ability to predict complexity and that creating
intelligence may require only exposure to complexity.
Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning
Authors: name: Weitai Kang, name: Haifeng Huang, name: Yuzhang Shang, name: Mubarak Shah, name: Yan Yan
Published: 2024-09-30 21:55:38

Recent advancements in 3D Large Language Models (3DLLMs) have highlighted
their potential in building general-purpose agents in the 3D real world, yet
challenges remain due to the lack of high-quality robust instruction-following
data, leading to limited discriminative power and generalization of 3DLLMs. In
this paper, we introduce Robin3D, a powerful 3DLLM trained on large-scale
instruction-following data generated by our novel data engine, Robust
Instruction Generation (RIG) engine. RIG generates two key instruction data: 1)
the Adversarial Instruction-following data, which features mixed negative and
positive samples to enhance the model's discriminative understanding. 2) the
Diverse Instruction-following data, which contains various instruction styles
to enhance model's generalization. As a result, we construct 1 million
instruction-following data, consisting of 344K Adversarial samples, 508K
Diverse samples, and 165K benchmark training set samples. To better handle
these complex instructions, Robin3D first incorporates Relation-Augmented
Projector to enhance spatial understanding, and then strengthens the object
referring and grounding ability through ID-Feature Bonding. Robin3D
consistently outperforms previous methods across five widely-used 3D multimodal
learning benchmarks, without the need for task-specific fine-tuning. Notably,
we achieve a 7.8\% improvement in the grounding task (Multi3DRefer) and a 6.9\%
improvement in the captioning task (Scan2Cap).
VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit
  Assignment
Authors: name: Amirhossein Kazemnejad, name: Milad Aghajohari, name: Eva Portelance, name: Alessandro Sordoni, name: Siva Reddy, name: Aaron Courville, name: Nicolas Le Roux
Published: 2024-10-02 15:49:30

Large language models (LLMs) are increasingly applied to complex reasoning
tasks that require executing several complex steps before receiving any reward.
Properly assigning credit to these steps is essential for enhancing model
performance. Proximal Policy Optimization (PPO), a state-of-the-art
reinforcement learning (RL) algorithm used for LLM finetuning, employs value
networks to tackle credit assignment. However, value networks face challenges
in predicting the expected cumulative rewards accurately in complex reasoning
tasks, often leading to high-variance updates and suboptimal performance. In
this work, we systematically evaluate the efficacy of value networks and reveal
their significant shortcomings in reasoning-heavy LLM tasks, showing that they
barely outperform a random baseline when comparing alternative steps. To
address this, we propose VinePPO, a straightforward approach that leverages the
flexibility of language environments to compute unbiased Monte Carlo-based
estimates, bypassing the need for large value networks. Our method consistently
outperforms PPO and other RL-free baselines across MATH and GSM8K datasets with
fewer gradient updates (up to 9x), less wall-clock time (up to 3.0x). These
results emphasize the importance of accurate credit assignment in RL finetuning
of LLM and demonstrate VinePPO's potential as a superior alternative.
Learning the Latent Rules of a Game from Data: A Chess Story
Authors: name: Ben Fauber
Published: 2024-10-03 12:19:49

We demonstrate that small pretrained foundational generative language models
with millions of parameters can learn the latent rules of a process from data
associated with the process. Inspired by Stefan Zweig's novella
"Schachnovelle," also known as "The Royal Game" in English, we show that 28M
and 125M parameter pretrained foundational small language models (SLMs) can be
instruction fine-tuned with 1,000-to-1,000,000 examples to learn the rules of
chess, propose legal moves, and accurately solve chess problems. We also
explore the impact of successive language model fine-tuning epochs on improved
outcomes and demonstrate reductions in model hallucinations by increasing the
number of instruction fine-tuning examples.
Large Language Models as Markov Chains
Authors: name: Oussama Zekri, name: Ambroise Odonnat, name: Abdelhakim Benechehab, name: Linus Bleistein, name: Nicolas Boullé, name: Ievgen Redko
Published: 2024-10-03 17:45:31

Large language models (LLMs) have proven to be remarkably efficient, both
across a wide range of natural language processing tasks and well beyond them.
However, a comprehensive theoretical analysis of the origins of their
impressive performance remains elusive. In this paper, we approach this
challenging task by drawing an equivalence between generic autoregressive
language models with vocabulary of size T and context window of size K and
Markov chains defined on a finite state space of size O(T^K). We
derive several surprising findings related to the existence of a stationary
distribution of Markov chains that capture the inference power of LLMs, their
speed of convergence to it, and the influence of the temperature on the latter.
We then prove pre-training and in-context generalization bounds and show how
the drawn equivalence allows us to enrich their interpretation. Finally, we
illustrate our theoretical guarantees with experiments on several recent LLMs
to highlight how they capture the behavior observed in practice.
CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified
  Multiplet Upcycling
Authors: name: Jihai Zhang, name: Xiaoye Qu, name: Tong Zhu, name: Yu Cheng
Published: 2024-09-28 09:28:51

In recent years, Contrastive Language-Image Pre-training (CLIP) has become a
cornerstone in multimodal intelligence. However, recent studies have identified
that the information loss in the CLIP encoding process is substantial, and CLIP
tends to capture only coarse-grained features from the input. This deficiency
significantly limits the ability of a single CLIP model to handle images rich
in visual detail. In this work, we propose a simple yet effective
model-agnostic strategy, Diversified Multiplet Upcycling (DMU), for CLIP. DMU
efficiently fine-tunes a series of CLIP models that capture different feature
spaces, from a dense pre-trained CLIP checkpoint, sharing parameters except for
the Feed-Forward Network (FFN). These models can then be transformed into a
CLIP-MoE with a larger model capacity, leading to significantly enhanced
performance with minimal computational overhead. To the best of our knowledge,
Diversified Multiplet Upcycling is the first approach to introduce sparsely
activated MoE into CLIP foundation models. Extensive experiments demonstrate
the significant performance of CLIP-MoE across various zero-shot retrieval,
zero-shot image classification tasks, and downstream Multimodal Large Language
Model (MLLM) benchmarks by serving as a vision encoder. Furthermore,
Diversified Multiplet Upcycling enables the conversion of any dense CLIP model
into CLIP-MoEs, which can seamlessly replace CLIP in a plug-and-play manner
without requiring further adaptation in downstream frameworks. Through
Diversified Multiplet Upcycling, we aim to provide valuable insights for future
research on developing more efficient and effective multimodal learning
systems.
Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short
  Videos
Authors: name: Jianrui Zhang, name: Mu Cai, name: Yong Jae Lee
Published: 2024-10-03 17:59:58

There has been growing sentiment recently that modern large multimodal models
(LMMs) have addressed most of the key challenges related to short video
comprehension. As a result, both academia and industry are gradually shifting
their attention towards the more complex challenges posed by understanding
long-form videos. However, is this really the case? Our studies indicate that
LMMs still lack many fundamental reasoning capabilities even when dealing with
short videos. We introduce Vinoground, a temporal counterfactual LMM evaluation
benchmark encompassing 1000 short and natural video-caption pairs. We
demonstrate that existing LMMs severely struggle to distinguish temporal
differences between different actions and object transformations. For example,
the best model GPT-4o only obtains ~50% on our text and video scores, showing a
large gap compared to the human baseline of ~90%. All open-source multimodal
models and CLIP-based models perform much worse, producing mostly random chance
performance. Through this work, we shed light onto the fact that temporal
reasoning in short videos is a problem yet to be fully solved. The dataset and
evaluation code are available at https://vinoground.github.io.
Eliminating Oversaturation and Artifacts of High Guidance Scales in
  Diffusion Models
Authors: name: Seyedmorteza Sadat, name: Otmar Hilliges, name: Romann M. Weber
Published: 2024-10-03 12:06:29

Classifier-free guidance (CFG) is crucial for improving both generation
quality and alignment between the input condition and final output in diffusion
models. While a high guidance scale is generally required to enhance these
aspects, it also causes oversaturation and unrealistic artifacts. In this
paper, we revisit the CFG update rule and introduce modifications to address
this issue. We first decompose the update term in CFG into parallel and
orthogonal components with respect to the conditional model prediction and
observe that the parallel component primarily causes oversaturation, while the
orthogonal component enhances image quality. Accordingly, we propose
down-weighting the parallel component to achieve high-quality generations
without oversaturation. Additionally, we draw a connection between CFG and
gradient ascent and introduce a new rescaling and momentum method for the CFG
update rule based on this insight. Our approach, termed adaptive projected
guidance (APG), retains the quality-boosting advantages of CFG while enabling
the use of higher guidance scales without oversaturation. APG is easy to
implement and introduces practically no additional computational overhead to
the sampling process. Through extensive experiments, we demonstrate that APG is
compatible with various conditional diffusion models and samplers, leading to
improved FID, recall, and saturation scores while maintaining precision
comparable to CFG, making our method a superior plug-and-play alternative to
standard classifier-free guidance.
Training Language Models on Synthetic Edit Sequences Improves Code
  Synthesis
Authors: name: Ulyana Piterbarg, name: Lerrel Pinto, name: Rob Fergus
Published: 2024-10-03 17:57:22

Software engineers mainly write code by editing existing programs. In
contrast, large language models (LLMs) autoregressively synthesize programs in
a single pass. One explanation for this is the scarcity of open-sourced edit
data. While high-quality instruction data for code synthesis is already scarce,
high-quality edit data is even scarcer. To fill this gap, we develop a
synthetic data generation algorithm called LintSeq. This algorithm refactors
existing code into a sequence of code edits by using a linter to procedurally
sample across the error-free insertions that can be used to sequentially write
programs. It outputs edit sequences as text strings consisting of consecutive
program diffs. To test LintSeq, we use it to refactor a dataset of instruction
+ program pairs into instruction + program-diff-sequence tuples. Then, we
instruction finetune a series of smaller LLMs ranging from 2.6B to 14B
parameters on both the re-factored and original versions of this dataset,
comparing zero-shot performance on code synthesis benchmarks. We show that
during repeated sampling, edit sequence finetuned models produce more diverse
programs than baselines. This results in better inference-time scaling for
benchmark coverage as a function of samples, i.e. the fraction of problems
"pass@k" solved by any attempt given "k" tries. For example, on HumanEval
pass@50, small LLMs finetuned on synthetic edit sequences are competitive with
GPT-4 and outperform models finetuned on the baseline dataset by +20% (+/-3%)
in absolute score. Finally, we also pretrain our own tiny LMs for code
understanding. We show that finetuning tiny models on synthetic code edits
results in state-of-the-art code synthesis for the on-device model class. Our
150M parameter edit sequence LM matches or outperforms code models with twice
as many parameters, both with and without repeated sampling, including Codex
and AlphaCode.
Distilling an End-to-End Voice Assistant Without Instruction Training
  Data
Authors: name: William Held, name: Ella Li, name: Michael Ryan, name: Weiyan Shi, name: Yanzhe Zhang, name: Diyi Yang
Published: 2024-10-03 17:04:48

Voice assistants, such as Siri and Google Assistant, typically model audio
and text separately, resulting in lost speech information and increased
complexity. Recent efforts to address this with end-to-end Speech Large
Language Models (LLMs) trained with supervised finetuning (SFT)
  have led to models ``forgetting" capabilities from text-only LLMs. Our work
proposes an alternative paradigm for training Speech LLMs without instruction
data, using the response of a text-only LLM to transcripts as self-supervision.
Importantly, this process can be performed without annotated responses. We show
that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question
Answering, Classification, and Translation. Furthermore, we show that DiVA
better meets user preferences, achieving a 72\% win rate compared with
state-of-the-art models like Qwen 2 Audio, despite using >100x less training
compute.
L-CiteEval: Do Long-Context Models Truly Leverage Context for
  Responding?
Authors: name: Zecheng Tang, name: Keyan Zhou, name: Juntao Li, name: Baibei Ji, name: Jianye Hou, name: Min Zhang
Published: 2024-10-03 00:38:12

Long-context models (LCMs) have made remarkable strides in recent years,
offering users great convenience for handling tasks that involve long context,
such as document summarization. As the community increasingly prioritizes the
faithfulness of generated results, merely ensuring the accuracy of LCM outputs
is insufficient, as it is quite challenging for humans to verify the results
from the extremely lengthy context. Yet, although some efforts have been made
to assess whether LCMs respond truly based on the context, these works either
are limited to specific tasks or heavily rely on external evaluation resources
like GPT-4.In this work, we introduce L-CiteEval, a comprehensive multi-task
benchmark for long-context understanding with citations, aiming to evaluate
both the understanding capability and faithfulness of LCMs. L-CiteEval covers
11 tasks from diverse domains, spanning context lengths from 8K to 48K, and
provides a fully automated evaluation suite. Through testing with 11
cutting-edge closed-source and open-source LCMs, we find that although these
models show minor differences in their generated results, open-source models
substantially trail behind their closed-source counterparts in terms of
citation accuracy and recall. This suggests that current open-source LCMs are
prone to responding based on their inherent knowledge rather than the given
context, posing a significant risk to the user experience in practical
applications. We also evaluate the RAG approach and observe that RAG can
significantly improve the faithfulness of LCMs, albeit with a slight decrease
in the generation quality. Furthermore, we discover a correlation between the
attention mechanisms of LCMs and the citation generation process.
MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to
  Enhance Medical Image Segmentation
Authors: name: Gurucharan Marthi Krishna Kumar, name: Aman Chadha, name: Janine Mendola, name: Amir Shmuel
Published: 2024-10-03 14:50:33

Large Language Models (LLMs), known for their versatility in textual data,
are increasingly being explored for their potential to enhance medical image
segmentation, a crucial task for accurate diagnostic imaging. This study
explores enhancing Vision Transformers (ViTs) for medical image segmentation by
integrating pre-trained LLM transformer blocks. Our approach, which
incorporates a frozen LLM transformer block into the encoder of a ViT-based
model, leads to substantial improvements in segmentation performance across
various medical imaging modalities. We propose a Hybrid Attention Mechanism
that combines global and local feature learning with a Multi-Scale Fusion Block
for aggregating features across different scales. The enhanced model shows
significant performance gains, including an average Dice score increase from
0.74 to 0.79 and improvements in accuracy, precision, and the Jaccard Index.
These results demonstrate the effectiveness of LLM-based transformers in
refining medical image segmentation, highlighting their potential to
significantly boost model accuracy and robustness. The source code and our
implementation are available at: https://bit.ly/3zf2CVs
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second
Authors: name: Aleksei Bochkovskii, name: Amaël Delaunoy, name: Hugo Germain, name: Marcel Santos, name: Yichao Zhou, name: Stephan R. Richter, name: Vladlen Koltun
Published: 2024-10-02 22:42:20

We present a foundation model for zero-shot metric monocular depth
estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with
unparalleled sharpness and high-frequency details. The predictions are metric,
with absolute scale, without relying on the availability of metadata such as
camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map
in 0.3 seconds on a standard GPU. These characteristics are enabled by a number
of technical contributions, including an efficient multi-scale vision
transformer for dense prediction, a training protocol that combines real and
synthetic datasets to achieve high metric accuracy alongside fine boundary
tracing, dedicated evaluation metrics for boundary accuracy in estimated depth
maps, and state-of-the-art focal length estimation from a single image.
Extensive experiments analyze specific design choices and demonstrate that
Depth Pro outperforms prior work along multiple dimensions. We release code and
weights at https://github.com/apple/ml-depth-pro
Contrastive Localized Language-Image Pre-Training
Authors: name: Hong-You Chen, name: Zhengfeng Lai, name: Haotian Zhang, name: Xinze Wang, name: Marcin Eichner, name: Keen You, name: Meng Cao, name: Bowen Zhang, name: Yinfei Yang, name: Zhe Gan
Published: 2024-10-03 17:56:09

Contrastive Language-Image Pre-training (CLIP) has been a celebrated method
for training vision encoders to generate image/text representations
facilitating various applications. Recently, CLIP has been widely adopted as
the vision backbone of multimodal large language models (MLLMs) to connect
image inputs for language interactions. The success of CLIP as a
vision-language foundation model relies on aligning web-crawled noisy text
annotations at image levels. Nevertheless, such criteria may become
insufficient for downstream tasks in need of fine-grained vision
representations, especially when region-level understanding is demanding for
MLLMs. In this paper, we improve the localization capability of CLIP with
several advances. We propose a pre-training method called Contrastive Localized
Language-Image Pre-training (CLOC) by complementing CLIP with region-text
contrastive loss and modules. We formulate a new concept, promptable
embeddings, of which the encoder produces image embeddings easy to transform
into region representations given spatial hints. To support large-scale
pre-training, we design a visually-enriched and spatially-localized captioning
framework to effectively generate region-text pseudo-labels at scale. By
scaling up to billions of annotated images, CLOC enables high-quality regional
embeddings for image region recognition and retrieval tasks, and can be a
drop-in replacement of CLIP to enhance MLLMs, especially on referring and
grounding tasks.