Orionnnn/hub-stats huggingface dataset:
The Semantic Hub Hypothesis: Language Models Share Semantic
  Representations Across Languages and Modalities
Authors: name: Zhaofeng Wu, name: Xinyan Velocity Yu, name: Dani Yogatama, name: Jiasen Lu, name: Yoon Kim
Published: 2024-11-07 18:55:09

Modern language models can process inputs across diverse languages and
modalities. We hypothesize that models acquire this capability through learning
a shared representation space across heterogeneous data types (e.g., different
languages and modalities), which places semantically similar inputs near one
another, even if they are from different modalities/languages. We term this the
semantic hub hypothesis, following the hub-and-spoke model from neuroscience
(Patterson et al., 2007) which posits that semantic knowledge in the human
brain is organized through a transmodal semantic "hub" which integrates
information from various modality-specific "spokes" regions. We first show that
model representations for semantically equivalent inputs in different languages
are similar in the intermediate layers, and that this space can be interpreted
using the model's dominant pretraining language via the logit lens. This
tendency extends to other data types, including arithmetic expressions, code,
and visual/audio inputs. Interventions in the shared representation space in
one data type also predictably affect model outputs in other data types,
suggesting that this shared representations space is not simply a vestigial
byproduct of large-scale training on broad data, but something that is actively
utilized by the model during input processing.
LLM2CLIP: Powerful Language Model Unlock Richer Visual Representation
Authors: name: Weiquan Huang, name: Aoqi Wu, name: Yifan Yang, name: Xufang Luo, name: Yuqing Yang, name: Liang Hu, name: Qi Dai, name: Xiyang Dai, name: Dongdong Chen, name: Chong Luo, name: Lili Qiu
Published: 2024-11-07 18:59:16

CLIP is one of the most important multimodal foundational models today. What
powers CLIP's capabilities? The rich supervision signals provided by natural
language, the carrier of human knowledge, shape a powerful cross-modal
representation space. However, with the rapid advancements in large language
models LLMs like GPT-4 and LLaMA, the boundaries of language comprehension and
generation are continually being pushed. This raises an intriguing question:
can the capabilities of LLMs be harnessed to further improve multimodal
representation learning? The potential benefits of incorporating LLMs into CLIP
are clear. LLMs' strong textual understanding can fundamentally improve CLIP's
ability to handle image captions, drastically enhancing its ability to process
long and complex texts, a well-known limitation of vanilla CLIP. Moreover, LLMs
are trained on a vast corpus of text, possessing open-world knowledge. This
allows them to expand on caption information during training, increasing the
efficiency of the learning process. In this paper, we propose LLM2CLIP, a novel
approach that embraces the power of LLMs to unlock CLIP's potential. By
fine-tuning the LLM in the caption space with contrastive learning, we extract
its textual capabilities into the output embeddings, significantly improving
the output layer's textual discriminability. We then design an efficient
training process where the fine-tuned LLM acts as a powerful teacher for CLIP's
visual encoder. Thanks to the LLM's presence, we can now incorporate longer and
more complex captions without being restricted by vanilla CLIP's text encoder's
context window and ability limitations. Our experiments demonstrate that this
approach brings substantial improvements in cross-modal tasks.
Improving the detection of technical debt in Java source code with an
  enriched dataset
Authors: name: Nam Le Hai, name: Anh M. T. Bui, name: Phuong T. Nguyen, name: Davide Di Ruscio, name: Rick Kazman
Published: 2024-11-08 10:12:33

Technical debt (TD) is a term used to describe the additional work and costs
that emerge when developers have opted for a quick and easy solution to a
problem, rather than a more effective and well-designed, but time-consuming
approach. Self-Admitted Technical Debts (SATDs) are a specific type of
technical debts that developers intentionally document and acknowledge,
typically via textual comments. While these self-admitted comments are a useful
tool for identifying technical debts, most of the existing approaches focus on
capturing crucial tokens associated with various categories of TD, neglecting
the rich information embedded within the source code itself. Recent research
has focused on detecting SATDs by analyzing comments embedded in source code,
and there has been little work dealing with technical debts contained in the
source code. To fill such a gap, in this study, through the analysis of
comments and their associated source code from 974 Java projects hosted in the
Stack corpus, we curated the first ever dataset of TD identified by code
comments, coupled with its associated source code. Through an empirical
evaluation, we found out that the comments of the resulting dataset help
enhance the prediction performance of state-of-the-art SATD detection models.
More importantly, including the classified source code significantly improves
the accuracy in predicting various types of technical debt. In this respect,
our work is two-fold: (i) We believe that our dataset will catalyze future work
in the domain, inspiring various research issues related to the recognition of
technical debt; (ii) The proposed classifiers may serve as baselines for other
studies on the detection of TD by means of the curated dataset.
DELIFT: Data Efficient Language model Instruction Fine Tuning
Authors: name: Ishika Agarwal, name: Krishna Killamsetty, name: Lucian Popa, name: Marina Danilevksy
Published: 2024-11-07 04:38:29

Fine-tuning large language models (LLMs) is essential for enhancing their
performance on specific tasks but is often resource-intensive due to redundant
or uninformative data. To address this inefficiency, we introduce DELIFT (Data
Efficient Language model Instruction Fine-Tuning), a novel algorithm that
systematically optimizes data selection across the three key stages of
fine-tuning: (1) instruction tuning, (2) task-specific fine-tuning (e.g.,
reasoning, question-answering), and (3) continual fine-tuning (e.g.,
incorporating new data versions). Unlike existing methods that focus on
single-stage optimization or rely on computationally intensive gradient
calculations, DELIFT operates efficiently across all stages. Central to our
approach is a pairwise utility metric that quantifies how beneficial a data
sample is for improving the model's responses to other samples, effectively
measuring the informational value relative to the model's current capabilities.
By leveraging different submodular functions applied to this metric, DELIFT
selects diverse and optimal subsets that are useful across all stages of
fine-tuning. Experiments across various tasks and model scales demonstrate that
DELIFT can reduce the fine-tuning data size by up to 70% without compromising
performance, offering significant computational savings and outperforming
existing methods in both efficiency and efficacy.
Balancing Pipeline Parallelism with Vocabulary Parallelism
Authors: name: Man Tsung Yeung, name: Penghui Qi, name: Min Lin, name: Xinyi Wan
Published: 2024-11-08 02:45:30

Pipeline parallelism is widely used to scale the training of
transformer-based large language models, various works have been done to
improve its throughput and memory footprint. In this paper, we address a
frequently overlooked issue: the vocabulary layers can cause imbalanced
computation and memory usage across pipeline stages, worsening pipeline bubbles
and the memory bottleneck. To tackle this, we partition the vocabulary layers
evenly across pipeline devices and group the computation into pipeline passes.
To reduce the activation memory overhead, we propose several algorithms to
reduce communication barriers within vocabulary layers. Additionally, we
utilize a generalizable method to integrate Vocabulary Parallelism with
existing pipeline schedules. By combining these techniques, our methods
effectively balance the computation and parameter memory, with only a small
constant activation memory overhead. Notably, when combined with activation
memory-balanced schedules like V-Half, our approach achieves perfect balance in
both memory and computation. Extensive evaluations demonstrate that our method
achieves computation and memory balance regardless of the vocabulary size,
resulting in a 5% to 51% improvement in throughput compared to naive
approaches, meanwhile significantly reducing peak memory usage especially for
large vocabulary scenarios. Our implementation is open-sourced at
https://github.com/sail-sg/VocabularyParallelism .
Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test
  Generation: An Empirical Study
Authors: name: André Storhaug, name: Jingyue Li
Published: 2024-11-04 09:03:18

The advent of large language models (LLMs) like GitHub Copilot has
significantly enhanced programmers' productivity, particularly in code
generation. However, these models often struggle with real-world tasks without
fine-tuning. As LLMs grow larger and more performant, fine-tuning for
specialized tasks becomes increasingly expensive. Parameter-efficient
fine-tuning (PEFT) methods, which fine-tune only a subset of model parameters,
offer a promising solution by reducing the computational costs of tuning LLMs
while maintaining their performance. Existing studies have explored using PEFT
and LLMs for various code-related tasks and found that the effectiveness of
PEFT techniques is task-dependent. The application of PEFT techniques in unit
test generation remains underexplored. The state-of-the-art is limited to using
LLMs with full fine-tuning to generate unit tests. This paper investigates both
full fine-tuning and various PEFT methods, including LoRA, (IA)^3, and prompt
tuning, across different model architectures and sizes. We use well-established
benchmark datasets to evaluate their effectiveness in unit test generation. Our
findings show that PEFT methods can deliver performance comparable to full
fine-tuning for unit test generation, making specialized fine-tuning more
accessible and cost-effective. Notably, prompt tuning is the most effective in
terms of cost and resource utilization, while LoRA approaches the effectiveness
of full fine-tuning in several cases.
StdGEN: Semantic-Decomposed 3D Character Generation from Single Images
Authors: name: Yuze He, name: Yanning Zhou, name: Wang Zhao, name: Zhongkai Wu, name: Kaiwen Xiao, name: Wei Yang, name: Yong-Jin Liu, name: Xiao Han
Published: 2024-11-08 17:54:18

We present StdGEN, an innovative pipeline for generating semantically
decomposed high-quality 3D characters from single images, enabling broad
applications in virtual reality, gaming, and filmmaking, etc. Unlike previous
methods which struggle with limited decomposability, unsatisfactory quality,
and long optimization times, StdGEN features decomposability, effectiveness and
efficiency; i.e., it generates intricately detailed 3D characters with
separated semantic components such as the body, clothes, and hair, in three
minutes. At the core of StdGEN is our proposed Semantic-aware Large
Reconstruction Model (S-LRM), a transformer-based generalizable model that
jointly reconstructs geometry, color and semantics from multi-view images in a
feed-forward manner. A differentiable multi-layer semantic surface extraction
scheme is introduced to acquire meshes from hybrid implicit fields
reconstructed by our S-LRM. Additionally, a specialized efficient multi-view
diffusion model and an iterative multi-layer surface refinement module are
integrated into the pipeline to facilitate high-quality, decomposable 3D
character generation. Extensive experiments demonstrate our state-of-the-art
performance in 3D anime character generation, surpassing existing baselines by
a significant margin in geometry, texture and decomposability. StdGEN offers
ready-to-use semantic-decomposed 3D characters and enables flexible
customization for a wide range of applications. Project page:
https://stdgen.github.io
Analyzing The Language of Visual Tokens
Authors: name: David M. Chan, name: Rodolfo Corona, name: Joonyong Park, name: Cheol Jun Cho, name: Yutong Bai, name: Trevor Darrell
Published: 2024-11-07 18:59:28

With the introduction of transformer-based models for vision and language
tasks, such as LLaVA and Chameleon, there has been renewed interest in the
discrete tokenized representation of images. These models often treat image
patches as discrete tokens, analogous to words in natural language, learning
joint alignments between visual and human languages. However, little is known
about the statistical behavior of these visual languages - whether they follow
similar frequency distributions, grammatical structures, or topologies as
natural languages. In this paper, we take a natural-language-centric approach
to analyzing discrete visual languages and uncover striking similarities and
fundamental differences. We demonstrate that, although visual languages adhere
to Zipfian distributions, higher token innovation drives greater entropy and
lower compression, with tokens predominantly representing object parts,
indicating intermediate granularity. We also show that visual languages lack
cohesive grammatical structures, leading to higher perplexity and weaker
hierarchical organization compared to natural languages. Finally, we
demonstrate that, while vision models align more closely with natural languages
than other models, this alignment remains significantly weaker than the
cohesion found within natural languages. Through these experiments, we
demonstrate how understanding the statistical properties of discrete visual
languages can inform the design of more effective computer vision models.
M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for
  Evaluating Foundation Models
Authors: name: Chuhan Li, name: Ziyao Shangguan, name: Yilun Zhao, name: Deyuan Li, name: Yixin Liu, name: Arman Cohan
Published: 2024-11-06 17:52:01

Existing benchmarks for evaluating foundation models mainly focus on
single-document, text-only tasks. However, they often fail to fully capture the
complexity of research workflows, which typically involve interpreting
non-textual data and gathering information across multiple documents. To
address this gap, we introduce M3SciQA, a multi-modal, multi-document
scientific question answering benchmark designed for a more comprehensive
evaluation of foundation models. M3SciQA consists of 1,452 expert-annotated
questions spanning 70 natural language processing paper clusters, where each
cluster represents a primary paper along with all its cited documents,
mirroring the workflow of comprehending a single paper by requiring multi-modal
and multi-document data. With M3SciQA, we conduct a comprehensive evaluation of
18 foundation models. Our results indicate that current foundation models still
significantly underperform compared to human experts in multi-modal information
retrieval and in reasoning across multiple scientific documents. Additionally,
we explore the implications of these findings for the future advancement of
applying foundation models in multi-modal scientific literature analysis.
M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page
  Multi-document Understanding
Authors: name: Jaemin Cho, name: Debanjan Mahata, name: Ozan Irsoy, name: Yujie He, name: Mohit Bansal
Published: 2024-11-07 18:29:38

Document visual question answering (DocVQA) pipelines that answer questions
from documents have broad applications. Existing methods focus on handling
single-page documents with multi-modal language models (MLMs), or rely on
text-based retrieval-augmented generation (RAG) that uses text extraction tools
such as optical character recognition (OCR). However, there are difficulties in
applying these methods in real-world scenarios: (a) questions often require
information across different pages or documents, where MLMs cannot handle many
long documents; (b) documents often have important information in visual
elements such as figures, but text extraction tools ignore them. We introduce
M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various
document contexts (closed-domain and open-domain), question hops (single-hop
and multi-hop), and evidence modalities (text, chart, figure, etc.). M3DocRAG
finds relevant documents and answers questions using a multi-modal retriever
and an MLM, so that it can efficiently handle single or many documents while
preserving visual information. Since previous DocVQA datasets ask questions in
the context of a specific document, we also present M3DocVQA, a new benchmark
for evaluating open-domain DocVQA over 3,000+ PDF documents with 40,000+ pages.
In three benchmarks (M3DocVQA/MMLongBench-Doc/MP-DocVQA), empirical results
show that M3DocRAG with ColPali and Qwen2-VL 7B achieves superior performance
than many strong baselines, including state-of-the-art performance in
MP-DocVQA. We provide comprehensive analyses of different indexing, MLMs, and
retrieval models. Lastly, we qualitatively show that M3DocRAG can successfully
handle various scenarios, such as when relevant information exists across
multiple pages and when answer evidence only exists in images.
Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion
  Models
Authors: name: Shuhong Zheng, name: Zhipeng Bao, name: Ruoyu Zhao, name: Martial Hebert, name: Yu-Xiong Wang
Published: 2024-11-07 18:59:53

Beyond high-fidelity image synthesis, diffusion models have recently
exhibited promising results in dense visual perception tasks. However, most
existing work treats diffusion models as a standalone component for perception
tasks, employing them either solely for off-the-shelf data augmentation or as
mere feature extractors. In contrast to these isolated and thus sub-optimal
efforts, we introduce a unified, versatile, diffusion-based framework,
Diff-2-in-1, that can simultaneously handle both multi-modal data generation
and dense visual perception, through a unique exploitation of the
diffusion-denoising process. Within this framework, we further enhance
discriminative visual perception via multi-modal generation, by utilizing the
denoising network to create multi-modal data that mirror the distribution of
the original training set. Importantly, Diff-2-in-1 optimizes the utilization
of the created diverse and faithful data by leveraging a novel self-improving
learning mechanism. Comprehensive experimental evaluations validate the
effectiveness of our framework, showcasing consistent performance improvements
across various discriminative backbones and high-quality multi-modal data
generation characterized by both realism and usefulness.
Needle Threading: Can LLMs Follow Threads through Near-Million-Scale
  Haystacks?
Authors: name: Jonathan Roberts, name: Kai Han, name: Samuel Albanie
Published: 2024-11-07 18:59:27

As the context limits of Large Language Models (LLMs) increase, the range of
possible applications and downstream functions broadens. In many real-world
tasks, decisions depend on details scattered across collections of often
disparate documents containing mostly irrelevant information. Long-context LLMs
appear well-suited to this form of complex information retrieval and reasoning,
which has traditionally proven costly and time-consuming. However, although the
development of longer context models has seen rapid gains in recent years, our
understanding of how effectively LLMs use their context has not kept pace. To
address this, we conduct a set of retrieval experiments designed to evaluate
the capabilities of 17 leading LLMs, such as their ability to follow threads of
information through the context window. Strikingly, we find that many models
are remarkably threadsafe: capable of simultaneously following multiple threads
without significant loss in performance. Still, for many models, we find the
effective context limit is significantly shorter than the supported context
length, with accuracy decreasing as the context window grows. Our study also
highlights the important point that token counts from different tokenizers
should not be directly compared -- they often correspond to substantially
different numbers of written characters. We release our code and long-context
experimental data.
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in
  Videos
Authors: name: Shehan Munasinghe, name: Hanan Gani, name: Wenqi Zhu, name: Jiale Cao, name: Eric Xing, name: Fahad Shahbaz Khan, name: Salman Khan
Published: 2024-11-07 17:59:27

Fine-grained alignment between videos and text is challenging due to complex
spatial and temporal dynamics in videos. Existing video-based Large Multimodal
Models (LMMs) handle basic conversations but struggle with precise pixel-level
grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed
for fine-grained pixel-level grounding in videos based on user-provided textual
inputs. Our design seamlessly connects three key components: a Large Language
Model, a dual vision encoder that emphasizes both spatial and temporal details,
and a spatio-temporal decoder for accurate mask generation. This connection is
facilitated via tunable V-L and L-V adapters that enable close Vision-Language
(VL) alignment. The architecture is trained to synchronize both spatial and
temporal elements of video content with textual instructions. To enable
fine-grained grounding, we curate a multimodal dataset featuring detailed
visually-grounded conversations using a semiautomatic annotation pipeline,
resulting in a diverse set of 38k video-QA triplets along with 83k objects and
671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded
Conversation Generation, Visual Grounding, and Referring Video Segmentation.
Experimental results show that our model consistently outperforms existing
approaches across all three tasks.
BitNet a4.8: 4-bit Activations for 1-bit LLMs
Authors: name: Hongyu Wang, name: Shuming Ma, name: Furu Wei
Published: 2024-11-07 18:41:50

Recent research on the 1-bit Large Language Models (LLMs), such as BitNet
b1.58, presents a promising direction for reducing the inference cost of LLMs
while maintaining their performance. In this work, we introduce BitNet a4.8,
enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid
quantization and sparsification strategy to mitigate the quantization errors
introduced by the outlier channels. Specifically, we utilize 4-bit activations
for inputs to the attention and feed-forward network layers, while sparsifying
intermediate states followed with 8-bit quantization. Extensive experiments
demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58
with equivalent training costs, while being faster in inference with enabling
4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of
parameters and supports 3-bit KV cache, further enhancing the efficiency of
large-scale LLM deployment and inference.