Orionnnn/hub-stats huggingface dataset:
Statically Contextualizing Large Language Models with Typed Holes
Authors: name: Andrew Blinn, name: Xiang Li, name: June Hyung Kim, name: Cyrus Omar
Published: 2024-09-02 03:29:00

Large language models (LLMs) have reshaped the landscape of program
synthesis. However, contemporary LLM-based code completion systems often
hallucinate broken code because they lack appropriate context, particularly
when working with definitions not in the training data nor near the cursor.
This paper demonstrates that tight integration with the type and binding
structure of a language, as exposed by its language server, can address this
contextualization problem in a token-efficient manner. In short, we contend
that AIs need IDEs, too! In particular, we integrate LLM code generation into
the Hazel live program sketching environment. The Hazel Language Server
identifies the type and typing context of the hole being filled, even in the
presence of errors, ensuring that a meaningful program sketch is always
available. This allows prompting with codebase-wide contextual information not
lexically local to the cursor, nor necessarily in the same file, but that is
likely to be semantically local to the developer's goal. Completions
synthesized by the LLM are then iteratively refined via further dialog with the
language server. To evaluate these techniques, we introduce MVUBench, a dataset
of model-view-update (MVU) web applications. These applications serve as
challenge problems due to their reliance on application-specific data
structures. We find that contextualization with type definitions is
particularly impactful. After introducing our ideas in the context of Hazel we
duplicate our techniques and port MVUBench to TypeScript in order to validate
the applicability of these methods to higher-resource languages. Finally, we
outline ChatLSP, a conservative extension to the Language Server Protocol (LSP)
that language servers can implement to expose capabilities that AI code
completion systems of various designs can use to incorporate static context
when generating prompts for an LLM.
Building Math Agents with Multi-Turn Iterative Preference Learning
Authors: name: Wei Xiong, name: Chengshuai Shi, name: Jiaming Shen, name: Aviv Rosenberg, name: Zhen Qin, name: Daniele Calandriello, name: Misha Khalman, name: Rishabh Joshi, name: Bilal Piot, name: Mohammad Saleh, name: Chi Jin, name: Tong Zhang, name: Tianqi Liu
Published: 2024-09-04 02:41:04

Recent studies have shown that large language models' (LLMs) mathematical
problem-solving capabilities can be enhanced by integrating external tools,
such as code interpreters, and employing multi-turn Chain-of-Thought (CoT)
reasoning. While current methods focus on synthetic data generation and
Supervised Fine-Tuning (SFT), this paper studies the complementary direct
preference learning approach to further improve model performance. However,
existing direct preference learning algorithms are originally designed for the
single-turn chat task, and do not fully address the complexities of multi-turn
reasoning and external tool integration required for tool-integrated
mathematical reasoning tasks. To fill in this gap, we introduce a multi-turn
direct preference learning framework, tailored for this context, that leverages
feedback from code interpreters and optimizes trajectory-level preferences.
This framework includes multi-turn DPO and multi-turn KTO as specific
implementations. The effectiveness of our framework is validated through
training of various language models using an augmented prompt set from the
GSM8K and MATH datasets. Our results demonstrate substantial improvements: a
supervised fine-tuned Gemma-1.1-it-7B model's performance increased from 77.5%
to 83.9% on GSM8K and from 46.1% to 51.2% on MATH. Similarly, a Gemma-2-it-9B
model improved from 84.1% to 86.3% on GSM8K and from 51.0% to 54.5% on MATH.
FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary
  Segmentation
Authors: name: Xi Chen, name: Haosen Yang, name: Sheng Jin, name: Xiatian Zhu, name: Hongxun Yao
Published: 2024-09-05 13:36:50

Open-vocabulary segmentation poses significant challenges, as it requires
segmenting and recognizing objects across an open set of categories in
unconstrained environments. Building on the success of powerful vision-language
(ViL) foundation models, such as CLIP, recent efforts sought to harness their
zero-short capabilities to recognize unseen categories. Despite notable
performance improvements, these models still encounter the critical issue of
generating precise mask proposals for unseen categories and scenarios,
resulting in inferior segmentation performance eventually. To address this
challenge, we introduce a novel approach, FrozenSeg, designed to integrate
spatial knowledge from a localization foundation model (e.g., SAM) and semantic
knowledge extracted from a ViL model (e.g., CLIP), in a synergistic framework.
Taking the ViL model's visual encoder as the feature backbone, we inject the
space-aware feature into the learnable queries and CLIP features within the
transformer decoder. In addition, we devise a mask proposal ensemble strategy
for further improving the recall rate and mask quality. To fully exploit
pre-trained knowledge while minimizing training overhead, we freeze both
foundation models, focusing optimization efforts solely on a lightweight
transformer decoder for mask proposal generation-the performance bottleneck.
Extensive experiments demonstrate that FrozenSeg advances state-of-the-art
results across various segmentation benchmarks, trained exclusively on COCO
panoptic data, and tested in a zero-shot manner. Code is available at
https://github.com/chenxi52/FrozenSeg.
CDM: A Reliable Metric for Fair and Accurate Formula Recognition
  Evaluation
Authors: name: Bin Wang, name: Fan Wu, name: Linke Ouyang, name: Zhuangcheng Gu, name: Rui Zhang, name: Renqiu Xia, name: Bo Zhang, name: Conghui He
Published: 2024-09-05 16:01:21

Formula recognition presents significant challenges due to the complicated
structure and varied notation of mathematical expressions. Despite continuous
advancements in formula recognition models, the evaluation metrics employed by
these models, such as BLEU and Edit Distance, still exhibit notable
limitations. They overlook the fact that the same formula has diverse
representations and is highly sensitive to the distribution of training data,
thereby causing the unfairness in formula recognition evaluation. To this end,
we propose a Character Detection Matching (CDM) metric, ensuring the evaluation
objectivity by designing a image-level rather than LaTex-level metric score.
Specifically, CDM renders both the model-predicted LaTeX and the ground-truth
LaTeX formulas into image-formatted formulas, then employs visual feature
extraction and localization techniques for precise character-level matching,
incorporating spatial position information. Such a spatially-aware and
character-matching method offers a more accurate and equitable evaluation
compared with previous BLEU and Edit Distance metrics that rely solely on
text-based character matching. Experimentally, we evaluated various formula
recognition models using CDM, BLEU, and ExpRate metrics. Their results
demonstrate that the CDM aligns more closely with human evaluation standards
and provides a fairer comparison across different models by eliminating
discrepancies caused by diverse formula representations.
Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with
  Image-Based Surface Representation
Authors: name: Slava Elizarov, name: Ciara Rowles, name: Simon Donné
Published: 2024-09-05 17:21:54

Generating high-quality 3D objects from textual descriptions remains a
challenging problem due to computational cost, the scarcity of 3D data, and
complex 3D representations. We introduce Geometry Image Diffusion
(GIMDiffusion), a novel Text-to-3D model that utilizes geometry images to
efficiently represent 3D shapes using 2D images, thereby avoiding the need for
complex 3D-aware architectures. By integrating a Collaborative Control
mechanism, we exploit the rich 2D priors of existing Text-to-Image models such
as Stable Diffusion. This enables strong generalization even with limited 3D
training data (allowing us to use only high-quality training data) as well as
retaining compatibility with guidance techniques such as IPAdapter. In short,
GIMDiffusion enables the generation of 3D assets at speeds comparable to
current Text-to-Image models. The generated objects consist of semantically
meaningful, separate parts and include internal structures, enhancing both
usability and versatility.
Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free
  Real Image Editing
Authors: name: Vadim Titov, name: Madina Khalmatova, name: Alexandra Ivanova, name: Dmitry Vetrov, name: Aibek Alanov
Published: 2024-09-02 15:21:46

Despite recent advances in large-scale text-to-image generative models,
manipulating real images with these models remains a challenging problem. The
main limitations of existing editing methods are that they either fail to
perform with consistent quality on a wide range of image edits or require
time-consuming hyperparameter tuning or fine-tuning of the diffusion model to
preserve the image-specific appearance of the input image. We propose a novel
approach that is built upon a modified diffusion sampling process via the
guidance mechanism. In this work, we explore the self-guidance technique to
preserve the overall structure of the input image and its local regions
appearance that should not be edited. In particular, we explicitly introduce
layout-preserving energy functions that are aimed to save local and global
structures of the source image. Additionally, we propose a noise rescaling
mechanism that allows to preserve noise distribution by balancing the norms of
classifier-free guidance and our proposed guiders during generation. Such a
guiding approach does not require fine-tuning the diffusion model and exact
inversion process. As a result, the proposed method provides a fast and
high-quality editing mechanism. In our experiments, we show through human
evaluation and quantitative analysis that the proposed method allows to produce
desired editing which is more preferable by humans and also achieves a better
trade-off between editing quality and preservation of the original image. Our
code is available at https://github.com/FusionBrainLab/Guide-and-Rescale.
Report Cards: Qualitative Evaluation of Language Models Using Natural
  Language Summaries
Authors: name: Blair Yang, name: Fuyang Cui, name: Keiran Paster, name: Jimmy Ba, name: Pashootan Vaezipoor, name: Silviu Pitis, name: Michael R. Zhang
Published: 2024-09-01 21:18:14

The rapid development and dynamic nature of large language models (LLMs) make
it difficult for conventional quantitative benchmarks to accurately assess
their capabilities. We propose report cards, which are human-interpretable,
natural language summaries of model behavior for specific skills or topics. We
develop a framework to evaluate report cards based on three criteria:
specificity (ability to distinguish between models), faithfulness (accurate
representation of model capabilities), and interpretability (clarity and
relevance to humans). We also propose an iterative algorithm for generating
report cards without human supervision and explore its efficacy by ablating
various design choices. Through experimentation with popular LLMs, we
demonstrate that report cards provide insights beyond traditional benchmarks
and can help address the need for a more interpretable and holistic evaluation
of LLMs.
From MOOC to MAIC: Reshaping Online Teaching and Learning through
  LLM-driven Agents
Authors: name: Jifan Yu, name: Zheyuan Zhang, name: Daniel Zhang-li, name: Shangqing Tu, name: Zhanxin Hao, name: Rui Miao Li, name: Haoxuan Li, name: Yuanchun Wang, name: Hanming Li, name: Linlu Gong, name: Jie Cao, name: Jiayin Lin, name: Jinchang Zhou, name: Fei Qin, name: Haohua Wang, name: Jianxiao Jiang, name: Lijun Deng, name: Yisi Zhan, name: Chaojun Xiao, name: Xusheng Dai, name: Xuan Yan, name: Nianyi Lin, name: Nan Zhang, name: Ruixin Ni, name: Yang Dang, name: Lei Hou, name: Yu Zhang, name: Xu Han, name: Manli Li, name: Juanzi Li, name: Zhiyuan Liu, name: Huiqin Liu, name: Maosong Sun
Published: 2024-09-05 13:22:51

Since the first instances of online education, where courses were uploaded to
accessible and shared online platforms, this form of scaling the dissemination
of human knowledge to reach a broader audience has sparked extensive discussion
and widespread adoption. Recognizing that personalized learning still holds
significant potential for improvement, new AI technologies have been
continuously integrated into this learning format, resulting in a variety of
educational AI applications such as educational recommendation and intelligent
tutoring. The emergence of intelligence in large language models (LLMs) has
allowed for these educational enhancements to be built upon a unified
foundational model, enabling deeper integration. In this context, we propose
MAIC (Massive AI-empowered Course), a new form of online education that
leverages LLM-driven multi-agent systems to construct an AI-augmented
classroom, balancing scalability with adaptivity. Beyond exploring the
conceptual framework and technical innovations, we conduct preliminary
experiments at Tsinghua University, one of China's leading universities.
Drawing from over 100,000 learning records of more than 500 students, we obtain
a series of valuable observations and initial analyses. This project will
continue to evolve, ultimately aiming to establish a comprehensive open
platform that supports and unifies research, technology, and applications in
exploring the possibilities of online education in the era of large model AI.
We envision this platform as a collaborative hub, bringing together educators,
researchers, and innovators to collectively explore the future of AI-driven
online education.
FuzzCoder: Byte-level Fuzzing Test via Large Language Model
Authors: name: Liqun Yang, name: Jian Yang, name: Chaoren Wei, name: Guanglin Niu, name: Ge Zhang, name: Yunli Wang, name: Linzheng ChaI, name: Wanxu Xia, name: Hongcheng Guo, name: Shun Zhang, name: Jiaheng Liu, name: Yuwei Yin, name: Junran Peng, name: Jiaxin Ma, name: Liang Sun, name: Zhoujun Li
Published: 2024-09-03 14:40:31

Fuzzing is an important dynamic program analysis technique designed for
finding vulnerabilities in complex software. Fuzzing involves presenting a
target program with crafted malicious input to cause crashes, buffer overflows,
memory errors, and exceptions. Crafting malicious inputs in an efficient manner
is a difficult open problem and the best approaches often apply uniform random
mutations to pre-existing valid inputs. In this work, we propose to adopt
fine-tuned large language models (FuzzCoder) to learn patterns in the input
files from successful attacks to guide future fuzzing explorations.
Specifically, we develop a framework to leverage the code LLMs to guide the
mutation process of inputs in fuzzing. The mutation process is formulated as
the sequence-to-sequence modeling, where LLM receives a sequence of bytes and
then outputs the mutated byte sequence. FuzzCoder is fine-tuned on the created
instruction dataset (Fuzz-Instruct), where the successful fuzzing history is
collected from the heuristic fuzzing tool. FuzzCoder can predict mutation
locations and strategies locations in input files to trigger abnormal behaviors
of the program. Experimental results show that FuzzCoder based on AFL (American
Fuzzy Lop) gain significant improvements in terms of effective proportion of
mutation (EPM) and number of crashes (NC) for various input formats including
ELF, JPG, MP3, and XML.
mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page
  Document Understanding
Authors: name: Anwen Hu, name: Haiyang Xu, name: Liang Zhang, name: Jiabo Ye, name: Ming Yan, name: Ji Zhang, name: Qin Jin, name: Fei Huang, name: Jingren Zhou
Published: 2024-09-05 11:09:00

Multimodel Large Language Models(MLLMs) have achieved promising OCR-free
Document Understanding performance by increasing the supported resolution of
document images. However, this comes at the cost of generating thousands of
visual tokens for a single document image, leading to excessive GPU memory and
slower inference times, particularly in multi-page document comprehension. In
this work, to address these challenges, we propose a High-resolution
DocCompressor module to compress each high-resolution document image into 324
tokens, guided by low-resolution global visual features. With this compression
module, to strengthen multi-page document comprehension ability and balance
both token efficiency and question-answering performance, we develop the
DocOwl2 under a three-stage training framework: Single-image Pretraining,
Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new
state-of-the-art across multi-page document understanding benchmarks and
reduces first token latency by more than 50%, demonstrating advanced
capabilities in multi-page questioning answering, explanation with evidence
pages, and cross-page structure understanding. Additionally, compared to
single-image MLLMs trained on similar data, our DocOwl2 achieves comparable
single-page understanding performance with less than 20% of the visual tokens.
Our codes, models, and data are publicly available at
https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.
Attention Heads of Large Language Models: A Survey
Authors: name: Zifan Zheng, name: Yezhaohui Wang, name: Yuxin Huang, name: Shichao Song, name: Bo Tang, name: Feiyu Xiong, name: Zhiyu Li
Published: 2024-09-05 17:59:12

Since the advent of ChatGPT, Large Language Models (LLMs) have excelled in
various tasks but remain largely as black-box systems. Consequently, their
development relies heavily on data-driven approaches, limiting performance
enhancement through changes in internal architecture and reasoning pathways. As
a result, many researchers have begun exploring the potential internal
mechanisms of LLMs, aiming to identify the essence of their reasoning
bottlenecks, with most studies focusing on attention heads. Our survey aims to
shed light on the internal reasoning processes of LLMs by concentrating on the
interpretability and underlying mechanisms of attention heads. We first distill
the human thought process into a four-stage framework: Knowledge Recalling,
In-Context Identification, Latent Reasoning, and Expression Preparation. Using
this framework, we systematically review existing research to identify and
categorize the functions of specific attention heads. Furthermore, we summarize
the experimental methodologies used to discover these special heads, dividing
them into two categories: Modeling-Free methods and Modeling-Required methods.
Also, we outline relevant evaluation methods and benchmarks. Finally, we
discuss the limitations of current research and propose several potential
future directions. Our reference list is open-sourced at
https://github.com/IAAR-Shanghai/Awesome-Attention-Heads.
WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild
Authors: name: Yuntian Deng, name: Wenting Zhao, name: Jack Hessel, name: Xiang Ren, name: Claire Cardie, name: Yejin Choi
Published: 2024-09-05 17:59:15

The increasing availability of real-world conversation data offers exciting
opportunities for researchers to study user-chatbot interactions. However, the
sheer volume of this data makes manually examining individual conversations
impractical. To overcome this challenge, we introduce WildVis, an interactive
tool that enables fast, versatile, and large-scale conversation analysis.
WildVis provides search and visualization capabilities in the text and
embedding spaces based on a list of criteria. To manage million-scale datasets,
we implemented optimizations including search index construction, embedding
precomputation and compression, and caching to ensure responsive user
interactions within seconds. We demonstrate WildVis's utility through three
case studies: facilitating chatbot misuse research, visualizing and comparing
topic distributions across datasets, and characterizing user-specific
conversation patterns. WildVis is open-source and designed to be extendable,
supporting additional datasets and customized search and visualization
functionalities.
LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via
  Hybrid Architecture
Authors: name: Xidong Wang, name: Dingjie Song, name: Shunian Chen, name: Chen Zhang, name: Benyou Wang
Published: 2024-09-04 17:25:21

Expanding the long-context capabilities of Multi-modal Large Language
Models~(MLLMs) is crucial for video understanding, high-resolution image
understanding, and multi-modal agents. This involves a series of systematic
optimizations, including model architecture, data construction and training
strategy, particularly addressing challenges such as degraded
performance with more images and high computational costs. In this
paper, we adapt the model architecture to a hybrid of Mamba and Transformer
blocks, approach data construction with both temporal and spatial dependencies
among multiple images and employ a progressive training strategy. The released
model LongLLaVA~(Long-Context Large
Language and Vision Assistant) is the first
hybrid MLLM, which achieved a better balance between efficiency and
effectiveness. LongLLaVA not only achieves competitive results across various
benchmarks, but also maintains high throughput and low memory consumption.
Especially, it could process nearly a thousand images on a single A100 80GB
GPU, showing promising application prospects for a wide range of tasks.
Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for
  Political Text
Authors: name: Michael Burnham, name: Kayla Kahn, name: Ryan Yank Wang, name: Rachel X. Peng
Published: 2024-09-03 17:26:17

Social scientists quickly adopted large language models due to their ability
to annotate documents without supervised training, an ability known as
zero-shot learning. However, due to their compute demands, cost, and often
proprietary nature, these models are often at odds with replication and open
science standards. This paper introduces the Political DEBATE (DeBERTa
Algorithm for Textual Entailment) language models for zero-shot and few-shot
classification of political documents. These models are not only as good, or
better than, state-of-the art large language models at zero and few-shot
classification, but are orders of magnitude more efficient and completely open
source. By training the models on a simple random sample of 10-25 documents,
they can outperform supervised classifiers trained on hundreds or thousands of
documents and state-of-the-art generative models with complex, engineered
prompts. Additionally, we release the PolNLI dataset used to train these models
-- a corpus of over 200,000 political documents with highly accurate labels
across over 800 classification tasks.
FastVoiceGrad: One-step Diffusion-Based Voice Conversion with
  Adversarial Conditional Diffusion Distillation
Authors: name: Takuhiro Kaneko, name: Hirokazu Kameoka, name: Kou Tanaka, name: Yuto Kondo
Published: 2024-09-03 19:19:48

Diffusion-based voice conversion (VC) techniques such as VoiceGrad have
attracted interest because of their high VC performance in terms of speech
quality and speaker similarity. However, a notable limitation is the slow
inference caused by the multi-step reverse diffusion. Therefore, we propose
FastVoiceGrad, a novel one-step diffusion-based VC that reduces the number of
iterations from dozens to one while inheriting the high VC performance of the
multi-step diffusion-based VC. We obtain the model using adversarial
conditional diffusion distillation (ACDD), leveraging the ability of generative
adversarial networks and diffusion models while reconsidering the initial
states in sampling. Evaluations of one-shot any-to-any VC demonstrate that
FastVoiceGrad achieves VC performance superior to or comparable to that of
previous multi-step diffusion-based VC while enhancing the inference speed.
Audio samples are available at
https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/fastvoicegrad/.