Orionnnn/hub-stats huggingface dataset:
Open Language Data Initiative: Advancing Low-Resource Machine
  Translation for Karakalpak
Authors: name: Mukhammadsaid Mamasaidov, name: Abror Shopulatov
Published: 2024-09-06 13:25:18

This study presents several contributions for the Karakalpak language: a
FLORES+ devtest dataset translated to Karakalpak, parallel corpora for
Uzbek-Karakalpak, Russian-Karakalpak and English-Karakalpak of 100,000 pairs
each and open-sourced fine-tuned neural models for translation across these
languages. Our experiments compare different model variants and training
approaches, demonstrating improvements over existing baselines. This work,
conducted as part of the Open Language Data Initiative (OLDI) shared task, aims
to advance machine translation capabilities for Karakalpak and contribute to
expanding linguistic diversity in NLP technologies.
Paper Copilot: A Self-Evolving and Efficient LLM System for Personalized
  Academic Assistance
Authors: name: Guanyu Lin, name: Tao Feng, name: Pengrui Han, name: Ge Liu, name: Jiaxuan You
Published: 2024-09-06 20:04:04

As scientific research proliferates, researchers face the daunting task of
navigating and reading vast amounts of literature. Existing solutions, such as
document QA, fail to provide personalized and up-to-date information
efficiently. We present Paper Copilot, a self-evolving, efficient LLM system
designed to assist researchers, based on thought-retrieval, user profile and
high performance optimization. Specifically, Paper Copilot can offer
personalized research services, maintaining a real-time updated database.
Quantitative evaluation demonstrates that Paper Copilot saves 69.92\% of time
after efficient deployment. This paper details the design and implementation of
Paper Copilot, highlighting its contributions to personalized academic support
and its potential to streamline the research process.
Insights from Benchmarking Frontier Language Models on Web App Code
  Generation
Authors: name: Yi Cui
Published: 2024-09-08 18:24:26

This paper presents insights from evaluating 16 frontier large language
models (LLMs) on the WebApp1K benchmark, a test suite designed to assess the
ability of LLMs to generate web application code. The results reveal that while
all models possess similar underlying knowledge, their performance is
differentiated by the frequency of mistakes they make. By analyzing lines of
code (LOC) and failure distributions, we find that writing correct code is more
complex than generating incorrect code. Furthermore, prompt engineering shows
limited efficacy in reducing errors beyond specific cases. These findings
suggest that further advancements in coding LLM should emphasize on model
reliability and mistake minimization.
Evaluating Multiview Object Consistency in Humans and Image Models
Authors: name: Tyler Bonnen, name: Stephanie Fu, name: Yutong Bai, name: Thomas O'Connell, name: Yoni Friedman, name: Nancy Kanwisher, name: Joshua B. Tenenbaum, name: Alexei A. Efros
Published: 2024-09-09 17:59:13

We introduce a benchmark to directly evaluate the alignment between human
observers and vision models on a 3D shape inference task. We leverage an
experimental design from the cognitive sciences which requires zero-shot visual
inferences about object shape: given a set of images, participants identify
which contain the same/different objects, despite considerable viewpoint
variation. We draw from a diverse range of images that include common objects
(e.g., chairs) as well as abstract shapes (i.e., procedurally generated
`nonsense' objects). After constructing over 2000 unique image sets, we
administer these tasks to human participants, collecting 35K trials of
behavioral data from over 500 participants. This includes explicit choice
behaviors as well as intermediate measures, such as reaction time and gaze
data. We then evaluate the performance of common vision models (e.g., DINOv2,
MAE, CLIP). We find that humans outperform all models by a wide margin. Using a
multi-scale evaluation approach, we identify underlying similarities and
differences between models and humans: while human-model performance is
correlated, humans allocate more time/processing on challenging trials. All
images, data, and code can be accessed via our project page.
MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge
  Discovery
Authors: name: Hongjin Qian, name: Peitian Zhang, name: Zheng Liu, name: Kelong Mao, name: Zhicheng Dou
Published: 2024-09-09 13:20:31

Retrieval-Augmented Generation (RAG) leverages retrieval tools to access
external databases, thereby enhancing the generation quality of large language
models (LLMs) through optimized context. However, the existing retrieval
methods are constrained inherently, as they can only perform relevance matching
between explicitly stated queries and well-formed knowledge, but unable to
handle tasks involving ambiguous information needs or unstructured knowledge.
Consequently, existing RAG systems are primarily effective for straightforward
question-answering tasks. In this work, we propose MemoRAG, a novel
retrieval-augmented generation paradigm empowered by long-term memory. MemoRAG
adopts a dual-system architecture. On the one hand, it employs a light
but long-range LLM to form the global memory of database. Once a task is
presented, it generates draft answers, cluing the retrieval tools to locate
useful information within the database. On the other hand, it leverages an
expensive but expressive LLM, which generates the ultimate answer
based on the retrieved information. Building on this general framework, we
further optimize MemoRAG's performance by enhancing its cluing mechanism and
memorization capacity. In our experiment, MemoRAG achieves superior performance
across a variety of evaluation tasks, including both complex ones where
conventional RAG fails and straightforward ones where RAG is commonly applied.
Benchmarking Chinese Knowledge Rectification in Large Language Models
Authors: name: Tianhe Lu, name: Jizhan Fang, name: Yunzhi Yao, name: Xin Xu, name: Ningyu Zhang, name: Huajun Chen
Published: 2024-09-09 17:11:51

While Large Language Models (LLMs) exhibit remarkable generative
capabilities, they are not without flaws, particularly in the form of
hallucinations. This issue is even more pronounced when LLMs are applied to
specific languages and domains. For example, LLMs may generate nonsense
information when handling Chinese ancient poetry, proverbs, or idioms, owing to
the lack of specific knowledge. To this end, this paper introduces a benchmark
for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically,
we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of
knowledge from various sources, including classical texts, idioms, and content
from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony,
antithesis, and logical constructs inherent in the Chinese language. Through
the analysis of this dataset, we uncover the challenges faced by current LLMs
in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge
editing techniques on this dataset unveil the substantial scope for advancement
in the rectification of Chinese knowledge. Code and dataset are available at
https://github.com/zjunlp/EasyEdit.
OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs
Authors: name: Jintian Zhang, name: Cheng Peng, name: Mengshu Sun, name: Xiang Chen, name: Lei Liang, name: Zhiqiang Zhang, name: Jun Zhou, name: Huajun Chen, name: Ningyu Zhang
Published: 2024-09-08 16:35:19

Despite the recent advancements in Large Language Models (LLMs), which have
significantly enhanced the generative capabilities for various NLP tasks, LLMs
still face limitations in directly handling retrieval tasks. However, many
practical applications demand the seamless integration of both retrieval and
generation. This paper introduces a novel and efficient One-pass Generation and
retrieval framework (OneGen), designed to improve LLMs' performance on tasks
that require both generation and retrieval. The proposed framework bridges the
traditionally separate training approaches for generation and retrieval by
incorporating retrieval tokens generated autoregressively. This enables a
single LLM to handle both tasks simultaneously in a unified forward pass. We
conduct experiments on two distinct types of composite tasks, RAG and Entity
Linking, to validate the pluggability, effectiveness, and efficiency of OneGen
in training and inference. Furthermore, our results show that integrating
generation and retrieval within the same context preserves the generative
capabilities of LLMs while improving retrieval performance. To the best of our
knowledge, OneGen is the first to enable LLMs to conduct vector retrieval
during the generation.
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
Authors: name: Run Luo, name: Haonan Zhang, name: Longze Chen, name: Ting-En Lin, name: Xiong Liu, name: Yuchuan Wu, name: Min Yang, name: Minzheng Wang, name: Pengpeng Zeng, name: Lianli Gao, name: Heng Tao Shen, name: Yunshui Li, name: Xiaobo Xia, name: Fei Huang, name: Jingkuan Song, name: Yongbin Li
Published: 2024-09-09 17:44:00

The development of Multimodal Large Language Models (MLLMs) has seen
significant advancements. However, the quantity and quality of multimodal
instruction data have emerged as significant bottlenecks in their progress.
Manually creating multimodal instruction data is both time-consuming and
inefficient, posing challenges in producing instructions of high complexity.
Moreover, distilling instruction data from black-box commercial models (e.g.,
GPT-4o, GPT-4V) often results in simplistic instruction data, which constrains
performance to that of these models. The challenge of curating diverse and
complex instruction data remains substantial. We propose MMEvol, a novel
multimodal instruction data evolution framework that combines fine-grained
perception evolution, cognitive reasoning evolution, and interaction evolution.
This iterative approach breaks through data quality bottlenecks to generate a
complex and diverse image-text instruction dataset, thereby empowering MLLMs
with enhanced capabilities. Beginning with an initial set of instructions,
SEED-163K, we utilize MMEvol to systematically broadens the diversity of
instruction types, integrates reasoning steps to enhance cognitive
capabilities, and extracts detailed information from images to improve visual
understanding and robustness. To comprehensively evaluate the effectiveness of
our data, we train LLaVA-NeXT using the evolved data and conduct experiments
across 13 vision-language tasks. Compared to the baseline trained with seed
data, our approach achieves an average accuracy improvement of 3.1 points and
reaches state-of-the-art (SOTA) performance on 9 of these tasks.
Towards a Unified View of Preference Learning for Large Language Models:
  A Survey
Authors: name: Bofei Gao, name: Feifan Song, name: Yibo Miao, name: Zefan Cai, name: Zhe Yang, name: Liang Chen, name: Helan Hu, name: Runxin Xu, name: Qingxiu Dong, name: Ce Zheng, name: Wen Xiao, name: Ge Zhang, name: Daoguang Zan, name: Keming Lu, name: Bowen Yu, name: Dayiheng Liu, name: Zeyu Cui, name: Jian Yang, name: Lei Sha, name: Houfeng Wang, name: Zhifang Sui, name: Peiyi Wang, name: Tianyu Liu, name: Baobao Chang
Published: 2024-09-04 15:11:55

Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of
the crucial factors to achieve success is aligning the LLM's output with human
preferences. This alignment process often requires only a small amount of data
to efficiently enhance the LLM's performance. While effective, research in this
area spans multiple domains, and the methods involved are relatively complex to
understand. The relationships between different methods have been
under-explored, limiting the development of the preference alignment. In light
of this, we break down the existing popular alignment strategies into different
components and provide a unified framework to study the current alignment
strategies, thereby establishing connections among them. In this survey, we
decompose all the strategies in preference learning into four components:
model, data, feedback, and algorithm. This unified view offers an in-depth
understanding of existing alignment algorithms and also opens up possibilities
to synergize the strengths of different strategies. Furthermore, we present
detailed working examples of prevalent existing algorithms to facilitate a
comprehensive understanding for the readers. Finally, based on our unified
perspective, we explore the challenges and future research directions for
aligning large language models with human preferences.
Spinning the Golden Thread: Benchmarking Long-Form Generation in
  Language Models
Authors: name: Yuhao Wu, name: Ming Shan Hee, name: Zhiqing Hu, name: Roy Ka-Wei Lee
Published: 2024-09-03 17:25:54

The abilities of long-context language models (LMs) are often evaluated using
the "Needle-in-a-Haystack" (NIAH) test, which comprises tasks designed to
assess a model's ability to identify specific information ("needle") within
large text sequences ("haystack"). While these benchmarks measure how well
models understand long-context input sequences, they do not effectively gauge
the quality of long-form text generation--a critical aspect for applications
such as design proposals and creative writing. To address this gap, we have
introduced a new long-form text evaluation benchmark, Spinning the Golden
Thread (SGT), which tests models' ability to identify specific events within
generated long text sequences. In this benchmark, we prompt long-context LMs to
create long-form text that must include particular events or constraints and
evaluate their ability to incorporate these elements. We evaluated ten
long-context LMs across four distinct scenarios, three types of prompt
instructions, and two different generation-length settings (16K and 32K).
Although these models perform well on NIAH benchmarks, none demonstrated
satisfactory performance on the Spinning the Golden Thread, raising concerns
about their ability to generate coherent long-form text that follows
instructions. Additionally, as the length of the generated text increases, all
models exhibit a significant drop in performance.
GST: Precise 3D Human Body from a Single Image with Gaussian Splatting
  Transformers
Authors: name: Lorenza Prospero, name: Abdullah Hamdi, name: Joao F. Henriques, name: Christian Rupprecht
Published: 2024-09-06 11:34:24

Reconstructing realistic 3D human models from monocular images has
significant applications in creative industries, human-computer interfaces, and
healthcare. We base our work on 3D Gaussian Splatting (3DGS), a scene
representation composed of a mixture of Gaussians. Predicting such mixtures for
a human from a single input image is challenging, as it is a non-uniform
density (with a many-to-one relationship with input pixels) with strict
physical constraints. At the same time, it needs to be flexible to accommodate
a variety of clothes and poses. Our key observation is that the vertices of
standardized human meshes (such as SMPL) can provide an adequate density and
approximate initial position for Gaussians. We can then train a transformer
model to jointly predict comparatively small adjustments to these positions, as
well as the other Gaussians' attributes and the SMPL parameters. We show
empirically that this combination (using only multi-view supervision) can
achieve fast inference of 3D human models from a single image without test-time
optimization, expensive diffusion models, or 3D points supervision. We also
show that it can improve 3D pose estimation by better fitting human models that
account for clothes and other variations. The code is available on the project
website https://abdullahamdi.com/gst/ .
Configurable Foundation Models: Building LLMs from a Modular Perspective
Authors: name: Chaojun Xiao, name: Zhengyan Zhang, name: Chenyang Song, name: Dazhi Jiang, name: Feng Yao, name: Xu Han, name: Xiaozhi Wang, name: Shuo Wang, name: Yufei Huang, name: Guanyu Lin, name: Yingfa Chen, name: Weilin Zhao, name: Yuge Tu, name: Zexuan Zhong, name: Ao Zhang, name: Chenglei Si, name: Khai Hao Moo, name: Chenyang Zhao, name: Huimin Chen, name: Yankai Lin, name: Zhiyuan Liu, name: Jingbo Shang, name: Maosong Sun
Published: 2024-09-04 17:01:02

Advancements in LLMs have recently unveiled challenges tied to computational
efficiency and continual scalability due to their requirements of huge
parameters, making the applications and evolution of these models on devices
with limited computation resources and scenarios requiring various abilities
increasingly cumbersome. Inspired by modularity within the human brain, there
is a growing tendency to decompose LLMs into numerous functional modules,
allowing for inference with part of modules and dynamic assembly of modules to
tackle complex tasks, such as mixture-of-experts. To highlight the inherent
efficiency and composability of the modular approach, we coin the term brick to
represent each functional module, designating the modularized structure as
configurable foundation models. In this paper, we offer a comprehensive
overview and investigation of the construction, utilization, and limitation of
configurable foundation models. We first formalize modules into emergent bricks
- functional neuron partitions that emerge during the pre-training phase, and
customized bricks - bricks constructed via additional post-training to improve
the capabilities and knowledge of LLMs. Based on diverse functional bricks, we
further present four brick-oriented operations: retrieval and routing, merging,
updating, and growing. These operations allow for dynamic configuration of LLMs
based on instructions to handle complex tasks. To verify our perspective, we
conduct an empirical analysis on widely-used LLMs. We find that the FFN layers
follow modular patterns with functional specialization of neurons and
functional neuron partitions. Finally, we highlight several open issues and
directions for future research. Overall, this paper aims to offer a fresh
modular perspective on existing LLM research and inspire the future creation of
more efficient and scalable foundational models.
Open-MAGVIT2: An Open-Source Project Toward Democratizing
  Auto-regressive Visual Generation
Authors: name: Zhuoyan Luo, name: Fengyuan Shi, name: Yixiao Ge, name: Yujiu Yang, name: Limin Wang, name: Ying Shan
Published: 2024-09-06 17:14:53

We present Open-MAGVIT2, a family of auto-regressive image generation models
ranging from 300M to 1.5B. The Open-MAGVIT2 project produces an open-source
replication of Google's MAGVIT-v2 tokenizer, a tokenizer with a super-large
codebook (i.e., 2^{18} codes), and achieves the state-of-the-art
reconstruction performance (1.17 rFID) on ImageNet 256 times 256.
Furthermore, we explore its application in plain auto-regressive models and
validate scalability properties. To assist auto-regressive models in predicting
with a super-large vocabulary, we factorize it into two sub-vocabulary of
different sizes by asymmetric token factorization, and further introduce "next
sub-token prediction" to enhance sub-token interaction for better generation
quality. We release all models and codes to foster innovation and creativity in
the field of auto-regressive visual generation.
How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with
  High-Quality Data
Authors: name: Yejie Wang, name: Keqing He, name: Dayuan Fu, name: Zhuoma Gongque, name: Heyang Xu, name: Yanxu Chen, name: Zhexu Wang, name: Yujia Fu, name: Guanting Dong, name: Muxi Diao, name: Jingang Wang, name: Mengdi Zhang, name: Xunliang Cai, name: Weiran Xu
Published: 2024-09-05 17:46:30

Recently, there has been a growing interest in studying how to construct
better code instruction tuning data. However, we observe Code models trained
with these datasets exhibit high performance on HumanEval but perform worse on
other benchmarks such as LiveCodeBench. Upon further investigation, we find
that many datasets suffer from severe data leakage. After cleaning up most of
the leaked data, some well-known high-quality datasets perform poorly. This
discovery reveals a new challenge: identifying which dataset genuinely qualify
as high-quality code instruction data. To address this, we propose an efficient
code data pruning strategy for selecting good samples. Our approach is based on
three dimensions: instruction complexity, response quality, and instruction
diversity. Based on our selected data, we present XCoder, a family of models
finetuned from LLaMA3. Our experiments show XCoder achieves new
state-of-the-art performance using fewer training data, which verify the
effectiveness of our data strategy. Moreover, we perform a comprehensive
analysis on the data composition and find existing code datasets have different
characteristics according to their construction methods, which provide new
insights for future code LLMs. Our models and dataset are released in
https://github.com/banksy23/XCoder
Qihoo-T2X: An Efficiency-Focused Diffusion Transformer via Proxy Tokens
  for Text-to-Any-Task
Authors: name: Jing Wang, name: Ao Ma, name: Jiasong Feng, name: Dawei Leng, name: Yuhui Yin, name: Xiaodan Liang
Published: 2024-09-06 03:13:45

The global self-attention mechanism in diffusion transformers involves
redundant computation due to the sparse and redundant nature of visual
information, and the attention map of tokens within a spatial window shows
significant similarity. To address this redundancy, we propose the Proxy Token
Diffusion Transformer (PT-DiT), which employs sparse representative token
attention (where the number of representative tokens is much smaller than the
total number of tokens) to model global visual information efficiently.
Specifically, in each transformer block, we randomly sample one token from each
spatial-temporal window to serve as a proxy token for that region. The global
semantics are captured through the self-attention of these proxy tokens and
then injected into all latent tokens via cross-attention. Simultaneously, we
introduce window and shift window attention to address the limitations in
detail modeling caused by the sparse attention mechanism. Building on the
well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a
variety of models for T2I, T2V, and T2MV tasks. Experimental results show that
PT-DiT achieves competitive performance while reducing the computational
complexity in both image and video generation tasks (e.g., a 48% reduction
compared to DiT and a 35% reduction compared to Pixart-alpha). Our source code
is available at https://github.com/360CVGroup/Qihoo-T2X.