Orionnnn/hub-stats huggingface dataset:
Modulated Intervention Preference Optimization (MIPO): Keep the Easy,
  Refine the Difficult
Authors: name: Cheolhun Jang
Published: 2024-09-26 05:24:14

Preference optimization methods typically begin training with a well-trained
SFT model as a reference model. In RLHF and DPO, a regularization term is used
during the preference optimization process to prevent the policy model from
deviating too far from the reference model's distribution, thereby avoiding the
generation of anomalous responses. When the reference model is already
well-aligned with the given data or only requires slight adjustments, this
approach can produce a well-aligned model. However, if the reference model is
not aligned with the given data and requires significant deviation from its
current state, a regularization term may actually hinder the model alignment.
In this study, we propose Modulated Intervention Preference
Optimization (MIPO) to address this issue. MIPO modulates the degree of
intervention from the reference model based on how well the given data is
aligned with it. If the data is well-aligned, the intervention is increased to
prevent the policy model from diverging significantly from reference model.
Conversely, if the alignment is poor, the interference is reduced to facilitate
more extensive training. We compare the performance of MIPO and DPO using
Mistral-7B and Llama3-8B in Alpaca Eval 2.0 and MT-Bench. The experimental
results demonstrate that MIPO consistently outperforms DPO across various
evaluation scenarios.
MinerU: An Open-Source Solution for Precise Document Content Extraction
Authors: name: Bin Wang, name: Chao Xu, name: Xiaomeng Zhao, name: Linke Ouyang, name: Fan Wu, name: Zhiyuan Zhao, name: Rui Xu, name: Kaiwen Liu, name: Yuan Qu, name: Fukai Shang, name: Bo Zhang, name: Liqun Wei, name: Zhihao Sui, name: Wei Li, name: Botian Shi, name: Yu Qiao, name: Dahua Lin, name: Conghui He
Published: 2024-09-27 15:35:15

Document content analysis has been a crucial research area in computer
vision. Despite significant advancements in methods such as OCR, layout
detection, and formula recognition, existing open-source solutions struggle to
consistently deliver high-quality content extraction due to the diversity in
document types and content. To address these challenges, we present MinerU, an
open-source solution for high-precision document content extraction. MinerU
leverages the sophisticated PDF-Extract-Kit models to extract content from
diverse documents effectively and employs finely-tuned preprocessing and
postprocessing rules to ensure the accuracy of the final results. Experimental
results demonstrate that MinerU consistently achieves high performance across
various document types, significantly enhancing the quality and consistency of
content extraction. The MinerU open-source project is available at
https://github.com/opendatalab/MinerU.
Emu3: Next-Token Prediction is All You Need
Authors: name: Xinlong Wang, name: Xiaosong Zhang, name: Zhengxiong Luo, name: Quan Sun, name: Yufeng Cui, name: Jinsheng Wang, name: Fan Zhang, name: Yueze Wang, name: Zhen Li, name: Qiying Yu, name: Yingli Zhao, name: Yulong Ao, name: Xuebin Min, name: Tao Li, name: Boya Wu, name: Bo Zhao, name: Bowen Zhang, name: Liangdong Wang, name: Guang Liu, name: Zheqi He, name: Xi Yang, name: Jingjing Liu, name: Yonghua Lin, name: Tiejun Huang, name: Zhongyuan Wang
Published: 2024-09-27 16:06:11

While next-token prediction is considered a promising path towards artificial
general intelligence, it has struggled to excel in multimodal tasks, which are
still dominated by diffusion models (e.g., Stable Diffusion) and compositional
approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a
new suite of state-of-the-art multimodal models trained solely with next-token
prediction. By tokenizing images, text, and videos into a discrete space, we
train a single transformer from scratch on a mixture of multimodal sequences.
Emu3 outperforms several well-established task-specific models in both
generation and perception tasks, surpassing flagship models such as SDXL and
LLaVA-1.6, while eliminating the need for diffusion or compositional
architectures. Emu3 is also capable of generating high-fidelity video via
predicting the next token in a video sequence. We simplify complex multimodal
model designs by converging on a singular focus: tokens, unlocking great
potential for scaling both during training and inference. Our results
demonstrate that next-token prediction is a promising path towards building
general multimodal intelligence beyond language. We open-source key techniques
and models to support further research in this direction.
PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
Authors: name: Shaowei Liu, name: Zhongzheng Ren, name: Saurabh Gupta, name: Shenlong Wang
Published: 2024-09-27 17:59:57

We present PhysGen, a novel image-to-video generation method that converts a
single image and an input condition (e.g., force and torque applied to an
object in the image) to produce a realistic, physically plausible, and
temporally consistent video. Our key insight is to integrate model-based
physical simulation with a data-driven video generation process, enabling
plausible image-space dynamics. At the heart of our system are three core
components: (i) an image understanding module that effectively captures the
geometry, materials, and physical parameters of the image; (ii) an image-space
dynamics simulation model that utilizes rigid-body physics and inferred
parameters to simulate realistic behaviors; and (iii) an image-based rendering
and refinement module that leverages generative video diffusion to produce
realistic video footage featuring the simulated motion. The resulting videos
are realistic in both physics and appearance and are even precisely
controllable, showcasing superior results over existing data-driven
image-to-video generation works through quantitative comparison and
comprehensive user study. PhysGen's resulting videos can be used for various
downstream applications, such as turning an image into a realistic animation or
allowing users to interact with the image and create various dynamics. Project
page: https://stevenlsw.github.io/physgen/
MIO: A Foundation Model on Multimodal Tokens
Authors: name: Zekun Wang, name: King Zhu, name: Chunpu Xu, name: Wangchunshu Zhou, name: Jiaheng Liu, name: Yibo Zhang, name: Jiashuo Wang, name: Ning Shi, name: Siyu Li, name: Yizhi Li, name: Haoran Que, name: Zhaoxiang Zhang, name: Yuanxing Zhang, name: Ge Zhang, name: Ke Xu, name: Jie Fu, name: Wenhao Huang
Published: 2024-09-26 09:57:16

In this paper, we introduce MIO, a novel foundation model built on multimodal
tokens, capable of understanding and generating speech, text, images, and
videos in an end-to-end, autoregressive manner. While the emergence of large
language models (LLMs) and multimodal large language models (MM-LLMs) propels
advancements in artificial general intelligence through their versatile
capabilities, they still lack true any-to-any understanding and generation.
Recently, the release of GPT-4o has showcased the remarkable potential of
any-to-any LLMs for complex real-world tasks, enabling omnidirectional input
and output across images, speech, and text. However, it is closed-source and
does not support the generation of multimodal interleaved sequences. To address
this gap, we present MIO, which is trained on a mixture of discrete tokens
across four modalities using causal multimodal modeling. MIO undergoes a
four-stage training process: (1) alignment pre-training, (2) interleaved
pre-training, (3) speech-enhanced pre-training, and (4) comprehensive
supervised fine-tuning on diverse textual, visual, and speech tasks. Our
experimental results indicate that MIO exhibits competitive, and in some cases
superior, performance compared to previous dual-modal baselines, any-to-any
model baselines, and even modality-specific baselines. Moreover, MIO
demonstrates advanced capabilities inherent to its any-to-any feature, such as
interleaved video-text generation, chain-of-visual-thought reasoning, visual
guideline generation, instructional image editing, etc.
VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large
  Language Models
Authors: name: Yifei Liu, name: Jicheng Wen, name: Yang Wang, name: Shengyu Ye, name: Li Lyna Zhang, name: Ting Cao, name: Cheng Li, name: Mao Yang
Published: 2024-09-25 16:25:45

Scaling model size significantly challenges the deployment and inference of
Large Language Models (LLMs). Due to the redundancy in LLM weights, recent
research has focused on pushing weight-only quantization to extremely low-bit
(even down to 2 bits). It reduces memory requirements, optimizes storage costs,
and decreases memory bandwidth needs during inference. However, due to
numerical representation limitations, traditional scalar-based weight
quantization struggles to achieve such extreme low-bit. Recent research on
Vector Quantization (VQ) for LLMs has demonstrated the potential for extremely
low-bit model quantization by compressing vectors into indices using lookup
tables.
  In this paper, we introduce Vector Post-Training Quantization (VPTQ) for
extremely low-bit quantization of LLMs. We use Second-Order Optimization to
formulate the LLM VQ problem and guide our quantization algorithm design by
solving the optimization. We further refine the weights using
Channel-Independent Second-Order Optimization for a granular VQ. In addition,
by decomposing the optimization problem, we propose a brief and effective
codebook initialization algorithm. We also extend VPTQ to support residual and
outlier quantization, which enhances model accuracy and further compresses the
model. Our experimental results show that VPTQ reduces model quantization
perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on Mistral-7B,
4.41-7.34 on LLaMA-3 over SOTA at 2-bit, with an average accuracy
improvement of 0.79-1.5% on LLaMA-2, 1% on Mistral-7B, 11-22% on
LLaMA-3 on QA tasks on average. We only utilize 10.4-18.6% of the
quantization algorithm execution time, resulting in a 1.6-1.8times
increase in inference throughput compared to SOTA.
MSI-Agent: Incorporating Multi-Scale Insight into Embodied Agents for
  Superior Planning and Decision-Making
Authors: name: Dayuan Fu, name: Biqing Qi, name: Yihuai Gao, name: Che Jiang, name: Guanting Dong, name: Bowen Zhou
Published: 2024-09-25 07:21:51

Long-term memory is significant for agents, in which insights play a crucial
role. However, the emergence of irrelevant insight and the lack of general
insight can greatly undermine the effectiveness of insight. To solve this
problem, in this paper, we introduce Multi-Scale Insight Agent (MSI-Agent), an
embodied agent designed to improve LLMs' planning and decision-making ability
by summarizing and utilizing insight effectively across different scales. MSI
achieves this through the experience selector, insight generator, and insight
selector. Leveraging a three-part pipeline, MSI can generate task-specific and
high-level insight, store it in a database, and then use relevant insight from
it to aid in decision-making. Our experiments show that MSI outperforms another
insight strategy when planning by GPT3.5. Moreover, We delve into the
strategies for selecting seed experience and insight, aiming to provide LLM
with more useful and relevant insight for better decision-making. Our
observations also indicate that MSI exhibits better robustness when facing
domain-shifting scenarios.
The Imperative of Conversation Analysis in the Era of LLMs: A Survey of
  Tasks, Techniques, and Trends
Authors: name: Xinghua Zhang, name: Haiyang Yu, name: Yongbin Li, name: Minzheng Wang, name: Longze Chen, name: Fei Huang
Published: 2024-09-21 16:52:43

In the era of large language models (LLMs), a vast amount of conversation
logs will be accumulated thanks to the rapid development trend of language UI.
Conversation Analysis (CA) strives to uncover and analyze critical information
from conversation data, streamlining manual processes and supporting business
insights and decision-making. The need for CA to extract actionable insights
and drive empowerment is becoming increasingly prominent and attracting
widespread attention. However, the lack of a clear scope for CA leads to a
dispersion of various techniques, making it difficult to form a systematic
technical synergy to empower business applications. In this paper, we perform a
thorough review and systematize CA task to summarize the existing related work.
Specifically, we formally define CA task to confront the fragmented and chaotic
landscape in this field, and derive four key steps of CA from conversation
scene reconstruction, to in-depth attribution analysis, and then to performing
targeted training, finally generating conversations based on the targeted
training for achieving the specific goals. In addition, we showcase the
relevant benchmarks, discuss potential challenges and point out future
directions in both industry and academia. In view of current advancements, it
is evident that the majority of efforts are still concentrated on the analysis
of shallow conversation elements, which presents a considerable gap between the
research and business, and with the assist of LLMs, recent work has shown a
trend towards research on causality and strategic tasks which are sophisticated
and high-level. The analyzed experiences and insights will inevitably have
broader application value in business operations that target conversation logs.
Instruction Following without Instruction Tuning
Authors: name: John Hewitt, name: Nelson F. Liu, name: Percy Liang, name: Christopher D. Manning
Published: 2024-09-21 22:36:22

Instruction tuning commonly means finetuning a language model on
instruction-response pairs. We discover two forms of adaptation (tuning) that
are deficient compared to instruction tuning, yet still yield instruction
following; we call this implicit instruction tuning. We first find that
instruction-response pairs are not necessary: training solely on responses,
without any corresponding instructions, yields instruction following. This
suggests pretrained models have an instruction-response mapping which is
revealed by teaching the model the desired distribution of responses. However,
we then find it's not necessary to teach the desired distribution of responses:
instruction-response training on narrow-domain data like poetry still leads to
broad instruction-following behavior like recipe generation. In particular,
when instructions are very different from those in the narrow finetuning
domain, models' responses do not adhere to the style of the finetuning domain.
To begin to explain implicit instruction tuning, we hypothesize that very
simple changes to a language model's distribution yield instruction following.
We support this by hand-writing a rule-based language model which yields
instruction following in a product-of-experts with a pretrained model. The
rules are to slowly increase the probability of ending the sequence, penalize
repetition, and uniformly change 15 words' probabilities. In summary,
adaptations made without being designed to yield instruction following can do
so implicitly.
Enhancing Structured-Data Retrieval with GraphRAG: Soccer Data Case
  Study
Authors: name: Zahra Sepasdar, name: Sushant Gautam, name: Cise Midoglu, name: Michael A. Riegler, name: Pål Halvorsen
Published: 2024-09-26 06:53:29

Extracting meaningful insights from large and complex datasets poses
significant challenges, particularly in ensuring the accuracy and relevance of
retrieved information. Traditional data retrieval methods such as sequential
search and index-based retrieval often fail when handling intricate and
interconnected data structures, resulting in incomplete or misleading outputs.
To overcome these limitations, we introduce Structured-GraphRAG, a versatile
framework designed to enhance information retrieval across structured datasets
in natural language queries. Structured-GraphRAG utilizes multiple knowledge
graphs, which represent data in a structured format and capture complex
relationships between entities, enabling a more nuanced and comprehensive
retrieval of information. This graph-based approach reduces the risk of errors
in language model outputs by grounding responses in a structured format,
thereby enhancing the reliability of results. We demonstrate the effectiveness
of Structured-GraphRAG by comparing its performance with that of a recently
published method using traditional retrieval-augmented generation. Our findings
show that Structured-GraphRAG significantly improves query processing
efficiency and reduces response times. While our case study focuses on soccer
data, the framework's design is broadly applicable, offering a powerful tool
for data analysis and enhancing language model applications across various
structured domains.
Reducing the Footprint of Multi-Vector Retrieval with Minimal
  Performance Impact via Token Pooling
Authors: name: Benjamin Clavié, name: Antoine Chaffin, name: Griffin Adams
Published: 2024-09-23 03:12:43

Over the last few years, multi-vector retrieval methods, spearheaded by
ColBERT, have become an increasingly popular approach to Neural IR. By storing
representations at the token level rather than at the document level, these
methods have demonstrated very strong retrieval performance, especially in
out-of-domain settings. However, the storage and memory requirements necessary
to store the large number of associated vectors remain an important drawback,
hindering practical adoption. In this paper, we introduce a simple
clustering-based token pooling approach to aggressively reduce the number of
vectors that need to be stored. This method can reduce the space & memory
footprint of ColBERT indexes by 50% with virtually no retrieval performance
degradation. This method also allows for further reductions, reducing the
vector count by 66%-to-75% , with degradation remaining below 5% on a vast
majority of datasets. Importantly, this approach requires no architectural
change nor query-time processing, and can be used as a simple drop-in during
indexation with any ColBERT-like model.
Pixel-Space Post-Training of Latent Diffusion Models
Authors: name: Christina Zhang, name: Simran Motwani, name: Matthew Yu, name: Ji Hou, name: Felix Juefei-Xu, name: Sam Tsai, name: Peter Vajda, name: Zijian He, name: Jialiang Wang
Published: 2024-09-26 06:27:26

Latent diffusion models (LDMs) have made significant advancements in the
field of image generation in recent years. One major advantage of LDMs is their
ability to operate in a compressed latent space, allowing for more efficient
training and deployment. However, despite these advantages, challenges with
LDMs still remain. For example, it has been observed that LDMs often generate
high-frequency details and complex compositions imperfectly. We hypothesize
that one reason for these flaws is due to the fact that all pre- and
post-training of LDMs are done in latent space, which is typically 8 times 8
lower spatial-resolution than the output images. To address this issue, we
propose adding pixel-space supervision in the post-training process to better
preserve high-frequency details. Experimentally, we show that adding a
pixel-space objective significantly improves both supervised quality
fine-tuning and preference-based post-training by a large margin on a
state-of-the-art DiT transformer and U-Net diffusion models in both visual
quality and visual flaw metrics, while maintaining the same text alignment
quality.
LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with
  3D-awareness
Authors: name: Chenming Zhu, name: Tai Wang, name: Wenwei Zhang, name: Jiangmiao Pang, name: Xihui Liu
Published: 2024-09-26 17:59:11

Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced
their proficiency in 2D visual understanding tasks, enabling them to
effectively process and understand images and videos. However, the development
of LMMs with 3D-awareness for 3D scene understanding has been hindered by the
lack of large-scale 3D vision-language datasets and powerful 3D encoders. In
this paper, we introduce a simple yet effective framework called LLaVA-3D.
Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D
efficiently adapts LLaVA for 3D scene understanding without compromising 2D
understanding capabilities. To achieve this, we employ a simple yet effective
representation, 3D Patch, which connects 2D CLIP patch features with their
corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs
and employing joint 2D and 3D vision-language instruction tuning, we establish
a unified architecture for both 2D image understanding and 3D scene
understanding. Experimental results show that LLaVA-3D converges 3.5x faster
than existing 3D LMMs when trained on 3D vision-language datasets. Moreover,
LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks
but also maintains comparable 2D image understanding and vision-language
conversation capabilities with LLaVA.
Lotus: Diffusion-based Visual Foundation Model for High-quality Dense
  Prediction
Authors: name: Jing He, name: Haodong Li, name: Wei Yin, name: Yixun Liang, name: Leheng Li, name: Kaiqiang Zhou, name: Hongbo Liu, name: Bingbing Liu, name: Ying-Cong Chen
Published: 2024-09-26 17:58:55

Leveraging the visual priors of pre-trained text-to-image diffusion models
offers a promising solution to enhance zero-shot generalization in dense
prediction tasks. However, existing methods often uncritically use the original
diffusion formulation, which may not be optimal due to the fundamental
differences between dense prediction and image generation. In this paper, we
provide a systemic analysis of the diffusion formulation for the dense
prediction, focusing on both quality and efficiency. And we find that the
original parameterization type for image generation, which learns to predict
noise, is harmful for dense prediction; the multi-step noising/denoising
diffusion process is also unnecessary and challenging to optimize. Based on
these insights, we introduce Lotus, a diffusion-based visual foundation model
with a simple yet effective adaptation protocol for dense prediction.
Specifically, Lotus is trained to directly predict annotations instead of
noise, thereby avoiding harmful variance. We also reformulate the diffusion
process into a single-step procedure, simplifying optimization and
significantly boosting inference speed. Additionally, we introduce a novel
tuning strategy called detail preserver, which achieves more accurate and
fine-grained predictions. Without scaling up the training data or model
capacity, Lotus achieves SoTA performance in zero-shot depth and normal
estimation across various datasets. It also significantly enhances efficiency,
being hundreds of times faster than most existing diffusion-based methods.