Orionnnn/hub-stats huggingface dataset:
Roadmap towards Superhuman Speech Understanding using Large Language
  Models
Authors: name: Fan Bu, name: Yuhao Zhang, name: Xidong Wang, name: Benyou Wang, name: Qun Liu, name: Haizhou Li
Published: 2024-10-17 06:44:06

The success of large language models (LLMs) has prompted efforts to integrate
speech and audio data, aiming to create general foundation models capable of
processing both textual and non-textual inputs. Recent advances, such as
GPT-4o, highlight the potential for end-to-end speech LLMs, which preserves
non-semantic information and world knowledge for deeper speech understanding.
To guide the development of speech LLMs, we propose a five-level roadmap,
ranging from basic automatic speech recognition (ASR) to advanced superhuman
models capable of integrating non-semantic information with abstract acoustic
knowledge for complex tasks. Moreover, we design a benchmark, SAGI Bechmark,
that standardizes critical aspects across various tasks in these five levels,
uncovering challenges in using abstract acoustic knowledge and completeness of
capability. Our findings reveal gaps in handling paralinguistic cues and
abstract acoustic knowledge, and we offer future directions. This paper
outlines a roadmap for advancing speech LLMs, introduces a benchmark for
evaluation, and provides key insights into their current limitations and
potential.
MobA: A Two-Level Agent System for Efficient Mobile Task Automation
Authors: name: Zichen Zhu, name: Hao Tang, name: Yansi Li, name: Kunyao Lan, name: Yixuan Jiang, name: Hao Zhou, name: Yixiao Wang, name: Situo Zhang, name: Liangtai Sun, name: Lu Chen, name: Kai Yu
Published: 2024-10-17 16:53:50

Current mobile assistants are limited by dependence on system APIs or
struggle with complex user instructions and diverse interfaces due to
restricted comprehension and decision-making abilities. To address these
challenges, we propose MobA, a novel Mobile phone Agent powered by multimodal
large language models that enhances comprehension and planning capabilities
through a sophisticated two-level agent architecture. The high-level Global
Agent (GA) is responsible for understanding user commands, tracking history
memories, and planning tasks. The low-level Local Agent (LA) predicts detailed
actions in the form of function calls, guided by sub-tasks and memory from the
GA. Integrating a Reflection Module allows for efficient task completion and
enables the system to handle previously unseen complex tasks. MobA demonstrates
significant improvements in task execution efficiency and completion rate in
real-life evaluations, underscoring the potential of MLLM-empowered mobile
assistants.
γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large
  Language Models
Authors: name: Yaxin Luo, name: Gen Luo, name: Jiayi Ji, name: Yiyi Zhou, name: Xiaoshuai Sun, name: Zhiqiang Shen, name: Rongrong Ji
Published: 2024-10-17 17:59:53

Despite the significant progress in multimodal large language models (MLLMs),
their high computational cost remains a barrier to real-world deployment.
Inspired by the mixture of depths (MoDs) in natural language processing, we aim
to address this limitation from the perspective of ``activated tokens''. Our
key insight is that if most tokens are redundant for the layer computation,
then can be skipped directly via the MoD layer. However, directly converting
the dense layers of MLLMs to MoD layers leads to substantial performance
degradation. To address this issue, we propose an innovative MoD adaptation
strategy for existing MLLMs called gamma-MoD. In gamma-MoD, a novel
metric is proposed to guide the deployment of MoDs in the MLLM, namely rank of
attention maps (ARank). Through ARank, we can effectively identify which layer
is redundant and should be replaced with the MoD layer. Based on ARank, we
further propose two novel designs to maximize the computational sparsity of
MLLM while maintaining its performance, namely shared vision-language router
and masked routing learning. With these designs, more than 90% dense layers of
the MLLM can be effectively converted to the MoD ones. To validate our method,
we apply it to three popular MLLMs, and conduct extensive experiments on 9
benchmark datasets. Experimental results not only validate the significant
efficiency benefit of gamma-MoD to existing MLLMs but also confirm its
generalization ability on various MLLMs. For example, with a minor performance
drop, i.e., -1.5%, gamma-MoD can reduce the training and inference time of
LLaVA-HR by 31.0% and 53.2%, respectively.
Minimum Tuning to Unlock Long Output from LLMs with High Quality Data as
  the Key
Authors: name: Yingda Chen, name: Xingjun Wang, name: Jintao Huang, name: Yunlin Mao, name: Daoze Zhang, name: Yuze Zhao
Published: 2024-10-14 07:09:02

As large language models rapidly evolve to support longer context, there is a
notable disparity in their capability to generate output at greater lengths.
Recent study suggests that the primary cause for this imbalance may arise from
the lack of data with long-output during alignment training. In light of this
observation, attempts are made to re-align foundation models with data that
fills the gap, which result in models capable of generating lengthy output when
instructed. In this paper, we explore the impact of data-quality in tuning a
model for long output, and the possibility of doing so from the starting points
of human-aligned (instruct or chat) models. With careful data curation, we show
that it possible to achieve similar performance improvement in our tuned
models, with only a small fraction of training data instances and compute. In
addition, we assess the generalizability of such approaches by applying our
tuning-recipes to several models. our findings suggest that, while capacities
for generating long output vary across different models out-of-the-box, our
approach to tune them with high-quality data using lite compute, consistently
yields notable improvement across all models we experimented on. We have made
public our curated dataset for tuning long-writing capability, the
implementations of model tuning and evaluation, as well as the fine-tuned
models, all of which can be openly-accessed.
Toward Guidance-Free AR Visual Generation via Condition Contrastive
  Alignment
Authors: name: Huayu Chen, name: Hang Su, name: Peize Sun, name: Jun Zhu
Published: 2024-10-12 03:31:25

Classifier-Free Guidance (CFG) is a critical technique for enhancing the
sample quality of visual generative models. However, in autoregressive (AR)
multi-modal generation, CFG introduces design inconsistencies between language
and visual content, contradicting the design philosophy of unifying different
modalities for visual AR. Motivated by language model alignment methods, we
propose Condition Contrastive Alignment (CCA) to facilitate
guidance-free AR visual generation with high performance and analyze its
theoretical connection with guided sampling methods. Unlike guidance methods
that alter the sampling process to achieve the ideal sampling distribution, CCA
directly fine-tunes pretrained models to fit the same distribution target.
Experimental results show that CCA can significantly enhance the guidance-free
performance of all tested models with just one epoch of fine-tuning (sim 1\%
of pretraining epochs) on the pretraining dataset, on par with guided sampling
methods. This largely removes the need for guided sampling in AR visual
generation and cuts the sampling cost by half. Moreover, by adjusting training
parameters, CCA can achieve trade-offs between sample diversity and fidelity
similar to CFG. This experimentally confirms the strong theoretical connection
between language-targeted alignment and visual-targeted guidance methods,
unifying two previously independent research fields. Code and model weights:
https://github.com/thu-ml/CCA.
TransAgent: Transfer Vision-Language Foundation Models with
  Heterogeneous Agent Collaboration
Authors: name: Yiwei Guo, name: Shaobin Zhuang, name: Kunchang Li, name: Yu Qiao, name: Yali Wang
Published: 2024-10-16 03:01:44

Vision-language foundation models (such as CLIP) have recently shown their
power in transfer learning, owing to large-scale image-text pre-training.
However, target domain data in the downstream tasks can be highly different
from the pre-training phase, which makes it hard for such a single model to
generalize well. Alternatively, there exists a wide range of expert models that
contain diversified vision and/or language knowledge pre-trained on different
modalities, tasks, networks, and datasets. Unfortunately, these models are
"isolated agents" with heterogeneous structures, and how to integrate their
knowledge for generalizing CLIP-like models has not been fully explored. To
bridge this gap, we propose a general and concise TransAgent framework, which
transports the knowledge of the isolated agents in a unified manner, and
effectively guides CLIP to generalize with multi-source knowledge distillation.
With such a distinct framework, we flexibly collaborate with 11 heterogeneous
agents to empower vision-language foundation models, without further cost in
the inference phase. Finally, our TransAgent achieves state-of-the-art
performance on 11 visual recognition datasets. Under the same low-shot setting,
it outperforms the popular CoOp with around 10% on average, and 20% on EuroSAT
which contains large domain shifts.
Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage
  Gaussian Splats
Authors: name: Chen Ziwen, name: Hao Tan, name: Kai Zhang, name: Sai Bi, name: Fujun Luan, name: Yicong Hong, name: Li Fuxin, name: Zexiang Xu
Published: 2024-10-16 17:54:06

We propose Long-LRM, a generalizable 3D Gaussian reconstruction model that is
capable of reconstructing a large scene from a long sequence of input images.
Specifically, our model can process 32 source images at 960x540 resolution
within only 1.3 seconds on a single A100 80G GPU. Our architecture features a
mixture of the recent Mamba2 blocks and the classical transformer blocks which
allowed many more tokens to be processed than prior work, enhanced by efficient
token merging and Gaussian pruning steps that balance between quality and
efficiency. Unlike previous feed-forward models that are limited to processing
1~4 input images and can only reconstruct a small portion of a large scene,
Long-LRM reconstructs the entire scene in a single feed-forward step. On
large-scale scene datasets such as DL3DV-140 and Tanks and Temples, our method
achieves performance comparable to optimization-based approaches while being
two orders of magnitude more efficient. Project page:
https://arthurhero.github.io/projects/llrm
Can MLLMs Understand the Deep Implication Behind Chinese Images?
Authors: name: Chenhao Zhang, name: Xi Feng, name: Yuelin Bai, name: Xinrun Du, name: Jinchang Hou, name: Kaixin Deng, name: Guangzeng Han, name: Qinrui Li, name: Bingli Wang, name: Jiaheng Liu, name: Xingwei Qu, name: Yifei Zhang, name: Qixuan Zhao, name: Yiming Liang, name: Ziqiang Liu, name: Feiteng Fang, name: Min Yang, name: Wenhao Huang, name: Chenghua Lin, name: Ge Zhang, name: Shiwen Ni
Published: 2024-10-17 17:59:24

As the capabilities of Multimodal Large Language Models (MLLMs) continue to
improve, the need for higher-order capability evaluation of MLLMs is
increasing. However, there is a lack of work evaluating MLLM for higher-order
perception and understanding of Chinese visual content. To fill the gap, we
introduce the **C**hinese **I**mage **I**mplication understanding
**Bench**mark, **CII-Bench**, which aims to assess the higher-order perception
and understanding capabilities of MLLMs for Chinese images. CII-Bench stands
out in several ways compared to existing benchmarks. Firstly, to ensure the
authenticity of the Chinese context, images in CII-Bench are sourced from the
Chinese Internet and manually reviewed, with corresponding answers also
manually crafted. Additionally, CII-Bench incorporates images that represent
Chinese traditional culture, such as famous Chinese traditional paintings,
which can deeply reflect the model's understanding of Chinese traditional
culture. Through extensive experiments on CII-Bench across multiple MLLMs, we
have made significant findings. Initially, a substantial gap is observed
between the performance of MLLMs and humans on CII-Bench. The highest accuracy
of MLLMs attains 64.4%, where as human accuracy averages 78.2%, peaking at an
impressive 81.0%. Subsequently, MLLMs perform worse on Chinese traditional
culture images, suggesting limitations in their ability to understand
high-level semantics and lack a deep knowledge base of Chinese traditional
culture. Finally, it is observed that most models exhibit enhanced accuracy
when image emotion hints are incorporated into the prompts. We believe that
CII-Bench will enable MLLMs to gain a better understanding of Chinese semantics
and Chinese-specific images, advancing the journey towards expert artificial
general intelligence (AGI). Our project is publicly available at
https://cii-bench.github.io/.
Remember, Retrieve and Generate: Understanding Infinite Visual Concepts
  as Your Personalized Assistant
Authors: name: Haoran Hao, name: Jiaming Han, name: Changsheng Li, name: Yu-Feng Li, name: Xiangyu Yue
Published: 2024-10-17 09:10:26

The development of large language models (LLMs) has significantly enhanced
the capabilities of multimodal LLMs (MLLMs) as general assistants. However,
lack of user-specific knowledge still restricts their application in human's
daily life. In this paper, we introduce the Retrieval Augmented Personalization
(RAP) framework for MLLMs' personalization. Starting from a general MLLM, we
turn it into a personalized assistant in three steps. (a) Remember: We design a
key-value database to store user-related information, e.g., user's name, avatar
and other attributes. (b) Retrieve: When the user initiates a conversation, RAP
will retrieve relevant information from the database using a multimodal
retriever. (c) Generate: The input query and retrieved concepts' information
are fed into MLLMs to generate personalized, knowledge-augmented responses.
Unlike previous methods, RAP allows real-time concept editing via updating the
external database. To further improve generation quality and alignment with
user-specific information, we design a pipeline for data collection and create
a specialized dataset for personalized training of MLLMs. Based on the dataset,
we train a series of MLLMs as personalized multimodal assistants. By
pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual
concepts without additional finetuning. Our models demonstrate outstanding
flexibility and generation quality across a variety of tasks, such as
personalized image captioning, question answering and visual recognition. The
code, data and models are available at https://github.com/Hoar012/RAP-MLLM.
MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic
  Synchronization
Authors: name: Ruiqi Li, name: Siqi Zheng, name: Xize Cheng, name: Ziang Zhang, name: Shengpeng Ji, name: Zhou Zhao
Published: 2024-10-16 18:44:56

Generating music that aligns with the visual content of a video has been a
challenging task, as it requires a deep understanding of visual semantics and
involves generating music whose melody, rhythm, and dynamics harmonize with the
visual narratives. This paper presents MuVi, a novel framework that effectively
addresses these challenges to enhance the cohesion and immersive experience of
audio-visual content. MuVi analyzes video content through a specially designed
visual adaptor to extract contextually and temporally relevant features. These
features are used to generate music that not only matches the video's mood and
theme but also its rhythm and pacing. We also introduce a contrastive
music-visual pre-training scheme to ensure synchronization, based on the
periodicity nature of music phrases. In addition, we demonstrate that our
flow-matching-based music generator has in-context learning ability, allowing
us to control the style and genre of the generated music. Experimental results
show that MuVi demonstrates superior performance in both audio quality and
temporal synchronization. The generated music video samples are available at
https://muvi-v2m.github.io.
Do LLMs Have Political Correctness? Analyzing Ethical Biases and
  Jailbreak Vulnerabilities in AI Systems
Authors: name: Isack Lee, name: Haebin Seong
Published: 2024-10-17 08:46:09

Although large language models (LLMs) demonstrate impressive proficiency in
various tasks, they present potential safety risks, such as `jailbreaks', where
malicious inputs can coerce LLMs into generating harmful content. To address
these issues, many LLM developers have implemented various safety measures to
align these models. This alignment involves several techniques, including data
filtering during pre-training, supervised fine-tuning, reinforcement learning
from human feedback, and red-teaming exercises. These methods often introduce
deliberate and intentional biases similar to Political Correctness (PC) to
ensure the ethical behavior of LLMs. In this paper, we delve into the
intentional biases injected into LLMs for safety purposes and examine methods
to circumvent these safety alignment techniques. Notably, these intentional
biases result in a jailbreaking success rate in GPT-4o models that differs by
20% between non-binary and cisgender keywords and by 16% between white and
black keywords, even when the other parts of the prompts are identical. We
introduce the concept of PCJailbreak, highlighting the inherent risks posed by
these safety-induced biases. Additionally, we propose an efficient defense
method PCDefense, which prevents jailbreak attempts by injecting defense
prompts prior to generation. PCDefense stands as an appealing alternative to
Guard Models, such as Llama-Guard, that require additional inference cost after
text generation. Our findings emphasize the urgent need for LLM developers to
adopt a more responsible approach when designing and implementing safety
measures.
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding
  and Generation
Authors: name: Chengyue Wu, name: Xiaokang Chen, name: Zhiyu Wu, name: Yiyang Ma, name: Xingchao Liu, name: Zizheng Pan, name: Wen Liu, name: Zhenda Xie, name: Xingkai Yu, name: Chong Ruan, name: Ping Luo
Published: 2024-10-17 17:58:37

In this paper, we introduce Janus, an autoregressive framework that unifies
multimodal understanding and generation. Prior research often relies on a
single visual encoder for both tasks, such as Chameleon. However, due to the
differing levels of information granularity required by multimodal
understanding and generation, this approach can lead to suboptimal performance,
particularly in multimodal understanding. To address this issue, we decouple
visual encoding into separate pathways, while still leveraging a single,
unified transformer architecture for processing. The decoupling not only
alleviates the conflict between the visual encoder's roles in understanding and
generation, but also enhances the framework's flexibility. For instance, both
the multimodal understanding and generation components can independently select
their most suitable encoding methods. Experiments show that Janus surpasses
previous unified model and matches or exceeds the performance of task-specific
models. The simplicity, high flexibility, and effectiveness of Janus make it a
strong candidate for next-generation unified multimodal models.
MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures
Authors: name: Jinjie Ni, name: Yifan Song, name: Deepanway Ghosal, name: Bo Li, name: David Junhao Zhang, name: Xiang Yue, name: Fuzhao Xue, name: Zian Zheng, name: Kaichen Zhang, name: Mahir Shah, name: Kabir Jain, name: Yang You, name: Michael Shieh
Published: 2024-10-17 16:52:28

Perceiving and generating diverse modalities are crucial for AI models to
effectively learn from and engage with real-world signals, necessitating
reliable evaluations for their development. We identify two major issues in
current evaluations: (1) inconsistent standards, shaped by different
communities with varying protocols and maturity levels; and (2) significant
query, grading, and generalization biases. To address these, we introduce
MixEval-X, the first any-to-any real-world benchmark designed to optimize and
standardize evaluations across input and output modalities. We propose
multi-modal benchmark mixture and adaptation-rectification pipelines to
reconstruct real-world task distributions, ensuring evaluations generalize
effectively to real-world use cases. Extensive meta-evaluations show our
approach effectively aligns benchmark samples with real-world task
distributions and the model rankings correlate strongly with that of
crowd-sourced real-world evaluations (up to 0.98). We provide comprehensive
leaderboards to rerank existing models and organizations and offer insights to
enhance understanding of multi-modal evaluations and inform future research.
MoH: Multi-Head Attention as Mixture-of-Head Attention
Authors: name: Peng Jin, name: Bo Zhu, name: Li Yuan, name: Shuicheng Yan
Published: 2024-10-15 17:59:44

In this work, we upgrade the multi-head attention mechanism, the core of the
Transformer model, to improve efficiency while maintaining or surpassing the
previous accuracy level. We show that multi-head attention can be expressed in
the summation form. Drawing on the insight that not all attention heads hold
equal significance, we propose Mixture-of-Head attention (MoH), a new
architecture that treats attention heads as experts in the Mixture-of-Experts
(MoE) mechanism. MoH has two significant advantages: First, MoH enables each
token to select the appropriate attention heads, enhancing inference efficiency
without compromising accuracy or increasing the number of parameters. Second,
MoH replaces the standard summation in multi-head attention with a weighted
summation, introducing flexibility to the attention mechanism and unlocking
extra performance potential. Extensive experiments on ViT, DiT, and LLMs
demonstrate that MoH outperforms multi-head attention by using only 50%-90% of
the attention heads. Moreover, we demonstrate that pre-trained multi-head
attention models, such as LLaMA3-8B, can be further continue-tuned into our MoH
models. Notably, MoH-LLaMA3-8B achieves an average accuracy of 64.0% across 14
benchmarks, outperforming LLaMA3-8B by 2.4% by utilizing only 75% of the
attention heads. We believe the proposed MoH is a promising alternative to
multi-head attention and provides a strong foundation for developing advanced
and efficient attention-based models.
SBI-RAG: Enhancing Math Word Problem Solving for Students through
  Schema-Based Instruction and Retrieval-Augmented Generation
Authors: name: Prakhar Dixit, name: Tim Oates
Published: 2024-10-17 07:46:49

Many students struggle with math word problems (MWPs), often finding it
difficult to identify key information and select the appropriate mathematical
operations.Schema-based instruction (SBI) is an evidence-based strategy that
helps students categorize problems based on their structure, improving
problem-solving accuracy. Building on this, we propose a Schema-Based
Instruction Retrieval-Augmented Generation (SBI-RAG) framework that
incorporates a large language model (LLM).Our approach emphasizes step-by-step
reasoning by leveraging schemas to guide solution generation. We evaluate its
performance on the GSM8K dataset, comparing it with GPT-4 and GPT-3.5 Turbo,
and introduce a "reasoning score" metric to assess solution quality. Our
findings suggest that SBI-RAG enhances reasoning clarity and problem-solving
accuracy, potentially providing educational benefits for students