Orionnnn/hub-stats huggingface dataset:
Visual Question Decomposition on Multimodal Large Language Models
Authors: name: Haowei Zhang, name: Jianzhe Liu, name: Zhen Han, name: Shuo Chen, name: Bailan He, name: Volker Tresp, name: Zhiqiang Xu, name: Jindong Gu
Published: 2024-09-28 12:49:16

Question decomposition has emerged as an effective strategy for prompting
Large Language Models (LLMs) to answer complex questions. However, while
existing methods primarily focus on unimodal language models, the question
decomposition capability of Multimodal Large Language Models (MLLMs) has yet to
be explored. To this end, this paper explores visual question decomposition on
MLLMs. Specifically, we introduce a systematic evaluation framework including a
dataset and several evaluation criteria to assess the quality of the decomposed
sub-questions, revealing that existing MLLMs struggle to produce high-quality
sub-questions. To address this limitation, we propose a specific finetuning
dataset, DecoVQA+, for enhancing the model's question decomposition capability.
Aiming at enabling models to perform appropriate selective decomposition, we
propose an efficient finetuning pipeline. The finetuning pipeline consists of
our proposed dataset and a training objective for selective decomposition.
Finetuned MLLMs demonstrate significant improvements in the quality of
sub-questions and the policy of selective question decomposition. Additionally,
the models also achieve higher accuracy with selective decomposition on VQA
benchmark datasets.
Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained
  Transformers
Authors: name: Lirui Wang, name: Xinlei Chen, name: Jialiang Zhao, name: Kaiming He
Published: 2024-09-30 17:39:41

One of the roadblocks for training generalist robotic models today is
heterogeneity. Previous robot learning methods often collect data to train with
one specific embodiment for one task, which is expensive and prone to
overfitting. This work studies the problem of learning policy representations
through heterogeneous pre-training on robot data across different embodiments
and tasks at scale. We propose Heterogeneous Pre-trained Transformers (HPT),
which pre-train a large, shareable trunk of a policy neural network to learn a
task and embodiment agnostic shared representation. This general architecture
aligns the specific proprioception and vision inputs from distinct embodiments
to a short sequence of tokens and then processes such tokens to map to control
robots for different tasks. Leveraging the recent large-scale multi-embodiment
real-world robotic datasets as well as simulation, deployed robots, and human
video datasets, we investigate pre-training policies across heterogeneity. We
conduct experiments to investigate the scaling behaviors of training
objectives, to the extent of 52 datasets. HPTs outperform several baselines and
enhance the fine-tuned policy performance by over 20% on unseen tasks in
multiple simulator benchmarks and real-world settings. See the project website
(https://liruiw.github.io/hpt/) for code and videos.
UniAff: A Unified Representation of Affordances for Tool Usage and
  Articulation with Vision-Language Models
Authors: name: Qiaojun Yu, name: Siyuan Huang, name: Xibin Yuan, name: Zhengkai Jiang, name: Ce Hao, name: Xin Li, name: Haonan Chang, name: Junbo Wang, name: Liu Liu, name: Hongsheng Li, name: Peng Gao, name: Cewu Lu
Published: 2024-09-30 17:52:05

Previous studies on robotic manipulation are based on a limited understanding
of the underlying 3D motion constraints and affordances. To address these
challenges, we propose a comprehensive paradigm, termed UniAff, that integrates
3D object-centric manipulation and task understanding in a unified formulation.
Specifically, we constructed a dataset labeled with manipulation-related key
attributes, comprising 900 articulated objects from 19 categories and 600 tools
from 12 categories. Furthermore, we leverage MLLMs to infer object-centric
representations for manipulation tasks, including affordance recognition and
reasoning about 3D motion constraints. Comprehensive experiments in both
simulation and real-world settings indicate that UniAff significantly improves
the generalization of robotic manipulation for tools and articulated objects.
We hope that UniAff will serve as a general baseline for unified robotic
manipulation tasks in the future. Images, videos, dataset, and code are
published on the project website at:https://sites.google.com/view/uni-aff/home
Ruler: A Model-Agnostic Method to Control Generated Length for Large
  Language Models
Authors: name: Jiaming Li, name: Lei Zhang, name: Yunshui Li, name: Ziqiang Liu, name: yuelin bai, name: Run Luo, name: Longze Chen, name: Min Yang
Published: 2024-09-27 17:44:58

The instruction-following ability of large language models enables humans to
interact with AI agents in a natural way. However, when required to generate
responses of a specific length, large language models often struggle to meet
users' needs due to their inherent difficulty in accurately perceiving
numerical constraints. To explore the ability of large language models to
control the length of generated responses, we propose the Target Length
Generation Task (TLG) and design two metrics, Precise Match (PM) and Flexible
Match (FM) to evaluate the model's performance in adhering to specified
response lengths. Furthermore, we introduce a novel, model-agnostic approach
called Ruler, which employs Meta Length Tokens (MLTs) to enhance the
instruction-following ability of large language models under length-constrained
instructions. Specifically, Ruler equips LLMs with the ability to generate
responses of a specified length based on length constraints within the
instructions. Moreover, Ruler can automatically generate appropriate MLT when
length constraints are not explicitly provided, demonstrating excellent
versatility and generalization. Comprehensive experiments show the
effectiveness of Ruler across different LLMs on Target Length Generation Task,
e.g., at All Level 27.97 average gain on PM, 29.57 average gain on FM. In
addition, we conduct extensive ablation experiments to further substantiate the
efficacy and generalization of Ruler. Our code and data is available at
https://github.com/Geaming2002/Ruler.
Coffee-Gym: An Environment for Evaluating and Improving Natural Language
  Feedback on Erroneous Code
Authors: name: Hyungjoo Chae, name: Taeyoon Kwon, name: Seungjun Moon, name: Yongho Song, name: Dongjin Kang, name: Kai Tzu-iunn Ong, name: Beong-woo Kwak, name: Seonghyeon Bae, name: Seung-won Hwang, name: Jinyoung Yeo
Published: 2024-09-29 14:14:25

This paper presents Coffee-Gym, a comprehensive RL environment for training
models that provide feedback on code editing. Coffee-Gym includes two major
components: (1) Coffee, a dataset containing humans' code edit traces for
coding questions and machine-written feedback for editing erroneous code; (2)
CoffeeEval, a reward function that faithfully reflects the helpfulness of
feedback by assessing the performance of the revised code in unit tests. With
them, Coffee-Gym addresses the unavailability of high-quality datasets for
training feedback models with RL, and provides more accurate rewards than the
SOTA reward model (i.e., GPT-4). By applying Coffee-Gym, we elicit feedback
models that outperform baselines in enhancing open-source code LLMs' code
editing, making them comparable with closed-source LLMs. We make the dataset
and the model checkpoint publicly available.
Can Models Learn Skill Composition from Examples?
Authors: name: Haoyu Zhao, name: Simran Kaur, name: Dingli Yu, name: Anirudh Goyal, name: Sanjeev Arora
Published: 2024-09-29 22:14:02

As large language models (LLMs) become increasingly advanced, their ability
to exhibit compositional generalization -- the capacity to combine learned
skills in novel ways not encountered during training -- has garnered
significant attention. This type of generalization, particularly in scenarios
beyond training data, is also of great interest in the study of AI safety and
alignment. A recent study introduced the SKILL-MIX evaluation, where models are
tasked with composing a short paragraph demonstrating the use of a specified
k-tuple of language skills. While small models struggled with composing even
with k=3, larger models like GPT-4 performed reasonably well with k=5 and
6.
  In this paper, we employ a setup akin to SKILL-MIX to evaluate the capacity
of smaller models to learn compositional generalization from examples.
Utilizing a diverse set of language skills -- including rhetorical, literary,
reasoning, theory of mind, and common sense -- GPT-4 was used to generate text
samples that exhibit random subsets of k skills. Subsequent fine-tuning of 7B
and 13B parameter models on these combined skill texts, for increasing values
of k, revealed the following findings: (1) Training on combinations of k=2
and 3 skills results in noticeable improvements in the ability to compose
texts with k=4 and 5 skills, despite models never having seen such examples
during training. (2) When skill categories are split into training and held-out
groups, models significantly improve at composing texts with held-out skills
during testing despite having only seen training skills during fine-tuning,
illustrating the efficacy of the training approach even with previously unseen
skills. This study also suggests that incorporating skill-rich (potentially
synthetic) text into training can substantially enhance the compositional
capabilities of models.
IDEAW: Robust Neural Audio Watermarking with Invertible Dual-Embedding
Authors: name: Pengcheng Li, name: Xulong Zhang, name: Jing Xiao, name: Jianzong Wang
Published: 2024-09-29 09:32:54

The audio watermarking technique embeds messages into audio and accurately
extracts messages from the watermarked audio. Traditional methods develop
algorithms based on expert experience to embed watermarks into the time-domain
or transform-domain of signals. With the development of deep neural networks,
deep learning-based neural audio watermarking has emerged. Compared to
traditional algorithms, neural audio watermarking achieves better robustness by
considering various attacks during training. However, current neural
watermarking methods suffer from low capacity and unsatisfactory
imperceptibility. Additionally, the issue of watermark locating, which is
extremely important and even more pronounced in neural audio watermarking, has
not been adequately studied. In this paper, we design a dual-embedding
watermarking model for efficient locating. We also consider the impact of the
attack layer on the invertible neural network in robustness training, improving
the model to enhance both its reasonableness and stability. Experiments show
that the proposed model, IDEAW, can withstand various attacks with higher
capacity and more efficient locating ability compared to existing methods.
Cottention: Linear Transformers With Cosine Attention
Authors: name: Gabriel Mongaras, name: Trevor Dohm, name: Eric C. Larson
Published: 2024-09-27 13:38:36

Attention mechanisms, particularly softmax attention, have been instrumental
in the success of transformer-based models such as GPT. However, the quadratic
memory complexity of softmax attention with respect to sequence length poses
significant challenges for processing longer sequences. We introduce
Cottention, a novel attention mechanism that replaces the softmax operation
with cosine similarity. By leveraging the properties of cosine similarity and
rearranging the attention equation, Cottention achieves native linear memory
complexity with respect to sequence length, making it inherently more
memory-efficient than softmax attention. We demonstrate that Cottention can be
reformulated as a recurrent neural network (RNN) with a finite hidden state,
allowing for constant memory usage during inference. We evaluate Cottention on
both the bidirectional BERT and causal GPT tasks, demonstrating comparable
performance to softmax attention while significantly reducing memory
requirements. To ensure efficient computation, we develop a custom CUDA kernel
for Cottention. Our results show that Cottention is a promising alternative to
softmax attention, enabling the processing of longer sequences without
sacrificing performance, due to its native linear memory complexity and ability
to maintain a constant memory footprint during inference.
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Authors: name: Haotian Zhang, name: Mingfei Gao, name: Zhe Gan, name: Philipp Dufter, name: Nina Wenzel, name: Forrest Huang, name: Dhruti Shah, name: Xianzhi Du, name: Bowen Zhang, name: Yanghao Li, name: Sam Dodge, name: Keen You, name: Zhen Yang, name: Aleksei Timofeev, name: Mingze Xu, name: Hong-You Chen, name: Jean-Philippe Fauconnier, name: Zhengfeng Lai, name: Haoxuan You, name: Zirui Wang, name: Afshin Dehghan, name: Peter Grasch, name: Yinfei Yang
Published: 2024-09-30 17:59:34

We present MM1.5, a new family of multimodal large language models (MLLMs)
designed to enhance capabilities in text-rich image understanding, visual
referring and grounding, and multi-image reasoning. Building upon the MM1
architecture, MM1.5 adopts a data-centric approach to model training,
systematically exploring the impact of diverse data mixtures across the entire
model training lifecycle. This includes high-quality OCR data and synthetic
captions for continual pre-training, as well as an optimized visual
instruction-tuning data mixture for supervised fine-tuning. Our models range
from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE)
variants, and demonstrate that careful data curation and training strategies
can yield strong performance even at small scales (1B and 3B). Additionally, we
introduce two specialized variants: MM1.5-Video, designed for video
understanding, and MM1.5-UI, tailored for mobile UI understanding. Through
extensive empirical studies and ablations, we provide detailed insights into
the training processes and decisions that inform our final designs, offering
valuable guidance for future research in MLLM development.
Hyper-Connections
Authors: name: Defa Zhu, name: Hongzhi Huang, name: Zihao Huang, name: Yutao Zeng, name: Yunyao Mao, name: Banggu Wu, name: Qiyang Min, name: Xun Zhou
Published: 2024-09-29 07:57:07

We present hyper-connections, a simple yet effective method that can serve as
an alternative to residual connections. This approach specifically addresses
common drawbacks observed in residual connection variants, such as the seesaw
effect between gradient vanishing and representation collapse. Theoretically,
hyper-connections allow the network to adjust the strength of connections
between features at different depths and dynamically rearrange layers. We
conduct experiments focusing on the pre-training of large language models,
including dense and sparse models, where hyper-connections show significant
performance improvements over residual connections. Additional experiments
conducted on vision tasks also demonstrate similar improvements. We anticipate
that this method will be broadly applicable and beneficial across a wide range
of AI problems.
DiaSynth -- Synthetic Dialogue Generation Framework
Authors: name: Sathya Krishnan Suresh, name: Wu Mengjun, name: Tushar Pranav, name: Eng Siong Chng
Published: 2024-09-25 07:03:31

The scarcity of domain specific dialogue datasets across various domains,
from academic topics to everyday conversations, limits the development of
dialogue systems for various applications. Existing research is often
constrained either by dialogue datasets that are too general or by niche domain
dialogue datasets whose scale does not match the required scale for training
dialogue systems. To address this gap, we introduce DiaSynth - a synthetic
dialogue generation framework capable of generating high quality, contextually
rich dialogues across a wide range of domains. Our approach differs from
existing frameworks by dynamically generating dialogues that incorporate
simulated personas, subtopics, and diverse conversational characteristics,
using a Large Language Model (LLM) with Chain of Thought (CoT) reasoning to
create contextually rich, domain-specific dialogues that closely mimic natural
human interactions. DiaSynth produces tailored dialogues that emulate realistic
conversations. We perform our experiments by generating synthetic data using
different LLMs and few-shot examples from DialogSum and SAMSum. The pretrained
language models fine-tuned on the synthetic data outperform the base models by
16.47%, while the comparison between models fine-tuned on in-domain data and
synthetic data shows that the synthetic data is able to capture 90.48% of the
distribution of the in-domain data. The quality of the data generated also
scales with the size of LLMs. These results validate DiaSynth's potential as a
robust alternative to traditional data collection methods.
Image Copy Detection for Diffusion Models
Authors: name: Wenhao Wang, name: Yifan Sun, name: Zhentao Tan, name: Yi Yang
Published: 2024-09-30 05:14:07

Images produced by diffusion models are increasingly popular in digital
artwork and visual marketing. However, such generated images might replicate
content from existing ones and pose the challenge of content originality.
Existing Image Copy Detection (ICD) models, though accurate in detecting
hand-crafted replicas, overlook the challenge from diffusion models. This
motivates us to introduce ICDiff, the first ICD specialized for diffusion
models. To this end, we construct a Diffusion-Replication (D-Rep) dataset and
correspondingly propose a novel deep embedding method. D-Rep uses a
state-of-the-art diffusion model (Stable Diffusion V1.5) to generate 40, 000
image-replica pairs, which are manually annotated into 6 replication levels
ranging from 0 (no replication) to 5 (total replication). Our method,
PDF-Embedding, transforms the replication level of each image-replica pair into
a probability density function (PDF) as the supervision signal. The intuition
is that the probability of neighboring replication levels should be continuous
and smooth. Experimental results show that PDF-Embedding surpasses
protocol-driven methods and non-PDF choices on the D-Rep test set. Moreover, by
utilizing PDF-Embedding, we find that the replication ratios of well-known
diffusion models against an open-source gallery range from 10% to 20%.
HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and
  Dynamic Workflows
Authors: name: Wenlin Yao, name: Haitao Mi, name: Dong Yu
Published: 2024-09-25 23:52:17

Despite recent advancements in large language models (LLMs), their
performance on complex reasoning problems requiring multi-step thinking and
combining various skills is still limited. To address this, we propose a novel
framework HDFlow for complex reasoning with LLMs that combines fast and slow
thinking modes in an adaptive manner. Our approach consists of two key
components: 1) a new approach for slow, deliberate reasoning called Dynamic
Workflow, which automatically decomposes complex problems into more manageable
sub-tasks and dynamically designs a workflow to assemble specialized LLM or
symbolic reasoning tools to solve sub-tasks; 2) Hybrid Thinking, a general
framework that dynamically combines fast and slow thinking based on problem
complexity. Finally, we propose an easy-to-scale method for automatically
synthesizing a large-scale dataset of 27K challenging reasoning problems for
complex reasoning and a hybrid thinking tuning method that trains smaller LLMs
on this dataset to internalize the fast/slow hybrid reasoning strategies.
Experiments on four reasoning benchmark datasets demonstrate that our slow
thinking with dynamic workflows significantly outperforms Chain-of-Thought, and
hybrid thinking achieves the highest accuracy while providing an effective
balance between computational efficiency and performance. Fine-tuning using our
hybrid thinking approach also significantly boosts the complex reasoning
capabilities of open-source language models. The results showcase the promise
of slow thinking, dynamic workflows, and hybrid thinking in expanding the
frontier of complex problem-solving with LLMsCode and data will be
released at \url{https://github.com/wenlinyao/HDFlow.}.
A Survey on the Honesty of Large Language Models
Authors: name: Siheng Li, name: Cheng Yang, name: Taiqiang Wu, name: Chufan Shi, name: Yuji Zhang, name: Xinyu Zhu, name: Zesen Cheng, name: Deng Cai, name: Mo Yu, name: Lemao Liu, name: Jie Zhou, name: Yujiu Yang, name: Ngai Wong, name: Xixin Wu, name: Wai Lam
Published: 2024-09-27 14:34:54

Honesty is a fundamental principle for aligning large language models (LLMs)
with human values, requiring these models to recognize what they know and don't
know and be able to faithfully express their knowledge. Despite promising,
current LLMs still exhibit significant dishonest behaviors, such as confidently
presenting wrong answers or failing to express what they know. In addition,
research on the honesty of LLMs also faces challenges, including varying
definitions of honesty, difficulties in distinguishing between known and
unknown knowledge, and a lack of comprehensive understanding of related
research. To address these issues, we provide a survey on the honesty of LLMs,
covering its clarification, evaluation approaches, and strategies for
improvement. Moreover, we offer insights for future research, aiming to inspire
further exploration in this important area.
LML: Language Model Learning a Dataset for Data-Augmented Prediction
Authors: name: Praneeth Vadlapati
Published: 2024-09-27 17:58:50

This paper introduces a new approach to using Large Language Models (LLMs)
for classification tasks, which are typically handled using Machine Learning
(ML) models. Unlike ML models that rely heavily on data cleaning and feature
engineering, this method streamlines the process using LLMs. This paper
proposes a new concept called "Language Model Learning (LML)" powered by a new
method called "Data-Augmented Prediction (DAP)". The classification is
performed by LLMs using a method similar to humans manually exploring and
understanding the data and deciding classifications using data as a reference.
Training data is summarized and evaluated to determine the features that lead
to the classification of each label the most. In the process of DAP, the system
uses the data summary to automatically create a query, which is used to
retrieve relevant rows from the dataset. A classification is generated by the
LLM using data summary and relevant rows, ensuring satisfactory accuracy even
with complex data. Usage of data summary and similar data in DAP ensures
context-aware decision-making. The proposed method uses the words "Act as an
Explainable Machine Learning Model" in the prompt to enhance the
interpretability of the predictions by allowing users to review the logic
behind each prediction. In some test cases, the system scored an accuracy above
90%, proving the effectiveness of the system and its potential to outperform
conventional ML models in various scenarios. The code is available at
https://github.com/Pro-GenAI/LML-DAP