Orionnnn/hub-stats huggingface dataset:
FürElise: Capturing and Physically Synthesizing Hand Motions of Piano
  Performance
Authors: name: Ruocheng Wang, name: Pei Xu, name: Haochen Shi, name: Elizabeth Schumann, name: C. Karen Liu
Published: 2024-10-08 08:21:05

Piano playing requires agile, precise, and coordinated hand control that
stretches the limits of dexterity. Hand motion models with the sophistication
to accurately recreate piano playing have a wide range of applications in
character animation, embodied AI, biomechanics, and VR/AR. In this paper, we
construct a first-of-its-kind large-scale dataset that contains approximately
10 hours of 3D hand motion and audio from 15 elite-level pianists playing 153
pieces of classical music. To capture natural performances, we designed a
markerless setup in which motions are reconstructed from multi-view videos
using state-of-the-art pose estimation models. The motion data is further
refined via inverse kinematics using the high-resolution MIDI key-pressing data
obtained from sensors in a specialized Yamaha Disklavier piano. Leveraging the
collected dataset, we developed a pipeline that can synthesize
physically-plausible hand motions for musical scores outside of the dataset.
Our approach employs a combination of imitation learning and reinforcement
learning to obtain policies for physics-based bimanual control involving the
interaction between hands and piano keys. To solve the sampling efficiency
problem with the large motion dataset, we use a diffusion model to generate
natural reference motions, which provide high-level trajectory and fingering
(finger order and placement) information. However, the generated reference
motion alone does not provide sufficient accuracy for piano performance
modeling. We then further augmented the data by using musical similarity to
retrieve similar motions from the captured dataset to boost the precision of
the RL policy. With the proposed method, our model generates natural, dexterous
motions that generalize to music from outside the training dataset.
AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to
  Jailbreak LLMs
Authors: name: Xiaogeng Liu, name: Peiran Li, name: Edward Suh, name: Yevgeniy Vorobeychik, name: Zhuoqing Mao, name: Somesh Jha, name: Patrick McDaniel, name: Huan Sun, name: Bo Li, name: Chaowei Xiao
Published: 2024-10-03 17:59:01

In this paper, we propose AutoDAN-Turbo, a black-box jailbreak method that
can automatically discover as many jailbreak strategies as possible from
scratch, without any human intervention or predefined scopes (e.g., specified
candidate strategies), and use them for red-teaming. As a result, AutoDAN-Turbo
can significantly outperform baseline methods, achieving a 74.3% higher average
attack success rate on public benchmarks. Notably, AutoDAN-Turbo achieves an
88.5 attack success rate on GPT-4-1106-turbo. In addition, AutoDAN-Turbo is a
unified framework that can incorporate existing human-designed jailbreak
strategies in a plug-and-play manner. By integrating human-designed strategies,
AutoDAN-Turbo can even achieve a higher attack success rate of 93.4 on
GPT-4-1106-turbo.
Multimodal Situational Safety
Authors: name: Kaiwen Zhou, name: Chengzhi Liu, name: Xuandong Zhao, name: Anderson Compalas, name: Dawn Song, name: Xin Eric Wang
Published: 2024-10-08 16:16:07

Multimodal Large Language Models (MLLMs) are rapidly evolving, demonstrating
impressive capabilities as multimodal assistants that interact with both humans
and their environments. However, this increased sophistication introduces
significant safety concerns. In this paper, we present the first evaluation and
analysis of a novel safety challenge termed Multimodal Situational Safety,
which explores how safety considerations vary based on the specific situation
in which the user or agent is engaged. We argue that for an MLLM to respond
safely, whether through language or action, it often needs to assess the safety
implications of a language query within its corresponding visual context. To
evaluate this capability, we develop the Multimodal Situational Safety
benchmark (MSSBench) to assess the situational safety performance of current
MLLMs. The dataset comprises 1,820 language query-image pairs, half of which
the image context is safe, and the other half is unsafe. We also develop an
evaluation framework that analyzes key safety aspects, including explicit
safety reasoning, visual understanding, and, crucially, situational safety
reasoning. Our findings reveal that current MLLMs struggle with this nuanced
safety problem in the instruction-following setting and struggle to tackle
these situational safety challenges all at once, highlighting a key area for
future research. Furthermore, we develop multi-agent pipelines to coordinately
solve safety challenges, which shows consistent improvement in safety over the
original MLLM response. Code and data: mssbench.github.io.
One Initialization to Rule them All: Fine-tuning via Explained Variance
  Adaptation
Authors: name: Fabian Paischer, name: Lukas Hauzenberger, name: Thomas Schmied, name: Benedikt Alkin, name: Marc Peter Deisenroth, name: Sepp Hochreiter
Published: 2024-10-09 17:59:06

Foundation models (FMs) are pre-trained on large-scale datasets and then
fine-tuned on a downstream task for a specific application. The most successful
and most commonly used fine-tuning method is to update the pre-trained weights
via a low-rank adaptation (LoRA). LoRA introduces new weight matrices that are
usually initialized at random with a uniform rank distribution across model
weights. Recent works focus on weight-driven initialization or learning of
adaptive ranks during training. Both approaches have only been investigated in
isolation, resulting in slow convergence or a uniform rank distribution, in
turn leading to sub-optimal performance. We propose to enhance LoRA by
initializing the new weights in a data-driven manner by computing singular
value decomposition on minibatches of activation vectors. Then, we initialize
the LoRA matrices with the obtained right-singular vectors and re-distribute
ranks among all weight matrices to explain the maximal amount of variance and
continue the standard LoRA fine-tuning procedure. This results in our new
method Explained Variance Adaptation (EVA). We apply EVA to a variety of
fine-tuning tasks ranging from language generation and understanding to image
classification and reinforcement learning. EVA exhibits faster convergence than
competitors and attains the highest average score across a multitude of tasks
per domain.
Personalized Visual Instruction Tuning
Authors: name: Renjie Pi, name: Jianshu Zhang, name: Tianyang Han, name: Jipeng Zhang, name: Rui Pan, name: Tong Zhang
Published: 2024-10-09 17:46:53

Recent advancements in multimodal large language models (MLLMs) have
demonstrated significant progress; however, these models exhibit a notable
limitation, which we refer to as "face blindness". Specifically, they can
engage in general conversations but fail to conduct personalized dialogues
targeting at specific individuals. This deficiency hinders the application of
MLLMs in personalized settings, such as tailored visual assistants on mobile
devices, or domestic robots that need to recognize members of the family. In
this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel
data curation and training framework designed to enable MLLMs to identify
target individuals within an image and engage in personalized and coherent
dialogues. Our approach involves the development of a sophisticated pipeline
that autonomously generates training data containing personalized
conversations. This pipeline leverages the capabilities of various visual
experts, image generation models, and (multi-modal) large language models. To
evaluate the personalized potential of MLLMs, we present a benchmark called
P-Bench, which encompasses various question types with different levels of
difficulty. The experiments demonstrate a substantial personalized performance
enhancement after fine-tuning with our curated dataset.
TweedieMix: Improving Multi-Concept Fusion for Diffusion-based
  Image/Video Generation
Authors: name: Gihyun Kwon, name: Jong Chul Ye
Published: 2024-10-08 01:06:01

Despite significant advancements in customizing text-to-image and video
generation models, generating images and videos that effectively integrate
multiple personalized concepts remains a challenging task. To address this, we
present TweedieMix, a novel method for composing customized diffusion models
during the inference phase. By analyzing the properties of reverse diffusion
sampling, our approach divides the sampling process into two stages. During the
initial steps, we apply a multiple object-aware sampling technique to ensure
the inclusion of the desired target objects. In the later steps, we blend the
appearances of the custom concepts in the de-noised image space using Tweedie's
formula. Our results demonstrate that TweedieMix can generate multiple
personalized concepts with higher fidelity than existing methods. Moreover, our
framework can be effortlessly extended to image-to-video diffusion models,
enabling the generation of videos that feature multiple personalized concepts.
Results and source code are in our anonymous project page.
T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through
  Data, Reward, and Conditional Guidance Design
Authors: name: Jiachen Li, name: Qian Long, name: Jian Zheng, name: Xiaofeng Gao, name: Robinson Piramuthu, name: Wenhu Chen, name: William Yang Wang
Published: 2024-10-08 04:30:06

In this paper, we focus on enhancing a diffusion-based text-to-video (T2V)
model during the post-training phase by distilling a highly capable consistency
model from a pretrained T2V model. Our proposed method, T2V-Turbo-v2,
introduces a significant advancement by integrating various supervision
signals, including high-quality training data, reward model feedback, and
conditional guidance, into the consistency distillation process. Through
comprehensive ablation studies, we highlight the crucial importance of
tailoring datasets to specific learning objectives and the effectiveness of
learning from diverse reward models for enhancing both the visual quality and
text-video alignment. Additionally, we highlight the vast design space of
conditional guidance strategies, which centers on designing an effective energy
function to augment the teacher ODE solver. We demonstrate the potential of
this approach by extracting motion guidance from the training datasets and
incorporating it into the ODE solver, showcasing its effectiveness in improving
the motion quality of the generated videos with the improved motion-related
metrics from VBench and T2V-CompBench. Empirically, our T2V-Turbo-v2
establishes a new state-of-the-art result on VBench, with a Total score of
85.13, surpassing proprietary systems such as Gen-3 and Kling.
Do great minds think alike? Investigating Human-AI Complementarity in
  Question Answering with CAIMIRA
Authors: name: Maharshi Gor, name: Hal Daumé III, name: Tianyi Zhou, name: Jordan Boyd-Graber
Published: 2024-10-09 03:53:26

Recent advancements of large language models (LLMs) have led to claims of AI
surpassing humans in natural language processing (NLP) tasks such as textual
understanding and reasoning. This work investigates these assertions by
introducing CAIMIRA, a novel framework rooted in item response theory (IRT)
that enables quantitative assessment and comparison of problem-solving
abilities of question-answering (QA) agents: humans and AI systems. Through
analysis of over 300,000 responses from ~70 AI systems and 155 humans across
thousands of quiz questions, CAIMIRA uncovers distinct proficiency patterns in
knowledge domains and reasoning skills. Humans outperform AI systems in
knowledge-grounded abductive and conceptual reasoning, while state-of-the-art
LLMs like GPT-4 and LLaMA show superior performance on targeted information
retrieval and fact-based reasoning, particularly when information gaps are
well-defined and addressable through pattern matching or data retrieval. These
findings highlight the need for future QA tasks to focus on questions that
challenge not only higher-order reasoning and scientific thinking, but also
demand nuanced linguistic interpretation and cross-contextual knowledge
application, helping advance AI developments that better emulate or complement
human cognitive abilities in real-world problem-solving.
ViBiDSampler: Enhancing Video Interpolation Using Bidirectional
  Diffusion Sampler
Authors: name: Serin Yang, name: Taesung Kwon, name: Jong Chul Ye
Published: 2024-10-08 03:01:54

Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V)
diffusion models has greatly enhanced video generation, especially in terms of
keyframe interpolation. However, current image-to-video diffusion models, while
powerful in generating videos from a single conditioning frame, need adaptation
for two-frame (start & end) conditioned generation, which is essential for
effective bounded interpolation. Unfortunately, existing approaches that fuse
temporally forward and backward paths in parallel often suffer from
off-manifold issues, leading to artifacts or requiring multiple iterative
re-noising steps. In this work, we introduce a novel, bidirectional sampling
strategy to address these off-manifold issues without requiring extensive
re-noising or fine-tuning. Our method employs sequential sampling along both
forward and backward paths, conditioned on the start and end frames,
respectively, ensuring more coherent and on-manifold generation of intermediate
frames. Additionally, we incorporate advanced guidance techniques, CFG++ and
DDS, to further enhance the interpolation process. By integrating these, our
method achieves state-of-the-art performance, efficiently generating
high-quality, smooth videos between keyframes. On a single 3090 GPU, our method
can interpolate 25 frames at 1024 x 576 resolution in just 195 seconds,
establishing it as a leading solution for keyframe interpolation.
MM-Ego: Towards Building Egocentric Multimodal LLMs
Authors: name: Hanrong Ye, name: Haotian Zhang, name: Erik Daxberger, name: Lin Chen, name: Zongyu Lin, name: Yanghao Li, name: Bowen Zhang, name: Haoxuan You, name: Dan Xu, name: Zhe Gan, name: Jiasen Lu, name: Yinfei Yang
Published: 2024-10-09 17:59:59

This research aims to comprehensively explore building a multimodal
foundation model for egocentric video understanding. To achieve this goal, we
work on three fronts. First, as there is a lack of QA data for egocentric video
understanding, we develop a data engine that efficiently generates 7M
high-quality QA samples for egocentric videos ranging from 30 seconds to one
hour long, based on human-annotated data. This is currently the largest
egocentric QA dataset. Second, we contribute a challenging egocentric QA
benchmark with 629 videos and 7,026 questions to evaluate the models' ability
in recognizing and memorizing visual details across videos of varying lengths.
We introduce a new de-biasing evaluation method to help mitigate the
unavoidable language bias present in the models being evaluated. Third, we
propose a specialized multimodal architecture featuring a novel "Memory Pointer
Prompting" mechanism. This design includes a global glimpse step to gain an
overarching understanding of the entire video and identify key visual
information, followed by a fallback step that utilizes the key visual
information to generate responses. This enables the model to more effectively
comprehend extended video content. With the data, benchmark, and model, we
successfully build MM-Ego, an egocentric multimodal LLM that shows powerful
performance on egocentric video understanding.
Pyramidal Flow Matching for Efficient Video Generative Modeling
Authors: name: Yang Jin, name: Zhicheng Sun, name: Ningyuan Li, name: Kun Xu, name: Kun Xu, name: Hao Jiang, name: Nan Zhuang, name: Quzhe Huang, name: Yang Song, name: Yadong Mu, name: Zhouchen Lin
Published: 2024-10-08 12:10:37

Video generation requires modeling a vast spatiotemporal space, which demands
significant computational resources and data usage. To reduce the complexity,
the prevailing approaches employ a cascaded architecture to avoid direct
training with full resolution. Despite reducing computational demands, the
separate optimization of each sub-stage hinders knowledge sharing and
sacrifices flexibility. This work introduces a unified pyramidal flow matching
algorithm. It reinterprets the original denoising trajectory as a series of
pyramid stages, where only the final stage operates at the full resolution,
thereby enabling more efficient video generative modeling. Through our
sophisticated design, the flows of different pyramid stages can be interlinked
to maintain continuity. Moreover, we craft autoregressive video generation with
a temporal pyramid to compress the full-resolution history. The entire
framework can be optimized in an end-to-end manner and with a single unified
Diffusion Transformer (DiT). Extensive experiments demonstrate that our method
supports generating high-quality 5-second (up to 10-second) videos at 768p
resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models
will be open-sourced at https://pyramid-flow.github.io.
Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for
  Text-to-Image Diffusion Model Unlearning
Authors: name: Saemi Moon, name: Minjong Lee, name: Sangdon Park, name: Dongwoo Kim
Published: 2024-10-08 03:30:39

As text-to-image diffusion models become advanced enough for commercial
applications, there is also increasing concern about their potential for
malicious and harmful use. Model unlearning has been proposed to mitigate the
concerns by removing undesired and potentially harmful information from the
pre-trained model. So far, the success of unlearning is mainly measured by
whether the unlearned model can generate a target concept while maintaining
image quality. However, unlearning is typically tested under limited scenarios,
and the side effects of unlearning have barely been studied in the current
literature. In this work, we thoroughly analyze unlearning under various
scenarios with five key aspects. Our investigation reveals that every method
has side effects or limitations, especially in more complex and realistic
situations. By releasing our comprehensive evaluation framework with the source
codes and artifacts, we hope to inspire further research in this area, leading
to more reliable and effective unlearning methods.
IterComp: Iterative Composition-Aware Feedback Learning from Model
  Gallery for Text-to-Image Generation
Authors: name: Xinchen Zhang, name: Ling Yang, name: Guohao Li, name: Yaqi Cai, name: Jiake Xie, name: Yong Tang, name: Yujiu Yang, name: Mengdi Wang, name: Bin Cui
Published: 2024-10-09 17:59:13

Advanced diffusion models like RPG, Stable Diffusion 3 and FLUX have made
notable strides in compositional text-to-image generation. However, these
methods typically exhibit distinct strengths for compositional generation, with
some excelling in handling attribute binding and others in spatial
relationships. This disparity highlights the need for an approach that can
leverage the complementary strengths of various models to comprehensively
improve the composition capability. To this end, we introduce IterComp, a novel
framework that aggregates composition-aware model preferences from multiple
models and employs an iterative feedback learning approach to enhance
compositional generation. Specifically, we curate a gallery of six powerful
open-source diffusion models and evaluate their three key compositional
metrics: attribute binding, spatial relationships, and non-spatial
relationships. Based on these metrics, we develop a composition-aware model
preference dataset comprising numerous image-rank pairs to train
composition-aware reward models. Then, we propose an iterative feedback
learning method to enhance compositionality in a closed-loop manner, enabling
the progressive self-refinement of both the base diffusion model and reward
models over multiple iterations. Theoretical proof demonstrates the
effectiveness and extensive experiments show our significant superiority over
previous SOTA methods (e.g., Omost and FLUX), particularly in multi-category
object composition and complex semantic alignment. IterComp opens new research
avenues in reward feedback learning for diffusion models and compositional
generation. Code: https://github.com/YangLing0818/IterComp
Multimodal Large Language Models for Inverse Molecular Design with
  Retrosynthetic Planning
Authors: name: Gang Liu, name: Michael Sun, name: Wojciech Matusik, name: Meng Jiang, name: Jie Chen
Published: 2024-10-05 16:35:32

While large language models (LLMs) have integrated images, adapting them to
graphs remains challenging, limiting their applications in materials and drug
design. This difficulty stems from the need for coherent autoregressive
generation across texts and graphs. To address this, we introduce Llamole, the
first multimodal LLM capable of interleaved text and graph generation, enabling
molecular inverse design with retrosynthetic planning. Llamole integrates a
base LLM with the Graph Diffusion Transformer and Graph Neural Networks for
multi-conditional molecular generation and reaction inference within texts,
while the LLM, with enhanced molecular understanding, flexibly controls
activation among the different graph modules. Additionally, Llamole integrates
A* search with LLM-based cost functions for efficient retrosynthetic planning.
We create benchmarking datasets and conduct extensive experiments to evaluate
Llamole against in-context learning and supervised fine-tuning. Llamole
significantly outperforms 14 adapted LLMs across 12 metrics for controllable
molecular design and retrosynthetic planning.
BroadWay: Boost Your Text-to-Video Generation Model in a Training-free
  Way
Authors: name: Jiazi Bu, name: Pengyang Ling, name: Pan Zhang, name: Tong Wu, name: Xiaoyi Dong, name: Yuhang Zang, name: Yuhang Cao, name: Dahua Lin, name: Jiaqi Wang
Published: 2024-10-08 17:56:33

The text-to-video (T2V) generation models, offering convenient visual
creation, have recently garnered increasing attention. Despite their
substantial potential, the generated videos may present artifacts, including
structural implausibility, temporal inconsistency, and a lack of motion, often
resulting in near-static video. In this work, we have identified a correlation
between the disparity of temporal attention maps across different blocks and
the occurrence of temporal inconsistencies. Additionally, we have observed that
the energy contained within the temporal attention maps is directly related to
the magnitude of motion amplitude in the generated videos. Based on these
observations, we present BroadWay, a training-free method to improve the
quality of text-to-video generation without introducing additional parameters,
augmenting memory or sampling time. Specifically, BroadWay is composed of two
principal components: 1) Temporal Self-Guidance improves the structural
plausibility and temporal consistency of generated videos by reducing the
disparity between the temporal attention maps across various decoder blocks. 2)
Fourier-based Motion Enhancement enhances the magnitude and richness of motion
by amplifying the energy of the map. Extensive experiments demonstrate that
BroadWay significantly improves the quality of text-to-video generation with
negligible additional cost.