Orionnnn/hub-stats huggingface dataset:
Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging
Authors: name: Hua Farn, name: Hsuan Su, name: Shachi H Kumar, name: Saurav Sahay, name: Shang-Tse Chen, name: Hung-yi Lee
Published: 2024-12-27 08:03:22

Fine-tuning large language models (LLMs) for downstream tasks is a widely
adopted approach, but it often leads to safety degradation in safety-aligned
LLMs. Currently, many solutions address this issue by incorporating additional
safety data, which can be impractical in many cases. In this paper, we address
the question: How can we improve downstream task performance while preserving
safety in LLMs without relying on additional safety data? We propose a simple
and effective method that maintains the inherent safety of LLMs while enhancing
their downstream task performance: merging the weights of pre- and
post-fine-tuned safety-aligned models. Experimental results across various
downstream tasks, models, and merging methods demonstrate that this approach
effectively mitigates safety degradation while improving downstream task
performance, offering a practical solution for adapting safety-aligned LLMs.
The Superposition of Diffusion Models Using the Itô Density Estimator
Authors: name: Marta Skreta, name: Lazar Atanackovic, name: Avishek Joey Bose, name: Alexander Tong, name: Kirill Neklyudov
Published: 2024-12-23 18:18:07

The Cambrian explosion of easily accessible pre-trained diffusion models
suggests a demand for methods that combine multiple different pre-trained
diffusion models without incurring the significant computational burden of
re-training a larger combined model. In this paper, we cast the problem of
combining multiple pre-trained diffusion models at the generation stage under a
novel proposed framework termed superposition. Theoretically, we derive
superposition from rigorous first principles stemming from the celebrated
continuity equation and design two novel algorithms tailor-made for combining
diffusion models in SuperDiff. SuperDiff leverages a new scalable It\^o density
estimator for the log likelihood of the diffusion SDE which incurs no
additional overhead compared to the well-known Hutchinson's estimator needed
for divergence calculations. We demonstrate that SuperDiff is scalable to large
pre-trained diffusion models as superposition is performed solely through
composition during inference, and also enjoys painless implementation as it
combines different pre-trained vector fields through an automated re-weighting
scheme. Notably, we show that SuperDiff is efficient during inference time, and
mimics traditional composition operators such as the logical OR and the logical
AND. We empirically demonstrate the utility of using SuperDiff for generating
more diverse images on CIFAR-10, more faithful prompt conditioned image editing
using Stable Diffusion, and improved unconditional de novo structure design of
proteins. https://github.com/necludov/super-diffusion
SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized
  Images
Authors: name: Risa Shinoda, name: Kuniaki Saito, name: Shohei Tanaka, name: Tosho Hirasawa, name: Yoshitaka Ushiku
Published: 2024-12-23 14:25:33

Building a large-scale figure QA dataset requires a considerable amount of
work, from gathering and selecting figures to extracting attributes like text,
numbers, and colors, and generating QAs. Although recent developments in LLMs
have led to efforts to synthesize figures, most of these focus primarily on QA
generation. Additionally, creating figures directly using LLMs often encounters
issues such as code errors, similar-looking figures, and repetitive content in
figures. To address this issue, we present SBSFigures (Stage-by-Stage Synthetic
Figures), a dataset for pre-training figure QA. Our proposed pipeline enables
the creation of chart figures with complete annotations of the visualized data
and dense QA annotations without any manual annotation process. Our
stage-by-stage pipeline makes it possible to create diverse topic and
appearance figures efficiently while minimizing code errors. Our SBSFigures
demonstrate a strong pre-training effect, making it possible to achieve
efficient training with a limited amount of real-world chart data starting from
our pre-trained weights.
VideoMaker: Zero-shot Customized Video Generation with the Inherent
  Force of Video Diffusion Models
Authors: name: Tao Wu, name: Yong Zhang, name: Xiaodong Cun, name: Zhongang Qi, name: Junfu Pu, name: Huanzhang Dou, name: Guangcong Zheng, name: Ying Shan, name: Xi Li
Published: 2024-12-27 13:49:25

Zero-shot customized video generation has gained significant attention due to
its substantial application potential. Existing methods rely on additional
models to extract and inject reference subject features, assuming that the
Video Diffusion Model (VDM) alone is insufficient for zero-shot customized
video generation. However, these methods often struggle to maintain consistent
subject appearance due to suboptimal feature extraction and injection
techniques. In this paper, we reveal that VDM inherently possesses the force to
extract and inject subject features. Departing from previous heuristic
approaches, we introduce a novel framework that leverages VDM's inherent force
to enable high-quality zero-shot customized video generation. Specifically, for
feature extraction, we directly input reference images into VDM and use its
intrinsic feature extraction process, which not only provides fine-grained
features but also significantly aligns with VDM's pre-trained knowledge. For
feature injection, we devise an innovative bidirectional interaction between
subject features and generated content through spatial self-attention within
VDM, ensuring that VDM has better subject fidelity while maintaining the
diversity of the generated video.Experiments on both customized human and
object video generation validate the effectiveness of our framework.
Task Preference Optimization: Improving Multimodal Large Language Models
  with Vision Task Alignment
Authors: name: Ziang Yan, name: Zhilin Li, name: Yinan He, name: Chenting Wang, name: Kunchang Li, name: Xinhao Li, name: Xiangyu Zeng, name: Zilei Wang, name: Yali Wang, name: Yu Qiao, name: Limin Wang, name: Yi Wang
Published: 2024-12-26 18:56:05

Current multimodal large language models (MLLMs) struggle with fine-grained
or precise understanding of visuals though they give comprehensive perception
and reasoning in a spectrum of vision applications. Recent studies either
develop tool-using or unify specific visual tasks into the autoregressive
framework, often at the expense of overall multimodal performance. To address
this issue and enhance MLLMs with visual tasks in a scalable fashion, we
propose Task Preference Optimization (TPO), a novel method that utilizes
differentiable task preferences derived from typical fine-grained visual tasks.
TPO introduces learnable task tokens that establish connections between
multiple task-specific heads and the MLLM. By leveraging rich visual labels
during training, TPO significantly enhances the MLLM's multimodal capabilities
and task-specific performance. Through multi-task co-training within TPO, we
observe synergistic benefits that elevate individual task performance beyond
what is achievable through single-task training methodologies. Our
instantiation of this approach with VideoChat and LLaVA demonstrates an overall
14.6% improvement in multimodal performance compared to baseline models.
Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across
various tasks, performing comparably to state-of-the-art supervised models. The
code will be released at https://github.com/OpenGVLab/TPO
From Elements to Design: A Layered Approach for Automatic Graphic Design
  Composition
Authors: name: Jiawei Lin, name: Shizhao Sun, name: Danqing Huang, name: Ting Liu, name: Ji Li, name: Jiang Bian
Published: 2024-12-27 16:13:08

In this work, we investigate automatic design composition from multimodal
graphic elements. Although recent studies have developed various generative
models for graphic design, they usually face the following limitations: they
only focus on certain subtasks and are far from achieving the design
composition task; they do not consider the hierarchical information of graphic
designs during the generation process. To tackle these issues, we introduce the
layered design principle into Large Multimodal Models (LMMs) and propose a
novel approach, called LaDeCo, to accomplish this challenging task.
Specifically, LaDeCo first performs layer planning for a given element set,
dividing the input elements into different semantic layers according to their
contents. Based on the planning results, it subsequently predicts element
attributes that control the design composition in a layer-wise manner, and
includes the rendered image of previously generated layers into the context.
With this insightful design, LaDeCo decomposes the difficult task into smaller
manageable steps, making the generation process smoother and clearer. The
experimental results demonstrate the effectiveness of LaDeCo in design
composition. Furthermore, we show that LaDeCo enables some interesting
applications in graphic design, such as resolution adjustment, element filling,
design variation, etc. In addition, it even outperforms the specialized models
in some design subtasks without any task-specific training.
HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs
Authors: name: Junying Chen, name: Zhenyang Cai, name: Ke Ji, name: Xidong Wang, name: Wanlong Liu, name: Rongsheng Wang, name: Jianye Hou, name: Benyou Wang
Published: 2024-12-25 15:12:34

The breakthrough of OpenAI o1 highlights the potential of enhancing reasoning
to improve LLM. Yet, most research in reasoning has focused on mathematical
tasks, leaving domains like medicine underexplored. The medical domain, though
distinct from mathematics, also demands robust reasoning to provide reliable
answers, given the high standards of healthcare. However, verifying medical
reasoning is challenging, unlike those in mathematics. To address this, we
propose verifiable medical problems with a medical verifier to check the
correctness of model outputs. This verifiable nature enables advancements in
medical reasoning through a two-stage approach: (1) using the verifier to guide
the search for a complex reasoning trajectory for fine-tuning LLMs, (2)
applying reinforcement learning (RL) with verifier-based rewards to enhance
complex reasoning further. Finally, we introduce HuatuoGPT-o1, a medical LLM
capable of complex reasoning, which outperforms general and medical-specific
baselines using only 40K verifiable problems. Experiments show complex
reasoning improves medical problem-solving and benefits more from RL. We hope
our approach inspires advancements in reasoning across medical and other
specialized domains.
Orient Anything: Learning Robust Object Orientation Estimation from
  Rendering 3D Models
Authors: name: Zehan Wang, name: Ziang Zhang, name: Tianyu Pang, name: Chao Du, name: Hengshuang Zhao, name: Zhou Zhao
Published: 2024-12-24 18:58:43

Orientation is a key attribute of objects, crucial for understanding their
spatial pose and arrangement in images. However, practical solutions for
accurate orientation estimation from a single image remain underexplored. In
this work, we introduce Orient Anything, the first expert and foundational
model designed to estimate object orientation in a single- and free-view image.
Due to the scarcity of labeled data, we propose extracting knowledge from the
3D world. By developing a pipeline to annotate the front face of 3D objects and
render images from random views, we collect 2M images with precise orientation
annotations. To fully leverage the dataset, we design a robust training
objective that models the 3D orientation as probability distributions of three
angles and predicts the object orientation by fitting these distributions.
Besides, we employ several strategies to improve synthetic-to-real transfer.
Our model achieves state-of-the-art orientation estimation accuracy in both
rendered and real images and exhibits impressive zero-shot ability in various
scenarios. More importantly, our model enhances many applications, such as
comprehension and generation of complex spatial concepts and 3D object pose
adjustment.
Molar: Multimodal LLMs with Collaborative Filtering Alignment for
  Enhanced Sequential Recommendation
Authors: name: Yucong Luo, name: Qitao Qin, name: Hao Zhang, name: Mingyue Cheng, name: Ruiran Yan, name: Kefan Wang, name: Jie Ouyang
Published: 2024-12-24 05:23:13

Sequential recommendation (SR) systems have evolved significantly over the
past decade, transitioning from traditional collaborative filtering to deep
learning approaches and, more recently, to large language models (LLMs). While
the adoption of LLMs has driven substantial advancements, these models
inherently lack collaborative filtering information, relying primarily on
textual content data neglecting other modalities and thus failing to achieve
optimal recommendation performance. To address this limitation, we propose
Molar, a Multimodal large language sequential recommendation framework that
integrates multiple content modalities with ID information to capture
collaborative signals effectively. Molar employs an MLLM to generate unified
item representations from both textual and non-textual data, facilitating
comprehensive multimodal modeling and enriching item embeddings. Additionally,
it incorporates collaborative filtering signals through a post-alignment
mechanism, which aligns user representations from content-based and ID-based
models, ensuring precise personalization and robust performance. By seamlessly
combining multimodal content with collaborative filtering insights, Molar
captures both user interests and contextual semantics, leading to superior
recommendation accuracy. Extensive experiments validate that Molar
significantly outperforms traditional and LLM-based baselines, highlighting its
strength in utilizing multimodal data and collaborative signals for sequential
recommendation tasks. The source code is available at
https://anonymous.4open.science/r/Molar-8B06/.
MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
Authors: name: Wan-Cyuan Fan, name: Tanzila Rahman, name: Leonid Sigal
Published: 2024-12-24 00:59:16

With advances in foundational and vision-language models, and effective
fine-tuning techniques, a large number of both general and special-purpose
models have been developed for a variety of visual tasks. Despite the
flexibility and accessibility of these models, no single model is able to
handle all tasks and/or applications that may be envisioned by potential users.
Recent approaches, such as visual programming and multimodal LLMs with
integrated tools aim to tackle complex visual tasks, by way of program
synthesis. However, such approaches overlook user constraints (e.g.,
performance / computational needs), produce test-time sample-specific solutions
that are difficult to deploy, and, sometimes, require low-level instructions
that maybe beyond the abilities of a naive user. To address these limitations,
we introduce MMFactory, a universal framework that includes model and metrics
routing components, acting like a solution search engine across various
available models. Based on a task description and few sample input-output pairs
and (optionally) resource and/or performance constraints, MMFactory can suggest
a diverse pool of programmatic solutions by instantiating and combining
visio-lingual tools from its model repository. In addition to synthesizing
these solutions, MMFactory also proposes metrics and benchmarks performance /
resource characteristics, allowing users to pick a solution that meets their
unique design constraints. From the technical perspective, we also introduced a
committee-based solution proposer that leverages multi-agent LLM conversation
to generate executable, diverse, universal, and robust solutions for the user.
Experimental results show that MMFactory outperforms existing methods by
delivering state-of-the-art solutions tailored to user problem specifications.
Project page is available at https://davidhalladay.github.io/mmfactory_demo.
A Silver Bullet or a Compromise for Full Attention? A Comprehensive
  Study of Gist Token-based Context Compression
Authors: name: Chenlong Deng, name: Zhisong Zhang, name: Kelong Mao, name: Shuaiyi Li, name: Xinting Huang, name: Dong Yu, name: Zhicheng Dou
Published: 2024-12-23 11:24:04

In this work, we provide a thorough investigation of gist-based context
compression methods to improve long-context processing in large language
models. We focus on two key questions: (1) How well can these methods replace
full attention models? and (2) What potential failure patterns arise due to
compression? Through extensive experiments, we show that while gist-based
compression can achieve near-lossless performance on tasks like
retrieval-augmented generation and long-document QA, it faces challenges in
tasks like synthetic recall. Furthermore, we identify three key failure
patterns: lost by the boundary, lost if surprise, and lost along the way. To
mitigate these issues, we propose two effective strategies: fine-grained
autoencoding, which enhances the reconstruction of original token information,
and segment-wise token importance estimation, which adjusts optimization based
on token dependencies. Our work provides valuable insights into the
understanding of gist token-based context compression and offers practical
strategies for improving compression capabilities.
YuLan-Mini: An Open Data-efficient Language Model
Authors: name: Yiwen Hu, name: Huatong Song, name: Jia Deng, name: Jiapeng Wang, name: Jie Chen, name: Kun Zhou, name: Yutao Zhu, name: Jinhao Jiang, name: Zican Dong, name: Wayne Xin Zhao, name: Ji-Rong Wen
Published: 2024-12-23 17:47:53

Effective pre-training of large language models (LLMs) has been challenging
due to the immense resource demands and the complexity of the technical
processes involved. This paper presents a detailed technical report on
YuLan-Mini, a highly capable base model with 2.42B parameters that achieves
top-tier performance among models of similar parameter scale. Our pre-training
approach focuses on enhancing training efficacy through three key technical
contributions: an elaborate data pipeline combines data cleaning with data
schedule strategies, a robust optimization method to mitigate training
instability, and an effective annealing approach that incorporates targeted
data selection and long context training. Remarkably, YuLan-Mini, trained on
1.08T tokens, achieves performance comparable to industry-leading models that
require significantly more data. To facilitate reproduction, we release the
full details of the data composition for each training phase. Project details
can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini.
VidTwin: Video VAE with Decoupled Structure and Dynamics
Authors: name: Yuchi Wang, name: Junliang Guo, name: Xinyi Xie, name: Tianyu He, name: Xu Sun, name: Jiang Bian
Published: 2024-12-23 17:16:58

Recent advancements in video autoencoders (Video AEs) have significantly
improved the quality and efficiency of video generation. In this paper, we
propose a novel and compact video autoencoder, VidTwin, that decouples video
into two distinct latent spaces: Structure latent vectors, which capture
overall content and global movement, and Dynamics latent vectors, which
represent fine-grained details and rapid movements. Specifically, our approach
leverages an Encoder-Decoder backbone, augmented with two submodules for
extracting these latent spaces, respectively. The first submodule employs a
Q-Former to extract low-frequency motion trends, followed by downsampling
blocks to remove redundant content details. The second averages the latent
vectors along the spatial dimension to capture rapid motion. Extensive
experiments show that VidTwin achieves a high compression rate of 0.20% with
high reconstruction quality (PSNR of 28.14 on the MCL-JCV dataset), and
performs efficiently and effectively in downstream generative tasks. Moreover,
our model demonstrates explainability and scalability, paving the way for
future research in video latent representation and generation. Our code has
been released at https://github.com/microsoft/VidTok/tree/main/vidtwin.
How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation
  System?
Authors: name: Sara Papi, name: Peter Polak, name: Ondřej Bojar, name: Dominik Macháček
Published: 2024-12-24 15:26:31

Simultaneous speech-to-text translation (SimulST) translates source-language
speech into target-language text concurrently with the speaker's speech,
ensuring low latency for better user comprehension. Despite its intended
application to unbounded speech, most research has focused on human
pre-segmented speech, simplifying the task and overlooking significant
challenges. This narrow focus, coupled with widespread terminological
inconsistencies, is limiting the applicability of research outcomes to
real-world applications, ultimately hindering progress in the field. Our
extensive literature review of 110 papers not only reveals these critical
issues in current research but also serves as the foundation for our key
contributions. We 1) define the steps and core components of a SimulST system,
proposing a standardized terminology and taxonomy; 2) conduct a thorough
analysis of community trends, and 3) offer concrete recommendations and future
directions to bridge the gaps in existing literature, from evaluation
frameworks to system architectures, for advancing the field towards more
realistic and effective SimulST solutions.
WavePulse: Real-time Content Analytics of Radio Livestreams
Authors: name: Govind Mittal, name: Sarthak Gupta, name: Shruti Wagle, name: Chirag Chopra, name: Anthony J DeMattee, name: Nasir Memon, name: Mustaque Ahamad, name: Chinmay Hegde
Published: 2024-12-23 21:42:31

Radio remains a pervasive medium for mass information dissemination, with
AM/FM stations reaching more Americans than either smartphone-based social
networking or live television. Increasingly, radio broadcasts are also streamed
online and accessed over the Internet. We present WavePulse, a framework that
records, documents, and analyzes radio content in real-time. While our
framework is generally applicable, we showcase the efficacy of WavePulse in a
collaborative project with a team of political scientists focusing on the 2024
Presidential Elections. We use WavePulse to monitor livestreams of 396 news
radio stations over a period of three months, processing close to 500,000 hours
of audio streams. These streams were converted into time-stamped, diarized
transcripts and analyzed to track answer key political science questions at
both the national and state levels. Our analysis revealed how local issues
interacted with national trends, providing insights into information flow. Our
results demonstrate WavePulse's efficacy in capturing and analyzing content
from radio livestreams sourced from the Web. Code and dataset can be accessed
at https://wave-pulse.io.