Orionnnn/hub-stats huggingface dataset:
LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for
  Parameter-Efficient Fine-Tuning
Authors: name: Yiming Shi, name: Jiwei Wei, name: Yujia Wu, name: Ran Ran, name: Chengwei Sun, name: Shiyuan He, name: Yang Yang
Published: 2024-10-17 14:51:17

The rapid growth of model scale has necessitated substantial computational
resources for fine-tuning. Existing approach such as Low-Rank Adaptation (LoRA)
has sought to address the problem of handling the large updated parameters in
full fine-tuning. However, LoRA utilize random initialization and optimization
of low-rank matrices to approximate updated weights, which can result in
suboptimal convergence and an accuracy gap compared to full fine-tuning. To
address these issues, we propose LoLDU, a Parameter-Efficient Fine-Tuning
(PEFT) approach that significantly reduces trainable parameters by 2600 times
compared to regular PEFT methods while maintaining comparable performance.
LoLDU leverages Lower-Diag-Upper Decomposition (LDU) to initialize low-rank
matrices for faster convergence and orthogonality. We focus on optimizing the
diagonal matrix for scaling transformations. To the best of our knowledge,
LoLDU has the fewest parameters among all PEFT approaches. We conducted
extensive experiments across 4 instruction-following datasets, 6 natural
language understanding (NLU) datasets, 8 image classification datasets, and
image generation datasets with multiple model types (LLaMA2, RoBERTa, ViT, and
Stable Diffusion), providing a comprehensive and detailed analysis. Our
open-source code can be accessed at
https://github.com/SKDDJ/LoLDU{https://github.com/SKDDJ/LoLDU}.
VidPanos: Generative Panoramic Videos from Casual Panning Videos
Authors: name: Jingwei Ma, name: Erika Lu, name: Roni Paiss, name: Shiran Zada, name: Aleksander Holynski, name: Tali Dekel, name: Brian Curless, name: Michael Rubinstein, name: Forrester Cole
Published: 2024-10-17 17:53:24

Panoramic image stitching provides a unified, wide-angle view of a scene that
extends beyond the camera's field of view. Stitching frames of a panning video
into a panoramic photograph is a well-understood problem for stationary scenes,
but when objects are moving, a still panorama cannot capture the scene. We
present a method for synthesizing a panoramic video from a casually-captured
panning video, as if the original video were captured with a wide-angle camera.
We pose panorama synthesis as a space-time outpainting problem, where we aim to
create a full panoramic video of the same length as the input video. Consistent
completion of the space-time volume requires a powerful, realistic prior over
video content and motion, for which we adapt generative video models. Existing
generative models do not, however, immediately extend to panorama completion,
as we show. We instead apply video generation as a component of our panorama
synthesis system, and demonstrate how to exploit the strengths of the models
while minimizing their limitations. Our system can create video panoramas for a
range of in-the-wild scenes including people, vehicles, and flowing water, as
well as stationary background features.
Movie Gen: A Cast of Media Foundation Models
Authors: name: Adam Polyak, name: Amit Zohar, name: Andrew Brown, name: Andros Tjandra, name: Animesh Sinha, name: Ann Lee, name: Apoorv Vyas, name: Bowen Shi, name: Chih-Yao Ma, name: Ching-Yao Chuang, name: David Yan, name: Dhruv Choudhary, name: Dingkang Wang, name: Geet Sethi, name: Guan Pang, name: Haoyu Ma, name: Ishan Misra, name: Ji Hou, name: Jialiang Wang, name: Kiran Jagadeesh, name: Kunpeng Li, name: Luxin Zhang, name: Mannat Singh, name: Mary Williamson, name: Matt Le, name: Matthew Yu, name: Mitesh Kumar Singh, name: Peizhao Zhang, name: Peter Vajda, name: Quentin Duval, name: Rohit Girdhar, name: Roshan Sumbaly, name: Sai Saketh Rambhatla, name: Sam Tsai, name: Samaneh Azadi, name: Samyak Datta, name: Sanyuan Chen, name: Sean Bell, name: Sharadh Ramaswamy, name: Shelly Sheynin, name: Siddharth Bhattacharya, name: Simran Motwani, name: Tao Xu, name: Tianhe Li, name: Tingbo Hou, name: Wei-Ning Hsu, name: Xi Yin, name: Xiaoliang Dai, name: Yaniv Taigman, name: Yaqiao Luo, name: Yen-Cheng Liu, name: Yi-Chiao Wu, name: Yue Zhao, name: Yuval Kirstain, name: Zecheng He, name: Zijian He, name: Albert Pumarola, name: Ali Thabet, name: Artsiom Sanakoyeu, name: Arun Mallya, name: Baishan Guo, name: Boris Araya, name: Breena Kerr, name: Carleigh Wood, name: Ce Liu, name: Cen Peng, name: Dimitry Vengertsev, name: Edgar Schonfeld, name: Elliot Blanchard, name: Felix Juefei-Xu, name: Fraylie Nord, name: Jeff Liang, name: John Hoffman, name: Jonas Kohler, name: Kaolin Fire, name: Karthik Sivakumar, name: Lawrence Chen, name: Licheng Yu, name: Luya Gao, name: Markos Georgopoulos, name: Rashel Moritz, name: Sara K. Sampson, name: Shikai Li, name: Simone Parmeggiani, name: Steve Fine, name: Tara Fowler, name: Vladan Petrovic, name: Yuming Du
Published: 2024-10-17 16:22:46

We present Movie Gen, a cast of foundation models that generates
high-quality, 1080p HD videos with different aspect ratios and synchronized
audio. We also show additional capabilities such as precise instruction-based
video editing and generation of personalized videos based on a user's image.
Our models set a new state-of-the-art on multiple tasks: text-to-video
synthesis, video personalization, video editing, video-to-audio generation, and
text-to-audio generation. Our largest video generation model is a 30B parameter
transformer trained with a maximum context length of 73K video tokens,
corresponding to a generated video of 16 seconds at 16 frames-per-second. We
show multiple technical innovations and simplifications on the architecture,
latent spaces, training objectives and recipes, data curation, evaluation
protocols, parallelization techniques, and inference optimizations that allow
us to reap the benefits of scaling pre-training data, model size, and training
compute for training large scale media generation models. We hope this paper
helps the research community to accelerate progress and innovation in media
generation models. All videos from this paper are available at
https://go.fb.me/MovieGenResearchVideos.
BenTo: Benchmark Task Reduction with In-Context Transferability
Authors: name: Hongyu Zhao, name: Ming Li, name: Lichao Sun, name: Tianyi Zhou
Published: 2024-10-17 17:41:15

Evaluating large language models (LLMs) is costly: it requires the generation
and examination of LLM outputs on a large-scale benchmark of various tasks.
This paper investigates how to efficiently reduce the tasks used to benchmark
LLMs without affecting the evaluation quality. Our study reveals that task
transferability and relevance provide critical information to identify the most
representative subset of tasks via optimizing a facility location function. We
propose a practically efficient metric for estimating the transferability
between two tasks via in-context learning (ICL). By analyzing the pairwise
transferability, we can reduce tasks in a modern LLM benchmark (e.g., MMLU or
FLAN) to 5% while inducing only a <4% difference to the evaluation on the
original benchmark. Compared to prior works, our method is training-free,
gradient-free, and highly efficient requiring ICL only.
PopAlign: Diversifying Contrasting Patterns for a More Comprehensive
  Alignment
Authors: name: Zekun Moore Wang, name: Shawn Wang, name: Kang Zhu, name: Jiaheng Liu, name: Ke Xu, name: Jie Fu, name: Wangchunshu Zhou, name: Wenhao Huang
Published: 2024-10-17 17:22:05

Alignment of large language models (LLMs) involves training models on
preference-contrastive output pairs to adjust their responses according to
human preferences. To obtain such contrastive pairs, traditional methods like
RLHF and RLAIF rely on limited contrasting patterns, such as varying model
variants or decoding temperatures. This singularity leads to two issues: (1)
alignment is not comprehensive; and thereby (2) models are susceptible to
jailbreaking attacks. To address these issues, we investigate how to construct
more comprehensive and diversified contrasting patterns to enhance preference
data (RQ1) and verify the impact of the diversification of contrasting patterns
on model alignment (RQ2). For RQ1, we propose PopAlign, a framework that
integrates diversified contrasting patterns across the prompt, model, and
pipeline levels, introducing six contrasting strategies that do not require
additional feedback labeling procedures. Regarding RQ2, we conduct thorough
experiments demonstrating that PopAlign significantly outperforms existing
methods, leading to more comprehensive alignment.
MedMobile: A mobile-sized language model with expert-level clinical
  capabilities
Authors: name: Krithik Vishwanath, name: Jaden Stryker, name: Anton Alaykin, name: Daniel Alexander Alber, name: Eric Karl Oermann
Published: 2024-10-11 17:32:59

Language models (LMs) have demonstrated expert-level reasoning and recall
abilities in medicine. However, computational costs and privacy concerns are
mounting barriers to wide-scale implementation. We introduce a parsimonious
adaptation of phi-3-mini, MedMobile, a 3.8 billion parameter LM capable of
running on a mobile device, for medical applications. We demonstrate that
MedMobile scores 75.7% on the MedQA (USMLE), surpassing the passing mark for
physicians (~60%), and approaching the scores of models 100 times its size. We
subsequently perform a careful set of ablations, and demonstrate that chain of
thought, ensembling, and fine-tuning lead to the greatest performance gains,
while unexpectedly retrieval augmented generation fails to demonstrate
significant improvements
Harnessing Webpage UIs for Text-Rich Visual Understanding
Authors: name: Junpeng Liu, name: Tianyue Ou, name: Yifan Song, name: Yuxiao Qu, name: Wai Lam, name: Chenyan Xiong, name: Wenhu Chen, name: Graham Neubig, name: Xiang Yue
Published: 2024-10-17 17:48:54

Text-rich visual understanding-the ability to process environments where
dense textual content is integrated with visuals-is crucial for multimodal
large language models (MLLMs) to interact effectively with structured
environments. To enhance this capability, we propose synthesizing general
multimodal instructions from webpage UIs using text-based large language models
(LLMs). Despite lacking direct visual input, text-based LLMs are able to
process structured text representations from webpage accessibility trees. These
instructions are then paired with UI screenshots to train multimodal models. We
introduce MultiUI, a dataset containing 7.3 million samples from 1 million
websites, covering diverse multimodal tasks and UI layouts. Models trained on
MultiUI not only excel in web UI tasks-achieving up to a 48\% improvement on
VisualWebBench and a 19.1\% boost in action accuracy on a web agent dataset
Mind2Web-but also generalize surprisingly well to non-web UI tasks and even to
non-UI domains, such as document understanding, OCR, and chart interpretation.
These results highlight the broad applicability of web UI data for advancing
text-rich visual understanding across various scenarios.
DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise
  Motion Control
Authors: name: Yujie Wei, name: Shiwei Zhang, name: Hangjie Yuan, name: Xiang Wang, name: Haonan Qiu, name: Rui Zhao, name: Yutong Feng, name: Feng Liu, name: Zhizhong Huang, name: Jiaxin Ye, name: Yingya Zhang, name: Hongming Shan
Published: 2024-10-17 17:52:57

Recent advances in customized video generation have enabled users to create
videos tailored to both specific subjects and motion trajectories. However,
existing methods often require complicated test-time fine-tuning and struggle
with balancing subject learning and motion control, limiting their real-world
applications. In this paper, we present DreamVideo-2, a zero-shot video
customization framework capable of generating videos with a specific subject
and motion trajectory, guided by a single image and a bounding box sequence,
respectively, and without the need for test-time fine-tuning. Specifically, we
introduce reference attention, which leverages the model's inherent
capabilities for subject learning, and devise a mask-guided motion module to
achieve precise motion control by fully utilizing the robust motion signal of
box masks derived from bounding boxes. While these two components achieve their
intended functions, we empirically observe that motion control tends to
dominate over subject learning. To address this, we propose two key designs: 1)
the masked reference attention, which integrates a blended latent mask modeling
scheme into reference attention to enhance subject representations at the
desired positions, and 2) a reweighted diffusion loss, which differentiates the
contributions of regions inside and outside the bounding boxes to ensure a
balance between subject and motion control. Extensive experimental results on a
newly curated dataset demonstrate that DreamVideo-2 outperforms
state-of-the-art methods in both subject customization and motion control. The
dataset, code, and models will be made publicly available.
MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language
  Models
Authors: name: Peng Xia, name: Kangyu Zhu, name: Haoran Li, name: Tianze Wang, name: Weijia Shi, name: Sheng Wang, name: Linjun Zhang, name: James Zou, name: Huaxiu Yao
Published: 2024-10-16 23:03:27

Artificial Intelligence (AI) has demonstrated significant potential in
healthcare, particularly in disease diagnosis and treatment planning. Recent
progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new
possibilities for interactive diagnostic tools. However, these models often
suffer from factual hallucination, which can lead to incorrect diagnoses.
Fine-tuning and retrieval-augmented generation (RAG) have emerged as methods to
address these issues. However, the amount of high-quality data and distribution
shifts between training data and deployment data limit the application of
fine-tuning methods. Although RAG is lightweight and effective, existing
RAG-based approaches are not sufficiently general to different medical domains
and can potentially cause misalignment issues, both between modalities and
between the model and the ground truth. In this paper, we propose a versatile
multimodal RAG system, MMed-RAG, designed to enhance the factuality of
Med-LVLMs. Our approach introduces a domain-aware retrieval mechanism, an
adaptive retrieved contexts selection method, and a provable RAG-based
preference fine-tuning strategy. These innovations make the RAG process
sufficiently general and reliable, significantly improving alignment when
introducing retrieved contexts. Experimental results across five medical
datasets (involving radiology, ophthalmology, pathology) on medical VQA and
report generation demonstrate that MMed-RAG can achieve an average improvement
of 43.8% in the factual accuracy of Med-LVLMs. Our data and code are available
in https://github.com/richard-peng-xia/MMed-RAG.
A Unified View of Delta Parameter Editing in Post-Trained Large-Scale
  Models
Authors: name: Qiaoyu Tang, name: Le Yu, name: Bowen Yu, name: Hongyu Lin, name: Keming Lu, name: Yaojie Lu, name: Xianpei Han, name: Le Sun
Published: 2024-10-17 17:56:53

Post-training has emerged as a crucial paradigm for adapting large-scale
pre-trained models to various tasks, whose effects are fully reflected by delta
parameters (i.e., the disparity between post-trained and pre-trained
parameters). While numerous studies have explored delta parameter properties
via operations like pruning, quantization, low-rank approximation, and
extrapolation, a unified framework for systematically examining these
characteristics has been lacking. In this paper, we propose a novel perspective
based on Riemann sum approximation of the loss function to elucidate delta
parameter editing operations. Our analysis categorizes existing methods into
three classes based on their post-editing performance: competitive, decreased,
and improved, explaining how they are expressed by the Riemann sum
approximation term and how they alter the model performance. Extensive
experiments on both visual and language models, including ViT, LLaMA 3, Qwen 2,
and Mistral, corroborate our theoretical findings. Furthermore, we introduce
extensions to existing techniques like DARE and BitDelta, highlighting their
limitations in leveraging the properties of delta parameters and reorganizing
them into general expressions to enhance the applicability and effectiveness of
delta parameter editing in post-trained models.
AERO: Softmax-Only LLMs for Efficient Private Inference
Authors: name: Nandan Kumar Jha, name: Brandon Reagen
Published: 2024-10-16 21:40:49

The pervasiveness of proprietary language models has raised privacy concerns
for users' sensitive data, emphasizing the need for private inference (PI),
where inference is performed directly on encrypted inputs. However, current PI
methods face prohibitively higher communication and latency overheads,
primarily due to nonlinear operations. In this paper, we present a
comprehensive analysis to understand the role of nonlinearities in
transformer-based decoder-only language models. We introduce AERO, a four-step
architectural optimization framework that refines the existing LLM architecture
for efficient PI by systematically removing nonlinearities such as LayerNorm
and GELU and reducing FLOPs counts. For the first time, we propose a
Softmax-only architecture with significantly fewer FLOPs tailored for efficient
PI. Furthermore, we devise a novel entropy regularization technique to improve
the performance of Softmax-only models. AERO achieves up to 4.23times
communication and 1.94times latency reduction. We validate the effectiveness
of AERO by benchmarking it against the state-of-the-art.
Retrospective Learning from Interactions
Authors: name: Zizhao Chen, name: Mustafa Omer Gul, name: Yiwei Chen, name: Gloria Geng, name: Anne Wu, name: Yoav Artzi
Published: 2024-10-17 17:59:03

Multi-turn interactions between large language models (LLMs) and users
naturally include implicit feedback signals. If an LLM responds in an
unexpected way to an instruction, the user is likely to signal it by rephrasing
the request, expressing frustration, or pivoting to an alternative task. Such
signals are task-independent and occupy a relatively constrained subspace of
language, allowing the LLM to identify them even if it fails on the actual
task. This creates an avenue for continually learning from interactions without
additional annotations. We introduce ReSpect, a method to learn from such
signals in past interactions via retrospection. We deploy ReSpect in a new
multimodal interaction scenario, where humans instruct an LLM to solve an
abstract reasoning task with a combinatorial solution space. Through thousands
of interactions with humans, we show how ReSpect gradually improves task
completion rate from 31% to 82%, all without any external annotation.
Failing Forward: Improving Generative Error Correction for ASR with
  Synthetic Data and Retrieval Augmentation
Authors: name: Sreyan Ghosh, name: Mohammad Sadegh Rasooli, name: Michael Levit, name: Peidong Wang, name: Jian Xue, name: Dinesh Manocha, name: Jinyu Li
Published: 2024-10-17 04:00:29

Generative Error Correction (GEC) has emerged as a powerful post-processing
method to enhance the performance of Automatic Speech Recognition (ASR)
systems. However, we show that GEC models struggle to generalize beyond the
specific types of errors encountered during training, limiting their ability to
correct new, unseen errors at test time, particularly in out-of-domain (OOD)
scenarios. This phenomenon amplifies with named entities (NEs), where, in
addition to insufficient contextual information or knowledge about the NEs,
novel NEs keep emerging. To address these issues, we propose DARAG (Data- and
Retrieval-Augmented Generative Error Correction), a novel approach designed to
improve GEC for ASR in in-domain (ID) and OOD scenarios. We augment the GEC
training dataset with synthetic data generated by prompting LLMs and
text-to-speech models, thereby simulating additional errors from which the
model can learn. For OOD scenarios, we simulate test-time errors from new
domains similarly and in an unsupervised fashion. Additionally, to better
handle named entities, we introduce retrieval-augmented correction by
augmenting the input with entities retrieved from a database. Our approach is
simple, scalable, and both domain- and language-agnostic. We experiment on
multiple datasets and settings, showing that DARAG outperforms all our
baselines, achieving 8\% -- 30\% relative WER improvements in ID and 10\% --
33\% improvements in OOD settings.
FlatQuant: Flatness Matters for LLM Quantization
Authors: name: Yuxuan Sun, name: Ruikang Liu, name: Haoli Bai, name: Han Bao, name: Kang Zhao, name: Yuening Li, name: Jiaxin Hu, name: Xianzhi Yu, name: Lu Hou, name: Chun Yuan, name: Xin Jiang, name: Wulong Liu, name: Jun Yao
Published: 2024-10-12 08:10:28

Recently, quantization has been widely used for the compression and
acceleration of large language models~(LLMs). Due to the outliers in LLMs, it
is crucial to flatten weights and activations to minimize quantization error
with the equally spaced quantization points. Prior research explores various
pre-quantization transformations to suppress outliers, such as per-channel
scaling and Hadamard transformation. However, we observe that these transformed
weights and activations can still remain steep and outspread. In this paper, we
propose FlatQuant (Fast and Learnable Affine Transformation), a new
post-training quantization approach to enhance flatness of weights and
activations. Our approach identifies optimal affine transformations tailored to
each linear layer, calibrated in hours via a lightweight objective. To reduce
runtime overhead, we apply Kronecker decomposition to the transformation
matrices, and fuse all operations in FlatQuant into a single kernel. Extensive
experiments show that FlatQuant sets up a new state-of-the-art quantization
benchmark. For instance, it achieves less than 1% accuracy drop for
W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by
7.5%. For inference latency, FlatQuant reduces the slowdown induced
by pre-quantization transformation from 0.26x of QuaRot to merely
0.07x, bringing up to 2.3x speedup for prefill and
1.7x speedup for decoding, respectively. Code is available at:
https://github.com/ruikangliu/FlatQuant.