Orionnnn/hub-stats huggingface dataset:
ClotheDreamer: Text-Guided Garment Generation with 3D Gaussians
Authors: name: Yufei Liu, name: Junshu Tang, name: Chu Zheng, name: Shijie Zhang, name: Jinkun Hao, name: Junwei Zhu, name: Dongjin Huang
Published: 2024-06-24 17:25:39

High-fidelity 3D garment synthesis from text is desirable yet challenging for
digital avatar creation. Recent diffusion-based approaches via Score
Distillation Sampling (SDS) have enabled new possibilities but either
intricately couple with human body or struggle to reuse. We introduce
ClotheDreamer, a 3D Gaussian-based method for generating wearable,
production-ready 3D garment assets from text prompts. We propose a novel
representation Disentangled Clothe Gaussian Splatting (DCGS) to enable separate
optimization. DCGS represents clothed avatar as one Gaussian model but freezes
body Gaussian splats. To enhance quality and completeness, we incorporate
bidirectional SDS to supervise clothed avatar and garment RGBD renderings
respectively with pose conditions and propose a new pruning strategy for loose
clothing. Our approach can also support custom clothing templates as input.
Benefiting from our design, the synthetic 3D garment can be easily applied to
virtual try-on and support physically accurate animation. Extensive experiments
showcase our method's superior and competitive performance. Our project page is
at https://ggxxii.github.io/clothedreamer.
WARP: On the Benefits of Weight Averaged Rewarded Policies
Authors: name: Alexandre Ramé, name: Johan Ferret, name: Nino Vieillard, name: Robert Dadashi, name: Léonard Hussenot, name: Pierre-Louis Cedoz, name: Pier Giuseppe Sessa, name: Sertan Girgin, name: Arthur Douillard, name: Olivier Bachem
Published: 2024-06-24 16:24:34

Reinforcement learning from human feedback (RLHF) aligns large language
models (LLMs) by encouraging their generations to have high rewards, using a
reward model trained on human preferences. To prevent the forgetting of
pre-trained knowledge, RLHF usually incorporates a KL regularization; this
forces the policy to remain close to its supervised fine-tuned initialization,
though it hinders the reward optimization. To tackle the trade-off between KL
and reward, in this paper we introduce a novel alignment strategy named Weight
Averaged Rewarded Policies (WARP). WARP merges policies in the weight space at
three distinct stages. First, it uses the exponential moving average of the
policy as a dynamic anchor in the KL regularization. Second, it applies
spherical interpolation to merge independently fine-tuned policies into a new
enhanced one. Third, it linearly interpolates between this merged model and the
initialization, to recover features from pre-training. This procedure is then
applied iteratively, with each iteration's final model used as an advanced
initialization for the next, progressively refining the KL-reward Pareto front,
achieving superior rewards at fixed KL. Experiments with GEMMA policies
validate that WARP improves their quality and alignment, outperforming other
open-source LLMs.
Scaling Laws for Linear Complexity Language Models
Authors: name: Xuyang Shen, name: Dong Li, name: Ruitao Leng, name: Zhen Qin, name: Weigao Sun, name: Yiran Zhong
Published: 2024-06-24 14:51:31

The interest in linear complexity models for large language models is on the
rise, although their scaling capacity remains uncertain. In this study, we
present the scaling laws for linear complexity language models to establish a
foundation for their scalability. Specifically, we examine the scaling
behaviors of three efficient linear architectures. These include TNL, a linear
attention model with data-independent decay; HGRN2, a linear RNN with
data-dependent decay; and cosFormer2, a linear attention model without decay.
We also include LLaMA as a baseline architecture for softmax attention for
comparison. These models were trained with six variants, ranging from 70M to 7B
parameters on a 300B-token corpus, and evaluated with a total of 1,376
intermediate checkpoints on various downstream tasks. These tasks include
validation loss, commonsense reasoning, and information retrieval and
generation. The study reveals that existing linear complexity language models
exhibit similar scaling capabilities as conventional transformer-based models
while also demonstrating superior linguistic proficiency and knowledge
retention.
Preference Tuning For Toxicity Mitigation Generalizes Across Languages
Authors: name: Xiaochen Li, name: Zheng-Xin Yong, name: Stephen H. Bach
Published: 2024-06-23 22:53:47

Detoxifying multilingual Large Language Models (LLMs) has become crucial due
to their increasing global use. In this work, we explore zero-shot
cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike
previous studies that show limited cross-lingual generalization for other
safety tasks, we demonstrate that Direct Preference Optimization (DPO) training
with only English data can significantly reduce toxicity in multilingual
open-ended generations. For example, the probability of mGPT-1.3B generating
toxic continuations drops from 46.8% to 3.9% across 17 different languages
after training. Our results also extend to other multilingual LLMs, such as
BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal
intervention and activation analysis, we identified the dual multilinguality
property of MLP layers in LLMs, which explains the cross-lingual generalization
of DPO. Finally, we show that bilingual sentence retrieval can predict the
cross-lingual transferability of DPO preference tuning.
Long Context Transfer from Language to Vision
Authors: name: Peiyuan Zhang, name: Kaichen Zhang, name: Bo Li, name: Guangtao Zeng, name: Jingkang Yang, name: Yuanhan Zhang, name: Ziyue Wang, name: Haoran Tan, name: Chunyuan Li, name: Ziwei Liu
Published: 2024-06-24 17:58:06

Video sequences offer valuable temporal information, but existing large
multimodal models (LMMs) fall short in understanding extremely long videos.
Many works address this by reducing the number of visual tokens using visual
resamplers. Alternatively, in this paper, we approach this problem from the
perspective of the language model. By simply extrapolating the context length
of the language backbone, we enable LMMs to comprehend orders of magnitude more
visual tokens without any video training. We call this phenomenon long context
transfer and carefully ablate its properties. To effectively measure LMMs'
ability to generalize to long contexts in the vision modality, we develop
V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark
inspired by the language model's NIAH test. Our proposed Long Video Assistant
(LongVA) can process 2000 frames or over 200K visual tokens without additional
complexities. With its extended context length, LongVA achieves
state-of-the-art performance on Video-MME among 7B-scale models by densely
sampling more input frames. Our work is open-sourced at
https://github.com/EvolvingLMMs-Lab/LongVA.
Efficient Continual Pre-training by Mitigating the Stability Gap
Authors: name: Yiduo Guo, name: Jie Fu, name: Huishuai Zhang, name: Dongyan Zhao, name: Yikang Shen
Published: 2024-06-21 02:28:37

Continual pre-training has increasingly become the predominant approach for
adapting Large Language Models (LLMs) to new domains. This process involves
updating the pre-trained LLM with a corpus from a new domain, resulting in a
shift in the training distribution. To study the behavior of LLMs during this
shift, we measured the model's performance throughout the continual
pre-training process. we observed a temporary performance drop at the
beginning, followed by a recovery phase, a phenomenon known as the "stability
gap," previously noted in vision models classifying new classes. To address
this issue and enhance LLM performance within a fixed compute budget, we
propose three effective strategies: (1) Continually pre-training the LLM on a
subset with a proper size for multiple epochs, resulting in faster performance
recovery than pre-training the LLM on a large corpus in a single epoch; (2)
Pre-training the LLM only on high-quality sub-corpus, which rapidly boosts
domain performance; and (3) Using a data mixture similar to the pre-training
data to reduce distribution gap. We conduct various experiments on Llama-family
models to validate the effectiveness of our strategies in both medical
continual pre-training and instruction tuning. For example, our strategies
improve the average medical task performance of the OpenLlama-3B model from
36.2% to 40.7% with only 40% of the original training budget and enhance the
average general task performance without causing forgetting. Furthermore, we
apply our strategies to the Llama-3-8B model. The resulting model,
Llama-3-Physician, achieves the best medical performance among current
open-source models, and performs comparably to or even better than GPT-4 on
several medical benchmarks. We release our models at
https://huggingface.co/YiDuo1999/Llama-3-Physician-8B-Instruct.
DreamBench++: A Human-Aligned Benchmark for Personalized Image
  Generation
Authors: name: Yuang Peng, name: Yuxin Cui, name: Haomiao Tang, name: Zekun Qi, name: Runpei Dong, name: Jing Bai, name: Chunrui Han, name: Zheng Ge, name: Xiangyu Zhang, name: Shu-Tao Xia
Published: 2024-06-24 17:58:47

Personalized image generation holds great promise in assisting humans in
everyday work and life due to its impressive function in creatively generating
personalized content. However, current evaluations either are automated but
misalign with humans or require human evaluations that are time-consuming and
expensive. In this work, we present DreamBench++, a human-aligned benchmark
automated by advanced multimodal GPT models. Specifically, we systematically
design the prompts to let GPT be both human-aligned and self-aligned, empowered
with task reinforcement. Further, we construct a comprehensive dataset
comprising diverse images and prompts. By benchmarking 7 modern generative
models, we demonstrate that DreamBench++ results in significantly more
human-aligned evaluation, helping boost the community with innovative findings.
VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in
  Large Video-Language Models
Authors: name: Yuxuan Wang, name: Yueqian Wang, name: Dongyan Zhao, name: Cihang Xie, name: Zilong Zheng
Published: 2024-06-24 06:21:59

Recent advancements in Multimodal Large Language Models (MLLMs) have extended
their capabilities to video understanding. Yet, these models are often plagued
by "hallucinations", where irrelevant or nonsensical content is generated,
deviating from the actual video context. This work introduces VideoHallucer,
the first comprehensive benchmark for hallucination detection in large
video-language models (LVLMs). VideoHallucer categorizes hallucinations into
two main types: intrinsic and extrinsic, offering further subcategories for
detailed analysis, including object-relation, temporal, semantic detail,
extrinsic factual, and extrinsic non-factual hallucinations. We adopt an
adversarial binary VideoQA method for comprehensive evaluation, where pairs of
basic and hallucinated questions are crafted strategically. By evaluating
eleven LVLMs on VideoHallucer, we reveal that i) the majority of current models
exhibit significant issues with hallucinations; ii) while scaling datasets and
parameters improves models' ability to detect basic visual cues and
counterfactuals, it provides limited benefit for detecting extrinsic factual
hallucinations; iii) existing models are more adept at detecting facts than
identifying hallucinations. As a byproduct, these analyses further instruct the
development of our self-PEP framework, achieving an average of 5.38%
improvement in hallucination resistance across all model architectures.
Video-Infinity: Distributed Long Video Generation
Authors: name: Zhenxiong Tan, name: Xingyi Yang, name: Songhua Liu, name: Xinchao Wang
Published: 2024-06-24 01:56:12

Diffusion models have recently achieved remarkable results for video
generation. Despite the encouraging performances, the generated videos are
typically constrained to a small number of frames, resulting in clips lasting
merely a few seconds. The primary challenges in producing longer videos include
the substantial memory requirements and the extended processing time required
on a single GPU. A straightforward solution would be to split the workload
across multiple GPUs, which, however, leads to two issues: (1) ensuring all
GPUs communicate effectively to share timing and context information, and (2)
modifying existing video diffusion models, which are usually trained on short
sequences, to create longer videos without additional training. To tackle
these, in this paper we introduce Video-Infinity, a distributed inference
pipeline that enables parallel processing across multiple GPUs for long-form
video generation. Specifically, we propose two coherent mechanisms: Clip
parallelism and Dual-scope attention. Clip parallelism optimizes the gathering
and sharing of context information across GPUs which minimizes communication
overhead, while Dual-scope attention modulates the temporal self-attention to
balance local and global contexts efficiently across the devices. Together, the
two mechanisms join forces to distribute the workload and enable the fast
generation of long videos. Under an 8 x Nvidia 6000 Ada GPU (48G) setup, our
method generates videos up to 2,300 frames in approximately 5 minutes, enabling
long video generation at a speed 100 times faster than the prior methods.
Towards Fast Multilingual LLM Inference: Speculative Decoding and
  Specialized Drafters
Authors: name: Euiin Yi, name: Taehyeon Kim, name: Hongseok Jeung, name: Du-Seong Chang, name: Se-Young Yun
Published: 2024-06-24 16:06:50

Large language models (LLMs) have revolutionized natural language processing
and broadened their applicability across diverse commercial applications.
However, the deployment of these models is constrained by high inference time
in multilingual settings. To mitigate this challenge, this paper explores a
training recipe of an assistant model in speculative decoding, which are
leveraged to draft and-then its future tokens are verified by the target LLM.
We show that language-specific draft models, optimized through a targeted
pretrain-and-finetune strategy, substantially brings a speedup of inference
time compared to the previous methods. We validate these models across various
languages in inference time, out-of-domain speedup, and GPT-4o evaluation.
ICAL: Continual Learning of Multimodal Agents by Transforming
  Trajectories into Actionable Insights
Authors: name: Gabriel Sarch, name: Lawrence Jang, name: Michael J. Tarr, name: William W. Cohen, name: Kenneth Marino, name: Katerina Fragkiadaki
Published: 2024-06-20 17:45:02

Large-scale generative language and vision-language models (LLMs and VLMs)
excel in few-shot in-context learning for decision making and instruction
following. However, they require high-quality exemplar demonstrations to be
included in their context window. In this work, we ask: Can LLMs and VLMs
generate their own prompt examples from generic, sub-optimal demonstrations? We
propose In-Context Abstraction Learning (ICAL), a method that builds a memory
of multimodal experience insights from sub-optimal demonstrations and human
feedback. Given a noisy demonstration in a new domain, VLMs abstract the
trajectory into a general program by fixing inefficient actions and annotating
cognitive abstractions: task relationships, object state changes, temporal
subgoals, and task construals. These abstractions are refined and adapted
interactively through human feedback while the agent attempts to execute the
trajectory in a similar environment. The resulting abstractions, when used as
exemplars in the prompt, significantly improve decision-making in
retrieval-augmented LLM and VLM agents. Our ICAL agent surpasses the
state-of-the-art in dialogue-based instruction following in TEACh, multimodal
web agents in VisualWebArena, and action anticipation in Ego4D. In TEACh, we
achieve a 12.6% improvement in goal-condition success. In VisualWebArena, our
task success rate improves over the SOTA from 14.3% to 22.7%. In Ego4D action
forecasting, we improve over few-shot GPT-4V and remain competitive with
supervised models. We show finetuning our retrieval-augmented in-context agent
yields additional improvements. Our approach significantly reduces reliance on
expert-crafted examples and consistently outperforms in-context learning from
action plans that lack such insights.
Data Contamination Can Cross Language Barriers
Authors: name: Feng Yao, name: Yufan Zhuang, name: Zihao Sun, name: Sunan Xu, name: Animesh Kumar, name: Jingbo Shang
Published: 2024-06-19 05:53:27

The opacity in developing large language models (LLMs) is raising growing
concerns about the potential contamination of public benchmarks in the
pre-training data. Existing contamination detection methods are typically based
on the text overlap between training and evaluation data, which can be too
superficial to reflect deeper forms of contamination. In this paper, we first
present a cross-lingual form of contamination that inflates LLMs' performance
while evading current detection methods, deliberately injected by overfitting
LLMs on the translated versions of benchmark test sets. Then, we propose
generalization-based approaches to unmask such deeply concealed contamination.
Specifically, we examine the LLM's performance change after modifying the
original benchmark by replacing the false answer choices with correct ones from
other questions. Contaminated models can hardly generalize to such easier
situations, where the false choices can be not even wrong, as all
choices are correct in their memorization. Experimental results demonstrate
that cross-lingual contamination can easily fool existing detection methods,
but not ours. In addition, we discuss the potential utilization of
cross-lingual contamination in interpreting LLMs' working mechanisms and in
post-training LLMs for enhanced multilingual capabilities. The code and dataset
we use can be obtained from https://github.com/ShangDataLab/Deep-Contam.
DELLA-Merging: Reducing Interference in Model Merging through
  Magnitude-Based Sampling
Authors: name: Pala Tej Deep, name: Rishabh Bhardwaj, name: Soujanya Poria
Published: 2024-06-17 15:02:45

With the proliferation of domain-specific models, model merging has emerged
as a set of techniques that combine the capabilities of multiple models into
one that can multitask without the cost of additional training. In this paper,
we propose a new model merging technique, Drop and rEscaLe via sampLing with
mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE,
which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the
parameters in order of their magnitude and assigns higher dropout probabilities
(p) to parameters with lower ranks corresponding to lower magnitudes. To
approximate the original embeddings, MAGPRUNE employs a rescaling operation on
the parameters that survive the random dropping by 1/(1 - p). On three
different expert models considered for merging (LM, Math, Code) and
corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an
average improvement of 2.4 points over baseline methods employing delta
parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over
DARE), and 11.1 points over the no-pruning baseline (TA). We release the source
code at: https://github.com/declare-lab/della.
Ruby Teaming: Improving Quality Diversity Search with Memory for
  Automated Red Teaming
Authors: name: Vernon Toh Yan Han, name: Rishabh Bhardwaj, name: Soujanya Poria
Published: 2024-06-17 15:36:14

We propose Ruby Teaming, a method that improves on Rainbow Teaming by
including a memory cache as its third dimension. The memory dimension provides
cues to the mutator to yield better-quality prompts, both in terms of attack
success rate (ASR) and quality diversity. The prompt archive generated by Ruby
Teaming has an ASR of 74%, which is 20% higher than the baseline. In terms of
quality diversity, Ruby Teaming outperforms Rainbow Teaming by 6% and 3% on
Shannon's Evenness Index (SEI) and Simpson's Diversity Index (SDI),
respectively.
4K4DGen: Panoramic 4D Generation at 4K Resolution
Authors: name: Renjie Li, name: Panwang Pan, name: Bangbang Yang, name: Dejia Xu, name: Shijie Zhou, name: Xuanyang Zhang, name: Zeming Li, name: Achuta Kadambi, name: Zhangyang Wang, name: Zhiwen Fan
Published: 2024-06-19 13:11:02

The blooming of virtual reality and augmented reality (VR/AR) technologies
has driven an increasing demand for the creation of high-quality, immersive,
and dynamic environments. However, existing generative techniques either focus
solely on dynamic objects or perform outpainting from a single perspective
image, failing to meet the needs of VR/AR applications. In this work, we tackle
the challenging task of elevating a single panorama to an immersive 4D
experience. For the first time, we demonstrate the capability to generate
omnidirectional dynamic scenes with 360-degree views at 4K resolution, thereby
providing an immersive user experience. Our method introduces a pipeline that
facilitates natural scene animations and optimizes a set of 4D Gaussians using
efficient splatting techniques for real-time exploration. To overcome the lack
of scene-scale annotated 4D data and models, especially in panoramic formats,
we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors
to animate consistently in 360-degree images, transforming them into panoramic
videos with dynamic scenes at targeted regions. Subsequently, we elevate the
panoramic video into a 4D immersive environment while preserving spatial and
temporal consistency. By transferring prior knowledge from 2D models in the
perspective domain to the panoramic domain and the 4D lifting with spatial
appearance and geometry regularization, we achieve high-quality Panorama-to-4D
generation at a resolution of (4096 times 2048) for the first time. See the
project website at https://4k4dgen.github.io.
Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework
Authors: name: Zackary Rackauckas, name: Arthur Câmara, name: Jakub Zavrel
Published: 2024-06-20 23:20:34

Challenges in the automated evaluation of Retrieval-Augmented Generation
(RAG) Question-Answering (QA) systems include hallucination problems in
domain-specific knowledge and the lack of gold standard benchmarks for company
internal tasks. This results in difficulties in evaluating RAG variations, like
RAG-Fusion (RAGF), in the context of a product QA task at Infineon
Technologies. To solve these problems, we propose a comprehensive evaluation
framework, which leverages Large Language Models (LLMs) to generate large
datasets of synthetic queries based on real user queries and in-domain
documents, uses LLM-as-a-judge to rate retrieved documents and answers,
evaluates the quality of answers, and ranks different variants of
Retrieval-Augmented Generation (RAG) agents with RAGElo's automated Elo-based
competition. LLM-as-a-judge rating of a random sample of synthetic queries
shows a moderate, positive correlation with domain expert scoring in relevance,
accuracy, completeness, and precision. While RAGF outperformed RAG in Elo
score, a significance analysis against expert annotations also shows that RAGF
significantly outperforms RAG in completeness, but underperforms in precision.
In addition, Infineon's RAGF assistant demonstrated slightly higher performance
in document relevance based on MRR@5 scores. We find that RAGElo positively
aligns with the preferences of human annotators, though due caution is still
required. Finally, RAGF's approach leads to more complete answers based on
expert annotations and better answers overall based on RAGElo's evaluation
criteria.