Orionnnn/hub-stats huggingface dataset:
AutoVFX: Physically Realistic Video Editing from Natural Language
  Instructions
Authors: name: Hao-Yu Hsu, name: Zhi-Hao Lin, name: Albert Zhai, name: Hongchi Xia, name: Shenlong Wang
Published: 2024-11-04 18:59:05

Modern visual effects (VFX) software has made it possible for skilled artists
to create imagery of virtually anything. However, the creation process remains
laborious, complex, and largely inaccessible to everyday users. In this work,
we present AutoVFX, a framework that automatically creates realistic and
dynamic VFX videos from a single video and natural language instructions. By
carefully integrating neural scene modeling, LLM-based code generation, and
physical simulation, AutoVFX is able to provide physically-grounded,
photorealistic editing effects that can be controlled directly using natural
language instructions. We conduct extensive experiments to validate AutoVFX's
efficacy across a diverse spectrum of videos and instructions. Quantitative and
qualitative results suggest that AutoVFX outperforms all competing methods by a
large margin in generative quality, instruction alignment, editing versatility,
and physical plausibility.
Constrained Diffusion Implicit Models
Authors: name: Vivek Jayaram, name: Ira Kemelmacher-Shlizerman, name: Steven M. Seitz, name: John Thickstun
Published: 2024-11-01 04:51:24

This paper describes an efficient algorithm for solving noisy linear inverse
problems using pretrained diffusion models. Extending the paradigm of denoising
diffusion implicit models (DDIM), we propose constrained diffusion implicit
models (CDIM) that modify the diffusion updates to enforce a constraint upon
the final output. For noiseless inverse problems, CDIM exactly satisfies the
constraints; in the noisy case, we generalize CDIM to satisfy an exact
constraint on the residual distribution of the noise. Experiments across a
variety of tasks and metrics show strong performance of CDIM, with analogous
inference acceleration to unconstrained DDIM: 10 to 50 times faster than
previous conditional diffusion methods. We demonstrate the versatility of our
approach on many problems including super-resolution, denoising, inpainting,
deblurring, and 3D point cloud reconstruction.
IGOR: Image-GOal Representations are the Atomic Control Units for
  Foundation Models in Embodied AI
Authors: name: Xiaoyu Chen, name: Junliang Guo, name: Tianyu He, name: Chuheng Zhang, name: Pushi Zhang, name: Derek Cathera Yang, name: Li Zhao, name: Jiang Bian
Published: 2024-10-17 13:41:16

We introduce Image-GOal Representations (IGOR), aiming to learn a unified,
semantically consistent action space across human and various robots. Through
this unified latent action space, IGOR enables knowledge transfer among
large-scale robot and human activity data. We achieve this by compressing
visual changes between an initial image and its goal state into latent actions.
IGOR allows us to generate latent action labels for internet-scale video data.
This unified latent action space enables the training of foundation policy and
world models across a wide variety of tasks performed by both robots and
humans. We demonstrate that: (1) IGOR learns a semantically consistent action
space for both human and robots, characterizing various possible motions of
objects representing the physical interaction knowledge; (2) IGOR can "migrate"
the movements of the object in the one video to other videos, even across human
and robots, by jointly using the latent action model and world model; (3) IGOR
can learn to align latent actions with natural language through the foundation
policy model, and integrate latent actions with a low-level policy model to
achieve effective robot control. We believe IGOR opens new possibilities for
human-to-robot knowledge transfer and control.
LoRA-Contextualizing Adaptation of Large Multimodal Models for Long
  Document Understanding
Authors: name: Jian Chen, name: Ruiyi Zhang, name: Yufan Zhou, name: Tong Yu, name: Franck Dernoncourt, name: Jiuxiang Gu, name: Ryan A. Rossi, name: Changyou Chen, name: Tong Sun
Published: 2024-11-02 02:09:01

Large multimodal models (LMMs) have recently shown great progress in
text-rich image understanding, yet they still struggle with complex,
multi-page, visually-rich documents. Traditional methods using document parsers
for retrieval-augmented generation suffer from performance and efficiency
limitations, while directly presenting all pages to LMMs leads to
inefficiencies, especially with lengthy documents. In this work, we present a
novel framework named LoRA-Contextualizing Adaptation of Large multimodal
models (LoCAL), which broadens the capabilities of any LMM to support
long-document understanding. We demonstrate that LMMs can effectively serve as
multimodal retrievers, fetching relevant pages to answer user questions based
on these pages. LoCAL is implemented with two specific LMM adapters: one for
evidence page retrieval and another for question answering. Empirical results
show state-of-the-art performance on public benchmarks, demonstrating the
effectiveness of LoCAL.
Multi-expert Prompting Improves Reliability, Safety, and Usefulness of
  Large Language Models
Authors: name: Do Xuan Long, name: Duong Ngoc Yen, name: Anh Tuan Luu, name: Kenji Kawaguchi, name: Min-Yen Kan, name: Nancy F. Chen
Published: 2024-11-01 10:06:52

We present Multi-expert Prompting, a novel enhancement of ExpertPrompting (Xu
et al., 2023), designed to improve the large language model (LLM) generation.
Specifically, it guides an LLM to fulfill an input instruction by simulating
multiple experts, aggregating their responses, and selecting the best among
individual and aggregated responses. This process is performed in a single
chain of thoughts through our seven carefully designed subtasks derived from
the Nominal Group Technique (Ven and Delbecq, 1974), a well-established
decision-making framework. Our evaluations demonstrate that Multi-expert
Prompting significantly outperforms ExpertPrompting and comparable baselines in
enhancing the truthfulness, factuality, informativeness, and usefulness of
responses while reducing toxicity and hurtfulness. It further achieves
state-of-the-art truthfulness by outperforming the best baseline by 8.69% with
ChatGPT. Multi-expert Prompting is efficient, explainable, and highly adaptable
to diverse scenarios, eliminating the need for manual prompt construction.
MVPaint: Synchronized Multi-View Diffusion for Painting Anything 3D
Authors: name: Wei Cheng, name: Juncheng Mu, name: Xianfang Zeng, name: Xin Chen, name: Anqi Pang, name: Chi Zhang, name: Zhibin Wang, name: Bin Fu, name: Gang Yu, name: Ziwei Liu, name: Liang Pan
Published: 2024-11-04 17:59:39

Texturing is a crucial step in the 3D asset production workflow, which
enhances the visual appeal and diversity of 3D assets. Despite recent
advancements in Text-to-Texture (T2T) generation, existing methods often yield
subpar results, primarily due to local discontinuities, inconsistencies across
multiple views, and their heavy dependence on UV unwrapping outcomes. To tackle
these challenges, we propose a novel generation-refinement 3D texturing
framework called MVPaint, which can generate high-resolution, seamless textures
while emphasizing multi-view consistency. MVPaint mainly consists of three key
modules. 1) Synchronized Multi-view Generation (SMG). Given a 3D mesh model,
MVPaint first simultaneously generates multi-view images by employing an SMG
model, which leads to coarse texturing results with unpainted parts due to
missing observations. 2) Spatial-aware 3D Inpainting (S3I). To ensure complete
3D texturing, we introduce the S3I method, specifically designed to effectively
texture previously unobserved areas. 3) UV Refinement (UVR). Furthermore,
MVPaint employs a UVR module to improve the texture quality in the UV space,
which first performs a UV-space Super-Resolution, followed by a Spatial-aware
Seam-Smoothing algorithm for revising spatial texturing discontinuities caused
by UV unwrapping. Moreover, we establish two T2T evaluation benchmarks: the
Objaverse T2T benchmark and the GSO T2T benchmark, based on selected
high-quality 3D meshes from the Objaverse dataset and the entire GSO dataset,
respectively. Extensive experimental results demonstrate that MVPaint surpasses
existing state-of-the-art methods. Notably, MVPaint could generate
high-fidelity textures with minimal Janus issues and highly enhanced cross-view
consistency.
PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance
Authors: name: Ruyang Liu, name: Haoran Tang, name: Haibo Liu, name: Yixiao Ge, name: Ying Shan, name: Chen Li, name: Jiankun Yang
Published: 2024-11-04 17:50:36

The past year has witnessed the significant advancement of video-based large
language models. However, the challenge of developing a unified model for both
short and long video understanding remains unresolved. Most existing video LLMs
cannot handle hour-long videos, while methods custom for long videos tend to be
ineffective for shorter videos and images. In this paper, we identify the key
issue as the redundant content in videos. To address this, we propose a novel
pooling strategy that simultaneously achieves token compression and
instruction-aware visual feature aggregation. Our model is termed Prompt-guided
Pooling LLaVA, or PPLLaVA for short. Specifically, PPLLaVA consists of three
core components: the CLIP-based visual-prompt alignment that extracts visual
information relevant to the user's instructions, the prompt-guided pooling that
compresses the visual sequence to arbitrary scales using convolution-style
pooling, and the clip context extension designed for lengthy prompt common in
visual dialogue. Moreover, our codebase also integrates the most advanced video
Direct Preference Optimization (DPO) and visual interleave training. Extensive
experiments have validated the performance of our model. With superior
throughput and only 1024 visual context, PPLLaVA achieves better results on
image benchmarks as a video LLM, while achieving state-of-the-art performance
across various video benchmarks, excelling in tasks ranging from caption
generation to multiple-choice questions, and handling video lengths from
seconds to hours. Codes have been available at
https://github.com/farewellthree/PPLLaVA.
"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM
  Quantization
Authors: name: Eldar Kurtic, name: Alexandre Marques, name: Shubhra Pandit, name: Mark Kurtz, name: Dan Alistarh
Published: 2024-11-04 18:21:59

Despite the popularity of large language model (LLM) quantization for
inference acceleration, significant uncertainty remains regarding the
accuracy-performance trade-offs associated with various quantization formats.
We present a comprehensive empirical study of quantized accuracy, evaluating
popular quantization formats (FP8, INT8, INT4) across academic benchmarks and
real-world tasks, on the entire Llama-3.1 model family. Additionally, our study
examines the difference in text generated by quantized models versus their
uncompressed counterparts. Beyond benchmarks, we also present a couple of
quantization improvements which allowed us to obtain state-of-the-art accuracy
recovery results. Our investigation, encompassing over 500,000 individual
evaluations, yields several key findings: (1) FP8 weight and activation
quantization (W8A8-FP) is lossless across all model scales, (2) INT8 weight and
activation quantization (W8A8-INT), when properly tuned, incurs surprisingly
low 1-3% accuracy degradation, and (3) INT4 weight-only quantization
(W4A16-INT) is competitive with 8-bit integer weight and activation
quantization. To address the question of the "best" format for a given
deployment environment, we conduct inference performance analysis using the
popular open-source vLLM framework on various GPU architectures. We find that
W4A16 offers the best cost-efficiency for synchronous deployments, and for
asynchronous deployment on mid-tier GPUs. At the same time, W8A8 formats excel
in asynchronous "continuous batching" deployment of mid- and large-size models
on high-end GPUs. Our results provide a set of practical guidelines for
deploying quantized LLMs across scales and performance requirements.
Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and
  Cross-Cultural Embedding Models and Benchmarks
Authors: name: Gagan Bhatia, name: El Moatez Billah Nagoudi, name: Abdellah El Mekki, name: Fakhraddin Alwajih, name: Muhammad Abdul-Mageed
Published: 2024-11-02 09:39:49

We introduce Swan, a family of embedding models centred around the Arabic
language, addressing both small-scale and large-scale use cases. Swan includes
two variants: Swan-Small, based on ARBERTv2, and Swan-Large, built on
ArMistral, a pretrained Arabic large language model. To evaluate these models,
we propose ArabicMTEB, a comprehensive benchmark suite that assesses
cross-lingual, multi-dialectal, multi-domain, and multi-cultural Arabic text
embedding performance, covering eight diverse tasks and spanning 94 datasets.
Swan-Large achieves state-of-the-art results, outperforming
Multilingual-E5-large in most Arabic tasks, while the Swan-Small consistently
surpasses Multilingual-E5 base. Our extensive evaluations demonstrate that Swan
models are both dialectally and culturally aware, excelling across various
Arabic domains while offering significant monetary efficiency. This work
significantly advances the field of Arabic language modelling and provides
valuable resources for future research and applications in Arabic natural
language processing. Our models and benchmark will be made publicly accessible
for research.
Survey of Cultural Awareness in Language Models: Text and Beyond
Authors: name: Siddhesh Pawar, name: Junyeong Park, name: Jiho Jin, name: Arnav Arora, name: Junho Myung, name: Srishti Yadav, name: Faiz Ghifari Haznitrama, name: Inhwa Song, name: Alice Oh, name: Isabelle Augenstein
Published: 2024-10-30 16:37:50

Large-scale deployment of large language models (LLMs) in various
applications, such as chatbots and virtual assistants, requires LLMs to be
culturally sensitive to the user to ensure inclusivity. Culture has been widely
studied in psychology and anthropology, and there has been a recent surge in
research on making LLMs more culturally inclusive in LLMs that goes beyond
multilinguality and builds on findings from psychology and anthropology. In
this paper, we survey efforts towards incorporating cultural awareness into
text-based and multimodal LLMs. We start by defining cultural awareness in
LLMs, taking the definitions of culture from anthropology and psychology as a
point of departure. We then examine methodologies adopted for creating
cross-cultural datasets, strategies for cultural inclusion in downstream tasks,
and methodologies that have been used for benchmarking cultural awareness in
LLMs. Further, we discuss the ethical implications of cultural alignment, the
role of Human-Computer Interaction in driving cultural inclusion in LLMs, and
the role of cultural alignment in driving social science research. We finally
provide pointers to future research based on our findings about gaps in the
literature.
DynaSaur: Large Language Agents Beyond Predefined Actions
Authors: name: Dang Nguyen, name: Viet Dac Lai, name: Seunghyun Yoon, name: Ryan A. Rossi, name: Handong Zhao, name: Ruiyi Zhang, name: Puneet Mathur, name: Nedim Lipka, name: Yu Wang, name: Trung Bui, name: Franck Dernoncourt, name: Tianyi Zhou
Published: 2024-11-04 02:08:59

Existing LLM agent systems typically select actions from a fixed and
predefined set at every step. While this approach is effective in closed,
narrowly-scoped environments, we argue that it presents two major challenges
when deploying LLM agents in real-world scenarios: (1) selecting from a fixed
set of actions significantly restricts the planning and acting capabilities of
LLM agents, and (2) this approach requires substantial human effort to
enumerate and implement all possible actions, which becomes impractical in
complex environments with a vast number of potential actions. In this work, we
propose an LLM agent framework that enables the dynamic creation and
composition of actions in an online manner. In this framework, the agent
interacts with the environment by generating and executing programs written in
a general-purpose programming language at each step. Furthermore, generated
actions are accumulated over time for future reuse. Our extensive experiments
on the GAIA benchmark demonstrate that this framework offers significantly
greater flexibility and outperforms previous methods. Notably, it allows an LLM
agent to recover in scenarios where no relevant action exists in the predefined
set or when existing actions fail due to unforeseen edge cases. At the time of
writing, we hold the top position on the GAIA public leaderboard. Our code can
be found in
https://github.com/adobe-research/dynasaur{https://github.com/adobe-research/dynasaur}.
Training-free Regional Prompting for Diffusion Transformers
Authors: name: Anthony Chen, name: Jianjin Xu, name: Wenzhao Zheng, name: Gaole Dai, name: Yida Wang, name: Renrui Zhang, name: Haofan Wang, name: Shanghang Zhang
Published: 2024-11-04 18:59:05

Diffusion models have demonstrated excellent capabilities in text-to-image
generation. Their semantic understanding (i.e., prompt following) ability has
also been greatly improved with large language models (e.g., T5, Llama).
However, existing models cannot perfectly handle long and complex text prompts,
especially when the text prompts contain various objects with numerous
attributes and interrelated spatial relationships. While many regional
prompting methods have been proposed for UNet-based models (SD1.5, SDXL), but
there are still no implementations based on the recent Diffusion Transformer
(DiT) architecture, such as SD3 and FLUX.1.In this report, we propose and
implement regional prompting for FLUX.1 based on attention manipulation, which
enables DiT with fined-grained compositional text-to-image generation
capability in a training-free manner. Code is available at
https://github.com/antonioo-c/Regional-Prompting-FLUX.
GenXD: Generating Any 3D and 4D Scenes
Authors: name: Yuyang Zhao, name: Chung-Ching Lin, name: Kevin Lin, name: Zhiwen Yan, name: Linjie Li, name: Zhengyuan Yang, name: Jianfeng Wang, name: Gim Hee Lee, name: Lijuan Wang
Published: 2024-11-04 17:45:44

Recent developments in 2D visual generation have been remarkably successful.
However, 3D and 4D generation remain challenging in real-world applications due
to the lack of large-scale 4D data and effective model design. In this paper,
we propose to jointly investigate general 3D and 4D generation by leveraging
camera and object movements commonly observed in daily life. Due to the lack of
real-world 4D data in the community, we first propose a data curation pipeline
to obtain camera poses and object motion strength from videos. Based on this
pipeline, we introduce a large-scale real-world 4D scene dataset: CamVid-30K.
By leveraging all the 3D and 4D data, we develop our framework, GenXD, which
allows us to produce any 3D or 4D scene. We propose multiview-temporal modules,
which disentangle camera and object movements, to seamlessly learn from both 3D
and 4D data. Additionally, GenXD employs masked latent conditions to support a
variety of conditioning views. GenXD can generate videos that follow the camera
trajectory as well as consistent 3D views that can be lifted into 3D
representations. We perform extensive evaluations across various real-world and
synthetic datasets, demonstrating GenXD's effectiveness and versatility
compared to previous methods in 3D and 4D generation.
Sparsing Law: Towards Large Language Models with Greater Activation
  Sparsity
Authors: name: Yuqi Luo, name: Chenyang Song, name: Xu Han, name: Yingfa Chen, name: Chaojun Xiao, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-11-04 17:59:04

Activation sparsity denotes the existence of substantial weakly-contributed
elements within activation outputs that can be eliminated, benefiting many
important applications concerned with large language models (LLMs). Although
promoting greater activation sparsity within LLMs deserves deep studies,
existing works lack comprehensive and quantitative research on the correlation
between activation sparsity and potentially influential factors. In this paper,
we present a comprehensive study on the quantitative scaling properties and
influential factors of the activation sparsity within decoder-only
Transformer-based LLMs. Specifically, we propose PPL-p% sparsity, a precise
and performance-aware activation sparsity metric that is applicable to any
activation function. Through extensive experiments, we find several important
phenomena. Firstly, different activation functions exhibit comparable
performance but opposite training-time sparsity trends. The activation ratio
(i.e., 1-sparsity ratio) evolves as a convergent increasing
power-law and decreasing logspace power-law with the amount of training data
for SiLU-activated and ReLU-activated LLMs, respectively. These demonstrate
that ReLU is more efficient as the activation function than SiLU and can
leverage more training data to improve activation sparsity. Secondly, the
activation ratio linearly increases with the width-depth ratio below a certain
bottleneck point, indicating the potential advantage of a deeper architecture
at a fixed parameter scale. Finally, at similar width-depth ratios, we
surprisingly find that the limit value of activation sparsity varies weakly
with the parameter scale, i.e., the activation patterns within LLMs are
insensitive to the parameter scale. These empirical laws towards LLMs with
greater activation sparsity have important implications for making LLMs more
efficient and interpretable.
AndroidLab: Training and Systematic Benchmarking of Android Autonomous
  Agents
Authors: name: Yifan Xu, name: Xiao Liu, name: Xueqiao Sun, name: Siyi Cheng, name: Hao Yu, name: Hanyu Lai, name: Shudan Zhang, name: Dan Zhang, name: Jie Tang, name: Yuxiao Dong
Published: 2024-10-31 15:25:20

Autonomous agents have become increasingly important for interacting with the
real world. Android agents, in particular, have been recently a
frequently-mentioned interaction method. However, existing studies for training
and evaluating Android agents lack systematic research on both open-source and
closed-source models. In this work, we propose AndroidLab as a systematic
Android agent framework. It includes an operation environment with different
modalities, action space, and a reproducible benchmark. It supports both large
language models (LLMs) and multimodal models (LMMs) in the same action space.
AndroidLab benchmark includes predefined Android virtual devices and 138 tasks
across nine apps built on these devices. By using the AndroidLab environment,
we develop an Android Instruction dataset and train six open-source LLMs and
LMMs, lifting the average success rates from 4.59% to 21.50% for LLMs and from
1.93% to 13.28% for LMMs. AndroidLab is open-sourced and publicly available at
https://github.com/THUDM/Android-Lab.