Orionnnn/hub-stats huggingface dataset:
Mind the Time: Temporally-Controlled Multi-Event Video Generation
Authors: name: Ziyi Wu, name: Aliaksandr Siarohin, name: Willi Menapace, name: Ivan Skorokhodov, name: Yuwei Fang, name: Varnith Chordia, name: Igor Gilitschenski, name: Sergey Tulyakov
Published: 2024-12-06 18:52:20

Real-world videos consist of sequences of events. Generating such sequences
with precise temporal control is infeasible with existing video generators that
rely on a single paragraph of text as input. When tasked with generating
multiple events described using a single prompt, such methods often ignore some
of the events or fail to arrange them in the correct order. To address this
limitation, we present MinT, a multi-event video generator with temporal
control. Our key insight is to bind each event to a specific period in the
generated video, which allows the model to focus on one event at a time. To
enable time-aware interactions between event captions and video tokens, we
design a time-based positional encoding method, dubbed ReRoPE. This encoding
helps to guide the cross-attention operation. By fine-tuning a pre-trained
video diffusion transformer on temporally grounded data, our approach produces
coherent videos with smoothly connected events. For the first time in the
literature, our model offers control over the timing of events in generated
videos. Extensive experiments demonstrate that MinT outperforms existing
open-source models by a large margin.
2DGS-Room: Seed-Guided 2D Gaussian Splatting with Geometric Constrains
  for High-Fidelity Indoor Scene Reconstruction
Authors: name: Wanting Zhang, name: Haodong Xiang, name: Zhichao Liao, name: Xiansong Lai, name: Xinghui Li, name: Long Zeng
Published: 2024-12-04 16:17:47

The reconstruction of indoor scenes remains challenging due to the inherent
complexity of spatial structures and the prevalence of textureless regions.
Recent advancements in 3D Gaussian Splatting have improved novel view synthesis
with accelerated processing but have yet to deliver comparable performance in
surface reconstruction. In this paper, we introduce 2DGS-Room, a novel method
leveraging 2D Gaussian Splatting for high-fidelity indoor scene reconstruction.
Specifically, we employ a seed-guided mechanism to control the distribution of
2D Gaussians, with the density of seed points dynamically optimized through
adaptive growth and pruning mechanisms. To further improve geometric accuracy,
we incorporate monocular depth and normal priors to provide constraints for
details and textureless regions respectively. Additionally, multi-view
consistency constraints are employed to mitigate artifacts and further enhance
reconstruction quality. Extensive experiments on ScanNet and ScanNet++ datasets
demonstrate that our method achieves state-of-the-art performance in indoor
scene reconstruction.
LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment
Authors: name: Yibin Wang, name: Zhiyu Tan, name: Junyan Wang, name: Xiaomeng Yang, name: Cheng Jin, name: Hao Li
Published: 2024-12-06 07:16:14

Recent advancements in text-to-video (T2V) generative models have shown
impressive capabilities. However, these models are still inadequate in aligning
synthesized videos with human preferences (e.g., accurately reflecting text
descriptions), which is particularly difficult to address, as human preferences
are inherently subjective and challenging to formalize as objective functions.
Therefore, this paper proposes LiFT, a novel fine-tuning method leveraging
human feedback for T2V model alignment. Specifically, we first construct a
Human Rating Annotation dataset, LiFT-HRA, consisting of approximately 10k
human annotations, each including a score and its corresponding rationale.
Based on this, we train a reward model LiFT-Critic to learn reward function
effectively, which serves as a proxy for human judgment, measuring the
alignment between given videos and human expectations. Lastly, we leverage the
learned reward function to align the T2V model by maximizing the
reward-weighted likelihood. As a case study, we apply our pipeline to
CogVideoX-2B, showing that the fine-tuned model outperforms the CogVideoX-5B
across all 16 metrics, highlighting the potential of human feedback in
improving the alignment and quality of synthesized videos.
Expanding Performance Boundaries of Open-Source Multimodal Models with
  Model, Data, and Test-Time Scaling
Authors: name: Zhe Chen, name: Weiyun Wang, name: Yue Cao, name: Yangzhou Liu, name: Zhangwei Gao, name: Erfei Cui, name: Jinguo Zhu, name: Shenglong Ye, name: Hao Tian, name: Zhaoyang Liu, name: Lixin Gu, name: Xuehui Wang, name: Qingyun Li, name: Yimin Ren, name: Zixuan Chen, name: Jiapeng Luo, name: Jiahao Wang, name: Tan Jiang, name: Bo Wang, name: Conghui He, name: Botian Shi, name: Xingcheng Zhang, name: Han Lv, name: Yi Wang, name: Wenqi Shao, name: Pei Chu, name: Zhongying Tu, name: Tong He, name: Zhiyong Wu, name: Huipeng Deng, name: Jiaye Ge, name: Kai Chen, name: Min Dou, name: Lewei Lu, name: Xizhou Zhu, name: Tong Lu, name: Dahua Lin, name: Yu Qiao, name: Jifeng Dai, name: Wenhai Wang
Published: 2024-12-06 18:57:08

We introduce InternVL 2.5, an advanced multimodal large language model (MLLM)
series that builds upon InternVL 2.0, maintaining its core model architecture
while introducing significant enhancements in training and testing strategies
as well as data quality. In this work, we delve into the relationship between
model scaling and performance, systematically exploring the performance trends
in vision encoders, language models, dataset sizes, and test-time
configurations. Through extensive evaluations on a wide range of benchmarks,
including multi-discipline reasoning, document understanding, multi-image /
video understanding, real-world comprehension, multimodal hallucination
detection, visual grounding, multilingual capabilities, and pure language
processing, InternVL 2.5 exhibits competitive performance, rivaling leading
commercial models such as GPT-4o and Claude-3.5-Sonnet. Notably, our model is
the first open-source MLLMs to surpass 70% on the MMMU benchmark, achieving a
3.7-point improvement through Chain-of-Thought (CoT) reasoning and showcasing
strong potential for test-time scaling. We hope this model contributes to the
open-source community by setting new standards for developing and applying
multimodal AI systems. HuggingFace demo see
https://huggingface.co/spaces/OpenGVLab/InternVL
APOLLO: SGD-like Memory, AdamW-level Performance
Authors: name: Hanqing Zhu, name: Zhenyu Zhang, name: Wenyan Cong, name: Xi Liu, name: Sem Park, name: Vikas Chandra, name: Bo Long, name: David Z. Pan, name: Zhangyang Wang, name: Jinwon Lee
Published: 2024-12-06 18:55:34

Large language models (LLMs) are notoriously memory-intensive during
training, particularly with the popular AdamW optimizer. This memory burden
necessitates using more or higher-end GPUs or reducing batch sizes, limiting
training scalability and throughput. To address this, various memory-efficient
optimizers have been proposed to reduce optimizer memory usage. However, they
face critical challenges: (i) reliance on costly SVD operations; (ii)
significant performance trade-offs compared to AdamW; and (iii) still
substantial optimizer memory overhead to maintain competitive performance.
  In this work, we identify that AdamW's learning rate adaptation rule can be
effectively coarsened as a structured learning rate update. Based on this
insight, we propose Approximated Gradient Scaling for Memory-Efficient LLM
Optimization (APOLLO), which approximates learning rate scaling using an
auxiliary low-rank optimizer state based on pure random projection. This
structured learning rate update rule makes APOLLO highly tolerant to further
memory reductions while delivering comparable pre-training performance. Even
its rank-1 variant, APOLLO-Mini, achieves superior pre-training performance
compared to AdamW with SGD-level memory costs.
  Extensive experiments demonstrate that the APOLLO series performs on-par with
or better than AdamW, while achieving greater memory savings by nearly
eliminating the optimization states of AdamW. These savings provide significant
system-level benefits: (1) Enhanced Throughput: 3x throughput on an 8xA100-80GB
setup compared to AdamW by supporting 4x larger batch sizes. (2) Improved Model
Scalability: Pre-training LLaMA-13B with naive DDP on A100-80GB GPUs without
system-level optimizations. (3) Low-End GPU Friendly Pre-training: Pre-training
LLaMA-7B on a single GPU using less than 12 GB of memory with weight
quantization.
MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at
  Scale
Authors: name: Jarvis Guo, name: Tuney Zheng, name: Yuelin Bai, name: Bo Li, name: Yubo Wang, name: King Zhu, name: Yizhi Li, name: Graham Neubig, name: Wenhu Chen, name: Xiang Yue
Published: 2024-12-06 18:14:24

Open-source multimodal large language models (MLLMs) have shown significant
potential in a broad range of multimodal tasks. However, their reasoning
capabilities remain constrained by existing instruction-tuning datasets, which
were predominately repurposed from academic datasets such as VQA, AI2D, and
ChartQA. These datasets target simplistic tasks, and only provide phrase-level
answers without any intermediate rationales. To address these challenges, we
introduce a scalable and cost-effective method to construct a large-scale
multimodal instruction-tuning dataset with rich intermediate rationales
designed to elicit CoT reasoning. Using only open models, we create a dataset
containing 12M instruction-response pairs to cover diverse, reasoning-intensive
tasks with detailed and faithful rationales. Experiments demonstrate that
training MLLMs on this dataset significantly improves reasoning capabilities,
achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%),
MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates
notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation
studies further highlight the importance of key components, such as rewriting
and self-filtering, in the dataset construction process.
EXAONE 3.5: Series of Large Language Models for Real-world Use Cases
Authors: name: LG AI Research, name: Soyoung An, name: Kyunghoon Bae, name: Eunbi Choi, name: Kibong Choi, name: Stanley Jungkyu Choi, name: Seokhee Hong, name: Junwon Hwang, name: Hyojin Jeon, name: Gerrard Jeongwon Jo, name: Hyunjik Jo, name: Jiyeon Jung, name: Yountae Jung, name: Hyosang Kim, name: Joonkee Kim, name: Seonghwan Kim, name: Soyeon Kim, name: Sunkyoung Kim, name: Yireun Kim, name: Yongil Kim, name: Youchul Kim, name: Edward Hwayoung Lee, name: Haeju Lee, name: Honglak Lee, name: Jinsik Lee, name: Kyungmin Lee, name: Woohyung Lim, name: Sangha Park, name: Sooyoun Park, name: Yongmin Park, name: Sihoon Yang, name: Heuiyeen Yeen, name: Hyeongu Yun
Published: 2024-12-06 08:53:46

This technical report introduces the EXAONE 3.5 instruction-tuned language
models, developed and released by LG AI Research. The EXAONE 3.5 language
models are offered in three configurations: 32B, 7.8B, and 2.4B. These models
feature several standout capabilities: 1) exceptional instruction following
capabilities in real-world scenarios, achieving the highest scores across seven
benchmarks, 2) outstanding long-context comprehension, attaining the top
performance in four benchmarks, and 3) competitive results compared to
state-of-the-art open models of similar sizes across nine general benchmarks.
The EXAONE 3.5 language models are open to anyone for research purposes and can
be downloaded from https://huggingface.co/LGAI-EXAONE. For commercial use,
please reach out to the official contact point of LG AI Research:
contact_us@lgresearch.ai.
SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step
  Diffusion
Authors: name: Trong-Tung Nguyen, name: Quang Nguyen, name: Khoi Nguyen, name: Anh Tran, name: Cuong Pham
Published: 2024-12-05 16:23:11

Recent advances in text-guided image editing enable users to perform image
edits through simple text inputs, leveraging the extensive priors of multi-step
diffusion-based text-to-image models. However, these methods often fall short
of the speed demands required for real-world and on-device applications due to
the costly multi-step inversion and sampling process involved. In response to
this, we introduce SwiftEdit, a simple yet highly efficient editing tool that
achieve instant text-guided image editing (in 0.23s). The advancement of
SwiftEdit lies in its two novel contributions: a one-step inversion framework
that enables one-step image reconstruction via inversion and a mask-guided
editing technique with our proposed attention rescaling mechanism to perform
localized image editing. Extensive experiments are provided to demonstrate the
effectiveness and efficiency of SwiftEdit. In particular, SwiftEdit enables
instant text-guided image editing, which is extremely faster than previous
multi-step methods (at least 50 times faster) while maintain a competitive
performance in editing results. Our project page is at:
https://swift-edit.github.io/
Moto: Latent Motion Token as the Bridging Language for Robot
  Manipulation
Authors: name: Yi Chen, name: Yuying Ge, name: Yizhuo Li, name: Yixiao Ge, name: Mingyu Ding, name: Ying Shan, name: Xihui Liu
Published: 2024-12-05 18:57:04

Recent developments in Large Language Models pre-trained on extensive corpora
have shown significant success in various natural language processing tasks
with minimal fine-tuning. This success offers new promise for robotics, which
has long been constrained by the high cost of action-labeled data. We ask:
given the abundant video data containing interaction-related knowledge
available as a rich "corpus", can a similar generative pre-training approach be
effectively applied to enhance robot learning? The key challenge is to identify
an effective representation for autoregressive pre-training that benefits robot
manipulation tasks. Inspired by the way humans learn new skills through
observing dynamic environments, we propose that effective robotic learning
should emphasize motion-related knowledge, which is closely tied to low-level
actions and is hardware-agnostic, facilitating the transfer of learned motions
to actual robot actions. To this end, we introduce Moto, which converts video
content into latent Motion Token sequences by a Latent Motion Tokenizer,
learning a bridging "language" of motion from videos in an unsupervised manner.
We pre-train Moto-GPT through motion token autoregression, enabling it to
capture diverse visual motion knowledge. After pre-training, Moto-GPT
demonstrates the promising ability to produce semantically interpretable motion
tokens, predict plausible motion trajectories, and assess trajectory
rationality through output likelihood. To transfer learned motion priors to
real robot actions, we implement a co-fine-tuning strategy that seamlessly
bridges latent motion token prediction and real robot control. Extensive
experiments show that the fine-tuned Moto-GPT exhibits superior robustness and
efficiency on robot manipulation benchmarks, underscoring its effectiveness in
transferring knowledge from video data to downstream visual manipulation tasks.
GenMAC: Compositional Text-to-Video Generation with Multi-Agent
  Collaboration
Authors: name: Kaiyi Huang, name: Yukun Huang, name: Xuefei Ning, name: Zinan Lin, name: Yu Wang, name: Xihui Liu
Published: 2024-12-05 18:56:05

Text-to-video generation models have shown significant progress in the recent
years. However, they still struggle with generating complex dynamic scenes
based on compositional text prompts, such as attribute binding for multiple
objects, temporal dynamics associated with different objects, and interactions
between objects. Our key motivation is that complex tasks can be decomposed
into simpler ones, each handled by a role-specialized MLLM agent. Multiple
agents can collaborate together to achieve collective intelligence for complex
goals. We propose GenMAC, an iterative, multi-agent framework that enables
compositional text-to-video generation. The collaborative workflow includes
three stages: Design, Generation, and Redesign, with an iterative loop between
the Generation and Redesign stages to progressively verify and refine the
generated videos. The Redesign stage is the most challenging stage that aims to
verify the generated videos, suggest corrections, and redesign the text
prompts, frame-wise layouts, and guidance scales for the next iteration of
generation. To avoid hallucination of a single MLLM agent, we decompose this
stage to four sequentially-executed MLLM-based agents: verification agent,
suggestion agent, correction agent, and output structuring agent. Furthermore,
to tackle diverse scenarios of compositional text-to-video generation, we
design a self-routing mechanism to adaptively select the proper correction
agent from a collection of correction agents each specialized for one scenario.
Extensive experiments demonstrate the effectiveness of GenMAC, achieving
state-of-the art performance in compositional text-to-video generation.
Establishing Task Scaling Laws via Compute-Efficient Model Ladders
Authors: name: Akshita Bhagia, name: Jiacheng Liu, name: Alexander Wettig, name: David Heineman, name: Oyvind Tafjord, name: Ananya Harsh Jha, name: Luca Soldaini, name: Noah A. Smith, name: Dirk Groeneveld, name: Pang Wei Koh, name: Jesse Dodge, name: Hannaneh Hajishirzi
Published: 2024-12-05 18:21:49

We develop task scaling laws and model ladders to predict the individual task
performance of pretrained language models (LMs) in the overtrained setting.
Standard power laws for language modeling loss cannot accurately model task
performance. Therefore, we leverage a two-step prediction approach: first use
model and data size to predict a task-specific loss, and then use this task
loss to predict task performance. We train a set of small-scale "ladder"
models, collect data points to fit the parameterized functions of the two
prediction steps, and make predictions for two target models: a 7B model
trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder
models only costs 1% of the compute used for the target models. On four
multiple-choice tasks written in ranked classification format, we can predict
the accuracy of both target models within 2 points of absolute error. We have
higher prediction error on four other tasks (average absolute error 6.9) and
find that these are often tasks with higher variance in task metrics. We also
find that using less compute to train fewer ladder models tends to deteriorate
predictions. Finally, we empirically show that our design choices and the
two-step approach lead to superior performance in establishing scaling laws.
NVILA: Efficient Frontier Visual Language Models
Authors: name: Zhijian Liu, name: Ligeng Zhu, name: Baifeng Shi, name: Zhuoyang Zhang, name: Yuming Lou, name: Shang Yang, name: Haocheng Xi, name: Shiyi Cao, name: Yuxian Gu, name: Dacheng Li, name: Xiuyu Li, name: Yunhao Fang, name: Yukang Chen, name: Cheng-Yu Hsieh, name: De-An Huang, name: An-Chieh Cheng, name: Vishwesh Nath, name: Jinyi Hu, name: Sifei Liu, name: Ranjay Krishna, name: Daguang Xu, name: Xiaolong Wang, name: Pavlo Molchanov, name: Jan Kautz, name: Hongxu Yin, name: Song Han, name: Yao Lu
Published: 2024-12-05 18:59:55

Visual language models (VLMs) have made significant advances in accuracy in
recent years. However, their efficiency has received much less attention. This
paper introduces NVILA, a family of open VLMs designed to optimize both
efficiency and accuracy. Building on top of VILA, we improve its model
architecture by first scaling up the spatial and temporal resolutions, and then
compressing visual tokens. This "scale-then-compress" approach enables NVILA to
efficiently process high-resolution images and long videos. We also conduct a
systematic investigation to enhance the efficiency of NVILA throughout its
entire lifecycle, from training and fine-tuning to deployment. NVILA matches or
surpasses the accuracy of many leading open and proprietary VLMs across a wide
range of image and video benchmarks. At the same time, it reduces training
costs by 4.5X, fine-tuning memory usage by 3.4X, pre-filling latency by
1.6-2.2X, and decoding latency by 1.2-2.8X. We will soon make our code and
models available to facilitate reproducibility.
4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion
Authors: name: Chaoyang Wang, name: Peiye Zhuang, name: Tuan Duc Ngo, name: Willi Menapace, name: Aliaksandr Siarohin, name: Michael Vasilkovsky, name: Ivan Skorokhodov, name: Sergey Tulyakov, name: Peter Wonka, name: Hsin-Ying Lee
Published: 2024-12-05 18:59:41

We propose 4Real-Video, a novel framework for generating 4D videos, organized
as a grid of video frames with both time and viewpoint axes. In this grid, each
row contains frames sharing the same timestep, while each column contains
frames from the same viewpoint. We propose a novel two-stream architecture. One
stream performs viewpoint updates on columns, and the other stream performs
temporal updates on rows. After each diffusion transformer layer, a
synchronization layer exchanges information between the two token streams. We
propose two implementations of the synchronization layer, using either hard or
soft synchronization. This feedforward architecture improves upon previous work
in three ways: higher inference speed, enhanced visual quality (measured by
FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency
(measured by VideoScore and Dust3R-Confidence).
p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay
Authors: name: Jun Zhang, name: Desen Meng, name: Ji Qi, name: Zhenpeng Huang, name: Tao Wu, name: Limin Wang
Published: 2024-12-05 18:58:03

Despite the remarkable performance of multimodal large language models
(MLLMs) across diverse tasks, the substantial training and inference costs
impede their advancement. The majority of computation stems from the
overwhelming volume of vision tokens processed by the transformer decoder. In
this paper, we propose to build efficient MLLMs by leveraging the
Mixture-of-Depths (MoD) mechanism, where each transformer decoder layer selects
essential vision tokens to process while skipping redundant ones. However,
integrating MoD into MLLMs is non-trivial. To address the challenges of
training and inference stability as well as limited training data, we adapt the
MoD module with two novel designs: tanh-gated weight normalization (TanhNorm)
and symmetric token reweighting (STRing). Moreover, we observe that vision
tokens exhibit higher redundancy in deeper layer and thus design a progressive
ratio decay (PRD) strategy, which gradually reduces the token retention ratio
layer by layer, employing a shifted cosine schedule. This crucial design fully
unleashes the potential of MoD, significantly boosting the efficiency and
performance of our models. To validate the effectiveness of our approach, we
conduct extensive experiments with two baseline models across 14 benchmarks.
Our model, p-MoD, matches or even surpasses the performance of the baseline
models, with only 55.6% TFLOPs and 53.8% KV cache storage during inference, and
77.7% GPU hours during training.