Orionnnn/hub-stats huggingface dataset:
Open-Sora Plan: Open-Source Large Video Generation Model
Authors: name: Bin Lin, name: Yunyang Ge, name: Xinhua Cheng, name: Zongjian Li, name: Bin Zhu, name: Shaodong Wang, name: Xianyi He, name: Yang Ye, name: Shenghai Yuan, name: Liuhan Chen, name: Tanghui Jia, name: Junwu Zhang, name: Zhenyu Tang, name: Yatian Pang, name: Bin She, name: Cen Yan, name: Zhiheng Hu, name: Xiaoyi Dong, name: Lin Chen, name: Zhang Pan, name: Xing Zhou, name: Shaoling Dong, name: Yonghong Tian, name: Li Yuan
Published: 2024-11-28 14:07:45

We introduce Open-Sora Plan, an open-source project that aims to contribute a
large generation model for generating desired high-resolution videos with long
durations based on various user inputs. Our project comprises multiple
components for the entire video generation process, including a Wavelet-Flow
Variational Autoencoder, a Joint Image-Video Skiparse Denoiser, and various
condition controllers. Moreover, many assistant strategies for efficient
training and inference are designed, and a multi-dimensional data curation
pipeline is proposed for obtaining desired high-quality data. Benefiting from
efficient thoughts, our Open-Sora Plan achieves impressive video generation
results in both qualitative and quantitative evaluations. We hope our careful
design and practical experience can inspire the video generation research
community. All our codes and model weights are publicly available at
https://github.com/PKU-YuanGroup/Open-Sora-Plan.
VisOnlyQA: Large Vision Language Models Still Struggle with Visual
  Perception of Geometric Information
Authors: name: Ryo Kamoi, name: Yusen Zhang, name: Sarkar Snigdha Sarathi Das, name: Ranran Haoran Zhang, name: Rui Zhang
Published: 2024-12-01 19:46:22

Errors in understanding visual information in images (i.e., visual perception
errors) remain a major source of mistakes in Large Vision Language Models
(LVLMs). While further analysis is essential, there is a deficiency in datasets
for evaluating the visual perception of LVLMs. In this work, we introduce
VisOnlyQA, a new dataset designed to directly evaluate the visual perception
capabilities of LVLMs on questions about geometric and numerical information in
scientific figures. Our dataset enables us to analyze the visual perception of
LVLMs for fine-grained visual information, independent of other capabilities
such as reasoning. The evaluation set of VisOnlyQA includes 1,200
multiple-choice questions in 12 tasks on four categories of figures. We also
provide synthetic training data consisting of 70k instances. Our experiments on
VisOnlyQA highlight the following findings: (i) 20 LVLMs we evaluate, including
GPT-4o and Gemini 1.5 Pro, work poorly on the visual perception tasks in
VisOnlyQA, while human performance is nearly perfect. (ii) Fine-tuning on
synthetic training data demonstrates the potential for enhancing the visual
perception of LVLMs, but observed improvements are limited to certain tasks and
specific models. (iii) Stronger language models improve the visual perception
of LVLMs. In summary, our experiments suggest that both training data and model
architectures should be improved to enhance the visual perception capabilities
of LVLMs. The datasets, code, and model responses are provided at
https://github.com/psunlpgroup/VisOnlyQA.
Long Video Diffusion Generation with Segmented Cross-Attention and
  Content-Rich Video Data Curation
Authors: name: Xin Yan, name: Yuxuan Cai, name: Qiuyue Wang, name: Yuan Zhou, name: Wenhao Huang, name: Huan Yang
Published: 2024-12-02 09:32:36

We introduce Presto, a novel video diffusion model designed to generate
15-second videos with long-range coherence and rich content. Extending video
generation methods to maintain scenario diversity over long durations presents
significant challenges. To address this, we propose a Segmented Cross-Attention
(SCA) strategy, which splits hidden states into segments along the temporal
dimension, allowing each segment to cross-attend to a corresponding
sub-caption. SCA requires no additional parameters, enabling seamless
incorporation into current DiT-based architectures. To facilitate high-quality
long video generation, we build the LongTake-HD dataset, consisting of 261k
content-rich videos with scenario coherence, annotated with an overall video
caption and five progressive sub-captions. Experiments show that our Presto
achieves 78.5% on the VBench Semantic Score and 100% on the Dynamic Degree,
outperforming existing state-of-the-art video generation methods. This
demonstrates that our proposed Presto significantly enhances content richness,
maintains long-range coherence, and captures intricate textual details. More
details are displayed on our project page: https://presto-video.github.io/.
TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any
  Point in Long Video
Authors: name: Jinyuan Qu, name: Hongyang Li, name: Shilong Liu, name: Tianhe Ren, name: Zhaoyang Zeng, name: Lei Zhang
Published: 2024-11-27 17:37:22

In this paper, we present TAPTRv3, which is built upon TAPTRv2 to improve its
point tracking robustness in long videos. TAPTRv2 is a simple DETR-like
framework that can accurately track any point in real-world videos without
requiring cost-volume. TAPTRv3 improves TAPTRv2 by addressing its shortage in
querying high quality features from long videos, where the target tracking
points normally undergo increasing variation over time. In TAPTRv3, we propose
to utilize both spatial and temporal context to bring better feature querying
along the spatial and temporal dimensions for more robust tracking in long
videos. For better spatial feature querying, we present Context-aware
Cross-Attention (CCA), which leverages surrounding spatial context to enhance
the quality of attention scores when querying image features. For better
temporal feature querying, we introduce Visibility-aware Long-Temporal
Attention (VLTA) to conduct temporal attention to all past frames while
considering their corresponding visibilities, which effectively addresses the
feature drifting problem in TAPTRv2 brought by its RNN-like long-temporal
modeling. TAPTRv3 surpasses TAPTRv2 by a large margin on most of the
challenging datasets and obtains state-of-the-art performance. Even when
compared with methods trained with large-scale extra internal data, TAPTRv3 is
still competitive.
WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent
  Video Diffusion Model
Authors: name: Zongjian Li, name: Bin Lin, name: Yang Ye, name: Liuhan Chen, name: Xinhua Cheng, name: Shenghai Yuan, name: Li Yuan
Published: 2024-11-26 14:23:53

Video Variational Autoencoder (VAE) encodes videos into a low-dimensional
latent space, becoming a key component of most Latent Video Diffusion Models
(LVDMs) to reduce model training costs. However, as the resolution and duration
of generated videos increase, the encoding cost of Video VAEs becomes a
limiting bottleneck in training LVDMs. Moreover, the block-wise inference
method adopted by most LVDMs can lead to discontinuities of latent space when
processing long-duration videos. The key to addressing the computational
bottleneck lies in decomposing videos into distinct components and efficiently
encoding the critical information. Wavelet transform can decompose videos into
multiple frequency-domain components and improve the efficiency significantly,
we thus propose Wavelet Flow VAE (WF-VAE), an autoencoder that leverages
multi-level wavelet transform to facilitate low-frequency energy flow into
latent representation. Furthermore, we introduce a method called Causal Cache,
which maintains the integrity of latent space during block-wise inference.
Compared to state-of-the-art video VAEs, WF-VAE demonstrates superior
performance in both PSNR and LPIPS metrics, achieving 2x higher throughput and
4x lower memory consumption while maintaining competitive reconstruction
quality. Our code and models are available at
https://github.com/PKU-YuanGroup/WF-VAE.
Yi-Lightning Technical Report
Authors: name: 01. AI, name: Alan Wake, name: Albert Wang, name: Bei Chen, name: C. X. Lv, name: Chao Li, name: Chengen Huang, name: Chenglin Cai, name: Chujie Zheng, name: Daniel Cooper, name: Ethan Dai, name: Fan Zhou, name: Feng Hu, name: Heng Ji, name: Howard Qiu, name: Jiangcheng Zhu, name: Jun Tian, name: Katherine Su, name: Lihuan Zhang, name: Liying Li, name: Ming Song, name: Mou Li, name: Peng Liu, name: Qichen Hu, name: Shawn Wang, name: Shijun Zhou, name: Shiyong Li, name: Tianhang Zhu, name: Wen Xie, name: Xiang He, name: Xiaobo Chen, name: Xiaohui Hu, name: Xiaoyi Ren, name: Xinyao Niu, name: Yanpeng Li, name: Yongke Zhao, name: Yongzhen Luo, name: Yuchi Xu, name: Yuxuan Sha, name: Zhaodong Yan, name: Zhiyuan Liu, name: Zirui Zhang
Published: 2024-12-02 08:22:56

This technical report presents Yi-Lightning, our latest flagship large
language model (LLM). It achieves exceptional performance, ranking 6th overall
on Chatbot Arena, with particularly strong results (2nd to 4th place) in
specialized categories including Chinese, Math, Coding, and Hard Prompts.
Yi-Lightning leverages an enhanced Mixture-of-Experts (MoE) architecture,
featuring advanced expert segmentation and routing mechanisms coupled with
optimized KV-caching techniques. Our development process encompasses
comprehensive pre-training, supervised fine-tuning (SFT), and reinforcement
learning from human feedback (RLHF), where we devise deliberate strategies for
multi-stage training, synthetic data construction, and reward modeling.
Furthermore, we implement RAISE (Responsible AI Safety Engine), a
four-component framework to address safety issues across pre-training,
post-training, and serving phases. Empowered by our scalable super-computing
infrastructure, all these innovations substantially reduce training, deployment
and inference costs while maintaining high-performance standards. With further
evaluations on public academic benchmarks, Yi-Lightning demonstrates
competitive performance against top-tier LLMs, while we observe a notable
disparity between traditional, static benchmark results and real-world, dynamic
human preferences. This observation prompts a critical reassessment of
conventional benchmarks' utility in guiding the development of more intelligent
and powerful AI systems for practical applications. Yi-Lightning is now
available through our developer platform at https://platform.lingyiwanwu.com.
DeMo: Decoupled Momentum Optimization
Authors: name: Bowen Peng, name: Jeffrey Quesnelle, name: Diederik P. Kingma
Published: 2024-11-29 17:31:47

Training large neural networks typically requires sharing gradients between
accelerators through specialized high-speed interconnects. Drawing from the
signal processing principles of frequency decomposition and energy compaction,
we demonstrate that synchronizing full optimizer states and model parameters
during training is unnecessary. By decoupling momentum updates and allowing
controlled divergence in optimizer states across accelerators, we achieve
improved convergence compared to state-of-the-art optimizers. We introduce
{De}coupled {Mo}mentum (DeMo), a fused optimizer and data
parallel algorithm that reduces inter-accelerator communication requirements by
several orders of magnitude. This enables training of large neural networks
even with limited network bandwidth and heterogeneous hardware. Our method is
topology-agnostic and architecture-independent and supports scalable
clock-synchronous distributed training with negligible compute and memory
overhead. Empirical results show that models trained with DeMo match or exceed
the performance of equivalent models trained with AdamW, while eliminating the
need for high-speed interconnects when pre-training large scale foundation
models. An open source reference PyTorch implementation is published on GitHub
at https://github.com/bloc97/DeMo
MATATA: a weak-supervised MAthematical Tool-Assisted reasoning for
  Tabular Applications
Authors: name: Vishnou Vinayagame, name: Gregory Senay, name: Luis Martí
Published: 2024-11-28 05:12:17

Mathematical reasoning capabilities are increasing with tool-augmented
language agents, but methods often rely either on closed-source or large
models, external data, or extensive prompt engineering. This work introduces
MATATA, a novel cost-effective method to train LLM agents for tabular data
problems through reasoning, planning, and tool use. With a progressive
self-improvement paradigm and an iterative weak supervision, it empowers
3.8B/8B Small Language Models (SLMs), particularly suited for local hosting and
sensitive business contexts where data privacy is crucial. By employing a
flexible and reusable tools across different datasets, it achieves robust
performance with effective scalability across shared tasks. Experiments show
that MATATA reaches state-of-the-art performances on FinQA and TAT-QA among
reasoning frameworks based on open-source models. Moreover, MATATA models
compete with GPT-4 based frameworks on TabMWP, while being SLMs.
GRAPE: Generalizing Robot Policy via Preference Alignment
Authors: name: Zijian Zhang, name: Kaiyuan Zheng, name: Zhaorun Chen, name: Joel Jang, name: Yi Li, name: Chaoqi Wang, name: Mingyu Ding, name: Dieter Fox, name: Huaxiu Yao
Published: 2024-11-28 18:30:10

Despite the recent advancements of vision-language-action (VLA) models on a
variety of robotics tasks, they suffer from critical issues such as poor
generalizability to unseen tasks, due to their reliance on behavior cloning
exclusively from successful rollouts. Furthermore, they are typically
fine-tuned to replicate demonstrations collected by experts under different
settings, thus introducing distribution bias and limiting their adaptability to
diverse manipulation objectives, such as efficiency, safety, and task
completion. To bridge this gap, we introduce GRAPE: Generalizing Robot Policy
via Preference Alignment. Specifically, GRAPE aligns VLAs on a trajectory level
and implicitly models reward from both successful and failure trials to boost
generalizability to diverse tasks. Moreover, GRAPE breaks down complex
manipulation tasks to independent stages and automatically guides preference
modeling through customized spatiotemporal constraints with keypoints proposed
by a large vision-language model. Notably, these constraints are flexible and
can be customized to align the model with varying objectives, such as safety,
efficiency, or task success. We evaluate GRAPE across a diverse array of tasks
in both real-world and simulated environments. Experimental results demonstrate
that GRAPE enhances the performance of state-of-the-art VLA models, increasing
success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%,
respectively. Additionally, GRAPE can be aligned with various objectives, such
as safety and efficiency, reducing collision rates by 44.31% and rollout
step-length by 11.15%, respectively. All code, models, and data are available
at https://grape-vla.github.io/
Training Noise Token Pruning
Authors: name: Mingxing Rao, name: Bohan Jiang, name: Daniel Moyer
Published: 2024-11-27 07:04:00

In the present work we present Training Noise Token (TNT) Pruning for vision
transformers. Our method relaxes the discrete token dropping condition to
continuous additive noise, providing smooth optimization in training, while
retaining discrete dropping computational gains in deployment settings. We
provide theoretical connections to Rate-Distortion literature, and empirical
evaluations on the ImageNet dataset using ViT and DeiT architectures
demonstrating TNT's advantages over previous pruning methods.
SpotLight: Shadow-Guided Object Relighting via Diffusion
Authors: name: Frédéric Fortier-Chouinard, name: Zitian Zhang, name: Louis-Etienne Messier, name: Mathieu Garon, name: Anand Bhattad, name: Jean-François Lalonde
Published: 2024-11-27 16:06:08

Recent work has shown that diffusion models can be used as powerful neural
rendering engines that can be leveraged for inserting virtual objects into
images. Unlike typical physics-based renderers, however, neural rendering
engines are limited by the lack of manual control over the lighting setup,
which is often essential for improving or personalizing the desired image
outcome. In this paper, we show that precise lighting control can be achieved
for object relighting simply by specifying the desired shadows of the object.
Rather surprisingly, we show that injecting only the shadow of the object into
a pre-trained diffusion-based neural renderer enables it to accurately shade
the object according to the desired light position, while properly harmonizing
the object (and its shadow) within the target background image. Our method,
SpotLight, leverages existing neural rendering approaches and achieves
controllable relighting results with no additional training. Specifically, we
demonstrate its use with two neural renderers from the recent literature. We
show that SpotLight achieves superior object compositing results, both
quantitatively and perceptually, as confirmed by a user study, outperforming
existing diffusion-based models specifically designed for relighting.
Scaling Transformers for Low-Bitrate High-Quality Speech Coding
Authors: name: Julian D Parker, name: Anton Smirnov, name: Jordi Pons, name: CJ Carr, name: Zack Zukowski, name: Zach Evans, name: Xubo Liu
Published: 2024-11-29 16:58:02

The tokenization of speech with neural audio codec models is a vital part of
modern AI pipelines for the generation or understanding of speech, alone or in
a multimodal context. Traditionally such tokenization models have concentrated
on low parameter-count architectures using only components with strong
inductive biases. In this work we show that by scaling a transformer
architecture with large parameter count to this problem, and applying a
flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to
reach state-of-the-art speech quality at extremely low bit-rates of 400 or
700 bits-per-second. The trained models strongly out-perform existing
baselines in both objective and subjective tests.
Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling
Authors: name: Junha Hyung, name: Kinam Kim, name: Susung Hong, name: Min-Jung Kim, name: Jaegul Choo
Published: 2024-11-27 15:59:48

Diffusion models have emerged as a powerful tool for generating high-quality
images, videos, and 3D content. While sampling guidance techniques like CFG
improve quality, they reduce diversity and motion. Autoguidance mitigates these
issues but demands extra weak model training, limiting its practicality for
large-scale models. In this work, we introduce Spatiotemporal Skip Guidance
(STG), a simple training-free sampling guidance method for enhancing
transformer-based video diffusion models. STG employs an implicit weak model
via self-perturbation, avoiding the need for external models or additional
training. By selectively skipping spatiotemporal layers, STG produces an
aligned, degraded version of the original model to boost sample quality without
compromising diversity or dynamic degree. Our contributions include: (1)
introducing STG as an efficient, high-performing guidance technique for video
diffusion models, (2) eliminating the need for auxiliary models by simulating a
weak model through layer skipping, and (3) ensuring quality-enhanced guidance
without compromising sample diversity or dynamics unlike CFG. For additional
results, visit https://junhahyung.github.io/STGuidance.
Reverse Thinking Makes LLMs Stronger Reasoners
Authors: name: Justin Chih-Yao Chen, name: Zifeng Wang, name: Hamid Palangi, name: Rujun Han, name: Sayna Ebrahimi, name: Long Le, name: Vincent Perot, name: Swaroop Mishra, name: Mohit Bansal, name: Chen-Yu Lee, name: Tomas Pfister
Published: 2024-11-29 17:27:05

Reverse thinking plays a crucial role in human reasoning. Humans can reason
not only from a problem to a solution but also in reverse, i.e., start from the
solution and reason towards the problem. This often enhances overall reasoning
performance as it enables consistency checks between their forward and backward
thinking. To enable Large Language Models (LLMs) to perform reverse thinking,
we introduce Reverse-Enhanced Thinking (RevThink), a framework composed of data
augmentation and learning objectives. In RevThink, we augment the dataset by
collecting structured forward-backward reasoning from a teacher model,
consisting of: (1) the original question, (2) forward reasoning, (3) backward
question, and (4) backward reasoning. We then employ three objectives to train
a smaller student model in a multi-task learning fashion: (a) generate forward
reasoning from a question, (b) generate a backward question from a question,
and (c) generate backward reasoning from the backward question. Experiments
across 12 datasets covering commonsense, math, and logical reasoning show an
average 13.53% improvement over the student model's zero-shot performance and a
6.84% improvement over the strongest knowledge distillation baselines.
Moreover, our method demonstrates sample efficiency -- using only 10% of the
correct forward reasoning from the training data, it outperforms a standard
fine-tuning method trained on 10x more forward reasoning. RevThink also
exhibits strong generalization to out-of-distribution held-out datasets.
FAM Diffusion: Frequency and Attention Modulation for High-Resolution
  Image Generation with Stable Diffusion
Authors: name: Haosen Yang, name: Adrian Bulat, name: Isma Hadji, name: Hai X. Pham, name: Xiatian Zhu, name: Georgios Tzimiropoulos, name: Brais Martinez
Published: 2024-11-27 17:51:44

Diffusion models are proficient at generating high-quality images. They are
however effective only when operating at the resolution used during training.
Inference at a scaled resolution leads to repetitive patterns and structural
distortions. Retraining at higher resolutions quickly becomes prohibitive.
Thus, methods enabling pre-existing diffusion models to operate at flexible
test-time resolutions are highly desirable. Previous works suffer from frequent
artifacts and often introduce large latency overheads. We propose two simple
modules that combine to solve these issues. We introduce a Frequency Modulation
(FM) module that leverages the Fourier domain to improve the global structure
consistency, and an Attention Modulation (AM) module which improves the
consistency of local texture patterns, a problem largely ignored in prior
works. Our method, coined Fam diffusion, can seamlessly integrate into any
latent diffusion model and requires no additional training. Extensive
qualitative results highlight the effectiveness of our method in addressing
structural and local artifacts, while quantitative results show
state-of-the-art performance. Also, our method avoids redundant inference
tricks for improved consistency such as patch-based or progressive generation,
leading to negligible latency overheads.