Orionnnn/hub-stats huggingface dataset:
DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow
  Decoding
Authors: name: Jungbin Cho, name: Junwan Kim, name: Jisoo Kim, name: Minseo Kim, name: Mingu Kang, name: Sungeun Hong, name: Tae-Hyun Oh, name: Youngjae Yu
Published: 2024-11-29 07:54:56

Human motion, inherently continuous and dynamic, presents significant
challenges for generative models. Despite their dominance, discrete
quantization methods, such as VQ-VAEs, suffer from inherent limitations,
including restricted expressiveness and frame-wise noise artifacts. Continuous
approaches, while producing smoother and more natural motions, often falter due
to high-dimensional complexity and limited training data. To resolve this
"discord" between discrete and continuous representations, we introduce
DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decoding, a
novel method that decodes discrete motion tokens into continuous motion through
rectified flow. By employing an iterative refinement process in the continuous
space, DisCoRD captures fine-grained dynamics and ensures smoother and more
natural motions. Compatible with any discrete-based framework, our method
enhances naturalness without compromising faithfulness to the conditioning
signals. Extensive evaluations demonstrate that DisCoRD achieves
state-of-the-art performance, with FID of 0.032 on HumanML3D and 0.169 on
KIT-ML. These results solidify DisCoRD as a robust solution for bridging the
divide between discrete efficiency and continuous realism. Our project page is
available at: https://whwjdqls.github.io/discord.github.io/.
Beyond Examples: High-level Automated Reasoning Paradigm in In-Context
  Learning via MCTS
Authors: name: Jinyang Wu, name: Mingkuan Feng, name: Shuai Zhang, name: Feihu Che, name: Zengqi Wen, name: Jianhua Tao
Published: 2024-11-27 16:19:00

In-context Learning (ICL) enables large language models (LLMs) to tackle
downstream tasks through sophisticated prompting and high-quality
demonstrations. However, this traditional ICL paradigm shows limitations when
facing complex mathematical reasoning tasks, primarily due to its heavy
dependence on example quality and the necessity for human intervention in
challenging scenarios. To address these limitations, this paper presents
HiAR-ICL, a High-level Automated Reasoning paradigm
in ICL that shifts focus from specific examples to abstract thinking
patterns, extending the conventional concept of context in ICL. HiAR-ICL
introduces five atomic reasoning actions as fundamental components for
constructing chain-structured patterns. Using Monte Carlo Tree Search, we
explore reasoning paths and construct thought cards to guide subsequent
inference. We then develop a cognitive complexity framework that dynamically
matches problems with appropriate thought cards. Experimental results
demonstrate HiAR-ICL's effectiveness, achieving state-of-the-art accuracy
(79.6%) on the MATH benchmark with Qwen2.5-7B-Instruct, surpassing GPT-4o
(76.6%) and Claude 3.5 (71.1%).
Video Depth without Video Models
Authors: name: Bingxin Ke, name: Dominik Narnhofer, name: Shengyu Huang, name: Lei Ke, name: Torben Peters, name: Katerina Fragkiadaki, name: Anton Obukhov, name: Konrad Schindler
Published: 2024-11-28 14:50:14

Video depth estimation lifts monocular video clips to 3D by inferring dense
depth at every frame. Recent advances in single-image depth estimation, brought
about by the rise of large foundation models and the use of synthetic training
data, have fueled a renewed interest in video depth. However, naively applying
a single-image depth estimator to every frame of a video disregards temporal
continuity, which not only leads to flickering but may also break when camera
motion causes sudden changes in depth range. An obvious and principled solution
would be to build on top of video foundation models, but these come with their
own limitations; including expensive training and inference, imperfect 3D
consistency, and stitching routines for the fixed-length (short) outputs. We
take a step back and demonstrate how to turn a single-image latent diffusion
model (LDM) into a state-of-the-art video depth estimator. Our model, which we
call RollingDepth, has two main ingredients: (i) a multi-frame depth estimator
that is derived from a single-image LDM and maps very short video snippets
(typically frame triplets) to depth snippets. (ii) a robust, optimization-based
registration algorithm that optimally assembles depth snippets sampled at
various different frame rates back into a consistent video. RollingDepth is
able to efficiently handle long videos with hundreds of frames and delivers
more accurate depth videos than both dedicated video depth estimators and
high-performing single-frame models. Project page: rollingdepth.github.io.
LLM Teacher-Student Framework for Text Classification With No Manually
  Annotated Data: A Case Study in IPTC News Topic Classification
Authors: name: Taja Kuzman, name: Nikola Ljubešić
Published: 2024-11-29 11:42:58

With the ever-increasing number of news stories available online, classifying
them by topic, regardless of the language they are written in, has become
crucial for enhancing readers' access to relevant content. To address this
challenge, we propose a teacher-student framework based on large language
models (LLMs) for developing multilingual news classification models of
reasonable size with no need for manual data annotation. The framework employs
a Generative Pretrained Transformer (GPT) model as the teacher model to develop
an IPTC Media Topic training dataset through automatic annotation of news
articles in Slovenian, Croatian, Greek, and Catalan. The teacher model exhibits
a high zero-shot performance on all four languages. Its agreement with human
annotators is comparable to that between the human annotators themselves. To
mitigate the computational limitations associated with the requirement of
processing millions of texts daily, smaller BERT-like student models are
fine-tuned on the GPT-annotated dataset. These student models achieve high
performance comparable to the teacher model. Furthermore, we explore the impact
of the training data size on the performance of the student models and
investigate their monolingual, multilingual and zero-shot cross-lingual
capabilities. The findings indicate that student models can achieve high
performance with a relatively small number of training instances, and
demonstrate strong zero-shot cross-lingual abilities. Finally, we publish the
best-performing news topic classifier, enabling multilingual classification
with the top-level categories of the IPTC Media Topic schema.
AlphaTablets: A Generic Plane Representation for 3D Planar
  Reconstruction from Monocular Videos
Authors: name: Yuze He, name: Wang Zhao, name: Shaohui Liu, name: Yubin Hu, name: Yushi Bai, name: Yu-Hui Wen, name: Yong-Jin Liu
Published: 2024-11-29 18:59:52

We introduce AlphaTablets, a novel and generic representation of 3D planes
that features continuous 3D surface and precise boundary delineation. By
representing 3D planes as rectangles with alpha channels, AlphaTablets combine
the advantages of current 2D and 3D plane representations, enabling accurate,
consistent and flexible modeling of 3D planes. We derive differentiable
rasterization on top of AlphaTablets to efficiently render 3D planes into
images, and propose a novel bottom-up pipeline for 3D planar reconstruction
from monocular videos. Starting with 2D superpixels and geometric cues from
pre-trained models, we initialize 3D planes as AlphaTablets and optimize them
via differentiable rendering. An effective merging scheme is introduced to
facilitate the growth and refinement of AlphaTablets. Through iterative
optimization and merging, we reconstruct complete and accurate 3D planes with
solid surfaces and clear boundaries. Extensive experiments on the ScanNet
dataset demonstrate state-of-the-art performance in 3D planar reconstruction,
underscoring the great potential of AlphaTablets as a generic 3D plane
representation for various applications. Project page is available at:
https://hyzcluster.github.io/alphatablets
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs
Authors: name: Akhiad Bercovich, name: Tomer Ronen, name: Talor Abramovich, name: Nir Ailon, name: Nave Assaf, name: Mohammad Dabbah, name: Ido Galil, name: Amnon Geifman, name: Yonatan Geifman, name: Izhak Golan, name: Netanel Haber, name: Ehud Karpas, name: Itay Levy, name: Shahar Mor, name: Zach Moshe, name: Najeeb Nabwani, name: Omri Puny, name: Ran Rubin, name: Itamar Schen, name: Ido Shahaf, name: Oren Tropp, name: Omer Ullman Argov, name: Ran Zilberstein, name: Ran El-Yaniv
Published: 2024-11-28 13:45:42

Large language models (LLMs) have demonstrated remarkable capabilities, but
their adoption is limited by high computational costs during inference. While
increasing parameter counts enhances accuracy, it also widens the gap between
state-of-the-art capabilities and practical deployability. We present Puzzle, a
framework to accelerate LLM inference on specific hardware while preserving
their capabilities. Through an innovative application of neural architecture
search (NAS) at an unprecedented scale, Puzzle systematically optimizes models
with tens of billions of parameters under hardware constraints. Our approach
utilizes blockwise local knowledge distillation (BLD) for parallel architecture
exploration and employs mixed-integer programming for precise constraint
optimization.
  We demonstrate the real-world impact of our framework through
Llama-3.1-Nemotron-51B-Instruct (Nemotron-51B), a publicly available model
derived from Llama-3.1-70B-Instruct. Nemotron-51B achieves a 2.17x inference
throughput speedup, fitting on a single NVIDIA H100 GPU while preserving 98.4%
of the original model's capabilities. Nemotron-51B currently stands as the most
accurate language model capable of inference on a single GPU with large batch
sizes. Remarkably, this transformation required just 45B training tokens,
compared to over 15T tokens used for the 70B model it was derived from. This
establishes a new paradigm where powerful models can be optimized for efficient
deployment with only negligible compromise of their capabilities, demonstrating
that inference performance, not parameter count alone, should guide model
selection. With the release of Nemotron-51B and the presentation of the Puzzle
framework, we provide practitioners immediate access to state-of-the-art
language modeling capabilities at significantly reduced computational costs.
Look Every Frame All at Once: Video-Ma^2mba for Efficient Long-form
  Video Understanding with Multi-Axis Gradient Checkpointing
Authors: name: Hosu Lee, name: Junho Kim, name: Hyunjun Kim, name: Yong Man Ro
Published: 2024-11-29 04:12:13

With the growing scale and complexity of video data, efficiently processing
long video sequences poses significant challenges due to the quadratic increase
in memory and computational demands associated with existing transformer-based
Large Multi-modal Models (LMMs). To address these issues, we introduce
Video-Ma^2mba, a novel architecture that incorporates State Space Models
(SSMs) within the Mamba-2 framework, replacing the attention mechanisms. This
allows the LMMs to scale linearly in terms of time and memory requirements,
making it feasible to handle long-duration video content. Furthermore, we
enhance the memory efficiency introducing the Multi-Axis Gradient Checkpointing
(MA-GC) method, which strategically manages memory by retaining only essential
activations across multiple computational axes. Our approach significantly
reduces the memory footprint compared to standard gradient checkpointing.
Empirical analyses show that Video-Ma^2mba can process extensive video
sequences-equivalent to millions of tokens or over two hours of continuous
sequences at 1 FPS-on a single GPU. By maintaining a detailed capture of
temporal dynamics, our model improves the accuracy and relevance of responses
in long video understanding tasks, demonstrating substantial advantages over
existing frameworks.
Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model
Authors: name: Feng Liu, name: Shiwei Zhang, name: Xiaofeng Wang, name: Yujie Wei, name: Haonan Qiu, name: Yuzhong Zhao, name: Yingya Zhang, name: Qixiang Ye, name: Fang Wan
Published: 2024-11-28 12:50:05

As a fundamental backbone for video generation, diffusion models are
challenged by low inference speed due to the sequential nature of denoising.
Previous methods speed up the models by caching and reusing model outputs at
uniformly selected timesteps. However, such a strategy neglects the fact that
differences among model outputs are not uniform across timesteps, which hinders
selecting the appropriate model outputs to cache, leading to a poor balance
between inference efficiency and visual quality. In this study, we introduce
Timestep Embedding Aware Cache (TeaCache), a training-free caching approach
that estimates and leverages the fluctuating differences among model outputs
across timesteps. Rather than directly using the time-consuming model outputs,
TeaCache focuses on model inputs, which have a strong correlation with the
modeloutputs while incurring negligible computational cost. TeaCache first
modulates the noisy inputs using the timestep embeddings to ensure their
differences better approximating those of model outputs. TeaCache then
introduces a rescaling strategy to refine the estimated differences and
utilizes them to indicate output caching. Experiments show that TeaCache
achieves up to 4.41x acceleration over Open-Sora-Plan with negligible (-0.07%
Vbench score) degradation of visual quality.
AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion
  Transformers
Authors: name: Sherwin Bahmani, name: Ivan Skorokhodov, name: Guocheng Qian, name: Aliaksandr Siarohin, name: Willi Menapace, name: Andrea Tagliasacchi, name: David B. Lindell, name: Sergey Tulyakov
Published: 2024-11-27 18:49:13

Numerous works have recently integrated 3D camera control into foundational
text-to-video models, but the resulting camera control is often imprecise, and
video generation quality suffers. In this work, we analyze camera motion from a
first principles perspective, uncovering insights that enable precise 3D camera
manipulation without compromising synthesis quality. First, we determine that
motion induced by camera movements in videos is low-frequency in nature. This
motivates us to adjust train and test pose conditioning schedules, accelerating
training convergence while improving visual and motion quality. Then, by
probing the representations of an unconditional video diffusion transformer, we
observe that they implicitly perform camera pose estimation under the hood, and
only a sub-portion of their layers contain the camera information. This
suggested us to limit the injection of camera conditioning to a subset of the
architecture to prevent interference with other video features, leading to 4x
reduction of training parameters, improved training speed and 10% higher visual
quality. Finally, we complement the typical dataset for camera control learning
with a curated dataset of 20K diverse dynamic videos with stationary cameras.
This helps the model disambiguate the difference between camera and scene
motion, and improves the dynamics of generated pose-conditioned videos. We
compound these findings to design the Advanced 3D Camera Control (AC3D)
architecture, the new state-of-the-art model for generative video modeling with
camera control.
On Domain-Specific Post-Training for Multimodal Large Language Models
Authors: name: Daixuan Cheng, name: Shaohan Huang, name: Ziyu Zhu, name: Xintong Zhang, name: Wayne Xin Zhao, name: Zhongzhi Luan, name: Bo Dai, name: Zhenliang Zhang
Published: 2024-11-29 18:42:28

Recent years have witnessed the rapid development of general multimodal large
language models (MLLMs). However, adapting general MLLMs to specific domains,
such as scientific fields and industrial applications, remains less explored.
This paper systematically investigates domain adaptation of MLLMs through
post-training, focusing on data synthesis, training pipelines, and task
evaluation. (1) Data Synthesis: Using open-source models, we develop a visual
instruction synthesizer that effectively generates diverse visual instruction
tasks from domain-specific image-caption pairs. Our synthetic tasks surpass
those generated by manual rules, GPT-4, and GPT-4V in enhancing the
domain-specific performance of MLLMs. (2) Training Pipeline: While the
two-stage training--initially on image-caption pairs followed by visual
instruction tasks--is commonly adopted for developing general MLLMs, we apply a
single-stage training pipeline to enhance task diversity for domain-specific
post-training. (3) Task Evaluation: We conduct experiments in two domains,
biomedicine and food, by post-training MLLMs of different sources and scales
(e.g., Qwen2-VL-2B, LLaVA-v1.6-8B, Llama-3.2-11B), and then evaluating MLLM
performance on various domain-specific tasks. To support further research in
MLLM domain adaptation, we will open-source our implementations.
Trajectory Attention for Fine-grained Video Motion Control
Authors: name: Zeqi Xiao, name: Wenqi Ouyang, name: Yifan Zhou, name: Shuai Yang, name: Lei Yang, name: Jianlou Si, name: Xingang Pan
Published: 2024-11-28 18:59:51

Recent advancements in video generation have been greatly driven by video
diffusion models, with camera motion control emerging as a crucial challenge in
creating view-customized visual content. This paper introduces trajectory
attention, a novel approach that performs attention along available pixel
trajectories for fine-grained camera motion control. Unlike existing methods
that often yield imprecise outputs or neglect temporal correlations, our
approach possesses a stronger inductive bias that seamlessly injects trajectory
information into the video generation process. Importantly, our approach models
trajectory attention as an auxiliary branch alongside traditional temporal
attention. This design enables the original temporal attention and the
trajectory attention to work in synergy, ensuring both precise motion control
and new content generation capability, which is critical when the trajectory is
only partially available. Experiments on camera motion control for images and
videos demonstrate significant improvements in precision and long-range
consistency while maintaining high-quality generation. Furthermore, we show
that our approach can be extended to other video motion control tasks, such as
first-frame-guided video editing, where it excels in maintaining content
consistency over large spatial and temporal ranges.
SelfSplat: Pose-Free and 3D Prior-Free Generalizable 3D Gaussian
  Splatting
Authors: name: Gyeongjin Kang, name: Jisang Yoo, name: Jihyeon Park, name: Seungtae Nam, name: Hyeonsoo Im, name: Sangheon Shin, name: Sangpil Kim, name: Eunbyung Park
Published: 2024-11-26 08:01:50

We propose SelfSplat, a novel 3D Gaussian Splatting model designed to perform
pose-free and 3D prior-free generalizable 3D reconstruction from unposed
multi-view images. These settings are inherently ill-posed due to the lack of
ground-truth data, learned geometric information, and the need to achieve
accurate 3D reconstruction without finetuning, making it difficult for
conventional methods to achieve high-quality results. Our model addresses these
challenges by effectively integrating explicit 3D representations with
self-supervised depth and pose estimation techniques, resulting in reciprocal
improvements in both pose accuracy and 3D reconstruction quality. Furthermore,
we incorporate a matching-aware pose estimation network and a depth refinement
module to enhance geometry consistency across views, ensuring more accurate and
stable 3D reconstructions. To present the performance of our method, we
evaluated it on large-scale real-world datasets, including RealEstate10K, ACID,
and DL3DV. SelfSplat achieves superior results over previous state-of-the-art
methods in both appearance and geometry quality, also demonstrates strong
cross-dataset generalization capabilities. Extensive ablation studies and
analysis also validate the effectiveness of our proposed methods. Code and
pretrained models are available at https://gynjn.github.io/selfsplat/
AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering
  Benchmark Dataset
Authors: name: Tobi Olatunji, name: Charles Nimo, name: Abraham Owodunni, name: Tassallah Abdullahi, name: Emmanuel Ayodele, name: Mardhiyah Sanni, name: Chinemelu Aka, name: Folafunmi Omofoye, name: Foutse Yuehgoh, name: Timothy Faniran, name: Bonaventure F. P. Dossou, name: Moshood Yekini, name: Jonas Kemp, name: Katherine Heller, name: Jude Chidubem Omeke, name: Chidi Asuzu MD, name: Naome A. Etori, name: Aimérou Ndiaye, name: Ifeoma Okoh, name: Evans Doe Ocansey, name: Wendy Kinara, name: Michael Best, name: Irfan Essa, name: Stephen Edward Moore, name: Chris Fourie, name: Mercy Nyamewaa Asiedu
Published: 2024-11-23 19:43:02

Recent advancements in large language model(LLM) performance on medical
multiple choice question (MCQ) benchmarks have stimulated interest from
healthcare providers and patients globally. Particularly in low-and
middle-income countries (LMICs) facing acute physician shortages and lack of
specialists, LLMs offer a potentially scalable pathway to enhance healthcare
access and reduce costs. However, their effectiveness in the Global South,
especially across the African continent, remains to be established. In this
work, we introduce AfriMed-QA, the first large scale Pan-African English
multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open
and closed-ended) sourced from over 60 medical schools across 16 countries,
covering 32 medical specialties. We further evaluate 30 LLMs across multiple
axes including correctness and demographic bias. Our findings show significant
performance variation across specialties and geographies, MCQ performance
clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general
models and smaller edge-friendly LLMs struggle to achieve a passing score.
Interestingly, human evaluations show a consistent consumer preference for LLM
answers and explanations when compared with clinician answers.
Free^2Guide: Gradient-Free Path Integral Control for Enhancing
  Text-to-Video Generation with Large Vision-Language Models
Authors: name: Jaemin Kim, name: Bryan S Kim, name: Jong Chul Ye
Published: 2024-11-26 02:14:47

Diffusion models have achieved impressive results in generative tasks like
text-to-image (T2I) and text-to-video (T2V) synthesis. However, achieving
accurate text alignment in T2V generation remains challenging due to the
complex temporal dependency across frames. Existing reinforcement learning
(RL)-based approaches to enhance text alignment often require differentiable
reward functions or are constrained to limited prompts, hindering their
scalability and applicability. In this paper, we propose Free^2Guide, a novel
gradient-free framework for aligning generated videos with text prompts without
requiring additional model training. Leveraging principles from path integral
control, Free^2Guide approximates guidance for diffusion models using
non-differentiable reward functions, thereby enabling the integration of
powerful black-box Large Vision-Language Models (LVLMs) as reward model.
Additionally, our framework supports the flexible ensembling of multiple reward
models, including large-scale image-based models, to synergistically enhance
alignment without incurring substantial computational overhead. We demonstrate
that Free^2Guide significantly improves text alignment across various
dimensions and enhances the overall quality of generated videos.