Orionnnn/hub-stats huggingface dataset:
VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained
  Video Reasoning via Core Frame Selection
Authors: name: Songhao Han, name: Wei Huang, name: Hairong Shi, name: Le Zhuo, name: Xiu Su, name: Shifeng Zhang, name: Xu Zhou, name: Xiaojuan Qi, name: Yue Liao, name: Si Liu
Published: 2024-11-22 08:33:36

The advancement of Large Vision Language Models (LVLMs) has significantly
improved multimodal understanding, yet challenges remain in video reasoning
tasks due to the scarcity of high-quality, large-scale datasets. Existing video
question-answering (VideoQA) datasets often rely on costly manual annotations
with insufficient granularity or automatic construction methods with redundant
frame-by-frame analysis, limiting their scalability and effectiveness for
complex reasoning. To address these challenges, we introduce VideoEspresso, a
novel dataset that features VideoQA pairs preserving essential spatial details
and temporal coherence, along with multimodal annotations of intermediate
reasoning steps. Our construction pipeline employs a semantic-aware method to
reduce redundancy, followed by generating QA pairs using GPT-4o. We further
develop video Chain-of-Thought (CoT) annotations to enrich reasoning processes,
guiding GPT-4o in extracting logical relationships from QA pairs and video
content. To exploit the potential of high-quality VideoQA pairs, we propose a
Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a
two-stage instruction fine-tuned reasoning LVLM. This framework adaptively
selects core frames and performs CoT reasoning using multimodal evidence.
Evaluated on our proposed benchmark with 14 tasks against 9 popular LVLMs, our
method outperforms existing baselines on most tasks, demonstrating superior
video reasoning capabilities. Our code and dataset will be released at:
https://github.com/hshjerry/VideoEspresso
A Flexible Large Language Models Guardrail Development Methodology
  Applied to Off-Topic Prompt Detection
Authors: name: Gabriel Chua, name: Shing Yee Chan, name: Shaun Khoo
Published: 2024-11-20 00:31:23

Large Language Models are prone to off-topic misuse, where users may prompt
these models to perform tasks beyond their intended scope. Current guardrails,
which often rely on curated examples or custom classifiers, suffer from high
false-positive rates, limited adaptability, and the impracticality of requiring
real-world data that is not available in pre-production. In this paper, we
introduce a flexible, data-free guardrail development methodology that
addresses these challenges. By thoroughly defining the problem space
qualitatively and passing this to an LLM to generate diverse prompts, we
construct a synthetic dataset to benchmark and train off-topic guardrails that
outperform heuristic approaches. Additionally, by framing the task as
classifying whether the user prompt is relevant with respect to the system
prompt, our guardrails effectively generalize to other misuse categories,
including jailbreak and harmful prompts. Lastly, we further contribute to the
field by open-sourcing both the synthetic dataset and the off-topic guardrail
models, providing valuable resources for developing guardrails in
pre-production environments and supporting future research and development in
LLM safety.
Efficient Long Video Tokenization via Coordinated-based Patch
  Reconstruction
Authors: name: Huiwon Jang, name: Sihyun Yu, name: Jinwoo Shin, name: Pieter Abbeel, name: Younggyo Seo
Published: 2024-11-22 06:50:44

Efficient tokenization of videos remains a challenge in training vision
models that can process long videos. One promising direction is to develop a
tokenizer that can encode long video clips, as it would enable the tokenizer to
leverage the temporal coherence of videos better for tokenization. However,
training existing tokenizers on long videos often incurs a huge training cost
as they are trained to reconstruct all the frames at once. In this paper, we
introduce CoordTok, a video tokenizer that learns a mapping from
coordinate-based representations to the corresponding patches of input videos,
inspired by recent advances in 3D generative models. In particular, CoordTok
encodes a video into factorized triplane representations and reconstructs
patches that correspond to randomly sampled (x,y,t) coordinates. This allows
for training large tokenizer models directly on long videos without requiring
excessive training resources. Our experiments show that CoordTok can
drastically reduce the number of tokens for encoding long video clips. For
instance, CoordTok can encode a 128-frame video with 128times128 resolution
into 1280 tokens, while baselines need 6144 or 8192 tokens to achieve similar
reconstruction quality. We further show that this efficient video tokenization
enables memory-efficient training of a diffusion transformer that can generate
128 frames at once.
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training
Authors: name: Nathan Lambert, name: Jacob Morrison, name: Valentina Pyatkin, name: Shengyi Huang, name: Hamish Ivison, name: Faeze Brahman, name: Lester James V. Miranda, name: Alisa Liu, name: Nouha Dziri, name: Shane Lyu, name: Yuling Gu, name: Saumya Malik, name: Victoria Graf, name: Jena D. Hwang, name: Jiangjiang Yang, name: Ronan Le Bras, name: Oyvind Tafjord, name: Chris Wilhelm, name: Luca Soldaini, name: Noah A. Smith, name: Yizhong Wang, name: Pradeep Dasigi, name: Hannaneh Hajishirzi
Published: 2024-11-22 18:44:04

Language model post-training is applied to refine behaviors and unlock new
skills across a wide range of recent language models, but open recipes for
applying these techniques lag behind proprietary ones. The underlying training
data and recipes for post-training are simultaneously the most important pieces
of the puzzle and the portion with the least transparency. To bridge this gap,
we introduce T\"ULU 3, a family of fully-open state-of-the-art post-trained
models, alongside its data, code, and training recipes, serving as a
comprehensive guide for modern post-training techniques. T\"ULU 3, which builds
on Llama 3.1 base models, achieves results surpassing the instruct versions of
Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and
Claude 3.5-Haiku. The training algorithms for our models include supervised
finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we
call Reinforcement Learning with Verifiable Rewards (RLVR). With T\"ULU 3, we
introduce a multi-task evaluation scheme for post-training recipes with
development and unseen evaluations, standard benchmark implementations, and
substantial decontamination of existing open datasets on said benchmarks. We
conclude with analysis and discussion of training methods that did not reliably
improve performance.
  In addition to the T\"ULU 3 model weights and demo, we release the complete
recipe -- including datasets for diverse core skills, a robust toolkit for data
curation and evaluation, the training code and infrastructure, and, most
importantly, a detailed report for reproducing and further adapting the T\"ULU
3 approach to more domains.
WildLMa: Long Horizon Loco-Manipulation in the Wild
Authors: name: Ri-Zhao Qiu, name: Yuchen Song, name: Xuanbin Peng, name: Sai Aneesh Suryadevara, name: Ge Yang, name: Minghuan Liu, name: Mazeyu Ji, name: Chengzhe Jia, name: Ruihan Yang, name: Xueyan Zou, name: Xiaolong Wang
Published: 2024-11-22 18:56:56

`In-the-wild' mobile manipulation aims to deploy robots in diverse real-world
environments, which requires the robot to (1) have skills that generalize
across object configurations; (2) be capable of long-horizon task execution in
diverse environments; and (3) perform complex manipulation beyond
pick-and-place. Quadruped robots with manipulators hold promise for extending
the workspace and enabling robust locomotion, but existing results do not
investigate such a capability. This paper proposes WildLMa with three
components to address these issues: (1) adaptation of learned low-level
controller for VR-enabled whole-body teleoperation and traversability; (2)
WildLMa-Skill -- a library of generalizable visuomotor skills acquired via
imitation learning or heuristics and (3) WildLMa-Planner -- an interface of
learned skills that allow LLM planners to coordinate skills for long-horizon
tasks. We demonstrate the importance of high-quality training data by achieving
higher grasping success rate over existing RL baselines using only tens of
demonstrations. WildLMa exploits CLIP for language-conditioned imitation
learning that empirically generalizes to objects unseen in training
demonstrations. Besides extensive quantitative evaluation, we qualitatively
demonstrate practical robot applications, such as cleaning up trash in
university hallways or outdoor terrains, operating articulated objects, and
rearranging items on a bookshelf.
Novel View Extrapolation with Video Diffusion Priors
Authors: name: Kunhao Liu, name: Ling Shao, name: Shijian Lu
Published: 2024-11-21 15:16:48

The field of novel view synthesis has made significant strides thanks to the
development of radiance field methods. However, most radiance field techniques
are far better at novel view interpolation than novel view extrapolation where
the synthesis novel views are far beyond the observed training views. We design
ViewExtrapolator, a novel view synthesis approach that leverages the generative
priors of Stable Video Diffusion (SVD) for realistic novel view extrapolation.
By redesigning the SVD denoising process, ViewExtrapolator refines the
artifact-prone views rendered by radiance fields, greatly enhancing the clarity
and realism of the synthesized novel views. ViewExtrapolator is a generic novel
view extrapolator that can work with different types of 3D rendering such as
views rendered from point clouds when only a single view or monocular video is
available. Additionally, ViewExtrapolator requires no fine-tuning of SVD,
making it both data-efficient and computation-efficient. Extensive experiments
demonstrate the superiority of ViewExtrapolator in novel view extrapolation.
Project page: https://kunhao-liu.github.io/ViewExtrapolator/.
MyTimeMachine: Personalized Facial Age Transformation
Authors: name: Luchao Qi, name: Jiaye Wu, name: Bang Gong, name: Annie N. Wang, name: David W. Jacobs, name: Roni Sengupta
Published: 2024-11-21 18:54:13

Facial aging is a complex process, highly dependent on multiple factors like
gender, ethnicity, lifestyle, etc., making it extremely challenging to learn a
global aging prior to predict aging for any individual accurately. Existing
techniques often produce realistic and plausible aging results, but the re-aged
images often do not resemble the person's appearance at the target age and thus
need personalization. In many practical applications of virtual aging, e.g. VFX
in movies and TV shows, access to a personal photo collection of the user
depicting aging in a small time interval (20sim40 years) is often available.
However, naive attempts to personalize global aging techniques on personal
photo collections often fail. Thus, we propose MyTimeMachine (MyTM), which
combines a global aging prior with a personal photo collection (using as few as
50 images) to learn a personalized age transformation. We introduce a novel
Adapter Network that combines personalized aging features with global aging
features and generates a re-aged image with StyleGAN2. We also introduce three
loss functions to personalize the Adapter Network with personalized aging loss,
extrapolation regularization, and adaptive w-norm regularization. Our approach
can also be extended to videos, achieving high-quality, identity-preserving,
and temporally consistent aging effects that resemble actual appearances at
target ages, demonstrating its superiority over state-of-the-art approaches.
OminiControl: Minimal and Universal Control for Diffusion Transformer
Authors: name: Zhenxiong Tan, name: Songhua Liu, name: Xingyi Yang, name: Qiaochu Xue, name: Xinchao Wang
Published: 2024-11-22 17:55:15

In this paper, we introduce OminiControl, a highly versatile and
parameter-efficient framework that integrates image conditions into pre-trained
Diffusion Transformer (DiT) models. At its core, OminiControl leverages a
parameter reuse mechanism, enabling the DiT to encode image conditions using
itself as a powerful backbone and process them with its flexible multi-modal
attention processors. Unlike existing methods, which rely heavily on additional
encoder modules with complex architectures, OminiControl (1) effectively and
efficiently incorporates injected image conditions with only ~0.1% additional
parameters, and (2) addresses a wide range of image conditioning tasks in a
unified manner, including subject-driven generation and spatially-aligned
conditions such as edges, depth, and more. Remarkably, these capabilities are
achieved by training on images generated by the DiT itself, which is
particularly beneficial for subject-driven generation. Extensive evaluations
demonstrate that OminiControl outperforms existing UNet-based and DiT-adapted
models in both subject-driven and spatially-aligned conditional generation.
Additionally, we release our training dataset, Subjects200K, a diverse
collection of over 200,000 identity-consistent images, along with an efficient
data synthesis pipeline to advance research in subject-consistent generation.
Style-Friendly SNR Sampler for Style-Driven Generation
Authors: name: Jooyoung Choi, name: Chaehun Shin, name: Yeongtak Oh, name: Heeseung Kim, name: Sungroh Yoon
Published: 2024-11-22 08:29:25

Recent large-scale diffusion models generate high-quality images but struggle
to learn new, personalized artistic styles, which limits the creation of unique
style templates. Fine-tuning with reference images is the most promising
approach, but it often blindly utilizes objectives and noise level
distributions used for pre-training, leading to suboptimal style alignment. We
propose the Style-friendly SNR sampler, which aggressively shifts the
signal-to-noise ratio (SNR) distribution toward higher noise levels during
fine-tuning to focus on noise levels where stylistic features emerge. This
enables models to better capture unique styles and generate images with higher
style alignment. Our method allows diffusion models to learn and share new
"style templates", enhancing personalized content creation. We demonstrate the
ability to generate styles such as personal watercolor paintings, minimal flat
cartoons, 3D renderings, multi-panel images, and memes with text, thereby
broadening the scope of style-driven generation.
DINO-X: A Unified Vision Model for Open-World Object Detection and
  Understanding
Authors: name: Tianhe Ren, name: Yihao Chen, name: Qing Jiang, name: Zhaoyang Zeng, name: Yuda Xiong, name: Wenlong Liu, name: Zhengyu Ma, name: Junyi Shen, name: Yuan Gao, name: Xiaoke Jiang, name: Xingyu Chen, name: Zhuheng Song, name: Yuhong Zhang, name: Hongjie Huang, name: Han Gao, name: Shilong Liu, name: Hao Zhang, name: Feng Li, name: Kent Yu, name: Lei Zhang
Published: 2024-11-21 17:42:20

In this paper, we introduce DINO-X, which is a unified object-centric vision
model developed by IDEA Research with the best open-world object detection
performance to date. DINO-X employs the same Transformer-based encoder-decoder
architecture as Grounding DINO 1.5 to pursue an object-level representation for
open-world object understanding. To make long-tailed object detection easy,
DINO-X extends its input options to support text prompt, visual prompt, and
customized prompt. With such flexible prompt options, we develop a universal
object prompt to support prompt-free open-world detection, making it possible
to detect anything in an image without requiring users to provide any prompt.
To enhance the model's core grounding capability, we have constructed a
large-scale dataset with over 100 million high-quality grounding samples,
referred to as Grounding-100M, for advancing the model's open-vocabulary
detection performance. Pre-training on such a large-scale grounding dataset
leads to a foundational object-level representation, which enables DINO-X to
integrate multiple perception heads to simultaneously support multiple object
perception and understanding tasks, including detection, segmentation, pose
estimation, object captioning, object-based QA, etc. Experimental results
demonstrate the superior performance of DINO-X. Specifically, the DINO-X Pro
model achieves 56.0 AP, 59.8 AP, and 52.4 AP on the COCO, LVIS-minival, and
LVIS-val zero-shot object detection benchmarks, respectively. Notably, it
scores 63.3 AP and 56.5 AP on the rare classes of LVIS-minival and LVIS-val
benchmarks, both improving the previous SOTA performance by 5.8 AP. Such a
result underscores its significantly improved capacity for recognizing
long-tailed objects.
Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable
  Single-stage Image-to-3D Generation
Authors: name: Yuanhao Cai, name: He Zhang, name: Kai Zhang, name: Yixun Liang, name: Mengwei Ren, name: Fujun Luan, name: Qing Liu, name: Soo Ye Kim, name: Jianming Zhang, name: Zhifei Zhang, name: Yuqian Zhou, name: Zhe Lin, name: Alan Yuille
Published: 2024-11-21 18:21:24

Existing feed-forward image-to-3D methods mainly rely on 2D multi-view
diffusion models that cannot guarantee 3D consistency. These methods easily
collapse when changing the prompt view direction and mainly handle
object-centric prompt images. In this paper, we propose a novel single-stage 3D
diffusion model, DiffusionGS, for object and scene generation from a single
view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to
enforce view consistency and allow the model to generate robustly given prompt
views of any directions, beyond object-centric inputs. Plus, to improve the
capability and generalization ability of DiffusionGS, we scale up 3D training
data by developing a scene-object mixed training strategy. Experiments show
that our method enjoys better generation quality (2.20 dB higher in PSNR and
23.25 lower in FID) and over 5x faster speed (~6s on an A100 GPU) than SOTA
methods. The user study and text-to-3D applications also reveals the practical
values of our method. Our Project page at
https://caiyuanhao1998.github.io/project/DiffusionGS/ shows the video and
interactive generation results.
UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs
  on Low-Resource Languages
Authors: name: Bethel Melesse Tessema, name: Akhil Kedia, name: Tae-Sun Chung
Published: 2024-11-21 17:41:08

Large language models (LLMs) under-perform on low-resource languages due to
limited training data. We present a method to efficiently collect text data for
low-resource languages from the entire Common Crawl corpus. Our approach,
UnifiedCrawl, filters and extracts common crawl using minimal compute
resources, yielding mono-lingual datasets much larger than previously available
sources. We demonstrate that leveraging this data to fine-tuning multilingual
LLMs via efficient adapter methods (QLoRA) significantly boosts performance on
the low-resource language, while minimizing VRAM usage. Our experiments show
large improvements in language modeling perplexity and an increase in few-shot
prompting scores. Our work and released source code provide an affordable
approach to improve LLMs for low-resource languages using consumer hardware.
Our source code is available here at
https://github.com/bethelmelesse/unifiedcrawl.
MagicDriveDiT: High-Resolution Long Video Generation for Autonomous
  Driving with Adaptive Control
Authors: name: Ruiyuan Gao, name: Kai Chen, name: Bo Xiao, name: Lanqing Hong, name: Zhenguo Li, name: Qiang Xu
Published: 2024-11-21 03:13:30

The rapid advancement of diffusion models has greatly improved video
synthesis, especially in controllable video generation, which is essential for
applications like autonomous driving. However, existing methods are limited by
scalability and how control conditions are integrated, failing to meet the
needs for high-resolution and long videos for autonomous driving applications.
In this paper, we introduce MagicDriveDiT, a novel approach based on the DiT
architecture, and tackle these challenges. Our method enhances scalability
through flow matching and employs a progressive training strategy to manage
complex scenarios. By incorporating spatial-temporal conditional encoding,
MagicDriveDiT achieves precise control over spatial-temporal latents.
Comprehensive experiments show its superior performance in generating realistic
street scene videos with higher resolution and more frames. MagicDriveDiT
significantly improves video generation quality and spatial-temporal controls,
expanding its potential applications across various tasks in autonomous
driving.
Do I Know This Entity? Knowledge Awareness and Hallucinations in
  Language Models
Authors: name: Javier Ferrando, name: Oscar Obeso, name: Senthooran Rajamanoharan, name: Neel Nanda
Published: 2024-11-21 16:05:58

Hallucinations in large language models are a widespread problem, yet the
mechanisms behind whether models will hallucinate are poorly understood,
limiting our ability to solve this problem. Using sparse autoencoders as an
interpretability tool, we discover that a key part of these mechanisms is
entity recognition, where the model detects if an entity is one it can recall
facts about. Sparse autoencoders uncover meaningful directions in the
representation space, these detect whether the model recognizes an entity, e.g.
detecting it doesn't know about an athlete or a movie. This suggests that
models can have self-knowledge: internal representations about their own
capabilities. These directions are causally relevant: capable of steering the
model to refuse to answer questions about known entities, or to hallucinate
attributes of unknown entities when it would otherwise refuse. We demonstrate
that despite the sparse autoencoders being trained on the base model, these
directions have a causal effect on the chat model's refusal behavior,
suggesting that chat finetuning has repurposed this existing mechanism.
Furthermore, we provide an initial exploration into the mechanistic role of
these directions in the model, finding that they disrupt the attention of
downstream heads that typically move entity attributes to the final token.
Patience Is The Key to Large Language Model Reasoning
Authors: name: Yijiong Yu
Published: 2024-11-20 07:20:48

Recent advancements in the field of large language models, particularly
through the Chain of Thought (CoT) approach, have demonstrated significant
improvements in solving complex problems. However, existing models either tend
to sacrifice detailed reasoning for brevity due to user preferences, or require
extensive and expensive training data to learn complicated reasoning ability,
limiting their potential in solving complex tasks. To bridge this gap,
following the concept of scaling test-time, we propose a simple method by
encouraging models to adopt a more patient reasoning style without the need of
introducing new knowledge or skills. To employ a preference optimization
approach, we generate detailed reasoning processes as positive examples and
simple answers as negative examples, thereby training the model to favor
thoroughness in its responses. Our results demonstrate a performance increase
of up to 6.7% on GSM8k with training just on a lightweight dataset.