Orionnnn/hub-stats huggingface dataset:
StrandHead: Text to Strand-Disentangled 3D Head Avatars Using Hair
  Geometric Priors
Authors: name: Xiaokun Sun, name: Zeyu Cai, name: Zhenyu Zhang, name: Ying Tai, name: Jian Yang
Published: 2024-12-16 09:17:36

While haircut indicates distinct personality, existing avatar generation
methods fail to model practical hair due to the general or entangled
representation. We propose StrandHead, a novel text to 3D head avatar
generation method capable of generating disentangled 3D hair with strand
representation. Without using 3D data for supervision, we demonstrate that
realistic hair strands can be generated from prompts by distilling 2D
generative diffusion models. To this end, we propose a series of reliable
priors on shape initialization, geometric primitives, and statistical haircut
features, leading to a stable optimization and text-aligned performance.
Extensive experiments show that StrandHead achieves the state-of-the-art
reality and diversity of generated 3D head and hair. The generated 3D hair can
also be easily implemented in the Unreal Engine for physical simulation and
other applications. The code will be available at
https://xiaokunsun.github.io/StrandHead.github.io.
Smaller Language Models Are Better Instruction Evolvers
Authors: name: Tingfeng Hui, name: Lulu Zhao, name: Guanting Dong, name: Yaqi Zhang, name: Hua Zhou, name: Sen Su
Published: 2024-12-15 16:07:48

Instruction tuning has been widely used to unleash the complete potential of
large language models. Notably, complex and diverse instructions are of
significant importance as they can effectively align models with various
downstream tasks. However, current approaches to constructing large-scale
instructions predominantly favour powerful models such as GPT-4 or those with
over 70 billion parameters, under the empirical presumption that such larger
language models (LLMs) inherently possess enhanced capabilities. In this study,
we question this prevalent assumption and conduct an in-depth exploration into
the potential of smaller language models (SLMs) in the context of instruction
evolution. Extensive experiments across three scenarios of instruction
evolution reveal that smaller language models (SLMs) can synthesize more
effective instructions than LLMs. Further analysis demonstrates that SLMs
possess a broader output space during instruction evolution, resulting in more
complex and diverse variants. We also observe that the existing metrics fail to
focus on the impact of the instructions. Thus, we propose Instruction
Complex-Aware IFD (IC-IFD), which introduces instruction complexity in the
original IFD score to evaluate the effectiveness of instruction data more
accurately. Our source code is available at:
https://github.com/HypherX/Evolution-Analysis{https://github.com/HypherX/Evolution-Analysis}
IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and
  Illuminations
Authors: name: Zhibing Li, name: Tong Wu, name: Jing Tan, name: Mengchen Zhang, name: Jiaqi Wang, name: Dahua Lin
Published: 2024-12-16 18:52:56

Capturing geometric and material information from images remains a
fundamental challenge in computer vision and graphics. Traditional
optimization-based methods often require hours of computational time to
reconstruct geometry, material properties, and environmental lighting from
dense multi-view inputs, while still struggling with inherent ambiguities
between lighting and material. On the other hand, learning-based approaches
leverage rich material priors from existing 3D object datasets but face
challenges with maintaining multi-view consistency. In this paper, we introduce
IDArb, a diffusion-based model designed to perform intrinsic decomposition on
an arbitrary number of images under varying illuminations. Our method achieves
accurate and multi-view consistent estimation on surface normals and material
properties. This is made possible through a novel cross-view, cross-domain
attention module and an illumination-augmented, view-adaptive training
strategy. Additionally, we introduce ARB-Objaverse, a new dataset that provides
large-scale multi-view intrinsic data and renderings under diverse lighting
conditions, supporting robust training. Extensive experiments demonstrate that
IDArb outperforms state-of-the-art methods both qualitatively and
quantitatively. Moreover, our approach facilitates a range of downstream tasks,
including single-image relighting, photometric stereo, and 3D reconstruction,
highlighting its broad applications in realistic 3D content creation.
MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes
Authors: name: Ruijie Lu, name: Yixin Chen, name: Junfeng Ni, name: Baoxiong Jia, name: Yu Liu, name: Diwen Wan, name: Gang Zeng, name: Siyuan Huang
Published: 2024-12-16 05:23:45

Repurposing pre-trained diffusion models has been proven to be effective for
NVS. However, these methods are mostly limited to a single object; directly
applying such methods to compositional multi-object scenarios yields inferior
results, especially incorrect object placement and inconsistent shape and
appearance under novel views. How to enhance and systematically evaluate the
cross-view consistency of such models remains under-explored. To address this
issue, we propose MOVIS to enhance the structural awareness of the
view-conditioned diffusion model for multi-object NVS in terms of model inputs,
auxiliary tasks, and training strategy. First, we inject structure-aware
features, including depth and object mask, into the denoising U-Net to enhance
the model's comprehension of object instances and their spatial relationships.
Second, we introduce an auxiliary task requiring the model to simultaneously
predict novel view object masks, further improving the model's capability in
differentiating and placing objects. Finally, we conduct an in-depth analysis
of the diffusion sampling process and carefully devise a structure-guided
timestep sampling scheduler during training, which balances the learning of
global object placement and fine-grained detail recovery. To systematically
evaluate the plausibility of synthesized images, we propose to assess
cross-view consistency and novel view object placement alongside existing
image-level NVS metrics. Extensive experiments on challenging synthetic and
realistic datasets demonstrate that our method exhibits strong generalization
capabilities and produces consistent novel view synthesis, highlighting its
potential to guide future 3D-aware multi-object NVS tasks.
RetroLLM: Empowering Large Language Models to Retrieve Fine-grained
  Evidence within Generation
Authors: name: Xiaoxi Li, name: Jiajie Jin, name: Yujia Zhou, name: Yongkang Wu, name: Zhonghua Li, name: Qi Ye, name: Zhicheng Dou
Published: 2024-12-16 16:03:25

Large language models (LLMs) exhibit remarkable generative capabilities but
often suffer from hallucinations. Retrieval-augmented generation (RAG) offers
an effective solution by incorporating external knowledge, but existing methods
still face several limitations: additional deployment costs of separate
retrievers, redundant input tokens from retrieved text chunks, and the lack of
joint optimization of retrieval and generation. To address these issues, we
propose RetroLLM, a unified framework that integrates retrieval and
generation into a single, cohesive process, enabling LLMs to directly generate
fine-grained evidence from the corpus with constrained decoding. Moreover, to
mitigate false pruning in the process of constrained evidence generation, we
introduce (1) hierarchical FM-Index constraints, which generate
corpus-constrained clues to identify a subset of relevant documents before
evidence generation, reducing irrelevant decoding space; and (2) a
forward-looking constrained decoding strategy, which considers the relevance of
future sequences to improve evidence accuracy. Extensive experiments on five
open-domain QA datasets demonstrate RetroLLM's superior performance across both
in-domain and out-of-domain tasks. The code is available at
https://github.com/sunnynexus/RetroLLM.
GaussianProperty: Integrating Physical Properties to 3D Gaussians with
  LMMs
Authors: name: Xinli Xu, name: Wenhang Ge, name: Dicong Qiu, name: ZhiFei Chen, name: Dongyu Yan, name: Zhuoyun Liu, name: Haoyu Zhao, name: Hanfeng Zhao, name: Shunsi Zhang, name: Junwei Liang, name: Ying-Cong Chen
Published: 2024-12-15 17:44:10

Estimating physical properties for visual data is a crucial task in computer
vision, graphics, and robotics, underpinning applications such as augmented
reality, physical simulation, and robotic grasping. However, this area remains
under-explored due to the inherent ambiguities in physical property estimation.
To address these challenges, we introduce GaussianProperty, a training-free
framework that assigns physical properties of materials to 3D Gaussians.
Specifically, we integrate the segmentation capability of SAM with the
recognition capability of GPT-4V(ision) to formulate a global-local physical
property reasoning module for 2D images. Then we project the physical
properties from multi-view 2D images to 3D Gaussians using a voting strategy.
We demonstrate that 3D Gaussians with physical property annotations enable
applications in physics-based dynamic simulation and robotic grasping. For
physics-based dynamic simulation, we leverage the Material Point Method (MPM)
for realistic dynamic simulation. For robot grasping, we develop a grasping
force prediction strategy that estimates a safe force range required for object
grasping based on the estimated physical properties. Extensive experiments on
material segmentation, physics-based dynamic simulation, and robotic grasping
validate the effectiveness of our proposed method, highlighting its crucial
role in understanding physical properties from visual data. Online demo, code,
more cases and annotated datasets are available on
https://Gaussian-Property.github.io{this https URL}.
Efficient Generative Modeling with Residual Vector Quantization-Based
  Tokens
Authors: name: Jaehyeon Kim, name: Taehong Moon, name: Keon Lee, name: Jaewoong Cho
Published: 2024-12-13 15:31:17

We explore the use of Residual Vector Quantization (RVQ) for high-fidelity
generation in vector-quantized generative models. This quantization technique
maintains higher data fidelity by employing more in-depth tokens. However,
increasing the token number in generative models leads to slower inference
speeds. To this end, we introduce ResGen, an efficient RVQ-based discrete
diffusion model that generates high-fidelity samples without compromising
sampling speed. Our key idea is a direct prediction of vector embedding of
collective tokens rather than individual ones. Moreover, we demonstrate that
our proposed token masking and multi-token prediction method can be formulated
within a principled probabilistic framework using a discrete diffusion process
and variational inference. We validate the efficacy and generalizability of the
proposed method on two challenging tasks across different modalities:
conditional image generation} on ImageNet 256x256 and zero-shot text-to-speech
synthesis. Experimental results demonstrate that ResGen outperforms
autoregressive counterparts in both tasks, delivering superior performance
without compromising sampling speed. Furthermore, as we scale the depth of RVQ,
our generative models exhibit enhanced generation fidelity or faster sampling
speeds compared to similarly sized baseline models. The project page can be
found at https://resgen-genai.github.io
GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong
  Prompt Optimizers
Authors: name: Sarkar Snigdha Sarathi Das, name: Ryo Kamoi, name: Bo Pang, name: Yusen Zhang, name: Caiming Xiong, name: Rui Zhang
Published: 2024-12-12 20:59:43

The effectiveness of large language models (LLMs) is closely tied to the
design of prompts, making prompt optimization essential for enhancing their
performance across a wide range of tasks. Many existing approaches to
automating prompt engineering rely exclusively on textual feedback, refining
prompts based solely on inference errors identified by large, computationally
expensive LLMs. Unfortunately, smaller models struggle to generate high-quality
feedback, resulting in complete dependence on large LLM judgment. Moreover,
these methods fail to leverage more direct and finer-grained information, such
as gradients, due to operating purely in text space. To this end, we introduce
GReaTer, a novel prompt optimization technique that directly incorporates
gradient information over task-specific reasoning. By utilizing task loss
gradients, GReaTer enables self-optimization of prompts for open-source,
lightweight language models without the need for costly closed-source LLMs.
This allows high-performance prompt optimization without dependence on massive
LLMs, closing the gap between smaller models and the sophisticated reasoning
often needed for prompt refinement. Extensive evaluations across diverse
reasoning tasks including BBH, GSM8k, and FOLIO demonstrate that GReaTer
consistently outperforms previous state-of-the-art prompt optimization methods,
even those reliant on powerful LLMs. Additionally, GReaTer-optimized prompts
frequently exhibit better transferability and, in some cases, boost task
performance to levels comparable to or surpassing those achieved by larger
language models, highlighting the effectiveness of prompt optimization guided
by gradients over reasoning. Code of GReaTer is available at
https://github.com/psunlpgroup/GreaTer.
LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation
  with Linear Computational Complexity
Authors: name: Hongjie Wang, name: Chih-Yao Ma, name: Yen-Cheng Liu, name: Ji Hou, name: Tao Xu, name: Jialiang Wang, name: Felix Juefei-Xu, name: Yaqiao Luo, name: Peizhao Zhang, name: Tingbo Hou, name: Peter Vajda, name: Niraj K. Jha, name: Xiaoliang Dai
Published: 2024-12-13 04:55:10

Text-to-video generation enhances content creation but is highly
computationally intensive: The computational cost of Diffusion Transformers
(DiTs) scales quadratically in the number of pixels. This makes minute-length
video generation extremely expensive, limiting most existing models to
generating videos of only 10-20 seconds length. We propose a Linear-complexity
text-to-video Generation (LinGen) framework whose cost scales linearly in the
number of pixels. For the first time, LinGen enables high-resolution
minute-length video generation on a single GPU without compromising quality. It
replaces the computationally-dominant and quadratic-complexity block,
self-attention, with a linear-complexity block called MATE, which consists of
an MA-branch and a TE-branch. The MA-branch targets short-to-long-range
correlations, combining a bidirectional Mamba2 block with our token
rearrangement method, Rotary Major Scan, and our review tokens developed for
long video generation. The TE-branch is a novel TEmporal Swin Attention block
that focuses on temporal correlations between adjacent tokens and medium-range
tokens. The MATE block addresses the adjacency preservation issue of Mamba and
improves the consistency of generated videos significantly. Experimental
results show that LinGen outperforms DiT (with a 75.6% win rate) in video
quality with up to 15times (11.5times) FLOPs (latency) reduction.
Furthermore, both automatic metrics and human evaluation demonstrate our
LinGen-4B yields comparable video quality to state-of-the-art models (with a
50.5%, 52.1%, 49.1% win rate with respect to Gen-3, LumaLabs, and Kling,
respectively). This paves the way to hour-length movie generation and real-time
interactive video generation. We provide 68s video generation results and more
examples in our project website: https://lineargen.github.io/.
TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for
  Generalist Robotic Policies
Authors: name: Ruijie Zheng, name: Yongyuan Liang, name: Shuaiyi Huang, name: Jianfeng Gao, name: Hal Daumé III, name: Andrey Kolobov, name: Furong Huang, name: Jianwei Yang
Published: 2024-12-13 18:40:51

Although large vision-language-action (VLA) models pretrained on extensive
robot datasets offer promising generalist policies for robotic learning, they
still struggle with spatial-temporal dynamics in interactive robotics, making
them less effective in handling complex tasks, such as manipulation. In this
work, we introduce visual trace prompting, a simple yet effective approach to
facilitate VLA models' spatial-temporal awareness for action prediction by
encoding state-action trajectories visually. We develop a new TraceVLA model by
finetuning OpenVLA on our own collected dataset of 150K robot manipulation
trajectories using visual trace prompting. Evaluations of TraceVLA across 137
configurations in SimplerEnv and 4 tasks on a physical WidowX robot demonstrate
state-of-the-art performance, outperforming OpenVLA by 10% on SimplerEnv and
3.5x on real-robot tasks and exhibiting robust generalization across diverse
embodiments and scenarios. To further validate the effectiveness and generality
of our method, we present a compact VLA model based on 4B Phi-3-Vision,
pretrained on the Open-X-Embodiment and finetuned on our dataset, rivals the 7B
OpenVLA baseline while significantly improving inference efficiency.
FluxSpace: Disentangled Semantic Editing in Rectified Flow Transformers
Authors: name: Yusuf Dalva, name: Kavana Venkatesh, name: Pinar Yanardag
Published: 2024-12-12 18:59:40

Rectified flow models have emerged as a dominant approach in image
generation, showcasing impressive capabilities in high-quality image synthesis.
However, despite their effectiveness in visual generation, rectified flow
models often struggle with disentangled editing of images. This limitation
prevents the ability to perform precise, attribute-specific modifications
without affecting unrelated aspects of the image. In this paper, we introduce
FluxSpace, a domain-agnostic image editing method leveraging a representation
space with the ability to control the semantics of images generated by
rectified flow transformers, such as Flux. By leveraging the representations
learned by the transformer blocks within the rectified flow models, we propose
a set of semantically interpretable representations that enable a wide range of
image editing tasks, from fine-grained image editing to artistic creation. This
work offers a scalable and effective image editing approach, along with its
disentanglement capabilities.
SynerGen-VL: Towards Synergistic Image Understanding and Generation with
  Vision Experts and Token Folding
Authors: name: Hao Li, name: Changyao Tian, name: Jie Shao, name: Xizhou Zhu, name: Zhaokai Wang, name: Jinguo Zhu, name: Wenhan Dou, name: Xiaogang Wang, name: Hongsheng Li, name: Lewei Lu, name: Jifeng Dai
Published: 2024-12-12 18:59:26

The remarkable success of Large Language Models (LLMs) has extended to the
multimodal domain, achieving outstanding performance in image understanding and
generation. Recent efforts to develop unified Multimodal Large Language Models
(MLLMs) that integrate these capabilities have shown promising results.
However, existing approaches often involve complex designs in model
architecture or training pipeline, increasing the difficulty of model training
and scaling. In this paper, we propose SynerGen-VL, a simple yet powerful
encoder-free MLLM capable of both image understanding and generation. To
address challenges identified in existing encoder-free unified MLLMs, we
introduce the token folding mechanism and the vision-expert-based progressive
alignment pretraining strategy, which effectively support high-resolution image
understanding while reducing training complexity. After being trained on
large-scale mixed image-text data with a unified next-token prediction
objective, SynerGen-VL achieves or surpasses the performance of existing
encoder-free unified MLLMs with comparable or smaller parameter sizes, and
narrows the gap with task-specific state-of-the-art models, highlighting a
promising path toward future unified MLLMs. Our code and models shall be
released.
ObjectMate: A Recurrence Prior for Object Insertion and Subject-Driven
  Generation
Authors: name: Daniel Winter, name: Asaf Shul, name: Matan Cohen, name: Dana Berman, name: Yael Pritch, name: Alex Rav-Acha, name: Yedid Hoshen
Published: 2024-12-11 18:59:53

This paper introduces a tuning-free method for both object insertion and
subject-driven generation. The task involves composing an object, given
multiple views, into a scene specified by either an image or text. Existing
methods struggle to fully meet the task's challenging objectives: (i)
seamlessly composing the object into the scene with photorealistic pose and
lighting, and (ii) preserving the object's identity. We hypothesize that
achieving these goals requires large scale supervision, but manually collecting
sufficient data is simply too expensive. The key observation in this paper is
that many mass-produced objects recur across multiple images of large unlabeled
datasets, in different scenes, poses, and lighting conditions. We use this
observation to create massive supervision by retrieving sets of diverse views
of the same object. This powerful paired dataset enables us to train a
straightforward text-to-image diffusion architecture to map the object and
scene descriptions to the composited image. We compare our method, ObjectMate,
with state-of-the-art methods for object insertion and subject-driven
generation, using a single or multiple references. Empirically, ObjectMate
achieves superior identity preservation and more photorealistic composition.
Differently from many other multi-reference methods, ObjectMate does not
require slow test-time tuning.
SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better
  Reasoning in SLMs
Authors: name: Sultan Alrashed
Published: 2024-12-11 12:41:36

We present SmolTulu-1.7b-Instruct, referenced in this report as
SmolTulu-DPO-1130, an instruction-tuned language model that adapts AllenAI's
Tulu 3 post-training pipeline to enhance Huggingface's SmolLM2-1.7B base model.
Through comprehensive empirical analysis using a 135M parameter model, we
demonstrate that the relationship between learning rate and batch size
significantly impacts model performance in a task-dependent manner. Our
findings reveal a clear split: reasoning tasks like ARC and GSM8K benefit from
higher learning rate to batch size ratios, while pattern recognition tasks such
as HellaSwag and IFEval show optimal performance with lower ratios. These
insights informed the development of SmolTulu, which achieves state-of-the-art
performance among sub-2B parameter models on instruction following, scoring
67.7% on IFEval (Delta11%), and mathematical reasoning with 51.6% on GSM8K
(Delta3.4%), with an alternate version achieving scoring 57.1% on ARC
(Delta5.4%). We release our model, training recipes, and ablation studies to
facilitate further research in efficient model alignment, demonstrating that
careful adaptation of optimization dynamics can help bridge the capability gap
between small and large language models.