Orionnnn/hub-stats huggingface dataset:
TEXGen: a Generative Diffusion Model for Mesh Textures
Authors: name: Xin Yu, name: Ze Yuan, name: Yuan-Chen Guo, name: Ying-Tian Liu, name: JianHui Liu, name: Yangguang Li, name: Yan-Pei Cao, name: Ding Liang, name: Xiaojuan Qi
Published: 2024-11-22 05:22:11

While high-quality texture maps are essential for realistic 3D asset
rendering, few studies have explored learning directly in the texture space,
especially on large-scale datasets. In this work, we depart from the
conventional approach of relying on pre-trained 2D diffusion models for
test-time optimization of 3D textures. Instead, we focus on the fundamental
problem of learning in the UV texture space itself. For the first time, we
train a large diffusion model capable of directly generating high-resolution
texture maps in a feed-forward manner. To facilitate efficient learning in
high-resolution UV spaces, we propose a scalable network architecture that
interleaves convolutions on UV maps with attention layers on point clouds.
Leveraging this architectural design, we train a 700 million parameter
diffusion model that can generate UV texture maps guided by text prompts and
single-view images. Once trained, our model naturally supports various extended
applications, including text-guided texture inpainting, sparse-view texture
completion, and text-driven texture synthesis. Project page is at
http://cvmi-lab.github.io/TEXGen/.
DreamMix: Decoupling Object Attributes for Enhanced Editability in
  Customized Image Inpainting
Authors: name: Yicheng Yang, name: Pengxiang Li, name: Lu Zhang, name: Liqian Ma, name: Ping Hu, name: Siyu Du, name: Yunzhi Zhuge, name: Xu Jia, name: Huchuan Lu
Published: 2024-11-26 08:44:47

Subject-driven image inpainting has emerged as a popular task in image
editing alongside recent advancements in diffusion models. Previous methods
primarily focus on identity preservation but struggle to maintain the
editability of inserted objects. In response, this paper introduces DreamMix, a
diffusion-based generative model adept at inserting target objects into given
scenes at user-specified locations while concurrently enabling arbitrary
text-driven modifications to their attributes. In particular, we leverage
advanced foundational inpainting models and introduce a disentangled
local-global inpainting framework to balance precise local object insertion
with effective global visual coherence. Additionally, we propose an Attribute
Decoupling Mechanism (ADM) and a Textual Attribute Substitution (TAS) module to
improve the diversity and discriminative capability of the text-based attribute
guidance, respectively. Extensive experiments demonstrate that DreamMix
effectively balances identity preservation and attribute editability across
various application scenarios, including object insertion, attribute editing,
and small object inpainting. Our code is publicly available at
https://github.com/mycfhs/DreamMix.
Star Attention: Efficient LLM Inference over Long Sequences
Authors: name: Shantanu Acharya, name: Fei Jia, name: Boris Ginsburg
Published: 2024-11-26 05:10:04

Inference with Transformer-based Large Language Models (LLMs) on long
sequences is both costly and slow due to the quadratic complexity of the
self-attention mechanism. We introduce Star Attention, a two-phase block-sparse
approximation that improves computational efficiency by sharding attention
across multiple hosts while minimizing communication overhead. In the first
phase, the context is processed using blockwise-local attention across hosts,
in parallel. In the second phase, query and response tokens attend to all prior
cached tokens through sequence-global attention. Star Attention integrates
seamlessly with most Transformer-based LLMs trained with global attention,
reducing memory requirements and inference time by up to 11x while preserving
95-100% of accuracy.
Find Any Part in 3D
Authors: name: Ziqi Ma, name: Yisong Yue, name: Georgia Gkioxari
Published: 2024-11-20 18:59:01

We study open-world part segmentation in 3D: segmenting any part in any
object based on any text query. Prior methods are limited in object categories
and part vocabularies. Recent advances in AI have demonstrated effective
open-world recognition capabilities in 2D. Inspired by this progress, we
propose an open-world, direct-prediction model for 3D part segmentation that
can be applied zero-shot to any object. Our approach, called Find3D, trains a
general-category point embedding model on large-scale 3D assets from the
internet without any human annotation. It combines a data engine, powered by
foundation models for annotating data, with a contrastive training method. We
achieve strong performance and generalization across multiple datasets, with up
to a 3x improvement in mIoU over the next best method. Our model is 6x to over
300x faster than existing baselines. To encourage research in general-category
open-world 3D part segmentation, we also release a benchmark for general
objects and parts. Project website: https://ziqi-ma.github.io/find3dsite/
Edge Weight Prediction For Category-Agnostic Pose Estimation
Authors: name: Or Hirschorn, name: Shai Avidan
Published: 2024-11-25 18:53:09

Category-Agnostic Pose Estimation (CAPE) localizes keypoints across diverse
object categories with a single model, using one or a few annotated support
images. Recent works have shown that using a pose graph (i.e., treating
keypoints as nodes in a graph rather than isolated points) helps handle
occlusions and break symmetry. However, these methods assume a static pose
graph with equal-weight edges, leading to suboptimal results. We introduce
EdgeCape, a novel framework that overcomes these limitations by predicting the
graph's edge weights which optimizes localization. To further leverage
structural priors, we propose integrating Markovian Structural Bias, which
modulates the self-attention interaction between nodes based on the number of
hops between them. We show that this improves the model's ability to capture
global spatial dependencies. Evaluated on the MP-100 benchmark, which includes
100 categories and over 20K images, EdgeCape achieves state-of-the-art results
in the 1-shot setting and leads among similar-sized methods in the 5-shot
setting, significantly improving keypoint localization accuracy. Our code is
publicly available.
Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline
Authors: name: Junlong Cheng, name: Bin Fu, name: Jin Ye, name: Guoan Wang, name: Tianbin Li, name: Haoyu Wang, name: Ruoyu Li, name: He Yao, name: Junren Chen, name: JingWen Li, name: Yanzhou Su, name: Min Zhu, name: Junjun He
Published: 2024-11-19 19:06:29

Interactive Medical Image Segmentation (IMIS) has long been constrained by
the limited availability of large-scale, diverse, and densely annotated
datasets, which hinders model generalization and consistent evaluation across
different models. In this paper, we introduce the IMed-361M benchmark dataset,
a significant advancement in general IMIS research. First, we collect and
standardize over 6.4 million medical images and their corresponding ground
truth masks from multiple data sources. Then, leveraging the strong object
recognition capabilities of a vision foundational model, we automatically
generated dense interactive masks for each image and ensured their quality
through rigorous quality control and granularity management. Unlike previous
datasets, which are limited by specific modalities or sparse annotations,
IMed-361M spans 14 modalities and 204 segmentation targets, totaling 361
million masks-an average of 56 masks per image. Finally, we developed an IMIS
baseline network on this dataset that supports high-quality mask generation
through interactive inputs, including clicks, bounding boxes, text prompts, and
their combinations. We evaluate its performance on medical image segmentation
tasks from multiple perspectives, demonstrating superior accuracy and
scalability compared to existing interactive segmentation models. To facilitate
research on foundational models in medical computer vision, we release the
IMed-361M and model at https://github.com/uni-medical/IMIS-Bench.
LLMs Do Not Think Step-by-step In Implicit Reasoning
Authors: name: Yijiong Yu
Published: 2024-11-24 14:38:59

It has been well-known that Chain-of-Thought can remarkably enhance LLMs'
performance on complex tasks. However, because it also introduces slower
inference speeds and higher computational costs, many researches have attempted
to use implicit CoT, which does not need LLMs to explicitly generate the
intermediate steps. But there is still gap between their efficacy and typical
explicit CoT methods. This leaves us a doubt that, does implicit CoT really
equal to explicit CoT? Therefore, in this study, we address this question
through experiments. We probe the information of intermediate steps from the
model's hidden states when it is performing implicit CoT. The results
surprisingly indicate that LLMs hardly think about intermediate steps,
suggesting they may just rely on experience rather than strict step-by-step
reasoning. Moreover, we find LLMs' implicit reasoning capabilities are
susceptible and unstable, reaffirming the necessity of explicit CoT to
effectively support complex tasks.
Knowledge Transfer Across Modalities with Natural Language Supervision
Authors: name: Carlo Alberto Barbano, name: Luca Molinaro, name: Emanuele Aiello, name: Marco Grangetto
Published: 2024-11-23 17:26:50

We present a way to learn novel concepts by only using their textual
description. We call this method Knowledge Transfer. Similarly to human
perception, we leverage cross-modal interaction to introduce new concepts. We
hypothesize that in a pre-trained visual encoder there are enough low-level
features already learned (e.g. shape, appearance, color) that can be used to
describe previously unknown high-level concepts. Provided with a textual
description of the novel concept, our method works by aligning the known
low-level features of the visual encoder to its high-level textual description.
We show that Knowledge Transfer can successfully introduce novel concepts in
multimodal models, in a very efficient manner, by only requiring a single
description of the target concept. Our approach is compatible with both
separate textual and visual encoders (e.g. CLIP) and shared parameters across
modalities. We also show that, following the same principle, Knowledge Transfer
can improve concepts already known by the model. Leveraging Knowledge Transfer
we improve zero-shot performance across different tasks such as classification,
segmentation, image-text retrieval, and captioning.
One Diffusion to Generate Them All
Authors: name: Duong H. Le, name: Tuan Pham, name: Sangho Lee, name: Christopher Clark, name: Aniruddha Kembhavi, name: Stephan Mandt, name: Ranjay Krishna, name: Jiasen Lu
Published: 2024-11-25 12:11:05

We introduce OneDiffusion, a versatile, large-scale diffusion model that
seamlessly supports bidirectional image synthesis and understanding across
diverse tasks. It enables conditional generation from inputs such as text,
depth, pose, layout, and semantic maps, while also handling tasks like image
deblurring, upscaling, and reverse processes such as depth estimation and
segmentation. Additionally, OneDiffusion allows for multi-view generation,
camera pose estimation, and instant personalization using sequential image
inputs. Our model takes a straightforward yet effective approach by treating
all tasks as frame sequences with varying noise scales during training,
allowing any frame to act as a conditioning image at inference time. Our
unified training framework removes the need for specialized architectures,
supports scalable multi-task training, and adapts smoothly to any resolution,
enhancing both generalization and scalability. Experimental results demonstrate
competitive performance across tasks in both generation and prediction such as
text-to-image, multiview generation, ID preservation, depth estimation and
camera pose estimation despite relatively small training dataset. Our code and
checkpoint are freely available at https://github.com/lehduong/OneDiffusion
VisualLens: Personalization through Visual History
Authors: name: Wang Bill Zhu, name: Deqing Fu, name: Kai Sun, name: Yi Lu, name: Zhaojiang Lin, name: Seungwhan Moon, name: Kanika Narang, name: Mustafa Canim, name: Yue Liu, name: Anuj Kumar, name: Xin Luna Dong
Published: 2024-11-25 01:45:42

We hypothesize that a user's visual history with images reflecting their
daily life, offers valuable insights into their interests and preferences, and
can be leveraged for personalization. Among the many challenges to achieve this
goal, the foremost is the diversity and noises in the visual history,
containing images not necessarily related to a recommendation task, not
necessarily reflecting the user's interest, or even not necessarily
preference-relevant. Existing recommendation systems either rely on
task-specific user interaction logs, such as online shopping history for
shopping recommendations, or focus on text signals. We propose a novel
approach, VisualLens, that extracts, filters, and refines image
representations, and leverages these signals for personalization. We created
two new benchmarks with task-agnostic visual histories, and show that our
method improves over state-of-the-art recommendations by 5-10% on Hit@3, and
improves over GPT-4o by 2-5%. Our approach paves the way for personalized
recommendations in scenarios where traditional methods fail.
Factorized Visual Tokenization and Generation
Authors: name: Zechen Bai, name: Jianxiong Gao, name: Ziteng Gao, name: Pichao Wang, name: Zheng Zhang, name: Tong He, name: Mike Zheng Shou
Published: 2024-11-25 18:59:53

Visual tokenizers are fundamental to image generation. They convert visual
data into discrete tokens, enabling transformer-based models to excel at image
generation. Despite their success, VQ-based tokenizers like VQGAN face
significant limitations due to constrained vocabulary sizes. Simply expanding
the codebook often leads to training instability and diminishing performance
gains, making scalability a critical challenge. In this work, we introduce
Factorized Quantization (FQ), a novel approach that revitalizes VQ-based
tokenizers by decomposing a large codebook into multiple independent
sub-codebooks. This factorization reduces the lookup complexity of large
codebooks, enabling more efficient and scalable visual tokenization. To ensure
each sub-codebook captures distinct and complementary information, we propose a
disentanglement regularization that explicitly reduces redundancy, promoting
diversity across the sub-codebooks. Furthermore, we integrate representation
learning into the training process, leveraging pretrained vision models like
CLIP and DINO to infuse semantic richness into the learned representations.
This design ensures our tokenizer captures diverse semantic levels, leading to
more expressive and disentangled representations. Experiments show that the
proposed FQGAN model substantially improves the reconstruction quality of
visual tokenizers, achieving state-of-the-art performance. We further
demonstrate that this tokenizer can be effectively adapted into auto-regressive
image generation. https://showlab.github.io/FQGAN
O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple
  Distillation, Big Progress or Bitter Lesson?
Authors: name: Zhen Huang, name: Haoyang Zou, name: Xuefeng Li, name: Yixiu Liu, name: Yuxiang Zheng, name: Ethan Chern, name: Shijie Xia, name: Yiwei Qin, name: Weizhe Yuan, name: Pengfei Liu
Published: 2024-11-25 15:31:27

This paper presents a critical examination of current approaches to
replicating OpenAI's O1 model capabilities, with particular focus on the
widespread but often undisclosed use of knowledge distillation techniques.
While our previous work explored the fundamental technical path to O1
replication, this study reveals how simple distillation from O1's API, combined
with supervised fine-tuning, can achieve superior performance on complex
mathematical reasoning tasks. Through extensive experiments, we show that a
base model fine-tuned on simply tens of thousands of samples O1-distilled
long-thought chains outperforms O1-preview on the American Invitational
Mathematics Examination (AIME) with minimal technical complexity. Moreover, our
investigation extends beyond mathematical reasoning to explore the
generalization capabilities of O1-distilled models across diverse tasks:
hallucination, safety and open-domain QA. Notably, despite training only on
mathematical problem-solving data, our models demonstrated strong
generalization to open-ended QA tasks and became significantly less susceptible
to sycophancy after fine-tuning. We deliberately make this finding public to
promote transparency in AI research and to challenge the current trend of
obscured technical claims in the field. Our work includes: (1) A detailed
technical exposition of the distillation process and its effectiveness, (2) A
comprehensive benchmark framework for evaluating and categorizing O1
replication attempts based on their technical transparency and reproducibility,
(3) A critical discussion of the limitations and potential risks of
over-relying on distillation approaches, our analysis culminates in a crucial
bitter lesson: while the pursuit of more capable AI systems is important, the
development of researchers grounded in first-principles thinking is paramount.
SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting
  Synthesis
Authors: name: Hyojun Go, name: Byeongjun Park, name: Jiho Jang, name: Jin-Young Kim, name: Soonwoo Kwon, name: Changick Kim
Published: 2024-11-25 14:46:17

Text-based generation and editing of 3D scenes hold significant potential for
streamlining content creation through intuitive user interactions. While recent
advances leverage 3D Gaussian Splatting (3DGS) for high-fidelity and real-time
rendering, existing methods are often specialized and task-focused, lacking a
unified framework for both generation and editing. In this paper, we introduce
SplatFlow, a comprehensive framework that addresses this gap by enabling direct
3DGS generation and editing. SplatFlow comprises two main components: a
multi-view rectified flow (RF) model and a Gaussian Splatting Decoder
(GSDecoder). The multi-view RF model operates in latent space, generating
multi-view images, depths, and camera poses simultaneously, conditioned on text
prompts, thus addressing challenges like diverse scene scales and complex
camera trajectories in real-world settings. Then, the GSDecoder efficiently
translates these latent outputs into 3DGS representations through a
feed-forward 3DGS method. Leveraging training-free inversion and inpainting
techniques, SplatFlow enables seamless 3DGS editing and supports a broad range
of 3D tasks-including object editing, novel view synthesis, and camera pose
estimation-within a unified framework without requiring additional complex
pipelines. We validate SplatFlow's capabilities on the MVImgNet and DL3DV-7K
datasets, demonstrating its versatility and effectiveness in various 3D
generation, editing, and inpainting-based tasks.
Best of Both Worlds: Advantages of Hybrid Graph Sequence Models
Authors: name: Ali Behrouz, name: Ali Parviz, name: Mahdi Karami, name: Clayton Sanford, name: Bryan Perozzi, name: Vahab Mirrokni
Published: 2024-11-23 23:24:42

Modern sequence models (e.g., Transformers, linear RNNs, etc.) emerged as
dominant backbones of recent deep learning frameworks, mainly due to their
efficiency, representational power, and/or ability to capture long-range
dependencies. Adopting these sequence models for graph-structured data has
recently gained popularity as the alternative to Message Passing Neural
Networks (MPNNs). There is, however, a lack of a common foundation about what
constitutes a good graph sequence model, and a mathematical description of the
benefits and deficiencies in adopting different sequence models for learning on
graphs. To this end, we first present Graph Sequence Model (GSM), a unifying
framework for adopting sequence models for graphs, consisting of three main
steps: (1) Tokenization, which translates the graph into a set of sequences;
(2) Local Encoding, which encodes local neighborhoods around each node; and (3)
Global Encoding, which employs a scalable sequence model to capture long-range
dependencies within the sequences. This framework allows us to understand,
evaluate, and compare the power of different sequence model backbones in graph
tasks. Our theoretical evaluations of the representation power of Transformers
and modern recurrent models through the lens of global and local graph tasks
show that there are both negative and positive sides for both types of models.
Building on this observation, we present GSM++, a fast hybrid model that uses
the Hierarchical Affinity Clustering (HAC) algorithm to tokenize the graph into
hierarchical sequences, and then employs a hybrid architecture of Transformer
to encode these sequences. Our theoretical and experimental results support the
design of GSM++, showing that GSM++ outperforms baselines in most benchmark
evaluations.
Predicting Emergent Capabilities by Finetuning
Authors: name: Charlie Snell, name: Eric Wallace, name: Dan Klein, name: Sergey Levine
Published: 2024-11-25 01:48:09

A fundamental open challenge in modern LLM scaling is the lack of
understanding around emergent capabilities. In particular, language model
pretraining loss is known to be highly predictable as a function of compute.
However, downstream capabilities are far less predictable -- sometimes even
exhibiting emergent jumps -- which makes it challenging to anticipate the
capabilities of future models. In this work, we first pose the task of
emergence prediction: given access to current LLMs that have random few-shot
accuracy on a task, can we predict whether future models (GPT-N+1) will have
non-trivial accuracy on that task? We then discover a simple insight for this
problem: finetuning LLMs on a given task can shift the point in scaling at
which emergence occurs towards less capable models. To operationalize this
insight, we can finetune LLMs with varying amounts of data and fit a parametric
function that predicts when emergence will occur (i.e., "emergence laws"). We
validate this approach using four standard NLP benchmarks where large-scale
open-source LLMs already demonstrate emergence (MMLU, GSM8K, CommonsenseQA, and
CoLA). Using only small-scale LLMs, we find that, in some cases, we can
accurately predict whether models trained with up to 4x more compute have
emerged. Finally, we present a case study of two realistic uses for emergence
prediction.