Orionnnn/hub-stats huggingface dataset:
CoLLM: A Large Language Model for Composed Image Retrieval
Authors: name: Chuong Huynh, name: Jinyu Yang, name: Ashish Tawari, name: Mubarak Shah, name: Son Tran, name: Raffay Hamid, name: Trishul Chilimbi, name: Abhinav Shrivastava
Published: 2025-03-25 17:59:50

Composed Image Retrieval (CIR) is a complex task that aims to retrieve images
based on a multimodal query. Typical training data consists of triplets
containing a reference image, a textual description of desired modifications,
and the target image, which are expensive and time-consuming to acquire. The
scarcity of CIR datasets has led to zero-shot approaches utilizing synthetic
triplets or leveraging vision-language models (VLMs) with ubiquitous
web-crawled image-caption pairs. However, these methods have significant
limitations: synthetic triplets suffer from limited scale, lack of diversity,
and unnatural modification text, while image-caption pairs hinder joint
embedding learning of the multimodal query due to the absence of triplet data.
Moreover, existing approaches struggle with complex and nuanced modification
texts that demand sophisticated fusion and understanding of vision and language
modalities. We present CoLLM, a one-stop framework that effectively addresses
these limitations. Our approach generates triplets on-the-fly from
image-caption pairs, enabling supervised training without manual annotation. We
leverage Large Language Models (LLMs) to generate joint embeddings of reference
images and modification texts, facilitating deeper multimodal fusion.
Additionally, we introduce Multi-Text CIR (MTCIR), a large-scale dataset
comprising 3.4M samples, and refine existing CIR benchmarks (CIRR and
Fashion-IQ) to enhance evaluation reliability. Experimental results demonstrate
that CoLLM achieves state-of-the-art performance across multiple CIR benchmarks
and settings. MTCIR yields competitive results, with up to 15% performance
improvement. Our refined benchmarks provide more reliable evaluation metrics
for CIR models, contributing to the advancement of this important field.
LookAhead Tuning: Safer Language Models via Partial Answer Previews
Authors: name: Kangwei Liu, name: Mengru Wang, name: Yujie Luo, name: Lin Yuan, name: Mengshu Sun, name: Ningyu Zhang, name: Lei Liang, name: Zhiqiang Zhang, name: Jun Zhou, name: Huajun Chen
Published: 2025-03-24 18:11:42

Fine-tuning enables large language models (LLMs) to adapt to specific
domains, but often undermines their previously established safety alignment. To
mitigate the degradation of model safety during fine-tuning, we introduce
LookAhead Tuning, which comprises two simple, low-resource, and effective
data-driven methods that modify training data by previewing partial answer
prefixes. Both methods aim to preserve the model's inherent safety mechanisms
by minimizing perturbations to initial token distributions. Comprehensive
experiments demonstrate that LookAhead Tuning effectively maintains model
safety without sacrificing robust performance on downstream tasks. Our findings
position LookAhead Tuning as a reliable and efficient solution for the safe and
effective adaptation of LLMs. Code is released at
https://github.com/zjunlp/LookAheadTuning.
When Words Outperform Vision: VLMs Can Self-Improve Via Text-Only
  Training For Human-Centered Decision Making
Authors: name: Zhe Hu, name: Jing Li, name: Yu Yin
Published: 2025-03-21 09:25:23

Embodied decision-making is fundamental for AI agents operating in real-world
environments. While Visual Language Models (VLMs) have advanced this
capability, they still struggle with complex decisions, particularly in
human-centered situations that require deep reasoning about human needs and
values. In this study, we systematically evaluate open-sourced VLMs on
multimodal human-centered decision-making tasks. We find that LLMs receiving
only textual descriptions unexpectedly outperform their VLM counterparts of
similar scale that process actual images, suggesting that visual alignment may
hinder VLM abilities. To address this challenge, we propose a novel text-only
training approach with synthesized textual data. This method strengthens VLMs'
language components and transfers the learned abilities to multimodal
inference, eliminating the need for expensive image-text paired data.
Furthermore, we show that VLMs can achieve substantial performance gains
through self-improvement, using training data generated by their LLM
counterparts rather than relying on larger teacher models like GPT-4. Our
findings establish a more efficient and scalable approach to enhancing VLMs'
human-centered decision-making capabilities, opening new avenues for optimizing
VLMs through self-improvement mechanisms.
Gumbel-Softmax Flow Matching with Straight-Through Guidance for
  Controllable Biological Sequence Generation
Authors: name: Sophia Tang, name: Yinuo Zhang, name: Alexander Tong, name: Pranam Chatterjee
Published: 2025-03-21 17:59:43

Flow matching in the continuous simplex has emerged as a promising strategy
for DNA sequence design, but struggles to scale to higher simplex dimensions
required for peptide and protein generation. We introduce Gumbel-Softmax Flow
and Score Matching, a generative framework on the simplex based on a novel
Gumbel-Softmax interpolant with a time-dependent temperature. Using this
interpolant, we introduce Gumbel-Softmax Flow Matching by deriving a
parameterized velocity field that transports from smooth categorical
distributions to distributions concentrated at a single vertex of the simplex.
We alternatively present Gumbel-Softmax Score Matching which learns to regress
the gradient of the probability density. Our framework enables high-quality,
diverse generation and scales efficiently to higher-dimensional simplices. To
enable training-free guidance, we propose Straight-Through Guided Flows
(STGFlow), a classifier-based guidance method that leverages straight-through
estimators to steer the unconditional velocity field toward optimal vertices of
the simplex. STGFlow enables efficient inference-time guidance using
classifiers pre-trained on clean sequences, and can be used with any discrete
flow method. Together, these components form a robust framework for
controllable de novo sequence generation. We demonstrate state-of-the-art
performance in conditional DNA promoter design, sequence-only protein
generation, and target-binding peptide design for rare disease treatment.
Global-Local Tree Search for Language Guided 3D Scene Generation
Authors: name: Wei Deng, name: Mengshi Qi, name: Huadong Ma
Published: 2025-03-24 09:21:13

Large Vision-Language Models (VLMs), such as GPT-4, have achieved remarkable
success across various fields. However, there are few studies on 3D indoor
scene generation with VLMs. This paper considers this task as a planning
problem subject to spatial and layout common sense constraints. To solve the
problem with a VLM, we propose a new global-local tree search algorithm.
Globally, the method places each object sequentially and explores multiple
placements during each placement process, where the problem space is
represented as a tree. To reduce the depth of the tree, we decompose the scene
structure hierarchically, i.e. room level, region level, floor object level,
and supported object level. The algorithm independently generates the floor
objects in different regions and supported objects placed on different floor
objects. Locally, we also decompose the sub-task, the placement of each object,
into multiple steps. The algorithm searches the tree of problem space. To
leverage the VLM model to produce positions of objects, we discretize the
top-down view space as a dense grid and fill each cell with diverse emojis to
make to cells distinct. We prompt the VLM with the emoji grid and the VLM
produces a reasonable location for the object by describing the position with
the name of emojis. The quantitative and qualitative experimental results
illustrate our approach generates more plausible 3D scenes than
state-of-the-art approaches. Our source code is available at
https://github.com/dw-dengwei/TreeSearchGen .
Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated
  Data Refinement Using Contrastive Learning
Authors: name: Sherry X. Chen, name: Misha Sra, name: Pradeep Sen
Published: 2025-03-24 07:25:44

Although natural language instructions offer an intuitive way to guide
automated image editing, deep-learning models often struggle to achieve
high-quality results, largely due to challenges in creating large, high-quality
training datasets. Previous work has typically relied on text-toimage (T2I)
generative models to produce pairs of original and edited images that simulate
the input/output of an instruction-guided image-editing model. However, these
image pairs often fail to align with the specified edit instructions due to the
limitations of T2I models, which negatively impacts models trained on such
datasets. To address this, we present Instruct-CLIP, a self-supervised method
that learns the semantic changes between original and edited images to refine
and better align the instructions in existing datasets. Furthermore, we adapt
Instruct-CLIP to handle noisy latent images and diffusion timesteps so that it
can be used to train latent diffusion models (LDMs) [19] and efficiently
enforce alignment between the edit instruction and the image changes in latent
space at any step of the diffusion pipeline. We use Instruct-CLIP to correct
the InstructPix2Pix dataset and get over 120K refined samples we then use to
fine-tune their model, guided by our novel Instruct-CLIP-based loss function.
The resulting model can produce edits that are more aligned with the given
instructions. Our code and dataset are available at
https://github.com/SherryXTChen/Instruct-CLIP.git.
QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on
  the Edge
Authors: name: Xuan Shen, name: Weize Ma, name: Jing Liu, name: Changdi Yang, name: Rui Ding, name: Quanyi Wang, name: Henghui Ding, name: Wei Niu, name: Yanzhi Wang, name: Pu Zhao, name: Jun Lin, name: Jiuxiang Gu
Published: 2025-03-20 21:03:10

Monocular Depth Estimation (MDE) has emerged as a pivotal task in computer
vision, supporting numerous real-world applications. However, deploying
accurate depth estimation models on resource-limited edge devices, especially
Application-Specific Integrated Circuits (ASICs), is challenging due to the
high computational and memory demands. Recent advancements in foundational
depth estimation deliver impressive results but further amplify the difficulty
of deployment on ASICs. To address this, we propose QuartDepth which adopts
post-training quantization to quantize MDE models with hardware accelerations
for ASICs. Our approach involves quantizing both weights and activations to
4-bit precision, reducing the model size and computation cost. To mitigate the
performance degradation, we introduce activation polishing and compensation
algorithm applied before and after activation quantization, as well as a weight
reconstruction method for minimizing errors in weight quantization.
Furthermore, we design a flexible and programmable hardware accelerator by
supporting kernel fusion and customized instruction programmability, enhancing
throughput and efficiency. Experimental results demonstrate that our framework
achieves competitive accuracy while enabling fast inference and higher energy
efficiency on ASICs, bridging the gap between high-performance depth estimation
and practical edge-device applicability. Code:
https://github.com/shawnricecake/quart-depth
Mind with Eyes: from Language Reasoning to Multimodal Reasoning
Authors: name: Zhiyu Lin, name: Yifei Gao, name: Xian Zhao, name: Yunfan Yang, name: Jitao Sang
Published: 2025-03-23 13:40:44

Language models have recently advanced into the realm of reasoning, yet it is
through multimodal reasoning that we can fully unlock the potential to achieve
more comprehensive, human-like cognitive capabilities. This survey provides a
systematic overview of the recent multimodal reasoning approaches, categorizing
them into two levels: language-centric multimodal reasoning and collaborative
multimodal reasoning. The former encompasses one-pass visual perception and
active visual perception, where vision primarily serves a supporting role in
language reasoning. The latter involves action generation and state update
within reasoning process, enabling a more dynamic interaction between
modalities. Furthermore, we analyze the technical evolution of these methods,
discuss their inherent challenges, and introduce key benchmark tasks and
evaluation metrics for assessing multimodal reasoning performance. Finally, we
provide insights into future research directions from the following two
perspectives: (i) from visual-language reasoning to omnimodal reasoning and
(ii) from multimodal reasoning to multimodal agents. This survey aims to
provide a structured overview that will inspire further advancements in
multimodal reasoning research.
Human Motion Unlearning
Authors: name: Edoardo De Matteis, name: Matteo Migliarini, name: Alessio Sampieri, name: Indro Spinelli, name: Fabio Galasso
Published: 2025-03-24 13:46:27

We introduce the task of human motion unlearning to prevent the synthesis of
toxic animations while preserving the general text-to-motion generative
performance. Unlearning toxic motions is challenging as those can be generated
from explicit text prompts and from implicit toxic combinations of safe motions
(e.g., ``kicking" is ``loading and swinging a leg"). We propose the first
motion unlearning benchmark by filtering toxic motions from the large and
recent text-to-motion datasets of HumanML3D and Motion-X. We propose baselines,
by adapting state-of-the-art image unlearning techniques to process
spatio-temporal signals. Finally, we propose a novel motion unlearning model
based on Latent Code Replacement, which we dub LCR. LCR is training-free and
suitable to the discrete latent spaces of state-of-the-art text-to-motion
diffusion models. LCR is simple and consistently outperforms baselines
qualitatively and quantitatively. Project page:
https://www.pinlab.org/hmu{https://www.pinlab.org/hmu}.
Revisiting Image Fusion for Multi-Illuminant White-Balance Correction
Authors: name: David Serrano-Lozano, name: Aditya Arora, name: Luis Herranz, name: Konstantinos G. Derpanis, name: Michael S. Brown, name: Javier Vazquez-Corral
Published: 2025-03-18 23:01:22

White balance (WB) correction in scenes with multiple illuminants remains a
persistent challenge in computer vision. Recent methods explored fusion-based
approaches, where a neural network linearly blends multiple sRGB versions of an
input image, each processed with predefined WB presets. However, we demonstrate
that these methods are suboptimal for common multi-illuminant scenarios.
Additionally, existing fusion-based methods rely on sRGB WB datasets lacking
dedicated multi-illuminant images, limiting both training and evaluation. To
address these challenges, we introduce two key contributions. First, we propose
an efficient transformer-based model that effectively captures spatial
dependencies across sRGB WB presets, substantially improving upon linear fusion
techniques. Second, we introduce a large-scale multi-illuminant dataset
comprising over 16,000 sRGB images rendered with five different WB settings,
along with WB-corrected images. Our method achieves up to 100\% improvement
over existing techniques on our new multi-illuminant image fusion dataset.
Rethinking Image Evaluation in Super-Resolution
Authors: name: Shaolin Su, name: Josep M. Rocafort, name: Danna Xue, name: David Serrano-Lozano, name: Lei Sun, name: Javier Vazquez-Corral
Published: 2025-03-17 11:25:48

While recent advancing image super-resolution (SR) techniques are continually
improving the perceptual quality of their outputs, they can usually fail in
quantitative evaluations. This inconsistency leads to a growing distrust in
existing image metrics for SR evaluations. Though image evaluation depends on
both the metric and the reference ground truth (GT), researchers typically do
not inspect the role of GTs, as they are generally accepted as `perfect'
references. However, due to the data being collected in the early years and the
ignorance of controlling other types of distortions, we point out that GTs in
existing SR datasets can exhibit relatively poor quality, which leads to biased
evaluations. Following this observation, in this paper, we are interested in
the following questions: Are GT images in existing SR datasets 100% trustworthy
for model evaluations? How does GT quality affect this evaluation? And how to
make fair evaluations if there exist imperfect GTs? To answer these questions,
this paper presents two main contributions. First, by systematically analyzing
seven state-of-the-art SR models across three real-world SR datasets, we show
that SR performances can be consistently affected across models by low-quality
GTs, and models can perform quite differently when GT quality is controlled.
Second, we propose a novel perceptual quality metric, Relative Quality Index
(RQI), that measures the relative quality discrepancy of image pairs, thus
issuing the biased evaluations caused by unreliable GTs. Our proposed model
achieves significantly better consistency with human opinions. We expect our
work to provide insights for the SR community on how future datasets, models,
and metrics should be developed.
Feather-SQL: A Lightweight NL2SQL Framework with Dual-Model
  Collaboration Paradigm for Small Language Models
Authors: name: Wenqi Pei, name: Hailing Xu, name: Hengyuan Zhao, name: Shizheng Hou, name: Han Chen, name: Zining Zhang, name: Pingyi Luo, name: Bingsheng He
Published: 2025-03-22 16:22:53

Natural Language to SQL (NL2SQL) has seen significant advancements with large
language models (LLMs). However, these models often depend on closed-source
systems and high computational resources, posing challenges in data privacy and
deployment. In contrast, small language models (SLMs) struggle with NL2SQL
tasks, exhibiting poor performance and incompatibility with existing
frameworks. To address these issues, we introduce Feather-SQL, a new
lightweight framework tailored for SLMs. Feather-SQL improves SQL executability
and accuracy through 1) schema pruning and linking, 2) multi-path and
multi-candidate generation. Additionally, we introduce the 1+1 Model
Collaboration Paradigm, which pairs a strong general-purpose chat model with a
fine-tuned SQL specialist, combining strong analytical reasoning with
high-precision SQL generation. Experimental results on BIRD demonstrate that
Feather-SQL improves NL2SQL performance on SLMs, with around 10% boost for
models without fine-tuning. The proposed paradigm raises the accuracy ceiling
of SLMs to 54.76%, highlighting its effectiveness.
CODA: Repurposing Continuous VAEs for Discrete Tokenization
Authors: name: Zeyu Liu, name: Zanlin Ni, name: Yeguo Hua, name: Xin Deng, name: Xiao Ma, name: Cheng Zhong, name: Gao Huang
Published: 2025-03-22 12:59:00

Discrete visual tokenizers transform images into a sequence of tokens,
enabling token-based visual generation akin to language models. However, this
process is inherently challenging, as it requires both compressing visual
signals into a compact representation and discretizing them into a fixed set of
codes. Traditional discrete tokenizers typically learn the two tasks jointly,
often leading to unstable training, low codebook utilization, and limited
reconstruction quality. In this paper, we introduce
CODA(COntinuous-to-Discrete Adaptation), a
framework that decouples compression and discretization. Instead of training
discrete tokenizers from scratch, CODA adapts off-the-shelf continuous VAEs --
already optimized for perceptual compression -- into discrete tokenizers via a
carefully designed discretization process. By primarily focusing on
discretization, CODA ensures stable and efficient training while retaining the
strong visual fidelity of continuous VAEs. Empirically, with 6
times less training budget than standard VQGAN, our approach achieves a
remarkable codebook utilization of 100% and notable reconstruction FID (rFID)
of 0.43 and 1.34 for 8 times and 16 times
compression on ImageNet 256times 256 benchmark.
DynamicVis: An Efficient and General Visual Foundation Model for Remote
  Sensing Image Understanding
Authors: name: Keyan Chen, name: Chenyang Liu, name: Bowen Chen, name: Wenyuan Li, name: Zhengxia Zou, name: Zhenwei Shi
Published: 2025-03-20 17:59:54

The advancement of remote sensing technology has improved the spatial
resolution of satellite imagery, facilitating more detailed visual
representations for diverse interpretations. However, existing methods exhibit
limited generalization capabilities across varied applications. While some
contemporary foundation models demonstrate potential, they are hindered by
insufficient cross-task adaptability and primarily process low-resolution
imagery of restricted sizes, thus failing to fully exploit high-resolution data
or leverage comprehensive large-scene semantics. Crucially, remote sensing
imagery differs fundamentally from natural images, as key foreground targets
(eg., maritime objects, artificial structures) often occupy minimal spatial
proportions (~1%) and exhibit sparse distributions. Efficiently modeling
cross-task generalizable knowledge from lengthy 2D tokens (~100,000) poses a
significant challenge yet remains critical for remote sensing image
understanding. Motivated by the selective attention mechanisms inherent to the
human visual system, we propose DynamicVis, a dynamic visual perception
foundation model for remote sensing imagery. The framework integrates a novel
dynamic region perception backbone based on the selective state space model,
which strategically balances localized detail extraction with global contextual
integration, enabling computationally efficient encoding of large-scale data
while maintaining architectural scalability. To enhance cross-task knowledge
transferring, we introduce a multi-instance learning paradigm utilizing
meta-embedding representations, trained on million-scale region-level
annotations. Evaluations across nine downstream tasks demonstrate the model's
versatility. DynamicVis achieves multi-level feature modeling with exceptional
efficiency, processing (2048x2048) pixels with 97 ms latency (6% of ViT's) and
833 MB GPU memory (3% of ViT's).