Talk in Pieces, See in Whole: Disentangling and Hierarchical Aggregating Representations for Language-based Object Detection

## Abstract

Abstract While vision-language models (VLMs) have made significant progress in multimodal perception (e.g., open-vocabulary object detection) with simple language queries, state-of-the-art VLMs still show limited ability to perceive complex queries involving descriptive attributes and relational clauses.
Our in-depth analysis shows that these limitations mainly stem from text encoders in VLMs.
Such text encoders behave like bags-of-words and fail to separate target objects from their descriptive attributes and relations in complex queries, resulting in frequent false positives.
To address this, we propose restructuring linguistic representations according to the hierarchical relations within sentences for language-based object detection.
A key insight is the necessity of disentangling textual tokens into core components—objects, attributes, and relations (“talk in pieces”)—and subsequently aggregating them into hierarchically structured sentence-level representations (“see in whole”).
Building on this principle, we introduce the TaSe framework with three main contributions: (1) a hierarchical synthetic captioning dataset spanning three tiers from category names to descriptive sentences; (2) Talk in Pieces , the three-component disentanglement module guided by a novel disentanglement loss function, transforms text embeddings into subspace compositions; and (3) See in Whole , which learns to aggregate disentangled components into hierarchically structured embeddings with the guide of proposed hierarchical objectives.
The proposed TaSe framework strengthens the inductive bias of hierarchical linguistic structures, resulting in fine-grained multimodal representations for language-based object detection.
Experimental results under the OmniLabel benchmark show a 24% performance improvement, demonstrating the importance of linguistic compositionality.

## 1 Introduction

Vision-language (VL) understanding, which aims to perceive each modality and form associations between them, is a long-standing and fundamental problem. Recently, foundational VLMs such as CLIP (Radford et al., 2021a) have leveraged web-scale image-text pairs to learn generic VL representations, achieving strong generalization performance on tasks like image classification and image-text retrieval. Building upon these advances, recent studies have actively explored grounding language queries into specific image regions (e.g., open-vocabulary object detection (Liu et al., 2024b; Zhao et al., 2024; Yin et al., 2025)). Many approaches (Liu et al., 2024a; Li et al., 2022) distill the general VL knowledge embedded in foundational models into object detectors and have demonstrated remarkable results in detecting previously unseen object categories—commonly referred to as open-vocabulary object detection (Gu et al., 2021).

Figure: (a) Detection results from w/o and w/ our hierarchy entailment
Refer to caption: x1.png

Despite these advances, current VL detectors often succeed only when the input queries are short and consist of simple category names. They still struggle to fully comprehend complex language queries and accurately localize the corresponding objects. To illustrate this limitation, we conduct a preliminary analysis using the state-of-the-art foundation model for visual grounding, GLEE (Wu et al., 2024) (see Fig. [1(a)](https://arxiv.org/html/2509.24192v1#S1.F1.sf1)). The model reliably detects objects given simple noun phrases (e.g., “segway”). However, it fails when faced with more complex and specific queries (e.g., “segway with a man”), indicating its limited compositional understanding.

To investigate the underlying cause of this limitation, we visualize sentence-level text features using t-SNE (Van der Maaten & Hinton, 2008) in Fig. [1(b)](https://arxiv.org/html/2509.24192v1#S1.F1.sf2). Interestingly, we observe that although some sentences (“a segway with a man” vs. “a segway with a woman”) refer to different target objects (“with a man” vs. “with a woman”), their embeddings remain close in the feature space due to shared tokens (“segway”).
The contrasts with the ideal cases, where embeddings of distinct target objects should be well-separated, while those of the same object with different attributes should be closer for object detection (“a segway” vs. “a black segway”).
These results indicate limited hierarchical and compositional understanding in current VLMs (Yuksekgonul et al., 2022).
Most VL detectors (Liu et al., 2024a; Li et al., 2022) are trained to align a few positives with image regions while distinguishing them from negatives using contrastive learning.
For example, training with augmented captions (Li et al., 2023b; Yuksekgonul et al., 2022) labeled as positive or negative is effective for coarse-grained concept analysis.
Still, detectors often struggle to handle tasks that require understanding of fine-grained text, such as reasoning over interactions between objects (e.g., “bigger than”).
Sentence embeddings obtained via simple pooling compress token-level information and fail to capture contextualization in language queries.
Beyond augmentation, sentence embeddings need to disentangle text tokens and encode compositional features.
We argue that VL detectors should go further: representations need to see the whole sentence from meaningful pieces.

In this paper, we propose a novel framework that disentangles component-wise text features (“Talk in Pieces”) and explicitly learns hierarchical knowledge (“See in Whole”) from these disentangled representations to construct sentence-level understanding for language-based object detection.
We refer to our framework as TaSe (Talk in Pieces, See in Whole).
We begin by formally defining a hierarchical structure tailored for language-based object detection.
Specifically, we design our new HiVG dataset, a three-tiered hierarchy, object–attribute–relation, where the first tier represents object category names, the second tier adds descriptive attributes, and the last tier includes relational phrases that describe interactions or contexts.
Our approach builds on phrase grounding datasets like Visual Genome (Krishna et al., 2017), which provide densely annotated phrases associated with images and object regions. Using a large language model (LLM) (Dubey et al., 2024), we abstract these phrases into a three-tier hierarchy—object, attribute, relation—by sequentially removing relational and attribute information in phrases to obtain the final object categories.
Unlike typical generation-based approaches (e.g., generating sentences from category names (Li et al., 2023b) or captions from images), our abstraction-based process performs reverse abstraction, effectively mitigating hallucination issues (Ji et al., 2023) common in generative models.

To effectively construct contextualized (i.e., holistic) sentence representations from the HiVG dataset, we disentangle into several key aspects (“Talk in Pieces”)—such as objects, attributes, and relations.
This design allows us to disentangle text representations into subspaces to adjust targeted token embeddings and preserve meaningful information in the remaining features.
For this purpose, we further design a lightweight learnable attention module for the TriDe (Three-component disentanglement), enabling efficient fine-tuning of conventional text encoders.
The key idea of TriDe is to leverage the hierarchical structure of the HiVG dataset to contrast component-wise tokens so that targeted tokens to be adjusted without loss of meaningful information.

Then, we guide the model to learn linguistic representations that capture these levels of abstraction.
This facilitates learning of sentence context enriched with descriptive attributes and relational clauses.
We introduce a hierarchical aggregation method (“See in Whole”) based on sentence-level hierarchy entailment, which effectively models sentence-level hierarchical relationships with our HiVG. Our learning hierarchical objective offers a richer and more structured alternative to naïve contrastive learning, which typically aligns image regions with positive tokens in a sentence while contrasting them with negative tokens. In contrast, our method models the full sentence hierarchy, promoting a more dense VL understanding.

To summarize, our main contributions are as follows:
1) We present an efficient hierarchical data generation pipeline that abstracts dense existing phrases into an explicit hierarchical structure of “object–attribute–relation.” 2) We introduce a novel framework for disentangling the three core components and employ the TriDe loss to guide this process. 3) We propose a method for learning disentangled and hierarchical representations that capture sentence-level inductive biases and can be integrated into conventional VL detectors. With hierarchical learning on our generated dataset HiVG, our model significantly outperforms strong baselines, including state-of-the-art VL detectors, on challenging language-based object detection benchmarks such as OmniLabel (Schulter et al., 2023) and D3 (Xie et al., 2023).

## 2 Related Works

### 2.1 Language-based Object Detection

Language-based object detection aims to locate and identify objects in images using free-form text.
One of the leading approaches is to transfer the pre-trained model and align images and texts using contrastive learning (Li et al., 2022; 2023b; Park et al., 2024).
Contrastive learning enhances compositionality in VLMs by capturing relationships with contextual entities and improves the understanding of object relationships (Li et al., 2023b; Minderer et al., 2022a; Gu et al., 2021; Gao et al., 2024; Liu et al., 2024b).
GLIP (Li et al., 2022) proposes to add deep fusion layers between different modalities and learn a language-aware visual representation based on reformulated alignment scores.

However, existing approaches overlook the need for contextualized sentence-level understanding of VL text embedding.
For example, APE (Shen et al., 2024); GLEE (Wu et al., 2024); Dino-x (Ren et al., 2024); and Zeng et al. (2024) explore VLM alignment challenges and highlight the need to improve reasoning capabilities in multimodal LLMs.
These works investigate model capabilities from restricted VL perspectives, with a primary focus on fine-grained textual details and inter-object relationships.
VL detectors still struggle to align images with syntactically intricate language queries (Wang et al., 2023), underscoring the need for a more grounded contextual understanding of text.

Disentangled representation learning is a method for enhancing linguistic understanding by learning fine-grained representations (Bengio et al., 2013; Wang et al., 2024).
Several approaches have been proposed for disentangled representation learning, including prompt learning (Lu et al., 2023; Zheng et al., 2024), learnable vectors (Qi et al., 2024), and attention-based mechanisms (Wu et al., 2025).
Prior works have introduced methods for designing object–attribute compositions, which improve compositional zero-shot learning.
In contrast to these concept-aware approaches that disentangle objects and attributes for recomposition, our method leverages disentanglement to capture hierarchical sentence structures and contextualized understanding.

### 2.2 Hierarchical Entailment for Vision-Language Models

To better represent the embedding space of VLMs, hyperbolic learning has highlighted the need to capture hierarchical structures and relationships in multimodal data.
Hyperbolic learning was formulated on the Poincaré ball by Ganea et al. (2018), learning entailment relations between embedded objects.
The formulation now extends the more common Lorentz model as Lou et al. (2020) due to its computationally heavy Gyrovector operations.
Hyperbolic learning maps the embedding into an entailment cone (EC) to represent hierarchical entailment in a continuous space.
Recent studies investigated the use of the EC embedding for vision tasks (Atigh et al., 2022; Kong et al., 2024; Khrulkov et al., 2020), multimodal learning Desai et al. (2023); Hong et al. (2023); Pal et al. (2024), and synthetic data generation (Kong et al., 2024).

However, the hyperbolic manifold needs to transpose features from Euclidean to hyperbolic and requires additional hyperparameter configurations.
To address this limitation, Alper & Averbuch-Elor (2024) proposed radial embedding (RE) optimization for learning hierarchical representations directly in Euclidean space.
Inspired by this approach, we extend RE optimization to language-based object detection based on hierarchical representation learning at the sentence level.
While previous works explore hierarchical manifolds to capture natural hierarchy (Lang et al., 2022), sentence-level hierarchy objectives remain underexplored.
This work introduces a hierarchical modeling approach to define the sentence-level hierarchy entailment with compositional learning, which captures inclusive relationships between hierarchy nodes in language-based object detection.

## 3 TaSe: Disentangled and Hierarchical Text Representation Learning for Language-based Object Detection

This section introduces TaSe, a framework for disentangling and hierarchy aggregating method.
Specifically, our approach comprises three components: 1) the HiVG dataset (Sec. [3.1](https://arxiv.org/html/2509.24192v1#S3.SS1)), a synthetic dataset re-captioned from VG to capture hierarchical entailment relations; 2) disentangling text representations into objects, attributes, and relations for a component-wise subspace for aligning semantic pieces within sentences (Sec. [3.2](https://arxiv.org/html/2509.24192v1#S3.SS2)); and 3) a hierarchical aggregation method to represent contextualized sentence embedding based on disentangled tokens (Sec. [3.3](https://arxiv.org/html/2509.24192v1#S3.SS3)).
Fig. [2](https://arxiv.org/html/2509.24192v1#S3.F2) outlines the TaSe to learn contextualized sentence representations within language-based object detection.

### 3.1 HiVG: Hierarchy Captioning Pipeline

Although augmented captions enhance fine-grained textual representations (Li et al., 2023b; Yuksekgonul et al., 2022), open-vocabulary detectors often rely on keywords and fail to separate target objects from their attributes and relations, owing to the absence of hard textual negatives that reflect linguistic hierarchy.
To address this problem, we propose a Hierarchical captioning pipeline that re-captions the Visual Genome dataset (HiVG) by leveraging pre-trained LLMs and lexical databases (e.g., WordNet (Miller, 1995) and ConceptNet (Speer et al., 2017)).
HiVG is a synthetic dataset constructed by spanning from category names to descriptive sentences and structuring hierarchical captions into three tiers: objects, attributes, and relations.
Each caption in the Visual Genome (Krishna et al., 2017) annotation is transformed into three positive ($e^{+}$) and negative tiers ($e^{-}$) where $e$ follows the notation introduced in Sec. [3.3](https://arxiv.org/html/2509.24192v1#S3.SS3). We show an example for the input image in Fig. [2](https://arxiv.org/html/2509.24192v1#S3.F2).

- •
Tier 1. Category names (object): containing the class name (e.g., woman ($e_{1}^{+}$) and man ($e_{1}^{-}$)).
- •
Tier 2. Enriched descriptions (w/ attribute): adding an attribute to the object (e.g., middle woman ($e_{2}^{+}$) and left woman ($e_{2}^{-}$) for learning fine-grained linguistic compositionality).
- •
Tier 3. Contextual understanding (w/ attribute and relation): emphasizing the relationships between objects by injecting a relation into the second-tier caption (e.g., middle woman with dark hair ($e_{3}^{+}$) and with red shirt ($e_{3}^{-}$)).

Further details of our re-captioning approach and examples are provided in the supplementary material (see Sec. [A](https://arxiv.org/html/2509.24192v1#A1), Fig. [13](https://arxiv.org/html/2509.24192v1#A4.F13)).

### 3.2 Talk in Piece: Component-wise Text Disentanglement

Textual descriptions typically contain not only descriptive attributes but also complex relational structures, which cause false positives in language-based object detection.
To address this, we propose the TriDe module to disentangle text embeddings into meaningful subspaces, which adaptively refines these components to enhance semantic representation.

Figure: Figure 2: The overall framework of TaSe. (Left) The text encoder is fine-tuned with LoRA (Hu et al., 2021) and the TriDe module to restructure text representations. (Right) Overview of the TriDe module and its embedding space. Top: Hierarchy aggregated embeddings with HiVG, where the recaptioned dataset passes through the TriDe module to learn linguistic hierarchy. Bottom: Architecture of the TriDe module.
Refer to caption: x3.png

Text embedding.
We extract text features by CLIP text encoder with low-rank adaptation (LoRA) (Hu et al., 2021) for efficiently evolving text embedding from the text encoder.
Let $\{v_{i},t_{i}\}_{i=1}^{B}$ be a batch of image–text pairs.
The text embedding $\mathbf{X}=\mathcal{T}_{\theta}(\text{text})$, where $\mathrm{X}\in\mathbb{R}^{B\times T\times d_{\text{model}}}$, is obtained using the text encoder.
Here, $B$, $T$, and $d_{model}$ denote the batch size, number of tokens, and embedding dimension, respectively.
A text projection layer maps the input into $\mathrm{X}\in\mathbb{R}^{B\times T\times D}$, where $D$ denotes the text embedding dimension.

Figure: Figure 3: Learning process for hierarchically structured positive and negative sentences. The model is trained with contrastive learning on HiVG, where text features are disentangled into subspaces across tiers and optimized with cosine distance.
Refer to caption: x4.png

Component-wise disentanglement.
We disentangle text representations into three components—objects, attributes, and relations.
This design mirrors the three-tier structure of HiVG and facilitates the learning of effective contextualized sentence embeddings.
We adjust learnable vectors $\mathrm{V_{O}},\mathrm{V_{A}},\mathrm{V_{R}}\in\mathbb{R}^{B\times T\times D}$ to disentangle the text embedding into the three components.
We employ a multi-head cross-attention layer between the learnable vectors and text embedding $\mathrm{X}$.
Let FFN, LN, and Proj denote the feed-forward network, layer normalization, and projection layer, respectively.
The TriDe module is defined as follows:

$$ $\begin{array}[]{c}\mathbf{X}=\text{LN}(\text{Proj}(\mathrm{X}+\text{FFN}(\mathrm{X}))),\\ \left[\text{O},\text{A},\text{R}\right]=\text{CrossAttn}(\mathrm{X},\left[\mathrm{V}_{\text{O}},\mathrm{V}_{\text{A}},\mathrm{V}_{\text{R}}\right]),\\ \mathbf{E}=\text{pooling}(\text{FFN}(\text{LN}(\mathrm{O}+\mathrm{A}+\mathrm{R}))),\\ \end{array}$ (1) $$

where $\mathrm{O}$, $\mathrm{A}$, $\mathrm{R}$, and $\mathrm{E}$ represent the object, attribute, relation components, and restructured text embedding.
Note that $\mathrm{E}$ is employed to learn hierarchical entailment for contextualized sentence representations in Sec. [3.3](https://arxiv.org/html/2509.24192v1#S3.SS3).
The CrossAttn($\mathrm{Q},\mathrm{V}$) with the scaling factor $d_{k}$ is defined as follows:

$$ $\begin{array}[]{c}\text{CrossAttn}(\mathrm{Q},\mathrm{V})=\text{Softmax}\Big(\dfrac{\text{Proj}(\mathrm{Q})\text{Proj}(\mathrm{V})^{T}}{\sqrt{d_{k}}}\Big)\text{Proj}(\mathrm{V}).\end{array}$ (2) $$

Objective for component disentanglement.
In Fig. [3](https://arxiv.org/html/2509.24192v1#S3.F3), components are aligned with their positive counterparts, while negatives are enforced to remain distant according to their tier.
Let $t$ be the tier of HiVG, and the disentanglement of text embedding is adjusted $\mathcal{L}_{\text{TriDe}}$ as follows:

$$ $\begin{array}[]{c}\mathcal{L}_{\text{TriDe}}=\lambda\sum_{(i,j)\in\{(\text{O},\text{A}),(\text{O},\text{R}),(\text{A},\text{R})}\}\big|\mathbf{i}\cdot\mathbf{j}\big|+\sum_{t=1}^{l}\Big(m+\cos(\text{O}^{+}_{t},\text{O}^{+}_{t+1})-\cos(\text{O}^{+}_{t},\text{O}^{-}_{t})\Big)\\ +\sum_{t=2}^{l}\Big(m+\cos(\text{A}^{+}_{t},\text{A}^{+}_{t+1})-\cos(\text{A}^{+}_{t},\text{A}^{-}_{t})\Big)+\Big(m-\cos(\text{R}^{+}_{l},\text{R}^{-}_{l})\Big).\\ \end{array}$ (3) $$

$\mathcal{L}_{\text{TriDe}}$ and $m$ represent compositional loss for the TriDe module and margin, respectively.
$\lambda$ represents a hyperparameter for stability adjustment.
Minimizing correlation among components promotes an inductive bias toward semantically grounded object representations.

### 3.3 See In Whole: Sentence-level Hierarchical Aggregation

We design a hierarchy aggregation method of disentangled features to serve fine-grained semantic distinction, which helps capture contextual meaning beyond simple word-level perception.

Background: hierarchical entailment in Euclidean space.
The goal of hierarchical entailment is to learn general concepts by representing entailment relations via low-dimensional embeddings (Ganea et al., 2018).
While conventional contrastive learning to learn embedding from pair-wise (i.e., positive and negative), the RE objective (Alper & Averbuch-Elor, 2024) aims to represent the hierarchy structure of embedding by exterior angle with respect to the reference point.
Two key advantages of this representation learning with the RE are: 1) modeling sentence-level structure; and 2) learning compositional generalization without requiring transformation into sphere space.
Let $\Xi$ denote the exterior angle in radians and $r\in\mathbb{R}^{d}$ denote a root embedding, the exterior angle between embedding $a$ and $b$ in the RE objective is defined:

$$ $\begin{array}[]{c}\Xi\langle a,b\rangle=\cos^{-1}\left(\frac{\mathbf{a^{\prime}}\cdot\mathbf{b^{\prime}}}{\|\mathbf{a^{\prime}}\|\|\mathbf{b^{\prime}}\|}\right)\leq\pi.\\ \text{where}\quad\mathbf{a^{\prime}}=\mathbf{a}-r,\quad\mathbf{b^{\prime}}=\mathbf{b}-r-\mathbf{a^{\prime}},\quad\mathbf{a},\mathbf{b}\in\mathbb{R}^{d}\end{array}$ (4) $$

The value $\langle a,b\rangle_{\Xi}\in[0,\pi]$ is bounded.
Given a root embedding $r$ and a reference embedding of a text embedding $e$, the objective function of the RE is represented as follows:

$$ $\mathcal{L}_{RE}=\sum_{i}\left(\langle e_{i}^{+},e_{>i}^{+}\rangle_{\Xi}-\langle e_{i}^{+},e_{i}^{-}\rangle_{\Xi}\right).\vskip-10.0pt$ (5) $$

Here, $e_{i}^{+}$ and $e_{>i}^{+}$ denote distinct positive embeddings, and $e_{i}^{-}$ is a corresponding negative.
The objective $\mathcal{L}_{RE}$ encourages smaller exterior angles between positive pairs while enforcing larger angles between positive and negative pairs.
This deviation reflects a misalignment from the reference anchor $r$ (frozen) and corresponds to a larger angular distance in the embedding space.

Figure: Figure 4: Limitation of the RE objective. (left) Solid - ideal angle, Dotted - distortion from larger upper-tier embedding. (right) Red - positive pairs, blue - negative pairs; with the root fixed, the “left woman” represents a trade-off.
Refer to caption: x5.png

Reference-based hierarchy induction.
The previous approach still has challenges to enhance compositional generalization from the perspective of sentence-level hierarchy entailment (see Fig. [4](https://arxiv.org/html/2509.24192v1#S3.F4)).
First, we introduce a regularization term to preserve the inherent embedding space, defining two objective functions for hierarchy entailment.
Since angular-only supervision may induce directional bias and degrade representational fidelity, we regularize the embedding space to retain knowledge.
Second, contrastive learning is formulated using exterior angles, where sentence pairs are trained with respect to a reference-conditioned point.
A key consideration in sentence-level contrastive learning with a text encoder is that a negative $e_{t}^{-}$ is also positive with respect to the previous tier $e_{t-1}^{+}$.
To account for this, we position samples in opposing directions based on a dynamic reference $r$ rather than a fixed root (see Eq. [6](https://arxiv.org/html/2509.24192v1#S3.E6)).

Hierarchical objectives.
Let $l$ denote the tier of HiVG in the object-attribute-relation hierarchy.
Let $e_{t}$ be an embedding at tier $t$ of HiVG, $e_{t}^{+}$ its positive counterpart at tier $l$, and $e_{t}^{-}$ a negative sample at the same tier.
We use two complementary losses—(i) alignment across tiers and (ii) within-tier discrimination—defined compactly as

$$ $\displaystyle\mathcal{L}_{\text{H}^{+}}$ $\displaystyle=\textstyle\sum_{t=1}^{l}\ \Xi\langle e_{t}^{+},e_{t+1}^{+}\rangle+\Xi\langle e_{t}^{+},e_{t+1}^{-}\rangle,\quad b^{\prime}=(b-r)-a^{\prime},$ (6) $\displaystyle\mathcal{L}_{\text{H}^{-}}$ $\displaystyle=\textstyle\sum_{t=1}^{l}\ \Xi\langle e_{t}^{+},e_{t}^{-}\rangle,\quad b^{\prime}=(r-b)-a^{\prime},\quad r=$ $$

with normalization $e=\frac{e-r}{\|e-r\|+\epsilon},\quad e\in\{e^{+},e^{-}\}$.
Note that normalization maintains directional consistency independent of embedding scale.
$r$ is adapted to expose directional differences not captured by global alignment and to account for locally meaningful variation.
The proposed formulation minimizes directional deviation from the reference point, which helps preserve the intrinsic structure of the embedding space while enforcing separation between negative pairs.

Training objectives.
For learning contextualized features, the objective function of hierarchical and disentangled representation learning, $\mathcal{L}_{\text{TaSe}}$, is defined as:

$$ $\mathcal{L}_{\text{TaSe}}=\mathcal{L}_{\text{TriDe}}+\mathcal{L}_{\text{H}^{+}}+\mathcal{L}_{\text{H}^{-}}.$ (7) $$

The loss function is formulated as a weighted sum of classification, localization, and hierarchy losses updated on the text encoder and VL fusion layers as follows:

$$ $\mathcal{L}=\mathcal{L}_{class}+\mathcal{L}_{bbox}+\mathcal{L}_{giou}+\mathcal{L}_{\text{TaSe}},$ (8) $$

where $\mathcal{L}_{class}$ represents Focal loss (Lin et al., 2017), $\mathcal{L}_{bbox}$ represents L1 loss, and $\mathcal{L}_{giou}$ represents generalized intersection over union (GIoU) loss (Rezatofighi et al., 2019).

## 4 Experiments

This section compares our method with baselines.
The following sections provide the implementation details (Sec. [B](https://arxiv.org/html/2509.24192v1#A2)), the main results for performance comparison (Sec. [4.2](https://arxiv.org/html/2509.24192v1#S4.SS2)), and ablation studies conducted to analyze the results in three benchmark datasets (Sec. [4.3](https://arxiv.org/html/2509.24192v1#S4.SS3)).
Additional experimental details can be found in the Sec [G](https://arxiv.org/html/2509.24192v1#A7) of the supplementary material.
The key findings of this study are as follows:
1) sentence-level hierarchical supervision enhances VL alignment by improving linguistic compositionality (Tab. [3](https://arxiv.org/html/2509.24192v1#S4.T3));
2) disentangling components with hierarchical structures leads to better modeling of the inductive biases of sentences (Tab. [3](https://arxiv.org/html/2509.24192v1#S4.T3)); and
3) compositional structure improves the discrimination of positive and negative pairs to represent descriptive sentences better (Tab. [1](https://arxiv.org/html/2509.24192v1#S4.T1)).

### 4.1 Experimental settings

Implementation details.
We build our method based on GLEE (Wu et al., 2024), a pre-trained foundation model composed of MaskDINO (Li et al., 2023a) and CLIP (Radford et al., 2021b) text-image encoders.
GLEE was selected as a baseline because, despite being a powerful vision–language foundation model in many benchmarks (e.g., RefCOCO (Yu et al., 2016)), it still faces challenges in contextualizing text embeddings.
This study demonstrates that a lightweight hierarchy entailment mechanism can address this limitation and yield further performance gains.
For implementations, we use only HiVG dataset for training, which contains 10 K hierarchy captions.
We provide more details of the experimental settings in Sec. [B](https://arxiv.org/html/2509.24192v1#A2) of the supplementary materials.

Benchmarks and evaluation metrics.
We evaluate the language-based object detection capabilities in two different benchmarks.
1) D^3 (Xie et al., 2023) dataset is a widely used benchmark for visual grounding tasks.
The dataset includes negative instances, multi-target scenarios, and long sentences.
2) Omnilabel (Schulter et al., 2023) dataset is an open-vocabulary detection dataset.
Omnilabel provides an evaluation of compositionality from perspectives such as spatial relationships, actions, and numeracy within referring objects.
We perform mean average precision (mAP), a standard evaluation metric, to validate the language-based object detection task.

**Table 1: Evaluation on D^3 (Xie et al., 2023) and OmniLabel (Schulter et al., 2023). D^3 provides three types of descriptions: absence (ABS), presence (PRES), and full (FULL). text length. For OmniLabel, the final AP is computed as the geometric mean of category-level (APc) and description-level (APd) scores. Note that the evaluation results of GLEE-Lite^∗ are reproduced.**
| Model | Backbone | D^3 (default) | D^3 (length) | OmniLabel (default) | Omnilabel (length) |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Full | Pres | Abs | S | M | L | XL | AP | APc | APd | S | M | L |  |  |
| OFA-L (Wang et al., 2022) | RN50 | 4.2 | 4.1 | 4.6 | 4.9 | 5.4 | 3.0 | 2.1 | 2.7 | 2.7 | 2.6 | 3.6 | 2.7 | 2.3 |
| OWL (Minderer et al., 2022b) | ViT-B | 9.6 | 10.7 | 6.4 | 20.7 | 9.4 | 6.0 | 5.3 | 8.0 | 15.6 | 5.4 | 5.7 | 5.4 | 6.2 |
| UNINEXT (Lin et al., 2023) | RN50 | 21.6 | 23.7 | 15.4 | 23.6 | 22.6 | 20.5 | 18.4 | 22.2 | 27.2 | 18.8 | - | - | - |
| G-DINO (Liu et al., 2024a) | Swin-T | 20.7 | 20.1 | 22.5 | 22.6 | 22.5 | 18.9 | 16.5 | 19.3 | 23.6 | 16.4 | 29.4 | 14.8 | 8.2 |
| GEN (Zhao et al., 2024) | Swin-T | 21.4 | 20.6 | 23.7 | 28.1 | 24.5 | 17.4 | 11.5 | 22.2 | 27.2 | 18.8 | - | - | - |
| GLIP (Li et al., 2022) | Swin-T | 19.1 | 18.3 | 21.5 | 22.4 | 22.0 | 16.6 | 10.6 | 19.3 | 23.6 | 16.4 | 29.4 | 14.8 | 8.2 |
| GLEE-Lite^∗ (Wu et al., 2024) | RN50 | 27.6 | 26.8 | 30.1 | 30.0 | 27.6 | 26.9 | 17.2 | 21.7 | 36.6 | 15.4 | 28.4 | 13.8 | 10.3 |
| GLIP + DesCo (Li et al., 2023b) | Swin-T | 24.2 | 22.9 | 27.8 | 24.3 | 21.9 | 16.4 | 11.5 | 23.8 | 27.4 | 21.0 | 33.7 | 19.0 | 13.7 |
| GLEE-Lite + DesCo | RN50 | 28.3 | 27.6 | 30.3 | 30.2 | 28.4 | 27.8 | 18.2 | 24.6 | 37.3 | 18.3 | 32.0 | 17.0 | 13.2 |
| GLEE-Lite + TaSe (ours) | RN50 | 30.7 | 29.9 | 33.2 | 31.8 | 31.2 | 30.3 | 19.8 | 26.9 | 36.8 | 21.2 | 33.1 | 19.3 | 14.8 |

### 4.2 Main Results

We investigate the impact of object detection on disentanglement and hierarchical representation learning through a set of research questions.

Does learning hierarchical entailment improve generalization?
As shown in Tab. [1](https://arxiv.org/html/2509.24192v1#S4.T1), the proposed model improves upon the baseline by fine-tuning only the LoRA and TriDe.
Compared to GLEE, which served as the vision foundation model, we observe improvements of $+3.1$ in D^3 and $+5.2$ in Omnilabel AP scores.
The AP scores in OmniLabel show that hierarchical learning improves performance in zero-shot evaluation, and the gains observed on open-vocabulary benchmarks further demonstrate its effectiveness.

Does hierarchical learning provide greater benefits than caption augmentation like DesCo?
We further evaluate the performance of GLEE with caption augmentation based on DesCo (Li et al., 2023b).
To apply this augmentation, we randomly sample from HiVG.
The selected sentence is concatenated with the original caption, and the augmented components are pooled separately and then averaged.
The DesCo improves the GLEE model $+2.4$ AP on D^3 and $+2.3$ on Omnilabel.
While caption augmentation increases textual diversity, our hierarchy learning further enhances TaSe by enabling accurate distinction of positives and negatives, even when sentences share category names or attributes.

Qualitative results.
We present two qualitative examples in Fig. [6](https://arxiv.org/html/2509.24192v1#S4.F6) to illustrate the effectiveness of our hierarchical entailment learning.
The first case shows negative cases containing an attribute (i.e., blue), and the second case presents a positive case with attributes and descriptive relations (e.g., numeracy and text in the image).
In the first case, GLEE incorrectly assigns a confident score of $0.92$ to the language query.
In the second case, GLEE predicts all bikes as positives, including those that do not correspond to the queried bike.
On the other hand, TaSe captures contextual information related to category names, and hierarchical entailment helps reduce false positives.
More qualitative results are provided in Figs. [11](https://arxiv.org/html/2509.24192v1#A3.F11) and [12](https://arxiv.org/html/2509.24192v1#A3.F12) of the supplementary material.

### 4.3 Ablation Studies

**Table 2: GLEE trained with hierarchy entailment. The final AP value is the geometric mean of categories (APc) and free-form descriptions (APd). $\mathcal{L}_{\text{CL}}$ and $\mathcal{L}_{\text{RE}}$ represnt contrastive loss (Oord et al., 2018) and RE embedding objective, respectively.**
|  | D^3 | OmniLabel |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | FULL | PRES | ABS | AP | APc | APd |
| Original GLEE | 27.6 | 27.1 | 30.5 | 21.7 | 36.6 | 15.4 |
| + LoRA (base) | 27.5 | 26.7 | 30.0 | 21.7 | 36.5 | 15.5 |
| + $\mathcal{L}_{\text{CL}}$ | 26.9 | 26.1 | 29.1 | 23.9 | 36.7 | 17.7 |
| + $\mathcal{L}_{\text{RE}}$ | 27.5 | 26.7 | 30.0 | 25.7 | 36.9 | 19.8 |
| + $\mathcal{L}_{\text{H}}$ (ours) | 28.6 | 27.8 | 31.6 | 26.2 | 38.9 | 19.2 |
| $\;\;\;\llcorner$ w/ $\mathcal{L}_{\text{H}^{+}}$ | 28.8 | 27.7 | 31.8 | 24.8 | 37.1 | 18.5 |
| $\;\;\;\llcorner$ w/ $\mathcal{L}_{\text{H}^{-}}$ | 27.7 | 27.0 | 30.1 | 23.6 | 36.4 | 18.4 |
| + Reverse $\mathcal{L}_{\text{H}}$ | 26.7 | 25.8 | 29.3 | 22.1 | 36.8 | 15.3 |

What advantages does our hierarchical loss offer over traditional contrastive loss?
In Tab. [3](https://arxiv.org/html/2509.24192v1#S4.T3), we conduct an ablation study to validate the effectiveness of our hierarchical loss.
Within the base setting (GLEE with LoRA), we evaluate three configurations: (1) conventional contrastive loss ($\mathcal{L}_{\text{CL}}$); (2) the RE objective ($\mathcal{L}_{\text{RE}}$); and (3) the proposed method.
For sentence-level hierarchy aggregation, our loss $\mathcal{L}_{\text{H}}$ outperforms contrastive baselines. Conventional contrastive learning causes embeddings of identical category names to diverge when descriptive information differs, whereas our reference-based hierarchy induction aligns them hierarchically and improves sentence-level meaning and performance.

In hierarchy entailment loss, is it better to learn positives or negatives?
We ablate the hierarchy entailment loss to compare learning with positive and negative pairs. Fine-tuning with positives improves OmniLabel by $+3.1$ AP, while negatives yield $+1.9$ AP.
Combining both provides the best generalization in zero-shot settings.
To validate this, we invert the objective and observe performance degradation when positives and negatives are aligned in opposite directions (Tab. [3](https://arxiv.org/html/2509.24192v1#S4.T3)).
These findings highlight the role of hierarchical entailment in building effective sentence-level embeddings and suggest that aligning semantically meaningful sentences with visual representations improves performance.

Ablation on where and how to disentangle in text representation.
Tab. [3](https://arxiv.org/html/2509.24192v1#S4.T3) reports ablation studies analyzing the design choices of the TriDe module.
Interestingly, we observe that where text embeddings are disentangled has the greatest influence on learning granularity.
We compare three modes for constructing compositional text embeddings: (1) no disentanglement, (2) token-level disentanglement, and (3) disentangled text embeddings after pooling.
Pooling compresses information and limits effective disentanglement, while the no-disentanglement approach is insufficient for capturing sentence-level contextualization.
Token-level disentanglement generalizes better and yields the best performance, with module initialization also having a substantial impact on the results.
In exploring how to design the TriDe module, we investigate disentanglement under three self-attention variants: direct text alignment, learnable queries, and key–value configurations.
Key–value attention outperforms query-only and self-attention mechanisms.
Key (indexing)–value (content) attention preserves independent subspaces and yields more structured semantic features than query-based approaches.
We provide the disentangled embedding results in
supplementary material, Fig. [14](https://arxiv.org/html/2509.24192v1#A5.F14).

Is it beneficial to disentangle the representation into three components?
Conventional detectors (Li et al., 2023b; Yuksekgonul et al., 2022) disentangle objects and attributes, whereas we separate representations into three components—object, attribute, and relation—and evaluate their effectiveness.
As shown at the bottom of Tab. [3](https://arxiv.org/html/2509.24192v1#S4.T3), overall, the performance of three-component disentanglement is higher than two-component (i.e., object and attribute) disentanglement.
These findings suggest that three-component disentanglement introduces an inductive bias for complex linguistic structures, making longer sentences more robust to negatives.
The granularity of text embeddings reveals features that characterize their representational properties.
To further disentangle these components, explicit criteria for dataset composition are required.

Figure: Figure 5: Qualitative analysis on Omnilabel data (Schulter et al., 2023). We visualize and compare the results between our baseline (GLEE) and TaSe. We select the scenario that includes attributes and relations for referring to a category name.
Refer to caption: x6.png

Figure: (a) GLEE embedding visualization
Refer to caption: x9.png

How are embeddings structured after disentanglement?
While a few negative pairs still exhibit large angles, Fig. [6](https://arxiv.org/html/2509.24192v1#S4.F6) confirms that positive and negative pairs are effectively aligned in the embedding space.
To examine whether this alignment follows the intended structure after disentanglement and hierarchical aggregation, we visualize the t-SNE projection of the trained TaSe embeddings.
As shown in Fig. [7(a)](https://arxiv.org/html/2509.24192v1#S4.F7.sf1), GLEE is dispersed around category names, whereas TaSe realigns embeddings around objects and preserves robust angular distance for negatives corresponding to attributes or relations. This is evident in the “segway” object, where the captions “black segway with a woman” and “black segway with a man” lie at different angles from the reference point “black segway.”

## 5 Conclusion

This study proposed a disentanglement and hierarchy aggregation framework for constructing contextualized sentence representations within language-based object detection.
Additionally, we generate re-captioned data for object detection using hierarchical concepts.
TaSe improved the linguistic compositionality, which serves as a key learning factor and leads to competitive results.
The results indicated that hierarchy entailment allows learning the granularity of text embedding to distinguish descriptive sentences.
This study highlights the need for further exploration of the underlying linguistic compositionality in future studies for downstream vision tasks.

#### Ethics Statement

During the preparation of this work, the author used ChatGPT (Hurst et al., 2024) in order to improve readability. After using this tool/service, the author reviewed and edited the content as needed and takes full responsibility for the content of the publication.

### Reproducibility Statement

We recaptioned our dataset, utilizing the publicly available Llama 3 (Dubey et al., 2024) model released on Hugging Face hub (Wolf et al., 2020).
Additional statistics and details of the dataset are presented in Sec. [A](https://arxiv.org/html/2509.24192v1#A1) of the supplementary material.
The code for the experiments can be found in the supplementary material.

## References

- Alper & Averbuch-Elor (2024)
Morris Alper and Hadar Averbuch-Elor.
Emergent visual-semantic hierarchies in image-text representations.
In *European Conference on Computer Vision*, pp. 220–238. Springer, 2024.
- Atigh et al. (2022)
Mina Ghadimi Atigh, Julian Schoep, Erman Acar, Nanne Van Noord, and Pascal Mettes.
Hyperbolic image segmentation.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 4453–4462, 2022.
- Bengio et al. (2013)
Yoshua Bengio, Aaron Courville, and Pascal Vincent.
Representation learning: A review and new perspectives.
*IEEE transactions on pattern analysis and machine intelligence*, 35(8):1798–1828, 2013.
- Deng et al. (2009)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In *2009 IEEE conference on computer vision and pattern recognition*, pp. 248–255. Ieee, 2009.
- Desai et al. (2023)
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam.
Hyperbolic image-text representations.
In *International Conference on Machine Learning*, pp. 7694–7731. PMLR, 2023.
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Ganea et al. (2018)
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann.
Hyperbolic entailment cones for learning hierarchical embeddings.
In *International conference on machine learning*, pp. 1646–1655. PMLR, 2018.
- Gao et al. (2024)
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.
Clip-adapter: Better vision-language models with feature adapters.
*International Journal of Computer Vision*, 132(2):581–595, 2024.
- Gu et al. (2021)
Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.
Open-vocabulary object detection via vision and language knowledge distillation.
*arXiv preprint arXiv:2104.13921*, 2021.
- Hong et al. (2023)
Jie Hong, Zeeshan Hayder, Junlin Han, Pengfei Fang, Mehrtash Harandi, and Lars Petersson.
Hyperbolic audio-visual zero-shot learning.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 7873–7883, 2023.
- Hu et al. (2021)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
*arXiv preprint arXiv:2106.09685*, 2021.
- Huang et al. (2023)
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu.
T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.
*Advances in Neural Information Processing Systems*, 36:78723–78747, 2023.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Ji et al. (2023)
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung.
Survey of hallucination in natural language generation.
*ACM computing surveys*, 55(12):1–38, 2023.
- Kamath et al. (2021)
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.
Mdetr-modulated detection for end-to-end multi-modal understanding.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 1780–1790, 2021.
- Khrulkov et al. (2020)
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky.
Hyperbolic image embeddings.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 6418–6428, 2020.
- Kong et al. (2024)
Fanjie Kong, Yanbei Chen, Jiarui Cai, and Davide Modolo.
Hyperbolic learning with synthetic captions for open-world detection.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 16762–16771, 2024.
- Krause et al. (2017)
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei.
A hierarchical approach for generating descriptive image paragraphs.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 317–325, 2017.
- Krishna et al. (2017)
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.
Visual genome: Connecting language and vision using crowdsourced dense image annotations.
*International journal of computer vision*, 123:32–73, 2017.
- Lang et al. (2022)
Christopher Lang, Alexander Braun, Lars Schillingmann, and Abhinav Valada.
On hyperbolic embeddings in object detection.
In *DAGM German Conference on Pattern Recognition*, pp. 462–476. Springer, 2022.
- Li et al. (2023a)
Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeung Shum.
Mask dino: Towards a unified transformer-based framework for object detection and segmentation.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 3041–3050, 2023a.
- Li et al. (2023b)
Liunian Li, Zi-Yi Dou, Nanyun Peng, and Kai-Wei Chang.
Desco: Learning object recognition with rich language descriptions.
*Advances in Neural Information Processing Systems*, 36:37511–37526, 2023b.
- Li et al. (2022)
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.
Grounded language-image pre-training.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 10965–10975, 2022.
- Lin et al. (2023)
Fangjian Lin, Jianlong Yuan, Sitong Wu, Fan Wang, and Zhibin Wang.
Uninext: Exploring a unified architecture for vision recognition.
In *Proceedings of the 31st ACM International Conference on Multimedia*, pp. 3200–3208, 2023.
- Lin et al. (2017)
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.
Focal loss for dense object detection.
In *Proceedings of the IEEE international conference on computer vision*, pp. 2980–2988, 2017.
- Lin et al. (2024)
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan.
Evaluating text-to-visual generation with image-to-text generation.
In *European Conference on Computer Vision*, pp. 366–384. Springer, 2024.
- Liu et al. (2024a)
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al.
Grounding dino: Marrying dino with grounded pre-training for open-set object detection.
In *European Conference on Computer Vision*, pp. 38–55. Springer, 2024a.
- Liu et al. (2024b)
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al.
Grounding dino: Marrying dino with grounded pre-training for open-set object detection.
In *European Conference on Computer Vision*, pp. 38–55. Springer, 2024b.
- Liu et al. (2019)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
Roberta: A robustly optimized bert pretraining approach.
*ArXiv*, abs/1907.11692, 2019.
- Loshchilov & Hutter (2017)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
*arXiv preprint arXiv:1711.05101*, 2017.
- Lou et al. (2020)
Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa.
Differentiating through the fréchet mean.
In *International conference on machine learning*, pp. 6393–6403. PMLR, 2020.
- Lu et al. (2023)
Xiaocheng Lu, Ziming Liu, Song Guo, Jingcai Guo, Fushuo Huo, Sikai Bai, and Tao Han.
Drpt: Disentangled and recurrent prompt tuning for compositional zero-shot learning.
*arXiv preprint arXiv:2305.01239*, 2023.
- Miller (1995)
George A Miller.
Wordnet: a lexical database for english.
*Communications of the ACM*, 38(11):39–41, 1995.
- Min et al. (2022)
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer.
Rethinking the role of demonstrations: What makes in-context learning work?
*arXiv preprint arXiv:2202.12837*, 2022.
- Minderer et al. (2022a)
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al.
Simple open-vocabulary object detection.
In *European Conference on Computer Vision*, pp. 728–755. Springer, 2022a.
- Minderer et al. (2022b)
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al.
Simple open-vocabulary object detection.
In *European conference on computer vision*, pp. 728–755. Springer, 2022b.
- Oord et al. (2018)
Aaron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding.
*arXiv preprint arXiv:1807.03748*, 2018.
- Pal et al. (2024)
Avik Pal, Max van Spengler, Guido Maria D’Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes.
Compositional entailment learning for hyperbolic vision-language models.
*arXiv preprint arXiv:2410.06912*, 2024.
- Park et al. (2024)
Kwanyong Park, Kuniaki Saito, and Donghyun Kim.
Weak-to-strong compositional learning from generative models for language-based object detection.
In *European Conference on Computer Vision*, pp. 1–19. Springer, 2024.
- Qi et al. (2024)
Tianhao Qi, Shancheng Fang, Yanze Wu, Hongtao Xie, Jiawei Liu, Lang Chen, Qian He, and Yongdong Zhang.
Deadiff: An efficient stylization diffusion model with disentangled representations.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 8693–8702, 2024.
- Radford et al. (2021a)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, pp. 8748–8763. PmLR, 2021a.
- Radford et al. (2021b)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, pp. 8748–8763. PmLR, 2021b.
- Ren et al. (2024)
Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, et al.
Dino-x: A unified vision model for open-world object detection and understanding.
*arXiv preprint arXiv:2411.14347*, 2024.
- Rezatofighi et al. (2019)
Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.
Generalized intersection over union: A metric and a loss for bounding box regression.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 658–666, 2019.
- Schulter et al. (2023)
Samuel Schulter, Yumin Suh, Konstantinos M Dafnis, Zhixing Zhang, Shiyu Zhao, Dimitris Metaxas, et al.
Omnilabel: A challenging benchmark for language-based object detection.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 11953–11962, 2023.
- Shen et al. (2024)
Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji.
Aligning and prompting everything all at once for universal visual perception.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 13193–13203, 2024.
- Speer et al. (2017)
Robyn Speer, Joshua Chin, and Catherine Havasi.
Conceptnet 5.5: An open multilingual graph of general knowledge.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 31, 2017.
- Van der Maaten & Hinton (2008)
Laurens Van der Maaten and Geoffrey Hinton.
Visualizing data using t-sne.
*Journal of machine learning research*, 9(11), 2008.
- Wang et al. (2022)
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.
Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.
In *International conference on machine learning*, pp. 23318–23340. PMLR, 2022.
- Wang et al. (2023)
Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang.
Equivariant similarity for vision-language foundation models.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 11998–12008, 2023.
- Wang et al. (2024)
Xin Wang, Hong Chen, Si’ao Tang, Zihao Wu, and Wenwu Zhu.
Disentangled representation learning.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 46(12):9677–9696, 2024.
- Wolf et al. (2020)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al.
Transformers: State-of-the-art natural language processing.
In *Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations*, pp. 38–45, 2020.
- Wu et al. (2024)
Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai.
General object foundation model for images and videos at scale.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 3783–3795, 2024.
- Wu et al. (2025)
Peng Wu, Xiankai Lu, Hao Hu, Yongqin Xian, Jianbing Shen, and Wenguan Wang.
Logiczsl: Exploring logic-induced representation for compositional zero-shot learning.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 30301–30311, 2025.
- Xie et al. (2023)
Chi Xie, Zhao Zhang, Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang.
Described object detection: Liberating object detection with flexible expressions.
*Advances in Neural Information Processing Systems*, 36:79095–79107, 2023.
- Yin et al. (2025)
Heng Yin, Yuqiang Ren, Ke Yan, Shouhong Ding, and Yongtao Hao.
Rod-mllm: Towards more reliable object detection in multimodal large language models.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 14358–14368, 2025.
- Yu et al. (2016)
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg.
Modeling context in referring expressions.
In *Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14*, pp. 69–85. Springer, 2016.
- Yuksekgonul et al. (2022)
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.
When and why vision-language models behave like bags-of-words, and what to do about it?
*arXiv preprint arXiv:2210.01936*, 2022.
- Zeng et al. (2024)
Yunan Zeng, Yan Huang, Jinjin Zhang, Zequn Jie, Zhenhua Chai, and Liang Wang.
Investigating compositional challenges in vision-language models for visual grounding.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 14141–14151, 2024.
- Zhao et al. (2024)
Shiyu Zhao, Long Zhao, Yumin Suh, Dimitris N Metaxas, Manmohan Chandraker, Samuel Schulter, et al.
Generating enhanced negatives for training language-based object detectors.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 13592–13602, 2024.
- Zheng et al. (2024)
Zhaoheng Zheng, Haidong Zhu, and Ram Nevatia.
Caila: concept-aware intra-layer adapters for compositional zero-shot learning.
In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pp. 1721–1731, 2024.

## Appendix A Details of HiVG

### A.1 Creating Hierarchical Positive Captions

Our hierarchical captioning pipeline is illustrated in Tab. [4](https://arxiv.org/html/2509.24192v1#A1.T4), which highlights key differences from conventional captioning approaches.
We generate a 10 K re-captioned dataset from the Visual Genome dataset.
We first filter out all Visual Genome captions with fewer than six words to ensure sufficient semantic richness.
For positive captions, we leverage in-context learning using LLaMA (Dubey et al., 2024) to transform the remaining Visual Genome captions into a three-tier hierarchical structure.
To enhance attribute diversity within the captions, we draw on common visual concepts (Huang et al., 2023; Lin et al., 2024) to define a set of visual attributes (spatial, color, number, and size) and randomly select one to modify the object for alternative object attributes.
Based on our experiments, we set the randomization ratio to 50%.
To fully align with the in-context learning demonstration format, samples that lack attributes or relations are also incorporated into the learning process, following the approach suggested by Min et al. (2022).

### A.2 Creating Hierarchical Negative Captions

One of the challenges in language-based object detection is effectively handling negative samples, which often report higher false negative rates compared to false positives.
To address the issue, we focus on both re-captioning hard and easy negative samples.
For hard negative samples in tier 1, we replace the positive object with an antonym (e.g., man is replaced by woman) or a random concrete noun.
Easy negative samples are generated by selecting nouns from ImageNet1000 (Deng et al., 2009) classes and lexical databases such as WordNet and ConceptNet.
Additionally, we insert a negative determiner to the object (e.g., dog is switched to no dog).

In tier 2, we reuse the same set of visual attributes from the positive captions but replace them with semantically different attributes (e.g., tall building is replaced by short building) for generating hard negative samples.
We use LLM-based mask-filling (Liu et al., 2019) to diversify attributes by substituting them with contextually plausible but semantically different terms or by prepending “not” to create hard negatives (e.g., tall building → not tall building).

In tier 3, we use a set of common spatial relations (e.g., above and beside) and object-specific relations from the Visual Genome dataset.
These pre-defined object-specific relations ensure that the relation is contextually relevant to the object in question.
To introduce hard negatives, we apply absence-based transformations by replacing affirmative relations with their negative counterparts (e.g., with is replaced by without).
We also leverage LLaMA’s (Dubey et al., 2024) sentence completion capability to generate further relation diversity.

Captions that do not adhere to the hierarchical structure are filtered out.
By re-captioning using a multi-tiered set of positive and negative captions, our approach is intended to facilitate the learning of hierarchical representations, thereby improving linguistic compositionality.

Figure: Figure 8: Overview of generating positive and negative captions. Positive captions are derived using in-context learning based on Llama3 (Dubey et al., 2024). We transform Visual Genome captions into structured forms: object (category name), attribute (category name with an attribute), and relation (category name with an attribute and a relation). Negative captions are constructed through a multi-step process: 1) retrieving antonyms or random concrete nouns from lexical databases for negative objects; 2) using LLM-based mask-filling combined with pre-defined visual attributes to generate semantically different negative attributes; and 3) using pre-defined object-specific relations to create negative relations.
Refer to caption: x12.png

**Table 4: Comparison of positive/negative caption strategies and entailment assumptions across datasets. Our method introduces grounded, logic-consistent supervision with object-level structure, unlike prior captioning datasets.**
| Dataset / Approach | Positive Captions | Negative Captions | Entailment Structure |
| --- | --- | --- | --- |
| Visual Genome<br>(Krishna et al., 2017) | Flat object-centric region descriptions | - | - |
| Image Paragraphs<br>(Krause et al., 2017) | Multi-sentence paragraphs per image | - | Narrative-level cohesion only |
| HierarCaps<br>(Alper & Averbuch-Elor, 2024) | LLM-generated hierarchical captions | LLM&NLI-based structure contradiction samples | Inferred via entailment prediction |
| HiVG | Explicit object $\rightarrow$ attribute $\rightarrow$ relation chains, used in in-context learning | WordNet-informed hard negatives and ImageNet-based visual diversity | Explicit tiered entailment |

### A.3 Static Dataset Construction

The combined dataset consists of 286,006 annotations, with the majority containing $8.75\pm 1.34$ words, as shown in Fig. [9](https://arxiv.org/html/2509.24192v1#A1.F9).
Caption length distributions across Positive and Negative samples are largely consistent within each tier.
Tier 3 contains the most linguistically diverse and structurally rich captions, which may be particularly beneficial for semantic reasoning.

Figure: Figure 9: Statistics of HiVG dataset. Top: Distribution of the number of words. (Left) The original Visual Genome dataset. (Middle) positive captions. (Right) Negative captions. Bottom: Distribution of the number of words per tier. (Left) Tier1 - category name. (Middle) Tier2 - attribute. (Right) - Tier3 relation.
Refer to caption: x13.png

## Appendix B Experimental Setup

Baselines
This paper compares the language-based object detection models on MDETR (Kamath et al., 2021), OFA (Wang et al., 2022), OWL (Minderer et al., 2022b), G-DINO (Liu et al., 2024a), GLIP (Li et al., 2022), UNINEXT (Lin et al., 2023), Desco (Li et al., 2023b), GLIP-GEN (Zhao et al., 2024), and GLEE (Wu et al., 2024).

### B.1 Implementation Details

We sample 16 images per batch and further select 6 corresponding sentences per image for hierarchy learning.
We employ AdamW (Loshchilov & Hutter, 2017) to optimize the trainable model, using a learning rate of $1\times 10^{-4}$ for the TriDe module and $5\times 10^{-6}$ for LoRA.
For the comparison with baselines, our detector was trained for 60 K iterations, the same as in the ablation studies.
Following Alper & Averbuch-Elor (2024), the RE loss was set with a positive-to-negative ratio of 10:4.
In case of $\mathcal{L}_{\text{H}}$, we conduct experiments wit positive-to-negative ratio of 2:1.
The values of $\gamma$ is set to 0.1.

**Table 5: Hyperparameters setting**
| Params. | Value |
| --- | --- |
| Batch size | 4 |
| Optimizer | AdamW |
| Optimizer momentum | $\beta_{1}=0.9$, $\beta_{2}=0.999$ |
| Rank of LoRA | 16 |
| scaling factor of LoRA | 16 |
| learning rate of LoRA | 5e-6 |
| learning rate of TriDe | 1e-4 |
| Input resolution | 800 $\times$ 800 |
| loss of class ($\mathcal{L}_{class}$) | 4.0 |
| loss of bbox ($\mathcal{L}_{bbox}$) | 5.0 |
| loss of gIoU ($\mathcal{L}_{giou}$) | 2.0 |
| loss of TaSe ($\mathcal{L}_{\text{TaSe}}$) | 5.0 |
| $\lambda$ | 0.1 |

### B.2 Model Size and Budget

For fine-tuning the pre-trained GLEE, we only train LoRA layers, TriDe module, and VL fusion layers.
We train a total of 5,447,680 parameters, which is an efficient approach that reduces memory usage by 2.93% of the model parameters.
Experiments were conducted using 4 NVIDIA A6000 GPUs for model training.

**Table 6: Model configuration.**
| Layers | # of Params. (M) |
| --- | --- |
| Image backbone | 23.5 |
| Text encoder | 126.3 |
| Detector | 31.5 |
| LoRA | 0.4 |
| TriDe | 1.9 |
| VL | 3.2 |
| Trainable params. | 5.4 (2.93%) |

## Appendix C Qualitative Results

Hierarchy training embedding analysis.
We validate the effectiveness of our proposed hierarchical learning approach by visualizing the impact of the angular loss on both inter-tier and intra-tier constraints.
For the experimental setup, we randomly initialize 50 two-dimensional embeddings and train them using the original hierarchy loss and our extended loss function.

Figure: Figure 10: Visualization of angular embeddings (without dynamic reference). Triangles illustrate learned pairs with respect to the root: positive pairs (red) and negative pairs (blue) are connected to depict directional behavior. Positive pairs are expected to align in similar directions from the root, while negative pairs should diverge. While the baseline tends to increase radial distance more than meaningful angular adjustment, our objective function encourages more structured representations guided by directional alignment.
Refer to caption: x14.png

Additional qualitative results.
To evaluate whether our model effectively learns sentence-level hierarchy, we compared its performance with baselines using scenarios including objects, attributes, and relations from the benchmark dataset.
As shown in Fig. [11](https://arxiv.org/html/2509.24192v1#A3.F11) and Fig. [12](https://arxiv.org/html/2509.24192v1#A3.F12), we visualize the results of two scenarios containing an absent example.
Given that sentences become longer, many VLMs focus on specific words, such as “running” to detect objects.
For example, our model improves performance by capturing richer semantic information, such as the attribute “pink,” and understanding contextual meaning, like recognizing the “girl” as the subject of “running.”
In contrast, our model demonstrates greater robustness in detecting complex relations and predicts bounding boxes more accurately by better understanding object states and relative information.

Figure: Figure 11: Qualitative analysis on Omnilabel data Schulter et al. (2023). We visualize and compare the results between GLEE and TaSe. We visualized the prediction results for both positive and negative captions of the same image.
Refer to caption: x15.png

Figure: Figure 12: Qualitative analysis on Omnilabel data Schulter et al. (2023). We visualize and compare the results between GLEE and TaSe.
Refer to caption: x17.png

## Appendix D HiVG statistic analysis

We report that our hierarchy dataset, HiVG, in Fig. [13](https://arxiv.org/html/2509.24192v1#A4.F13).
Leveraging Visual Genome data for re-captioning, we create a more diverse dataset by incorporating a wider range of classes and the LLM and other datasets.

Figure: Figure 13: Re-captioning data examples
Refer to caption: x19.png

## Appendix E Details of Compositional Learning

We provide the details of the disentanglement modes employed for compositional learning as shown in Alg. [1](https://arxiv.org/html/2509.24192v1#algorithm1).
The first mode adopts traditional mean-pooling and uses the resulting representation for contrastive learning.
The second and third modes involve disentanglement via a TriDe module, followed by contrastive learning based on the aggregated compositional embedding.
Specifically, the second mode applies the TriDe module at the token level to leverage information across all tokens, whereas the third mode applies the module after pooling, focusing on sentence-level semantics.

Figure: Algorithm 1 Pseudo code for disentangling and hierarchical aggregating paradigm

Figure: (a) Embedding visualization of the three disentangled components for the language query “Segway with a man”
Refer to caption: x20.png

## Appendix F Analysis of Representation Disentanglement

To verify whether disentangled embeddings contain distinct embedding representations for each component, we visualize the embedding of each component using t-SNE.
For the t-SNE visualization, we construct the embedding space using our HiVG dataset of 10K samples.
We then visualize the embeddings based on language queries, which are our motivation.
As shown in Fig. [14](https://arxiv.org/html/2509.24192v1#A5.F14), while there are slight variations depending on token information, we observe that the embeddings for each component cluster relatively well.
This validates that when a sentence is input, each component holds disentangled representations.

## Appendix G Additional Experimental Results

We conducted an ablation study on the model parameters.
We compared and analyzed the effects of adjusting the LoRA (Hu et al., 2021) rank (see Fig. [17](https://arxiv.org/html/2509.24192v1#A7.F17)), the number of tokens (see Fig. [17](https://arxiv.org/html/2509.24192v1#A7.F17), and the margin parameter (see Fig. [17](https://arxiv.org/html/2509.24192v1#A7.F17)) for the disentangled loss.

Figure: Figure 15: Analysis on the effect of different LoRA ranks
Refer to caption: x22.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: The paper directly tackles the "bag-of-words" behavior of current VLMs, which often fail to distinguish target objects from descriptive attributes and relations in complex queries.


The proposed TaSe framework demonstrates significant performance improvements over strong baselines like GLEE. It achieved a 24% performance improvement on the OmniLabel benchmark and increased AP scores by +3.1 on D³ and +5.2 on OmniLabel.

It introduces a framework that disentangles text into three core components (objects, attributes, relations) and aggregates them hierarchically, strengthening the inductive bias for linguistic structures.


The paper proposes the HiVG dataset.
summary: Current VLMs struggle with complex language queries involving descriptive attributes and relational clauses. Their text encoders often behave like bags-of-words, failing to distinguish target objects from their attributes and relations, which leads to frequent false positives (e.g., confusing "segway with a man" for just "segway"). The paper proposes a framework designed to restructure linguistic representations according to hierarchical relations within sentences. The core principle is to disentangle text tokens into components and aggregate them into hierarchically structured sentence-level representations. The paper proposes:

HiVG Dataset: A hierarchical synthetic captioning dataset derived from Visual Genome. It structures captions into three tiers using a "reverse abstraction" process to mitigate hallucination:

A module that transforms text embeddings into three distinct subspaces, objects, attributes, and relations, guided by a disentanglement loss function.

A learning method that aggregates these disentangled components into structured embeddings. Unlike contrastive learning, it models full sentence hierarchy using dynamic references to enforce separation between negative pairs while preserving intrinsic embedding structure.
weaknesses: 1: The proposed HiVG dataset and TriDe module strictly enforce a three-tier hierarchy: Object, Attribute and Relation. While this addresses simple compound queries, natural language often involves deeper, recursive nesting that exceeds three tiers (e.g., "the man holding the cup that is on the table"). The current framework may struggle to represent these deeper recursive structures efficiently because it is designed purely around these three predefined components.

2. Ambiguity in "Attribute" vs. "Relation" Definitions. The distinction between Tier 2 (Attribute) and Tier 3 (Relation) appears conflated in the examples provided. For instance, "middle woman" is Tier 2, while "middle woman with dark hair" is classified as Tier 3 (Relation). Linguistically, "dark hair" is often considered an attribute of the entity, similar to "middle." By contrast, other Tier 3 examples involve distinct objects, such as "segway with a man". Treating intrinsic attributes (like hair color) and extrinsic relations (like accompanying objects) identically in Tier 3 might limit the model's ability to disentangle complex scenes where both distinct object interactions and fine-grained attributes are present simultaneously.

3. The fine-tuning relies on a relatively small synthetic dataset. The authors explicitly state they "use only HiVG dataset for training, which contains 10 K hierarchy captions". While efficient (training only 2.93% of parameters) , basing generic open-vocabulary improvements on such a small, specifically structured dataset risks overfitting to the specific syntactic patterns generated by Llama 3 during the re-captioning process. The claimed robustness to general complex queries might be limited to the specific domain covered by these 10K Visual Genome-derived captions.


4. The qualitative results presented focus exclusively on successes where TaSe outperforms the GLEE baseline by reducing false positives. There is no analysis of where the strict hierarchical enforcement might cause errors—for example, in idiomatic expressions that do not neatly decompose into object-attribute-relation, or where disentanglement might over-segment a coherent concept.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: - In the introduction, the authors reveal an observation that descriptions with partially-shared and partiall-different words have over-close embedding distances. However, recent SoTA VLMs are already trained with fine-grained discriptions. I am interested if the authors have deeper analysis on the reason why this phenomenon still happens.

- How does your method perform on general OVD dataset (e.g., LVIS). You may use the Out-of-Domain setting for easy comparison,i.e., you train a baseline method on your dataset and compare this baseline against your method.

- Will the proposed hierarchical and disengtangled representation learning be good for the VLM training (besides the OVD as in the paper)?
soundness: 3
strengths: - The observation on the text embedding well illustrates the reason that OVD-VLMs fail to understand complex queries. 

- The solution is comprehensive, covering from the data construction, the model architecture design, as well as the corresponding training objectives. 

- The data construction part is reasonable, and the method seems novel.

- The ablation study is comprehensive.
summary: This paper aims to tackle the problem that VLM-based OVD models fall short in understanding complex queries. The authors make an observation that the text embedding of VLM can not well distinguish different attributes / relations for the same object, and consider this phenomenon as the main reason that causes the insufficient ability. To tackle this problem, they first construct a new data pipeline that has structured descriptions with "object - attribute - relation" hierarchy. Under this hierarchy, each data point (the bbox) has multiple hard negative text names that have different attribute / relation.  They also propose a new method that first disentangles  the embedding into three components (object / attribute / relation) and then re-aggregates them into a new embedding. The corresponding training method has some novelty for accommodating the hierarchy, as well.
weaknesses: - The presentation and writing needs improvement. Some parts are difficult to follow. 

- According to the example given in Section 3.1, the most import aspect of HIVG is to generate HARD negative text discription of each box, i.e., the description sharing the same object but have different attribute (Tier 2) or realtion (Tier 3). It would be more clear for the authors to describe this principle into the introduction.

- In Eq. 1, 
    - 1) What is the text embedding X in Line 1?
    - 2) In Line 2, Which one is the query / value for the cross attention, respectively?
    - 3) It seems that TriDe not only disentangles the original embeddings, but also re-aggregates the derived O, A, R into the output E (Line 3). What is the motivation of this design?

- The improvement on D^3 dataset is very small, e.g., +1.0 FULL AP over the GLEE baseline.

- The employed datasets are insufficient. The authors have not includes a popolar dataset RefCoCo which also has long and fine-grained descriptions, and is popular for visual grounding evaluation. Is there any specific reason?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Lines 103–104, the authors state that the proposed reverse abstraction process “effectively mitigates hallucination issues.” Could the authors provide quantitative evidence or more systematic analysis to support this claim?
2. Can TaSe generalize to other VLM models?
3. How does the baseline trained on HiVG perform with complex but non-hierarchically annotated language?
soundness: 3
strengths: 1. The TaSe framework is well-designed that competently integrates disentanglement via attention and hierarchical learning via a Euclidean-space objective.
2. It achieves clear performance improvements over a strong baseline on the OmniLabel and D³ benchmarks. The ablation studies provide useful insights into the function of each component of the proposed framework.
3. The writing is clear, and the presentation of figures and diagrams is good.
summary: This paper introduces TaSe (Talk in Pieces, See in Whole), a framework designed to enhance language-based object detection by explicitly modeling the hierarchical and compositional structure of language. It first introduces a disentanglement module to break down text representations into core components of objects, attributes, and relations ('Talk in Pieces'), using a novel module called TriDe. It then designs a hierarchical aggregation mechanism to learn structured, sentence-level embeddings from these disentangled parts ('See in Whole'). To facilitate this, the authors introduce HiVG, a new dataset generated by using an LLM to re-caption the Visual Genome dataset with an explicit three-tier hierarchical structure. Experiment results on challenging language-based object detection benchmarks demonstrate that linguistic compositionality is crucial for the task of language-driven object detection.
weaknesses: 1. In the HiVG dataset generation process, does the diversity of the generated negative captions influence the final performance? For example, how would the results vary if different large language models (LLMs) were used for re-captioning?
2. The authors describe their approach as “lightweight”. it would be helpful to provide a quantitative analysis of the computational overhead.
3. Given the framework's strong reliance on the O-A-R structure, how does it perform if queries that violate or deviate from this structure (eg. comparative phrases or multi-step relations)?
4. The TriDe module uses learnable queries to disentangle objects, attributes, and relations. Can the authors provide some qualitative visualizations to demonstrate that the module indeed learns meaningful disentanglement?

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. The final text embedding E=pooling(O+A+R). What pooling function is used (mean, [CLS] token, attention)? Have you compared different pooling methods to see how they affect compositional sensitivity? 
2. How is the “reference point”  r in Eq. 6 chosen or updated?
3. Why were structure-aware or compositional baselines like CLIP-Adapter, LogicZSL, or MDETR not included?
4. Have you identified scenarios where the hierarchical structure fails — e.g., ambiguous relational clauses or overlapping attributes?
soundness: 3
strengths: 1. The paper convincingly diagnoses a key weakness of current VL detectors: poor sentence-level compositionality and contextual grounding. The empirical t-SNE analysis in Fig. 1 is effective in illustrating why text embeddings collapse across different relational phrases.
2. Comprehensive experiments on multiple benchmarks, with both quantitative (Tables 1–3) and qualitative (Figs. 5–7  results. 
3. The writing is clear and methodically organized.
summary: The paper addresses a key limitation in vision-language models: their poor compositional understanding of complex textual queries in language-based object detection (LBOD). Current models behave like “bags-of-words,” often misinterpreting sentences that combine object names, attributes, and relations. The authors propose TaSe (Talk in Pieces, See in Whole), which consists of:
1. HiVG Dataset: A hierarchical synthetic captioning dataset derived from Visual Genome, organized into object–attribute–relation tiers.
2. TriDe Module: A component-wise disentanglement mechanism that separates text embeddings into object, attribute, and relation subspaces.
3. Hierarchical Aggregation: A new loss based on hierarchical entailment using a Euclidean-space “Radial Embedding (RE)” objective to reconstruct contextualized sentence representations.
TaSe integrates into GLEE (a MaskDINO + CLIP-based VL detector) and achieves substantial gains on OmniLabel and D^3 benchmarks.
weaknesses: 1. HiVG, the proposed hierarchical dataset, is central to the paper but only briefly described in the main text. There’s no quantitative analysis of HiVG’s diversity, quality, or balance (e.g., number of distinct attributes/relations, comparison with Visual Genome). Since HiVG is synthetic and LLM-generated, potential hallucination or linguistic bias needs evaluation.
2. Consider including a human evaluation of caption quality or a robustness test (e.g., sensitivity to paraphrases).
3. The framework combines multiple non-trivial modules (TriDe, RE, hierarchical losses). While each part is motivated, the cumulative complexity may limit reproducibility. It’s unclear how much improvement comes from disentanglement vs. better data vs. hierarchical supervision, though ablations partly address this.
 


Minor Presentation Issues： 
1. Some equations (e.g., Eq. 3, 6) are dense and could use more intuition.

2. Figures could label components more clearly (especially in Fig. 2).


Decision: Reject