Open-Det: An Efficient Learning Framework for Open-Ended Detection
Authors: Guiping Cao
Tao Wang
Wenjian Huang
Xiangyuan Lan
Jianguo Zhang
Dongmei Jiang
Venue: ICML 2025 poster (2025)
Keywords: Open-Ended Detection; Framework; Prompts Distillation; Loss Functions
Primary area: deep_learning
TL;DR: Open-Ended Detection

Abstract: Open-Ended object Detection (OED) is a novel and challenging task that detects objects and generates their category names in a free-form manner, without requiring additional vocabularies during inference. However, the existing OED models, such as GenerateU, require large-scale datasets for training, suffer from slow convergence, and exhibit limited performance. To address these issues, we present a novel and efficient Open-Det framework, consisting of four collaborative parts. Specifically, Open-Det accelerates model training in both the bounding box and object name generation process by reconstructing the Object Detector and the Object Name Generator. To bridge the semantic gap between Vision and Language modalities, we propose a Vision-Language Aligner with V-to-L and L-to-V alignment mechanisms, incorporating with the Prompts Distiller to transfer knowledge from the VLM into VL-prompts, enabling accurate object name generation for the LLM. In addition, we design a Masked Alignment Loss to eliminate contradictory supervision and introduce a Joint Loss to enhance classification, resulting in more efficient training. Compared to GenerateU, Open-Det, using only 1.5% of the training data (0.077M vs. 5.077M), 20.8% of the training epochs (31 vs. 149), and fewer GPU resources (4 V100 vs. 16 A100), achieves even higher performance (+1.0% in APr). The source codes are available at: https://github.com/Med-Process/Open-Det.

Open-Det: An Efficient Learning Framework for Open-Ended Detection
Guiping Cao 12
Tao Wang 12
Wenjian Huang' Xiangyuan Lan
Jianguo Zhang 124
Dongmei Jiang"
Abstract
Zero Zhrt-shat Performancee on Lins Minivall
0.30
AP Generatel
Ge-Gereratel
Open-Ended object Detection (OED) is a novel and challenging task that detects objects and gen- erates their category names in a free-form manner, without requiring additional vocabularies during inference. However, the existing OED models, such as GenerateU, require large-scale datasets for training, suffer from slow convergence, and exhibit limited performance. To address these is- sues, we present a novel and efficient Open-Det framework, consisting of four collaborative parts. Specifically, Open-Det accelerates model training in both the bounding box and object name genera- tion process by reconstructing the Object Detector and the Object Name Generator. To bridge the se- mantic gap between Vision and Language modal- ities, we propose a Rision-Languen Aligner with V-to-L and L-to-V alignment mechanisms, incor- porating with the Prompts Distiller to transfer knowledge from the VLM into VL-prompts, en- abling accurate object name generation for the LLM. In addition, we design a Masked Alignment Loss to eliminate contradictory supervision and introduce a Joint Loss to enhance classification, resulting in more efficient training. Compared to GenerateU, Open-Det, using only 1.5% of the training data (0.077M vs. 5.077M), 20.8% of the training epochs (31 vs. 149), and fewer GPU resources (4 V100 vs. 16 A100), achieves even higher performance (+1.0% in AP,). The source codes are available at: https: //gi thub. com httm/Meed-rrocess/ppeeee
9:25
APr
GenerateU
AP-Open Det
9.20
AP, -ppen-DD
AP.eOpe
AP-Oppn-DDt
9:15
0.10
30
Training Epochs
Figure 1: Performance curves of GenerateU and Open-Det, trained on the VG and evaluated on zero-shot LVIS MiniVal.
1. Introduction
Open Vocabulary object Detection (OVD) (Zareian et al., 2021a; Gu et al., 2021; Lin et al., 2022; Wu et al., 2023b; Cheng et al., 2024) is a fundamental task in computer vision that largely extends detection abilities from conventional closed-set to open-set detection, allowing the location and identification of objects beyond fixed categories of training data. However, OVD still depends on additional vocabu- laries as input obtain detection results during inference. This reliance creates a necessary dependency supplemen- tary language knowledge priors, significantly restricting the model's 's detection capabilities in the open-world scenario.
Recently, Open-Ended object Detection (OED) (Lin et al., 2024) has emerged as a more general and practical object detection task that eliminates the need for predefined object categories during the inference stage. The end-to-end frame- work of GenerateU (Lin et al., 2024) formulates the OED as generative problem, enabling dense object detection and associated name generation in a free-form manner.
Ressearch Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, South- University of Science and Technology, Shenzhen 518055, China. *Pengcheng Laboratory, Shenzhen, China. 'Pazhou Laboratory (Huangpu). *Guangdong Provincial Key Labora- tory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering. Southern University of Sci- ence and Technology, Shenzhen 518055, China. Correspon- dence to: Xiangyuan Lan <lanxy@ppl.ac..m @pcl.ac.cn> Jianguo Zhang <zhangig cchangi@@sustcch.dduu sustoch.cdu.cn>>
Despite its open-ended nature, this model's detection ca- pabilities face notable challenges: it requires large-scale datasets and substantial GPU resources (e.g.. A A100 GPUs) for training, and it suffers from low training efficiency, slow convergence, and suboptimal detection performance. These challenges arise from three main issues: (I) Semantic Gap: This framework directly feeds vision queries from the object detector into the Large Language Model (LLM) to gener- ate object names. Although it employs Vision-Lan Vision-Language
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
alignment loss for these queries, the approach still faces challenges in effectively bridging the semantic gap between the Vision and Language modalities in high-dimensional fea-
of directly using vision queries as input for the LLM in GenerateU, we utilize these VI.pproppts, which effectively bridge the semantic gap between Vision and Language rep- resentations, to boost the training convergence of the LLM.
ture space. This limitation negatively impacts both model training and the accuracy of object name generation. (2) Contradictory Supervision: The framework fails to con- sider the relationships between object categories within the image, resulting contradictory supervisory losses and gradients that degrade training efficiency (the causes of Con- tradictory Alignment Loss are detailed in Sec. A.3). (3) The Heavy-weight LLM Head and Noisy Alignment: Gener- ative LLMs typically operate with a large vocabulary size for text generation (e.g. 32,122 128 tokens in the T5 (Raffel et al., 2020) model, including a heavy-weight head of linear layer with 24. 24 parameters), making training highly chal- lenging. Additionally, during the early stages of training, vision queries are not well-aligned with language modali- ties, leading to noisy alignment. This sub-optimal alignment supervision can disrupt the pre-trained weights of the LLM Head, further reducing the training efficiency and slowing convergence, as detailed in Sec. 3.5.
To further strengthen the alignment between Vision and Lan- guage modalities, we present a new Bidirectional Vision- Language Alignment module VBVL within the Vision- Language Aligner. This module is designed to improve the alignment of vision queries with text embeddings, thereby enhancing training efficiency and overall performance. Ad- ditionally, we introduce a novel Masked Alignment Loss to prevent the emergence of contradictory losses and their as- sociated gradients, as detailed in Sec. 3.6 and Sec. A.3. We also present a Joint Loss for improving positive and nega- tive queries classification (detailed in Sec. 3.6 and Sec. A.4). The main contributions summarized as follows:
We develop a novel and efficient end-to-end OED framework, referred to Open-Det. It significantll ac- celerates training convergence in both the box Detector and the Object Name Generator, and it further distills the knowledge of Vision-Lannuage alignment from a frozen VLM into VL-prompts using the proposed VLD-M, greatly enhancing the training for LLM.
Based on the analysis, a natural question arises: Is there a more efficient OED framewoorrr framework that can accelerate train- ing convergence, enhance training efficiency, and improve detection performance, while eliminating the reliance on large-scale datasets?
We improve training efficiency and performance by op- timizing the alignment of Vision- LanguuLe modalities with BVLA-M, correcting contradictory supervision using Masked Aligument Loss and enhancing classi- fication through a Joint Loss that associates IoU and alignment scores with binary classification scores.
To answer this question, we propose a novel and efficient Open- Been- Detection framework, termed Open-Det. As presented in Fig. 2. Open-Det -Det consists of four key compo- nents: (1) Object Detector (ODR); (2) Prompts Distiller; (3) Object Name Generator; and (4) Vision-Language Aligner. Specifically, Open-Det enhances overall convergence speed in two key aspects: 1) Accelerating the training for the box detector. Inspired by the one-to-many matching ap- proach to improve training convergence (Jia et al., 2023; Zong et al., 2023: Hu al.. 2023), we further design a decoder in Object Detector that incorporates a decoupled one-to-many and one-to-one matching structure. This de- sign accelerates training without requiring an additional branch, reducing training complexity. 2)Accelerating the training for the object name generation. We enhance the training of the Object Name Generator in three ways: firstly, we freeze the heavy-weight head of the LLM (while keeping other weights active) in the early training stage and intro- duce LoRa (Hu et al., 2021) head to accelerate training, which preserves the original pre-trained weights of the head while significantly reducing the trainable parameters of the LLM head; secondly, we design a Text Denoising training approach to facilitate the training and increase the robust- ness for the LLM; thirdly, we propose a Vision-to- La-gguage Distillation Module (VLD-M) in Prompts Distiller to trans- fer the Vision- Language Alignment knowledge from the VLM into newly introduced VL-prompts for LLM. Instead
Open- outperforms OVD models like GLIP(A), achieving +6.8% in AP, and +8.5 % in AP with only 11.7% of training data. Additionally, it demonstrates significant superior efficiency compared to the OED model GenerateU: using just 1.5% of training data, 20.8% of training epochs, and fewer GPUs (4 V100 vs. 16 A100), it achieves a +1.0% improvement in AP,.
2. Related Work
Open- Open-Voca Vocabulary Object Detection (OVD). Traditional closed-set object detection (Ren et al., 2015; He et al., 2017; Carion et al., 2020; Zhang et al., 2022a; Cao et al., 2024) (COD) can identify only fixed object categories, requiring additional bounding box annotations and costly model re- training when new categories are introduced. To address this issue, the OVD task is introduced in OVR-CNN (Zareian et al., 2021b). OVD enables the model to generalize to new object categories without the need for annotations by learning a rich vocabulary during pretraining on large image- caption datasets. Then, a series of methods related to the large-scale Eision-Language Model (VLM) like CLIP (Rad- ford et al., 2021) are utilized to address the challenge of
2

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
limited training data and advance the performance of OVD, including distilling the knowledge from CLIP into detec- tor of ViLD (Gu et al., 2021), learning continuous shared prompt representations of DetPro (Du et al., 2022), extend- ing CLIP to learn region-level visual representations of Re- gionCLIP (Zhong et al., 2022), and forming bags of regions de-
Open-Ended Object Detection (OED). LLM (Raffel et al., 2020; Touvron et al., 2023; Achiam et al., 2023) and multi-modal VLM models (Radford et al., 2021; Li et al., 2022a; 2023), trained on extensive datasets of text or image- text pairs, have demonstrated remarkable model capacity and generalization performance in classification tasks. Nat- urally, integrating these models with detection frameworks presents new opportunities for achieving vocabulary-free detection for arbitrary classes. GenerateU (Lin et al., 2024) first introduces the OED problem, proposing an end-to-end framework that reformulates object detection as a generative task incorporating an LLM. The framework integrates an object detector with an encoder-decoded bbasdd generative L.M,, using vision queries from the detector as inputs for the LLM to enable free-form object name generation. However, this approach struggles with slow training convergence, low efficiency, large GPU resource demands, and high training costs. Thus, efficiently leveraging existing L.L.ss to train OED models with limited detection data remains a critical challenge that needs to be further addressed.
in BARON (Wu et al., 2023a). The transformer-bast tector is also extended to the OVD task. MDETR (Kamath et al., 2021) utilizes wansformer-bassee
architecture that allows for detecting objects conditioned on raw text queries. OV-DETR (Zang et al., 2022) achieves OVD through condi- tional matching. VL-DET (Lin et al., 2022) builds a unified framework to formulate object-language alignment as a set matching problem. CORA (Wu et al., 2023b) enhances the region-text distribution by introducing the region prompting and anchor pre-matching. Although these methods achieve advanced performance, they encounter common challenges related to region-text alignment and limited generalization ability in detecting new categories. Open World Object De- tection (Joseph et al., 2021) introduces a framework that enables the detector to label unknown objects as "unknown" " and incrementally learn these identified unknown categories without forgetting previously learned classes. However, this approach still requires new label priors for each incremental learning phase, restricting its practical applicabilityy
3. Method
3.1. Main Architecture of the Open-Det Framework
As illustrated in Fig. 2, the Open-Det framework comprises four collaborative components. During the training stage, the input image / E R ## x3 (where H and W denote height and width) and its corresponding object names T are fed into the Object Detector and the VLM, respectively. This generates decoder queries Qa for bounding box predic- tion and text embeddings T. for Vision-Language feature alignment. To enhance this alignment, we propose novel Bidirectional Vision-Language Alignment module (BVLA- M) within the Vision-Language Aligner. This module en- hances alignment scores (detailed in Sec. 3.3) and generates alignment indices V Latign for query-text matching, en- abling supervision for the Object Detector.
Recently, learning from a diverse range of data sources, such as image-text pairs and grounding data, has gained more attention in OVD to enhance the generalization ability of visual concepts. Detic (Zhou et al., 2022) employs a dual-branch approach for classification and box prediction, where the class branch is trained on large-scale dataset to achieve OVD through image-level supervision. GLIP (Li et al., 2022b) unifies object detection and phrase ground- ing for pre-training with grounding data and image-text pairs. GLIPv2 (Zhang et al., 2022b) further streamlines the training pipeline and enhances the synergy between local- ization and understanding by integrating localization and Vision-Language pre-training. Based GLIP, Ground- ing DINO (Liu et al., 2023) combines DINO (Zhang et al., 2022a) with grounded pre-training to effectively fuse lan- guage and vision modalities. DetCLIP (Yao et al., 2022) enhances cinuel-concept modeling through a dictionary- enriched framework for parallel OVD pre-training. Its successor, DetCLIPv2 (Yao al., 2023), unifies detection, grounding, and image-text pair data under hybrid supervi- sion approach. The latest iteration, DetCL.PP33 (Yao et al., 2024), advances the architecture with a nigh-performance detector capable of excelling in OVD and generating hier- archical labels for detected objects. Although DetCLIPy3 possesses generative capabilities, its reliance on large-scale datasets (over 50M), complex multi-stage training processes, and substantial GPU resource requirements (32/64 V V100) greatly limit its further development and applicability.
To improve the object name generation, we introduce a novel Prompts Distiller into the OED framework, replac- ing the direct use of vision queries input for LLM in GenerateU. The Prompts Distiller utilizes VLD-M which takes backbone features B. encoder features E, queries Qa. text embeddings T., and V Lalign as inputs, to transfer the Vision-Language alignment knowledge from the VLM (such as CLIP (Radford al., 2021)) to decoder queries Qa. producing VL-prompts (Pot). These VL-prompts are then fed into the LLM (e.g., T5 (Raffel et al., 2020)) to generate more accurate object names. Finally, bounding box predic- tions and binary classification scores are derived from the decoder queries Qu and the object names are generated by the generative LLM. During the inference phase, the Vision- Language Aligner is omitted, achieving vocabulary-free detection and effectively reducing inference complexity.
3

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
Open-Ended
Object Detector
Prompts Distiller
4
Vision-Language Aligner (only used for training)
Detection
backbone features
encoderfeatures E
Masked Aligument
input text
T
Vision- Language Model
Loss
Transformer encoder-decoder
Backbone
VLD Module
V
input image
Joint Loss
T add noise,
binary scores
Binary Head
Bi- Hi-Vision-Languae Alignment
VL-prompts
bounding boxes
Box Head
nolece No0.
N xxd
V-to-L
M xxd,
I.too
object names
LoRa Head
Generative Language Model
decoder queries
text embeddings
output
Object Name Generator
2.
Main architecture of the Open-Det framework. consists of collaborative components: (1) Object Detector (ODR) for accelerating the bounding box training; (2) Prompts Distiller with Vision-to- Vision-to-Langee -anguugg Distillation module (VLD-M) bridge the semantic gap between Vision and Language: (3) Object Name Generator with the Text Denoising approach to accelerate the training of the LoRa Head; (4) Vision-Language Aligner with BVLA-M enhance the alignment of Vision and Language. The Masked Alignment Loss and Joint Loss are introduced for correcting the supervision information and enhancing binary classification consistency, respectively. Please refer to Sec. A. for simplified pipeline.
Figure 2:
3.2. Object Detector
numbers of objects. The process can be formulated as:
The Object Detector (ODR) plays a crucial role in gener- ating bounding boxes and vision queries, which form the foundation for object name generation. Inspired by the ef- ficiency of DINO (Zhang al., 2022a), which enhances training through anchor box denoising, and the success of one-to-many matching in accelerating convergence (Jia et al., 2023; Zong et al., 2023; Hu et al., 2023), we further design novel decoder structure, improving training con- vergence without requiring additional decoder branch and enabling flexible number of objects detection.
Qia = fe; E ((iiearrr)) > A}
where Qia indexes each token e, from the encoder features E, and the decoder queries Qa are selected from E accord- ing Qid- The probability of each token e, being chosen as query is calculated by a Linear head layer, where a the sigmoid function and À the threshold (default: 0.05).
3.3. Vision-Languaae Aligner with BVLA-M
Vision- Langun-Language modalities alignment is a core component in the label-matching process. The alignment score com- puted by measuring the similarity between the vision queries Qa and the text embeddings Te. However, the channel di- mension of Qa is usually smaller than that of Te: for exam- ple, Qa has 256 channels while T. has 768. This significant dimensionality gap results in insufficient information in the vision queries compared to text embeddings. Consequently, directly aligning these two features by mapping Qu into a higher-dimensionall space along the channel dimension, as done in existing methods (Gu et al., 2021: Lin al., 2022: 2024), may lead suboptimal Vision- LanguunLa alignment.
Unlike existing methods (Jia et al., 2023; Zong al., 2023; Hu al., 2023) that rely additional branches heads to combine one-to-one and one-to- matching, we sim- plify the decoder design via decoupling these two matching approaches. Specifically, the first four layers employ one-to- many matching with cross-attention to enhance box local- ization, while the last two layers use one-to-one matching with self-attention eliminate duplicate detections. Addi- flexible OED framework
tionally, aim develop capable of detecting variable number of objects, to address the limitation of existing COD, VOD, and OED frameworks that are restricted to a fixed number of objects due to their reliance on predefined queries. To achieve this, we introduce a threshold- based query selection method that adaptively chooses queries Q& from encoder tokens. This design en- sures both efficiency and flexibility in detecting varying
To address this issue, we propose simple yet effective Bidi- rectional Vision-Language Alignment module (BVLA-M) to enhance the Vision-Language alignment supervision dur- ing training. Specifically, as shown in Fig. 2, we calculate the alignment scores from both Vision-to-Laam Langugg (V-to-L.)

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
text embeddings T,
slligned indice
P.
VL-prompts
Specifically, VLD-M adaptively enriches the background information around Qa by interacting it with the back- bone B and encoder features E through deformable cross- attention (Zhu et al., 2020) operations. This process adap- tively samples token offsets around the query object, allow- ing it to learn relevant background information, as illus- trated Fig. 3. The feed-forward network (FFN) is used to re-weight queries, producing updated Qo and Qe- These updated queries, along with the original Qd, are then passed through MLP soyeer--omppising two linear layers and an activation layer-to project the lower-dimensions queries into the same dimensional space as T. along the channel dimension. Finally, a linear layer fuses these queries, gener- ating the VI.-prompts Put.
Vl
Cusine Similarity Loss
VLM knowledge distillation
Linear
MLP
ML.P
MLP
10
lo,
T
LayerNorm
LayerNorm
FFN
PFN
LayerNorm
e
LayerNorm
Detiormable Cross- Attention with backhone and aseader fisttures for l
Deformable
Deformable Cross-Attentonn
Cross-Attente
Knowledge distillation from the VLM into VL-prompts Pur is achieved using Cosine Similarity Loss supervision to fa- cilitate object name generation. Given the text embeddings Te and aligned indices V Lalign from the BVLA-M, we first select the VL-prompts and text embedding pairs based on these aligned indices and compute the Cosine Similarity Loss between Per and Te, enabling effective knowledge distillation from the VLM the VL-prompis Pet.
backhane features
encuder features
Figure 3: Overall architecture of the proposed VLD-M.
and Language-to- Vision (L-to-V) perspectives to strengthen the alignment between the Vision and Language modalities. This process can be formulated as:
Salign = cos(Quxx MVL. +o0+ Qs. T. x MLV) (2)
3.5. Object Name Generator
where Qu E Nxd, (N is the number of queries and dg denotes the channel dimension), and T. E Mdx (M is the number of text embeddings and de the channel dimen- sion). S Salign N is the alignment score matrix, and cos denotes the cosine similarity. My and MLv are the transformation matrices for V-to-L and L-to-V, respectively.
the early stages of training, the de- coder queries Qa are under-trained and exhibit a large se- mantic gap compared to the text embeddings, leading to noisy and low-quality query-text alignment. Training with such noises risks disrupting the pre-trained weights of the LLM, especially the heavy-weight head layer responsible for text generation, leading to inefficient and noisy training.
Noisy Alignment.
3.4. Vision-to-Language Distillation Module
To address this issue, we freeze the head of the LLM (while keeping other weights active) in early training stage and in- troduce LoRa Head (Hu et al., 2021) to accelerate training (detailed in Sec.A.2), This approach prevents disruption of the pre-trained LLM head weights and significantly reduces the trainable parameters of the LLM head. Furthermore, we observe that when text embeddings T. are fed into the LLM (e.g., T5 (Raffel et al., 2020)) for text reconstruction, the training loss decreases rapidly, indicating T5 model can effectively process these embeddings to reconstruct the text. Leveraging this insight, as shown in Fig. 2, we enhance the robustness of the T5 model by adding Gaussian noise into the text embeddings and feeding them into T5 for text recon- struction. The noise follows a normal distribution (00 where a is the standard deviation of the text embeddings Te. This Text Denoising approach stabilizes training and improves the model's 's ability handle noisy inputs.
The existing VLM models (Radford et al., 2021; Jia et al., 2021a) are usually trained with easily accessible web-scale image-text paired data. In contrast, the ine-ggrined region- text data is hard to obtain due to the challenges of complex labeling and its high cost (Zhong et al., 2022; Wu et al., 2023a;b). This difficulty poses a significant challenge for achieving effective region-text alignment in VLMS.
To address this issue, we propose a Vision-to-Languaag Distillation Module (VLD-M), as presented in Fig. 3. Ex- isting methods like RegionCLIP (Zhong et al., 2022) gen- erate additional lseudo-labels for region-text data using VLMs, while DVDet (Jin et al., 2024) transforms regional embeddings into image-like representations by cropping expanded predicted bounding boxes with handcrafted pa- rameters to include background areas. In contrast, our VLD-M enhances queries by transforming regional em- beddings Qa into image-like representations and further transferring knowledge from the VLM into these queries. This approach effectively bridges the gap between vision queries and lmage-level text embeddings from the VLM.
3.6. Masked Alignment Loss and Joint Loss
In GenerateU, the Vision- Language alignment loss (namely the BCE loss) is used to enhance the alignment between
5

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
queries and text embeddings. This loss strengthens the similarity between matched query-text pairs while applying negative constraints to increase the dissimilarity between the query and all unmatched text embeddings in the mini-bate batch. However, it fails to account for the relationships among object names within the images or mini-batch, leading to contradictory losses and gradients that ultimately reduce training efficiency, as detailed in Sec. A.3.
IoU and alignment scores are more likely to match ground- truth boxes, while those with lower scores are more likely to remain unmatched. Based on this insight, we propose novel Joint Loss to improve binary predictions by inte- grating the binary score with IoU and alignment scores to enhance their consistency. This loss can be formulated as:
1 N Lu
(V Mf -i)' log(p;)+
To address this issue, we introduce a Masked Alignment Loss (MAL), which generates a binary mask M = T. x T. (where M E I MM and M is the number of text embeddings) to compute the similarity between all paired text embeddings. The mask is binarized using a threshold т (default: 0.99), assigning a value of 1 for the same categories and value of 0 to others. The MAL can be formulated as:
(4)
pf V pf of ui 22) - log(1 pi)
where y E {0. 13 is the binary label, while pt, 8,, and uy denote the binary score, alignment score, and IoU score, respectively. N denotes the number of queries. or (default: 0.25) is the scale factor. The square root operation applied to prevent the product of these three scores from becoming too small, which helps maintain numerical stability for train- ing. 'he effectiveness analysis of Joint Loss, along with its differences from Focal loss (Lin et al., 2017) and BCE loss, is detailed in Sec. A.4 of the Appendix. The total losses for model training are detailed See. A.5.
1
LMAL=
NM [(V Latign x M) C log(Satign)
(3)
(1 Lutign x M) @ log(1 Satign))
where N and M are the number of queries and text em- beddings, respectively. The matrix Valiggn E N M contains the aligned format, while Salign E R x represents the alignment score matrix. Our MAL loss works by utilizing this mask to calibrate the alignment labels, thereby preventing the occurrence of contradictory losses and gradients.
indices in one-hot
4. Experiments
We train our model with small set of detection data Visual Genome (VG) (Krishna et al., 2017), which con- tains 77,398 images for training. Following the pioneering work of GenerateU (Lin et al., 2024) in OED, our model is evaluated on the commonly used zero-shot LVIS (Gupta et al., 2019) dataset, which contains 1,203 categories. The COC02017 (Lin et al., 2014) and Objects3665 (Shao et al., 2019) are also used for performance evaluation.
Datasets.
Additionally, our MAL approach differs fundamentally from ScaleDet (Chen et al., 2023) in both methodology and ob- jectives: ScaleDet unifies text labels via semantic similarity (as soft label in MSE) to combine multi-datase training: in contrast, MAL resolves query-text matching conflicts through similarity- BCE updates.
Joint Loss for Positive and Negative Object Predictions. The OED framework detects objects and generates their names automatically during inference, meaning it cannot compute the Vision-I Langugge alignment scores of detected objects for matching and selection of results. Additionally, the number of detected objects is typically large, necessitat- ing deduplication. Therefore, additional head is applied to perform binary classification on the decoder queries Qd. identifying and filtering out duplicate detections.
Evaluation Metrics. Following the evaluation methodol- ogy of the GenerateU framework, we compute the similarity score between generated object names and annotated cat- egory names using a fixed pre-trained text encoder. The evaluation metrics include: average precision for rare cat- egories (AP.), common categories (AP.)). frequency cat- egories (APy), and all categories (AP), respectively. To ensure fair comparison, we adhere to the protocols es- tablished in popular works (Kamath et al., 2021; Li et al., 2022b; Lin et al., 2024), evaluating on the 5k Mini Val subset of the LVIS (Gupta et al., 2019) dataset.
However, the binary labels for each query are dynami- cally assigned based on the Vision-Language alignment and matching process: matched queries are labeled as positive samples, while unmatched queries are labeled as negative samples. This dynamic labeling increases the difficulty of binary prediction for this head. In the Open- -et framework, similar to DETR-like models (Carion et al., 2020; Zhang et al., 2022a; Hu et al., 2023), the matching mechanism eval- uates both the loU score and the alignment score (analogous to the classification score in COD detectors) between pre- dicted boxes and ground truth boxes. Queries with higher
Implementation Details. For a fair comparison, Open- Det utilizes the same backbone model (e.g.. Swin-Tiny and Swin- Large (Liu et al., 2021)) and FlanT5-base (Chung et al., 2024) generative language model as used in the Gen- erateU framework. Unless otherwise specified, our models are trained with a mini-batch size of 8 on 4 Tesla V100 GPUs, using the AdamW optimizer (Loshchilov, 2017) with weight decay of 0.05. The learning rates are configured as
6

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
Table 1: Comparison results of the zero-shot domain transfer on LVIS MiniVal dataset. In row 2, LVIS" indicates that the model is initially trained on GoldG (Kamath et al., 2021) and then fine-tuned using 10% of the LVIS data in MDETR. The GRITSM used for GenerateU training includes million images, selected from the larger grounding dataset of GRIT (Peng al., 2023), which contains total of 90.6M images. The details about the training epoch are presented in Sec. B.1.
Model
Backbone
Pre-Truin Data
Data Size Vocabulary- Freel Epochs
AP.
AP.
AP/
AP
MDETR (Kamath ct al., 2021)
ResNet101
GoIS
0.812M
X
20.9
24.9
24,3
24.2
MaskRCNN (He 2017)
ResNet101
LVIS
0.118M
X
26.3
34.0
33.9
33.3
Deformable DETR (Zhu et al,, 2020)
Swin-Ting
LVIS
0.118M
X
24.2
36.0
38.2
36.0
GLIP(A) (Li et al., 2022b)
Swin-Tiny
O365
0.660M
X
-
14.2
13.9
23.4
18.5
GLIP(C) (Li et al., 2022h)
Swin-Ting Tiny
G365,
1.460M
X
X
17.7
19.5
31.0
24.9
GLJP(C) (1. et al., 2022h)
Swin-Tiny Tiny
0365, GioMdG
5.456M
20.8
21.4
31.0
26.0
Grunding- DINO (Liu al., 2023)
Swin-Ting Tiny
G3l5,
1.460M
X
14.4
19.6
32.2
25.6
Grounding-DiNe DINO (Liu et al., 2023)
Swin-Tiny Tiny
O365..oolGG..asta Cap4M
5.460M
X
18.1
23.3
32.7
27.4
GenerateU (Lin al., 2024)
Swin-Ting
VG
0.077M
149
17.4
22,4
29.6
25,4
GenerateU (Lin al., 2024)
Wwnn-Tiny
VG,GRITSM
5.077M
20.0
24.9
29.8
26.8
Open-Det (ours)
Swin-Tiny
VG
0.077M
31
21.0,3.6
24.8
30.1
27.0
Open-Det (ours)
Swin-Tiny
VG
0.077M
50
21.9
25.
30.4/o.8
27.
GenerateU (Lin et al., 2024)
Swin-Lurge Swin-Small
VG,GRIT5M
5.077M
22.3
25.2
31.4
27.9
Open-Det (ours)
VG
0.077M
31
26.0
28.6
32.8
30.4
Open-Det (ours)
Swin-Large
VG
0.077M
31
31.2
32.1
34.3 33.1
follows: x 10-4 for both the Object Detector and Prompts Distiller, and 2 x 10 for the Object Name Generator.
efficiency, scalability, and effectiveness in the OED task.
Extending Open-Det's training from 31 to 50 epochs yielded a +0.9% improvement in AP, and +0.4% % in AP. Notably, Open-Det trained solely on VG dataset outperforms Gen- erateU (trained on both VG and GRITSM), demonstrating +1.9% % higher AP, and +0.6% higher AP.
4.1. Main Results
Compared to OVD models like GLIP (Li et al., 2022b), Open-Det demonstrates significant advantages by eliminat- ing the need for additional vocabulary priors in inference and achieving higher model efficiency. As shown in Table I, Open-Dec achieves superior performance with substantially reduced training data. When trained using only 0.077M data (11.7% of the data used by GLIP(A)). Open-Det out- performs GLIP(A) by +6.8% in AP, and +8.5% in AP. Even compared to GLIP(C), which is trained on a much larger dataset of 5.456M, Ope still achieves a + +1.0% higher AP while using just 1.4% of the training data. Ad- ditionally, Open-Det's s data efficiency is further validated against Grounding DINO (Liu al., 2023), maintaining consistent advantages with fewer training resources.
We further evaluate Open-Det with two larger backbones: Swin-Small (using 4 V100 GPUs) and Swin-Large (using 4 A800 GPUs). As shown in Table 1, using only 1.5% training data, Open-Det-Swin-S achieves an improvement of +3.7% % in AP, (26.0% vs. 22.3%) and +2.5% in AP (30.4% vs. 27.9%) than GenerateU-SSinnLareee strating its efficiency. When utilizing the larger backbone of Swin-Large, Open-Det-Swinn DeentSwin--arge significantly outper- forms Generatel--Swin-I.e vs. 22.3%) and +5.2% in AP (33.1% 1% vs. 27.9%), further confirming its superior effectiveness and efficiency.
demon-
by +8.9% in AP, (31.2%
Evaluation on COCO and Objects 365. Compared to OVD, OED can directly detect objects in novel data under shot setting without requiring text priors. The results in Table 2 indicate that using less training data, Open-Det -Det outperforms GenerateU by +2.2% AP on COC02017 (Lin et al., 2014) and +3.3% % AP on Objects365 (Shao et al., 2019), demonstrating both higher data efficiency and im- proved detection accuracy.
Compared to the pioneering OED framework of Genera- teU (Lin et al., 2024), Open-Det framework demonstrates faster convergence and higher performance. Specifically, Open-Dec achieves significant performance improvements, with +3.6% % in AP, and +1.6% % in AP, while requiring only 20.8% of the training epochs compared GenerateU. Fur- thermore, even when GenerateU is trained with additional grounding data from GRIT5M (Peng et al., 2023), Open- Det still outperforms it by +1.0% % in APr, while using only 1.5% of the training data. Fig 1 illustrates the performance curves in relation to the training epochs, clearly highlight- ing Open-Det's advantages in both performance and faster convergence speed. These results underscore Open-Det's
4.2. Ablation Studies
Effectiveness of Open-Det Components. In Table 3, the ablation results for each component in Open-Det demon- strate their contributions: (1) The proposed ODR and

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
Table 2: Evaluation on COCO and Objects 365 in a zero- shot setting. The evaluation metric is AP.
Table 4: Ablations on the proposed loss functions.
Methods
Backbone
Pre-Train Data
COCO
Objects365
Masked Alignment Loss
Joint Loss
AP,
APc
AP,
AP
16.9
24.5
29.7
26.3
GenerateU Swin-Large
VG
33.0
10.1
18.6
24.2
30.1
26.6
GenerateU Swin-Large
VG,GRIT
33.6
10.5
19.5
23.9
30.0
26.5
Open-Det Swin-Large
VG
35.8
13.8,3.3
21.0
24.8
30.1
27.0
BVLA-M improve the performance by enhancing both the box convergence and the alignment between Vision and Language modalities. (2) The VLD-M further boosts perfor- mance by transferring knowledge from the VLM to the VL- prompts, achieving notable gains of +1.6% in AP,, +3.9% APc. and +2.8% in AP. This result demonstrates its criti- cal effectiveness in bridging the semantic gap between vi- sion and language domains. (3) The Object Name Generator (ONG), incorporating the LoRa Head and Text Denoising training approach, plays critical role in improving the rare class and overall performance. When combined with spe- cially designed loss Pnaetions-Masker Alignment Loss and Joint Loss Open-Det achieves significant improve- ments of +4.1% % AP, and +0.7% in AP. These results effectively confirm their roles in accelerating convergence, enhancing training efficiency, and improving performance.
than baseline, with AP, and AP increasing by +5.7% % and +1.0%, respectively, confirming their effectiveness.
Table 5: Ablations on elements of Object Name Generator.
LoRa Head
Text Denoising
AP,
AP.
AP;
AP
15.3
23,5
30.1
26,0
v
16.5
24.5
29.9
26.4
19.9
23.2
29.9
26.2
/
21.0
24.8
30.1
27.0
4.3. Improvements in VL Alignment Scores
The Open-Det framework enhances Vision-Language (VL) alignment through two innovative components: the BVLA- M and the VLD-M. The BVLA-M focuses on improving query-text alignment, while the VI.D-M distills image- text knowledge from a pre-trained VLM into aligned VL- prompts, enabling more accurate object name generation. To quantify the improvements in VI. alignment, we compared the alignment scores for Open-Det and GenerateU on the LVIS Mini dataset (over 50,000 object instances) under zero-shot transfer setting. As shown in Fig. 8, Open-Dec Det achieves a higher alignment score of 0.555 & 0.074, sig- nificantly outperforming GenerateU (0.448 = 0.026). This demonstrates the superior capability of Open-Det in achiev- ing robust VL alignment.
Table 3: Ablations the components of the Open-Det. "Losses" means the combination of the MAL and Joint Loss.
ODR BVLA-M VLD-M ONG Losses Ap, AP, AP. AP,
31
10.2
17.4
23.2
19.6
31
13.9
19.8 27.6
23.1
31
14.7
20.3
27.9 23.5
16.3
24.2 29.9 26.3
/
31
16.9
24.5
29.7 26.3
31
21.0
24.8
30.1 27.0
Furthermore, Open-Det outperforms GenerateU (as shown in Table 1), which can be further supported by the higher similarity score between the generated texts and ground truth category names, as illustrated in Fig. 9 of the Ap- pendix. Specifically, Open-Det achieves a similarity score of 0. 0.628 = 0.067, while GenerateU scores 0.620 + 0.077, This demonstrates that the VLD-M distillation method for VL-prompts in Open-Det Det is highly effective, enabling the LLM to generate more precise and accurate object names.
Ablations on the Losses. We propose the Masked Align- ment Loss to eliminate contradictory losses and the Joint Loss to enhance the binary classification. Table 4 indicates that both losses contribute to performance improvements, particularly for rare classes with limited training samples: the Masked Alignment Loss increases AP, by +1.7%, while the Joint Loss boosts AP, by +2.6%. When the two losses combined, the model achieves the best performance, with significant gains of +4.1% in AP, and +0.7% in AP.
4.4. Visualization and Analysis
Ablations Object Name Generator Components. We conduct ablation studies on the LoRa Head and Text De- noising training approach, both designed to enhance LLM convergence efficiency. As presented in Table 5, each com- ponent individually improves performance, especially for rare classes. When combined, they achieve optimal results
As an efficient OED framework, Open-Det provides sig- nificant advantages by detecting potential objects for any images without requiring additional vocabulary priors, mak- ing it highly practical and applicable in real-world scenarios. In Fig. 4, we visualize the detection results on the LVIS Mini data. Compared human labeled annotations and
8

---

Open-Det: Det: An Efficient Learning Framework for Open-Ended Detection
Ground Truth
STOP
LITT EXTIEAS
SiZAT 22.P
GenerateU
dois
CHEE
RARH BATH
bby
pen- Det
aiis
(d)
Figure 4: Visualization results for Ground Truth, GenerateU, and Open- on the LVIS MiniVal dataset. Open-Det demonstrates superior capability in detecting a broader range of potential objects in images (indicated by yellow arrows), covering large-scale objects, small objects, and fie-ggrained details, such as cabinet, rug, light, radiator, and fireplace in (a); sidewalk, shadows, street, and small sign in (b); and sky, sky lift, head, and pole (c); wall, ear, eye, and pillow in (d).
the results of GenerateU, our model shows the superior ca- pability to detect a broader spectrum of potential objects and accurately identify objects across varying sizes and granu- larity in the images, effectively highlighting its advanced capabilities and overall effectiveness. More visualizations and analysis are presented in Sec. C.2 of the Appendix.
Furthermore, integrating a segmentation decoder module into Open-Det framework will offer mask priors for more precise regional and semantic feature extraction. This enhancement can further address semantic gap in cross- modal representations, transforming Open-Det into a uni- fied detection setttion-sgmmttattoon framework and boosting perfor- mance in both tasks.
4.5. Limitations and Future Works
5. Conclusion
Similarly to the existing OED framework of GenerateU, Open-Det's performance primarily constrained by cross- modal semantic discrepancies between visual regions and image-like textual embeddings. These discrepancies arise from the interactions among the backbone, the detector, the VLM, and the LLM. To mitigate this limitation, em- ploying stronger foundation models, such as InternIm- age (Wang et al., 2023) and Strip-MLP (Cao et al., 2023) for vision backbone models, ALIGN (Jia et al., 2021b) and CLM (Wang et al., 2024) for VLM models and DeepSeek (Guo al., 2025) for generative language model, an efficient method for further performance improving. Additionally, training supplementary datasets serve as an effective approach to enhance performance.
This paper presents Open-Det, novel and efficient genera- tive framework for the OED task. It accelerates the model training for both the box detector and the LLM through the specifically designed architecture and enhancing region- text alignment between Vision and Language modalities. This is achieved through key components: the Bidirec- tional Vision-Language Alignment module and the Vision- to-Language Distillation module, incorporated by the pro- posed Masked Aligmment Loss and Joint Loss to further improve training efficiency and performance. In summary, Open-Det advances OED research by offering robust and efficient framework, paving the way for future exploration of more flexible and practical OED approaches.

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
Acknowledgements
Cheng, T., Song, L., Ge, Y., Liu, W., Wang. X., and Shan, Y. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 16901-16911, 2024.
This work was supported in part by the National Key Re- search and Development Program of China (Grant No. 2021YFF1200800 in part by the National Natural Sci- ence Foundation of China (Grant No. 62276121), in part by the National Natural Science Foundation of China (Grant No. 62402252), in part by the TianYua funds for Mathematics of the National Science Foundation of China (Grant No. 12326604). in part by the Shenzhen International Research Cooperation Project (Grant No. GJHZ202209 13 14261 122)) and in part by the Pengcheng Laboratory Research Project (Grants No. PCL2023A08 and PCL2024Y02).
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction finetuned language models. Journal of Machine Learning Research, 25(70):1 1-3, 2024.
Du, Y., Wei, F,, Zhang, Z., Shi, M., Gao, Y., and Li, G. Learning prompt for vocabuauyyl object detec- tion with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14084-14093, 2022.
Impact Statement
This paper introduces a novel and efficient Open-Ended Detection framework that significantly accelerates model convergence, enhances training efficiency and performance, and reduces reliance on large-scale datasets and high GPU resources for training, This enables practical and flexible vocabulary-free open-world detection. These advancements hold significant potential for real-world applications, such as autonomous driving and security, while also advancing research in Open-Ended Ended Detection.
Gu, X., Lin, T.-Y., Kuo, W., and Cui, Y. Open-vocabulaty object detection via vision and language knowledge dis- tillation. In International Conference on Learning Repre- sentations, 2021.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Gupta, A., Dollar, P., and Girshick, R. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r- cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Cao, G., Luo, S., Huang, W., Lan, X., Jiang, D., Wang. Y., and Zhang, Strip-mip: Efficient token interaction for vi- sion mlp. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10....99..999 1494-1504, 2023.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference on Learn- ing Representations, 2021.
Cao, G., Huang, W., Lan, X., Zhang, J., Jiang, D., and Wang, Y. Mlp-dino: Category modeling and query graphing with deep mlp for object detection. In Proceedings of the Thirty-Thine International Joint Conference on Artificial Intelligence, pp. 605-613, 2024.
Hu, Z., Sun, Y., Wang, J., and Yang, Y. Dac-detr: Divide the attention layers and conquer. Advances in Neural Information Processing Systems, 36:75189-75200 2023.
Jia, C., Yang, Y., Xia, Y., Chen, Y.T.,, Parekh, Z., Pham, H., Le, Q. Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
PMLR, 2021a.
Chen, Y., Wang, M., Mittal, A., Xu, Z., Favaro, P., Tighe, J., and Modolo, D. Scaledet: A scalable multi-dataset object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7288- 7297, 2023.
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T, Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and vanion-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904 4916. PMLR, 2021b.
10

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
Jia, D., Yuan, Y., He, H., Wu, X., Yu, H., Lin, W., Sun, L., Zhang, C., and Hu, H. Detrs with hybrid matching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19702-19713 2023.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L.. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Jin, S., Jiang X., Huang, J., Lu, L., and Lu, Lims meet vims: Boost open vocabulary object detection with fine- grained descriptors. arXiv preprint arXiv:2402.04630, 2024.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollar, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2980-990022 298802 2017.
Joseph, K., Khan, S., Khan, F. S., and Balasubramanian, N. Towards open world object detection. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5830-5840, 2021.
Liu, S., Zeng. Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. Grounding dino: Marrying dino with grounded pre- pre-training for open-set object detection. arXiv preprins trXiv:2333.05499 2023.
Kamath, A., Singh, M., LeCun, Y. Synnaeve, G., Misra, I,, and Carion, N. Mdetr- moduuaedddl detection for end-to- end multi-modal understanding. In Proceedings of the IEEF/CVF international conference computer vision, pp. 1780-1790, 2021.
Liu, Z., Lin, Y., Cao. Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, I..-J,, Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In- lernational journal of computer vision, 123:32-73, 2017.
Loshchilov. I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0511 2017.
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Li, J., Li, D., Xiong, C.,, and Hoi, S. Blip: Bootstrapping language-ing -mngge pre-training for unified vision-language understanding and generation. In International confer- ence on machine learning, pp. 12888-12900. PMLR, 2022a.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763. PMLR, 2021.
J., Li, D., Savarese, S., and Hoi, S. 21 Bootstrapping language-image image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19742, PMLR, 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1667, 2020.
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L.,, Zhang, L.., Hwang, J.-N., et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975, 2022b.
Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Lin, C., Sun, P., Jiang, Y., Luo, P., Qu, L., Haffari, G., Yuan, Z., and Cai, J. Learning object-language alignments for ooca-vocabulary object detection. In The Eleventh International Conference on Learning Representations, 2022.
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., and Savarese, S. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEFF/CVF conference computer vision and pattern recognition, pp. 658-6668666 2019.
Lin, C., Jiang, Y., Qu, L.,, Yuan, Z., and Cai, J. Genera- tive region-language pretraining for open-ended object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13958- 13968, 2024.
Shao, S., Li, Z.,, Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., and Sun, J. Objects365: A large-scale, high- quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8430-8439, 2019.

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Zareian, A., Rosa, K. D., Hu, D. H., and Chang, S.-F. Open- vocabulary object detection using captions. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14393-14400 2021b.
Zhang, H., Li, F,, Liu, S., Zhang, L., Su, H., Zhu, J., Ni,
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L.., Li, H., al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVE confer- ence on computer vision and pattern recognition, pp. 14408-14419, 2023.
L. M., and Shum, H.-Y. Dino: Detr with improved denois- ing anchor boxes for end-to-end object detection. arXiv preprint wrXiv:2203.03600 2022a.
Zhang, H., Zhang, P., Hu, X., Chen, Y-CC,, Li, L., Dai, X.,
Wang, L.., Yuan, L.., Hwang, J.-N., and Gao, J. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems, 35: 36067-36080, 2022b.
Wang, W. Lv, Q. Yu, W. Hong. W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., XiXuan, S., et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475- 121499, 2024.
Zhong, Y., Yang. J., Zhang, P., Li, C.,, Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al. Regionclip: Region-based language-image pretraining. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. (6793-16803, 2022.
Wu, S., Zhang, W., Jin, S., Liu, W., and Loy, C. C. Aligning bag of regions for open-vocabulary object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15254-15266 2023a.
Zhou, X., Girdhar, R., Joulin, A., Krähenbühl, P., and Misra, I. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vi- sion, pp. 350-368. Springer, 2022.
Wu, X., Zhu, F. Zhao, R., and Li, H. Cora: Adapting clip for open- open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7031-7040, 2023b.
Zhu, X., Su, W., Lu, L.., Li, B., Wang, X., and Dai, J. De- formable detr: Deformable transformers for end-to-end object detection. arXiv preprint 201.0009..,419 2020.
Yao, L., Han, J., Wen, Y., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, C., and Xu, H. Detelip: ceetionary-enricher visual-concept paralleled pre-training for open-world de- tection. Advances in Neural Information Processing Sys- tems, 55:9125-91353 2022.
Zong, Z., Song, G., and Liu, Y. Detrs with collabora- tive hybrid assignments training. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2023.
Yao, L., Han, J., Liang, X., Xu, D., Zhang, W., Li, Z., and Xu, H. Detelipv2: Scalable open-vocabulary ob- ject detection pre-training via word-region alignment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23497-23506, 2023.
Yao, L., Pi, R., Han, J., Liang, X., Xu, H., Zhang, W., Li, Z., and Xu, D. Detclipv3: Towards versatile generative open-vocabulary object detection. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Zang, Y., Li, W., Zhou, K., Huang, C., and Loy, C. C. Open- vocabulary detr with conditional matching. In European Conference on Computer Vision, pp. 106-122. Springer, 2022.
Zareian, A., Rosa, K. D., Hu, D. H., and Chang, S.-F. Open- vocabulary object detection using captions. In Proceed- ings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18393-144422 2021a.
12

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
A. Appendix for Method.
A.1. Main Pipeline of the Open-Det Framework
In Fig. 2 of the main text, we present the main architecture of Open-Det. To further clarify its workflow, we provide a simplified pipeline in Fig. 5. During training, the input image processed by the Detector to generate queries, which predict bounding boxes and binary classification (positive and negative samples) results. Simultaneously, the text corresponding the objects in the image fed into frozen encoder of the Vision-Language Model (VLM) produce text embeddings. Then, Open-Det distills Vision-Language alignment knowledge from the VLM into queries, generating VL-prompts. These VL-prompts, along with text embeddings augmented with Gaussian noise, are input into the Large Language Model (LLM) for text reconstruction. Finally, the L.M automatically generates the names of the objects detected by the box Detector.
The Text Denoising training approach improves the LLM's training efficiency and robustness. The distillation module bridges the semantic gap between VL-prompts and text embeddings, reducing reliance on large-scale datasets through prompt distillation. The LoRa Head in the LLM reduces trainable parameters, further enhancing the overall training efficiency. As result, model achieves significantly higher performance than Generate while being trained on just V100 GPUs (compared to 16 A100 GPUs used by GenerateU).
queries
bounding boxes
binary classifications
VL-prompts
LoRa Head
Detector
LLM
object names
denoising training
input image
distillation
'person'
'ski pole
VLM
text embeddings
moises
-NO,aa*
only used in training phase
Figure 5: The simplified pipeline of the Open-Det framework. The Vision-Langura Model (VLM) model and input texts are only used in the training phase. The symbols and respectively.
represent that the model weights are activated and frozen,
A.2. Effectiveness of the LoRa Head in the LLM
Generative language models, such as T5, often feature a heavh- head, making training challenging. For example, in the popular T5 (Raffel et al., 2020) model, the head layer responsible for generating object names contains 24.7M parameters, mapping an input dimension of 768 an output dimension of 32,128 (the vocabulary size of T5). By replacing the standard head with LoRa Head, the number of trainable parameters is reduced to 0.526M, which is only 2.12% of the original parameter count. Additionally, the original pre-trained weights of the LLM head are frozen, preventing disruption when noisy training occurs in the early stages. This significant reduction in trainable parameters of LLM Head enhances computational efficiency while maintaining model performance.
A.3. Causes of Contradictory Alignment Loss
In Sec. 3.6 of the main text, we analyze the contradictory losses that arise in the existing query-text alignment loss (Lin et al., 2024) for the OED task. In this section, we provide a detailed explanation of how these contradictory losses and their associated gradients are generated.
The existing alignment loss in Generatel is calculated using the binary cross-entropy (BCE) loss, based on the similarity
13

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
Contradictory Loss
matched
not matched
'hat'
'hat'
postive gradient
'sock'
negtive gradient
'hat'
Matrix of Vision- Language Alignment
'hat'
Score:
"sock"
S aligo E R NxM
text embeddings
T. E Midd
queries 2, R
(a) image with ground truth
(b) alignment process
Figure 6: Illustration of contradictory loss generation in the query-text alignment process.
between the query feature and text embeddings from fixed CLIP (Radford et al., 2021) text encoder. For each object query, the corresponding word in the text encoder is treated as positive sample, while all other words in the same mini-butch are treated as negative samples. This arrangement establishes a contrastive learning framework, where the model maximizes the similarity between the query and its positive text embedding while simultaneously minimizing the similarity with negative text embeddings.
However, this formulation introduces a potential conflict in the optimization process, leading contradictory losses. As shown in Fig. 6 (a), the image contains multiple instances of the same category, such some people wearing hats. For the alignment loss in GenerateU, as illustrated in Fig. 6 (b), only text embedding is matched with each single query (indicated by the red line as positive sample), while all other text embeddings are treated as unmatched (represented by black dashed lines as negative samples). Consequently, the query generates positive loss with the matched text embedding for hat, while simultaneously generating negative losses with multiple unmatched text embeddings (including har). However, the text embeddings generated by the CL.IP encoder for the same text (like "hat") are identical. Therefore, the contradiction arises because the gradients computed for the query feature during back packppropagation are influenced by two opposing forces:
Positive Gradient: The gradient from the positive sample (the corresponding text embedding) encourages the query feature to move closer to the positive text embedding in the latent space.
Negative Gradients: The gradients from the negative samples (all other text embeddings in the mini-batch) push the query feature away from the negative text embeddings.
In summary, when positive and negative text embeddings are semantically similar (e.g., multiple instances of "hat"), contradictory aligmment loss arises, resulting in opposing gradients. This conflict leads to unstable updates and suboptimal aligmment. To address this issue, we introduce the Masked Alignment Loss, which corrects the labels to avoid generating contradictory losses. Experimental results Table and Table 4 demonstrate its effectiveness.
A.4. Joint Loss: Effectiveness Analysis and Comparisons with Focal Loss and BCE Loss
As discussed in Sec. 3.6 of the main text, the positive and negative attributes of detected bounding boxes are predicted by a binary head. During training, the binary classification label for each query is dynamically determined through the matching process with ground truth boxes: matched queries are labeled as positive samples, while unmatched queries are labeled as negative samples. This dynamic matching process is determined by the cost of the GloU (Rezatofighi al., 2019), the LI distance between predicted boxes and ground truth boxes, and the alignment score between queries and text embeddings. During model training, the matching status of each query can dynamically shift between matched and unmatched. This fluctuation may result in conflicting supervision, generating contradictory gradients and causing unstable training.
To address this issue, we propose a novel Joint Loss, defined in Eq. 4, which integrates the loU score u4, Alignment Score 8;s and the binary classification score Pi. To further illustrate its effectiveness, Fig. 7 visualizes the positive weights, negative
14

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
weights, and Joint Loss in relation variations in the IoU score, Alignment score, and binary classification probability.
Surface joint
Surface Negazive joint
Surface Pock joint Loss
0.6
0.6
ci 22 R0
p.0 by and Aggnnnnn g
B 0
0.2
02 Ca 2 Probassily P
02 by
end 4
Reguiiyy P
0
1.
1.0
1.0
1.0
(a)
(b)
(c)
Figure 7: 3D surface plots of (a) positive weights, (b) negative weights, and (c) Joint Loss. To simplify the visualization in 3D space, we assume that the IoU score u, and the alignment score s; in Joint Loss have the same value (i.e. u, 8;) for queries.
In Fig. 7. we simplify the three-dimensional problem (pi. Ui s;) to a two-dimensional plane by assuming un = St This assumption implies that the model exhibits consistent correlation between the predicted bounding boxes and the target categories, meaning that the localization accuracy (IoU) and semantic alignment (alignment score) are aligned (for convenience, refer to these two scores as the IA Score). This simplification facilitates easier visualization and analysis of the relationships among the variables. Specifically, the effectiveness of our Joint Loss can be summarized as follows:
As shown in Fig. (a). when the IA Score and probability p: are close (or consistent) in value, it indicates that the binary head accurately predicts the binary classification, aligning well with both localization accuracy and semantic alignment. In this scenario, the positive weight remains relatively small (in the diagonal area at the center of the figure), which contributes to training stability. This consistency allows the model to quickly adjust its gradients and converge when the matching state changes. Conversely, when the IA Score or probability is significantly higher than the other-indicating inconsistent-i suggests an incorrect probability prediction. This inconsistency stimulates an increase in positive weights (in both the left and right corners of the figure), thereby enhancing the loss of weight for positive samples.
Positive Weights:
During training, positive samples are typically scarce, whereas negative samples are more abundant, leading to a significant imbalance of samples. This imbalance challenges the model to effectively detect the rare positive samples. In addition, large negative weights can drive the binary head's 's predicted probability closer to O, potentially further reducing the prediction of positive samples. To address this, we carefully design the negative weights in the Joint Loss, illustrated in Fig. (b): (1) When the probability p; is close the TA score (indicating they consistent), it suggests that the binary head predicts accurate binary classification score, regardless of whether the current dynamic matching is correct or not. Specifically: if both scores are high or low (along the diagonal in the figure), the probability prediction is relatively accurate; if the IA score is high but the probability score is low (in the right corner of the figure), likely indicates an incorrect label, and the negative weights should be small to avoid introducing incorrect supervision. In both scenarios, small negative weights help mitigate training instability caused by label changes in dynamic matching and maintain the model's 's ability to detect positive samples. (2) Only when the IA score is low and the probability score is high (in the left corner of the figure), do the negative weights increase significantly. This indicates that the current matching and labels are correct, but the binary head's prediction is incorrect. In this case, the negative weights are increased to reduce the predicted probability score for the negative sample, improving the model's predictions.
Negative Weights:
Joint Loss: The 3D surface plot of the Joint Loss is presented in Fig. 7 (c). From the plot, we can observe that: (I) When the IA score and the probability score are consistent (along the diagonal of the figure), it indicates accurate probability predictions, resulting in a smaller loss. (2) In contrast, when they are inconsistent (in both the left and right corners), it suggests inaccurate probability predictions, leading to larger loss.
15

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
In summary, the Joint Loss effectively integrates the alignment score, IoU score, and probability score, mitigating incorrect supervision and training instability caused by dynamic matching. This results in significant improvement in model performance, as demonstrated in Table 3 and Table of the main text. Both theoretical analysis and experimental results validate the effectiveness of the Joint Loss.
Differences from Focal Loss and BCE Loss: Compared to Focal loss (Lin et al., 2017) and BCE loss, the proposed Joint loss adjusts the binary loss weights using consistent weights by associating the loU score, alignment score, and binary score O, adaptively generating a "soft loss". For positive samples, the weight increases as the difference between the binary score and the IA Score grows; for negative samples, the weight increases only when the IA Score is notably high. This design effectively mitigates the negative impact of noisy labels from matched queries during dynamic matching and benefits rare-class detection, enhancing both training efficiency and accuracy. In contrast, Focal Loss and BCE Loss are limited to computing "hard loss" based solely on the matching results of queries, This hard loss can lead to contradictory supervisory and gradients when the matching state of a query changes during the dynamic matching process in model training.
A.5. Composition of Multiple Loss Functions
As presented in Fig. 2, the end-to-end OED framework of Open-Det consists of four collaborative components. The loss functions are also comprised of four parts: the loss from the Object Detector, the loss from the Prompts Distiller, the loss from the Vision-Language Aligner, and the loss from the Object Name Generator. The total loss can be formulated as:
L = LODR + LVLD + LVLA + LONG
(5)
where CODR represents the loss associated with the Object Detector for bounding box generation; LVLD denotes the distillation loss from the Vision-Language Model to the VL-prompts; LVLA indicates the alignment loss between the query and the text embeddings; and LONG refers to the loss from the Large Language Model for generating object names.
The details of these loss functions are as follows:
(1) The Object Detector Loss. This loss is similar those used in most transformer- based detectors (Carion et al., 2020; Zhang et al., 2022a; Hu et al., 2023; Lin et al., 2024), including the GloU (Rezatofighi et al., 2019), L1 loss for bounding box generation. Additionally, it incorporates our proposed Joint Loss for positive and negative object predictions:
CODR = LaoU + LL + LJL
(6)
(2) The Vision-Language Distillation Loss. This loss is the cosine similarity loss for mapping the query to its associate text-embedding:
LVLD = - cosine (Qa, Te)
where Qu and T. represent the query features and text-embeddlnngs, respectively.
(3) The Vision-Language Alignment Loss. This loss is the proposed Masked Alignment Loss:
LVLA = MALL
(8)
(4) The Object Name Generator Loss. Similar to the language modeling loss used in GenerateU (Lin et al., 2024), we adopt cross-entropy loss to train its sequence-to-sequeet Name Generator. This loss measures the difference between the model's predicted token probabilities and the actual target tokens. is a standard loss function for text generation tasks and is widely used in natural language processing (NLP). This loss can be formulated as:
architecture of the T5 (Raffel et al., 2020) model in our Object
N Σ
LONG
Σ i=1
Uh. log((s.))
where N and V represent the number of tokens in the output sequence and the size of the model's vocabulary, respectively. yi.j denotes the ground truth label for the i-th position in the output sequence and the j-th token in the vocabulary. It is a one-hot encoded vector where the correct token is 1, and all others are O. Di.j represents the predicted probability for the i-th position in the output sequence and the j-th token in the vocabulary. This is the probability assigned by the model to each token in the vocabulary.
16

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
B. Appendix for Experiments.
B.1. Details of Training Epochs
As shown in Table of the main text, the results in rows 8, 9, and 11 correspond to the GenerateU and Open-Det models, both trained using Iteration-base -asoodd training on the same VG (Krishna al., 2017) dataset. However, due differences in their training devices and batch sizes, directly comparing the number of iterations is not feasible. To enable fair comparison, convert iterations into epochs, providing consistent measure of how many times each model has the entire dataset, independent of batch size hardware differences.
Specifically, GenerateU (Lin et al., 2024) is trained using 16 A100 GPUs with a batch size of 64 for 180,000 iterations, while Open-Det is trained using V100 GPUs with a batch size of 8 for 300,000 iterations. As presented in Table 6, Open-Det achieves higher performance with only training epochs, which 20.8% of the epochs required by GenerateU (149 training epochs). This highlights that our model significantly accelerates training convergence while maintaining superior performance.
Table Training epochs for GenerateU and Open-Det.
Model
Training Data
Image Size Training Devices Batch Size
Iteration
Epochs
AP,
AP,
AP/
AP
GenereteU (Lin et al. 2024) VG (Krishna al., 2017)
77.398
16 A100
64
180,000
149
17.4
22.4
29.6
25.4
Open-Det (ours)
VG (Krishna 2017)
77.398
4 V100
8
300,000
31
21.0:..
24.8/24
30.1
27.0-1.
B.2. Ablation Study: Freezing vs. Training LLM
The LLM is core component of the Open-Det framework, enabling object category recognition and name generation. To assess its impact, we compare two training strategies for the OED task:
Freezing the LLM weights (only activating the head);
Activating all of the LLM weights.
As shown in Table 7, Open-Det Det framework achieves superior performance compared to GenerateU in both training strategies, with improvements of +5.6 % and +3.6% in APr, and +7.7% and +1.6% % in AP, demonstrating consistent and significant superiority over GenerateU.
Notably, activating the LLM weights of Open-Det further boosts performance, yielding additional gains of +10.5% in AP, and +7.0% in AP. These performance stem from the domain shift between the image-text data used for LLM pre-training and the region-text alignment required for detection tasks. Experimental results demonstrate that Open-Det effectively enhances the region-text alignment, leading superior overall performance.
Table Ablation results on the training strategy of Freezing LLM vs. Training LLM. The symbol *t" indicates that the model was trained by us using the public official code of GenerateU, utilizing 4 V100 GPUs.
Model
Backhone
L.M
Pre-train Data
Data Size
Epochs
AP,
AP.
AP;
AP
Ge (I et al., 2024)
Swin-Tiny
Freeze
VG
0.077M
31
4.9
6.1
19.2
12,3
Open-Det(ours)
Swin-Tiny
Freeze
VG
0.077M
31
10.5
15.0s.n
26.16.9
20.07.7
GenerateU (Lin et al., 2024)
Swin- Tiny
Train
VG
0.077M
149
17.4
22.4
29.6
25.4
Open-Det(ours)
Swin-Tiny
Train
VG
0.077M
31
21.0;a.8
24.8
30.1
27.0
C. Appendix for Results Visualization.
C.1. Visualization of Query-Text Alignment Score and Text Similarity Score
For each matched query, we can compute the query-text alignment score and text similarity score, simultaneously. The query-text alignment score measures the cosine similarity between the query and the matched text embedding. The text similarity score evaluates the cosine similarity of text embeddings between the generated text and ground truth object names.
17

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
In Sec. 4.3 of the main text, we compare the Vision-Language alignment scores (query-text alignment) and text similarity scores of Open-Det and GenerateU (Lin et 2024). The results show that:
Vision-Language Alignment: Open-Det achieves significantly higher query- alignment scores than Generate with scores of 0.555 compared to 0,448, as shown in Fig. 8. This represents +23.88% improvement over GenerateU, confirming Open-Det Deens superior ability to align visual and textual information.
Text Generation Accuracy. Open-Det also achieves higher text-embedding similarity scores between generated object names and ground truth object names, demonstrating its ability to generate more accurate object names.
Query- TTxt Alignment Scores
Query-Text Text Alignment Scores
0.8
0.7
0.6
0.6
Query-Text
0.4
0.5
0.2-
0.4-
GenerateU!
GenerateU Open-Det (Ours) Model Types
GenerateU"
GenerateU Open-Det (Ours) Model Types
Figure 8: Comparison of VL alignment scores between GenerateU and Open-Det using violin and box plots.
In Fig. 9 (a), we visualize the alignment scores and text similarity scores for matched queries in a 2D space. The figure illustrates that Open-Det net achieves higher alignment scores and text similarity scores, with points being more densely distributed. This suggests that Ope is more robust in both query-text alignment and object name generation.
Score (the product the two scores for each query) in a 3D space. The figure shows that most points for Open- ppen-Det are consistently positioned above those of GenerateU, further confirming our model's higher consistency and superior ability accurate alignment and object name generation.
In Fig. (b), we further visualize these two scores alongside the Integrated Algnment-Similarne
C.2. Visualization and Analysis for Open-Ended Detection Results
Due to space constraints, Fig. 4 in the main text includes partial images of zero-shot domain transfer results the LVTS Mini dataset. To further demonstrate the effectiveness of the Open-Det model, provide additional visual comparisons in Fig. 10 and Fig. 11, showcasing results from Ground Truth, GenerateU, and Open-Det. These visualizations offer comprehensive evaluation of the model's 's performance across various conditions, as detailed below:
Variety of Object Types:
The evaluation includes both single and multiple objects, covering specific categories such as humans, animals, vehicles, buildings,fodd and various other objects.
Diverse Scenarios: The results span a wide range of environments, including indoor and outdoor settings, as well as scenes featuring skies, streets, oceans, static scenes, and dynamic scenes.
Scale and Granularity: The visualized objects vary in scale, from small to large, and include both coarse-grained and fine-grained object types.
Specifically, detection results that surpass GenerateU are marked with yellow arrows. Open-Det demonstrates superior performance in the following key aspects of Open-Epenee Ended Detection:
Superior detection ability from coarse-grained to fine-grained objects detection: Open-Det is capable of detecting both coarse-grained and fine-grained objects, demonstrating superior region-text alignment compared to GeneratelJ.
18

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
GenerateU
Open-Det (ours)
GenerateU
Open-Det (ours)
0.8
Alignment Scores Query
X Algnnneet
0.6
0.6
0.5
0.4
0.3
0.0
0.3
0.8
0.2 0,3 0.4 0.5 0,6 7 0g8
Text Text
Text Similarity Scores
0.6
Alignment Scores of Query
0.4
0.2
0.0
0.2
0.4
0.6
0.8
0.0
Text Similarity Scores
(a)
(b)
Figure 9: Visualization of query-text alignment score and text similarity score in zero-shot domain transfer on LVIS Mini Val dataset. (a) Visualization of alignment scores and text similarity scores for each matched query in 2D space. (b) Visualization both scores along with the Integrated Alignment-Similn Score in a 3D space.
For instance, as shown Fig. 10 (a) Open-Det detects coarse-grained objects like large-scale grass and fine- -ne-ggained details such as the bear's eye and mouth, outperforming GenerateU. Similar detection results can be observed Fig. 10 (b) (h) and Fig. 11, across objects of humans, animals, vehicles, buildings, and on.
Capability to detect more potential objects: For example, as shown Fig. 11 (e), Open-Det identifies various objects on the table, such as a lanyard, pen, and even the woman depicted within the book cover, which is not detected by Ground Truth or GenerateU. Similar detection results can be observed in Fig. 10 and Fig. 11.
More accurate object name generation: For example, in Fig. 11 (b), Open-Det generates more precise object names, such as detecting the yellow traffic line on the street as a "yellow line", and identifying the yellow school bus as a "school bus" and "yellow bus", providing more accurate and descriptive categories.
Enhanced detection accuracy for occluded and blurred objects: We also observe that Open-Det excels in detecting occluded and blurred objects. Examples include: 1) In Fig. 10: the occluded elephant in (b), the blurred plants and occluded person in (e), the occluded car in (f), and the blurred shoes (h); 2) In Fig. 11: the occluded skis (with more accurate detection boxes than Ground Truth and GenerateU) in (f) and the occluded person in (h). These instances are highlighted with red arrows for clarity.
In conclusion, Open- -Det framework demonstrates several key advantages: faster training convergence, higher training efficiency with smaller datasets and fewer GPU resources, and superior overall performance. These strengths, supported by the accompanying visualizations, highlight the model's ability to automatically detect a wide range of potential objects from coarse-grained to fine-grained without requiring additional vocabulary priors. This capability holds significant practical value for real-world upplications, such as autonomous driving, security, and intelligent transportation systems.
19

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
Ground Truth
GenerateU
Open-Def
(a)
(c)
(d)
Ground Truth
GenerateU
Open-Det
(
(h)
Figure 10: Visualization of detection results for Ground Truth, GenerateU, and Open-Det models on the LVIS Vil dataset. The scenes feature a variety of object types and levels of granularity.
20

---

Open-Det: An Efficient Learning Framework for Open-Ended Detection
Ground Truth
Generate
Open-Det Det
(b)
(c)
(d)
Ground Truth
GenerateU
Open-Dee Det
(e)
(I)
Figure 11: Visualization detection results for Ground Truth, GenerateU, and Open-Det on the LVIS Mini dataset. The results feature diverse scenarios, such as streets, oceans, skies, and so on.
21


Reviews:
[{"review_id": "DWbnrHbsMp", "reviewer": "Reviewer_EV8Z", "summary": "This paper proposes a new framework for open-ended detection. To address the problem of alignment between vision queries and text embeddings, the authors adopts a Bidirectional Vision-L;anguage Alignment module to obtain alignment score and uses a masked alignment loss to supervise the alignment beween queires and text embeddings. Additionally, a Vision-to-Language Distillation module is used to enhance the query feature to make more align to text embedding. The author conducts experiments on LVIS MiniVal dataset and show the proposed method can achieve better performances to GenerateU with the same pre-train data.\n\n## update after rebuttal\nThe author's rebuttal address my concerns about the comparison with GenerateU. I keep my rating to Weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer EV8Z,\n\n\nWe sincerely appreciate your time and constructive suggestions. We have carefully integrated your suggestions into the manuscript and will include these updates in the next version.\n\n**Q1: (1) Can the proposed ... more training data? (2) Comparing to ... improvement. It would be interesting to see the performance with VG and GRIT5M.**\n\nThank you for your valuable suggestion.\n\n(1) Intuitively and empirically, it is commonly observed that larger datasets can enhance model accuracy in deep learning, as demonstrated in studies such as [1,2,3]. This principle is applicable to the Open-Det model as well. \n\n(2) There are two key reasons why Open-Det was not trained using the combined VG and GRIT5M datasets. **Firstly**, Open-Det's design prioritizes model efficiency, minimizing reliance on large datasets and training resources. Remarkably, when trained on the single VG dataset for only **20.8%** of GenerateU's training epochs, our model *outperforms* GenerateU which used both datasets. This result effectively demonstrates the effectiveness of our approach. **Secondly**, the GRIT5M dataset is **64.6$\\times$** larger than VG dataset, which would require either: \n\n* **64.6$\\times$** more GPU resources for the same training time (for instance, GenerateU was trained using *16 A100 GPUs*), or\n* **64.6$\\times$** longer training time on our *4 V100 GPUs*\n\nThese demands would result in substantial resource requirements and costs for model training. As a result, we excluded GRIT5M training from our initial submitted manuscript.\n\nUnfortunately, due to current limitations in available time (within the rebuttal days) and GPU resources, conducting experiments with the large-scale GRIT5M dataset presents significant challenges. However, following the reviewer's suggestion, we implemented a more computationally efficient alternative by extending Open-Det's training on the VG dataset from 31 to 50 epochs. This modified experimental design serves to:\n\n* Investigate potential performance improvements under practical resource limitations;\n* Validate the model's effectiveness without requiring additional combined dataset training;\n\n\nThe experimental results are as follows:\n\n| Model | Backbone | Pre-Train Data | Data Size | Epochs | AP$_{r}$ | AP$_{c}$ | AP$_f$ | AP |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | \n| GenerateU | Swin-T | VG | 0.077M | 149 | 17.4 | 22.4 | 29.6 | 25.4 |\n| GenerateU | Swin-T | VG,GRIT5M | 5.077M | - | 20.0 | 24.9 | 29.8 | 26.8 |\n| **Open-Det (ours)** | Swin-T | VG | 0.077M |  31 | 21.0 | 24.8 | 30.1 | 27.0 |\n| **Open-Det (ours)** | Swin-T | VG | 0.077M |  50 | **21.9** | **25.1** | **30.4** | **27.4** |\n\nOur extended training experiments yielded *consistent improvements* across all evaluation metrics. Compared to the 31-epoch baseline, additional training achieved: **+0.9%** improvement in AP$_r$ and **+0.4%** improvement in AP. Notably, Open-Det trained **solely** on VG dataset outperforms GenerateU (trained on both VG and GRIT5M), demonstrating **+1.9%** higher AP$_r$ and **+0.6%** higher AP. These comparative results provide further evidence of Open-Det's superior efficiency and effectiveness in the OED task.\n\n**Experiments on Larger Backbones:** In addition, when using larger backbones, such as *Swin-S* and *Swin-L*, Open-Det achieves significant improvements across all metrics. For instance, **Open-Det-Swin-S** obtains an improvement of **+5.0%** in AP$_r$ and **+3.4%** in AP than Open-Det-Swin-T. Using only **1.5%** training data, **Open-Det-Swin-L** *significantly outperforms GenerateU-Swin-L* by ``+8.9%`` in AP$_r$（31.2% vs. 22.3%）and ``+5.2%`` in AP（33.1% vs. 27.9%）, further confirming Open-Det's significantly superior effectiveness and efficiency. For more information, please refer to our responses to the Reviewer **JNJc** in **Q3**.\n\n\n\n**Q2: Figure 2 is complex and it is not easy to understand the entire pipeline.**\n\nThank you for your valuable suggestion. Figure ``2`` may appear complex due to the *extensive detailed information* it contains. In Appendix ``A.1`` (page 12) of the submitted manuscript, we have presented a ``simplified version`` of the Open-Det pipeline (Figure ``5``), which highlights the core workflow and provides a clearer illustration of the entire process for better clarity. Additionally, the caption of Figure ``2`` has explicitly indicated the relationships between the two figures, contributing to a better understanding of the main pipeline. We deeply acknowledge the reviewer's suggestion and will further modify Figure ``2`` by reorganizing the figure layout to make it more understandable. \n\n[1] Grounded language-image pre-training. CVPR 20222.\n\n[2] Grounding dino: Marrying dino with grounded pre-training for open-set object detection. ECCV 2024.\n\n[3] Detclipv3: Towards versatile generative open-vocabulary object detection. CVPR 2024.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. Zero-shot domain transfer on LVIS MiniVal dataset is a popular choice for open set detection.", "theoretical_claims": "No. There are not theoretical claims.", "experimental_designs_or_analyses": "Yes. I check the main experiments and ablation studies.", "relation_to_broader_scientific_literature": "The whole pipeline is similar to GenerateU which follows a pipeline of region proposal generator with a language model to generate object names for visual regions.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n\n1. This paper proposes several reasonable modules to improve the performances of open-ended detection and conduct detailed ablation studies to validate the effectiveness of this modules. These modules contribute consistent performance improvement.\n2. The idea of adopting IoU and alignment scores to enhance the loss function is interesting and the proposed approach inprove the performances significantly.\n3. The proposed framework achieve better performances with less training cost comparing to recent SOTA methods.\n\nWeaknesses\n1. Can the proposed method achieve better performances with more training data? Comparing to GenerateU with VG and GRIT5M dataset, the proposed method trained with VG achieve not very significant performance improvement. It would be interesting to see the performance with VG and GRIT5M.", "other_comments_or_suggestions": "1. The Figure 2 is too complex and it not easy to understand the whole pipeline with Figure 2.", "questions_for_authors": "Please refer to the Weaknesses in Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j2xSrmOt3r", "reviewer": "Reviewer_E2VT", "summary": "This paper presents Open-Det, a novel and efficient framework for Open-Ended Object Detection (OED), addressing the issues of slow convergence, low efficiency, and reliance on large-scale datasets found in existing models like GenerateU. Open-Det consists of four core components: the Object Detector (ODR), the Prompts Distiller, the Object Name Generator, and the Vision-Language Aligner. By introducing the Bidirectional Vision-Language Alignment module (BVLA-M) and the Vision-to-Language Distillation Module (VLD-M), Open-Det effectively bridges the semantic gap between vision and language. Additionally, the Masked Alignment Loss and Joint Loss further optimize training efficiency and classification consistency. Compared to existing models, Open-Det achieves superior performance using only 1.5% of the training data and 20.8% of the training epochs, while also detecting a broader range of objects, including smaller ones. Open-Det provides a highly efficient and scalable solution for OED tasks, with significantly reduced resource requirements.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer E2VT,\n\nWe deeply appreciate your suggestions and will include them into our revisions.\n\n**Q1: Training free VL-SAM .. better.**\n\nA direct performance comparison requires careful consideration of **model scales**:\n\n* **VL-SAM**: VLM: 17B, LLM: 7B, SAM: 0.632B, totaling ``24.632B`` \n* **Open-Det**: VLM: 0.307B, LLM: 0.250B, Detector: 0.049B,  totaling ``0.606B``  (ONLY **2.46%**)\n\nWith significantly **less** parameters (**40.6$\\times$** ↓), Open-Det is still on par with VL-SAM in AP$_c$ (24.8% vs. 25.3%) and even **higher** in AP$_f$ (30.1% vs. 30.0%), indicating better trade-off between resouce and performance.\n\n\n**Q2: (1) How ... ambiguous? (2) What's ... accuracy? (3) How ... consistent results?**\n\nLanguage (text) inherently exhibits ambiguity, a characteristic **independent of the design of OED models**. \n\n(1) Similar to GenerateU, Open-Det also adopts ''*beam search*'' strategy to generate ``multi-level object names`` , effectively addressing the ambiguity issue and enhancing accuracy. \n\n(2) For fair evaluation, we adopt GenerateU's **standardized protocol**: comparing latent space similarities score between generated and annotated texts, *eliminating evaluation biases* from textual variability. \n\n(3) The adopted *beam search* strategy ensures consistent results, and Open-Det achieves higher text similarity scores (Appendix Sec. ``C.1``) than GenerateU.\n\n\n**Q3: (1) How  ... same category? (2) Does ... conflicting gradients, or (3) could  ... detection tasks?**\n\n(1) MAL transfers one-to-one matching (query to category, Fig. ``6``) to one-to-many matching (single query to multiple same-category instances) via:\n\n* **Computes text similarities** between all object-name pairs, offering a quantitative basis for dynamic label matching correction.\n\n* **Corrects dynamic label matching** (from 0 to 1) by transforming unmatched same-category pairs into matched ones (one-to-many) based on text similarities, resolving contradictory loss terms during optimization.\n\n(2) Yes, MAL efficiently addresses contradictory loss generation at its source. \n\n(3) MAL depends **solely** on textual ground truth, making it invariant to scene variations, but may be related to the annotations accuracy and text encoder.\n\n\n**Q4: (1) The use of AP ... category names? (2) how ...  descriptions (e.g., ...)?**\n\n(1) The evaluation assesses **both** box accuracy (IoU) and object naming performance (via cosine similarity (CS) score between **generated** and **annotated** texts). If object name is wrongly predicted, the CS score will be lower, thereby reducing the performance of AP.\n\n(2) Generated name granularity varies by training data annotations. Similar to GenerateU, Open-Det focuses on OED task, using **detection annotations** (i.e.,  object names and boxes) without utilizing *complex object descriptions*. Thus, Open-Det mainly predicts object categories (Fig.``4``, ``10``, ``11``), rather than fine-grained descriptions (i.e., obtaining relations between objects). We view the *task* of finer-grained name generation as an important area for future research.\n\n\n**Q5: The ... Grounding DINO or ... CoDET.**\n\nThe comparison between Grounding DINO and Open-Det are as follows:\n\n| Model | Train Data | Data Size | AP$_{r}$ | AP$_{c}$ | AP$_f$ | AP |\n| :---: |  :---: | :---: | :---: | :---: | :---: | :---: | \n| Grounding-DINO-SwinT | O365,GoldG | 1.460M | 14.4 | 19.6 | 32.2 | 25.6 |\n| **Open-Det-SwinT(ours)** | VG | 0.077M | 21.0 | 24.8 | 30.1 | 27.0 |\n\n\nUsing only **5.3%** training data, Open-Det-SwinT still significantly **outperforms** Grounding-DINO-SwinT by ``+6.6%`` in AP$_r$, ``+5.2%`` in AP$_c$,  and ``+1.4%`` in AP, further confirming its effectiveness and efficiency. \n\nCoDET [1] is designed for programming problems and has NOT been evaluated on object detection tasks. Due to fundamentally different objectives (**Code Generation** vs. **Object Detection**), a direct detection performance comparison is invalid. We also cross-check Co-DETR [2], which is designed for closed-set detection and can't be directly compared to our OED model.\n\n\n**Q6: (1) While ..., ... low-data regimes, such as ...? (2) Does ... training set?**\n\n\n(1) Like most OVD and OED models, Open-Det also faces common challenges related to low-data regimes and few-instance learning. \n\n(2) However, Open-Det demonstrates superiorities in both **efficiency** and **generalization** via VLM knowledge distillation into proposed VL-prompts. Notably, using significant **less** training data, it obtains superior performance on ``rare`` categories (namely only few instances of new category): **+6.8%** AP$_r$ over GLIP(A), **+6.6%** AP$_r$ over Grounding-DINO-SwinT, and **+3.6%** AP$_r$ over GenerateU. Using ``1.5%`` training data, Open-Det-SwinL outperforms GenerateU-SwinL by ``+8.9%`` in AP$_r$ (Please see our responses to **JNJc** in **Q3**)\n\n\n[1] Codet: Code generation with generated tests. ICLR 2023.\n\n[2] Detrs with collaborative hybrid assignments training. ICCV 2023.", "claims_and_evidence": "- The claims made in the submission are generally supported by clear and convincing evidence. The paper provides experimental results that demonstrate the superiority of the Open-Det framework over existing models such as GenerateU and GLIP in various aspects, including performance, training data requirements, and convergence speed. Specifically, the evidence shows that Open-Det achieves higher performance with significantly fewer resources, requiring only 1.5% of the training data, 20.8% of the training epochs, and fewer GPUs, which is supported by quantitative results in the tables and performance curves.\n- I have find a work named VL-SAM, they achieved 23.4 APr in LVIS, they are training free, so the resource and performance is better than yours.", "methods_and_evaluation_criteria": "- In the Open-Ended Detection task, objects can belong to multiple categories or have ambiguous semantics (e.g., \"dog\" vs. \"pet\"). How does Open-Det handle ambiguous or overlapping categories during inference, especially when there are multiple plausible names for a detected object? What is the impact of this ambiguity on the accuracy of object name generation, and how does the model ensure consistent results under these circumstances?\n- The proposed Masked Alignment Loss (MAL) aims to address contradictory supervision, but how does this loss behave in practice when applied to images with multiple objects of the same category (e.g., several \"cars\" in a street scene)? Does the MAL genuinely prevent conflicting gradients, or could there still be scenarios where the supervision conflicts due to the inherent ambiguity in object detection tasks (e.g., detecting small objects in cluttered scenes)?\n- The use of average precision (AP) and related metrics is standard, but how well do these metrics capture the true performance of Open-Det in open-ended object detection, where the focus is not just on detection but also on generating the correct category names? For example, how would Open-Det fare in tasks that require fine-grained distinctions between objects or the generation of complex object descriptions (e.g., detecting \"a small red ball on the table\" vs. \"a ball\")?", "theoretical_claims": "This work does not involve too much new mathematical theory or proofs.", "experimental_designs_or_analyses": "In the comparison experiments, the authors chose GLIP, which performs relatively poorly in OV detection, as a baseline. This seems somewhat unfair, as there have been many OV detectors that have achieved higher accuracy on LVIS after GLIP, such as the computationally intensive GroundingDINO or the more efficient CoDET, among others.", "relation_to_broader_scientific_literature": "- The generative nature of Open-Det (which generates object names in addition to bounding boxes) relates to the work done on generative approaches in vision-language tasks, such as **image captioning** and **visual question answering (VQA)**, where models generate descriptive text based on visual inputs. The paper’s approach of using a **Large Language Model (LLM)** for name generation is in line with recent advancements in **transformer-based generative models** for language, like **T5** (Raffel et al., 2020).\n- The paper incorporates a **Vision-to-Language Distillation Module (VLD-M)** for transferring knowledge from pre-trained Vision-Language Models (VLMs) into VL-prompts. where knowledge from a teacher model is transferred to a smaller or more efficient student model. In this case, the VLD-M distills alignment knowledge from powerful pre-trained VLMs (like CLIP) to improve the training of the Object Name Generator.", "essential_references_not_discussed": "Essential references are discussed.", "other_strengths_and_weaknesses": "While Open-Det demonstrates impressive results with significantly reduced data, could the approach still face limitations in very low-data regimes, such as when there is an extremely low number of labeled examples for certain object categories? Does the framework struggle to generalize in scenarios where only a few instances of a new category are available, or when the object categories are underrepresented in the training set?", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t98tLOduI9", "reviewer": "Reviewer_AMke", "summary": "This paper introduces **Open-Det**, which addresses the inefficiencies of previous open-ended methods, including slow training and high memory consumption. Open-Det achieves improved efficiency and performance through:  \n\n1. Enhancing the object detector,  \n2. A vision-language (VL) alignment module,  \n3. A VL distillation module,  \n4. LoRA for parameter-efficient fine-tuning,  \n5. A noise-adding and denoising strategy,  \n6. Masked Alignment Loss.\n\n## update after rebuttal\nDuring the rebuttal, the authors repeatedly used terms like “factual error” in an attempt to assert the originality of their method, which I find unconvincing. In my view, the proposed approach does not appear substantially different from prior work—though this may require further verification by other reviewers or the AC. That said, I believe there is nothing wrong with building upon existing methods; standing on the shoulders of giants is how we advance the community. However, in the method description section, the authors failed to acknowledge this connection. I trust that the proposed modules were likely inspired by previous work, and this omission is a notable weakness. I will maintain my original score and leave the final judgment to other reviewers and the AC.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer AMke,\n\nWe thank your time and feedback. However, we respectfully disagree with comments (Q1 to Q7) and the assertions（the omitted citations and misleading novelty) for the following reasons:\n\n**Q1: The ..., yet ... cite ... novelty.**\n\n(1) **Reviewer's Factual Error:** The reviewer claimed ***3 times***: we **failed to cite any** of the 5 specified papers. In fact, we have explicitly **cited** ``3`` out of 5 in **multiple places** in main paper: Deformable DETR [1] (*Line 338*), DINO [2] (*Lines 87, 132, 152, 295, 796*), and DAC-DETR [4] (*Lines 79, 155, 161, 295, 797*). The assertion that *we deliberately omitted citations - and thus misleading novelty* - has ``no ground and constitutes a factual inaccuracy requiring correction``.\n\n(2) **Exclusion of [3,5]:** DAB-DETR [3] differs from Open-Det in *task* and *method* (e.g., Q2). Similarly, ScaleDet [5] tackles label unification for multiple-dataset training, differing from our focus in **Q7**. We initially excluded them but may be included in Related Works if space permits. \n\n(3) Open-Det enhances detection across object granularities (coarse-to-fine) and scales (large-to-small) (not solely in training acceleration). Notably, it demonstrates *additional vocabulary priors are not necessary in inference*, with ***11.7%*** training data achieving **+6.8%** AP$_r$ and **+8.5%** AP over GLIP(A) of OVD. We believe our work establishes a **foundational framework** for future OED research.\n\n**Q2: Object is ... found in [1][2][3].**\n\n**Reviewer's Factual Error:** \n* **[1]**: Obtains Content Query (CQ)/Positional Query (PQ) via Top-K selected sinusoidal-encoded anchors (**no threshold**);\n\n* **[2]**: Initializes CQ statically, derives PQ from Top-K boxes (**no threshold**);\n* **[3]**: Uses static initialization for both CQ/PQ (**no threshold**)\n\n**Innovation of Open-Det:** Different from [1-3], Open-Det introduces a NOVEL single adaptive query type (rather than CQ and PQ), transforming fixed number of objects prediction into flexible.\n\n\n**Q3: Object ... DAC-DETR.**\n\n\n* **Reviewer's Factual Error:** DAC-DETR [4] employs a dual-decoder architecture (standard O-Decoder and auxiliary C-Decoder) to enhance training efficacy, rather than dividing the first and last two decoder layers.\n\n* **Advantages of Open-Det:** In contrast, our Open-Det directly optimizes the architecture of standard single-decoder (Sec.``3.2``), enabling lower training costs and superior efficiency.\n\n\n**Q4: Vision ... deformable attention (DA).**\n\nThis claim **misinterprets** our contribution. We did **NOT** claim DA is our contribution. It is not suprise that DA is widely used. We also follow this practice. However, one of our novelty is the design of **VLD-M** (Fig.``3``), leveraging it to construct **image-like** queries for visual-text alignment. This represents both a **novel architectural design** and **distinct application**.\n\n \n**Q5: Vision ... supervise ...**\n\n* This claim without gives any references. **Text-embedding to supervise query feature is NOT a common trick but a priciple paradiam for vision-text alignment.** Exisiting mehtods differ in how the supervison is done, *rather than whether or not simply using it*. The reviewer's claim is NOT professional. \n\n* **Innovation of Proposed VLD-M:** Our VLD-M aims to adaptively construct **image-like** queries to reduce **region-text annotations** (Sec.``3.4``), improving AP$_c$ by **+3.9%** and AP by **+2.8%** (Table ``2``). \n\n**Q6: ONG: Noisy ... [2].**\n\nDINO adds *biased coordinates noise into box* to accelerate *box convergence*. However, Open-Det adds *Gaussian noise into text embeddings* of VLM to speed up *LLM*'s convergence. They **differ fundamentally** in both objectives (**box vs. LLM**) and noise types (**biased coordinates vs. Gaussian**).\n\n**Q7: MAL: ... ScaleDet [5].**\n\n* **Reviewer's Factual Error:** MAL is NOT used for text-classifier but for query-text alignment. \n\n* **Text Label Unification ≠ Multimodal Alignment:** They are **fundamentally different** in motivation and method. ScaleDet unifies text labels via semantic similarity (**as soft label in MSE**) to combine multi-dataset training, while MAL resolves query-text matching conflicts through **similarity-binarized BCE** updates (Sec.``3.6``). \n\n**Q8: Missing ... .**\n\n* We conducted test on COCO. Open-Det still outperforms GenerateU by **+2.8%** AP (35.8% vs. 33.0%).\n\n* Furthermore, additional experiment shows that Open-Det-SwinL surpasses GenerateU-SwinL by ``+8.9%`` in AP$_r$ on LVIS (see **Q3** responses to **JNJc**), demonstrating significant superiorities.\n***\n\nAccording to our responses to Q1~Q8, our method is **NOT** an adaptation of existing techniques, but rather respresents a fundamentally different approach, innovating in framework, modules, loss functions, performance and efficiency.\n\n\n``Overall, we respectfully request the reviewer to cross-check the citations, methodologies, as well as all our responses, and we appreciate a fair judgement.``", "claims_and_evidence": "The analysis and claim of **GeneratU** in the paper is clear and convincing.", "methods_and_evaluation_criteria": "**My biggest concern is that the proposed acceleration methods for open-ended detection are adaptations of techniques already introduced in traditional object detection, yet the authors fail to cite any of them. This omission gives the impression of misleading novelty.**\n\n3.2 Object Detector: **a threshold-based query selection** is essentially the common two-stage approach found in [1][2][3] and so on.\n\n3.2 Object Detector: *divide the first layer and last two layer* is similar to the idea presented in [4].\n\n3.4. Vision-to-Language Distillation: *using deformable cross-attention to attend encoder feature* is exactly the same as [1]\n\n3.4. Vision-to-Language Distillation: *using text-embedding to supervise query feature* is a common trick used in open-vocabulary (Almost every method uses this approach.)\n\n3.5. Object Name Generator: *Noisy Alignment.* The noise-adding and denoising strategy is the same as in [2].\n\n3.6. Masked Alignment Loss: *Mask text-classifier based on feature similarity* is the same as [5].\n\n*The key issue is that the authors did not cite any of the aforementioned works, which raises serious concerns. Given that many of these methods are well-known, it is unlikely that the authors are unaware of them. Instead, it appears that they may have deliberately omitted these citations, which could be misleading.*\n\n[1] Deformable Transformers for End-to-End Object Detection\n\n[2] DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection\n\n[3] Dynamic Anchor Boxes are Better Queries for DETR\n\n[4] DAC-DETR: Divide the Attention Layers and Conquer\n\n[5] Scaledet: A scalable multi-dataset object detector", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Missing results on COCO and Obje365, which is common Open-end dataset and evaluated in in GenerateU", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Please see method section. I firmly believe that the vast majority of contributions presented in this paper were originally proposed and validated in the object detection domain. However, the authors neither discuss nor cite these prior works, which raises concerns about the completeness and transparency of their literature review.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fEegJ8Ci5H", "reviewer": "Reviewer_JNJc", "summary": "In this paper, the authors present a novel learning framework for open-ended detection. Open-ended detection consists in detecting objects that are not known a priori. The proposed framework is based on four main components: 1) an open object detector, 2) a prompt distiller, 3) an object name generator and 4) a vision language alignment module. the model is trained with a Visual Genome dataset (about 77K images). The pproposed model is compared to GenerateU, the SOTA model in open-ended detection, a new task in Deep Learning based Computer Vision. The proposed model slightly outperforms GenerateU in terms of AP. Moreover, Open-Det is more efficient in terms of training convergence (5 less epochs than GenerateU) and training data. The ablation study shows the effectiveness of each component of the proposed model. The paper is well written and the experiments are well conducted. This paper is a good contribution.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer JNJc,\n\nWe sincerely appreciate your valuable feedback and have incorporated all suggestions into the manuscript for the next version.\n\n\n**Q1: OED is close to OWD. It could ... section.**\n\nWe have carefully checked the OWD [1], which enables the detector to label unknown objects as ''unknown'' and incrementally learn these identified unknown categories without forgetting previously learned classes. In contrast, our method could directly detect novel/unseen categories without requiring any prior human knowledge. In other words, both OWD and OED are capable of detecting and recognizing new categories, demonstrating their relevance. Following your suggestion, we will include this work in the related work section in the new version.\n\n**Q2: The authors ... insights ... improve it.**\n\nSimilarly to the existing OED framework of GenerateU, Open-Det's performance is primarily constrained by cross-modal semantic discrepancies between **visual regions** and image-like **textual embeddings**. These discrepancies arise from the *interactions* among the detector, the VLM, and the LLM. To mitigate this limitation, employing stronger foundation models, such as ALIGN and CogVLM-17B for VLM models and DeepSeek for generative language model, is an efficient method for further performance improving. Additionally, training on supplementary datasets can serve as an effective approach to enhance performance.\n\nFurthermore, integrating a segmentation module into Open-Det framework will offer mask priors for more precise **regional** and semantic feature extraction. This enhancement can further address semantic gap in cross-modal representations, transforming Open-Det into a unified **detection-segmentation** framework and boosting performance in both tasks. We will incorporate this discussion in the new version and propose it as a direction for future research.\n\n\n\n**Q3: Why the ... Swin-L model ... GenerateU?**\n\nCompared to Swin-T, the Swin-L model has a significantly larger number of parameters (**197M vs. 29M, a difference of 6.8$\\times$**) and supports a higher input image size (384 $\\times$ 384 vs. 224 $\\times$ 224, with GenerateU using a pretrained Swin-L at a resolution of 384 $\\times$ 384). This results in a substantial increase in GPU memory usage during model training (e.g., Swin-L requires approximately 4 $\\sim$ 5 times more memory than Swin-T), making it **infeasible** for Open-Det to train on 4 V100 GPUs. \n\nIt is fair and effective to evaluate the performance of GenerateU and Open-Det models with the **same backbone** (such as Swin-T). Therefore, we did not use Swin-L for performance comparisons in the first instance. \n\nHowever, as suggested, we managed to conduct **additional experiments** with two backbones: Swin-S (using 4 V100 GPUs) and Swin-L (using 4 A800 GPUs). The main experimental results are as follows:\n\n| Model | Backbone | Train Data | Data Size | Epochs | AP$_{r}$ | AP$_{c}$ |  AP$_f$ | AP |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | \n| GenerateU | Swin-L | VG,GRIT5M | 5.077M | - | 22.3 | 25.2 | 31.4 | 27.9 |\n| **Open-Det (ours)** | Swin-T | VG | 0.077M | 31 | 21.0 | 24.8 | 29.8 | 27.0 |\n| **Open-Det (ours)** | Swin-S | VG | 0.077M | 31 | 26.0 (+3.7%) | 28.6 (+3.4%) | 32.8 (+1.4%) | 30.4 (+2.5%)|\n| **Open-Det (ours)** | Swin-L | VG | **0.077M** | 31 | **31.2 （+8.9%）** | **32.1 （+6.9%）** | **34.3 （+2.9%）** | **33.1 （+5.2%）** |\n\n\n**Results:**\n* Open-Det-Swin-S achieves an improvement of **+5.0%** in AP$_r$ (26.0% vs. 21.0%) and **+3.4%** in AP (30.4% vs. 27.0%) than Open-Det-Swin-T, demonstrating its efficiency. \n\n* Using only **1.5%**  training data, **Open-Det-Swin-S** outperforms *GenerateU-Swin-L* by **+3.7%** in AP$_r$ and **+2.5%** in AP. \n\n* When utilizing the larger backbone of Swin-L, **Open-Det-Swin-L** significantly outperforms *GenerateU-Swin-L* by ``+8.9%`` in AP$_r$（31.2% vs. 22.3%）and ``+5.2%`` in AP（33.1% vs. 27.9%）, further confirming its superior effectiveness and efficiency. \n\nWe will include these results in the new version.\n\n\n\n**Q4: Did you try to ... is improved?**\n\nIntuitively, more training data would further improve the performance of Open-Det. Unfortunately, due to current limitations in available time (within the rebuttal days) and GPU resources, conducting experiments with the large-scale dataset (such as GRIT5M) presents significant challenges. \n\nAs an implementable scheme, we further extended Open-Det's training on the VG dataset from 31 to 50 epochs to investigate potential performance improvements under practical resource limitations. The results show that extended training further improves the performance with **+0.9%** (21.9% vs. 21.0%) in AP$_r$ and **+0.4%** (27.4% vs. 27.0%) in AP, confirming its effectiveness. However, we believe it remains valuable to investigate the impact of training data scale on performance for furture studies when more GPU resources become available.\n\n\n[1] Towards open world object detection. CVPR 2021.", "claims_and_evidence": "The authors claim that the proposed model is more efficient than GenerateU in terms of training convergence and size of training data. The authors provide evidence that the proposed model outperforms GenerateU in terms of AP. The ablation study shows the effectiveness of each contribution of the proposed model.", "methods_and_evaluation_criteria": "/", "theoretical_claims": "/", "experimental_designs_or_analyses": "/", "relation_to_broader_scientific_literature": "/", "essential_references_not_discussed": "To my point of view, OED is close to Open Word Object Detection (OWD). Joseph, K. J., Khan, S., Khan, F. S., and Balasubramanian, V. N. Towards open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5830–5840, June 2021.\n\nIt could be interesting to cite OWD in the related work section.", "other_strengths_and_weaknesses": "++ Strong Points:\n- The paper is well written and the experiments are well conducted.\n- The proposed model is more efficient than GenerateU in terms of training convergence and size of training data.\n- The ablation study shows the effectiveness of each component of the proposed model.\n- There are many good insights in the paper: 1) a variable number of queries from encoder tokens in the object detector part, 2) the vision-language alignment module with the bidirectional alignment, 3) the joint loss function that  merges binary score with IoU and alignment score. 3) improves the performance for rare objects by 4%.\n\n\n-- Weak Points:\n\n- Although the proposed model is more efficient than GenerateU, the performance gain is not significant. The authors should provide more insights on the limitations of the proposed model and how to improve it.\n- In the GenerateU paper, the authors achieved a better performance by using a Swin-L model. Why the authors did not use this model in order to show if Open-Det can also benefit from this BB and outperform GenerateU?\n- The training data is small (77K images). Did you try to train the model with a larger dataset and to see if the performance is improved?", "other_comments_or_suggestions": "/", "questions_for_authors": "Did you try to train the model with a larger dataset and to see if the performance is improved?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by four experts in the field, receiving mixed evaluations: two Weak Accepts, one Accept, and one Reject. Overall, most reviewers provided positive feedback. The authors have addressed the majority of the concerns raised during the review process. The main remaining issue, raised by reviewer AMke, pertains to citation practices. However, the AC acknowledges that the authors have provided a detailed response to this concern. It is recommended that the authors carefully revise the final version by adding appropriate citations and clearly highlighting the differences from related work. As a result, the final decision is to accept the paper.
Decision: Accept (poster)