Uni-YOLO: Vision-Language Model-Guided YOLO for Robust and Fast Universal Detection in the Open World

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 08:58:05

Abstract: Universal object detectors aim to detect any object in any scene without human annotation, exhibiting superior generalization. However, the current universal object detectors show degraded performance in harsh weather, and their insufficient real-time capabilities limit their application. In this paper, we present Uni-YOLO, a universal detector designed for complex scenes with real-time performance. Uni-YOLO is a one-stage object detector that uses general object confidence to distinguish between objects and backgrounds, and employs a grid cell regression method for real-time detection. To improve its robustness in harsh weather conditions, the input of Uni-YOLO is adaptively enhanced with a physical model-based enhancement module. During training and inference, Uni-YOLO is guided by the extensive knowledge of the vision-language model CLIP. An object augmentation method is proposed to improve generalization in training by utilizing multiple source datasets with heterogeneous annotations. Furthermore, an online self-enhancement method is proposed to allow Uni-YOLO to further focus on specific objects through self-supervised fine-tuning in a given scene. Extensive experiments on public benchmarks and a UAV deployment are conducted to validate its superiority and practical value.

Main paper: 
 introduction: 
 The dependence on human annotations and the numerous categories present in the open world significantly limit the universality of traditional object detectors. In complex and variable environments, it is unfeasible to collect and annotate all data for every scene [6, 27, 28, 53, 56] . To address these data limitations, a universal visual object detector is necessary [44] . The universal detector aims to detect any object (open vocabulary) in any scene (open world) and can refine itself in a new scene without human annotations.Recently, some large language models (LLMs), such as GPT [1] , and ERNIE [40] , demonstrate superior generalization performance in natural language processing. Researchers are now exploring how to extend the generalization capabilities of LLMs to visual models. Some large-scale vision-language pre-training models, such as CLIP [31] , have been proposed. In the field of object detection, open vocabulary detectors (ov-detectors) [10, 17, 26, 44, 51, 57] use CLIP [31] to recognize unknown categories. However, these current ov-detectors face limitations in addressing two important aspects, which restrict their universality. 1) The universal detector should have better generalization and robustness to detect objects in the open world. Although some novel categories can be detected by existing ovdetectors, detecting a wide range of unknown categories in the open world remains a challenge [44] . Moreover, current ov-detectors exhibit poor robustness in harsh weather conditions, and real-world scenes are inevitably affected by factors such as scattering and low illuminance, which lead to unsatisfactory generalization of existing detectors.2) The universal detector should also have better real-time performance to enable deployment on mobile platforms. Most existing ov-detectors rely on the Faster RCNN architecture [35] , which has a two-stage architecture that prioritizes detection accuracy in a closed set, albeit at the expense of real-time performance. A more efficient architecture for universal detectors will facilitate deployment on mobile platforms and enhance the practical application value. Thus, unlike ov-detectors, a universal detector should have better openworld generalization, robustness, and efficient architecture.To achieve a universal detector, we focus on further improving open-world generalization, and we need to address the following three technical challenges. 1) How to generalize to the complex open world without category supervision. Given the abundance and variety of object categories, it is not feasible to provide complete annotation for each object in training. Additionally, the open-world environment is complex and variable, and the imaging process by vision sensors is susceptible to various harsh weather conditions, resulting in less robust detection. We propose general object confidence to directly learn the general features of all objects, and enhance object features based on physical imaging models in harsh weather conditions. 2) How to train a universal detector using multiple source datasets with heterogeneous annotations. A universal detector should be trained on large datasets for better generalization. However, existing datasets such as COCO [21] and Object365 [36] are based on different human annotation criteria, resulting in cases where some objects may be annotated as background in another dataset. Such inconsistent annotation prevents the detector from learning the general features of objects. We propose an object augmentation method to generate consistent annotation. 3) How to adapt and improve itself in a new scene without human annotation. At the core of intelligence is adaptation and learning, as if even a child can generalize rapidly in a new environment [44] . When a well-trained universal detector is applied to a given scene, the categories of objects to be detected are usually also given. In this case, the universal detector should be able to adapt itself to improve the detection of given categories while minimizing the focus on irrelevant categories. We propose a self-enhancement method to further improve the detection of specific objects in a given scene.In this paper, we propose Uni-YOLO, a robust universal object detector for the complex open world with real-time performance. Unlike most existing ov-detectors, Uni-YOLO is designed as a onestage detector. The input of Uni-YOLO is enhanced with a physical model-based enhancement module (EHM) to provide adaptive enhancement for various complex weather conditions, rather than directly using the original degraded images as in previous methods. To detect more unknown objects, we propose a General Object Confidence (GOC). Based on GOC, Uni-YOLO learns the general features of objects to effectively discriminate between numerous categories and backgrounds. For zero-shot classification, Uni-YOLO is designed with a Matching Head to perform contrastive classification with the text embeddings of candidate objects. During training and inference, Uni-YOLO is guided by the extensive knowledge of the vision-language model CLIP. An object augmentation method is proposed to achieve consistent annotation for multiple source datasets. Based on generalization training with multiple source datasets, Uni-YOLO learns the general features of innumerable objects to achieve better generalization. To further improve the detection in a given scene, an online self-enhancement method is proposed. Uni-YOLO assigns pseudo-labels exclusively to given objects and performs fine-tuning based on these labels. Based on Uni-YOLO, we develop a UAV platform for multimedia interaction detection. It is demonstrated that Uni-YOLO can maintain real-time detection based on a low computational platform in the open world.The main contributions of this work are summarized as follows:• A new one-stage universal detector named Uni-YOLO is proposed. It includes three detection Heads operating in parallel to perform contrastive classification with the text embeddings of candidates, and uses a physical model-based EHM to improve its robustness in harsh weather conditions. • For training Uni-YOLO, an object augmentation training method is proposed to achieve better zero-shot generalization. The method addresses the heterogeneous problem of multiple source datasets and achieves large-scale training. • For online fine-tuning Uni-YOLO, a self-enhancement method is proposed. The method enables the detector to improve the detection of given objects in any given scene.Extensive experiments are conducted to validate the superiority of Uni-YOLO on various public object detection benchmarks. 
related work 2.1 traditional object detection: 
 Object detection tasks, involving object classification and localization, are crucial in computer vision. Current learning-based detectors can be broadly classified into three categories: two-stage methods, one-stage methods, and transformer-based methods. Two-stage detectors first extract a set of region proposals and then perform classification and regression, such as Faster-RCNN [35] . One-stage detectors perform classification and localization directly on the input images. One-stage detectors, especially YOLO [34] , exhibit remarkable real-time detection performance. Transformer-based detectors, such as DETR [2] and others [4, 22, 52] , are rapidly evolving. However, these traditional detectors only work on a closed set [9, 19, 48] , which requires a lot of human annotations for training.Open-vocabulary detectors aim to detect categories that do not appear in the training dataset. Various open-set object detection methods [23, 27, 33, 39, 59, 60] are proposed to generalize to unknown categories from the base categories. For example, Liang 𝑒𝑡 𝑎𝑙 . propose UnSniffer [20] , which uses two detector heads for both base and unknown categories and introduces general object confidence to achieve unknown localization. However, these methods provide only the localization, but not the classification for unknown categories. Subsequently, the advent of large-scale visionlanguage pre-training models has led to advances in ov-detectors [10, 17, 26, 37, 44, 51, 57] . The ov-detectors use vision-language models to achieve classification for novel categories. Zhong 𝑒𝑡 𝑎𝑙 .propose RegionCLIP [57] , an extension of CLIP, to learn regional visual information for finer-grained alignment of images and text. Wang 𝑒𝑡 𝑎𝑙 . propose a universal object detector, named UniDetector Most methods first design an image enhancement network to enhance degraded images and then perform object detection based on these enhanced images [3, 8, 30, 38, 41, 42, 49, 54] . However, these enhancement modules are originally designed for human vision, as measured by image quality metrics, and not for detection accuracy [16, 24] . For this reason, some studies introduce enhancement methods that are specifically tailored to object detection tasks. IA-YOLO [24] proposes a trainable image processor and uses detection loss training to improve object detection performance. BAD-Net [16] uses an attention fusion module to combine the features of the original images with those of the enhanced images, assuming that the original images contain valuable information for detection. 2PCNet [12] introduces a two-stage consistent unsupervised domain-adaptive network and applies domain-adaptive methods to enable detection in low illuminance conditions. However, these enhancement and detection methods work only in a closed set and cannot detect any unknown object in an open-world scene. 
method 3.1 the pipeline of uni-yolo: 
 The pipeline of Uni-YOLO is illustrated in Figure 2 . Uni-YOLO is designed as a new one-stage architecture with three detection heads operating in parallel. The detection results are based on the localization coordinates provided by the General Object Confidence Head (GOC Head) and the Box Regression Head (Box Head), and the classification results provided by the Matching Head.Object Localization: We utilize Efficient-Net [43] as the backbone network for feature extraction. Uni-YOLO divides the input image into 𝑆 × 𝑆 grid cells, and each grid cell is responsible for detecting objects that fall within its center. Each grid cell predicts three sizes of bounding boxes (𝑥, 𝑦, 𝑤, ℎ) provided by the Box Head, which represent the location of the bounding boxes (center coordinates, width, and height). Each bounding box has a corresponding general object confidence provided by the GOC Head. We determine whether there is any object in each candidate bounding box by a GOC threshold and output the localization coordinates.Object Classification: Uni-YOLO achieves zero-shot classification through the Matching Head. Each bounding box has a corresponding object embedding provided by the Matching Head, denoted as 𝑉 = [Ψ 𝑉 (𝑏𝑜𝑥 1 ), Ψ 𝑉 (𝑏𝑜𝑥 2 ), ..., Ψ 𝑉 (𝑏𝑜𝑥 𝑗 )]. Uni-YOLO uses CLIP's text encoder to generate text embeddings for retrieving candidate objects. Specifically, we construct a vocabulary list of objects that are of interest to the user for the candidates, such as 𝐿𝑖𝑠𝑡 = [𝑝𝑒𝑟𝑠𝑜𝑛, 𝑙𝑖𝑜𝑛, ..., 𝑒𝑙𝑒𝑝ℎ𝑎𝑛𝑡, |𝑏𝑎𝑐𝑘𝑔𝑟𝑜𝑢𝑛𝑑, 𝑜𝑏 𝑗𝑒𝑐𝑡]1 . We use a prompt template, i.e., "A photo of a/an object," and feed the prompts into the text encoder to obtain a set of text embeddings 𝐿 = [Ψ 𝐿 (𝑝𝑒𝑟𝑠𝑜𝑛), Ψ 𝐿 (𝑙𝑖𝑜𝑛), ..., Ψ 𝐿 (𝑒𝑙𝑒𝑝ℎ𝑎𝑛𝑡)]. If a bounding box contains any object (GOC above threshold), we perform similarity matching in the image-text space to classify the object corresponding to the maximum similarity between the text embeddings and features, thus achieving zero-shot classification for any object:EQUATIONwhere the maximum probability is "background" or "object, " the candidate box is discarded for further classification corrections.𝑝The Architecture of Uni-YOLO 3.2.1 General Object Confidence. Given the abundance and diversity of object categories, providing specific and consistent descriptions for the features of each object is challenging. Although humans struggle to provide specific descriptions, we can still identify overarching distinguishing features of objects and backgrounds; for example, there is usually an obvious boundary between them.It is suggested that the detector can also learn the overarching features from a large dataset spanning various categories.We propose the General Object Confidence (GOC) to discriminate whether a bounding box contains objects. The GOC is defined as the prediction results of the detection head based on the RepConv structure [43] , denoted as Φ(𝑏 𝑖 ). The range of GOC is [0, 1], with a higher value indicating a higher probability that the bounding box contains an object. In inference, we filter all bounding boxes based on a GOC threshold to identify those that may contain potential objects. In training, we design three losses to train GOC. The design of these losses takes into account three practical situations aimed at detecting more objects with better generalization.Case 1: Complete Confidence Objects. This case contains complete confidence objects (manually annotated boxes in training datasets), and their corresponding bounding box should have a GOC value of 1. Thus, the first GOC loss is expressed as:EQUATIONwhere Φ(𝑏 𝑖 ) represents the predicted probability of GOC in the proposal bounding boxes 𝑏 𝑖 . 𝐵 𝑐𝑐𝑜 is the set of complete confidence annotations. 𝑁 is the number of proposal bounding boxes. Case 2: Contrastive Confidence. This case involves the contrastive confidence between bounding boxes, i.e., the more precise the predicted localization of the proposals, the higher the value of their GOC. Thus, the second GOC loss is expressed as: (3) where 𝛼 = 1 if 𝐼𝑜𝑈 ( Φ(𝑏 𝑘 ), 𝑦 𝑖 ) > 𝐼𝑜𝑈 ( Φ(𝑏 𝑗 ), 𝑦 𝑖 ); otherwise, 𝛼 = -1, 𝑦 𝑖 is the annotation. 𝜁 is a tiny constant that set to 0.01.𝐿 𝑐𝑜 = 1 𝑁 ∑︁ 𝑖 ∈ [1,𝑁 ] 2 |𝐵 𝑐𝑐𝑜 | ∑︁ 𝑏 𝑗 ,𝑏 𝑘 ∈𝐵 𝑐𝑐𝑜 𝑚𝑎𝑥 ( Φ(𝑏 𝑗 ) -Φ(𝑏 𝑘 ) 𝛼 + 𝜁 , 0),Case 3: Complete Confidence Background. This case involves a complete confidence background (discriminated as background by the CLIP), and its corresponding bounding box should have a GOC value of 0. Thus, the third GOC loss is expressed as:EQUATIONwhere 𝐵 𝑐𝑐𝑏 represents the set of complete confidence backgrounds, determined based on the proposed object augmentation training method (subsection 3.3). Thus, the total GOC loss is:EQUATIONAdditionally, we use a regression loss, denoted as 𝐿𝑜𝑠𝑠 𝑟 , to perform bounding box regression training. The regression loss regresses all bounding boxes containing potential objects to achieve more accurate localization, and it can be roughly expressed as:EQUATIONwhere 𝐶𝐼𝑜𝑈 (•) represents the Complete-IOU, as proposed by [55] .3.2.2 Contrastive Classification. We design a contrastive classification loss to train the Matching Head. It is designed to compute the similarity between the proposal bounding boxes containing potential objects and the object embeddings of the annotated regions extracted by the vision encoder of CLIP. It is expressed as:EQUATION)where 𝐴𝑛 is the set of annotated object regions. 𝑆𝑖𝑚(•) is the cosine similarity. The contrastive classification loss represents the knowledge transfer from the CLIP model to the Matching Head.We design a physical model-based enhancement module (EHM) to perform image enhancement in harsh weather conditions for more robust detection. We mainly consider two common degradations: scattering and low illuminance.Scattering Degradation. The atmospheric scattering model [29] is used to describe the degradation in scattering environments:EQUATIONwhere To reduce the computational complexity of two parameter estimation, we combine the parameters, 𝑡 (𝑥) and 𝐴(𝑥), into a dehazing map 𝐷 𝑚 (𝑥), by referring [14] . The reformulated model is:EQUATIONEQUATIONThus, we can use one network to estimate the dehazing map 𝐷 𝑚 (𝑥) to perform enhancement in scattering environments.Illuminance Degradation. We design a learnable 𝐺𝑎𝑚𝑚𝑎 corrector to achieve enhance for low illuminance conditions. The 𝐺𝑎𝑚𝑚𝑎 corrector improves the contrast of degraded images through a nonlinear transformation, defined as follows:EQUATIONwhere 𝛾 𝑟,𝑔,𝑏 (𝑥) is the correction map and different values correspond to different mappings for illuminance degraded images. We also use one network to estimate the correction map 𝛾 𝑟,𝑔,𝑏 (𝑥) to perform enhancement in low illuminance environments. Enhancement Network. For real-time detection performance, we design a lightweight two-branch network, denoted as 𝜙 𝐷 𝑚 ,𝛾 [𝐼 (𝑥)],Figure 3 : Illustration for the EHM. The degraded image is firstly extracted features through a dense backbone and then passes through a two-branch structure to obtain the dehazing map and correction map. Finally, the enhancement is performed based on physical models. The EHM is pre-trained based on the perceptual loss [11] , and then is jointly trained with our detector. The two-branch network allows the two sets of parameters to be estimated using one single backbone. The specifics of the designed two-branch network are shown in Figure 3 , and more parameter and training details are provided in our supplementary material.The generalization performance of large models improves with the increased availability of training data [44] . Ensuring that Uni-YOLO is trained to its full potential using existing datasets is crucial for its generalization. However, the challenge arises from the heterogeneous annotations present in multiple source datasets. For example, in the COCO dataset [21] , the object "book" is annotated as an object, while in the Pascal VOC dataset [5] , it is ignored as background. This inconsistency hampers the detector's ability to learn the overarching features of general objects when simply merging multiple source datasets for large-scale training.To address this inconsistency, we propose an object augmentation method to achieve consistent annotations across multiple source datasets for Uni-YOLO training. As shown in the left part of Figure 4 , we first train our Uni-YOLO using the most densely annotated dataset. The pre-trained Uni-YOLO is then used to perform inference on a sparser dataset for obtaining more annotations, with the correctness of the pseudo-labels determined under the guidance Update 𝐷𝑆 = {𝐷𝑆, 𝐷 𝑖 }.Update Θ(𝑖𝑚𝑎𝑔𝑒) with 𝐿𝑜𝑠𝑠 𝑔𝑜𝑐 , 𝐿𝑜𝑠𝑠 𝑟 and 𝐿𝑜𝑠𝑠 𝑐𝑜𝑛 on 𝐷𝑆. 14: end for Output: The trained Θ(𝑖𝑚𝑎𝑔𝑒). And given any image, it has the ability to provide detection for all objects.of the large vision-language model CLIP. The pseudo-labels, along with the original annotations, form more dense and consistent annotations, and Uni-YOLO is retrained on the augmented datasets.The process is applied iteratively to multiple datasets for training. The specific steps are summarized in Algorithm 1.Input: A specific scene set 𝑆 with a list of objects of interest to users: [𝑖𝑚𝑎𝑔𝑒 (𝑖), 𝑙𝑎𝑏𝑒𝑙 (𝑖)] = 𝑆 (𝑖). Update Θ(𝑖𝑚𝑎𝑔𝑒) with 𝐿𝑜𝑠𝑠 𝑔𝑜𝑐 , 𝐿𝑜𝑠𝑠 𝑟 and 𝐿𝑜𝑠𝑠 𝑐𝑜𝑛 on 𝑆 (𝑖). 11: end for Output: The self-enhanced Θ(𝑖𝑚𝑎𝑔𝑒). It will focus on specific objects of interest even more, in specific scenes.𝐿𝑖𝑠𝑡 𝑖𝑛 = [𝑂The trained Uni-YOLO can detect a vast array of categories, but when applied to a given scene, it is not necessary to focus on all categories. Our goal is to enhance the detection of given objects while minimizing the detection of irrelevant ones, resulting in a more adaptable and universal detection system for specific scenes. We propose an online self-enhancement method to improve detection performance for given objects in any given scene. As shown in the right part of The selected objects serve as pseudo-labels for further online finetuning of our Uni-YOLO to achieve better detection. The specific steps of this proposed method are summarized in Algorithm 2.We develop a multimedia interaction UAV platform for object detection as shown in Figure 5 . The system's inputs include the user's voice and real-time images captured by the visual sensors. Users can specify objects of interest to the system via voice commands. The voice is then converted into a list of candidate objects using Whisper JAX, a real-time Acoustic-to-Text model [32] . Subsequently, the text embeddings of the candidate objects are obtained from the text encoder of CLIP, and the input images are processed by our Uni-YOLO to obtain detection results in the open world. The detection platform includes the DJI MATRICE M300 RTK as the base UAV, complemented by the MATRICE 350 RTK serving as the human-machine interactive remote controller. The primary imaging sensor used in our UAV system is the ZENMUSE H20 T camera.To process the information collected by the UAV and perform realtime detection, we use the DJI MANIFOLD 2 as the main processor. This computation platform is equipped with both a CPU and GPU, providing the necessary computing power for our object detector Uni-YOLO. The specified computation platform has a theoretical  
experiments 4.1 implementation details: 
 Uni-YOLO is trained on various large-scale object detection datasets, as summarized in Table 1 . We conduct training and inference using PyTorch 1.8.1 on an i9-13900K CPU and an NVIDIA 4090 GPU. Details of the UAV system configuration are provided in subsection 3.5. We employ an SGD optimizer with a learning rate of 0.01. The Enhancement Module (EHM) is pre-trained on the scattering dataset RESIDE [15] and the low illuminance dataset LOL [45] .We first evaluate Uni-YOLO in the open world using the low illuminance dataset ExDark [25] and the scattering dataset RTTS [15] to test its generalization and robustness under harsh weather conditions. Additionally, we use the 13 ODinW datasets [17] to further evaluate generalization performance across various complex scenes. Comparisons with existing ov-detectors are also conducted on the OV-COCO dataset [50] in normal environments. Detection performance is measured using mean Average Precision (mAP). [17] .Training Datasets mAP ↑ GCLIP-T (A) [18] Object365 28.8 GCLIP-T (B) [ RN50 backbone and without additional caption supervision. We perform comparison experiments with other ov-detectors on the low illuminance dataset ExDark [25] and the scattering dataset RTTS [15] . A summary of the test results can be found in Table 2 . Uni-YOLO achieves 53.4% mAP on ExDark, surpassing the previous best by 9.0%, and 52.5% mAP on RTTS, outperforming the previous This experiment demonstrates the zero-shot performance of Uni-YOLO in normal environments, and its real-time performance. The common public benchmark OV-COCO [50] [17] . First, Uni-YOLO is trained on the relatively densely annotated dataset, Object365 [36] . Two methods are used to introduce additional datasets for training. The Compound method involves the simple sequential blending of these datasets, the Augment method uses the proposed object augmentation (Algorithm 1). We gradually include the Pascal VOC [5] , COCO [21] , and OpenImages [13] datasets for further training. The results of the experiments are summarized in Table 5 . In most cases, Augment-based training produces better results than the simple Compound method. On average, the Augment-based multi-sources data training achieves a 39.2% mAP, representing an 8.6% increase over using the single Object365 dataset, while the Compound method only achieves a 33.7% mAP. The results demonstrate the importance of using multi-source data and the effectiveness of using the consistent annotations provided by the proposed augmentation.4.4.2 Online Self-Enhancement Method. The above test results in Table 3 demonstrate the effectiveness of self-enhancement (S.E.), achieving 8.8% (from 39.4% to 48.2%) and 7.0% (from 30.6% to 37.6%) improvement. Additionally, we sequentially perform five categories self-enhanced (S.E.) based on the RTTS dataset [15] to evaluate the effectiveness of the method in scattering scenes.The results are summarized in Table 6 . The results show that selfenhancement method enables Uni-YOLO to improve the detection performance of given objects. We observe a sequential improvement in the detection accuracy for the five categories. Thus, based on this method, Uni-YOLO can improve the detection of specific objects in specific scenes by itself, without human annotation.Enhancement Module. This experiment evaluates the proposed enhancement module (EHM). Table 2 provides the experimental results of the two types of Uni-YOLO, with EHM and without EHM. The detection performance is improved with the EHM under harsh conditions, achieving 53.4% mAP on the low illuminance dataset, which outperforms the without EHM method (45.1%) by 8.3%, and achieving 52.5% mAP on the scattering dataset, which outperforms the without EHM method (46.4%) by 6.1%. Based on the proposed EHM, the robustness of Uni-YOLO's detection performance under different weather conditions is effectively improved. 
conclusions: 
 In this paper, we propose Uni-YOLO, a robust and fast universal object detector. It is a new one-stage detector for object detection in the open world. Uni-YOLO utilizes general object confidence to effectively distinguish between objects and backgrounds, incorporating a grid cell method for precise bounding box regression. We design a physical model-based EHM to provide adaptive enhancement for Uni-YOLO in harsh weather conditions. We also propose the object augmentation method to train Uni-YOLO and design the self-enhancement method to online fine-tune Uni-YOLO. Comprehensive experiments on public benchmarks and the deployment of a UAV demonstrate its real-time robust detection performance.  
 
 Appendix: Different from the candidates of the traditional CLIP zero-shot classification method[31], our list introduces two additional candidates, "background" and "object", to mitigate the potential misclassification of the background region.The DJI MANIFOLD 2 website: https://www.dji.com/cn/manifold-2. 


Decision: This paper was reviewed by four experts in the field. The recommendations are (Weak Accept, Borderline Accept x 3). Based on the reviewers' feedback, the decision is to recommend the acceptance of the paper. The reviewers did raise some valuable concerns (especially more detailed ablations experiments, numerical results for model efficiency, more comparisons with related works, etc.) that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper.

Reviews: review: paper_summary: paper_summary: This paper proposes Uni-YOLO, a universal detector designed for complex scenes with real-time performance.
strength_weakness: strength_weakness: 1. This paper studies universal object detection in harsh weather, which has not been studied in existing works.
2. This paper realizes real-time object detection.
3. The experimental results are good.
limitations: limitations: 1. The article proposes EHM to overcome harsh weather conditions, mainly on two aspects: scattering and illuminance. However, harsh weather contains a wide variety of conditions. Restricting the discussion of harsh weather to these two aspects may limit the method's universal ability in harsh weather scenarios. The authors should conduct experiments under more diverse harsh weather conditions and compare with previous approaches to demonstrate its universality in handling harsh weather.
2. Table 2 should specify the training data used for each method. It seems that the training data for these methods is not consistent.
3. Since it focuses on real-time detection, I recommend that the authors include a detailed table about model efficiency. This table should compare metrics such as GFLOPs, parameters, and FPS with existing methods.
4. The ablation study should be more detailed. About EHM, the authors should separately discuss the impact of scattering degradation and illuminance degradation on the results.
paper_summary: paper_summary: This paper develops Uni-YOLO, a robust and fast detector that would deal with open-vocabulary objects in challenging weather. It develops several techniques to improve the robustness of the model, e.g., input enhancement, object augmentation, etc. Uni-YOLO achieves good performance on several benchmarks with fast inference speed.
strength_weakness: strength_weakness: 1. The proposed method reaches a decent balance between speed and performance across different benchmarks. 
2. This paper is clearly written, and the figures and supplementary videos are well demonstrated, making this paper easy to understand.
3. Some proposed novel modules are effective like object augmentation. It makes sense to deal with the annotation conflicts among different datasets when conducting joint training.
limitations: limitations: 1. The novelty is limited. 
The proposed modules are straightforward combinations of existing techniques.  Like the open-vocabulary pipeline design and the object augmentation methods.

2. Lacks ablation studies. 
Could the input enhancement module be replaced by simple data augmentations?   
The effectiveness of the object score is not verified.
A step-by-step ablation from the naive baseline to the full-version model is expected. We could not figure out where does the improvement come from.

3. Lacks discussions for similar works.
The author should add [A1]  for discussion and comparisons.

[A1]YOLO-World: Real-Time Open-Vocabulary Object Detection
paper_summary: paper_summary: This paper introduces a one-stage detector for object detection in the open world. Uni-YOLO utilizes general object confidence to effectively distinguish between objects and backgrounds, incorporating a grid cell method for precise bounding box regression. The authors also design a physical model-based EHM to provide adaptive enhancement for Uni-YOLO in harsh weather conditions.
strength_weakness: strength_weakness: The concept of Universal Detection in the Open World is practical and reasonable. The authors have considered both Harsh Weather and Open Vocabulary settings, and the experimental results are impressive.
limitations: limitations: 1-Although the paper presents excellent work, the impact is somewhat limited by the fact that the code and models for Uni-YOLO are not open-sourced, especially considering that most of the compared methods (e.g., UniDetector, CORA, BARON, Detic) have released their code.

2-To demonstrate the adaptability of the proposed method to harsh weather, it would be better to include comparisons with specific harsh weather detectors (Table 2 seems to include only general detectors).

3-In this work, harsh weather conditions appear to be limited to poor lighting scenarios (using EHM to enhance images). How does the method handle other conditions such as rain or snow?

4-In Table 4, Uni-YOLO does not show an advantage over previous works in the "Base" and "All" settings. What could be the possible reasons for this?
paper_summary: paper_summary: This paper presents a new single-stage generalized object detector. it employs a generalized target confidence level to discriminate between target and background, and uses a grid cell regression approach to achieve real-time detection. In the article, EHM is designed to enhance the robustness of Uni-YOLO under adverse weather conditions. During training and inference, Uni-YOLO is guided by a large-scale visual language model, CLIP.. In addition, the article proposes an online self-enhancement method that allows Uni-YOLO to further focus on specific targets in a given scene through self-supervised fine-tuning. The superiority and practical value of the method are verified in public benchmarking and UAV deployment experiments.
strength_weakness: strength_weakness: 1. The paper is written in a logical and clear manner, with a clear hierarchy and visual presentation of graphs and charts. It stimulates the reader's interest in the study.
2. The proposed one-stage Uni-YOLO architecture with parallel detection heads and EHM provides a novel and efficient approach for open-world object detection under various conditions.
3. The object augmentation method addresses the important problem of inconsistent annotations across source datasets, enabling effective large-scale training.
4. The paper includes thorough experiments on a variety of datasets (ExDark, RTTS, ODinW, OV-COCO) and real-world UAV testing, providing strong empirical support for Uni-YOLO's effectiveness and practical value.
limitations: limitations: Weaknesses:
1. There are concerns regarding the fairness of the comparative experiments. The proposed method utilizes EfficientNet for feature extraction, yet it does not specify the backbone used by the comparative methods. Additionally, the comparison does not appear to be conducted using the same benchmark detectors, such as the MaskRCNN utilized by GLIP. 
2. A significant enhancement in the performance of the proposed method is contingent upon the EHM and the S.E. strategy. These components seem applicable to other open-vocabulary detectors as well. However, the paper does not validate their potential as a general contribution to the field.
3. The paper does not mention or compare with other ov and single-stage object detection works, which is a notable omission considering the relevance to the study's focus.

[1] Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation. CVPR2022.

[2] EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignment. ICCV 2023.

[3] Zero-shot object detection through vision-language embedding alignment. ICDMW 2022.

4. In comparison experiments with other state-of-the-art open vocabulary detectors (Table 4), the real-time performance of Uni-YOLO (33 ms) is much better than previous methods (e.g., OV-DETR 148 ms, CORA 156 ms, etc.). I encourage the authors to explain the reasons.

Questions:

1. How was the vocabulary list for zero-shot classification (Section 3.1) constructed in the experiments - manually or automatically? Are there any limitations or sensitivities around this?
2. What criteria or thresholds are used to select the pseudo-labels from CLIP during object augmentation and self-enhancement? How do these impact precision/recall?
3. Can the EHM architecture and training process be elaborated? E.g. what datasets were used, how were the branches designed?

I will revise my rating based on other comments and the author's rebuttals.