YOLO-RD: Introducing Relevant and Compact Explicit Knowledge to YOLO by Retriever-Dictionary
Authors: Hao-Tang Tsui
Chien-Yao Wang
Hong-Yuan Mark Liao
Venue: ICLR 2025 Poster (2025)
Keywords: Real-time Object Detection
Object Detection
Dictionary Learning
Retrieval-Augmented Generation
Representation Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Identifying and localizing objects within images is a fundamental challenge, and numerous efforts have been made to enhance model accuracy by experimenting with diverse architectures and refining training strategies. Nevertheless, a prevalent limitation in existing models is overemphasizing the current input while ignoring the information from the entire dataset. We introduce an innovative $\textbf{R}etriever-\textbf{D}ictionary$ (RD) module to address this issue. This architecture enables YOLO-based models to efficiently retrieve features from a Dictionary that contains the insight of the dataset, which is built by the knowledge from  Visual Models (VM), Large Language Models (LLM), or Visual Language Models (VLM). The flexible RD enables the model to incorporate such explicit knowledge that enhances the ability to benefit multiple tasks, specifically, segmentation, detection, and classification, from pixel to image level. The experiments show that using the RD significantly improves model performance, achieving more than a 3\% increase in mean Average Precision for object detection with less than a 1\% increase in model parameters. Beyond 1-stage object detection models, the RD module improves the effectiveness of 2-stage models and DETR-based architectures, such as Faster R-CNN and Deformable DETR. Code is released at https://github.com/henrytsui000/YOLO.

Published as a conference paper at ICLR 2025
YOLO-RD: INTRODUCING RELEVANT AND COMPACT EXPLICIT KNOWLEDGE TO YOLO BY RETRIEVER- DICTIONARY
Hao-Tang Tsui, Chien-Yao Wang & Hong-Von Yuan Mark Liao Institute of Information Science, Academia Sinica (henryt sui, kinyiu, feiiseiiis..............aaiiiiei sinica. edu.
ABSTRACT
Identifying and localizing objects within images is fundamental challenge, and numerous efforts have been made to enhance model accuracy by experimenting with diverse architectures and refining training strategies. Nevertheless, a preva- lent limitation in existing models overemphasizing the current input while ignor- ing the information from the entire dataset. We introduce innovative Retriever- Dictionary (RD) module to address this issue. This architecture enables YOLO- based models to efficiently retrieve features from Dictionary that contains the insight of the dataset, which is built by the knowledge from Visual Models (VM), Large Language Models (LLM), or Visual Language Models (VLM). The flexible RD enables the model to incorporate such explicit knowledge that enhances the ability benefit multiple tasks, specifically, segmentation. detection, and clas- sification, from pixel to image level. The experiments show that using the RD significantly improves model performance, achieving more than a 3% increase mean Average Precision for object detection with less than a 1% increase in model parameters. Beyond YOLO, the RD module improves the effectiveness 2-stage models and BETR-based architectures, such as Faster R-CNN and Deformable DETR. Code released at attttps: //ttt::://ittbbb..// http://gtttu....o/eeoeeeeeeeeeee cww wttp://nytttuuut0////0000
INTRODUCTION
In the field of computer vision, object detection models play pivotal role, these models are de- signed to precisely locate objects within images. They are used in applications such as medical image analysis and autonomous driving. Additionally, they also be used backbone models for downstream tasks like multi-object tracking (Zhang et al. 2022 Cao et al. 2023]] Aharon et al., 2022). and crowd counting (Zhang al. [2166] (ong al., (022)) As the basis for these extended
tasks, object detection models must combine high accuracy with low latency to allow downstream tasks to stand the shoulders of giants.
Among object detection models, the YOLO (Redmon et al. 2016), FasterRCNN (Ren et al. 2015), and DETR (Carion et al. 2020) are notably prevalent. The YOLO series primarily utilizes Con- volutional Neural Networks (CNN) (LeCun et al. 1998). providing a balance between inference speed and accuracy. From YOLOv1 through YOLOv (Redmon al. 2016 Redmon & Farhadi Bochkovskiy et al. 2020 Jocher 2020 Li et al.) 2022] Wang et 2003aa] Jocher al. 2023 Wang et al.J 2024b a). there has been consistent focus on refining the architecture
2017 2018
and training methods to reduce the model's parameters while enhancing accuracy. Beyond the main YOLO series, variants such as YOLOR (Wang et al.] 2023b) incorporate implicit knowledge and other techniques to further boost model performance.
As shown in Figure I while both CNNs and Transformers (Vaswani et al. 2017) concentrate on the input image, CNNs are restricted to local input data, and Transformers, despite considering interac- tions among various inputs, are still confined to the given inputs or other model branches. However, the above-mentioned models often overlook a crucial aspect of explicit knowledge- the comprehen- sive dataset information. On the other hand, some contrastive methods, e.g. SimCLR (Chen et al.] 2020), DINO (Caron al. 2021) have demonstrated that cross-referencing data is beneficial.

---

Published as a conference paper at ICLR 2025
Traditional Model
Our Architecture
Predict
Predict
Input
Input
input
Other inputs
Knowledge Dictionary
& Retriever Module
comparison between traditional models and our proposed Retriever Dictionary module. On the left, CNN- models focus local regions (blue box), while transformer-basee models tend to utilize the entire image (red box). However, both methods fail to leverage the information from the entire dataset, illustrated in the bottom-right corner. Our module enhances the model's resource utilization by incorporating knowledge from other parts of the dataset.
Figure 1:
Moreover, in the realm of natural language processing, some models incorporate the Retrieval Aug- mented Generation (RAG) architecture (Lewis et al. 2020). which stores knowledge in pre-built database and retrieves this information during inference to pass to the generator for encoding, This allows the model to access external information from a large, pre-established dataset. This approach significantly enhances the capabilities of large language models, although it typically requires sub- stantial computational resources. However, applying this technique to object detection or other computer vision tasks still faces significant challenges, particularly in preparing and retrieving ex- ternal data. Object detection models must carefully balance accuracy, model parameters, and latency when handling external information.
To address these challenges, we introduce a compact external module composed of a Retriever and Dictionary, designed to enhance dataset utilization in computer vision models during training. This module effectively filters out irrelevant information and amplifies crucial data. The Retriever aggregates region features generate query, while the Dictionary, containing comprehensive dataset information, enables the query to select relevant atoms. Notably, this pre-built Dictionary extends beyond the YOLO backbone, incorporating data encoders like VLMs LLMs, which bring extensive training data and knowledge for more precise and comprehensive information.
This module allows models reinforce data during the forward process, benefiting not only region- level tasks like object detection but also pixel-level tasks like segmentation and whole-image tasks like classification. Furthermore, our module can be extended various model architectures, such as the FPN network in Faster RCNN and the backbone-encoder regions in Detection Transform- ers, providing higher-quality information during downsampling, ultimately leading much better performance.
In this paper, we make several key contributions:
We introduce Retriever-Diccionary module that enables efficient utilization of external information without the need for an external loss function, while still maintaining the Dictionary's properties and allowing updates to its parameters.
We demonstrate that incorporating external knowledge from models such VLMs and LLMs can significantly enhance model performance.
We further show that integrating external information improves not only YOLO's object detection performance but also other mainstream vision tasks and architectures.
These improvements substantially enhance the capabilities of object detection models and demon- strate that our module exhibits All-to-All properties, allowing it to utilize external knowledge to improve performance across multiple tasks and model architectures with minimal parameters.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Real-time object detection. Real-time object detection is foundational problem in computer vi- sion, with focus on achieving low latency and high accuracy. Traditional approaches have primarily focused on CNN architectures, with seminal works such as the OverFeat (Sermanet et al. 2014) and
2013, Ren et al. 2015). Since the introduction of Vision Transformers (ViT) Dosovitskiy et al. 2021. there have been notable follow-up works, including DETR (Carion ct al. 2020) and RT-DETR (Zhao et al. 2024). However, the CNN-based YOLO series models hold a crucial position in the field of real-time detection due to their ease of training from scratch, lightweight design, and ability perform high-speed inference.
Faster R-CNN
Each version of the YOLO model introduces different architectures and training strategies. For example, YOLOv7 (Wang et al. (023a) employs ELAN and trainable bag-of-freebies techniques to enhance performance, while YOLOv9 Waag et al.
2024b) incorporates Generalized Efficient Layer Aggregation Networks (G-ELAN) and Programmabe mll Gradient Information for improved efficiency and learning capability. YOLOv10 (Wang et al. 2024a) further introduces the compact inverted block to optimize model size and computation. On the theoretical front, works like YOLOR (Wang et al., (023b) leverage the shared characteristics across multiple computer vision tasks, allowing the model to learn implicit knowledge and relax the prediction head, thus generalizing the YOLO architecture to various tasks. Despite these architectural and strategic differences, all YOLO models share a conceptual framework comprising three core modules: a Backbone for downsampling, a Neck (e.g. FPN) for feature fusion, and Detection Head for final prediction. In this paper, also utilize the Backbone as an image encoder.
Dictionary learning. Dictionary learning is fundamental technique in signal processing and machine learning, aimed at learning a set of basis functions (or atoms) that can efficiently represent signals. This technique has been extensively explored in the context of sparse coding, where signals are approximated as sparse linear combinations of dictionary atoms. Additionally, it was discovered that natural images can be effectively represented using sparse coding models (Olshausen & Field 1997]. laying the foundation for further development of dictionary learning algorithms.
Several algorithms have been proposed to optimize dictionaries and sparse coefficients. Among these, K-SVD (Abaron et al. (2066) has become a standard due to its effectiveness in applications such as image denoising, compression, and inpainting. With the rise of CNNs, dictionary learning has also seen new developments, such as designing convolutional blocks and defining loss func- tions to achieve dictionary learning objectives (Garcia-Cardona & Wohlberg 2018 Zheng et al.) 2021). Additionally, dictionary learning has been applied to tasks like content-based image retrieval (CBIR), ) as seen in works like Şaban Öztürk (2021); Tarawneh et al. 2019). this paper, we focus on dictionary learning rather than sparse dictionary learning and emphasize dictionaries that can robustly represent information and retain critical, relevant signals.
Retrieval-augmeented generation (RAG) Retrieval-Augmented Generation (RAG) (Lewis al., 2020) is a technique that first appeared in large language models. It primarily involves three steps: Indexing, where the database is split into chunks, encoded into vectors, and stored in vector database; Retrieval, which retrieves relevant information based on similarity to the input; and Gen- eration, where both the original input and the retrieved information are fed into the model for further processing. This approach enables LLM to handle unseen information and has been successfully applied in T5 (Raffel et al. ) or certain versions of ChatGPT al. 2023).
Due to the Lime-consuming nature of the retrieval process, and the real-time requirements of most computer vision tasks, RAG has seen limited application in vision-based models. Recently, some work has been done to mitigate the bottleneck in Wu & Xie (2024); Kim et al. (2024); [iu et al.) (002).. For example, RALF (Kim et al. 2024) leverages LLM to embed a huge vocabulary set and find similar meaning words to refine features; REACT (Liu et al. 2023) utilizes the World Wide Web as the information source to extend model knowledge. However, they still require an undetachable huge dataset or Language model which leads extremely high training and inference loading. In this work, we provide the vision-based model with not only a light-weight database but also refinable features.
3

---

Published as a conference paper at ICLR 2025
Input
Prediction
Pre- Peocerocss
Back bone
Neck
Head
Whole Dataset
Information Encoder
Retriever
Dictionary
VLM
K-Means
Positional Norm
coefficient
VM
LLM
Coefficient
tom
Generator
d'anc X
riytom
Initialize
Global Information
4
Exchanger
d'suk
agttom
03
Figure 2: Dictionary initialized by encoding the dataset using an image encoder, from which N embeddings are selected as Dictionary atoms. During training, for each input feature Xw, the Retriever core comprisin the Coefficient Generator (G) and the Global Information Exchanger (E) generates coefficients for each atom a in the Dictionary D. Then, the normalized coefficients are used as weights for each Dictionary atom. Finally, by concatenating the residual X X the output Y.,h obtained.
METHOD
In this work, we introduced the Retriever-Dictionne module, as shown in Figure 2 which enables computer vision models to utilize comprehensive dataset knowledge with minimal extra parameters quickly. This plug-in stores encoded information from various models, enhancing the model's 's ability to identify which features of the input data should be emphasized diminished, thereby improving overall performance. The RD-module is composed of two main components; the Retriever and the Dictionary. The Dictionary consists of N elements, each represented as R vectors, known as atoms The Retriever generates the coefficient of a for each pixel according to the input. The Dictionary be generated using the YOLO backbone initialized with diverse models like the visual language model CLIP (Radford et al. (02T) and large language models such as GPT (Radford al.) 2018). This approach allows the model align visual and linguistic representations, leading to more balanced and valuable distribution of atoms. Furthermore, the incorporation of linguistic knowledge helps the module retain erucial information. The main goal to adjust the distribution of the Dictionary and allow the Retriever core to find the best Atoms" weight for each input pixel.
3.1 MODULE STRUCTURE.
The Retriever core aims to efficiently generate the coefficients of each a in the Dictionary. In- spired by depthwise convolution Chollet (017). we separate the Retriever into two components: the Coefficient Generator G and the Global Information Exchanger E. The Coefficient Generator, denoted as G //xWX + Nx WwxWxl computes coarse coefficients based on the input feature map X E //xW xH where f is the input feature dimension and W, H represent the width and height of the feature map, respectively. The coarse coefficients are calculated as follows:
Y = G(X) = WG -Xw.hs
(1)
where WG E Nx is the projection matrix of G, and Xm E R is the feature vector at spatial location (w, h).
The Global Information Exchanger, denoted as E NXWWH R NxWxH refines and exchanges information across neighboring pixels, and defined as:
E(Y) = 5 (
(2)
where W E R NxIxko is the depthwise convolution filter with kernel size x k. and E N) indexes the channels of Y. The term (t) e gw* refers to the spatial feature map of the i-th channel.
4

---

Published as a conference paper at ICLR 2025
The combined operation of G and E generates the final coefficients vector c for each pixel:
((),).,)))
This separation of tasks minimizes direct computation of the coefficient vectors at each pixel location, significantly reducing the parameter count while maintaining high performance (see Ex- periments 4.3 and Appendix A.8 for further discussion).
To prevent the coefficient vectors e from simply replicating the input features, which would make the Dictionary become an identity matrix, we normalize each c. We apply normalization process equivalent to Positional Normalization (PONO) (Li et al. 2019), which is defined as:
PONO(X) X Pe 7 3
Voc +
where the mean pe and variance a. are calculated as:
N He Σ X,, O. Σ (Xn
-o Mo)²
with X, representing the n-th feature at a fixed spatial location, and % 8 are learned parameters. Although PONO was initially designed to preserve structural information in generative networks, here ensures that the feature vectors e are properly scaled and centered, preventing the Dictionary from collapsing into an identity matrix.
The normalized coefficients c' are then used to select atoms from the Dictionary, either enhancing or diminishing specific features. This selection is weighted summation of the atoms and integration with the input residuals to produce the final output. To preserve the dictionary's 's learning dynamics, each atom normalized to unit length during training. The resulting formula follows:
Yi,. = A Xho + (1 Σ c' i,b, ai.
(5)
where |a,| = for all a, E D, and À is the residual weight. As can be seen in Equation 5 the weighted summation of Dictionary atoms mathematically equivalent a convolutional layer with a kernel size of 1, stride of 1, and no bias term. To ensure that the sum of each atom's 's components equals 1, as required in dictionary learning, we employ weight normalization (Salimans & Kingma 2016). This eliminates the need for an external objective function to enforce this condition. Weight normalization is defined as:
Va E ED
WN(D)
(6)
aa
Finally, we combine the Retriever core (G and E), the Dictionary, and the residual connection to express the entire Retriever Dictionary process RD /f + x as:
7 RD(X) A X + (1 A) PONO(E(G(X))) WN(D),
where * *WN(D) denotes the convolution operation using the weight-normalized Dictionary atoms set WN(D) as filters.
3.2 Dictionary INITIALIZATION
By pre-initializing the Dictionary, we embed knowledge into the module atoms of the Dictionary. More precisely, we use the selected encoder to map the entire dataset into a high-dimensional space, by incorporating insights from various modality models, as illustrated in Figure 3 This high-dimensional space shares the same dimension as the original YOLO backbone middle layer. Obviously, this will have huge number of vectors and present multiple groups in nigh-dimensional space, we employ k-means (Macqueen] 1967) to leave representative vectors, which ultimately serve as Dictionary atoms. Through these operations, we can map the dataset pairs into high-dimensional space in short time (this operation is equivalent to using the model to traverse the entire dataset at the speed of inference), and find vectors that can represent most of the dataset. For features that not in the Dictionary, utilize the Retriever Dictionary to bring closer atoms of the same category to the outlier feature. Next, discuss using different modality models encoders for encoding knowledge:
the feature through the residual mechanism in the module, and
5

---

Published as a conference paper at ICLR 2025
Dataset Space
Language Space
Vision Space
Vision Language Space
imput
imput
input
imput
(a) Dataset distribution
(b) Init with YOLO
(c) Init with GPT
(d) Init with CLIP
Figure 3: Illustrates the distribution of the dataset in the model's middle layer, where the blue square represents the current input feature. In traditional models, only the input feature is used, neglecting the rich information available the dataset. In contrast, with our Retriever-Dictiona model. additional data information is retrieved from the dataset. The dictionary can be initialized from different models: vision models, language models, or vision-language models. The latter provides a more comprehensive and integrated representation of the dataset.
Vision model. In the YOLO architecture, the backbone layer is designed to transform the origi- nal image into high-dimensional embeddings, providing the FPN with enriched regional informa- tion, which makes the backbone an ideal image encoder. Therefore, we use the modern prainee YOLOv9 (Wang et al. (002b) as the vision model encoder. This model is leveraged to traverse the entire training set, converting all the dataset's data into feature-dimensional distributions.
Vision language model. As a visual language model, CLIP possesses a strong understanding of input image features and can map these features into a semantically related space for compari- son. Therefore, we utilize CLIP for visual language initialization. However, according to Dense- CLIP (Rao et al. 2022), although CLIP's output embeddings are global representations of the image, the output of CLIP's VIT blocks retains information about corresponding regions. CLIP primarily uses the first patches as embeddings, leaving out local information from the other patches. As a result, leverage all the output patches from CLIP's image encoder embed the dataset more comprehensively.
Large language model. For initialization from an LLM, since language models cannot directly convert images into feature embeddings and image captions typically describe the entire image rather than specific regions, we use class names from MSCOCO (Lin et al. (2044) and ImageNet-2 kk (Rid- nik al. 2021), along with image captions from MSCOCO, as prompts for GPT (Radford al. 2018). However, because some feature dimensions deviate significantly from the unit interval, we apply standard normalization to scale each dimension, aiming to stabilize the training process while preserving the relative positioning of the vectors.
3.3 Dictionary COMPRESSION
The objective of designing Retriever Dictionary (RD) retain the most critical information from the dataset. However, even after training, the number of atoms in the Dictionary may exceed the required amount, with some atoms being infrequently used or irrelevant to the specific dataset do- main. Drawing inspiration from knowledge distillation and the Macher-SSudent model (Hinton ctt al. 2015), we condense the original Dictionary into smaller and more efficient version, de- noted d. To align the output features of d with those of D, employ contrastive learning (Chen et al. 2020) to provide the d soft labels instead of the traditional cross-entropy loss. In this process,
the model backbone and RD are frozen, while the optimization is focused the smaller Retriever Dictionary module d. The objective function is defined as:
Liven = - log exp(sim(=dl and
RD L, RR,n)/T))
(8)
Σ/= Σ W. exp(sim ( A j. Maunke) x)/T)
RD

---

Published as a conference paper at ICLR 2025
Comparing the performance and the improvements across different modalities of RD- module and structures on the COCO 2017 validation set.
Table 1:
BackBone
Initializer
Params
Latency
mAPeel
55.95 (%)
mAP (%)
YOLOv7
Origin
37.2M
3.59
50.04
68.65
YOLOv9
Origin
25.3M
4.00
52.64
69.56
Paster-RCNN
Origin
43.1M
41.00
38.40
59.00
Deformable DETR
Origin
40.1M
41.10
43.80
62.60
YOLOv7
VM
37.4M
3.70
R
51.37 (1 2.66%)
69.42 (1 1.13%)
YOLOv9
VM
25.5M
4.16
IR512
53.41 ( 1.46%)
70.57 (1 1.46%)
Faster-RCNN
VM
44.1M
41.00
IR512
40.50 (1 5.47%)
60.30 (1 2.20%
Deformable DETR
VM
41.2M
41.10
IR512
44.10 (1 0.68%)
63.30 1.12%)
YOLOv7
VLM
37.4M
3.70
TR
51.75 (1.2%)
70.12 (1 2.15%)
YOLOv9
VLM
25.5M
4.16
R
53.36 (f 1.37%)
70.55 1.43%)
Faster-RCNN
VLM
44.1M
41.02
IR512
40.50 5.47%)
60.40 (1 2.37%)
Deformable DETR
VLM
41.2M
41.28
RR512
44.40 (f 1.37%)
63.30 (1 1.12%)
YOLOv7
LLM
38.2M
3.79
R 1024
51.36
(1 (164%)
69.40 (1 1.17%)
YOLOv9
LLM
25.8M
4.20
R101
53.28
(f 1.22%)
70.48 (1 1.33%)
RCster-RCNN
LLM
44.6M
41.03
R I.024
40.70
5.99%)
60.80 3.05%)
Deformable DETR
LLM
41.7M
41.35
R 1024
44.16
(1 0.91%)
63.10 (1 1.12%)
add where i,,,h E R represents the i-th batch output feature of d at position hw, B is the batch size, the temperature parameter, and sim(.,)) denotes the cosine similarity between two vectors.
This distillation process ensures that, within the specific domain, the atoms in d can effectively ap- proximate various potential linear combinations found in the original Dictionary D. By selectively removing atoms from RD that are not pertinent to the dataset domain, we achieve a significant re- duction in atom count by at least 50%. This reduction not only increases the model's 's efficiency but also maintains the performance and expressiveness within the targeted domain.
EXPERIMENT
4.1 SETUPS
Experimental setup. We primarily validated the method on the Microsoft COCO dataset (Lin al. 2014), training on the COCO 2017 train set and evaluating on the COCO 2017 validation set. For Object Detection and Segmentation, we respectively used mAP and mAP@ 55 as evaluation metries, testing on YOLOv7, YÓLOv9, Faster RCNN, and Deformable DETR. For the Classifica- tion task, we used the CIFAR-100 dataset with the YOLOv9-classify model, using top-1 and top-5 accuracy as metrics.
Implementation details. All experiments were conducted using 8 Nvidia V100 GPUs. In the main series of experiments, we trained a modified YOLOv7 model, which included the addition of a Retriever-Diccionary Module, for 300 epochs in 2 days. The YOLOv9-based model was trained for 5 days with 500 epochs. We also trained a modified Faster RCNN, based on the mm-detection framework, for maximum of 120 epochs over 3 days. For Deformable DETR, trained for approximately 120 epochs over days. The classification task on CIFAR- (Krizhevsky et al. 2009) took 2 hours on single Nvidia 4090 GPU for 100 epochs. Latency measured single Nvidia 3090 GPU without any external acceleration tools, using milliseconds per batch with a batch size of 32 on the MSCOCO validation set.
4.2 COMPARISION WITH RD

---

Published as a conference paper at ICLR 2025
Apply to state-of-the-art detectors. As demonstrated in Table 1 evaluate the proposed mod- ule mainly on YOLOv7 and also integrate it with both fundamental and SOTA real-time object detection models. The Dictionary is initialized separately by three distinct models: Vision Model (VM) with the YOLOv7 backbone, Vision-Language Model (VLM) employing CLIP, and Large Language Model (LL.)) based GPTv2. Among these, CLIP provides the most significant im- provement, likely due to its well-balanced performance across both the vision and language domains.
The RD module consistently yields notable improvements. In YOLOV7 and YOLOv9, the intro- duction of the module increases the parameter count by less than 1%, yet results in substantial performance gains across key metries. The improvement comparable to the performance boost achieved by moving to the next model size (e.g.. YOLOv7-x, YOLOv9-e), which typically requires 100% increase in parameters. For more traditional architectures, we incorporate the RD module into Faster R-CNN with a ResNet-50 backbone He et al. (20T6). Despite modest 2% increase in param- eters, the model outperforms Faster R- R-CNN with ResNet-101 backbone. Furthermore, we extend the module transformer-basee architecture, specifically Deformable DETR Zhu et al. (2020) with a ResNet-50 backbone. Similar to the previous results, the RD module yields improvements equivalent to upgrading to the 101 backbone.
These experiments conclusively demonstrate that leveraging dataset information and incorporating knowledge from VM, VLM, and LLM significantly enhances the performance of wide range of base models, while requiring only minimal additional parameters.
Apply to other tasks. The Retriever Dictionary (RD) module enhances pixel-level features, and its potential benefits extend beyond detection tasks to include other vision tasks, such as segmenta- tion and classification. To validate this, we conducted segmentation experiments on the MSCOCO dataset and classification experiments on the CIFAR-100 dataset, demonstrating the effectiveness of the RD module across both pixel-Ievel and image-level tasks. Table 2 multi-task structure with the one incorporating our proposed module. The results clearly show that the Retriever Dictionary module upgrades performance across classification, detection, and segmen- tation tasks, demonstrating its effectiveness enhancing overall multi nulti-tasking performance.
compares the original YOLO
Table 2: Comparing RD at different tasks.
Table 3: Comparing with different knowledge-baadd based methods.
Task
Metrics(%)
w/o RD
RD
Improve
Classification
Top-1 Top-5
74.86
75.70
1.12%
Method
mAP
mAP.s
+Params
93.72
94.28
0.60%
baseline
52.64
69.56
Delection
mAp Box
50.04
51.75
t 3.42%
KD
52.52
69.14
57.3M
MAPBos
68.65
69.51
f 2.15%
YOLO-World
51.00
67.70
66.
RALF
51.40
68.07
37.8M
SApSeg
40.53
41.56
t 2.54%
Segmentation
RD (ours)
53.36
70.55
0.2M
mApSeg
64.00
64.64
1.00%
Comparison of knowledge integration methods for YOLO. Table compares RD with various methods for integrating external knowledge into YOLO. Specifically, knowledge distillation (Hinton al. 2015) YOLOv9-e the teacher model and YOLOv9-c the student model, denoted as KD. Another approach, YOLO-World (Cheng et al. 2024), incorporates visual-language concepts from CLIP into YOLO to enhance its understanding of both domains. Additionally, RALF (Kim) ct al. (202)) utilizes CLIP's text encoder to create vocabulary set as database in RAG-based method. The "+Params" column represents the additional parameters introduced by the knowledge provider or supervisor model compared to the baseline. Overall, RD not only offers the lightest solution, with only 0.2 additional parameters but also delivers the best performance, making it highly efficient and effective approach.
4.3 ABLATION STUDIES
Fuse coefficient generator and global information exchanger. In Section 3.1 we discussed splitting the Retriever core into pointwise convolution G and depthwise convolution can sig- nificantly reduce the number of parameters. Without split Retriever core, the 1-stage coefficient
8

---

Published as a conference paper at ICLR 2025
Table 4: Analysis of fuse Retriever core.
Retriever
Weight size
Params.
mAP
Fuse E, (X) E G())
R/xNxkxk
41.9M
50.95%
R N
4.3M
50.64%
Table 5: Performance comparison different models on the COCO 2017 validation set.
BackBone
Initializer
algorithm
Params
Atoms
Feature
mAP
mAPeal
.5:95
YOLOv7
Random
37.4M
512
I2522
50.02%
69.06%
YOLOv7
YOLOv7
37.4M
512
IR612
51.34%
69.37%
YOLOv7
CLIP
37.4M
512
R512
51.75%
70.12%
YOLOv7
GPT
38.2M
512
R0024
51.33%
69.40%
YOLOv7
YOLOv7
k-means
37.4M
512
1R512
51.34%
69.37%
YOLOv7
YOLOv7
VAE
37.4M
512
RR52
51.19%
69.25%
YOLOv7
CLIP
k-means
37.4M
512
R212
51.75%
70%
YOLOv7 CLIP
Convex Hull
37.8M
1024
RR12
49.58%
68.92%
YOLOv7
GPT
Normalize
38.2M
512
1024
51.33%
69.40%
YOLOV7
GPT
Tanh
38.2M
512
R 1024
51.27%
69.35%
YOLOV7
CLIP
37.4M
512
R212
51.75%
70.12%
YOLOV7
GPT
38.2M
512
R 1024
51.33%
69.40%
YOLOV7
Mix
(LLM, VLM}
37.4M
512
12512
51.38%
69.59%
generates the process as follows:
R'(X) = Fuse (E, G) = Xw.h * Wo9
where We E B is the convolution matrix, R', RRxWXH _ IR NxWxH is the 1-stage generate process. This operation will require an extra of NSL22 (fN + NR parameters. And the computational complexity is of O(W H HNNK- sk2), compared to G and E, whose complexity is of O (WH(JN NI.2), R' requires a huge amount of operations and parameters, but only receives slightly better performance, as demonstrated in Table
Different Dictionary construction strategies. Table 5 presents an ablation study more differ- ent strategies for constructing the Dictionary. For VM, a VAE (Kingma & Welling 2014) is used to capture the dataset's general distribution and build the Dictionary. In VLM, we leverage whole CLIP (Radford et al. 2021y's s image encoder for global embeddings, as outlined in Section 3.2
Additionally, we explore using convex hull of the dataset's feature distribution address outliers, though this limits the RD-module's ability to represent common features. For LLM, normaliza- tion techniques like tanh and standard normalization are applied to manage outliers. While tanh compresses large feature distances, standard normalization is more effective in handling outliers.
Finally, we combine feature distributions from LLM and VLM to create blended representation. Despite yielding positive results, these methods fail to produce uniform distribution and underper- form compared the strategy described in the methodology.
4.4 VISUALIZATION
In this section, we employ visualization to demonstrate the impact of the RD-module object detection models and to visualize the Retriever's selection atoms from the Dictionary.
Visualization of Dictionary atom coefficients. To gain deeper understanding of the behavior of the Retriever and Dictionary core, we visualized the atom coefficients and their distribution during the forward pass. In Figure 4b) the X-axis represents the correlation with the current input and each

---

Published as a conference paper at ICLR 2025
044
gd
STOR
PUIS
(c) Positive
(d) Positive
-24-
-0.4
-2.2
0.0
0.1
07
Ed1
with More Ditionary
(a) Input Image
(b) Correlation-ceeffiicee
(c) Negative
(f) Negative 2
Figure 4: Visualization of Dictionary Atom Coefficients
STOP
(a) Model's 's input image
(b) The output of backbone
(c) The output of RD module
Figure 5: Visualization of w/o RD module
a, while the Y-axis displays the corresponding coefficient for each atom. The surrounding points retrieved from the dataset, relevant to the current input, are marked as x on the plot.
Using the bounding box area in Figure 4a as the input, we generate correlation-coefficii depicted Figure 4b most high-correlation and high-coefficient pairs (Figures 4d) correspond to traffic signs, while the low-correlation and low-coefficient pairs (Figures 4i) do not. This visualization demonstrates that the Retriever Dictionary effectively selects the relevant atoms to enhance input features while attenuating non-relevant atoms, thereby reinforcing the input's key characteristics.
map. As
Visualizing model backbone output with and without Dictionary. Using the same traffic signals as the model input (Figure 5a). we visualize the feature map generated by the origin backbone network (Figure (5b) and the backbone output including the RD-module (Figure [5c]. In Figure (5c the Traffic Sign pattern is distinctly retained, and background information is preserved. In contrast, Figure 55 retains only parts of the railings and the input photo padding block. This comparison demonstrates that the RD-module helps the model retain important information while eliminating unimportant details.
CONCLUSION
The Retriever Dictionary module offers lightweight and efficient approach incorporating dataset knowledge into YOLO through various modality models. By leveraging pre-stored explicit knowl- edge within the Dictionary, the Retriever effectively retrieves relevant information while the Dictio- nary learning mechanism enables fine-tuning of the atoms. This module demonstrates its versatility, providing improvements not only in YOLO-based tasks but also across a range foundational object detection models and broader computer vision tasks. We believe that this work lays the foun- dation for further exploration of real-time computer vision models that integrate explicit or external knowledge sources.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, al. GPT-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Michal Aharon, Michael Elad, and Alfred Bruckstein. K-SVD: An algorithm for designing over- complete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54, 2006.
Netzer Aharon, Eyal Gazit, and Itzik Bibi. BoT-SORT: Robust associations multi-pedessrian track- ing. arXiv preprint arXiv:2206. 14651, 2022.
Alexey Bochkovskiy, Chien-Ye Yao Wang, and Hong- Yuan Mark Liao. YOLOv4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
Jinkun Cao, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkar, and Kris Kitani. Observation-centric SORT: Rethinking SORT for robust multi-object tracking. In Proceedings the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the Eu- ropean Conference on Computer Vision (ECCV), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML). PMLR, 2020.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. YOLO-World: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations (ICLR), 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. International journal of computer vision, 88:303-338, 2010.
Cristina
and Brendt Wohlberg, Convolutional dictionary learning: A comparative review and new algorithms. IEEE Transactions on Computational Imaging, 2018.
Ross Girshick. Fast R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 0rXiv=1503.02531, 2015.
Glenn Jocher. ultralyticsyyolov5. v2.0, 2020.
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics YOLO, 2023. URL htttt:tts//gttthe
com/ultra oo
11

---

Published as a conference paper at ICLR 2025
Jooyeon Kim, Eulrang Cho, Sehyung Kim, and Hyunwoo J Kim. Retrieval-augmenn open- vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Confer- on Learning Representations (ICLR), 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR- 100 (canadian institute for advanced research). URL http://www.c..toro/ee hwtp::/ww...tttotoooe.errrrrrrra 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bart based learning applied to document recognition. Proceedings of the IEEE, (6(11))2278)
1998.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Rerrieval-augmented generation for knowledge-intensive NL.P tasks. In Proceedings of the Conference on Neural Information Processing Systems (NeurlPS), 2020.
Boyi Li, Felix Wu, Kilian Q Weinberger, and Serge Belongie. Positional normalization. In Proceed- ings of the Conference on Neural Information Processing Systems (NeurIPS). 2019.
Cheng Li, Qiaofeng Ye, Tongda Zhang, and Kai Chen. YOLOv6: A single-stage object detection framework for industrial applications. arXiv preprint 2xii:2209.009766, 2022.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2014.
Haotian Liu, Kilho Son, Jianwei Yang. Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li. Learning customized visual models with strieval-augmente knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15148-15158, 2023.
J Macqueen. Some methods for classification and analysis of multivariate observations. In Pro- ceedings of 5-th Berkeley Symposium on Mathematical Statistics and ProbabilityUUniversity of California Press, 1967.
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research, 37, 1997.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. In OpenAl, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning (ICML). PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 2020.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. DenseCLIP: tonggugge-gidee dense prediction with context-aware prompting, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Joseph Redmon and Ali Farhadi. YOLO9000: Better, faster, stronger. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Joseph Redmon and Ali Farhadi. YOLOv3: An incremental improvement. arXiv preprint arXiv: (00v:880.0000 02767, 2018.
12

---

Published as a conference paper at ICLR 2025
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: towards real-time object
detection with region proposal networks. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2015.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. ImageNet-21k pretraining for the masses. In Proceedings of the Conference Neural Information Processing Systems (NeurIPS), 2021.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems (NeurIPS), 29, 2016.
Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, and Yann LeCun. Over- Feat: Integrated recognition, localization and detection using convolutional networks. In Interna- tional Conference on Learning Representations (ICLR), 2014.
Qingyu Song, Changan Wang, Zhengkai Jiang, Yabiao Wang. Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yang Wu. Rethinking counting and localization in crowds: A purely point- based framework. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Ahmad S. Tarawneh, Ahmad B. Hassanat, Ceyhun Celik, Dmitry Chetverikov, M. Sohel Rahman, and Chaman Verma. Deep face image retrieval: a comparative study with dictionary learning. In 2019 10th International Conference on Information and Communication Systems (ICICS), 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the Confer- ence on Neural Information Processing Systems (NeurIPS), 2017.
Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, and Guiguang Ding. YOLOv10: Real-time end-to-end object detection. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2024a.
Chien Yao Wang, Hong-Yuan Yuan Mark Liao, Yuch-Hua Wu, Ping- Yang Chen, Wei Hsich, and I- Hau Yeh. CSPNet: A new backbone that can enhance learning capability of CNN. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRw), 2020.
Chien- Yao Wang, Alexey Bochkovskiy, and Hong- Yuan Mark Liao. YOLOV7: Trainable bag-of- freebies sets new state-of-the-ar for real-time object detectors. 2023a.
Chien Yao Wang, I-Hau Ych, and Hong-Yuse Yuan Mark Liao. You only learn one representation: Unified network for multiple tasks. Journal of Information Science and Engineering, 2023b.
Chien Yao Wang, I-Hau Yeh, and Hong- Yuan Mark Liao. YOLOv9: Learning what you want to learn using programmable gradient information. In Proceedings of the European Conference on Computer Vision (ECCV), 2024b.
Penghao Wu and Saining Xic. V*: Guided visual search as a core mechanism in multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2024.
Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Ping Luo, and Zehuan Yuan. ByteTrack: Multi- object tracking by associating every detection box. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting via multi-column convolutional neural network. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
13

---

Published as a conference paper at ICLR 2025
Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. DETRs beat YOLOs on real-time object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16965-16974, 2024.
Hongyi Zheng, Hongwei Yong, and Lei Zhang. Deep convolutional dictionary learning for image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: De- formable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2020.
Şaban Özüürk. Convolutional neural network based dictionary learning to create hash codes for content-based image retrieval. Procedia Computer Science, 2021. Proceedings of the 10th Inter- national Conference of Information and Communication Technology.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 REPRODUCIBILITY AND TRAINING SETUP
In 5 we observed that some of the mAP values for the original YOLOv7 and YOLOv9 models differ from those reported in the original papers. As noted in the corresponding GitHub issue, this discrepancy may be attributed to differences in the number of GPUs used, as well as the reduced effectiveness of batch normalization when training with smaller batch sizes. To ensure more con- sistent and fair comparison, we followed the official code instructions and re-trained the models on hardware setup, using the same batch sizes and number of GPUs specified in the original papers. Despite adhering to the official training guidelines, our re-run results for YOLOv7 and YOLOv9 produced slightly lower mAP values than those reported in the original papers. Nevertheless, implementation of the Retriever Dictionary (RD) method is still higher than those reported in the original papers, demonstrating the effectiveness of our approach.
For the Deformable DETR and Faster R-CNN models, we used the MMDetection training code. In this case, our results were consistent with the values reported in the original papers, likely due the fact that MMDetection provides standardized training batch configuration, minimizing the impact of hardware differences on model performance.
A.2 ARCHITECTURES OF THE MODEL WITH Retriever Dictionary
Backbone
Backbone
Retriever Girtionary
Ausillary
INPUT
INPUT
Network
A1
B2
Value
A2
B3
FPN
PAN
Head
A3
RD
F1
P1
PM
BBOK
BA
A4
B4
F2
P2
P2
B5
A5
86
F3
PS
Class
Figure 6: YOLOv7 with RD
Figure 7: YOLOv9 with RD
Backbone
Backhone
B1
INPUT
INPUT
Decoder
Encoder
BBox
B2
Self-Attn
Self-Attin
Crose-Attn
83
RD
F1
Self-Attn
RD
Self
Crose-Altn
BBox
F2
Pooling
Class
Self Attn
Self-Attn
Crose-Attn
85
F3
R3
B5
Object Querise
Figure 8: Faster-RCNN with RD
Figure 9: Deformable DETR with RD
A.3 ADDITIONAL EXPERIMENT RESULTS
Detailed mAP Results for All Experiments. Table 6 provides the detailed mAP values for each experiment discussed in the experiment section. It includes metrics such as AP, AP AP AP.75. APs- small object; APM. middle object; and APt, large object.
15

---

Published as a conference paper at ICLR 2025
Table 6: Comprehensive mAP Metries Across Experiments
BackBone
Initializer
Params
AP
AP.s
AP.rs
APs
APM
AP1
YOLOv7
VM
37.4M
51.34
69.37
56.42
35,86
56.10
66.78
YOLOv7
VLM
37.4M
51.75
70.12
56.70
36.00
57.12
66.32
YOLOv7
LLM
38.2M
51.33
69.40
56.40
34.72
56.70
67.13
YOLOv7
VAE
37.4M
51.34
69.37
56.42
35.86
56.10
66.78
YOLOv7
Convex
37.4M
49.58
68,92
55.60
34.72
55.14
66.02
YOLOv7
Tanh
37.4M
51.27
69.35
56.28
35.51
56.34
66.09
YOLOv7
Mix
37.4M
51.37
69.49
56.26
35.17
56.43
66.94
YOLOv9
VM
25.8M
53.41
70.57
58.40
36.29
58.93
69.94
YOLOvv
VLM
25.8M
53.36
70.55
57.98
36.39
59.04
69.95
YOLOv9
L.M
25.8M
53.28
70.48
57.81
35.91
58.80
69.55
Faster-RCNN
VM
44.1M
40.50
60.30
44,20
24.90
43.20
$1.80
Faster-RCNN
VLM
44.1M
40.50
60.40
44.40
25.10
43.30
52.50
Faster-RCNN
LLM
44.6M
40.70
60.80
44.70
25.60
43.70
52.60
Deformable DETR
VM
41.2M
44.10
63.00
48.20
26.30
47.00
59.00
Deformable DETR
VLM
41.2M
44,40
63.30
48.30
26.50
47.90
58.70
Deformable DETR
LLM
41.7M
44.20
63.10
47.80
26.20
47.60
58.70
YOLOv7 YOLOv7-x
baseline
37.2M
50.04
68.95
55.10
34.20
55,70
66.20
baseline
71.3M
51.29
70.27
56.78
35.84
56.63
67.59
Comparison of Frozen vs. Fully Trained Dictionary Strategies. Table compares two training strategies: one where only the model B and Retriever are trained while the Dictionary remains frozen, and another where the model, Retriever, and Dictionary are all fully trained. In the table, indicates that the corresponding component is trainable. Table shows results using pre-trained weights, with row displaying the fine-tuning of the original model. Table 7 reports results from training the model from scratch. The full training strategy slightly outperforms the frozen Dictionary method in both scenarios, with both approaches surpassing the performance of the original model. These findings highlight fine-tuning the Dictionary effectively helps the model's output distribution.
Table 7: Comparison of Model Performance with and without Freezing D during Training.
(a) Fine-tune with pre-trained weight.
Training from scratch.
B
5:.95
s
AP, 5:.95
AP.s
V 52.73
69.61
$1.72
70.12
/
52.64
69.57
51.35
69.52
51.66
68.12
Performance Improvements in Classification with RD Module. Table 8 demonstrates the per-
formance improvement of the Retriever Dictionary (RD) module in the classification task. In the YOLO classification task, YOLOv8 employs CSPNet (Wang al. 2020) the backbone, while YOLOv9 uses GELAN as the backbone. We tested both backbones with our RD module, and the results show that the RD module provides performance improvements in both structures.
Table 9: Transfer learning on small dataet.
Table 8: More classification task.
RD
Pretrained
Epoch
mAP(%)
mAP.:
RD
Model
Epoch
Top-1
Top-5
88.48
65.87
ELAN
100
71.85
92.93
/
10
91.54 (1 3.46%)
74.63 (1 13.30%)
ELAN
100
74.18
93.14
V
92.28
76.79
GELAN GELAN
100
74.86
93.72
/
/
100
92.93 (1 0.70%
78.02 (1 1.60%
100
75.70
94.28
84.44
65.49
100
85.15 (1 0.84%)
66.33 (1 1.28%)
16

---

Published as a conference paper at ICLR 2025
A.4 TRANSFER LEARNING WITH Retriever Dictionary ON VOC DATASET
In T we demonstrate the effectiveness of the Reiriever Dictionary (RD) on a transfer learning task. Using pre-trained weights from the MSCOCO dataset, trained the model on the VOC (Ev- eringham et al. 20T0) dataset with three learning rate schedules: 10-epoch fast training, 100-epoch full-tuning, and training from scratch. In the fast training scenario, the model with the RD mod- ule showed significantly faster convergence compared to the model without the module. In the full-tuning scenario, the RD-enhanced model achieved higher performance. Lastly, in the training from scratch scenario, our RD module provided the model with better information, yielding superior results even on a smaller dataset.
A.5 FURTHER VISUALIZATIONSS ORIGINAL VS. RD-MODDLEE MODELS
We present additional examples in Figures and I1 illustrating input images, the outputs from the original model, and the outputs from the model with the RD module. The results demonstrate that the RD-Model outputs noticeably clearer. For example, in Figure 10 (ID 1 and 2). the edges of objects are significantly sharpened. Similarly, Figures [10 and [1 exhibits higher accuracy and fewer false positives in the object's bounding boxes, as indicated by the red arrows.
(ID 3, 4, 5, and 6), our model
A.6 PSEUDO CODE OF FULL TRAINING PROCESS OF Retriever- Diireveer-Diccionary MODEL
The complete training process, from initialization final model, follows the pseudo-code pro- vided in Algorithm [ This process includes Dictionary initialization, regular model training, and Dictionary compression. The overall training time is approximately equivalent to the original train- ing epochs, with an additional 2 epochs allocated for setup and compression.
A.7 VISUALIZATION OF INITIAL DISTRIBUTIONS ACROSS DIFFERENT MODALITY MODELS
Figure 12 and 13 visualizes the t-SNE distributions VM, VLM, and LLM dictionaries. Vision and Language dictionaries occupy distinct regions, while Vision-Language overlaps with Vision. Notably, the Vision-Language dictionary is more uniformly distributed, showcasing its ability to provide richer information.
A.8 DEEPER DISCUSSION OF THE Retriever CORE
Two Convolutions Without Activation Functions. Consider two consecutive x 1 convolutional layers without activation functions. The first layer has weights W RMXN, and the second has weights 2 E xM with an input X E x
For the first convolutional layer, the output spatial location (h, w) is defined as:
Z
W Wh.
Xn,how
where W11 is the weight of the first convolution, E N) is the channels, and the second convo- lution's output is follows:
M
Z2
W-2
ZZ.
n.m
m==
M
N
W2
Σ n=l
W1
Xn.h.v
n.m
m,n
m=1
M
W n.m
Wl
Xnnh.w
17

---

Published as a conference paper at ICLR 2025
MD
Ground Truth
without RD
with RD
SALT BE 201
OUT Boorrd
2
Bookk
DRESON
pOrSON
Figure 10: More visualization of RD
18

---

Published as a conference paper at ICLR 2025
Ground Truth
without RD
with RD
4
Y9OD NODO
perSORTee Corrot
sprott
dring
lable
bedyyy beon
reddy beor
leddy
ludely bace
persoirant
person
Figure 11: More visualization of RD
19

---

Published as a conference paper at ICLR 2025
Algorithm 1: Train model with Retriever Dictionary
Data: Dataset with images and bounding boxes Result: Trained model with Retriever Dictionary of the Dictionary foreach (img, box) E Dataset do [eeatures encoder(img)
// Initializatia
RD & new aetiraeeeDDiciiooraayyaaaaaaatee
// Standard Training Method backbone, head + new Model()
for epoch = 1 to num-epochs do foreach (img, box) Dataset do
output <- heac(RDD(ackkbome (img))) loss + loss function output, box) update(loss, (backbone, RD, head))
10
// Dictionary Compression rd <- new Retriever-Dictionary choice.frome (D)) foreach (img, box) E Dataset do freeze(backbone)
13
teacher-feature <- RI((backbone(img))
15
student-featun <- rd(backbone(img))
loss t- cosine similarity teacher feature, student feature) update(loss, rd)
17
19
// Final Model: FullModel <- merge(backbone, rd, head)
20
1:80 Vosallation VLM VM. andiLMM
VLM
(a) 2D distributions.
(b) 3D distributions.
Figure 12: The initial distributions of dictionaries derived from different modality models.
20

---

Published as a conference paper at ICLR 2025
(a) X-axis
y-axis
(c) z-axis
Figure 13: Orthographic projections of the initial distributions using 3D t-SNE. Each subfigure
represents a different axis projection better illustrate the structure of the distributions.
Thus, the equivalent weight matrix is given by:
Wey D.n
W
W m.n' Vn,n' E N.
n,m
m=m
The original two-layer architecture has 2NM parameters. The equivalent layer has N parameters. Without an activation function, the number of parameters in the combined layer wasted by 2N M N N2. Even if M < 2 the rank of W will be limited by min [M, N), leading a significant drop abilities of the network. Therefore, activation functions are essential in most cases maintain the representational capacity of the sequential convolutional layers.
Forward Pass with the Retriever Core. We now extend this discussion to our Retriever core, consisting of two convolutional layers: pointwise (1 x 1) convolution for channel projection and depthwise convolution, with number of groups equal to the number channels. And there no activation function between layers.
For the pointwise convolution (first layer), we compute:
Y Ne
W
Xi,h
(10)
where the W., is the pointwise convolution weight, well as the Coefficient Generator weight. For the depthwise convolution (second layer), the operation is given by:
Beh.ww
Σ
W c.m,n Y Yohh+mwmt
(1I)
where the W.E... c.m.n the depthwise convolution weight, as well as Global Information Exchanger, c is the channel dimension. Since non-linear operation is applied between these two layers, combine them into a single equivalent convolution. Substituting the output of the first layer into the second, get:
21

---

Published as a conference paper at ICLR 2025
k
f
Zed = Σ
Σ
W
Σ
WC os
(12)
m,m,
m= R n=-k
W&
Σ
Σ
W
e,n,n X Ximtmmt
(13)
c,5.0,0
Mh= kn==
WG c,3,0,0
WE
c,m.n Xite+mmt
(14)
This results a combined convolution operation:
Wee C.l.m, c,1,0,0
W. W.f.m,
c,i,m,
where C, i, m, and n represent the input channels, output channels, and two spatial dimensions of the kernels, respectively. W c.s.0.0 denotes the pointwise convolution (with both spatial indices set to 0). The resulting equivalent kernel has size of k x k, with equivalent weights 1a E Rxxxxkxk x where f is the input feature dimension and N is the number of output channels. While this equivalent convolution maintains the functionality of the original two layers, the combined weights are computationally heavier due the different dimensions involved (channel dimension and kernel size). Nevertheless, even without an activation function, the model behaves as normal convolution operation.
Gradient Descent and Weight Update. Following the forward pass, the weight update rules for the pointwise and depthwise convolutional layers can expressed as:
aL WG' = WC - 11 DwG.
WEE = W 1) OWE'
OL
where ela represents the learning rate, and WG' and WG' denote the updated weights for the point- wise and depthwise convolutions, respectively. Retriever core update is then given by:
OL oWG
aL WE 1 OWE åL owG W
WG', WE'
W 7)
= GG WE 11 W ôL
ôL aL OWG
17
OWE
aWE'
Since (the learning rate) is generally small during training, especially when using FP16 preci- sion, the second-order term can be ignored. The simplified weight update for the equivalent kernel becomes:
ŌL
àL OWG
ØL
+ àL OWC
W F. ~ WWW.WWW.. WG
W. = Wa wy
"W
OWE
OWE
This follows the structure the Taylor expansion:
0f Dr
of
f(x + gx, σy) f(x,
8x
+
8y.
(15)
Dy
this gives the updated equivalent weight, which closely approximates the equivalent convolution:
aL W/ Weg 1) aaw..
G W.y
A.9 NOTATIONS
22

---

Published as a conference paper at ICLR 2025
Table 10: Notation Reference Table for Symbols Used in the Paper
Notation
Default Value
Description
D
The Dictionary, composed of learned atoms Number of atoms a the Dictionary D Each individual element (atom) in the Dictionary D The i-th atom the Dictionary
N
512
Og
W
80
The width dimension of the input to the RD module The height dimension of the input to the RD module Input feature map to the RD module
H
80
X
XA,
The pixel value at position (h, w) of the input X Dimensionality of each atom in the Dictionary Global Information Exchanger kernel size Coefficient matrix before normalized used weight atoms in the RD module Coefficient value before normalized for the i-th atom in the Dictionary D, at pixel (h, w) Normalized coefficient value for the i-th atom in the Dictionary D, at pixel (h, w)
512
5
h,,00
The feature of generate by backbone and module & for the i-th mini batch, at pixel (h, w)
0.8
Residual weight in the forward pass of the RD module Coefficient Generator function in the RD module
G(-)
WG
Convolutional matrix used in the Coefficient Generator G(.) Global Information Exchanger function in the RD module
E())
WE
Convolutional matrix used in the Global Information Exchanger E(-)
23


Reviews:
[{"review_id": "EvLC4RVLYV", "reviewer": "Reviewer_BYfS", "summary": "This paper aims to improve object detection with YOLO by introducing explicit knowledge in the form of a Retriever-Dictionary (RD) module. The dictionary is created using features from LLMs, VLMs, or VMs. The additional module helps the model retrieve relevant features from a dictionary that contains distilled information about the training dataset. The paper shows that such an addition leads to improved model performance.", "questions": "Please see the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The main strength of the paper is the increase in performance that is achieved by the proposed approach. The paper shows that the proposed approach leads to more than 3% performance improvements for object detection with YOLO. This result is quite good and demonstrates that the proposed approach might be useful. The paper also shows that the proposed approach can be used with other object detection models.", "weaknesses": "The major weaknesses of the paper are related to the lack of clarity about the motivation for the proposed approach and lack of clarity about several decisions. The authors should address the following in their response:\n\n1. It's not clear, why is the dataset dictionary needed at all. Why doesn’t batch gradient descent already incorporate dataset information during training? The paper does not make it clear why/how is the dataset information not used during standard model training. \n\n2. In lines 262-263, the paper mentions that the dimension of the dictionary features are the same dimension as YOLO middle layer dimension. Did the authors try incorporating dataset information at multiple locations in the network? It would help to see whether adding features at multiple levels (perhaps using dictionaries of different feature dimensions) improves performance.\n\n3. Further, why add dictionary feature information at only the middle layer? What about any other layer?  Are there any draw-backs to using another layer? \n\n4. The objective function written in section 3.3 (no equation number), needs to be explained better. It's not clear at all what is the objective trying to achieve and how does it achieve that. Such an explanation would help in understanding the motivation/approach more readily.\n\n\nEdit: The authors have addressed most of my and other reviewers' questions satisfactorily in their answers and updates. Therefore, I am raising the rating to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oDw7Y8VIte", "reviewer": "Reviewer_itZp", "summary": "This paper proposes the Retriever Dictionary module, which improves the model’s resource utilization by incorporating knowledge from other parts of the dataset, extending beyond the local region or individual image. The Retriever module aggregates regional features to generate a query, while the Dictionary can be an unimodal or multimodal model, such as YOLOv7, CLIP, or GPT. Experiments with multiple detectors, including YOLOv7, YOLOv9, Faster R-CNN, and Deformable DETR, demonstrate performance improvements.", "questions": "- The statement, \"We demonstrate that incorporating external knowledge from models such as VLMs and LLMs can significantly enhance model performance,\" reflects an already established idea in object detection [1, 2]. How does this paper uniquely contribute to this area?\n\n- The training strategy appears time-intensive. For example, Faster R-CNN is trained for 120 epochs, though a comparable baseline performance can typically be reached with 36 epochs (3x schedule). How do the authors justify this extended training duration?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-  The idea of leveraging external knowledge from the dataset for object detection is promising, especially with foundational large language models (LLMs) and vision-language models (VLMs) enabling this approach.\n\n- Applying dictionary learning to enhance object detection while maintaining a minimal increase in parameters is impressive.", "weaknesses": "- The primary weakness of this paper is the lack of computational analysis. The authors repeatedly emphasize \"carefully balancing accuracy, model parameters, and latency when handling external information.\" However, latency is not addressed in the experiments. Although the added trainable parameters are minimal, the integration of a large foundational model significantly impacts the pipeline. It’s crucial to include runtime/latency and FLOPs metrics, particularly for YOLOv7 and YOLOv9 in Tables 1-3. This trade-off will help verify the effectiveness of the proposed approach. \n\n- Employing different dictionaries demonstrates different behaviors. For instance, VLM improves YOLO models more than Faster R-CNN with LLM. This variation lacks justification, which seems essential to understand the behavior.\n\n\n[1] Wasim, Syed Talal, et al. \"VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Liu, Shilong, et al. \"Grounding dino: Marrying dino with grounded pre-training for open-set object detection.\" arXiv preprint arXiv:2303.05499 (2023).\n\n\nMinor:\n- The conclusion should be separated from the Experiment section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pRvDVCdCSm", "reviewer": "Reviewer_DigY", "summary": "The paper introduces the Retriever Dictionary (RD) module, a lightweight and efficient approach to integrating explicit knowledge into existing object detectors like YOLO, Faster R-CNN and DETR. RD outperforms other knowledge integration methods by introducing only 0.2M additional parameters while achieving superior performance. The module leverages pre-stored explicit knowledge within the Dictionary to enhance input features, demonstrating improvements across various YOLO-based tasks and broader computer vision applications like segmenation. The article also discusses ablation studies and visualizations that highlight the effectiveness of the RD module in retaining important information while eliminating unimportant details.", "questions": "see Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Innovation: It presents a approach to integrate explicit knowledge into YOLO-based models by \"efficiently\" retrieving features from a dictionary built from insights across the entire dataset.\n* Efficiency & Performance: RD achieves superior performance in object detection tasks, introducing only a minimal additional computational overhead~(with 0.2M additional parameters). And the paper shows it the lightest solution among compared methods.\n* Visualization: The article includes visualizations that show RD's effectiveness in retaining important information while eliminating unimportant details.", "weaknesses": "1. While the paper effectively demonstrates the RD module's potential to enhance model accuracy with minimal computational increase, the inclusion of speed metrics would provide a more holistic view of the module's practicality and readiness for real-world deployment.\n2. The knowledge integration methods compared in the article not only introduce external knowledge but also possess the capability for open-world detection. In contrast, the current work is primarily tailored for specific datasets. Although the paper demonstrates improved accuracy, the overall significance is diminished as a result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2fF0DzCuo1", "reviewer": "Reviewer_hT7W", "summary": "The paper purposed a Retriever-Dictionary (RD) module to solve the balancing problem between current input and whole dataset information in object identification and localization. Its retriever processes region features to generate a query, and the dictionary that saves dataset information could prompt the query to select relevant atoms. The dictionary incorporates different modalities as encoder, ensuring a good overall understanding of the dataset in various modality spaces.", "questions": "Have you tried incorporating multple modality initialization and alignment for the dictionary? For example, VM and VLM initialization together? For more questions, please refer to the weaknesses listed above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Rigorousity of pipeline design.** The whole Retriever Dictionary design is theoretically robust. The authors have considered the overall operation of Retriever and designed Global Information Exchanger with Coefficient Generator. The introduction of PONO also helps to prevent Dictionary from collapsing into an identity matrix. \n\n2. **Innovative incorporation of multimodal external knowledge in dictionary building.** The authors purposed that the dictionary could be initialized by multiple modalities of models, ensuring a comprehensive representation of the dataset. In response to huge amount of feature vectors involved, operations including k-means and contrasive learning distillation have made the dictionary condensed and efficient.\n\n3. **Congeneric methodology design.** The framework is extendable to RCNNs, Transformer-based detection models and other kindred models. The authors have incorporated experiments on these models. And their experiments supports their theory basically well.", "weaknesses": "1. The paper's figure lacks generalization for the whole work, i.e. Figure 1 and 2 cannot fully describe the complete functionality of the RD module. From the dictionary and module structural description, the whole pipeline is hard to follow as the authors have mentioned \n\n2. The authors refer to RAG in their related work and mention its limitations in vision applications. RAG is a system that relies on external multi-modules and is mainly applied to Large language models.The article tries to compare its consistency with RAG in terms of knowledge base ideas, but lacks further explanations on this. e.g. How exactly does it differ from a RAG as part of the network structure?\n\n3. Fig.5, the role of the RD module could have a richer and more well annotated picture explanation. Depending on the modality of initialisation (VM, VLM, LLM), there is also a strong need to visualize the effect of different modalities for initialization. \n\n4. Although the model introduces minimal parameter increase, there is limited discussion on its impact on computational speed and inference latency, especially for deployment in real-time applications.\n\n5. The dictionary initialization process, particularly when using VLMs or LLMs, is not straightforward and may demand significant computational resources during setup. More details on optimizing or reducing this process would be beneficial. \n\n6. The module's efficiency in managing larger datasets or dynamic, continuously updated dictionaries remains unclear. Further analysis on handling diverse data scales could improve its adaptability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)