SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Vishal Asnani
asnanivi@msu.edu
Abhinay Kumar
kumarab6@msu.edu
Suya You
DEVCOM Army Research Laboratory
suya.you.civ@army.mil
Xiaoming Liu
liuxm@cse.msu.edu
Previous research in  $2D$  object detection focuses on various tasks, including detecting objects in generic and camouflaged images. These works are regarded as passive works for object detection as they take the input image as is. However, convergence to global minima is not guaranteed to be optimal in neural networks; therefore, we argue that the trained weights in the object detector are not optimal. To rectify this problem, we propose a wrapper based on proactive schemes, PrObeD, which enhances the performance of these object detectors by learning a signal. PrObeD consists of an encoder-decoder architecture, where the encoder network generates an image-dependent signal termed templates to encrypt the input images, and the decoder recovers this template from the encrypted images. We propose that learning the optimum template results in an object detector with an improved detection performance. The template acts as a mask to the input images to highlight semantics useful for the object detector. Finetuning the object detector with these encrypted images enhances the detection performance for both generic and camouflaged. Our experiments on MS-COCO, CAMO, COD10K, and NC4K datasets show improvement over different detectors after applying PrObeD. Our models/codes are available at https://github.com/vishal3477/Proactive-Object-Detection.
Generic  $2D$  object detection (GOD) has improved from earlier traditional detectors [15, 20, 64, 65] to the deep-learning-based object detectors [8, 10, 26, 32, 52, 58]. Advancements in deep-learning-based methods underwent many architectural change over recent years, including one-stage [5, 43, 46, 52-54], two-stage [23, 24, 58], CNN-based [5, 14, 16, 21-23, 52, 54], transformer-based [8, 74], and diffusion-based [10] methods. All these methods aim to predict the  $2D$  bounding box of the objects in the images and their category class.
Another emerging area related to generic object detection is camouflaged object detection [17, 18, 27-29, 34, 40] (COD). COD aims to detect and segment objects blended with the background [17, 18] via object-level mask supervision. Applications of COD include medical [19, 45], surveillance [11] and autonomous driving [69]. Early COD detectors exploit hand-crafted features [50, 61] and optical flow [33], while current methods are deep-learning-based. These methods utilize attention [9, 63], joint learning [40], image gradient [34], and transformers [48, 70].
All these methods take input images as is for the detection task and hence are called passive methods. However, there is a line of research on proactive methods for a wide range of vision tasks such as disruption [59, 60], tagging [68], manipulation detection [1], and localization [2]. Proactive methods
Figure 1: (a) Passive vs. Proactive object detection. A learnable template encrypts the input images, which are further used to train the object detector. (b) PrObeD serves as a wrapper on both generic and camouflaged object detectors, enhancing the detection performance. (c) For the linear regression model under additive noise and other assumptions, the converged weights of the proactive detector are closer to the optimal weights as compared to the converged weights of the passive detector. See Sec. 3.2 for details and proof.
use signals, called templates, to encrypt the input images and pass the encrypted images as the input to the network. These are trained in an end-to-end manner by using either a fixed [68] or learnable template [1, 2, 59, 60] to improve the performance. A major advantage of proactive schemes is that such methods generalize better on unseen data/models [1, 2]. Motivated by this, we propose a plug-and-play Proactive Object Detection wrapper, PrObeD, to improve GOD and COD detectors.
Designing PrObeD as a proactive scheme involves several challenges and key factors. First, the proactive wrapper needs to be a plug-and-play module that can be applied to both GOD and COD detectors. Secondly, the encryption process should be intuitive to benefit the object detection task. e.g., an ideal template for detection should highlight the foreground objects in the input image. Lastly, the choice of supervision to estimate the template for encryption is hard to formulate.
Previous proactive methods [1, 2] use learnable but image-independent templates for manipulation and localization tasks. However, the object detection task is scene-specific; therefore, the ideal template should be image-dependent. Based on this key insight, we propose a novel plug-and-play proactive wrapper in which we apply object detectors to enhance detection performance. The PrObeD wrapper utilizes an encoder network to learn an image-dependent template. The learned template encrypts the input images by applying a transformation, defined as an element-wise multiplication between the template and the input image. The decoder network recovers the templates from the encrypted images. We utilize regression losses for supervision and leverage the ground-truth object map to guide the learning process, thereby imparting valuable object semantics to be integrated into the template. We then fine-tune the proactive wrapper with the GOD and COD detectors to improve their detection performance. Extensive experiments on MS-COCO, CAMO, COD10K, and NC4K datasets show that PrObeD improves the detection performance for both GOD and COD detectors.
In summary, the contributions of this work include:
- We propose a novel proactive approach  $PrObeD$  for the object detection task. To the best of our knowledge, this is the first work to develop a proactive approach to  $2D$  object detection.
- We mathematically prove that the proactive method results in a better-converged model than the passive detector under assumptions and, consequently, a better object detector.
- PrObeD wraps around both GOD and COD detectors and improves detection performance on MS-COCO, CAMO, COD10K, and NC4K datasets
Proactive Schemes. Earlier works adopt to add signals like perturbation [60], adversarial noise [59], and one-hot encoding [68] messages while focusing on tasks like disruption [59, 60] and deepfake tagging [68]. Asnani et al. [1] propose to learn an optimized template for binary detection by unseen generative models. Recently, MaLP [2] adds the learnable template to perform generalized
Table 1: Comparison of PrObeD with prior works.
Method	Proactive	Task	Template	COD	GOD	Plug-Play
Number	Type
Faster R-CNN [58]	×	Object Detection	-	-	×	✓	×
YOLO [52]	×	Object Detection	-	-	×	✓	×
DeTR [8]	×	Object Detection	-	-	×	✓	×
DGNet [34]	×	Object Detection	-	-	✓	×	×
SINet-v2 [17]	×	Object Detection	-	-	✓	×	×
JCSOD [40]	×	Object Detection	-	-	✓	×	×
OGAN [60]	✓	Disrupt	1	Learnable	-	-	×
Ruiz et al. [59]	✓	Disrupt	1	Learnable	-	-	×
Yeh et al. [71]	✓	Disrupt	1	Learnable	-	-	×
FakeTagger [68]	✓	Tagging	≥ 1	Fixed, Id-dependent	-	-	×
Asnani et al. [1]	✓	Manipulation Detection	≥ 1	Learnable set, Image-independent	-	-	✓
MaLP [2]	✓	Manipulation Localization	≥ 1	Learnable set, Image-independent	-	-	✓
PrObeD (Ours)	✓	Object Detection	≥ 1	Learnable, Image-dependent	✓	✓	✓
The manipulation localization for unknown generative models. Unlike these works, PrObeD uses image-dependent templates and is a plug-and-play wrapper for a different task of object detection.
Generic Object Detection Detection of generic objects, instead of specific object categories such as pedestrians [7], apples [13], and others [4, 37, 38], has been a long-standing objective of computer vision. RCNN [24, 25] employs the extraction of object proposals. He et al. [31] propose a spatial pooling layer to extract a fixed-length representation of all the objects. Modifications of RCNN [23, 41, 58, 72] increase the inference speed. Feature pyramid network [42] detects objects with a wide variety of scales. The above methods are mostly two-stage, so inference is an issue. Single-stage detectors likeYOLO [5, 52-54, 66], SSD [46], HRNet [67] and RetinaNet [43] increase the speed and simplicity of the framework compared to the two-stage detector. Recently, transformer-based methods [8, 74] use a global-scale receptive field. Chen et al. [10] use diffusion models to denoise noisy boxes at every forward step. PrObeD functions as a wrapper around the pre-existing object detector, facilitating its transformation into an enhanced object detector. The comparison of PrObeD with prior works is summarized in Tab. 1.
**Camouflaged Object Detection** Early COD works rely on hand-crafted features like co-occurrence matrices [61],  $3D$  convexity [50], optical flow [33], covariance matrix [35], and multivariate calibration components [57]. Later on, [9, 63] incorporate an attention-based cross-level fusion of multi-scale features to recover contextual information. Mei et al. [49] take motivation by predators to identify camouflaged objects using a position and focus ideology. SINet [18] uses a search and identification module to perform localization. SINET-v2 [17] uses group-reversal attention to extract the camouflaged maps. [36] explores uncertainty maps and [75] utilizes cube-like architecture to integrate multi-layer features. ANet [39], LSR [47], and JCSOD [40] employ joint learning with different tasks to improve COD. Lately, [12, 48, 70] apply a transformer-based architecture for difficult-aware learning, uncertainty modeling, and temporal consistency. Zhai et al. [73] use a graph learning model to disentangle input into different features for localization. DGNet [34] uses image gradients to exploit intensity changes in the camouflaged object from the background. Unlike these methods, PrObeD uses proactive methods to improve camouflaged object detection.
Our method originates from understanding what makes proactive schemes effective. We first overview the two detection problems: GOD and COD in Sec. 3.1. We next derive Lemma 1, where we show that the proactive schemes with the multiplicative transformation of images are better than passive schemes by comparing the deviation of trained network weights from the optimal. Based on this result, we derive that Average Precision (AP) from the proactive model is better than AP from the passive model in Theorem 1. At last, we present our proactive scheme-based wrapper, PrObeD, in Sec. 3.3, which builds upon the Theorem 1 to improve generic 2D objects and camouflaged detection.
3.1.1 Passive Object Detection
Although generic  $2D$  object detection and camouflage detection are similar problems, they have different objective functions. Therefore, we treat them as two different problems and define their objectives separately.
Generic 2D Object Detection. Let  $I_{j}$  be the set of input images given to the generic 2D object detector  $\mathcal{O}$  with trainable parameters  $\theta$ . Most of these detectors output two sets of predictions per image: (1) bounding box coordinates,  $\mathcal{O}(I_j)_1 = \hat{T} \in \mathbb{R}^4$ , (2) class logits,  $\mathcal{O}(I_j)_2 = \hat{C} \in \mathbb{R}^C$ , where  $N$  is the number of foreground object categories. If the ground-truth bounding box coordinates are  $T_{j}$ , and the ground-truth category label is  $C$ , the objective function of such detector is:
$$
\min  _ {\theta} \left\{\sum_ {j} \left(\left| \left| \mathcal {O} \left(\boldsymbol {I} _ {j}; \theta\right) _ {1} - T _ {j} \right| \right| _ {2}\right) - \sum_ {j} \sum_ {i = 1} ^ {N} \left(C _ {j} ^ {i} \cdot \log \left(\mathcal {O} \left(\boldsymbol {I} _ {j}; \theta\right) _ {2}\right)\right) \right\}. \tag {1}
$$
Camouflaged Object Detection. Let  $I_{j}$  be the input image set given to the camouflaged object detector  $\mathcal{O}$  with trainable parameters  $\theta$ , and  $G_{j}$  be the ground-truth segmentation map. Prior passive works predict a segmentation map with the following objective:
$$
\min  _ {\theta} \left\{\sum_ {j} \left(\left| \left| \mathcal {O} \left(I _ {j}; \theta\right) - G _ {j} \right| \right| _ {2}\right) \right\}. \tag {2}
$$
3.1.2 Proactive Object Detection
Proactive schemes [1, 2] encrypt the input images with the template to aid manipulation detection/localization. Such schemes take an input image  $\mathbf{I}_j \in \mathbb{R}^{H \times W \times 3}$  and learns a template  $S_j \in \mathbb{R}^{H \times W}$ . PrObeD uses image-dependent templates to improve object detection. Given an input image  $\mathbf{I}_j \in \mathbb{R}^{H \times W \times 3}$ , PrObeD learns to output a template  $S_j \in \mathbb{R}^{H \times W}$ , which can be used by a transformation  $\mathcal{T}$  resulting in encrypted images  $\mathcal{T}(I_j)$ . PrObeD uses element-wise multiplication as the transformation  $\mathcal{T}$ , which is defined as:
$$
\mathcal {T} \left(\boldsymbol {I} _ {j}\right) = \mathcal {T} \left(\boldsymbol {I} _ {j}; \boldsymbol {S} _ {j}\right) = \boldsymbol {I} _ {j} \odot \boldsymbol {S} _ {j}. \tag {3}
$$
3.2 Mathematical Analysis of Passive and Proactive Detectors
PrObeD optimizes the template to improve the performance of the object detector. We argue that this template helps arrive at a better global minima representing the optimal parameters  $\theta$ . We now define the following lemma to support our argument:
Lemma 1. Converged weights of proactive and passive detectors. Consider a linear regression model that regresses an input image  $\mathbf{I}_j$  under an additive noise setup to obtain the 2D coordinates. Assume the noise under consideration is a normal random variable  $\mathcal{N}(0, \sigma^2)$ . Let  $\mathbf{w}$  and  $\mathbf{w}^*$  denote the trained weights of the pretrained linear regression model and the optimal weights of the linear regression model. Also, assume SGD optimizes the model parameters with decreasing step size  $s$  such that the steps are square summable i.e.,  $S = \lim_{t \to \infty} \sum_{k=1}^{t} s_k^2$  exist, and the noise is independent of the image. Then, there exists a template  $S_j \in [0,1]$  for the image  $\mathbf{I}_j$  such that the multiplicative transformation of images as the input results in a trained weight  $\mathbf{w}'$  closer to the optimal weight than the originally trained weight  $\mathbf{w}$ . In other words,
$$
\mathbb {E} \left(\left| \left| \boldsymbol {w} ^ {\prime} - \boldsymbol {w} ^ {*} \right| \right| _ {2}\right) <   \mathbb {E} \left(\left| \left| \boldsymbol {w} - \boldsymbol {w} ^ {*} \right| \right| _ {2}\right). \tag {4}
$$
The proof of Lemma 1 is in supplementary. We use the variance of the gradient of the encrypted images to arrive at this lemma. We next use Lemma 1 to derive the following theorem:
Theorem 1. AP comparison of proactive and passive detectors. Consider a linear regression model that regresses an input image  $\mathbf{I}_j$  under an additive noise setup to obtain the 2D coordinates. Assume the noise under consideration  $e$  is a normal random variable  $\mathcal{N}(0, \sigma^2)$ . Let  $\mathbf{w}$  and  $\mathbf{w}^*$  denote the
Proactive wrapper
Figure 2: Overview of PrObeD. PrObeD consists of three stages: (1) template generation, (2) template recovery, and (3) detector fine-tuning. The templates are generated by encoder network  $\mathcal{E}$  to encrypt the input images. The decoder network  $\mathcal{D}$  is used to recover the template from the encrypted images. Finally, the encrypted images are used to fine-tune the object detector to perform detection. We train all the stages in an end-to-end manner. However, for inference, we only use stages 1 and 3. Best viewed in color.
trained weights of the pretrained linear regression model and the optimal weights of the linear regression model. Also, assume SGD optimizes the model parameters with decreasing step size  $s$  such that the steps are square summable i.e.,  $S = \lim_{t\to \infty}\sum_{k = 1}^{t}s_k^2$  exist, and the noise is independent of the image. Then, the AP of the proactive detector is better than the AP of the passive detector.
The proof of Theorem 1 is in the supplementary. We use the Lemma 1 and the non-decreasing nature of AP w.r.t. IoU to arrive at this theorem. Next, we adapt the objectives of Eqs. (1) and (2) to incorporate the proactive methods as follows:
$$
\left. \min  _ {\theta , S _ {j}} \left\{\sum_ {j} \left(\left| \left| \mathcal {O} \left(\mathcal {T} \left(\boldsymbol {I} _ {j}; \boldsymbol {S} _ {j}\right); \theta\right) _ {1} - T _ {j} \right| \right| _ {2}\right) - \sum_ {j} \sum_ {i = 1} ^ {N} \left(C _ {j} ^ {i} \cdot \log \left(\mathcal {O} \left(\mathcal {T} \left(\boldsymbol {I} _ {j}; \boldsymbol {S} _ {j}\right); \theta\right) _ {2}\right)\right) \right\}, \right. \tag {5}
$$
$$
\left. \min  _ {\theta , \mathbf {S} _ {j}} \left\{\sum_ {j} \left(\left| \left| \mathcal {O} \left(\mathcal {T} \left(\mathbf {I} _ {j}; \mathbf {S} _ {j}\right); \theta\right) - \mathbf {G} _ {j} \right| \right| _ {2}\right) \right\}. \right. \tag {6}
$$
3.3 PrObeD
Our proposed approach comprises of three stages: template generation, template recovery, and detector fine-tuning. First, we use an encoder network to generate an image-dependent template for image encryption. This encrypted image is further used to recover the template through a decoder network. Finally, the object detector is fine-tuned using the encrypted images. All three stages are trained in an end-to-end fashion. While all the stages are used for training PrObeD, we specifically use only stages 1 and 3 for inference. We will now describe each stage in detail.
3.3.1 Proactive Wrapper
Our proposed approach consists of three stages, as shown in Fig. 2. However, only the first two stages are part of our proposed proactive wrapper, which can be applied to object detector to improve its performance.
Stage 1: Template Generation. Prior works learn a set of templates [1,2] in their proactive schemes. This set of templates is enough to perform the respective downstream tasks as the generative model manipulates the template, which is easy to capture with a set of learnable templates. However, for object detection tasks, every image has unique object characteristics such as size, appearance, and color that can vary significantly. This variability present in the images may exceed the descriptive capacity of a finite set of templates, thereby necessitating the use of image-specific templates to
accurately represent the range of object features present in each image. In other words, a fixed set of templates may not be sufficiently flexible to capture the diversity of visual features across the given set of input images, thus demanding more adaptable, image-dependent templates.
Motivated by the above argument, we propose to generate the template  $S_{j}$  for every image using an encoder network. We hypothesize that highlighting the area of the key foreground objects would be beneficial for object detection. Therefore, for GOD, we use the ground-truth bounding boxes  $T^{G}$  to generate the pseudo ground-truth segmentation map. Specifically, for any image  $I_{j}$ , if the bounding box coordinates are  $T_{j}^{G} = \{x_{1}, x_{2}, y_{1}, y_{2}\}$ , we define the pseudo ground-truth segmentation map as:
$$
\forall m \in [ 0, H ], n \in [ 0, W ], \text {w e h a v e}
$$
$$
\boldsymbol {G} _ {j} (m, n) = 1 \text {i f} x _ {1} \leq m \leq x _ {2} \text {a n d} y _ {1} \leq n \leq y _ {2}, \text {o t h e r w i s e} 0
$$
However, for COD, the dataset already has the ground-truth segmentation map  $G_{j}$ , which we use as the supervision for the encoder to output the templates with semantic information of the image to be restricted only in the region of interest for the detector. For both GOD and COD, we minimize the cosine similarity (Cos) between  $S_{j}$  and  $G_{j}$  as the supervision for the encoder network. The encoder loss  $J_{E}$  is as follows:
$$
J _ {E} = 1 - \operatorname {C o s} \left(\boldsymbol {S} _ {j}, \boldsymbol {G} _ {j}\right) = 1 - \operatorname {C o s} \left(\mathcal {E} \left(\boldsymbol {I} _ {j}\right), \boldsymbol {G} _ {j}\right). \tag {7}
$$
This generated template acts as a mask for the input image to highlight the object region of interest for the detector. We use this template with the transformation  $\mathcal{T}$  to encrypt the input image as  $\mathcal{T}(I_j;S_j) = I_j\odot S_j$ . As we start from the pretrained model of object detector  $\mathcal{O}$ , we initialize the bias of the last layer of the encoder as 0 so that for the first few iterations,  $S_{j}\approx \mathbf{1}$ . This is to ensure that the distribution of  $I_{j}$  and  $\mathcal{T}(\boldsymbol {I}_j;\boldsymbol {S}_j)$  remains similar for the first few iterations, and  $\mathcal{O}$  doesn't encounter a sudden change in its input distribution.
Stage 2: Template Recovery. So far, we have discussed the generation of template  $S_{j}$  using  $\mathcal{E}$ , which will be used as a mask to encrypt the input image. The encrypted images are used for two purposes: (1) recovery of templates and (2) fine-tuning of the object detector. The main intuition of recovering the templates is from the prior works on image steganalysis [55, 56] and proactive schemes [1, 2]. Motivated by these works, we draw the following insight: "To properly learn the optimal template and embed it onto the input images, it is beneficial to recover the template from encrypted images."
To perform recovery, we exploit an encoder-decoder approach. Using this approach leverages the strengths of the encoder network  $\mathcal{E}$  for feature extraction, capturing the most useful salient details, and the decoder network  $\mathcal{D}$  for information recovery, allowing for efficient and effective encryption and decryption of the template. We also empirically show that not using the decoder to recover the templates harms the object detection performance.
To supervise  $\mathcal{D}$  in recovering  $S_{j}$  from  $\mathcal{T}(I_j;S_j)$ , we propose to maximize the cosine similarity between the recovered template,  $S_{j}^{\prime}$  and  $S_{j}$ . The decoder loss is as follows:
$$
J _ {D} = 1 - \operatorname {C o s} \left(\boldsymbol {S} _ {j} ^ {\prime}, \boldsymbol {S} _ {j}\right) = 1 - \operatorname {C o s} \left(\mathcal {D} \left(\mathcal {T} \left(\boldsymbol {I} _ {j}; \boldsymbol {S} _ {j}\right)\right), \boldsymbol {S} _ {j}\right). \tag {8}
$$
Stage 3: Detector Fine-tuning. Due to our encryption, the distribution of the images input to the pretrained  $\mathcal{O}$  changes. Thus, we fine-tune  $\mathcal{O}$  on the encrypted images  $\mathcal{T}(\boldsymbol{I}_j; \boldsymbol{S})$ . As proposed in Theorem 1, given the encrypted images  $\mathcal{T}(\boldsymbol{I}_j; \boldsymbol{S})$ , we use the pretrained detector  $\mathcal{O}$  with parameters  $\theta$  to arrive at a better local minima. Therefore, the general objective of GOD and COD in Eq. (5) and Eq. (6) change to as follows:
$$
\left. \right. \min  _ {\theta , \theta_ {\mathcal {E}}, \theta_ {\mathcal {D}}} \left\{ \right.\sum_ {j} \left( \right.\left|\left| \mathcal {O} \left(\mathcal {T} \left(\boldsymbol {I} _ {j}; \mathcal {E} \left(\boldsymbol {I} _ {j}; \theta_ {\mathcal {E}}\right)\right); \theta , \theta_ {\mathcal {D}}\right) _ {1} - T _ {j} \right|\right| _ {2} - \sum_ {i = 1} ^ {N} \left(C _ {j} ^ {i}. \log \left(\mathcal {O} \left(\mathcal {T} \left(\boldsymbol {I} _ {j}; \mathcal {E} \left(\boldsymbol {I} _ {j}; \theta_ {\mathcal {E}}\right)\right); \theta , \theta_ {\mathcal {D}}\right) _ {2}\right)\right)\left. \right\}, \tag {9}
$$
$$
\left. \min  _ {\theta , \theta_ {\mathcal {E}}, \theta_ {\mathcal {D}}} \left\{\sum_ {j} \left(\left| \left| \mathcal {O} \left(\mathcal {T} \left(\boldsymbol {I} _ {j}; \mathcal {E} \left(\boldsymbol {I} _ {j}; \theta_ {\mathcal {E}}\right)\right); \theta , \theta_ {\mathcal {D}}\right) - \boldsymbol {G} _ {j} \right| \right| _ {2}\right) \right\}. \right. \tag {10}
$$
We use the detector-specific loss function  $J_{OBJ}$  of  $\mathcal{O}$  along with the encoder and decoder loss in Eq. (7) and Eq. (8) to train all the three stages. The overall loss function  $J$  to train PrObeD is as follows:
$$
J = \lambda_ {O B J} J _ {O B J} + \lambda_ {E} J _ {E} + \lambda_ {D} J _ {D}. \tag {11}
$$