SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: GOD results on MS-COCO val split. PrObeD improves the performance of all GOD at all thresholds and across all categories.
Method	AP ↑	\( AP_{50} \) ↑	\( AP_{75} \) ↑	\( AP_S \) ↑	\( AP_M \) ↑	\( AP_L \) ↑
Faster R-CNN [58]	19.3	42.5	16.9	1.8	17.9	39.3
Faster R-CNN [58]+PrObeD	31.7	52.6	33.3	11.0	35.5	51.1
Faster R-CNN + FPN [42]	37.3	58.0	40.6	21.4	41.0	48.4
Faster R-CNN + FPN [42] + Seg. Mask [30]	38.2	60.3	41.7	22.1	43.2	51.2
Faster R-CNN + FPN [42] + PrObeD	38.5	60.4	41.9	22.5	43.4	49.8
Sparse R-CNN [62]	37.6	55.6	40.2	20.5	39.6	52.9
Sparse R-CNN [62] + PrObeD	39.2	57.5	41.5	21.7	40.1	53.6
YOLOv5 [52]	48.9	67.6	53.1	31.8	54.4	62.3
YOLOv5 [52] + PrObeD	49.4	67.9	53.5	32.0	55.1	62.6
DeTR [8]	41.9	62.3	44.1	20.3	45.8	61.0
DeTR [8] + PrObeD	42.1	62.6	44.4	20.4	46.0	61.3
Table 3: COD results on CAMO, COD10K and NC4K datasets. PrObeD outperforms DGNet on all datasets and metrics.
Method	CAMO	COD10K	NC4K
Em↑	Sm↑	wFβ↑	MAE↓	Em↑	Sm↑	wFβ↑	MAE↓	Em↑	Sm↑	wFβ↑	MAE↓
DGNet [34]	0.859	0.791	0.681	0.079	0.833	0.776	0.603	0.046	0.876	0.815	0.710	0.059
+ PrObeD	0.871	0.797	0.702	0.071	0.869	0.803	0.661	0.037	0.900	0.838	0.755	0.049
We apply PrObeD for two categories of object detectors: GOD and COD.
GOD Baselines. For GOD, we apply PrObeD on four detectors with varied architectures: two-stage, one-stage, and transformer-based detectors, namely, Faster R-CNN [58], YOLO [52], Sparse R-CNN, and DeTR [8]. We use these works as baselines for three reasons: (1) varied architecture types, (2) their increased prevalence in the community, and (3) varied timelines (from earlier to recent detectors). We use the PyTorch [51] code of the respective detectors for our GOD experiments and use the corresponding GODs as our baseline. For YOLOv5 and DeTR, we use the official repositories released by the authors; for Faster R-CNN, we use the public repository "Faster R-CNN.pytorch". For other GOD detectors, we use Detectron2 library as the pre-trained detector. We use the ResNet101 backbone for Faster R-CNN, Sparse R-CNN and DeTR, and CSPDarknet53 for YOLOv5.
COD Baselines. For COD, we apply PrObeD on the current SoTA camouflage detector DGNet [34] and use DGNet as our baseline. For all object detectors, we use the pretrained model released by the authors and fine-tune them with PrObeD. Please see the supplementary for more details.
Datasets. Our experiments use the MS-COCO 2017 [44] dataset for GOD, while we use CAMO [39], COD10K [17], and NC4K [47] datasets for COD. We use the following splits of these datasets:
- MS-COCO 2017 Val Split [44]: It includes 118,287 images for training and  $5K$  for testing.
- COD10K Val Split [17]: It includes 4,046 camouflaged images for training and 2,026 for testing.
- CAMO Val Split [39]: It includes  $1K$  camouflaged images for training and 250 for testing.
- NC4K Val [47]: It includes 4,121 NC4K images. We use it for generalization testing as in [34].
Evaluation Metrics. We use mean average precision average at multiple thresholds in [0.5, 0.95] (AP) for GOD as in [44]. We also report results at threshold of 0.5  $(\mathrm{AP}_{50})$ , threshold of 0.75  $(\mathrm{AP}_{75})$  and at different object sizes: small  $(\mathrm{AP}_S)$ , medium  $(\mathrm{AP}_M)$ , and large  $(\mathrm{AP}_L)$ . For COD, we use E-measure  $E_{m}$ , S-measure  $S_{m}$ , weighted F1 score  $wF_{\beta}$  and mean absolute error  $MAE$  as [34].
4.1 GOD Results
Quantitative Results. Tab. 2 shows the results of applying PrObeD on GOD networks. PrObeD improves the average precision of all three detectors. The performance gain is significant for Faster R-CNN. As Faster R-CNN is an older detector, it was at a worse minima to start with. PrObeD improves the convergence weight of Faster R-CNN by a significant margin, thereby improving the performance. We further experiment with two variations of Faster R-CNN, namely, Faster R-CNN +
Figure 3: Qualitative GOD Results on MS-COCO 2017 dataset. (a) ground-truth annotations, (b) Faster R-CNN [58] predictions, (c) Faster R-CNN [58] + PrObeD predictions, (d) generated template, and (e) recovered template. We highlight the objects responsible for improvement in (c) as compared to (b). The yellow box represents better localization, the blue box represents false positives, and the red box represents missed predictions. PrObeD improves on all these errors made by (b).
Figure 4: Qualitative COD Results on CAMO, COD10K, and NC4K datasets from top to bottom, after applying PrObeD. (a) input images, (b) ground-truth camouflaged map, (c) DGNet [34] predictions, (d) DGNet [34] + PrObeD predictions, (e) generated PrObeD template, and (f) recovered PrObeD template. PrObeD template has the semantics of the camouflaged object, which aids DGNet in detection.
FPN and Sparse-RCNN. We observe an increase in the performance of both detectors. PrObeD also improves newer detectors like YOLOv5 and DeTR, although the gains are smaller compared to Faster R-CNN. We believe this happens because the newer detectors leave little room for improvement due to which PrObeD improves the performance slightly. We next compare PrObeD with a work that leverage segmentation map as a mask for object detection. We compare our performance with Mask R-CNN [30], which uses an image segmentation branch to help with object detection. Tab. 2 shows that the gains using Mask R-CNN are lower than using our proactive wrapper.
Qualitative Results. Fig. 3 shows qualitative results for the MS-COCO 2017 dataset. PrObeD clearly improves the performance of pretrained Faster R-CNN for three types of errors: Missed predictions, false negatives, and localization errors. PrObeD has a lower number of missed predictions, fewer false positives, and better bounding box localization. We also visualize the generated and recovered templates. We see that the template has object semantics of the input images. When the template is multiplied with the input image, it highlights the foreground objects, thereby making the task of object detector easier.
Error Analysis. We show the error analysis [6] for GOD section 4 of the supplementary. We observe that all GOD detectors make mistakes mainly due to five types of errors: classification, localization, duplicate detection, background detection, and missed detection. The main reason for the degraded performance is the errors in which the foreground-background boundary is missed. These errors include localization, background detection, and missed detection. Our proactive wrapper significantly corrects these errors, as the template has object semantics, which, when multiplied with the input image, highlights the foreground objects, consequently simplifying the task of object detection.
Table 4: Performance comparison with proactive works. MaLP [2] has a significantly deteriorated performance than PrObeD.
Method	CAMO	COD10K	NC4K
Em↑	Sm↑	wFβ↑	MAE↓	Em↑	Sm↑	wFβ↑	MAE↓	Em↑	Sm↑	wFβ↑	MAE↓
MaLP [2]	0.474	0.514	0.218	0.254	0.491	0.520	0.150	0.202	0.503	0.548	0.228	0.222
PrObeD	0.871	0.797	0.702	0.071	0.869	0.803	0.661	0.037	0.900	0.838	0.755	0.049
Table 5: Ablation studies of PrObeD using Faster R-CNN GOD on MS-COCO 2017 dataset. Removing the encoder/decoder network or adding the template results in degraded performance.
Changed	From→To	AP ↑	AP50 ↑	AP75 ↑	APS ↑	APM ↑	APL ↑
Template	Image Dependent→Fixed	17.6	37.9	15.1	1.3	15.4	39.5
Image Dependent→Universal	19.4	42.6	17.1	1.9	18.0	39.4
Decoder	Yes→No	25.2	46.1	26.2	5.3	26.6	24.1
Transformation	Multiply→Add	19.2	42.3	20.1	1.7	17.9	39.1
PrObeD	-	31.7	52.6	33.3	11.0	35.5	51.1
4.2 COD Results
Quantitative Results. Tab. 3 shows the result of applying PrObeD to DGNet [34] on three different datasets. PrObeD, when applied on top of DGNet, outperforms DGNet on all four metrics for all datasets. The biggest gain appears in COD10K and NC4K datasets. This is impressive as these datasets have more diverse testing images than CAMO. As NC4K is only a testing set, the higher performance of PrObeD demonstrates its superior generalizability as compared to DGNet [34]. This result agrees with the observation in [1, 2], where proactive-based approaches exhibit improved generalization on manipulation detection and localization tasks.
Qualitative Results. Fig. 4 visualizes the predicted camouflaged map for DGNet before and after applying PrObeD on testing samples of all three datasets. PrObeD improves the predicted camouflaged map, with less blurriness along the boundaries and better localization of the camouflaged object. As observed before for GOD, the generated and recovered template has the semantics of the camouflaged objects, which after multiplication intensifies the foreground object, resulting in better segmentation by DGNet.
Comparison with Proactive Works. The prior proactive works perform a different task of image manipulation detection and localization. Therefore, these works are not directly comparable to our proposed proactive wrapper, which performs a different task of object detection as described in Tab. 1. However, manipulation localization and COD both involve a prediction of a localization map, segmentation, and fakeness map, respectively. This inspires us to experiment with MaLP [2] for the task of COD. We train the localization module of MaLP supervised with the COD datasets. The results are shown in Tab. 4. We see that MaLP is not able to perform well for all three datasets. MaLP is designed for estimating universal templates rather than templates tailored to specific images. It shows the significance of image-specific templates in object detection. While MaLP's design with image-independent templates is effective for localizing image manipulation, applying it to object detection has a negative impact on performance.
**Framework Design.** PrObeD consists of blocks to improve the object detector. Tab. 5 ablates different versions of PrObeD to highlight the importance of each block in our design. PrObeD utilizes an encoder network  $\mathcal{E}$  to learn image-dependent templates aiding the detector. We remove the encoder  $\mathcal{E}$  from our network, replacing it with a fixed template. We observe that the performance deteriorates by a large margin. Next, we make this template learnable as proposed in PrObeD, but only a single template would be used for all the input images. This choice also results in worse performance, highlighting that image-dependent templates are necessary for object detection. Finally, we remove the decoder network  $\mathcal{D}$ , which is used to recover the template from the encrypted images. Although this results in a better performance than the pretrained Faster R-CNN, we observe a drop as compared to PrObeD. Therefore, as discussed in Sec. 3.3, the recovery of templates is indeed a necessary and beneficial step for boosting the performance of the proactive schemes.
Table 6: Ablation of training iterations on Faster R-CNN. YOLOv5, and DeTR for more iterations similar to after applying PrObeD. We also report the inference time for all the detectors before and after applying PrObeD. Training object detectors proactively with PrObeD results in more performance gain compared to training passively for more iterations. PrObeD adds an overhead cost on top of the inference cost of detectors.
Method	Iterations	AP ↑	\( AP_{50} \) ↑	\( AP_{75} \) ↑	\( AP_S \) ↑	\( AP_M \) ↑	\( AP_L \) ↑	Time (ms)
Faster R-CNN [58]	1×	19.3	42.5	16.9	1.8	17.9	39.3	161.1
Faster R-CNN [58]	2×	20.1	46.6	21.5	3.3	20.3	41.2
Faster R-CNN [58] + PrObeD	2×	31.7	52.6	33.3	11.0	35.5	51.1	175.3 (↑ 8.7%)
YOLOv5 [52]	1×	48.9	67.6	53.1	31.8	54.4	62.3	48.5
YOLOv5 [52]	2×	48.8	67.7	53.0	31.8	54.7	62.4
YOLOv5 [52] + PrObeD	2×	49.4	67.9	53.5	32.0	55.1	62.6	62.7 (↑ 29.1%)
DeTR [8]	1×	41.9	62.3	44.1	20.3	45.8	61.0	194.2
DeTR [8]	2×	41.9	62.4	44.0	20.1	45.9	61.1
DeTR [8] + PrObeD	2×	42.1	62.6	44.4	20.4	46.0	61.3	208.4 (↑ 7.2%)
Encryption Process. PrObeD includes an encryption process as described in Eq. (3), which involves multiplying the template with the input image. This process makes the template act as a mask, highlighting the foreground for better detection. However, prior proactive works [1, 2] consider adding templates to achieve better results. Thus, we ablate by changing the encryption process to template addition. Tab. 5 shows that template addition degrades performance by a significant margin w.r.t. our multiplication scheme. This shows that encryption is a key step in formulating proactive schemes, and the same encryption process may not work for all tasks.
More Training Time. We perform an ablation to show that the performance gain of the detector is due to our proactive wrapper instead of training for more iterations of the pretrained object detector. Results in Tab. 6 show that although more training iterations for the detector has a performance gain, it's not enough to get the significant margin in performance as achieved by PrObeD. This shows that extra training can help, but only up to a certain extent.
Inference Time. We evaluate the overhead computational cost after applying PrObeD on different object detectors are shown in Tab. 6, averaged across 1,000 images, on a NVIDIA V100 GPU. Our encoder network has 17 layers, which adds extra cost for inference. For detectors with bulky architectures like Faster R-CNN (ResNet101) and DeTR (transformer), the overhead computational cost is quite small,  $8.7\%$  and  $7.2\%$ , respectively. This additional cost is minor compared to the performance gain of detectors, especially Faster R-CNN. For a lighter detector like YOLOv5, our overhead computational cost increases to  $29.1\%$ . So, there is a trade-off of applying PrObeD to different detectors with varied architectures. PrObeD is more beneficial to bulky detectors like two-staged/transformer-based as compared to one-stage detectors.
We mathematically prove that the proactive method results in a better-converged model than the passive detector under assumptions and, consequently, a better 2D object detector. Based on this finding, we propose a proactive scheme wrapper, PrObeD, which enhances the performance of camouflaged and generic object detectors. The wrapper outputs an image-dependent template using an encoder network, which encrypts the input images. These encrypted images are then used to fine-tune the object detector. Extensive experiments on MS-COCO, CAMO, COD10K, and NC4K datasets show that PrObeD improves the overall object detection performance for both GOD and COD detectors.
Limitations. Our proposed scheme has the following limitations. First, PrObeD does not provide a significant gain for recent object detectors such as YOLO and DeTR. Second, the proactive wrapper should be thoroughly tested on other object detectors to show the generalizability of PrObeD. Finally, we only experiment with simple multiplication and addition as the encryption scheme. A more sophisticated encryption process might further improve the object detectors' performance. We leave these for our future avenues.