SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We consider two ways of generating query costs: (i)  $c(i,j) = 1$  for all  $i,j$ , where  $\mathcal{C}$  is just the number of queries, and (ii)  $c(i,j)$  is based on Manhattan distance between  $i$  and  $j$ . Most of the results we present in the main paper reflect the Manhattan distance based setting; we also report the results for uniform query costs in the Supplement. We use  $\lambda = 0.1$  in all the experiments. We present the details of policy architecture and hyper-parameter details for each different experimental settings in the Supplement.
Baselines We compare the proposed PSVAS and MPS-VAS policy learning framework to the following baselines.
- Random Search (RS), in which unexplored grid cells are selected uniformly at random.
- Conventional Active Search (AS) proposed by Jiang et. al. [9], using a low-dimensional feature representation for each image grid from the same feature extraction network as in our approach.
- Greedy Classification (GC), in which we train a classifier  $\psi_{GC}$  to determine whether a grid contains a target object. We then prioritize the search in grids with the highest probability of containing the target object until the search budget is depleted.
- Active Learning (AL), in which the first grid is selected randomly for querying, and the remaining grids are chosen using a state-of-the-art active learning approach by Yoo et al. [13] until the search budget is saturated.
- Greedy Selection (GS), proposed by Uzkent et al. [15], that trains a policy  $\phi_{GS}$  to assign a probability of zooming into each grid cell  $j$ . We use this policy to select grids greedily until the search budget  $\mathcal{C}$  is exhausted.
- End-to-end visual active search (E2EVAS), the state-of-the-art approach for VAS proposed by Sarkar et al. [6].
When dealing with a multi-query scenario, we compare the effectiveness of MPS-VAS-TOPK and MPS-VAS-MQ.
Datasets We evaluate the proposed approach using two datasets: xView [16] and DOTA [17]. xView is a satellite imagery dataset which consists of large satellite images representing 60 categories, with approximately 3000 pixels in each dimensions. We use  $67\%$  and  $33\%$  of the large satellite
Table 1: ANT comparisons when trained with small car as target on xView in single-query setting.
Test with Helicopter as Target	Test with SB as Target	Test with Building as Target
Method	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
RS	0.16	0.39	0.72	0.35	0.71	1.02	2.41	3.56	4.62
GC	0.29	0.55	0.93	0.52	0.95	1.21	3.94	5.14	6.61
GS [15]	0.41	0.68	1.08	0.61	1.03	1.26	4.51	5.80	6.82
AL [13]	0.27	0.54	0.92	0.52	0.93	1.18	3.92	5.12	6.60
AS [9]	0.25	0.46	0.83	0.51	0.95	1.20	3.79	5.01	6.34
E2EVAS [6]	0.53	0.83	1.25	0.67	1.10	1.30	5.85	9.26	11.96
OnlineTTA[6]	0.54	0.84	1.26	0.68	1.10	1.32	5.86	9.26	11.98
PSVAS	0.87	1.08	1.28	0.93	1.23	1.66	6.81	10.53	13.44
MPS-VAS	0.92	1.13	1.38	1.07	1.67	2.10	6.83	10.59	13.64
Test with CC as Target	Test with SC as Target	Test with Helipad as Target
Method	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
RS	0.57	1.18	1.72	1.80	3.40	5.11	0.19	0.42	0.72
GC	1.05	1.81	2.14	2.64	4.88	7.04	0.41	0.78	1.04
GS [15]	1.12	1.97	2.48	3.35	5.39	7.52	0.54	0.93	1.12
AL [13]	1.04	1.77	2.12	2.62	4.88	7.03	0.40	0.76	1.03
AS [9]	1.02	1.61	2.03	2.43	4.61	6.95	0.38	0.75	0.98
E2EVAS [6]	1.43	2.31	2.98	4.73	7.43	9.59	0.81	1.20	1.46
OnlineTTA[6]	1.43	2.33	2.99	4.75	7.44	9.59	0.83	1.21	1.46
PSVAS	1.62	2.49	3.14	5.51	8.33	10.52	0.91	1.22	1.47
MPS-VAS	1.74	2.64	3.47	5.55	8.40	10.69	0.96	1.30	1.63
images to train and test the policy network respectively. DOTA is also a satellite imagery dataset. We rescale the original  $3000 \times 3000px$  images to  $1200 \times 1200px$ .
4.2 Single Query Setting
Table 2: ANT comparisons when trained with large vehicle as target on DOTA in single-query setting.
Test with Ship as Target	Test with LV as Target	Test with Harbor as Target
Method	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
Random	1.08	2.11	3.06	1.48	2.96	3.91	1.41	2.72	3.90
GC	1.52	3.04	4.19	2.59	3.77	5.48	1.90	3.77	5.02
GS[15]	1.79	3.56	4.47	2.72	4.10	5.77	2.31	4.14	5.87
AL[13]	1.51	3.02	4.18	2.57	3.74	5.47	1.89	3.74	5.01
AS[9]	1.43	2.87	4.01	1.64	3.15	4.23	1.73	3.45	4.68
VAS[6]	2.45	4.37	5.87	5.33	8.47	10.51	3.12	5.04	6.82
OnlineTTA[6]	2.46	4.38	5.89	5.33	8.47	10.52	3.12	5.06	6.83
PSVAS	2.46	4.41	6.00	5.33	8.52	10.59	3.15	5.24	6.94
MPS-VAS	3.08	5.25	7.13	5.34	8.53	10.63	3.82	6.77	9.00
Test with Helicopter as Target	Test with Plane as Target	Test with Roundabout as Target
Method	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
Random	0.27	0.46	0.71	1.37	2.55	3.62	1.47	2.53	3.81
GC	0.38	0.59	0.91	1.96	3.14	4.02	1.62	2.86	4.23
GS[15]	0.42	0.66	1.01	2.26	3.56	4.71	1.91	3.24	4.68
AL[13]	0.37	0.58	0.89	1.94	3.14	3.99	1.61	2.82	4.21
AS[9]	0.34	0.53	0.82	1.89	3.06	3.92	1.55	2.71	4.06
E2EVAS[6]	0.47	0.72	1.05	3.07	4.87	6.34	3.05	4.94	6.34
OnlineTTA[6]	0.48	0.72	1.06	3.08	4.89	6.34	3.05	4.95	6.36
PSVAS	0.50	0.73	1.10	3.09	4.96	6.38	3.09	4.96	6.39
MPS-VAS	0.63	1.03	1.60	3.46	5.57	7.71	3.46	5.57	7.71
Figure 3: Query sequences, and corresponding heat maps (darker indicates higher probability), obtained using E2EVAS (top row), PSVAS (middle row), and MPS-VAS (bottom row).
4.3 Multi Query Setting
Next, we evaluate the proposed MPS-VAS-MQ approach on the xView and DOTA dataset in multi query setting. In Table 3, we present the results with varying search budget  $\mathcal{C} = \{25,50,75\}$  and the number of equal sized grid cells  $N = 49$ . We consider  $K = 3$  in all the experiments we perform in different target settings. We observe two general consistent trends across different target settings. First, the search performance of MPS-VAS in single query setting is always better than the performance of MPS-VAS-MQ in multi query settings, due to the fact that in single query setting the task-specific prediction module is updated  $K$  times more frequently than in the multi-query setting. Second, across various target settings, the performance improvement of MPS-VAS-MQ over MPS-VAS-TOPK ranges from  $0.08\%$  to  $3.5\%$ . In Table 4, we present similar results with the number of grid cells  $N = 64$  and train the policy with large vehicle as the target class. We report the results with different values of  $N$  in the Supplement. Here the improvement of MPS-VAS-MQ over MPS-VAS-TOPK is up to  $3.5\%$  across different target settings, suggesting that there is added value from learning to capture interdependence in greedy search decisions.
4.4 Effect of  $\lambda$  on Search Performance
We perform experiments with different choices of  $\lambda$  and found  $\lambda = 0.1$  to be the best choice across all different experimental setup. For comparison, here we report the results in the case when we train the policy with different values of  $\lambda$  using small car as a target class and test the policy with
Table 3: ANT comparisons when trained with small car as target on xView in multi-query setting.
Test with Helicopter as Target	Test with SB as Target	Test with Building as Target
Method	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
MPS-VAS-TOPK	0.55	0.75	1.12	0.97	1.38	1.93	6.68	10.35	13.30
MPS-VAS-MQ	0.63	0.83	1.17	0.97	1.40	1.97	6.82	10.48	13.31
Test with CC as Target	Test with SC as Target	Test with Helipad as Target
Method	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
MPS-VAS-TOPK	1.63	2.59	3.36	5.41	8.28	10.30	0.81	1.07	1.23
MPS-VAS-MQ	1.67	2.60	3.38	5.47	8.34	10.38	0.84	1.14	1.30
Table 4: ANT comparisons when trained with large vehicle as target on DOTA in multi-query setting.
Test with Ship as Target	Test with LV as Target	Test with Harbor as Target
Method	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
MPS-VAS-TOPK	3.03	5.14	6.83	5.32	8.46	10.56	3.69	6.49	8.71
MPS-VAS-MQ	3.05	5.18	6.88	5.33	8.50	10.61	3.72	6.59	8.75
Test with Helicopter as Target	Test with Plane as Target	Test with RB as Target
Method	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
MPS-VAS-TOPK	0.57	0.96	1.39	3.38	5.43	7.64	3.29	5.43	7.62
MPS-VAS-MQ	0.60	1.01	1.46	3.41	5.48	7.66	3.38	5.50	7.66
small car, building, and sail boat as target on xView. We evaluate the policy with varying search budgets  $\mathcal{C} \in \{25, 50, 75\}$  and the number of equal sized grid cells  $N = 49$ . In table 5, we provide the result for the PSVAS framework with small car as target. In table 6, we provide similar results for the MPS-VAS framework. Our empirical findings across all the experimental settings are quite consistent, and justify the choice of  $\lambda = 0.1$ .
Table 5: ANT comparisons when trained with small car as target for different values of  $\lambda$  using the PSVAS Framework.
Test with Small Car as Target	Test with Building as Target	Test with Sail Boat as Target
λ	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
0.001	4.96	7.75	9.74	6.08	9.64	12.35	0.74	1.12	1.43
0.01	5.02	7.87	9.96	6.37	9.95	12.77	0.88	1.19	1.54
0.1	5.51	8.33	10.52	6.81	10.53	13.44	0.93	1.23	1.66
1.0	5.10	7.98	10.04	6.39	10.16	12.81	0.89	1.20	1.59
Table 6: ANT comparisons when trained with small car as target for different values of  $\lambda$  using the MPS-VAS Framework.
Test with Small Car as Target	Test with Building as Target	Test with Sail Boat as Target
λ	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
0.001	4.99	7.82	9.90	6.15	9.74	12.44	0.83	1.22	1.53
0.01	5.06	7.93	10.03	6.41	10.09	12.89	0.98	1.46	1.87
0.1	5.55	8.40	10.69	6.83	10.59	13.64	1.07	1.67	2.10
1.0	5.12	8.01	10.12	6.46	10.21	12.96	1.01	1.52	1.90
4.5 Effectiveness of Task Specific Prediction Module on Search Performance
We analyse the importance of the task-specific prediction module in Sections D.1 and D.2 of Supplementary Material by freezing the prediction module parameters during inference time. Here, we additionally analyse the efficacy of the task-specific prediction module by setting  $\lambda = 0$  while training the policy. We call the resulting policy USVAS (Un-Supervised VAS). We observe a significant drop in performance across all settings, demonstrating the importance of the Supervised prediction module in order to learn an effective search policy. Specifically, in the following table 7, we present the results when the policy is trained with small car on xView as a target, while the performance of the policy is evaluated for the following target classes: Small Car (SC), Helicopter, SailBoat (SB), Construction Site (CS), Building, and Helipad. We evaluate the policy with varying search budgets  $\mathcal{C} \in \{25, 50, 75\}$  and the number of equal sized grid cells  $N = 49$ .
Table 7: ANT comparisons when trained with small car as target on xView.
Test with Small Car as Target	Test with Building as Target	Test with Sail Boat as Target
Method	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
USVAS	4.77	7.46	9.61	5.86	9.37	12.05	0.64	1.08	1.27
PSVAS	5.51	8.33	10.52	6.81	10.53	13.44	0.93	1.23	1.66
MPS-VAS	5.55	8.40	10.69	6.83	10.59	13.64	1.07	1.67	2.10
Test with Helicopter as Target	Test with CS as Target	Test with Helipad as Target
Method	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75	C = 25	C = 50	C = 75
USVAS	0.53	0.84	1.19	1.44	2.27	2.99	0.80	1.16	1.42
PSVAS	0.87	1.08	1.28	1.62	2.49	3.14	0.91	1.22	1.47
MPS-VAS	0.92	1.13	1.38	1.74	2.64	3.47	0.96	1.30	1.63
RL for Visual Navigation RL has found broad applicability in visual navigation tasks [18, 19, 20, 21]. While these tasks share some similarities at a high level, such as requiring a sequence of visual navigation steps based on a local view of the environment, they often do not involve search budget constraints and rely on a predetermined kinematic model of motion. In contrast, our approach involves observing the full environment, albeit potentially at a lower resolution, and sequentially determining which regions to query without being constrained to a particular kinematic model. This highlights the distinctive nature of our approach compared to traditional visual navigation tasks and demonstrates the potential value of active search strategies in addressing budget-constrained settings.
Active Search Active Search was first introduced by Garnett et al. [7] as a means to discover members of valuable and rare classes rather than solely focusing on learning an accurate model as in Active Learning [11]. Subsequently, Jiang et al. [9, 22] proposed efficient nonmyopic active search techniques and incorporated search cost into the problem. Sarkar et al. [6] demonstrate that prior active search techniques do not scale well in high-dimensional visual space, and instead propose a DRL based visual active search framework for geo-spatial broad area search. However, the efficacy of the proposed approach by Sarkar et al. [6] can be limited when the search task varies between training and testing, which is often the case in real-world applications. In this work, we propose a novel framework that enables efficient and adaptive search in any previously unseen search task.
Meta Learning The concept of meta-learning, has consistently attracted attention in the field of machine learning [23, 24, 25, 26]. Finn et al. [25] present Model Agnostic Meta-Learning, a technique that utilizes SGD updates to rapidly adapt to new tasks. This approach, based on gradient-based meta-learning, can be seen as learning an effective parameter initialization, enabling the network to achieve good performance with just a few gradient updates. Wortsman et al. [27] introduces a self-adaptive visual navigation approach, which has the ability to learn and adapt to novel environments without the need for explicit supervision. Our work is significantly different than all these prior works as we observe true label at each step during search and hence the main challenge is how to leverage the supervised information in order to learn an efficient adaptive search policy.
Foveated Processing of Large Images Several studies have investigated the utilization of low-resolution images for guiding the selection of high-resolution regions to process [28, 29, 30, 31, 32, 33, 34, 35], with some employing reinforcement learning techniques [36, 15] to improve this process. However, our work differs significantly, as we focus on selecting a sequence of regions to query, where each query provides the true label, instead of a higher resolution image region. These labels are crucial for guiding further search and serving as an ultimate objective. As such, our approach tackles a unique challenge that differs from existing methods that rely on low-resolution imagery.
We present a novel approach for visual active search in which we decompose the search policy into a prediction and search modules. This decomposition enables us to combine supervised and reinforcement learning for training, and make use of supervised learning even during execution. Moreover, we propose a novel meta-learning framework to jointly learn a policy and initialization parameters for the supervised prediction module. Our findings demonstrate the significance of the proposed frameworks for conducting efficient search, particularly in real-world situations where search tasks may differ significantly from those utilized during policy training. We hope our framework will find its applicability in many practical scenarios ranging from human trafficking to animal poaching.
This research was partially supported by the NSF (IIS-1905558, IIS-1903207, and IIS-2214141), ARO (W911NF-18-1-0208), Amazon, NVIDIA, and the Taylor Geospatial Institute managed by Saint Louis University.
[1] Elizabeth Bondi, Debadeepta Dey, Ashish Kapoor, Jim Piavis, Shital Shah, Fei Fang, Bistra Dilkina, Robert Hannaford, Arvind Iyer, Lucas Joppa, et al. Airsim-w: A simulation environment for wildlife conservation with uavs. In ACM SIGCAS Conference on Computing and Sustainable Societies, pages 1-12, 2018.
[2] Elizabeth Bondi, Fei Fang, Mark Hamilton, Debarun Kar, Donnabell Dmello, Jongmoo Choi, Robert Hannaford, Arvind Iyer, Lucas Joppa, Milind Tambe, et al. Spot poachers in action: Augmenting conservation drones with automatic detection in near real time. In AAAI Conference on Artificial Intelligence, 2018.
[3] Elizabeth Bondi, Raghav Jain, Palash Aggrawal, Saket Anand, Robert Hannaford, Ashish Kapoor, Jim Piavis, Shital Shah, Lucas Joppa, Bistra Dilkina, et al. Birdsai: A dataset for detection and tracking in aerial thermal infrared videos. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1747-1756, 2020.
[4] Fei Fang, Peter Stone, and Milind Tambe. When security games go green: Designing defender strategies to prevent poaching and illegal fishing. In International Joint Conference on Artificial Intelligence, 2015.
[5] Fei Fang, Thanh Hong Nguyen, Rob Pickles, Wai Y Lam, Gopalasamy R Clements, Bo An, Amandeep Singh, Milind Tambe, and Andrew Lemieux. Deploying paws: Field optimization of the protection assistant for wildlife security. In AAAI Conference on Artificial Intelligence, volume 16, pages 3966-3973, 2016.
[6] Anindya Sarkar, Michael Lanier, Scott Alfeld, Roman Garnett, Nathan Jacobs, and Yevgeniy Vorobeychik. A visual active search framework for geospatial exploration. arXiv preprint arXiv:2211.15788, 2022.
[7] Roman Garnett, Yamuna Krishnamurthy, Xuehan Xiong, Jeff Schneider, and Richard Mann. Bayesian optimal active search and surveying. arXiv preprint arXiv:1206.6406, 2012.
[8] Roman Garnett, Thomas Gartner, Martin Vogt, and Jürgen Bajorath. Introducing the 'active search' method for iterative virtual screening. Journal of Computer-Aided Molecular Design, 29(4):305-314, 2015.
[9] Shali Jiang, Gustavo Malkomes, Geoff Converse, Alyssa Shofner, Benjamin Moseley, and Roman Garnett. Efficient nonmyopic active search. In International Conference on Machine Learning, pages 1714-1723. PMLR, 2017.
[10] Shali Jiang, Roman Garnett, and Benjamin Moseley. Cost effective active search. Advances in Neural Information Processing Systems, 32, 2019.
[11] Burr Settles. Active learning literature survey. 2009.
[12] Burr Settles. Active learning. Synthesis lectures on artificial intelligence and machine learning, 6(1):1-114, 2012.
[13] Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 93-102, 2019.
[14] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[31] Athanasios Papadopoulos, Pawel Korus, and Nasir Memon. Hard-attention for scalable image classification. In Advances in Neural Information Processing Systems, volume 34, pages 14694–14707, 2021.
[32] Chittesh Thavamani, Mengtian Li, Nicolas Cebron, and Deva Ramanan. Fovea: Foveated image magnification for autonomous navigation. In IEEE/CVF International Conference on Computer Vision, pages 15539-15548, 2021.
[33] Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12309-12318, 2022.
[34] Chenlin Meng, Enci Liu, Willie Neiswanger, Jiaming Song, Marshall Burke, David Lobell, and Stefano Ermon. Is-count: Large-scale object counting from satellite images with covariate-based importance sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12034-12042, 2022.
[35] Chenhongyi Yang, Zehao Huang, and Naiyan Wang. Querydet: Cascaded sparse query for accelerating high-resolution small object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13668-13677, June 2022.
[36] Burak Uzkent, Christopher Yeh, and Stefano Ermon. Efficient object detection in large images using deep reinforcement learning. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
A Partially Supervised Reinforcement Learning Framework for Visual Active Search: Supplementary Material
A Policy Network Architecture and Hyperparameter Details
Recall that the policy network  $\pi$  is composed of two parts: (1) a task specific prediction module, and (2) a task-agnostic search module. The task specific prediction module consists of an encoder  $e(x;\eta)$  that maps the aerial image  $x$  to a low-dimensional latent feature representation  $z$ , and a grid prediction network  $p(z,o;\kappa)$  that predicts the probabilities of grids containing a target by leveraging the latent semantic feature  $z$  and the outcomes of previous search queries  $o$ . Note that the task specific prediction module is represented as  $f(x,o,\theta) = p(z = e(x;\eta),o;\kappa)$ , where  $\theta = (\eta ,\kappa)$ . Following [6], we use frozen ResNet-34, pre-trained on ImageNet, followed by a learnable  $1\times 1$  convolution layer with a ReLU activation as a feature extraction component of the task specific prediction module that we refer as encoder  $e(.)$ . We then combine the latent semantic feature  $z$  with the previous query information  $o$ . We apply the tiling operation in order to convert  $o$  into a representation with the same dimensions as the extracted features  $z$ , enabling us to effectively apply channel-wise concatenation of latent image feature and auxiliary state feature while preserving the grid specific spatial and query related information. This combined representation is then fed to a grid prediction network comprises of a  $1\times 1$  convolution layer, flattening, and a MLP block consists of 2 fully connected layers with ReLU activations. Note that the output of grid prediction network is of dimension  $N$ . We finally apply sigmoid activation to each output neuron to convert them into a probability value representing the probability of the grids containing target. The proposed policy architecture is depicted in figure 2 of the main paper.
We re-shape the output of task specific prediction module by converting it back from 1D to 2D of shape  $(m\times n) = N$  before feeding it to the task agnostic search module  $g(\cdot)$  that takes the following three inputs: (1) the reshaped 2D output of the task specific prediction module, which is the probabilities of grids containing target; (2) the remaining search budget  $B$ , which is a scalar but we apply tiling to the scalar budget  $B$  to transform it to match the size of the reshaped 2D output of the task specific prediction module; (3) we also apply the tiling operation to  $o$  in a way that allows us to concatenate the features  $(z,o,B)$  along the channels dimension to finally obtain the combined representation that serves as a input to task agnostic search module. The task agnostic search module is composed of a flattening, a MLP block consists of 2 fully connected layer with ReLU activations, and a final softmax layer to convert the output to a probability distribution that guides us in selecting the grid to query next.
In Table 8, we detail the architecture of task specific prediction module  $(f)$  of PSVAS policy network. In Table 9, we detail the architecture of task agnostic search module  $(g)$  of PSVAS policy network. Note that, the task specific prediction module and task agnostic search module remains unchanged in MPS-VAS framework.
Table 8: Task Specific Prediction Module Architecture with number of grid cell  $N = \left( {m \times  n}\right)$
Layers	Configuration	o/p Feature Map size
Input	RGB Image	3 × 3500 × 3500
Encoder	ResNet-34	512 × 14 × 14
Conv1	Channel:N; kernel size:1 × 1	N × 14 × 14
2D MaxPool	Pooling size:2 × 2	N × 7 × 7
Tile1	Grid State (o)	N × 7 × 7
Channelwise Concat	Conv1,Tile1	(2N) × 7 × 7
Conv2	Channel:3; kernel size: 1 × 1	3 × 7 × 7
Flattened	Conv2	147
FC1+ReLU	(147– > 2N)	2N
FC2+Sigmoid	(2N– > N)	N
Table 9: Task Agnostic Search Module Architecture with number of grid cell  $N = \left( {m \times  n}\right)$
Layers	Configuration	o/p Feature Map size
Input 1	2D Reshape of Task Specific Prediction Module Output	1 × m × n
Input 2: Tile2	Grid State (o)	1 × m × n
Input 3: Tile3	Query Budget Left (B)	1 × m × n
Input: Channelwise Concat	Input 1, Input 2, Input 3	(3) × m × n
Flattened	Input: Channelwise Concat	K = (3) × m × n
FC1+ReLU	(K- > 2N)	2N
FC2+Softmax	(2N- > N)	N