Title: A Partially-Supervised Reinforcement Learning Framework for Visual Active Search
Venue: NeurIPS 2023
Accepted: yes
Keywords: Visual Active Search

Reinforcement Learning
Abstract: Visual active search (VAS) has been proposed as a  modeling framework in which visual cues are used to guide exploration, with the goal of identifying regions of interest in a large geospatial area. Its potential applications include identifying hot spots of rare wildlife poaching activity, search-and-rescue scenarios, identifying illegal trafficking of weapons, drugs, or people, and many others. State of the art approaches to VAS include applications of deep reinforcement learning (DRL), which yield end-to-end search policies, and traditional active search, which combines predictions with custom algorithmic approaches. While the DRL framework has been shown to greatly outperform traditional active search in such domains, its end-to-end nature does not make full use of supervised information attained either during training, or during actual search, a significant limitation if search tasks differ significantly from those in the training distribution. We propose an approach that combines the strength of both DRL and conventional active search approaches by decomposing the search policy into a prediction module, which produces a geospatial distribution of regions of interest based on task embedding and search history, and a search module, which takes the predictions and search history as input and outputs the search distribution. In addition, we develop a novel meta-learning approach for jointly learning the resulting combined policy that can make effective use of supervised information obtained both at training and decision time. Our extensive experiments demonstrate that the proposed representation and meta-learning frameworks significantly outperform state of the art in visual active search on several problem domains.

A Partially Supervised Reinforcement Learning Framework for Visual Active Search

Abstract

Visual active search (V AS) has been proposed as a modeling framework in which visual cues are used to guide exploration, with the goal of identifying regions of interest in a large geospatial area. Its potential applications include identifying hot spots of rare wildlife poaching activity, search-and-rescue scenarios, identifying illegal trafficking of weapons, drugs, or people, and many others. State of the art approaches to VAS include applications of deep reinforcement learning (DRL), which yield end-to-end search policies, and traditional active search, which combines predictions with custom algorithmic approaches. While the DRL framework has been shown to greatly outperform traditional active search in such domains, its end-to-end nature does not make full use of supervised information attained either during training, or during actual search, a significant limitation if search tasks differ significantly from those in the training distribution. We propose an approach that combines the strength of both DRL and conventional active search by decomposing the search policy into a prediction module, which produces a geospatial distribution of regions of interest based on task embedding and search history, and a search module, which takes the predictions and search history as input and outputs the search distribution. We develop a novel meta-learning approach for jointly learning the resulting combined policy that can make effective use of supervised information obtained both at training and decision time. Our extensive experiments demonstrate that the proposed representation and meta-learning frameworks significantly outperform state of the art in visual active search on several problem domains.

1 Introduction

Consider a scenario where a child is abducted and law enforcement needs to scan across hundreds of potential regions from a helicopter for a particular vehicle. An important strategy in such a search and rescue portfolio is to obtain aerial imagery using drones that helps detect a target object of interest (e.g., the abductor's car) [1, 2, 3, 4, 5]. The quality of the resulting photographs, however, is generally somewhat poor, making the detection problem extremely difficult. Moreover, security officers can only inspect relatively few small regions to confirm search and rescue activity, doing so sequentially.

We can distill some key generalizable structure from this scenario: given a broad area image (often with a relatively low resolution), sequentially query small areas within it (e.g., by sending security officers to the associated regions, on the ground) to identify as many target objects as possible. The number of queries we can make is typically limited, for example, by budget or resource constraints. Moreover, query results (e.g., detected search and rescue activity in a particular region) are highly informative about the locations of target objects in other regions , for example, due to spatial correlation. We refer to this general modeling framework as visual active search (VAS) . Numerous other scenarios share this broad structure, such as identification of drug or human trafficking sites, anti-poaching enforcement activities, identifying landmarks, and many others. Sarkar et al. [6]

recently proposed a visual active search (VAS) framework for geospatial exploration, as well as a deep reinforcement learning (DRL) approach for learning a search policy. However, the efficacy of this DRL approach remains limited by its inability to adapt to search tasks that differ significantly from those in the training data, with the DRL approach struggling in such settings even when combined with test-time adaptation methods.

To gain intuition about the importance of domain adaptivity in VAS, consider Figure 1. Suppose, we pre-train a policy by leveraging a fully annotated search tasks with large vehicle as a target class to learn a search policy. Now the challenge in visual active search problem is how to utilize such a policy for our current task, that is to search for small car given a broad aerial image. As depicted in Figure 1, an adaptive policy (right) initially makes a mistake but then quickly adapts to the current task by efficiently leveraging information obtained in response to queries to learning from its mistakes. In contrast, a non-adaptive policy (left) keeps repeating its

mistakes and ultimately fails to find the region containing the target object.

Indeed, traditional active search approaches have been designed precisely with such adaptivity in mind [7, 8, 9, 10] by combining an explicit machine learning model that predicts labels over inputs with custom algorithmic approaches aiming to balance exploration (which improves prediction efficacy) and exploitation (to identify as many actual objects of interest as possible within a limited budget). However, as Sarkar et al. [6] have shown, such approaches perform poorly in VAS settings compared to DRL, despite the lack of effective domain adaptivity of the latter.

We combine the best of traditional active search and the state-of-the-art DRL-based V AS approach by developing a partially-supervised reinforcement learning framework for VAS (PSVAS). A PSVAS policy architecture is comprised of two module: (i) A task-specific prediction module that learns to predict the locations of the target object based on the task aerial image and labels resulting from queries during the search process, and (ii) a task-agnostic search module that learns a search policy given the predictions provided by the prediction module and prior search results. The key advantage of this decomposition is that it enables us to use supervised information observed at decision time to update the parameters of the prediction module, without changing the search module. Furthermore, to learn search policies that are effective in the context of evolving predictions during the search, we propose a novel meta-learning approach to jointly learning the search module with the initialization parameters of the prediction module (used at the beginning of each task). Finally, we generalize the original VAS framework to allow for multiple simultaneous queries (a common setting in practice), and develop the PSVAS framework and a meta-learning approach for such settings.

In summary, we make the following contributions:

A novel partially supervised reinforcement learning (PSVAS) framework that enables effective adaptation of VAS search policies to out-of-distribution search tasks.

A novel meta-learning approach (MPS-VAS) to learn initialization parameters of the prediction module jointly with a search policy that is robust to the evolving predictions during a search task.

A generalization of the VAS problem to domains in which we can make multiple simultaneous queries, and a variant of MPS-VAS that learns how to choose a subset of queries to make in each search iteration.

An extensive experimental evaluation on two publicly available satellite imagery datasets, xView and DOTA, in a variety of unknown target settings, demonstrating that the proposed approaches significantly outperform all baselines. Our code is publicly available at this link.

2 Preliminaries

We consider a generalization of the visual active search (VAS) problem proposed by Sarkar et al. [6]. The basic building block of VAS is a task , which centers around an aerial image x divided into

N grid cells, so that x = ( x ( 1 ) , x ( 2 ) , ..., x ( N ) ) , with each grid cell a subimage. Broadly, the goal is to identify as many target objects of interest through iterative exploration of these grid cells as possible, subject to a budget constraint C . To this end, we represent the subset of grids containing the target object by associating each grid cell j with a binary label y ( j ) ∈ { 0 , 1 } , where y ( j ) = 1 if the grid cell j contains the target object, and 0 otherwise. The complete label vector associated with the task is y = ( y ( 1 ) , y ( 2 ) , ..., y ( N ) ) . When we are faced with the task at decision time, we have no direct information about y , but when we query a grid cell j , we obtain the ground truth label y ( j ) for this cell. Moreover, if y ( j ) = 1 , we also accrue utility from exploring j . In the original variant of VAS, we can make a single query j in each time step. Here, we consider a natural generalization where we have R query resources (for example, R < N patrol units identifying traps in a wildlife conservation setting), so that we can make R queries in each time step. We assume that R is constant for convenience; it is straightforward to generalize our approach below if R is time-varying.

Let c ( j, k ) be the cost of querying grid cell k if we start in grid cell j . For the very first query, we can define a dummy initial grid cell d , so that cost function c ( d, k ) captures the initial query cost. Let q r t denote the set of queries performed in step t by a query resource r . Our ultimate goal is to solve the following optimization problem:

Finally, we assume that we possess a collection of tasks (in this case, aerial images) for which we have annotated whether each grid cell contains the target object or not. This collection, which we will refer to as D = {( x i , y i )} , is comprised of images x i with corresponding grid cell labels y i , where each x i is composed of N elements ( x ( 1 ) i , x ( 2 ) i , . . . , x ( N ) i ) representing the cells in the image, and each y i contains N corresponding labels ( y ( 1 ) i , y ( 2 ) i , . . . , y ( N ) i ) .

The central technical goal in VAS is to learn an effective search policy that maximizes the total number of targets discovered, on average, for a sequence of tasks x on which we have no prior direct experience, given a set of resources R , exploration cost function c ( j, k ) , and total exploration budget C . Sarkar et al. [6] proposed a deep reinforcement learning (DRL) approach in which they learned a search policy π ( x, o, B ) that outputs at each time step t the grid we should explore at the next step t + 1 , given the task x , remaining budget B , and information produced from the sequence of previous queries encoded into an observation vector o with o ( j ) = 2 y ( j ) -1 if j has been explored, and o ( j ) = 0 otherwise. The reward function for this DRL approach is naturally captured by R ( x, o, j ) = y ( j ) .

Note that active search (including VAS) is qualitatively distinct from active learning [11, 12, 13]: in the latter, the goal is solely to learn to predict well, so that the entire query process serves the goal of improving predictions. In active search, in contrast, we aim to learn a search policy that balances exploration (improving our ability to predict where target objects are) and exploitation (actually finding such objects) within a limited budget. Indeed, it has been shown that active learning approaches are not competitive in the V AS context [6].

3 Partially-Supervised Reinforcement Learning Approach for VAS

The DRL approach for VAS proposed by Sarkar et al. [6] is end-to-end, producing a policy that is only partly adaptive to observations o made during exploration for each task. In particular, this end-to-end policy aims to capture the tension between exploration and exploitation fundamental in active search [7, 9], without explicitly representing the central aim of exploration, which is to improve our ability to predict which grid cells in fact contain the target object. In conventional active search, in contrast, this is directly represented by learning a prediction function f ( x ( j ) ) that is updated each time a grid j is explored during the search process, so that exploration directly impacts our ability to predict target locations, and thereby make better query choices in the future. While the end-to-end approach implicitly learns this, it would only do so effectively so long as tasks x we face at prediction time are closely related to those used in training. However, it does not take advantage of the supervised information we obtain about which grid cells actually contain the target object, either at training or

Task Specific Prediction Module

Input with

N Grids

Past Search

Outcomes

[o (1)

, o

(2)

,…, o

(N)

]

ResNet 34

(frozen)

Dx14x14

1x1

Conv

+

MLP

2D

Pool

Dx7x7

Nx7x7

Tile

1x1

Conv

Dx7x7

Kx7x7

1xMxN

1xMxN

Task Agnostic Search Module

+

MLP

Remaining Query Budget

Update

Tile

Tile

Reshape

1xMxN

test time, potentially reducing the efficacy of learning as well as ability to adapt when task distribution changes.

To address this issue, we propose a novel partially-supervised reinforcement learning approach for visual active search, which we refer to as PSVAS. In PSVAS, the search policy we obtain is a composition of two modules: 1) the task-specific prediction module f θ ( x, o ) and 2) the task-agnostic search module g ϕ ( p, o, B ) , where θ and ϕ are trainable parameters, where p = f θ ( x, o ) is the vector of predicted probabilities with p ( j ) the predicted probability of a target in grid cell j (see Figure 2, and Supplement Section A for further details about the policy network architecture). Conceptually, f θ makes predictions based solely on the task x and the prediction-relevant information gathered during the search o , while g relies solely on the information relevant to the search itself: predicted locations of objects p , observations acquired during the search o , and remaining budget B . The search policy is then the composition of these modules, π ( x, o, B ) = g ϕ ( f θ ( x, o ) , o, B ) .

Although in principle we can still train the policy π above end-to-end (effectively collapsing the composition), a key advantage of PSVAS is that it enables us to directly make use of supervised information about true labels y as they are observed either at training or decision time, using these to update f θ in both cases. In prior work that relies solely on end-to-end policy representation, there was no straightforward way to take advantage of this information. Specifically, we train the policy using the following objective function:

where λ is a hyperparameter. We represent L RL loss as follows:

Where M is the number of example search task seen during training and R t i is the discounted cumulative reward defined as R t i = ∑ T k = t γ k -t R k i with a discount factor γ ∈ [ 0 , 1 ] . Note that the RL loss in this approach also updates the parameters of the prediction module , θ , in an end-to-end fashion. We also represents L BCE as follows: L BCE = ∑ i M = 1 -( y i log ( p i ) + ( 1 -y i ) log ( 1 -p i )) .

The PSVAS algorithm is more formally presented in Algorithm 1. The combined RL and supervised loss yields a balance between using supervised information to improve the quality of initial predictions at the beginning of the episode and ensuring that these serve the goal of producing the best episodelevel policies. However, it is still crucial to note that f θ serves solely an instrumental role in the process, with learning a search policy that effectively adapts to each task the primary goal. Consequently, we ultimately wish to jointly learn g ϕ and f θ in such a way that f θ facilitates adaptive search as it is updated during a search task at decision time. To address this, we propose a metalearning approach, MPS-VAS, that learns g ϕ along with an initial parametrization θ 0 of f θ for each task. At the beginning of each task, then, θ is initialized to θ 0 , and then updated as labels are observed after each query.

At the high level, MPS-VAS trains over a series of episodes , where each episode corresponds to a fully labeled training task ( x i , y i ) and budget constraint C (we vary the budget constraints during training). We begin an episode i with the current prediction function parameters θ i = θ and search policy parameters ϕ . We fix g ϕ over the length of the episode, and use it to generate a sequence of queries (since the policy is stochastic, it naturally induces exploration). After observing the label y ( j )

Algorithm 1 The PSVAS algorithm.

Require: A search task instance ( x i , y i ) ; budget constraint C ; Prediction function ( f ) with current parameters θ i , i.e., f θ ; Search policy ( g ) with current parameters ϕ i , i.e., g ϕ ;

i i 1: Initialize o 0 = [ 0 ... 0 ] ; B 0 = C ; step t = 0 2: while B t > 0 do 3: ˜ y = f θ i ( x i , o t ) 4: j ← /leftrightline Sample j ∈{ Unexplored Grids } [ g ϕ i ( ˜ y, o t , B t )] 5: Query grid cell with index j and observe true label y ( j ) . 6: Obtain reward R t = y ( j ) , Update o t to o t + 1 with o ( j ) = 2 y ( j ) -1 , and update B t to B t + 1 with B t + 1 = B t -c ( k, j ) (assuming we query k 'th grid at ( t -1 ) ). 7: Collect transition tuple ( τ ) at step t, i.e., τ t = ( state = ( x i , o t , B t ) , action = j , reward = R t , next state = ( x i , o t + 1 , B t + 1 ) ) . 8: t ← /leftrightline t + 1 9: end while

10: Update the prediction and search policy parameters, i.e., θ i and ϕ i using ( L PSVAS ) based on the collected transition tuples ( τ t ) and the collected labels ( y ( j ) ) throughout the episode.

11: Return updated prediction and search policy parameters, i.e., θ i + 1 and ϕ i + 1 respectively.

for each queried grid cell j during the episode, we update f θ i using a standard supervised (binary cross-entropy) loss. At the completion of the episode (once we have exhausted the search budget C ), we update the policy parameters, as well as the initialization prediction function parameters θ by combining RL and supervised loss. For the search module, we use the accumulated sum of rewards R i = ∑ j y ( j ) over grids j explored during the episode, with the RL loss L RL based on the REINFORCE algorithm [14]. For the prediction module, we use the collected labels y ( j ) during the episode and the standard binary cross-entropy loss. Finally, the MPS-VAS loss also explicitly trades off the RL and supervised loss: L MLPARVS = (L RL + λ L BCE ) . The proposed meta-learning approach thus explicitly trains the policy to account for the evolution of the prediction during the episode . The full MPS-VAS is presented more formally in Algorithm 2.

Algorithm 2 The MPS-VAS meta-learning algorithm.

Require: A search task instance ( x i , y i ) ; budget constraint C ; Prediction function ( f ) with current parameters θ 0 i = θ i , i.e., f θ 0 i ; Search policy ( g ) with current parameters ϕ i , i.e., g ϕ i ;

1: Initialize o 0 = [ 0 ... 0 ] ; B 0 = C ; step t = 0

2: while B t > 0 do

3: ˜ y = f θ t i ( x i , o t )

4: j ← /leftrightline Sample j ∈{ Unexplored Grids } [ g ϕ i ( ˜ y, o t , B t )]

5: Query grid cell with index j and observe true label y ( j ) .

6: Update θ t i to θ t + 1 i using L BCE loss between ˜ y and pseudo label ˆ y , defined as

˜ y ( ) if y ( ) is Unobserved.

7: Obtain reward R t = y ( j ) , Update o t to o t + 1 with o ( j ) = 2 y ( j ) -1 , and update B t to B t + 1 with B t + 1 = B t -c ( k, j ) (assuming we query k 'th grid at ( t -1 ) ).

8: Collect transition tuple τ at step t, i.e., τ t = ( state = ( x i , o t , B t ) , action = j , reward = R t , next state = ( x i , o t + 1 , B t + 1 ) ) .

9: t ← /leftrightline t + 1

10: end while

11: Update search policy parameters ϕ i using ( L RL ) based on the collected transition tuples ( τ t ) throughout the episode and update initial prediction function parameters θ i using ( L BCE ) based on the collected labels ( y ( j ) ) throughout the episode.

12: Return updated prediction and search policy parameters, i.e., θ i + 1 and ϕ i + 1 respectively.

The search policy π produces a probability distribution over grid cells. However, since in our setting no advantage can be gained by querying previously queried grid cells, we simply renormalize the probability distribution induced by π over the remaining grid cells, both during training and at decision time. Note that during inference, in both PSV AS and MPS-VAS framework, we freeze the parameters of the search module ( ϕ ) and update the parameters of the prediction module ( θ ) after

observing query outcomes at each step using L BCE loss between predicted label and pseudo label as shown in step ( 6 ) of Algorithm 2.

The discussion thus far effectively assumed that R = 1 , that is, we make only a single query in each search time step. Next, we describe the generalization of our approach when we have multiple query resources R > 1 . First, note that the prediction module f θ is unaffected, since the number of query resources only pertain to the actual search strategy g ϕ . One way to handle R queries is to simply sample the search module (which is stochastic) iteratively R times without replacement during training, and to greedily choose the most probable R grid cells to query at decision time. We refer to this as MPS-VAS-TOPK. However, since the underlying problem is now combinatorial (the choice of R queries out of N grid cells), such a greedy policy may fail to capture important interdependencies among such search decisions.

To address this, we propose a novel policy architecture which is designed to learn how to optimize such a heuristic greedy approach for combinatorial grid cell selection in a way that is non-myopic and accounts for the interdependent effects of sequential greedy choices. Specifically, let ψ be a vector corresponding to the grid cells, with ψ j = 1 if grid cell j has either been queried in the past (during previous search steps), or has been already chosen to be queried in the current search step, and ψ j = 0 otherwise. Thus, ψ encodes the choices that have already been made, and enables the policy to learn the best next sequential choice to make using a greedy procedure through the same RL framework that we described above. We refer to this approach as MPS-VAS-MQ (in reference to multiple queries).

4 Experiments

4.1 Experiment Setup

Since the goal of active search is to maximize the number of target objects identified, we use average number of targets identified through exploration ( ANT ) as our evaluation metric.

We consider two ways of generating query costs: (i) c ( i, j ) = 1 for all i, j , where C is just the number of queries, and (ii) c ( i, j ) is based on Manhattan distance between i and j . Most of the results we present in the main paper reflect the Manhattan distance based setting; we also report the results for uniform query costs in the Supplement. We use λ = 0 . 1 in all the experiments. We present the details of policy architecture and hyper-parameter details for each different experimental settings in the Supplement.

Baselines We compare the proposed PSVAS and MPS-VAS policy learning framework to the following baselines.

Random Search (RS) , in which unexplored grid cells are selected uniformly at random.

Conventional Active Search (AS) proposed by Jiang et. al. [9], using a low-dimensional feature representation for each image grid from the same feature extraction network as in our approach.

Greedy Classification (GC) , in which we train a classifier ψ GC to determine whether a grid contains a target object. We then prioritize the search in grids with the highest probability of containing the target object until the search budget is depleted.

Active Learning (AL) , in which the first grid is selected randomly for querying, and the remaining grids are chosen using a state-of-the-art active learning approach by Yoo et al. [13] until the search budget is saturated.

Greedy Selection (GS) , proposed by Uzkent et al. [15], that trains a policy ϕ GS to assign a probability of zooming into each grid cell j . We use this policy to select grids greedily until the search budget C is exhausted.

End-to-end visual active search (E2EVAS) , the state-of-the-art approach for V AS proposed by Sarkar et al. [6].

When dealing with a multi-query scenario, we compare the effectiveness of MPS-VAS-TOPK and MPS-VAS-MQ.

Datasets We evaluate the proposed approach using two datasets: xView [16] and DOTA [17]. xView is a satellite imagery dataset which consists of large satellite images representing 60 categories, with approximately 3000 pixels in each dimensions. We use 67% and 33% of the large satellite

images to train and test the policy network respectively. DOTA is also a satellite imagery dataset. We rescale the original 3000 × 3000 px images to 1200 × 1200 px .

4.2 Single Query Setting

We begin by considering a setting with a single query resource, as in most prior work. We first evaluate the proposed method on the xView dataset with varying search budget C ∈ { 25 , 50 , 75 } and the number of equal sized grid cells N = 49 . We train the policy with small car as target and evaluate the performance of the policy with the following target classes : Small Car (SC), Helicopter , Sail Boat (SB), Construction Cite (CC), Building , and Helipad . As the dataset contains variable size images, we take random crops of 3500 × 3500 for N = 49 , ensuring equal grid cell sizes. We present the results with different values of N in the Supplement. The results with N = 49 are presented in Table 1. We observe significant improvements in performance of the proposed PSVAS approach compared to all baselines in each different target setting, ranging from 3 to 25% improvement relative to the most competitive E2EVAS method. The significance of utilizing supervised information of true labels y , which are observed after each query at inference time, is supported by the obtained results. This highlights the effectiveness of the PSV AS framework, which enables us to update task-specific prediction module f by leveraging such crucial information in an efficient manner. The experimental outcomes also indicates two consistent trends. In each target setting, the overall search performance improves as C increases, and the relative advantage of MPS-V AS over PSV AS increases, as it is better able to exploit the observed outcomes and in turn improve the policy further for the larger search budget C . We also observe that the extent of improvement in performance is greater when there is a greater difference between the target class used in training and the one used during inference. For example, when the target class is a sail boat , the improvement in performance of MPS-VAS in comparison to PSVAS ranges between 15% to 35% . However, if the target class is a small car , the improvement in performance of MPS-VAS is only between 1% to 2% . Considering all different target settings, performance improvement of MPS-VAS in comparison to PSVAS, ranges from 1% to 35% . The results demonstrate the efficacy of the MPS-VAS framework in learning a search policy that enables adaptive search. Figure 3 demonstrates the exploration strategies of different policies that are trained on small car as the target class and test on sail boat as the target. The figure showcases the different exploration behaviors exhibited by each policy in response to the target class, highlighting the impact of the proposed adaptive search framework on the resulting exploration strategies. In Table 2, we present similar results on the DOTA dataset with N as 64. Here, we train the policy with Large Vehicle as target and evaluate the policy with the following target classes: Ship , Large Vehicle (LV), Harbor , Helicopter , Plane , and Roundabout (RB). We observe that MPS-VAS approach significantly outperforms all other baseline methods across different target scenarios. This shows the importance of MPS-VAS framework for deploying visual active search when search tasks differ significantly from those that are used for training.

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

4.3 Multi Query Setting

Next, we evaluate the proposed MPS-VAS-MQ approach on the xView and DOTA dataset in multi query setting. In Table 3, we present the results with varying search budget C = { 25 , 50 , 75 } and the number of equal sized grid cells N = 49 . We consider K = 3 in all the experiments we perform in different target settings. We observe two general consistent trends across different target settings. First, the search performance of MPS-VAS in single query setting is always better than the performance of MPS-VAS-MQ in multi query settings, due to the fact that in single query setting the task-specific prediction module is updated K times more frequently than in the multi-query setting. Second, across various target settings, the performance improvement of MPS-VAS-MQ over MPS-VAS-TOPK ranges from 0 . 08% to 3 . 5% . In Table 4, we present similar result with the number of grid cells N = 64 and train the policy with large vehicle as the target class. We report the results with different values of N in the Supplement. Here the improvement of MPS-VAS-MQ over MPS-VAS-TOPK is up to 3 . 5% across different target settings, suggesting that there is added value from learning to capture interdependence in greedy search decisions.

4.4 Effect of λ on Search Performance

We perform experiments with different choices of λ and found λ = 0.1 to be the best choice across all different experimental setup. For comparison, here we report the results in the case when we train the policy with different values of λ using small car as a target class and test the policy with

small car , building , and sail boat as target on xView. We evaluate the policy with varying search budgets C ∈ { 25 , 50 , 75 } and the number of equal sized grid cells N = 49 . In table 5, we provide the result for the PSV AS framework with small car as target. In table 6, we provide similar result for the MPS-VAS framework. Our empirical findings across all the experimental settings are quite consistent, and justify the choice of λ = 0 . 1 .

4.5 Effectiveness of Task Specific Prediction Module on Search Performance

We analyse the importance of the task-specific prediction module in Sections D.1 and D.2 of Supplementary Material by freezing the prediction module parameters during inference time. Here, we additionally analyse the efficacy of the task-specific prediction module by setting λ = 0 while training the policy. We call the resulting policy USVAS (Un-Supervised VAS). We observe a significant drop in performance across all settings, demonstrating the importance of the Supervised prediction module in order to learn an effective search policy. Specifically, in the following table 7, we present the results when the policy is trained with small car on xView as a target, while the performance of the policy is evaluated for the following target classes: Small Car (SC), Helicopter, SailBoat (SB), Construction Site (CS), Building, and Helipad. We evaluate the policy with varying search budgets C ∈ { 25 , 50 , 75 } and the number of equal sized grid cells N = 49 .

5 Related Work

RLfor Visual Navigation RLhas found broad applicability in visual navigation tasks [18, 19, 20, 21]. While these tasks share some similarities at a high level, such as requiring a sequence of visual navigation steps based on a local view of the environment, they often do not involve search budget constraints and rely on a predetermined kinematic model of motion. In contrast, our approach involves observing the full environment, albeit potentially at a lower resolution, and sequentially determining which regions to query without being constrained to a particular kinematic model. This highlights the distinctive nature of our approach compared to traditional visual navigation tasks and demonstrates the potential value of active search strategies in addressing budget-constrained settings.

Active Search Active Search was first introduced by Garnett et al. [7] as a means to discover members of valuable and rare classes rather than solely focusing on learning an accurate model as in Active Learning [11]. Subsequently, Jiang et al. [9, 22] proposed efficient nonmyopic active search techniques and incorporated search cost into the problem. Sarkar et al. [6] demonstrate that prior active search techniques do not scale well in high-dimensional visual space, and instead propose a DRL based visual active search framework for geo-spatial broad area search. However, the efficacy of the proposed approach by Sarkar et al. [6] can be limited when the search task varies between training and testing, which is often the case in real-world applications. In this work, we propose a novel framework that enables efficient and adaptive search in any previously unseen search task.

Meta Learning The concept of meta-learning, has consistently attracted attention in the field of machine learning [23, 24, 25, 26]. Finn et al. [25] present Model Agnostic Meta-Learning, a technique that utilizes SGD updates to rapidly adapt to new tasks. This approach, based on gradient-based metalearning, can be seen as learning an effective parameter initialization, enabling the network to achieve good performance with just a few gradient updates. Wortsman et al. [27] introduces a self-adaptive visual navigation approach, which has the ability to learn and adapt to novel environments without the need for explicit supervision. Our work is significantly different than all these prior works as we observe true label at each step during search and hence the main challenge is how to leverage the supervised information in order to learn an efficient adaptive search policy.

Foveated Processing of Large Images Several studies have investigated the utilization of lowresolution images for guiding the selection of high-resolution regions to process [28, 29, 30, 31, 32, 33, 34, 35], with some employing reinforcement learning techniques [36, 15] to improve this process. However, our work differs significantly, as we focus on selecting a sequence of regions to query, where each query provides the true label, instead of a higher resolution image region. These labels are crucial for guiding further search and serving as an ultimate objective. As such, our approach tackles a unique challenge that differs from existing methods that rely on low-resolution imagery.

6 Conclusions

We present a novel approach for visual active search in which we decompose the search policy into a prediction and search modules. This decomposition enables us to combine supervised and reinforcement learning for training, and make use of supervised learning even during execution. Moreover, we propose a novel meta-learning framework to jointly learn a policy and initialization parameters for the supervised prediction module. Our findings demonstrate the significance of the proposed frameworks for conducting efficient search, particularly in real-world situations where search tasks may differ significantly from those utilized during policy training. We hope our framework will find its applicability in many practical scenarios ranging from human trafficking to animal poaching.

Acknowledgments

This research was partially supported by the NSF (IIS-1905558, IIS-1903207, and IIS-2214141), ARO (W911NF-18-1-0208), Amazon, NVIDIA, and the Taylor Geospatial Institute managed by Saint Louis University.

References

Elizabeth Bondi, Debadeepta Dey, Ashish Kapoor, Jim Piavis, Shital Shah, Fei Fang, Bistra Dilkina, Robert Hannaford, Arvind Iyer, Lucas Joppa, et al. Airsim-w: A simulation environment for wildlife conservation with uavs. In ACM SIGCAS Conference on Computing and Sustainable Societies , pages 1-12, 2018.

Elizabeth Bondi, Fei Fang, Mark Hamilton, Debarun Kar, Donnabell Dmello, Jongmoo Choi, Robert Hannaford, Arvind Iyer, Lucas Joppa, Milind Tambe, et al. Spot poachers in action: Augmenting conservation drones with automatic detection in near real time. In AAAI Conference on Artificial Intelligence , 2018.

Elizabeth Bondi, Raghav Jain, Palash Aggrawal, Saket Anand, Robert Hannaford, Ashish Kapoor, Jim Piavis, Shital Shah, Lucas Joppa, Bistra Dilkina, et al. Birdsai: A dataset for detection and tracking in aerial thermal infrared videos. In IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1747-1756, 2020.

Fei Fang, Peter Stone, and Milind Tambe. When security games go green: Designing defender strategies to prevent poaching and illegal fishing. In International Joint Conference on Artificial Intelligence , 2015.

Fei Fang, Thanh Hong Nguyen, Rob Pickles, Wai Y Lam, Gopalasamy R Clements, Bo An, Amandeep Singh, Milind Tambe, and Andrew Lemieux. Deploying paws: Field optimization of the protection assistant for wildlife security. In AAAI Conference on Artificial Intelligence , volume 16, pages 3966-3973, 2016.

Anindya Sarkar, Michael Lanier, Scott Alfeld, Roman Garnett, Nathan Jacobs, and Yevgeniy Vorobeychik. A visual active search framework for geospatial exploration. arXiv preprint arXiv:2211.15788 , 2022.

Roman Garnett, Yamuna Krishnamurthy, Xuehan Xiong, Jeff Schneider, and Richard Mann. Bayesian optimal active search and surveying. arXiv preprint arXiv:1206.6406 , 2012.

Roman Garnett, Thomas Gärtner, Martin Vogt, and Jürgen Bajorath. Introducing the 'active search'method for iterative virtual screening. Journal of Computer-Aided Molecular Design , 29(4):305-314, 2015.

Shali Jiang, Gustavo Malkomes, Geoff Converse, Alyssa Shofner, Benjamin Moseley, and Roman Garnett. Efficient nonmyopic active search. In International Conference on Machine Learning , pages 1714-1723. PMLR, 2017.

Shali Jiang, Roman Garnett, and Benjamin Moseley. Cost effective active search. Advances in Neural Information Processing Systems , 32, 2019.

Burr Settles. Active learning literature survey. 2009.

Burr Settles. Active learning. Synthesis lectures on artificial intelligence and machine learning , 6(1):1-114, 2012.

Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 93-102, 2019.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Burak Uzkent and Stefano Ermon. Learning when and where to zoom with deep reinforcement learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1234512354, 2020.

Darius Lam, Richard Kuzma, Kevin McGee, Samuel Dooley, Michael Laielli, Matthew Klaric, Yaroslav Bulatov, and Brendan McCord. xview: Objects in context in overhead imagery. arXiv preprint arXiv:1802.07856 , 2018.

Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3974-3983, 2018.

Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems , 33:4247-4258, 2020.

Bar Mayo, Tamir Hazan, and Ayellet Tal. Visual navigation with spatial attention. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16898-16907, June 2021.

Mahdi Kazemi Moghaddam, Ehsan Abbasnejad, Qi Wu, Javen Qinfeng Shi, and Anton Van Den Hengel. Foresi: Success-aware visual navigation agent. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 691-700, January 2022.

Kshitij Dwivedi, Gemma Roig, Aniruddha Kembhavi, and Roozbeh Mottaghi. What do navigation agents learn about their environment? In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10276-10285, 2022.

Shali Jiang, Gustavo Malkomes, Matthew Abbott, Benjamin Moseley, and Roman Garnett. Efficient nonmyopic batch active search. Advances in Neural Information Processing Systems , 31, 2018.

Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. Learning to learn , pages 3-17, 1998.

Jürgen Schmidhuber, Jieyu Zhao, and Marco Wiering. Shifting inductive bias with successstory algorithm, adaptive levin search, and incremental self-improvement. Machine Learning , 28:105-130, 1997.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Harkirat Singh Behl, Atılım Güne¸ s Baydin, and Philip HS Torr. Alpha maml: Adaptive model-agnostic meta-learning. arXiv preprint arXiv:1905.07435 , 2019.

Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6750-6759, 2019.

Zuxuan Wu, Caiming Xiong, Yu-Gang Jiang, and Larry S Davis. Liteeval: A coarse-tofine framework for resource efficient video recognition. In Advances in Neural Information Processing Systems , volume 32, 2019.

Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020.

Yi Wang, Youlong Yang, and Xi Zhao. Object detection using clustering algorithm adaptive searching regions in aerial images. In European Conference on Computer Vision , pages 651-664, 2020.

Athanasios Papadopoulos, Pawel Korus, and Nasir Memon. Hard-attention for scalable image classification. In Advances in Neural Information Processing Systems , volume 34, pages 14694-14707, 2021.

Chittesh Thavamani, Mengtian Li, Nicolas Cebron, and Deva Ramanan. Fovea: Foveated image magnification for autonomous navigation. In IEEE/CVF International Conference on Computer Vision , pages 15539-15548, 2021.

Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and SerNam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12309-12318, 2022.

Chenlin Meng, Enci Liu, Willie Neiswanger, Jiaming Song, Marshall Burke, David Lobell, and Stefano Ermon. Is-count: Large-scale object counting from satellite images with covariatebased importance sampling. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 12034-12042, 2022.

Chenhongyi Yang, Zehao Huang, and Naiyan Wang. Querydet: Cascaded sparse query for accelerating high-resolution small object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13668-13677, June 2022.

Burak Uzkent, Christopher Yeh, and Stefano Ermon. Efficient object detection in large images using deep reinforcement learning. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , March 2020.

A Partially Supervised Reinforcement Learning Framework for Visual Active Search: Supplementary Material

A Policy Network Architecture and Hyperparameter Details

Recall that the policy network π is composed of two parts : (1) a task specific prediction module, and (2) a task-agnostic search module. The task specific prediction module consists of an encoder e ( x ; η ) that maps the aerial image x to a low-dimensional latent feature representation z , and a grid prediction network p ( z, o ; κ ) that predicts the probabilities of grids containing a target by leveraging the latent semantic feature z and the outcomes of previous search queries o . Note that the task specific prediction module is represented as f ( x, o, θ ) = p ( z = e ( x ; η ) , o ; κ ) , where θ = ( η, κ ) . Following [6], we use frozen ResNet-34, pre-trained on ImageNet, followed by a learnable 1 × 1 convolution layer with a ReLU activation as a feature extraction component of the task specific prediction module that we refer as encoder e ( . ) . We then combine the latent semantic feature z with the previous query information o . We apply the tiling operation in order to convert o into a representation with the same dimensions as the extracted features z , enabling us to effectively apply channel-wise concatenation of latent image feature and auxiliary state feature while preserving the grid specific spatial and query related information. This combined representation is then fed to a grid prediction network comprises of a 1 × 1 convolution layer, flattening, and a MLP block consists of 2 fully connected layer with ReLU activations. Note that the output of grid prediction network is of dimension N . We finally apply sigmoid activation to each output neuron to convert them into a probability value representing the probability of the grids containing target. The proposed policy architecture is depicted in figure 2 of the main paper.

We re-shape the output of task specific prediction module by converting it back from 1D to 2D of shape ( m × n ) = N before feeding it to the task agnostic search module g ( . ) that takes the following three inputs: (1) the reshaped 2D output of the task specific prediction module, which is the probabilities of grids containing target; (2) the remaining search budget B , which is a scalar but we apply tiling to the scalar budget B to transform it to match the size of the reshaped 2D output of the task specific prediction module; (3) we also apply the tiling operation to o in a way that allows us to concatenate the features ( z, o, B ) along the channels dimension to finally obtain the combined representation that serves as a input to task agnostic search module. The task agnostic search module is composed of a flattening, a MLP block consists of 2 fully connected layer with ReLU activations, and a final softmax layer to convert the output to a probability distribution that guides us in selecting the grid to query next.

In Table 8, we detail the architecture of task specific prediction module ( f ) of PSVAS policy network. In Table 9, we detail the architecture of task agnostic search module ( g ) of PSVAS policy network. Note that, the task specific prediction module and task agnostic search module remains unchanged in MPS-VAS framework.

In MPS-VAS-MQ framework, the network architecture of task specific prediction module remains unaltered, but the additional dependence of task agnostic search module ( g ) on ψ enforce a slight modification of its architecture as detailed in Table 10.

We use a learning rate of 10 -4 , batch size of 16, number of training epochs 200, and the Adam optimizer to train the policy network in all experimental settings. During Inference, in all experimental settings, we update the parameters of task specific prediction module f after each query step using a learning rate of 10 -4 and the Adam optimizer. We use 1 NVidia A100 and 3 GeForce GTX 1080Ti GPU servers for all our experiments.

B Results with Uniform Query Cost

B.1 Single Query Setting

Here we present the results by considering a setting with a single query resource and query costs c ( i, j ) = 1 for all i, j , where C is the number of queries. We evaluate PSVAS and MPS-VAS on the xView dataset with varying search budget C ∈ { 12 , 15 , 18 } and the number of grid cells N = 49 . We train the policy with small car as the target and test the performance of the policy with the following target classes : Small Car (SC) , Helicopter , Sail Boat (SB) , Construction Cite (CC) , Building , and Helipad . The results are presented in Table 11. We observe noticeable improvement in performance of the proposed PSVAS approach compared to all baselines in each different target setting, ranging from approximately 0 . 50 to 52 . 0% relative to the most competitive E2EV AS baseline. In Table 12, we report the results on DOTA dataset with N = 64 . In this setting, we train the policy with large vehicle as the target and evaluate the performance with the following target classes : Ship , large vehicle (LV), Harbor , Helicopter , Plane , and Roundabout . Here, we notice significant improvement in performance of PSVAS compared to all the baselines including E2EVAS, ranging from approximately 3 . 5 to 25 . 0% . The effectiveness of the PSVAS framework becomes evident as it allows us to efficiently update the task-specific prediction module f by leveraging the crucial supervised information. We

also observe a consistent trend, i.e., the performance of MPS-V AS is significantly better than PSV AS across different target settings, ranging from approximately 0 . 6 to 60 . 0% . The significance of the MPS-VAS framework becomes apparent when deploying visual active search in scenarios where the search tasks differ substantially from those encountered during training.

B.2 Multi Query Setting

In Table 13, we present the results of MPS-VAS-MQ and compare its performance with MPSVAS-TOPK with varying search budget C ∈ { 12 , 15 , 18 } and the number of grid cell N=49. Here, we train the policy with small car as the target and evaluate the performance of the policy with the following target classes : Small Car (SC), Helicopter , Sail Boat (SB), Construction Cite (CC), Building , and Helipad . In table 14, we present similar results with the number of grid cell N = 64 . In this setting, we train the policy with Large Vehicle as the target and evaluate the policy with the

following target classes: Ship , Large Vehicle (LV), Harbor , Helicopter , Plane , and Roundabout (RB). We consider K = 3 in all these experiments. We observe a consistent improvement in performance of MPS-VAS-MQ over MPS-VAS-TOPK across different target setting, ranging from approximately 0 . 1 to 15% . The experimental results indicate that there are additional benefits in learning to capture the interdependence in greedy search decisions.

C Results with Different Number of grid cells

Here, we present the results of PSVAS and MPS-VAS and compare the performance with the most competitive E2EVAS approach for different choices of N .

C.1 Results with Number of Grid cell N = 99

In this setting, we train the policy with small car as the target and evaluate the performance of the policy with the following target classes : Small Car (SC), Helicopter , Sail Boat (SB), Construction Cite (CC), Building , and Helipad . In Table 15, we present the results with Manhattan distance based query cost in single query setting. The similar results with multi query setting are presented in Table 16. In Table 17 and 18, we present the results with uniform query cost in single and multi query setting respectively. We notice a very similar trend in performance as observed in the settings with other choices of N . Specifically, We observe PSVAS significantly outperforms E2EVAS across different target settings, and MPS-V AS further improves the search performance universally. These results highlights the effectiveness of our proposed PSV AS and MPS-V AS framework for visual active search in practical scenarios when search tasks differ from those that are used for policy training.

C.2 Results with Number of Grid cell N = 36

In this setting, we train the policy with large vehicle as the target and evaluate the performance with the following target classes : Ship , large vehicle (LV), Harbor , Helicopter , Plane , and Roundabout . In Table 19, we present the results with Manhattan distance based query cost in single query setting. The results with multi query setting are presented in Table 20. In Table 21 and 22, we present the the results with uniform query cost in single and multi query setting respectively. We observe a consistent performance trend across various target settings. Specifically, PSVAS consistently outperforms E2EVAS in different target settings, and the introduction of MPS-VAS further enhances the search performance across the board. These results emphasize the effectiveness of our proposed PSV AS and

MPS-VAS framework for visual active search in real-world scenarios where the search tasks differ from the ones used during policy training.

D Effect of Inference Time Adaptation of Task Specific Prediction Module on Search Performance

D.1 Effect on PSVAS Framework

First, we analyze the impact of inference time adaptation of task specific prediction module on PSV AS framework across different target settings. To this end, we first train a policy using our proposed PSVAS approach and then during inference we freeze the task specific prediction module along with task agnostic search module unlike PSVAS approach. We call the resulting policy as PSVAS-F. In Table 23, we compare the search performance of PSVAS and PSVAS-F with number of grid cell N = 36 across different target settings. In Table 24, we present similar results with number of grid cell N = 49 . We observe a significant improvement in performance of PSVAS compared to PSVAS-F across different target settings, justifying the importance of inference time adaptation of task specific prediction module after every query.

In Figure 4, the distinct exploration strategy behaviors of PSV AS and PSVAS-F are depicted when both policies are trained with a large vehicle as the target and tested with a ship as the target. Out of a total of 15 queries, PSVAS-F achieves 6 successful searches, while PSVAS achieves 8 successful searches. Figure 5 illustrates the contrasting exploration strategy behaviors between PSVAS and

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

PSVAS-F in the case when both the policies are trained with large vehicle as the target and test

with plane as the target. We observe PSVAS-F yields 9 successful searches, while PSVAS yields 12 successful search out of 15 total query.

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

Figure 6 illustrates the contrasting exploration strategy behaviors between PSV AS and PSVAS-F in the case when both the policies are trained with large vehicle as the target and test with roundabout as the target. We observe PSVAS-F yields 5 successful searches, while PSVAS yields 7 successful search out of 15 total query.

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

D.2 Effect on MPS-VAS Framework

Next, we examine the influence of inference time adaptation of the task-specific prediction module on the MPS-VAS framework across various target settings. For this purpose, we train a policy using our proposed MPS-VAS approach. But during inference, we freeze both the task-specific prediction module and the task-agnostic search module, which differs from the standard MPS-V AS approach. We refer the resulting policy as MPS-VAS-F. Table 26 presents a comparison of the search performance between MPS-VAS and MPS-VAS-F, considering a grid cell count of N = 36 , across various target settings. Similarly, in Table 25, we provide corresponding results with a grid cell count of N = 49 . Across various target settings, we observe a notable enhancement in the performance of MPS-VAS compared to MPS-VAS-F. This finding underscores the significance of adapting the task-specific prediction module during inference after each query, validating its importance on adaptive visual active search. Following Figures demonstrate the divergent exploration strategy behaviors exhibited by MPS-VAS and MPS-VAS-F.

Figure 7 illustrates the contrasting exploration strategy behaviors of MPS-V AS and MPS-VAS-F when both policies are trained with a large vehicle as the target and tested with a plane as the target. Among a total of 15 queries, MPS-VAS-F achieves 2 successful searches, while MPS-VAS achieves

4 successful searches. In Figure 8, the distinct exploration strategy behaviors of MPS-VAS and

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

MPS-VAS-F are depicted when both policies are trained with a large vehicle as the target and tested with a ship as the target. Out of a total of 15 queries, MPS-VAS-F achieves 7 successful searches,

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

while MPS-VAS achieves 9 successful searches. Figure 9 showcases the contrasting exploration strategy behaviors of MPS-VAS and MPS-VAS-F when both policies are trained with a large vehicle as the target and tested with a roundabout as the target. Among a total of 15 queries, MPS-VAS-F achieves 6 successful searches, while MPS-VAS achieves 8 successful searches.

E More Visualizations of Comparative Exploration Strategies of Different Approaches

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

The showcased visualizations (10, 11, 12, 13, 14) in all these examples demonstrate the superiority of our PSVAS and MPS-VAS framework compared to the E2EVAS baseline, especially in scenarios where search tasks vary from those employed in policy training.

F Analyzing Search Performance Across Multiple Trials

Here, we compare the search performance of E2EVAS, PSVAS, and MPS-VAS across multiple trials. In Figure 15, we present the results when the polices are trained with small car as the target and evaluate the performance under Manhattan distance based query cost C = 25 with the following target classes: Small Car (SC) , Helicopter , Sail Boat (SB) , Construction Cite (CC) , Building , and Helipad . In figure 16, we present similar results with Manhattan distance based query cost budget C = 50 . In figure 17, we also present similar results with Manhattan distance based query cost budget C = 75 .

In Figure 18, we present the results when the polices are trained with large vehicle as the target and evaluate the performance under Manhattan distance based query cost C = 25 with the following target

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

classes: Ship , large vehicle (LV), Harbor , Helicopter , Plane , and Roundabout . In figure 19, we present similar results with Manhattan distance based query cost budget C = 50 . In figure 20, we also present similar results with Manhattan distance based query cost budget C = 75 .

step 1

step 3

step 5

step 7

step 9

step 11

step 13

step 15

Trained withSmall Car as the target,N=49,and Distance based Query Cost Budget(C)=25

Helicopter as Target

Building as Target

CC as Target

SC as Target

Helipad as Target

1.0

1.2

6.8

1.8

5.6

T

1.00

0.9

1.1

6.6

1.7

5.4

0.95

0.8

1.0

ANT

6.4

1.6

5.2

0.90

0.7

0.9

0.6

0.8

6.2

1.5

5.0

0.85

0.5

0.7

6.0

1.4

4.8

0.80

0.6

5.8

4.6

0.75

E2EVAS PSVAS MPS-VAS

E2EVASPSVASMPS-VAS

E2EVASPSVASMPS-VAS

E2EVASPSVASMPS-VAS

E2EVASPSVASMPS-VAS

E2EVAS PSVAS MPS-VAS

Trained withSmallCaras the target,N=49,and Distancebased QueryCostBudget(C)=50

Helicopter as Target

Sail Boatas Target

Building as Target

CC as Target

SC as Target

Helipad as Target

1.40

1.2

1.7

10.6

2.7

8.4

1.1

1.6

10.4

8.2

1.35

ANT

1.5

10.2

2.6

1.30

1.0

1.4

10.0

2.5

8.0

1.3

9.8

7.8

1.25

0.9

9.6

2.4

1.2

7.6

1.20

0.8

1.1

9.4

9.2

2.3

7.4

1.15

E2EVAS PSVAS MPS-VAS

E2EVASPSVASMPS-VAS

E2EVAS PSVAS MPS-VAS

E2EVAS PSVAS MPS-VAS

E2EVAS PSVAS MPS-VAS

E2EVASPSVASMPS-VAS

G Search Performance Comparisons Across Datasets

Our experimental outcomes indicate that it is possible to apply our method directly across different datasets without requiring any further modifications or hyperparameter tuning. In the following table 27, we demonstrate this by presenting results of training on one dataset for one target class while evaluating on another dataset and for another target class. We use the number of equal sized grid cells N = 64 and varying search budgets C = { 25 , 50 , 75 } .

Trained with SmallCar as the target,N=49,and Distance based Query CostBudget(C)=75

Helicopter as Target

Building as Target

CC as Target

SC asTarget

Helipad as Target

2.2

13.75

10.8

1.70

1.45

1.40

2.0

13.50

13.25

3.5

3.4

10.6

1.65

10.4

1.60

ANT

1.35

1.8

13.00

3.3

10.2

1.30

1.6

12.75

3.2

10.0

1.55

12.50

3.1

1.50

1.25

9.8

1.4

12.25

3.0

1.45

1.20

12.00

2.9

9.6

1.40

E2EVAS PSVAS MPS-VAS

E2EVASPSVASMPS-VAS

E2EVASPSVASMPS-VAS

E2EVASPSVASMPS-VAS

E2EVASPSVASMPS-VAS

E2EVASPSVASMPS-VAS

Trained with Large Vehicle as the target,N=64,and Distance based Query Cost Budget(C)=25

Ship as Target

LV as Target

Harbor as Target

HelicopterasTarget

Plane as Target

RB as Target

3.9

0.70

3.1

5.425

8'

3.5

3.5

3.0

5.400

3.7

0.65

ANT

2.8

2.9

5.375

5.350

3.5

3.6

0.60

0.55

3.4

3.3

3.3

3.4

2.7

5.325

3.4

3.2

3.2

2.6

2.5

5.300

5.275

3.2

3.3

0.50

0.45

3.1

3.1

2.4

3.1

3.0

3.0

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

Trained withLarge Vehicle as the target,N=64,and Distancebased Query CostBudget(C)=50

5.4

Ship as Target

8.65

LV as Target

6.75

Harbor as Target

1.1

Helicopter as Target

5.6

Plane as Target

5.6

RB as Target

5.2

8.60

6.50

1.0

6.25

5.4

5.4

ANT

5.0

8.55

6.00

0.9

4.8

8.50

5.75

5.2

5.2

0.8

4.6

8.45

5.50

5.0

5.0

5.25

0.7

4.4

8.40

5.00

4.8

4.8

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

Trained with Large Vehicle as the target,N=64,and Distance based Query Cost Budget (C)= 75

ShipasTarget

LV as Target

HarborasTarget

HelicopterasTarget

Plane as Target

RB as Target

7.2

10.70

1.7

7.8

7.8

9.0

7.0

10.65

1.6

7.6

7.6

6.8

10.60

8.5

1.5

7.4

7.4

ANT

6.2

6.6

6.4

10.45

10.55

10.50

8.0

7.5

1.4

1.3

1.2

7.0

6.8

6.6

7.2

7.0

8'9

6.6

7.2

6.0

10.40

7.0

1.1

6.4

6.4

5.8

1.0

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

E2EVAS

PSVAS

MPS-VAS

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel Reinforcement Learning (RL) framework for visual active search. In contrast to previous methods, the authors suggest integrating visual supervised signals to enhance performance. Additionally, a meta-learning based approach is proposed to allow the model to adapt to varying test cases. The authors also expand their methodology to support multiple queries. Experimental results demonstrate that the proposed method significantly outperforms both the simple baseline approaches and previous RL-based methods.\n\nSTRENGTHS:\n- The proposed method is both intuitive and technically sound.\n- The motivation for test-time adaptation and multi-query adaptation is logically sound and well-justified.\n- A variety of datasets and settings were utilized for the experiments, which substantiate the superior performance of the proposed method compared to other approaches.\n- The manuscript is well-written and presented, enhancing its clarity and comprehension.\n\nWEAKNESSES:\n- Regarding PSVAS, while the concept of simply adding supervised loss is reasonable, it seems more like a straightforward engineering extension to an existing method rather than a novel contribution.\n- The authors should conduct more ablation studies to provide readers with a deeper understanding of their method. For instance, an ablation study on hyper-parameters and different modules of the proposed method would be valuable.\n- In the proposed method, the supervised loss is merely a binary classification loss that signifies the presence of a target in the query image patch. I wonder if integrating a more fine-grained supervised loss could enhance the overall performance. For example, would adding a detection or segmentation loss be beneficial?\n- The paragraph from lines 128 to 140 appears to lack logical consistency. I recommend that the authors restructure this section for improved clarity.\n\nQUESTIONS:\nComments: Although I don't directly work on this area, I perceive the research problem to be valuable, and the proposed method appears to be sound. However, given my limited expertise, I find it challenging to assess the significance of this paper's technical contributions. Therefore, I've initially rated it as \"Borderline Accept\". My final rating will depend on feedback from other reviewers, as well as the author's rebuttal.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Partially Supervised Reinforcement Learning to search for the objects of interest in geospatial images. The paper uses a meta-learning approach to learn initialization parameters of the prediction module jointly with a search policy.  A variant of this approach, MPS-VAS, learns  to choose a subset of queries to make in each search iteration. The proposed method outperforms the state-of-the-art and other baselines on xView and DOTA benchmarks.\n\nSTRENGTHS:\nThis paper proposes a Partially Supervised Reinforcement Learning to search for the objects of interest in geospatial images. The paper uses a meta-learning approach to learn initialization parameters of the prediction module jointly with a search policy.  A variant of this approach, MPS-VAS, learns  to choose a subset of queries to make in each search iteration. The proposed method outperforms the state-of-the-art and other baselines on xView and DOTA benchmarks. The results show drastically better results than the state-of-the-art model OnlineTTA.\n\nWEAKNESSES:\nWhile the proposed method is promising and novel, it does limit itself to only two benchmarks on satellite images. It would increase the contribution significantly if it was tested on some traditional vision benchmarks. At least, a discussion on it would be useful.\n\nQUESTIONS:\n1. Can we train on a dataset (xView) and generalize to the classes on another dataset (DOTA)?\n\n2. Can we use the same model for image classification task on the fMoW dataset?\n\nLIMITATIONS:\nThis paper proposes a Partially Supervised Reinforcement Learning to search for the objects of interest in geospatial images. The paper uses a meta-learning approach to learn initialization parameters of the prediction module jointly with a search policy.  A variant of this approach, MPS-VAS, learns  to choose a subset of queries to make in each search iteration. The proposed method outperforms the state-of-the-art and other baselines on xView and DOTA benchmarks. The results show drastically better results than the state-of-the-art model OnlineTTA. However, the method is not tested on traditional vision benchmarks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposes a partially supervised reinforcement learning framework (PSVAS) for visual active search tasks (VAS). Specifically, it decompose previous end-to-end deep RL method for VAS into a task-specific prediction module and a task-agnostic search module. In such way, the prediction module can be supervised with observed ground truth during training or inference time. Further, a meta-learning setting is added to adapt to different tasks quickly. Results on xView and DOTA datasets show the proposed methods and variants outperform previous baselines.

STRENGTHS:
The idea to utilize the target-location labels as additional supervision signal besides RL loss is easy to follow and makes sense. The method shows much better adaptation capability when applying to search target that is not trained on, demonstrating the advantages of decomposing the problem into prediction and searching separately. The meta-learning variants and multi-query variants further enhanced the performance. Overall the method is techniquely sound and empirically proved.

WEAKNESSES:
How different balancing factor lambda between RL and BCE loss would affect the performance is worth to investigate in ablation study section.

QUESTIONS:
Can authors address the weaknesses part.

LIMITATIONS:
Whether it is possible to apply such method across different datasets is not discusses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

comments: SUMMARY:
The paper addresses visual active search problem for geospatial exploration. For an effective adaptation of search policies to out-of-distribution tasks, the paper proposes partially supervised reinforcement learning framework. A meta-learning approach is also proposed that focuses on jointly learning a policy and initialization parameters for the supervised prediction module. Experiments are presented on two datasets.

STRENGTHS:
+ Proposed partially supervised reinforcement learning framework allows effective adaptation of search policies to out-of-distribution tasks.
+ Proposed meta-learning framework focuses on effective learning of supervised prediction module
+ Experiments show promising results

WEAKNESSES:
-- Not sure of the reproducibility of this work. Also, no code is provided in the supplementary.
-- If there is fully labeled training task, why imitation learning (IL) is not adopted? IL and RL loss can be used together for the task. 
-- It will be good to provide the formulation of L_BCE and L_RL loss in the paper?

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-16