Learning Transferable Reward for Query Object Localization with Policy Adaptation
Authors: Tingfeng Li
Shaobo Han
Martin Renqiang Min
Dimitris N. Metaxas
Venue: ICLR 2022 Poster (2022)

Abstract: We propose a reinforcement learning based approach to query object localization, for which an agent is trained to localize objects of interest specified by a small exemplary set. We learn a transferable reward signal formulated using the exemplary set by ordinal metric learning. Our proposed method enables test-time policy adaptation to new environments where the reward signals are not readily available, and outperforms fine-tuning approaches that are limited to annotated images. In addition, the transferable reward allows repurposing the trained agent from one specific class to another class. Experiments on corrupted MNIST, CU-Birds, and COCO datasets demonstrate the effectiveness of our approach.

Published as a conference paper at ICLR 2022
LEARNING TRANSFERABLE REWARD FOR QUERY OBJЕCT LOCALIZATION WITH POLICY ADAPTATION
Tingfeng Litg; Shaobo Han Martin Rengiang Min Dimitris Metaxas NEC Labs America, 'Department of Computer Science, Rutgers University (t1601, dnm) @cs. rutgers.... edu, (shaobo, renqiang @nec- com
ABSTRACT
We propose a reinforcement learning based approach to query object localization, for which an agent is trained localize objects of interest specified by a small exemplary set. We learn a transferable reward signal formulated using the exem- plary set by ordinal metric learning. Our proposed method enables test-time policy adaptation to new environments where the reward signals are not readily available, and outperforms fine-tuning approaches that are limited to annotated images. In addition, the transferable reward allows repurposing the trained agent from one specific class to another class. Experiments on corrupted MNIST, CU-Birds, and COCO datasets demonstrate the effectiveness of our approach !
INTRODUCTION
There is increasing interest in designing machine learning models that can be adapted to unseen environments or repurposed for new tasks, with only minimal human guidance (Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017; Sun et al., 2020; Hansen et al., 2021). To this end, a small set of examples can not only serve the purpose of implicitly generalizing a trained model to new environment during training time, but also enable the learner to update its learning objectives during test time.
In this paper, we focus on a reinforcement learning (RL) formulation to the problem of query object localization, in which an agent is trained to localize the target object specified by a small set of exemplary images. Instead of using fixed bounding box proposals. our vision-basse agent can be viewed as proactive information gatherer (Guo, 2003), which actively interacts with an image environment; Furthermore, it follows a class-specific localization policy, and thus is more suitable for acrial imagery (Xia et al., 2018), robotic manipulation (Kalashnikov et al., 2018), or embodied AI (Savva et al., 2019) tasks.
During the test time of query object localization, the queried object class to localize may be novel, or the background environment may undergo substantial changes, hindering the applicability of class-agnostic agents with fixed policy. In addition, in standard RL settings for object localization, the reward signal is often available, so fine-tuning methods (Julian et al., 2020) can effectively adapt agents to new environments and yield improved performance; On the contrary. in query object localization, the reward signal is not available test time, because the bounding box annotations to be found by the localization agent on test images.
To address these problems, we propose an ordinal metric learning based framework for learning an implicitly transferable reward signal defined with a small exemplary set. An ordinal embedding network pre-trained with data augmentation under loss function designed to be relevant to the RL task. The reward signal allows explicit updates of the controller in the policy network with continual training during test time. Compared fine-tuning approaches, the agent can get exposed to new environments more extensively with unlimited usage of test images. Informed by the exemplary set precisely, the agent is adapted to the changes of the localization target.
Our contributions in this paper are summarized as follows: (1) We propose novel ordinal metric learning based RL framework to learn transferable reward signal defined with small exemplary
Work done as an intern at NEC Labs America
Code available https: //gi wht.bb...uugggogggg////gtg com/ /1 1it ittingffng///oon /oocl izat by-by-tn OrdKmbed

---

Published as a conference paper at ICLR 2022
set for query object localization; (2) Our RL framework enables test-time policy adaptation to new environments where there is no readily available reward signal; (3) Our learned transferable reward allows repurposing trained agent from localizing specific class to another; (4) Extensive experiments on several datasets demonstrate the effectiveness of our proposed approach.
2
RELATED WORK
2.1 OBJECT LOCALIZATION
Compared bounding-box regression approaches (Redmon al., 2016; Ren al., 2016), deep RL based object localization approaches (Caicedo & Lazebnik, 2015; Jie et al., 2016) have the advantage of being agiion-propssas free, with customized search paths for each image environment. The specificity of an agent purely depends on the classes of bounding- boxes used in the reward. They can be made class-specific (Caicedo & Lazebnik, 2015), but the agent for each class would need to be trained separately.
Despite the rise of crowdsourcing platforms, obtaining ample amount bounding-box annotations remains costly and error-prone. Furthermore, the quality of annotations often varies, and precise annotations for certain object class may require special expertise from annotators. The emergence of weakly supervised object localization (WSOL) (Song et al., 2014; Oquab et al., 2015; Zhou et al., 2016) methods alleviates the situation, in which image class labels are used to derive bounding box annotations. It is known that WSOL methods have drawbacks of overly relying inter-class discriminative features and failing to generalize to classes unseen during the training phase.
We argue that intra-class similarity is a more natural objective for the problem of localizing objects belonging to a target class. "Query object localization" emphasizes more about the class-specific nature of localization task, with the relevance defined by the similarity query images. similar problem is image co-localization (Tang et al., 2014; Wei et al., 2017), in which the task is to identify the common objects within set images. Co-localization approaches exploit the common characteristics across images to localize objects. Being unsupervised, co-localization approaches could suffer from ambiguity if there exist multiple common objects or parts, e.g.. bird head and body (Jaderberg et al., 2015), which may provide unwanted common objects as output.
2.2 POLICY ADAPTATION
There seems to a contradiction between the goals of training an agent with high task specifiitt and better generalization performance to new situations the same time. The key to reconcile these two goals lies in the usage of a small set of examples. There has been a paradigm shift from training static models defined with parameters dynamic ones defined together with a support set (Vinyals et al., 2016; Snell et al., 2017), proved to be very effective in few-shot classification.
Besides the efforts of employing meta learning to adjust models, fine-tuning on a pre-trained model (Oquab et al., 2014; Finn et al., 2017; Julian et al., 2020) has also been widely used in transferring knowledge from data-abir aata-abbudant to data-scare scarce tasks. When reward signal is not available, (Hansen
Training Exemp ary Set
Test Exemplary
Reward
Training
Test-time Adaptation
Traiking Image Ewironment
Test Image Enviromennt
Localization Agent
Figure 1: RL-based query object localization: training and adaptation. Image- pairs are only available in the training phase. Test environments include images without box annotations and an exemplary specifying a different task objective. Thus, the trained policy needs be adapted.
2

---

Published as a conference paper at ICLR 2022
et al., 2021) proposed policy adaptation approach, in which the intermediate representation is fine-tuned via optimizing a self-superrvsed auxiliary loss while the controller is kept fixed. Our work shares the same motivation of test-time training (Sun et al., 2020), but focus instead on the settings where the controller needs to be adapted or even repurposed for new tasks.
3 LEARNING TRANSFERABLE REWARD FOR LOCALIZING QUERY OBJECTS
3.1 RL FORMULATION FOR QUERY OBJECT LOCALIZATION
Given set of images I with known class labels, the task of query object localization is to find the location b of the bounding box most relevant to the query object, specified in a small set of exemplary images E. Following the Markov Decision Process (MDP) framework for object localization (Caicedo & Lazebnik, 2015; Jie et al., 2016), this task can be formulated as a RL. problem. In the training phase, localization agent learns take actions of moving the bounding box to maximize a reward function reflecting localization accuracy. In the test phase, the agent is informed by small test exemplary set about a new task objective. Figure shows an overview of the query object localization problem and the RL formulation, with components listed as follows.
Environment: Raw pixels of single image 7, without candidate proposal boxes.
Discrete actions facilitating a top-down search, including five scaling: scale towards top left/right, bottom left/right, center; eight translation transformations: move left/right/up/- down, enlarge width/height, reduce width/height; plus one stay action.
Action:
State:
Pooled features from the current box and an internal state within RNN that encodes information from history observations.
Reward: Improvements the agent made in localizing the query images (in different tasks).
3.2 PRE-TNINNING ORDINAL REWARD SIGNAL
For training an agent to localize different objects according to queries, the reward function needs to be transferable. Existing deep RL approaches for object localization (Caicedo & Lazebnik, 2015; Jie et al., 2016) use its ground- truth object bounding box 9i the reward signal,
(1) where (b, 9.) denotes the Intersection-over- Unven (IoU) between the current window b, and the corresponding ground-truth box g.?, and loU g) area(b (b n g)/ aaaa U 9). Similar to the bounding box regression approaches which learn a mapping Φ I g. the image and box must be paired. However, annotated image-box pairs (I,9) may be scarce in both the training and testing phases. The reward signal defined in Eq. is instance-wise, which is not transferable across images.
(lgulob 9:) IoU(bi-1.g.))
To address this problem, a natural idea is to define the reward signal based on learned representations Given their M-
of the cropped images by current window b, and the ground-truth window dimensional representations b, and g produced by an embedding function f : RR. D + M from D- dimensional image feature vectors, a distance function d' IR x IR M + [0, c0)) returns the embedding distance d(b,,g) g). However, an embedding distance based on off-the-shelf pre-trained networks is not adequate, since it may not decrease monotonically as the agent approaches to the ground-truth box g. As a result, the embedding distance based reward signal may be less effective than Eq. I (as shown in Section 4). Therefore, an embedding function customized for the downstream RL-based localization task is needed.
Ordinal metric learning. We propose to use an ordinal embedding based reward signal. For any two perturbed boxes 6,, by from ground truth g, embeddings by. bk g are learned, such that the relative preferences between any pair of boxes are preserved in the Euclidean space.
Pj > Pk b gll < [bb - gll.
Vj, k C,
where Pj and Pk denote the preference (derived from either IoU to ground-truth box or ordinal feedback from user), and C is an ordinal constraint set constructed by sampling box pairs around
Boundingg box are denoted by lowercase lightface letters, e.g. g. while embedding vectors are denoted by holdface letters, e.g. g E R. is d-dimensional vector.
3

---

Published as a conference paper at ICLR 2022
g. This problem is originally posed non-men metrriii multidimensional scaling (Agarwal al., 2007; Jamieson & Nowak, 2011). Although we apply a very simple pairwise- basiee approach, there exist other extensions such as the listwise-based (Cao et al., 2007), quadruplet- saadruplet-based (Terada & Luxburg, 2014) and
(Anderton & Aslam, 2019; Ghosh et al., 2019) approaches.
Loss function. In this paper, we define preference p the IoU of box b to the ground-truth box g. i.e., (b, g). We choose to optimize a triplet loss (Hermans et al., 2017) for learning the desired embeddings,
miplet
Σ
max (m d(a. p) d(a,n),,) n),
(3)
anchor, ordinal pains
where a is the "anchor" embedding that will be discussed in detail later, n are the "positive" and "negative" embeddings for boxes with larger and smaller IoUs with ground truth box g. respectively, m is margin on positive and negative data obtained via data augmentation such as box perturbation.
We learn an embedding space consistent with local ordinal constraints
Box Perturbation Assuming the training exemplary set Eiain contains both image I and box we adopt tailored data augmentation scheme box perturbation, which a pair of boxes are randomly sampled from / with different IoUs to g. We compare the efficiency of sampling schemes on generating augmented bounding box pairs, (a) Random sampling, where the pair boxes are generated completely randomly, and (b) Group- scales are first generated, and then divided into M = 10 groups according to the IoU with ground- truth box. Each group has IoU interval of 1/M. The sampling is first done the group level, i.e., 2 out of M, then sample one box from each group. Thus, the sampled boxes are likely to cover more diverse cases comparing random sampling. We have found that using IoU-based partition scheme is more effective than random sampling.
sampling, where dense boxes with variant
The choice of anchor. The anchor embedding in Eq. 3 is not restricted the cropped image from the same image. For example, it could be replaced by the prototype embedding (Snell et al., 2017) of the exemplary set E, 1/(8] E gi where gi is the embedding of the cropped image I, by ground- truth box 9:- If images from multiple classes are of interest, the prototype can be further made to class-dependent, or peatering--bsedee A prototype as the anchor enables test-time policy adaptation without image-box pairs in the exemplary set Ejeat In comparison, supervised methods such as Faster RCNN would require image-hox pairs for fine-tuning on new tasks. some experiments (Appendix B.I), we find that using prototype-base embedding as the anchor could lead to better generalization performance than using g. is also interesting to explore other treatment on the exemplary set, such as permutation-invarian architectures (Zaheer et al., 2017; Ilse et al., 2018; Lee et al., 2019).
3.3 LOCALIZATION AGENT TRAINING
As discussed in Section 3.2, we use the ordinal embedding rather than the bounding box coordinates to quantify the improvement that an agent makes, and the continuous-valued reward for the agent
(x)
Images
(b)
Rol
Rol
Roi Encoder
Encoder
states
Projection
RNN
head
head
Triplet Loav(u, p.n)
actions
Eyriptex
steps
Figure 2: (a) Modules for ordinal representation learning. The Rol encoder extracts Rol feature that will be used as the state representation for localization. The projection head learns ordinal embedding b for computing reward. (b) Controller network. In each step, the agent takes the output from the Rol encoder as state, while it also maintains an internal hidden state h² within a RNN, which encodes information from history observations. Then it outputs the action for next step.
4

---

Published as a conference paper at ICLR 2022
moving from state s' to takes the following form,
R(s,s') = [bb-1 cll
||b
-cll
(4)
where с is the prototype embedding, by is the embedding of the cropped region at step t. We expect positive reward the embedding distance between cropped image b, and query image a is decreased.
State representation. Note that a good representation for defining reward may not necessarily be a good state representation at the same time - may not contain enough information guiding the agent taking the right actions. Chen et al. (2020) suggests that adding a projection head between the representation and the contrastive loss substantially improves the quality of the learned representation. We find that the use of projection head is crucial in balancing the two objectives in our task. The network architecture is shown Figure 2(a), in which an MLP projection head is attached after an (Region of Interest) Rol encoder. The ROIAlign module handles boxes of different sizes.
Starting from a whole image as input, the agent is trained to select actions to transform the current box at each step, maximizing the total discounted reward. We use policy gradient based on a recurrent neural network (RNN) (Mnih et al., 2014) with a vector of history actions and states. More detailed architecture can be found in Figure 2(b) and Appendix Figure 6. For all experiments, we apply policy gradient (REINFORCE) (Williams, 1992) with entropy loss regularization encourage exploration. The main algorithm is outlined in Algorithm 1.
Policy network.
Training localization agent using the proposed ordinal reward signal. Require initial policy Te, batch size N, pre-trained Rol encoder and projection head leaminn rate a Output :policy network Te
Algorithm 1:
for sampled minibatch [=x) N do
for k= I=.....
Construct an exemplary set Enach
+ h(e(In, 9;)) // compute the ordinal embodding in Figure 2 (a)
C₁ f Email Ejeemo gi // compute the prototype vector Th f { s1, a1, .T. ar) by running policy its // sample trajectory
b, + take action az and compute embedding h(e(T,, be)) for current window b
R + l - cx
lbb;
cxll // compute reward in equation 4
end
0 - + a En
log Te (Er - - A no log // policy update
end
3.4 TEST-TIME ADAPTATION
During test time, the agent has the option of further updating the policy network using the received reward from Eq. 4. To match test conditions, the training batch is split into two groups, and c is computed small subset that does overlap with the training images to localize; During test-time adaptation, c becomes the prototype of the test exemplary set Evest- Different from E only cropped images are needed in Ess as shown in Figure 1.
EXPERIMENTAL RESULTS
In this section, we evaluate the generalization ability of the ordinal embedding as well as the perfor- mance of the trained localization agent on both source and target domains. The embedding- baaed reward not only improves the RI. training, but also enables test-time adaptation the learned policy. Experimental results demonstrate the effectiveness of our approach, with empirical comparisons to fine-tuning, co-localization, few-shot and Faster RCNN object detection baselines.
Implementation details. To evaluate the learned ordinal embedding, we use OrdAce defined as the percentage of images within which the preference between a pair of perturbed boxes is correctly predicted. To evaluate object localization performance, (Deselaers et al., 2012) metric, which is defined as the percentage of images correctly localized according the criterion loU(br:9) g) 0.5, where by is the predicted box and is the ground-truth
the Correct Localization (CorLoc)
5

---

Published as a conference paper at ICLR 2022
box. Mean and standard deviation (displayed as subscript) are reported from 10 independent runs. We evaluate our approach on distorted versions of the MNIST handwriting, the CUB-200-2 20B-200-2011 birds (Wah et al., 201 1), and the COCO (Lin et al., 2014) dataset. For the MNIST dataset, we use three convolutional layers with ReLU activation after each layer as the image encoder. For the CUB and the COCO datasets, we adopt layers before comv5_3 of VGG-16 pre-trained on ImageNet as the encoder, unless otherwise specified. More details are presented in Appendix A.2.
4.1 POLICY ADAPTATION DURING TEST TIME
We demonstrate the performance improvement with test-time policy adaptation. Through all the experiments, assume source domain contains abundant data annotations, and target domain annotations are only available an exemplary set of size 5. We compare our policy adaptation scheme with a standard fine-tuning scheme on the pre-trained policy network.
Results on the corrupted MNIST dataset. For the new class adaptation experiment, we use 50 "digit images under random patch background noises" to train the ordinal embedding and the localization agent. The results on policy adaptation to new digits (other than 4) are shown in Table 1. Row 1 illustrates the transferability of the ordinal embedding reward, trained prototype embedding of subgroup without the training instance, and evaluated using instance embedding from the same test image ("OrdAce"). Rows to list the resulting localization accuracy after direct generalization ("before"), fine-tuning on the exemplary set "ffne--tune),, and adaptation using all test images ("adapt"). respectively. Our policy adaptation approach produces a substantial improvement over direct generalization, while fine-tuning approach experiences overfitting on the limited exemplary set. For the background adaptation experiment, we train on 50 digit-3 images under random patch noise, and test d images under all four noises. The localization accuracy both source and new backgrounds environment are shown in Table 2, significant improvements are achieved using our policy adaptation scheme.
Table OrdAce (%) and Cor Loc (%) on new digits environment.
2
5
6
OrdAce 91.9p.7
90.32.9
92.11.8
92.0p.2
92.70.6
92.6u.2
90.7u.7
92.0o.7
90.50.7 91.6u.5
before
94.20.
84.11.5
88.71.7
86.51.8
81.21.4
91.90.3
89.50.7
93.01.0
90.80.4 88.91.0
fine-tune
93.32.5
80.43.9
84.53.5
84.91.5
78.82.7
87.32.9
82.25.4
87.74.5
85. 65.3 83.74.5
adapt
99.8p.2
95.60.6
98.10.4
97.90.4
88.30.5
99.10.2
98.60.9
99.80.3
99. 20.4 97.40.4
Table 2:
Loc (%)
when adapted to other background on corrupted MNIST dataset.
adapt
random patch
clutter
impulse noise
gaussian noise
mean
97.60.4 100.0o.o
39.6u.s
22. 22.1a.7
66.22.0 100.0..0
56.4
97.40.3
199.90.1
99.3
Results the CUB dataset. We also evaluate the policy adaptation performance the CUB dataset. The localization agent is trained on 15 species from the "Warbler" class, and adapted to different classes of "Warbler" (5 new species), "Wren", "Sparrow", "Oriole", "Kingfisher", "Vireo". and "Gull". Each test class contains a single bird class. We also implement deep descriptor transforming (DDT) (Wei al., 2017), a deep learning based so-localization approach, and add it to the comparison.
Table 3: Cor Loc (%) when adapted to other species/classes on CUB dataset.
adapt
warbler (new)
wren
spartow
oriole
kingfisher
vireo
gull
mean
DDT
73.8
78.6
71.2
74.5
78.0
69.2
93.3
76.9
85.51.1
82.92
81.83.7
77.90.7
78.9p
82.24.6
86.33.5
82.1
Ours
89.71.1
91.01.1
89.31.8
85.0p.a
85.91 25.94.
90.0p.9
93.9p.a
89.3
Results on the COCO dataset. Given a target domain that one would like to deploy models to, a natural question is whether one should collect labeled data from an abundant number of source domains from one specific class with potential lower data collection cost. On the COCO dataset, we train on one of the five classes: cat, cow, dog, horse, zebra, then adapt to another four classes. The results shown in Figure 3. The models perform best using ordinal embedding reward with

---

Published as a conference paper at ICLR 2022
ours-before
ours-adapt
iou-before
iou-adapt
CorLoc (%)
zelbra
20
Car home
Coar home
Dog ftrree
Zabia a hora
Cut oa cow
Dog cow
Hores cow
Zator cow
Cow old Dog Horse Zahra
Caks dog Cow dag Horse dog Zebra dog
Cal arbig Cos zrbre Dog sethra
Horen zetra
Figure 3: Cor Loe (%) Comparison with loU-based reward on COCO dataset. "ours": ordinal embedding based reward; "iou": IoU based reward, ImageNet pre-trained embedding. Dotted lines the results of directly training target class, using embedding based reward but from ImageNet pre-trained model, indicating the advantage of our approach, with ordinal embedding learned from a single source class and policy adaptation.
adaptation (see ""urss-ddpt"). It shows that, although an agent is trained in highly specialized way on only one-source tasks, it can still be flexibly generalized to different target tasks. One interesting observation is that, it is easy to transfer from other four classes to zebra, but not vice versa. A possible explanation might be that, the embedding net is biased on textures, and texture information less adaptable than shape information.
4.2 COMPARISON TO FEW-SHOT OBJECT DETECTORS
Few-shot object detection (Kang et al., 2019; Yan et al., 2019; Wang et al., 2019; 2020) methods similar to our work in adapting models to a new test case with the help of limited annotations, but they have a different requirement on the number of training classes. Being different in the generalization mechanism, they usually require multiple classes in both stages of training and fine-tuning. On the COCO dataset, we compared the performance of our methods with TFA (Wang al., 2020), and Meta-RCNN (Yan et al., 2019), on the same one-way 5-shot test setting. TFA adopts two-stage finetuning of object detector, while Meta-RCNN incorporates additional meta-learner acquire class-level meta knowledge for generalization to novel classes. These two few-shot baselines are trained on base classes and fine-tuned on all 80 classes, while our model is trained from one single class of the 5 classes randomly selected from the base classes set: elephant, sheep, giraffe, laptop and suitcase. Table 4 shows that with ordinal embedding, our agent achieves better performance even without adaptation, and the performance can be further improved after adaptation.
Table 4: Cor Loe (%) Adaptation comparison with other methods on COCO dataset. "TFA w/ fc": TEA with a cosine similarity based box classifier; "TFA w/ cos": TFA with normal FC-based classifier; "FRCN+ft-full feature extractor and the box predictor.
MCNa with Faster RCNN as detector and finetuning both the
target
cat
dog
COW
horse
mean
FfCN+ (Yan et al., 2019)
13.1
3.1
3.7
7.4
6.8
TFA w/ fe (Wang et al., 2020)
29.1
13.0
5.0
10.7
14.4
TFA w/ cos (Wang et al., 2020)
28.0
10.3
4.5
8.9
12.9
Ours-beford Ours-adapt
23.0
20.6
24.5
21.2
22.3
40.3
33.5
43.1
40.2
39.3
4.3 COMPARE WITH SUPERVISED BASELINE FASTER RCNN
We compare our framework with a strong supervised object localization baseline, Faster RCNN (Ren et al., 2016). Both methods trained in one class (foreground background the classification labels in Faster RCNN) and adapted to a different class. We fine-tune the pre-trained VGG- 16 model and test on each of the five classes: cow, cat, dog, horse, zebra. The results source domain
7

---

Published as a conference paper at ICLR 2022
shown in Table 5. It shows that Faster RCNN can also be made into class-specific model and it still yields superior performance on source domain. On the target domain, we fine-tune Faster RCNN using only query set for each target class. The results shown in Table 6. It can be seen that our method works better on new classes with test-time adaptation over the traditional fine-tuning of Faster RCNN. Note that fine-tuning requires image-box pairs in the target domain while policy adaptation approach does not.
Table 5: Cor (%) comparison with Faster RCNN source domain.
method
cow
cat
dog
horse
zebra
Faster RCNN (Ren al., 2016) ours
70.37
89.82
85,81
92.65
85.71
70.37
68.46
61.26
61.28
79.36
Table 6: Loc (%) comparison with Faster RCNN fine-tuned on target domain.
before fine-tune
before adapt
after fine-tune
after adapt
Faster RCNN
ours
Faster RCNN
ours
->horse
20.93
33.32
37.73
51.89
cow ->horse
54.79
48.41
68.04
46.80
cog ->horse
38.52
41.50
58.01
55.89
zebra ~>horse
1.12
10.29
6,04
39.22
>>cow
40.52
50.85
58.55
58.58
dog->con -Scow
54.55
54.63
70.11
62.86
horse ->cow
72.11
59.52
75.35
64.83
zebra->com
1.23
8.14
5.86
35,56
cow ->cat
46.12
39.84
53.85
46.42
dog->cu ->cat
67.62
57.97
77.07
63.12
horse ->eat
36.67
44,25
36.67
53.39
zebra ->eat
0.55
10.45
4.09
35.73
cat->dog
58.98
47,81
66.50
48.94
cow >>dog
45.85
37.28
51.69
42.33
horse-2dog
41.04
39.07
47.69
44.77
zebra ->dog
0.68
7.74
3.4
17.73
cat ->zehra
10.64
57.58
37.97
70.28
cow ->zebra
4.42
39.64
19.64
65.80
dog >>zebra
2.29
35.27
15.88
63.91
horse >>zebra
7.86
66,82
29.3
72.83
30.32.12.00
39.51
41.17125.33
52.04.11..9
4.4 ABLATION STUDIES
We analyze the effectiveness of ordinal embedding and RL component separately COCO dataset (Appendix B.I provides more in-depth analysis on the corrupted MNIST dataset, including policy gradient deep Q-Network, continuous vs. binary reward). First, remove RI., and substitute it with simple linear search approach. Specifically, we adopt selective search (Uijlings et al., 2013) for candidate boxes generation. The candidate boxes are ranked according to their embedding distances to the prototype, and the one with the smallest embedding distance is returned as the final output. We consider two backbone networks as the embedding. including ImageNet pre-trained VGG-16 and Faster RCNN VGG-16 trained locally on COCO dataset. We also compare both with and without ordinal component, making it 2 x ablation study. It can be seen from the blue and green bars from Figure 4 and Appendix Figure 10 that with ordinal structure, the ranking method performs much better. We find that pre- pre-trainne training with the proposed ordinal loss significantly improves the rank consistency of these backbone networks (Appendix B.
In contrast, being able to learn across different images, the RL localization agent shows more tolerance to the defects of the reward function. We analyze the benefits of RL component. From both Figure 4 and Figure 10, when using ordinal embedding, the model with RL generally better than ranking, especially after adaptation.
Ordinal component is necessary for several possible reasons. First, the ordinal component makes the generic pre-trained network more specific to the downstream object localization task. Second, it
8

---

Published as a conference paper at ICLR 2022
rank ord rank ord
before
ord adept
40
CorLoc (%)
20
Cat.a torse Cow horee Dog forte
Zetra hore
Car cor
Dor core Hote cowe Zelbra cop
Cow aft Dog cat Homse ost Zahra
Caso dog Con did Home Buy Zebra dog
Cat abdra Cow aatra Drgg radira Horee Datia
Figure 4: Cor (%) comparison with ranking method using Faster RCNN pre-trained backbone.
helps remove the dataset bias if the pre-trained network is non-native. Third, ordinal pre-training encourages the reward signal to reflect step-by-step improvements.
Encouraged by the strong baseline performance of Faster RCNN on both source and target domain (detailed in Table and Table 6), we investigate whether it is adequate to use Faster RCNN embedding in the RL reward function, with without ordinal pre-training. Since Faster RCNN emebedding is also trained for object localization on the COCO dataset, it can separate out the effects of the ordinal pre-training component better. The result are shown in Table 7. It can be seen that without ordinal pre-training, the performance degrades significantly and the Cor is much lower even in source domain. More results of different backbones are presented in Appendix Table 15 and Table 16, from which the test-time adaptation still brings a large margin of improvement.
Table 7: Loc (%) RL with Faster RCNN embedding w/ and w/o ordinal vre-training on source domain.
method
Cow
cat
dog
horse
zebra
Faster RCNN backbone
25.93 66.67
13.17
12.16
16.18
28.57
Faster RCNN backbone + Ord
72.46
61.59
60.29
71.25
CONCLUSION AND FUTURE WORK
We propose ordinal representation learning based reward, for training localization agent to search queried objects of interest. In particular, we use a small exemplary set as a guidance signal for delivering learning objectives without ambiguity. Meanwhile, we use test image environments to inform the agent about the domain shifts, without requiring image-box pairs during test time.
Instead of jointly training a localization and classification model, learn box annotations from image class labels, in similar spirit to weakly-supervist learning. Empirically, we focus extensively on the transfer learning setting from one single data abundant training task to data-scarce (one- way few-shot) test tasks (Taylor & Stone, 2009; Zhu et al., 2020). To make a trained RL agent generalizable new unseen test cases, one can either expose the agent to as many cases as possible during training, or make the agent more specific but adaptable upon changed scenarios. Our approach belongs to the latter.
The transferability of our reward signal from training to test crucially relies on the generalization ability of the learned ordinal representation. If the ordinal preference does not hold in the test domain, the proposed test-time policy adaptation scheme will not work. By adapting the representation with self-supervised objectives (Hansen et al., 2021), this issue might remedied.
Although our work focuses on the generalization ability of single object localizer, extensions to multiple objects can possibly be done by applying it multiple times and marking the found regions, in similar manner (Caicedo & Lazebnik, 2015). Curriculum learning with designed sequence of targets in the exemplary set may guide the agent toward solving challenging localization tasks. We leave those extensions for future study.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This research has been partially funded by the following grants ARO MURI 805491, NSF IIS- 1793883, NSF CNS- NSF IIS 1763523, DOD-ARO ACC-W91INE, NSF 0IA-2040638 to D. Metaxas. We would like to thank anonymous reviewers for their comments and suggestions.
REFERENCES
Sameer Agarwal, Josh Wills, Lawrence Cayton, Gert Lanckriet, David Kriegman, and Serge Belongie. Generalized non- non-metric multidimensional scaling. Artificial Intelligence and Statistics, pp. 11-18. PMLR, 2007.
Jesse Anderton and Javed Aslam. Scaling up ordinal embedding: A landmark approach. In Interna- lional Conference on Machine Learning, pp. 282-290. PMLR, 2019.
Juan C Caicedo and Svetlana Lazebnik. Active object localization with deep reinforcement learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2488-248820 2496, 2015,
Zhe Cao, Tao Qin, Tie Tin Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In International Conference on Machine Learning, pp. 129-136, 2007.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp. 1597-1607. PMLR, 2020.
Thomas Deselaers, Bogdan Alexe, and Vittorio Ferrari. Weakly supervised localization and learning with generic knowledge. International Journal of Computer Vision, 100(3):275-293, 2012.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pp. 1126-113 1135. PMLR, 2017.
Nikhil Ghosh, Yuxin Chen, and Yisong Yue. Landmark ordinal embedding. In Advances in Neural Information Processing Systems, volume 32, 2019.
AnYuan Guo. Decision-theorctic active sensing for autonomous agents. In International Joint
Conference Autonomous Agents and Multiagent Systems, pp. 1002-1003, 2003.
Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Pieter Abbeel, Alexei Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. In International Conference on Learning Representations, 2021.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re-identification. arXiv preprint IrXiv 2017.
Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learning. In International Conference on Machine Learning, pp. 2127-2136. PMLR, 2018.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volese Volume 2, pp. 2017-2025, 2015.
Kevin G Jamieson and Robert D Nowak. Low-dimensional embedding using adaptively selected ordinal data. In 2 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1077-1084. IEEE, 2011,
Zequn Jie, Xiaodan Liang, Jiashi Feng, Xiaojie Jin, Wen Lu, and Shuicheng Yan. Tree-structured reinforcement learning for sequential object localization. In Advances in Neural Information Processing Systems, pp. 127-135, 2016.
10

---

Published as a conference paper at ICLR 2022
Ryan Julian, Benjamin Swanson, Gaurav S Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning.
In Conference on Robot Learning. 2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning, pp. 651-673. PMLR, 2018.
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8420-8429, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 13326 1338, 2015.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: A framework for attention- based permutation-invariee neural networks. In International Conference on Machine Learning, pp. 3744-3753 PMLR, 2019.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, pp. 740-755, 2014.
Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In Proceedings of the 27th International Conference on Neural Information Processing Kestems-Volumee 2, pp. 2204-220200 2212, 2014.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, pp. 1717-1724, 2014.
Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic. object localization for free? weakly- supervised learning with convolutional neural networks. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 685-694, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, (966):11371111 2016.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9939-9347, 2019.
Jake Snell. Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 4080-4090, 2017.
Hyun Oh Song, Ross Girshick, Stefanie Jegelka, Julien Mairal, Zaid Harchaoui, and Trevor Darrell. On learning to localize objects with minimal supervision. In International Conference on Machine Learning, pp. 1611-1619. PMLR, 2014.
11

---

Published as a conference paper at ICLR 2022
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. International Conference on Machine Learning, pp. 9229-9248 PMLR, 2020.
Kevin Tang, Armand Joulin, Li-Jia Li, and Li Fei-Fei. Co-localization in real-world images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1464-1471, 2014.
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Yoshikazu Terada and Ulrike Luxburg. Local ordinal embedding. In International Conference Machine Learning, pp. 847-855. PMLR, 2014.
Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Amoll WM Smeulders. Selective search for object recognition. International Journal of Computer Vision, 104 2013.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29:3630-3638, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-UCSD birds-200-2011 dataset, 2011.
Xin Wang, Thomas Huang, Joseph Gonzalez, Trevor Darrell, and Fisher Yu. Frustratingly simple few-shot object detection. In International Conference on Machine Learning, pp. 9919-9928, 2020.
Yu- Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Meta-learning to detect rare objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9925-9934, 2019.
Xiu-Shen Wei, Chen-Lin Zhang, Yao Li, Chen-Wei Xie, Jianxin Wu, Chunhua Shen, and Zhi- Hua Zhou. Deep descriptor transforming for image co-localization. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pp. 3048-30055 2017.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning. 8(3-4):229-25 1992.
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. DOTA: large-scale dataset for object detection in aerial images. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3974-3983, 2018.
Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang. Xiaodan Liang, and Liang Lin. Meta R-CNN: Towards general solver for instance-level low-shot learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9577-9586, 2019.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan R Salakhutdinov, and Alexander Smola. Deep sets. Advances in Neural Information Processing Systems, 2017.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2921-2929, 2016.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. CoRR, abs/2009.0788 078888, 2020.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX: IMPLEMENTATION DETAILS
A.1 ALGORITHM PIPELINE
General supervised training methods are usually class-agnostic and require exposure to large number of training classes, box-image pairs, and foreground and background variations in order to generalize well. In contrast, we allow specialized agent to trained, with the ability of adaptation to changes during the test time. Our approach is based on the feature similarity with query images, which departs from previous bounding-box regression and RL approaches based on objectiveness. Compared to general supervised training or fine-tuning methods, our approach is able to flexibly make use of various types of data in these phases. This summarized in Table 8.
Table 8: Breaking up the requirements on data and labels in training and adaptation
Supervised methods
Our approach
Training Fine-tuning
Ordinal emhedding
Agent training
Test-time adaptation
Image-box pairs
Unlabeled images X
X
X
Exemplar images
X
X
Our agent can both take human feedback in terms the exemplary set and perform test-time policy adaptation using unlabeled test data. It includes three stages: ordinal representation pre-training, RL agent training, and test-time adaptation. Details are as follows:
Stage 1: Pre-train ordinal embedding for state representation and reward In this stage, assume pairs of ground-truth bounding box and training image are available. We train the ordinal embedding by attaching a projection head after Rol encoder, as Figure shows. Rol encoder is composed of image encoder and a RolAlign layer (He et al., 2017). It extracts corresponding bounding box feature directly from image feature output by image encoder. After training, all the modules are fixed. The output of Rol Encoder then becomes the state for agent, and output of projection head is then used for reward computation.
On Corrupted MNIST (cMNIST) dataset, we attach decoder in order to train the encoder, but discard during inference. The loss function defined as
L = Freconstruct + a Liriplet:
(5)
where we set A 0.1, and Crecomstruct mean square error loss measure the reconstruction ability of the autoencoder. On CUB and COCO dataset, we adopt the pre-trained ImageNet encoder, thus no additional decoder is needed. The triplet loss Ltriplet is optimized, such that the embedding of positive bounding boxes are closer to the embedding of anchor than negative boxes. Different from traditional definition of triplet loss, our positive and negative boxes are not fixed. When comparing two perturbed boxes, the positive box is always the one with larger IoU. Thus, box with IoU=0.2 could be positive box if the negative box has even smaller IoU. The anchor box is not restricted to the mean vector of ordinal embeddings
ground truth box of the image instance. For example, in this paper, computed from ground-truth boxes of images from the exemplary set.
by b
Rol Encoder
Projection Head
g
b b
Images
Encoder
Rol Align
fc
Triplet (as
Ltriplet
Decoder
byi perturbed box, (o, by.)
Dei perturbed box, b,) v,
MSE Loss(img. (img)
a: average groundtruth embedding g
p: b,. embedding of the box with larger loU
n:: b,, embedding of the box with smaller IoU
free
Figure 5: Learning RoI Encoder and Projection Head.
13

---

Published as a conference paper at ICLR 2022
Table 9: Network and loss updating details for different stages of the method
Comiiguration
Training
Testing
Modules
Objective
Network
Exemplary set
Urdinal pre training
Folicy training
before adipeation
after osaptation
KG1 Encoder
NA
VGG-IVViT
Prozen
Froten
Frozen
Prozen
Head
Ordinal Eapht
MLP
Frozen
Centroller
Poi
Upeited
For cMNIST dalaset, ROI encoder trained under loss Eq For other datasets, direcily load the the thell pre trained network.
Stage 2:
Localization agent training In this stage, we assume the ground-truth bounding box available for each training image. All the models are trained and tested on the same datasets as stage 1. We fix all the layers in Rol encoder and projection head. The adopted RNN model is shown Figure In each step, the agent takes the Rol encoder output eai and the hidden state of RNN 1 concatenated the state representation. Then it predicts the action for next step by sampling according to the logits. The loss defined as
Lugent Lipolicy + X2 Lentropy
Centropy = - π log nT Cpolicy = - R) log
(6)
where Centropy prevents the agent from being stuck in the local minimum, is the policy, and R is the reward defined in Eq. 4.
Rol Encoder
Policy Net
image
encoder
align
Ral
COTVV
fc
(bs,
42,42)
(bs:
(be, 32, 7, ?)
frz
(bs, 48)
actions
(bs, 48)
(bs, 10)
Projection head
embedding
(ba.)12)
(bs,1024)
Figure 6: Policy network architecture.
Stage Test-time adaptation To adapt new class, we make use of a limited number of groundtruth-bor box cropped images in the test exemplary set. We leverage abundant unlabelled images and this exemplary set to update the agent. We fix the Rol encoder and projection head, and only update the agent. The process is almost the same as stage 2, except that in reward (Eq. 4) is the prototype of the test exemplary set.
In summary, the configuration of different model components and how they are updated is listed in Table The losses for each dataset and training stage are Table 10.
Table Summary of losses used on different datasets.
dataset
loss
cMNIST
L Creconstruct + A1 Liriplets A1 = 0.1
stage
CUB
L Liriptet
COCO
L Lrripter
cMNIST
Lagent policy + A2
Lentropy A2 6
stage 2
CUB
Lagent Epolicy + A2 Lentropy: x2 0..5
COCO
Lagent policy + X2 Lentropy: X2 0.5
cMNIST
Lagent Epolicy X2 Lentropy d2 0.5
stage
CUB
Lagent
pollic + A Lentropy- A2 0.5
COCO
Lagent Epolioy X2 Lentropy 0.5
14

---

Published as a conference paper at ICLR 2022
A.2 EXPERIMENTS DETAILS
For MNIST, we use three convolutional layers with ReLU activation after each layer as image encoder, while the same but mirrored structure decoder to learn an autoencoder, and then attach ROIAlign layer followed by two fully connected (fc) layers as projection head for ordinal reward learning. For the CUB and the COCO datasets, we adopt layers before conv5 3 of VGG-16 pre-trained on ImageNet as encoder unless otherwise specified. The projection head uses the same structure as before but with more units for each fc layer. All of our models were trained with the Adam optimizer (Kingma & Ba, 2015). We set margin m = 60 in all the experiments heuristically. All the models take less than one hour to finish training, implemented on PyTorch on single NVIDIA A100 GPU.
We list experiment details for the datasets used in Section 4 as follows.
Corrupted MNIST Dataset. Examples the four types of corrupted MNIST (cMNIST) images shown in Figure 7. In stage I and 2, we randomly sample 50 images of one class in original MNIST training set, and add noise as our training set. During testing, the models are evaluated on all the corrupted MNIST test set. In stage 3, to adapt of images as exemplary set for adaptation and use all the (corrupted) MNIST training set from that digit. We then the agent all (corrupted) MNIST test set of the same digit. There is overlap between the training and test set used in stage 3.
digit, annotate limited number
2
Random patch
Cluttered
Gaussian noise
Impulse noise
Figure 7: Corrupted MNIST Datasets: 28 x 28 digits on four kinds of 84 84 noisy background.
CUB Dataset. In stage and 2, we train on 15 warbler classes with class id between 158 and 172. There are 896 images in total. Then test the models on new warbler classes with class between 178 and 182, resulting in 294 images in total. In stage 3, the number and class ids of images for cach class are presented Table 11. also randomly seleet limited number of images as exemplary set and use all unlabled data for adaptation. The Cor Loc is calculated using all the images of this class.
Table 11: Number of images for training and testing in stage 3 on CUB dataset.
warbler
wren
sparrow
oriole
kingfisher
vireo
gull
cls id #images
[178, 182]
[193, 197]
[114, 119]
[95, 98]
[79, 83]
[151. 157]
[59, 64] 359
294
299
357
239
300
409
COCO Dataset. For the results of Figure 3, we train one of the five classes: cat, cow, dog, horse, zebra, then adapt to another four classes. The detailed number of each class for training and testing in stage 1 and 2 shown in Table 12.
Table 12: Number of images for training and testing in stage 2 COCO dataset.
cat
cow
dog
horse
zebra
elephant
giraffe
laptop
sheep
suitcase
train
3619
649
3701
1524
611
973
1146
2844
260
1402
test
167
27
148
68
21
32
40
163
18
In stage 3, the agent is tested new classes in target domain, within which we annotate limited number of images for adaptation. In comparison with few-shot object detection experiment, the models in stage and 2 are trained using one single class of the 5 classes: elephant, sheep, giraffe, laptop and suitcase. Then being adapted to the four classes in Table Note that the five classes are in the base classes, and the four classes are in the novel classes used in other methods. Thus, it's
15

---

Published as a conference paper at ICLR 2022
harder to transfer from one single class to another due to scarcity of training data and training class. Table 4 reports the average Cor from the five classes to each target class. We also provide the results of each source class in Table 13. From this table, we classes with target class usually performs better. For example, the Cor Loc from laptop and suitcase lower than other three animal classes, especially before adaptation. After adaptation, the gap becomes smaller.
that transferring from related
Table 13: CorLoo(% Adaptation comparison with other methods COCO dataset per class results.
before adapt
after adapt
cat
dog
Cow
horse
cat
dog
cow
horse
elephant giraffe
41.25
38.45
59.94
54.27
48,00
45.66
65.64
59.19
17.91
19.18
22.19
26.25
42.89
31.88
35.59
53.68
laptop
13.87
4.57
3.24
1.58
34.93
15.67
29.43
20.34
sheep
29.10
34.13
36.98
33.14
46.01
36.64
49.92
46.26
suitcase
12.99
6.43
11.71
4.99
37.99
37.56
43.14
25.98
Selective localization. We investigate the agent's ability localizing the object specified by the query set, when the set of images have two common objects. We random patched MNIST, where each image has digit? 3 and digit 4. First, the Rol encoder and projection head are trained with an additional contrastive loss to enlarge the distance between the two digits in emhedding space,
losSembed lossree Xerip losstrip
Acontr losScontr:
where 1osstrip = 1os8trips + losstripp- learning two local ordinal structure around each class center embedding space. We set the margin for both triplet losses as 10, and the margin for contrastive loss as 320 heuristically. We found that the larger the gap between the two margins, the better the performance. Detailed results can be found in Table and its related discussion. After learning the Rol encoder and projection head, we train the agent with the reward defined with Eq. 4 in Sect. 3.3, where с the prototype embedding of the targeted digit exemplary set (exemplary size is 5).
33
3
4
3
4
3
4
3
3
3
3
3
4
Ours
4
4
4
3
3
4
4
4
Exemplary Set
Outputs
DDT
Outputs
Figure 8: Selective localization vs. co-localization on two-digits data with random patch background.
Figure 8 shows an illustrative example of query object localization with task adaptation. Given two common objects coexisting in the image environment, the agent is trained to pick up one from the two during training, while quickly adapt to pick another one during testing. Different from unsupervised co-localization approaches (Wei et al., 2017) relying on low-level image clues to distinguish foreground and background, our learning objectives can be specified explicitly by switching the train/test exemplary sets. With test-time adaptation, agent can handle even partially conflicting learning objectives.
16

---

Published as a conference paper at ICLR 2022
B ADDITIONAL EXPERIMENTS
B.1 ABLATION OF ORDINAL EMBEDDING AND RL
Setting and main results on corrupted MNIST dataset. We analyze the effectiveness of using ordinal embedding in terms of representation and reward first on synthetic datasets of corrupted MNIST digits under cluttered, random patch, Gaussian, or impulse noises (see Figure 7 in Appendix for examples). We want to find the answers to two main questions listed below:
Is the embedding distance based reward signal as effective as the IoU counterpart?
2. Does the ordinal triplet loss benefit the state representation learning?
Accordingly, we consider baselines in which embeddings are trained with only autoencoder or jointly with the ordinal projection head. Besides, we also compare our embedding distance based reward against the IoU based reward used in Caicedo & Lazebnik (2015). Without loss of generality, the agent is trained on images of digit 4, and tested on images from all rest classes. In particular, interested in evaluating the sample efficiency of different approaches. The results under different training set sizes are shown in Figure 9. To distract the localization agent, cluttered noises with random 6 6 crops of digits added the background. All comparisons are based the policy gradient (PG) method. With ordinal embedding present in both the representation and reward Emmbedd, our model performance is consistently better than other settings, especially when the training set size is small. The benefit of triplet loss is demonstrated by the comparison between "AE+Ord+IoU" and "AE+IoU". Intuitively, the similarity a queried object is also more natural goal than objectives defined by bounding boxes IoU, which is used in both generic object detection or localization and previous RL-based approaches.
100
Cortoc (%) of other digits
95
CorLoc 5%6 of 4
95
90
85-
AE+IoU
80
AE+loU
Emeed
AE+Embed
80
75
AE+Ord
Em+erd+Embed
70
AE+Ord+Ember
50
100
300
600
50
100
300
600
Number of images in train set
Number images in
Figure 9: Comparison under different training set sizes. "AE+IoU": autoencoder, IoU based reward; "AE+Embed": autoencoder, embedding distance based reward; "AE+Ord+ooU"; autoencoder and ordinal projection head, IoU based reward; AE++rd++Ember autoencoder and ordinal projection head, embedding distance based reward. (a) Test performance on the trained digits (b) Average test performance on the other nine digits.
Signed reward or not. Another intriguing question is that other than benefiting state representation learning, whether the embedding- based reward itself can bring any additional benefits than the IoU- based reward. From Figure 9, there is a large gap between "AE+Ord+Embed" and "AE#Ord+IoU"" especially when the training set size is small. This is counter- inntutr-intuitiee as the ordinal reward is trained with supervision from IoU in embedding space, supposably it shall not bring any additional information. To further analyze this phenomenon, we take the sign operation off in Eq. I and retrain the agent. In an experiment with 50 training images, without sign operation, the localization accuracy increases from 90.92% to 94.36% on test digit 4, and from 82.43% to 88.64% on other digit test set. Perhaps, this improvement can be attributed to more informative feedback provided by the continuous-valued, unsigned reward.
On policy vs. off policy. Many deep RL approaches are in favor of using deep Q-Network (DQN) to train an agent. Different from Caicedo & Lazebnik (2015) and Jie et al. (2016), we apply Policy adopt top-down search strategy through a RNN, while
Gradient (PG) to optimize it. Besides, they used a vector of history actions to encode memory. We evaluate these design choices with four baselines, with "AE+IoU" setting, and trained the same 600 sampled cluttered digit 4 images. As
17

---

Published as a conference paper at ICLR 2022
Table 14 shows, the agent achieves the best performance with "PG+RNN". We find that empirically, with history action vectors the accuracy becomes worse when the agent is trained by DQN.
Table 14: Cor Lo)) DQN (with/without history action vector) vs. PG (with/without RNN)
Method
DQN
DQN+History
PG
PG+RNN
Digit 4
88.801.6
86.544.3 81.75a.4
88.982.9 81.912.7
94,68%.9
Other digits
84.212.0
89.051.7
imagenet rank
imagenet ard rank
Imagenet ard bafore
imagenet ard adapt
50
40
30
20
cow Sorraf
dagi Horto
zebit For de
cat: soctp
dogi Mocdy
herse Pcc
cow scat
torcet ood
crocodde calooddo
zebra dodg
car aettbb
como aezttra
forse zebra
dog
Figure 10: Loe %) comparison with ranking method using ImageNet pre-trained backbone.
More results on COCO dataset. Figure 10 provides results using ImageNet pre-trained VGG- 16 network backbone with the same training strategy as Figure 4. To further demonstrate the effectiveness of ordinal embedding, we compute the Spearman's Rho rank correlation between embedding distance to the prototype and IoU to the trounddtruth.. The results shown in Figure 11. Here we also add CLIP (Radford et al., 2021) pre-trained ViT as backbone for comparison. The rank correlation is smaller than -0.7 on all backbones with ordinal embedding, exhibiting ordinal embedding preserves the loU order, thus is better for the ranking purpose. Although pretty effective, embedding distance is still not perfect indicator of the ranking of IoU. Thus, directly formulate the object localization problem as search problem leads to suboptimal localization accuracy.
Table 15: Cor Loc()) compare backbones on source domain.
backbone
cow
cat
dog
horse
zebra
ImageNet pre-trained VGG-16
70.37
68,46
61.26
61.28
79.36
Faster RCNN pre-trained VGG-16
66.67
72.46
61.59
60.29
71.25
CLIP pre-traine -r-trrined VIT
74.07
82.64
70.95
76.47
80.95
B.2 COMPARE DIFFERENT OFF-THE-SHELF NETWORKS AS THE BACKBONE
It is interesting to study the choices of off-the-shelf pre-trained networks as the backbone, such as CLIP or supervised embedding provided by Faster RCNN or classification network. Since these networks have been exposed to large-scale dataset, it is interesttigg to see whether policy adaptation still needed. We compare different backbones on both source domain and target domain using our method. Table 15 reports the Cor Loc of training and testing on source domain. The large-scale pre-traind VIT backbone consistently performs the best, comparing to the other two VGG-16 models. Table 16 compares the backbones target domain with new classes. The test-time adaptation still brings a large margin of improvement. Interestingly, we also found that the Faster-RCNn embedding the target domain before adaptation, while the ViT network trained
offers the best performance on CLIP dataset provides the best performance after adaptation, indicating different generalization mechanisms. They both outperform the ImageNet backbone initially considered.
18

---

Published as a conference paper at ICLR 2022
cst
COW
-0.5
-0.55
0.60
0.5
0.75
ingnet.ord Ingnzt fuster ord
fister
vi,ood
vt
ingnet.ord Ingnet hostr ord fister
witoodd
vit
harse
-0.50
-0.55
-0.5
-0.60
0.65
-0.70
Enghat Engiat
Inghet Inghat
zobra
-0.35
-0.05
00.0
-0.75
00.00
and
fealer
Tester
Figure 11: Rank correlations between embedding distance and IoU: using different embedding functions such as ImageNet and Faster RCNN, with or without the ordinal pre-training, and ViT.
Table 16: Coc compare backbones on target domain.
before adapt
after adapt
ImageNet VGG-16
Faster RCNN
CLIP VIT
ImageNet VGG-16
Faster RCNN
CLIP VIT
VGG-16
VGG-16
cat->horse
33.32
35.50
18.42
51.89
47.64
56.41
->horse
48.41
54.55
53.67
46.80
59.61
63.06
dog >Shorse
41.50
46.48
15.70
55.89
56.83
58.62
zebra >hhosse
10.29
16,86
6.74
39.22
34.19
46.39
cat ->cow
50.85
42.99
36.26
58,58
53.26
55.52
dog >>cow
54.63
58.65
43.50
62.86
64.15
58.76
Horse ->onw
59.52
61.32
52.54
64.83
65.23
68.16
Zehra ->cow
11.92
7.19
35,56
38.26
52.65
cow ->eat
39.84
47.39
38.79
46.42
51.15
61.67
dog >>eat
57.97
63.84
66.60
63.12
65.18
76.83
horse >>eat
44.25
47.67
27.80
53.39
52.96
63.87
zebra >>eat
10.45
17.67
2.47
35.73
31.40
49.12
cat->dog
47.81
45.61
49.69
48.94
49.83
61.75
cow >dog
37.28
37.64
30.13
42.33
37.10
50,94
horse >>dog
39.07
40.76
23.89
44.77
40.69
55.68
zebra- ->dog
7.74
11.83
2.88
17.73
30.64
36.48
cat ->zebra
57.58
15.82
22.59
70.28
45.83
69.39
cow >>zebra
39.64
60.55
37.75
65.80
64.21
72.18
dog >>zebra
35.27
18.25
15.33
63,91
58.16
67.59
horse >>zebra
66.82
56.63
61.37
72.83
68.74
75.01
19

---

Published as a conference paper at ICLR 2022
B.3 THE EFFECTS OF MARGIN
The margin in triplet loss is selected heuristically. is not sensitive except in the selective localization experiment (Figure 8), where there are two different digits each image. For this experiment, we trained two ordinal structures around each digit using triplet loss with margin m.. and add additional contrastive loss with margin my to separate the centers of the two different digits as far as possible. And we found out that the model works best when m2 > In our experiment, we set my = 10, m2 = 320. The results of using different set of m and m2 are presented Table 17.
Table 17: Results of different margin configuration in selective localization.
m1
10
10
10
10
10
m2
60
70
80
160
320
Loc())
86.54
87.92
88.32
91.39
98.52
B.4 SIZE OF TRAINING SET
In this experiment, we train on class giraffe in stage and 2, then adapt cat, cow, dog and horse. We set the training set size as [200, 500, 700, 1146], exemplary set size as 5. We compare our after adaptation results with TFA w/ fc (Wang al., 2020) one-way 5-shot setting, where their model trained on all 60 base classes, while ours is only trained on one of the base classes. Figure 12 shows results, in which the dotted lines are results of TFA w/ fc. Ours performs much better than their method. Except on cat class, ours is better than theirs with only 200 images for training.
60
ours cat
ours cow
dog
ours horse
40
TFA w/ fo dog
Corto
TFA fo cow
20
0
200
400
600
800
1000
train set size of giraffe
Figure 12: Results using different training set size in stage 1, 2.
B.5 SIZE OF EXEMPLARY SET
We also compare the effect of different size of exemplary set during training and adaptation on dataset. During training stage, we use shuffle proto training strategy, and set exemplary set size 2, 5, 15, 25. The results without adaptation on test set are in Table 18. Both OrdAcc and Cor Loc increase with exemplary set size. For adaptation stage, the range exemplary set size is from 2 to 200. And the results are in Table 19. The test performance does not increase much with the exemplary set size. One possible explanation is that the data points embedding space are compact, thus prototype doesn't 't change much when increasing exemplary set size. We will analyze the influence of multiple prototypes per class in future experiments.
B.6 PROTOTYPE SELECTION
We further evaluate the choice of anchor in the triplet loss for both the pre-training of state representa- tion and the ordinal reward for the training of agent. We study (i) whether ordinal embedding can be
20

---

Published as a conference paper at ICLR 2022
Table 18: Effect of exemplary set size during training stage.
Size
OrdAcc(%)
Cor CocLo)(%)
2
94.39+17
84,18
5
94.83
88. 18.10
15
95.69
89. 1..
93.82+100
89.6422.3
Table 19: Effeet of exemplary set size during adaptation.
Size
2
5
50
100
150
200
(o)
89.12
89.67
90.15
90.36
89. 19.6344.2
90. 40.1
trained in reference to an anchor from different image instances, (it) whether it is advantageous to use the prototype embedding of exemplary set, rather than instance embeddings, and (tii) whether mimicking the test condition in training yields any improvement.
Table 20: Anchor choice comparison. Note that evaluations the OrdAcc is always computed using instance as anchor.
Mode
OrdAco(%)
CorLoc(%)
Self
97. 97.210.7
61.022.0
Proto
95.2+1.a
77.9
Shuffle self
92. 12.4
73.8
Shuffle proto
96.2+1.5
88.1
We use the MNIST dataset. The training and test set contains 15 and 5 disjoint fine-grained classes respectively, resulting 896 images for training (viewed as a single class) and 294 for testing. Table 20 shows the OrdAcc and Cor Loc in four settings. "Self" uses the embedding from images cropped by the ground-truth box from the same instance; "Shuffle self" uses the ground-truth box cropped image emebedding from different instance; Similarly. "Proto" uses the prototype of a subgroup containing the training instance within the same batch; "Shuffle proto (SP)" uses the prototype of subgroup from a different batch without the training instance. Results suggest that this training strategy brings compactness to the training set, constructing proto", while the OrdAce is lower than others, its CorLoe the best with large margin. Matching the condition between training and testing indeed improves generalization to new classes on this dataset.
dataset with more foreground background variations than the corrupted
ordinal structure around the cluster. For "Shuffle
B.7 ANNOTATION REFINEMENT
Annotation collection plays important role in building machine learning systems. It task that could benefit greatly from automation, especially in cost-sensitive applications. The human labeling efforts can be reduced, only terms of the number of annotate samples per class, number of annotate classes (one-class at the minimum), but also the level of accuracy required. We explore the direction of iterative refinement of annotation quality with minimal human guidance. Given an agent trained extensively with inexact human annotations, can we adapt it to refine the annotation with few cell-annotated examples? We purposefully enlarge the ground truth bounding boxes in the corrupted MNIST and CUB datasets, and adapt the trained agent using the original ground truth box an exemplary set of size 5.
For corrupted MNIST dataset, the agent, Rol encoder and projection head are first trained with 50 digit 3 images with loose (38x38) bounding box annotation. Then the agent adapted to digit I images, where the exemplary size=5, and the ground-truth boxes are 28 x8. For CUB dataset, during training, the ground truth box in each resized 224x224 image is expanded 10 pixels each side. During adaptation we use the original ground-truth boxes. In adaptation from entire body to head experiment on CUB dataset, we train on original entire body ground -trtth bounding box. During adaptation, 15 randomly sampled exemplar images annotated with head bounding box. The agent
21

---

Published as a conference paper at ICLR 2022
trained to adapt localize head by minimizing the distance between embedding of predicted box and prototype of head embeddings. The results are shown in Figure 13 (left panel). Preliminary results with agent trained on the entire bird body and adapt to bird head are shown in Figure 13 (right panel), with a few successful cases on the first row and failure cases on the second row.
100
100000
before sdepation
efter adepation
Cortoc 190
89
$3.57
NNST
CUB
Figure 13: Annotation refinement. Left: Loose to tight transfer. Right: entire body to head transfer.
B.8 FEW-SHOT LOCALIZATION ON OMNIGLOT
The proposed ordinal reward signal also makes our approach amenable to few-shot training, when only a small subset training images per class are annotated. Different from the transfer learning setting, in few-shot setting limited annotations across multiple classes available during training. The ordinal reward can be viewed as meta information. We evaluate our method under few-shot setting corrupted Omniglot dataset (Lake et al., 2015) and CUB-warbler dataset. For Omniglot, We put each 28 28 character in 84 84 random patch background. The train and test set contains 25 different classes respectively, thus 500 images for each set. We randomly sample 100 iterations for training and testing. For CUB-warblee datset, as we did in Sect. 4.2 we train on the same 15 species from the "Warbler" class, and adapted to 5 new species of "Warbler", thus 896 and 294 images respectively. We randomly sample 100 and 50 iterations for training and testing. We use 5-shot 5 5-way, set exemplary set size 5, and use proto training strategy for both dataset. The results are shown in Table 21. As an implicit meta learning method, our approach achieves 99.9 99.94% and 90.52% Cor on two datasets. We can also leverage explicit meta learning method, such as MAML. (Finn et al., 2017) to further improve the results. We will leave this part future work. Although initial results are promising, more efforts are nceded to validate whether the proposed RL approach can achieve state-of-the-al performance, but it is beyond the scope of this work.
Table 21: Evaluation under few-shot localization setting.
Dataset
OrdAce(%)
Coc(o)())
Omniglot
95.1 15,11
99.9440.1
CUB-warbles
91.28
90.52+0.7
22

---

Published as a conference paper at ICLR 2022
C INTERMEDIATE RESULTS
The agent tends to predict expected actions to transform the box so that it can finally get to the target location. It also be observed from Figure 14 that the embedding distance between predicted boxes and ground-truth boxes is in agreement with the corresponding IoU. This further demonstrates the effectiveness of the localization agent.
y
Figure 14: Starting from the whole image, the agent is able to localize digit 4 within 10 steps. Each row shows localizing one image, and each column is time step. Green box means ground-truth bounding box.
mmg1
embedaing dctance gt
1000
ing
mg2
0.8
img2
mm33
img5
0.6
800
200
Figure 15: The IoU relation between ground-truth and predicted box is accurately represented in embedding space. Left: the embedding distance between ground-truth and predicted box in each step. Right: the loU between ground-truth truth and predicted box in each step.
23


Reviews:
[{"review_id": "mNhFzJe93pz", "reviewer": "Reviewer_DiRh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper formulates the query object localization problem as an RL task. The task itself has many applications in other fields such as robotic manipulation. Prior work directly uses IoU as the reward measure for training localization agents while this paper proposes to first learn an embedding space that respects the IoU-based ordering and then directly use the distance between embedded features as the reward signal for training the localization agent, i.e. the more similar a state is with the query image in the embedding space, the higher the reward. Experiments on several query object localization tasks in MNIST, CUB and COCO datasets demonstrate the proposed method’s ability to transfer learned rewards from one task to another.\n\nOne question I have is on the evaluation protocol. The authors report the percentage of correct localization as the evaluation protocol (which counts a localization as correct if IoU is greater than 0.5). Why not use standardized measures such as mean IoU or mean average precision? How does CorLoc compare with mAP iou=0.5? What is the advantage of using CorLoc?\n\nThe authors mention that the method can be easily modified to deal with multiple queries objects or no queried object in the image, it does not seem to be trivial for readers that are not familiar with this task setting and it would be great if the authors could elaborate on this.\n\nThe baseline methods compared in Table 6 are not clearly explained and it can be hard to see whether the comparison is fair/meaningful. It would help a lot for understanding the results if the authors could discuss the differences between the baseline methods and the proposed algorithm under test.\n\nOne interesting baseline/ablation I think could be added here is to use an off-the-shelf pre-trained network as the embedding model/RoI encoder. Large-scale models such as CLIP could be tested here to see if these large-scale models offer better representation for transfer than task-specific ones learned in the proposed way. \n\n", "summary_of_the_paper": "This paper presents a new metric learning method on the RL formulation of query object localization. The metric learning method is based on a contrastive learning formulation of ordinal embeddings, which are pretrained with data augmentations and a loss formulation that respects IoU orderings. The experiment results have demonstrated that the proposed embedding metric increased the task performance compared with the baseline metric (IoU), and shown better generalization behavior on novel image categories.\n", "summary_of_the_review": "This paper presents a novel method for solving query object localization as an RL task. The authors propose to learn an embedding space that respects the IoU ordering of bounding boxes and then use the distance in the embedding space as reward signal instead of raw IoU scores. The writing and presentation of the paper is clear. However, it is not easy to understand the similarities and differences between the proposed method and the baselines it compared with in the transferring task. It would better help the audience understand the significance of the results if the baseline methods are explained with more details. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "h8PMwPoD_lb", "reviewer": "Reviewer_tkLp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The idea of using a transferable reward is interesting and novel. But as explained in the weakness section, the design of an RL agent for object localization is not empirically justified.\n- Experiments on few-shot object detection show good improvements over recent methods (e.g. TFA).\n- The paper presents some good investigation on the design of the loss function for the ordinal embedding network (Figure 4). In addition, various RL algorithms have been tested.\n\nWeaknesses:\n- The paper has not studied the separate effect of the RL component and the ordinal embedding component. Since during ordinal embedding training the model has access to a pool of bounding box labels, the authors could use the same data to train a class agnostic bounding box proposal generator (like the first stage in Faster-RCNN), or it could leverage selective search bounding boxes. Then, with the ordinal embedding module, it can choose the most similar region in the ordinal embedding space as the final box prediction. Such a pipeline removes the complicated procedures of RL and sequential box refining, but preserves the ordinal embedding component, and should be studied as a baseline to corroborate the claim that a transferable “reward” is required for such a task. I ask this because the problem structure is static and the state will not change based on the action taken by the agent, and instead of framing it as RL, people can view it simply as a search problem.\n- In addition to the baseline above, the ordinal embedding component can be replaced by a supervised embedding provided by Faster-RCNN or simply a pretrained classification network using the labeled portion of the data. These baselines will separate the effect of RL and ordinal embedding in a 2x2 experiment matrix.\n- Another concern with the current RL agent setup is that it is only configured to output a single object, and cannot handle if the image contains multiple objects. This is a limitation compared to more widely adopted settings of object detection. \n- As a result of the simplified setup, the paper uses the CorLoc metric to measure the percentage of images with IoU > 0.5, and it is not the same metric used in few-shot object detection (which uses mAP over different splits). The discrepancy in the evaluation protocol may make it difficult for follow-up works to compare to this work.\n- Section 3.3 should be named “ordinal embedding network architecture” instead of “model architecture” since it only describes the network that provides the ordinal embedding.\n- In the COCO few-shot detection experiment, although it is encouraging to see that the proposed model only needs to learn from one class, the setup seems a bit non-standard and not directly comparable to few-shot learning baselines (it’s making the task harder for the proposed method). It would be good to investigate the amount of training data needed, by testing on a varying number of training classes. Perhaps this can show that the proposed transferable reward is more data efficient and more generalizable.", "summary_of_the_paper": "This paper proposes a new method for the task of query object localization. It learns an embedding network such that the image crop that is closer to the query object will have a closer embedding distance as well, and the improvement of the embedding distance will be used as the reward function. The RL agent is then employed to maximize the reward. The paper found that the proposed approach compares favorably to DDT on object location on CUB and FTA on few-shot object detection on COCO.", "summary_of_the_review": "The paper presents an interesting idea for object localization using RL. My main concern is that the current experimental designs are weak, because of 1) less sufficient studies on the effect of individual components, and 2) non-standard object detection protocol. Therefore my initial rating is “weak reject”.\n\nAfter seeing the authors' response, I decided to increase my rating from 5 to 6 since the added experiments addressed my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CyXk7XNvv2-", "reviewer": "Reviewer_pHvg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n* reinforcement learning approaches are interesting in this domain\n* results seem promising\n* experiments thorough and give some depth to the methodology\n\nweaknesses:\n* the paper is overly complicated and lacks clarity on many details.\n\nobservations:\n* I cannot understand what figure 1 is meant to show. Is it the computation ? I find that very unclear. I find it insufficient to give a clear overview of the method (which i assume is the goal).\n* sec. 3.1 needs a rewrite. The actions are unclear, the role of the reward should be clearly stated, the environment is not an image it's a set of states and transitions etc. \n* sec. 3.2 it is unclear what box perturbation means. Are the features perturbed or just the crop ? \n* \\rho in section 3.2 is explained in section 3.3 What is m in eq 3 ? What is an anchor in this context ? \n* in algo 1 the entropy is not added so the equation is not correct", "summary_of_the_paper": "The paper proposes a reinforcement learning approach for localization in images. The idea is to consider and environment where the states are crops of an image, the actions are changes in the current coordinates of the current crops starting at the full image. The method consists of learning both a reward function and a policy that maximizes that reward function. The policy is learnt with REINFORCE with entropy regularisation. The manner in which the reward function is formulated and learnt is quite appealing. To allow for more flexibility at test time, the reward function is made conditional on some exemplars and learnt in a few shot manner based on state features. Using a triplet loss to ensure the reward function is increasing as the overlap is increasing is quite good.\n\nThe experiments are quite extensive and I find compelling for the most part. Ablation of the elements of the method is well executed and benefits of few short learning are compelling enough in my opinion. The results on COCO are a bit limited but I would be willing to overlook that.\n\n\n", "summary_of_the_review": "The paper has enough novelty and importance but the rewriting needs a lot of work to make more precise and cohesive. With a solid improvement to clarity I would support acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4pk6567Ts-O", "reviewer": "Reviewer_yBs4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found this work interesting (especially as I wasn’t aware of RL-based object localization), and it covers a rather specific problem setting well.\n\nIt is quite hard to find details of decisions and specific model choices however, so perhaps the authors can improve on that and clarify some of these points:\n1. The early sections provide a good overview of the problem setting and of the particular way this will be addressed. However, some details of exactly what modules are trained, with which losses, under which data distribution are lacking. I also found myself confused when understanding exactly how the policy was set up and what it was receiving.\n   1. Figure 2 is clear, but Algorithm 1 does not reuse the same modules clearly. Figure 9 in the Appendix was also quite helpful, but the ROI encoder outputs 3 vectors in Figure 2, hence it isn’t directly clear what is happening. I understand you probably just use the Encoder with a RoIAlign using the changing bounding box, but this should be clarified.\n   2. What is trained and frozen in the different tasks? For example in Figure 4, was the Encoder/L_triplet trained on just digits 4 or everything?\n   3. Network and loss details are hidden in the text in random places, could you add a clear table in the Appendix for all of these?\n   4. There is no mention of how the margin `m` was chosen.\n2. As a more meta-discussion point: why is it beneficial to treat this as a RL problem, instead of “just” outputting the whole sequence of actions in one go?\n   1. i.e. consider using a sequential action model which just spits out a variable number of bounding box modification actions, and treat the problem as a bandit setup. Could you discuss and contrast why that isn’t appropriate?\n3. The way that perturbed bounding boxes (one good, one bad) are sampled for a given image wasn’t extremely easy to find (it is mentioned without details in 3.2, then again in 4.1, with perhaps alternatives considered per dataset if I understood the Appendix well).\n   1. It might be useful to spell this out clearly in the Appendix.\n   2. I would expect this to have a rather strong effect, did you find it to be sensitive?\n4. Figure 4 uses 50 images in the training set as the smallest number.\n   1. Was the encoder trained on 50 images as well?\n   2. Did you try using fewer? It would make a stronger point if your method could still cope well.\n   3. How dependent is this on the size of the examplar set?\n5. Did you explore other ways to use the exemplar set, instead of just averaging them into a prototype?\n   1. For example, Transformers or other set-invariant modules?\n6. The “signed reward or not” section is not the clearest. There is a simple explanation for why “removing the sign operation” is helpful, which is that it gives continuous (instead of binary -1/+1) rewards at every steps, which makes the overall RL problem easier as the rewards provided are denser and more informative?\n7. What is the loss_embed presented in Appendix A.2.1 and is it used for any results apart from the selective localisation one? It is more complicated and less general than the loss present in the main text (e.g. it is aware that only 2 digits will ever be present in an image).\n", "summary_of_the_paper": "This paper extends works performing RL-based object localization by:\n1. Conditioning the localization on a exemplary set of images, instead of more classical hardcoded finite set of classes\n2. Introducing a new reward signal which doesn’t explicitly use IoU but instead enforces a rank-preserving metric space (where distances reflect how well 2 bounding boxes match up).\n\nThis is shown on multi-MNIST data, CUB and COCO. They are targeting rather challenging generalisation setups, where a policy is only trained to localize 1 digit class (in the MNIST case say) and extrapolates to all other ones.\n", "summary_of_the_review": "Overall, I find this work relevant and well executed, although some details of how novel it is and details of its implementation might deserve to be improved. Results are good on toy domains and acceptable on COCO, but it is unclear how well it would cope with more complex situations. For now, I’d qualify this as borderline and tend to accept given it was an interesting read, but I am not extremely familiar with the literature and related work.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a reinforcement learning based approach for object localization given an exemplary set of images.  The paper shows that test-time policy adaptation to new environments is possible with object detection experiments. All four reviewers find the proposed approach interesting. Most of the reviewers feel their initial concerns (including the clarity of the paper and the approach details) addressed after the discussion with the authors and the revision. One reviewer still finds the experiments limited even after the author rebuttal, but the reviewers all agree that there is value in the paper.

We recommend accepting the paper.
Decision: Accept (Poster)