Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper extends works performing RL-based object localization by:
1. Conditioning the localization on a exemplary set of images, instead of more classical hardcoded finite set of classes
2. Introducing a new reward signal which doesn’t explicitly use IoU but instead enforces a rank-preserving metric space (where distances reflect how well 2 bounding boxes match up).

This is shown on multi-MNIST data, CUB and COCO. They are targeting rather challenging generalisation setups, where a policy is only trained to localize 1 digit class (in the MNIST case say) and extrapolates to all other ones.

main_review: Overall, I found this work interesting (especially as I wasn’t aware of RL-based object localization), and it covers a rather specific problem setting well.

It is quite hard to find details of decisions and specific model choices however, so perhaps the authors can improve on that and clarify some of these points:
1. The early sections provide a good overview of the problem setting and of the particular way this will be addressed. However, some details of exactly what modules are trained, with which losses, under which data distribution are lacking. I also found myself confused when understanding exactly how the policy was set up and what it was receiving.
   1. Figure 2 is clear, but Algorithm 1 does not reuse the same modules clearly. Figure 9 in the Appendix was also quite helpful, but the ROI encoder outputs 3 vectors in Figure 2, hence it isn’t directly clear what is happening. I understand you probably just use the Encoder with a RoIAlign using the changing bounding box, but this should be clarified.
   2. What is trained and frozen in the different tasks? For example in Figure 4, was the Encoder/L_triplet trained on just digits 4 or everything?
   3. Network and loss details are hidden in the text in random places, could you add a clear table in the Appendix for all of these?
   4. There is no mention of how the margin `m` was chosen.
2. As a more meta-discussion point: why is it beneficial to treat this as a RL problem, instead of “just” outputting the whole sequence of actions in one go?
   1. i.e. consider using a sequential action model which just spits out a variable number of bounding box modification actions, and treat the problem as a bandit setup. Could you discuss and contrast why that isn’t appropriate?
3. The way that perturbed bounding boxes (one good, one bad) are sampled for a given image wasn’t extremely easy to find (it is mentioned without details in 3.2, then again in 4.1, with perhaps alternatives considered per dataset if I understood the Appendix well).
   1. It might be useful to spell this out clearly in the Appendix.
   2. I would expect this to have a rather strong effect, did you find it to be sensitive?
4. Figure 4 uses 50 images in the training set as the smallest number.
   1. Was the encoder trained on 50 images as well?
   2. Did you try using fewer? It would make a stronger point if your method could still cope well.
   3. How dependent is this on the size of the examplar set?
5. Did you explore other ways to use the exemplar set, instead of just averaging them into a prototype?
   1. For example, Transformers or other set-invariant modules?
6. The “signed reward or not” section is not the clearest. There is a simple explanation for why “removing the sign operation” is helpful, which is that it gives continuous (instead of binary -1/+1) rewards at every steps, which makes the overall RL problem easier as the rewards provided are denser and more informative?
7. What is the loss_embed presented in Appendix A.2.1 and is it used for any results apart from the selective localisation one? It is more complicated and less general than the loss present in the main text (e.g. it is aware that only 2 digits will ever be present in an image).

summary_of_the_review: Overall, I find this work relevant and well executed, although some details of how novel it is and details of its implementation might deserve to be improved. Results are good on toy domains and acceptable on COCO, but it is unclear how well it would cope with more complex situations. For now, I’d qualify this as borderline and tend to accept given it was an interesting read, but I am not extremely familiar with the literature and related work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method for the task of query object localization. It learns an embedding network such that the image crop that is closer to the query object will have a closer embedding distance as well, and the improvement of the embedding distance will be used as the reward function. The RL agent is then employed to maximize the reward. The paper found that the proposed approach compares favorably to DDT on object location on CUB and FTA on few-shot object detection on COCO.
main_review: Strengths:
- The idea of using a transferable reward is interesting and novel. But as explained in the weakness section, the design of an RL agent for object localization is not empirically justified.
- Experiments on few-shot object detection show good improvements over recent methods (e.g. TFA).
- The paper presents some good investigation on the design of the loss function for the ordinal embedding network (Figure 4). In addition, various RL algorithms have been tested.

Weaknesses:
- The paper has not studied the separate effect of the RL component and the ordinal embedding component. Since during ordinal embedding training the model has access to a pool of bounding box labels, the authors could use the same data to train a class agnostic bounding box proposal generator (like the first stage in Faster-RCNN), or it could leverage selective search bounding boxes. Then, with the ordinal embedding module, it can choose the most similar region in the ordinal embedding space as the final box prediction. Such a pipeline removes the complicated procedures of RL and sequential box refining, but preserves the ordinal embedding component, and should be studied as a baseline to corroborate the claim that a transferable “reward” is required for such a task. I ask this because the problem structure is static and the state will not change based on the action taken by the agent, and instead of framing it as RL, people can view it simply as a search problem.
- In addition to the baseline above, the ordinal embedding component can be replaced by a supervised embedding provided by Faster-RCNN or simply a pretrained classification network using the labeled portion of the data. These baselines will separate the effect of RL and ordinal embedding in a 2x2 experiment matrix.
- Another concern with the current RL agent setup is that it is only configured to output a single object, and cannot handle if the image contains multiple objects. This is a limitation compared to more widely adopted settings of object detection. 
- As a result of the simplified setup, the paper uses the CorLoc metric to measure the percentage of images with IoU > 0.5, and it is not the same metric used in few-shot object detection (which uses mAP over different splits). The discrepancy in the evaluation protocol may make it difficult for follow-up works to compare to this work.
- Section 3.3 should be named “ordinal embedding network architecture” instead of “model architecture” since it only describes the network that provides the ordinal embedding.
- In the COCO few-shot detection experiment, although it is encouraging to see that the proposed model only needs to learn from one class, the setup seems a bit non-standard and not directly comparable to few-shot learning baselines (it’s making the task harder for the proposed method). It would be good to investigate the amount of training data needed, by testing on a varying number of training classes. Perhaps this can show that the proposed transferable reward is more data efficient and more generalizable.
summary_of_the_review: The paper presents an interesting idea for object localization using RL. My main concern is that the current experimental designs are weak, because of 1) less sufficient studies on the effect of individual components, and 2) non-standard object detection protocol. Therefore my initial rating is “weak reject”.

After seeing the authors' response, I decided to increase my rating from 5 to 6 since the added experiments addressed my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a reinforcement learning approach for localization in images. The idea is to consider and environment where the states are crops of an image, the actions are changes in the current coordinates of the current crops starting at the full image. The method consists of learning both a reward function and a policy that maximizes that reward function. The policy is learnt with REINFORCE with entropy regularisation. The manner in which the reward function is formulated and learnt is quite appealing. To allow for more flexibility at test time, the reward function is made conditional on some exemplars and learnt in a few shot manner based on state features. Using a triplet loss to ensure the reward function is increasing as the overlap is increasing is quite good.

The experiments are quite extensive and I find compelling for the most part. Ablation of the elements of the method is well executed and benefits of few short learning are compelling enough in my opinion. The results on COCO are a bit limited but I would be willing to overlook that.



main_review: strengths:
* reinforcement learning approaches are interesting in this domain
* results seem promising
* experiments thorough and give some depth to the methodology

weaknesses:
* the paper is overly complicated and lacks clarity on many details.

observations:
* I cannot understand what figure 1 is meant to show. Is it the computation ? I find that very unclear. I find it insufficient to give a clear overview of the method (which i assume is the goal).
* sec. 3.1 needs a rewrite. The actions are unclear, the role of the reward should be clearly stated, the environment is not an image it's a set of states and transitions etc. 
* sec. 3.2 it is unclear what box perturbation means. Are the features perturbed or just the crop ? 
* \rho in section 3.2 is explained in section 3.3 What is m in eq 3 ? What is an anchor in this context ? 
* in algo 1 the entropy is not added so the equation is not correct
summary_of_the_review: The paper has enough novelty and importance but the rewriting needs a lot of work to make more precise and cohesive. With a solid improvement to clarity I would support acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new metric learning method on the RL formulation of query object localization. The metric learning method is based on a contrastive learning formulation of ordinal embeddings, which are pretrained with data augmentations and a loss formulation that respects IoU orderings. The experiment results have demonstrated that the proposed embedding metric increased the task performance compared with the baseline metric (IoU), and shown better generalization behavior on novel image categories.

main_review: 
This paper formulates the query object localization problem as an RL task. The task itself has many applications in other fields such as robotic manipulation. Prior work directly uses IoU as the reward measure for training localization agents while this paper proposes to first learn an embedding space that respects the IoU-based ordering and then directly use the distance between embedded features as the reward signal for training the localization agent, i.e. the more similar a state is with the query image in the embedding space, the higher the reward. Experiments on several query object localization tasks in MNIST, CUB and COCO datasets demonstrate the proposed method’s ability to transfer learned rewards from one task to another.

One question I have is on the evaluation protocol. The authors report the percentage of correct localization as the evaluation protocol (which counts a localization as correct if IoU is greater than 0.5). Why not use standardized measures such as mean IoU or mean average precision? How does CorLoc compare with mAP iou=0.5? What is the advantage of using CorLoc?

The authors mention that the method can be easily modified to deal with multiple queries objects or no queried object in the image, it does not seem to be trivial for readers that are not familiar with this task setting and it would be great if the authors could elaborate on this.

The baseline methods compared in Table 6 are not clearly explained and it can be hard to see whether the comparison is fair/meaningful. It would help a lot for understanding the results if the authors could discuss the differences between the baseline methods and the proposed algorithm under test.

One interesting baseline/ablation I think could be added here is to use an off-the-shelf pre-trained network as the embedding model/RoI encoder. Large-scale models such as CLIP could be tested here to see if these large-scale models offer better representation for transfer than task-specific ones learned in the proposed way. 


summary_of_the_review: This paper presents a novel method for solving query object localization as an RL task. The authors propose to learn an embedding space that respects the IoU ordering of bounding boxes and then use the distance in the embedding space as reward signal instead of raw IoU scores. The writing and presentation of the paper is clear. However, it is not easy to understand the similarities and differences between the proposed method and the baselines it compared with in the transferring task. It would better help the audience understand the significance of the results if the baseline methods are explained with more details.
review_title: null

Meta-review: The paper presents a reinforcement learning based approach for object localization given an exemplary set of images.  The paper shows that test-time policy adaptation to new environments is possible with object detection experiments. All four reviewers find the proposed approach interesting. Most of the reviewers feel their initial concerns (including the clarity of the paper and the approach details) addressed after the discussion with the authors and the revision. One reviewer still finds the experiments limited even after the author rebuttal, but the reviewers all agree that there is value in the paper.

We recommend accepting the paper.
Decision: Accept (Poster)