Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper describes a technique for estimating a loss for object detection that is well-aligned with the AP loss used at evaluation time. The work builds on a prior formulation of the AP loss as a function on classification scores, but converts the formulation to account for per-bounding box results, and also replaces the Heaviside function with parametrized, searchable functions. Experiments show the gain from the proposed technique using three backbone detectors, and compares to different prior losses.

Post-rebuttal: The rebuttal addressed my concerns.
main_review: 1. The work is well-written and sound. All steps are well-explained and justified. The results are strong and extensive, including detailed ablations that verify the contribution of each method component. 

2. The choice of baseline losses seems reasonable, and techniques from each family described in Sec. 2 are included in the experiments, except for the "Direct Optimization for the AP Metric" family-- why are results from one of those methods not included? 

3. Some statements are subjective, e.g. L91-92. Others are a bit unclear, i.e. L97-98, 104-105. Some text on the other hand is repetitive, e.g. L168-170 have been previously stated multiple times.

4. The authors explain why they use different theta parameters for different parts of Eq. 6, but one choice especially stands out, e.g. the use of theta_5 as opposed to theta_1 for different occurrences of the exact same l(b_i). How important is this experimentally?

5. A few places in the text indicate the approach relies a bit too much on experimental settings, e.g. L179. In general, using a searchable, tunable metric is great, but is there a risk of overfitting? 

6. In general, the proposed approach works well, but is there any cost to pay? Does some part of the method's workings get slower or less stable compared to baseline losses?

7. Minor questions:
- Eq. 7, what is the value of M, i.e. how computed?
- L215, which exact equation is used? 
limitations_and_societal_impact: Discussed
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors propose a loss for object detection, that is a relaxation of mAP suitable for use in gradient ascent. Where AP includes Heaviside (step) functions, the authors replace these by learnt piecewise-linear functions. These functions are trained to maximise AP on a held-out set achieved by a detection model that was itself trained using them as the loss (so they do not aim to approximate AP closely, but to derive a surrogate loss that is 'structurally similar' to AP and achieves good training performance). Experiments on COCO 2017 show accuracy improvements over baselines (trained with focal/XE + regression losses) and other techniques approximating AP.
main_review: Strengths:

The proposed method is reasonably well-motivated, in terms of limitations of existing methods that do not optimise the surrogate itself to be 'good' as a training loss.

The precise structure of the proposed surrogate loss is novel; also I don't know of any existing methods for object detection that aim to tune the surrogate.

Empirical results indicate some improvement over earlier methods; these gains are consistent across various different detection architectures. There is also a detailed ablation study. There is plenty of detail on the experimental details and training protocols.


Concerns / suggestions:

There are no error bars / standard deviations on the main experiments. This makes it difficult to judge how much better the proposed method really is.

By replacing the Heaviside functions with an arbitrary monotonic function, the loss ceases to be AP; rather, it becomes a proxy loss chosen to yield high test-set AP. But given this, there is not really any justification for retaining the full structure of AP in the loss -- larger chunks of eq. 2 than just the H(...) terms could be replaced. What is the justification for not doing so?

"A Unified Framework of Surrogate Loss by Refactoring and Interpolation" [Liu, ECCV 20] is very relevant, and should be cited (and either evaluated against, or discussed why it is not applicable).

"Learning Surrogates via Deep Embedding" [Patel, ECCV 20] is relevant and should be cited.

"SoDeep: a Sorting Deep net to learn ranking loss surrogates" [Engilberge, CVPR 19] is relevant and should be cited.

It would be useful to add some insights on why the various learnt piecewise-linear functions look so different.

What happens if the Heaviside functions are instead replaced by sigmoids of varying slope/offset? This results in many fewer parameters in the loss, but gives more flexibility than just sigmoid.

There is no discussion of how long the PPO training of the surrogate loss takes (e.g. does it dominate the main detector training time)?

The same training process used for the loss parameters (i.e. PPO2) could, theoretically, also be used to train the main detection model. There should be some discussion of why this is not done (or even better, an evaluation).

42: "helps to represent the numerous AP approximations" -- clarify -- is this supposed to refer back to the AP approximations discuss in the previous bit of text?

46: "more effective" -- than what?

---

## Post-rebuttal

The authors' comment clarified several concerns raised in my review (e.g. the search does not need to be repeated for each dataset; the standard deviations are small). I therefore still broadly favor acceptance. However, the rebuttal simply says that UniLoss [Liu, ECCV '20] -- which is one of the most relevant recent works -- will be discussed in the final version; there is no attempt to justify why it was ignored so far, and no promise to include an evaluation against it. Therefore, I cannot raise my score higher. Concerns raised by the other reviewers seem to have been adequately addressed by the authors' comments.

limitations_and_societal_impact: There is only very minimal discussion of limitations of the method. Societal impact is discussed sufficiently.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a Parameterized AP Loss to better align the network training and evaluation in object detection. The main technical contributions include:
1) It proposes a parameterized loss function by reformulating the AP metric and replacing the non-differentiable component with parameterized functions.
2) It automatically searches for the optimal parameters of the proposed loss function through a reinforcement-learning-based search process.
3) Finally, it demonstrates competitive results on COCO.

main_review: Pros:
1. The idea of introducing localization score into AP loss and parameterizing the reformulated AP loss is novel and interesting.
2. The paper is clearly written and easy to follow with detailed equations.
3. The authors conducted adequate experiments and ablation studies, verifying the efficacy of their method.
Cons:
There is no big flaw in this paper, here are some small issues:
1. In lines 285-292 and Table 2, what is the value of the gradient scale $\lambda$ used in the experiments with hand-crafted substitutions? The experiments with hand-crafted substitutions share H(.), what is the performance of our method when using the same $lambda$ and shared parameters of H(.) as the hand-crafted ones?
2. What is the difference of the shape of the five parameterized functions $f(x; \theta_{i})$ searched by our method. Will they change dramatically in different object detectors (e.g., RetinaNet, Faster R-CNN, and Deformable DETR)? It would be better to visualize the shape of the parameterized functions.
3. How is the generalization of the Parameterized AP Loss. Does it also achieve a good performance when using the hyper-parameters searched on different datasets? For example, searching the hyper-parameters on COCO and using the same hyper-parameters to train the object detector on VOC.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a parameterized AP loss. Parameterizd functions are introduced to substitute the non-differentiable components in the AP calculation. Then automatic parameter search algorithms can be employed to search the optimal parameters.
main_review: The method proposed by this paper is novel to some extent and technically sound. The written is also clear.

Strong points:

The parameterized AP Loss can produce better performance, while hand-crafted AP approximated loss produce lower performance than the commonly used cross-entropy and L1/smooth-L1 losses.

Weak points:

1.Parameterization makes the optimal parameters of the proposed AP Loss different on different datasets, so it is necessary to search for the optimal parameters for each dataset. However, the process of finding optimal parameters is computationally expensive.

2.The AP loss proposed in this paper uses a series of parameterized functions to approximate the calculation of AP, but this paper does not show whether the finally searched functions are really close to the Heaviside step function. If the searched function curve can be drawn, it will give readers a more intuitive understanding.

limitations_and_societal_impact: The searched losses in this paper share the same societal issues with other hand-crafted ones, that the trained object detectors may have inexplicable detection failures and suffer from data bias. This problem mainly needs to be solved from the perspective of dataset.
review_title: null

Meta-review: All the reviewers have supported the acceptance of this paper, so I recommend accepting as well.
Decision: Accept (Poster)