Title: Differentiable Average Precision Loss in DETR
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: AP loss; Object Detection; Rank-based loss

Abstract: Average Precision (AP) and mean AP (mAP) remain the dominant metrics for evaluating object detectors, yet most training objectives optimize surrogates that are not well aligned with these ranking-based measures.
Most prior AP surrogates either rely on pairwise comparisons, incurring quadratic complexity, or only optimize classification, necessitating additional localization losses. We introduce differentiable average precision (DAP) loss, a smooth AP loss that directly optimizes a differentiable approximation to COCO-style mAP for one-to-one detectors.
Our key idea is to (i) replace non-differentiable sorting by modeling detection scores as continuous distributions and sweeping a series of thresholds to obtain a differentiable precision–recall curve, and (ii) use interpolation techniques to optimize localization task.
This yields a differentiable mAP approximation with linear time ($(O(N))$) in the number of predictions, enabling seamless integration with Hungarian matching.
We prove that, with respect to prediction scores, the gradient of DAP is sign-consistent—positive for positives and negative for negatives.
Empirically, fine-tuning pretrained DETR-family models with DAP for a small number of epochs delivers consistent COCO mAP gains without auxiliary losses or architectural changes.
DAP is simple to implement, computationally efficient, reduces hyperparameter tuning, and bridges the gap between training and evaluation for one-to-one detection. 
From-scratch training also delivers modest but positive improvements, albeit smaller than those obtained through fine-tuning.