Average precision (AP) - weird results if val set has easy examples; how to calculate?

martinr: <p>I am computing average precision (AP) for object detection as in <a href="http://host.robots.ox.ac.uk/pascal/VOC/voc2012/htmldoc/devkit_doc.html#SECTION00044000000000000000" rel="nofollow noopener">Pascal VOC</a> dataset. My results were too good and I suspected that I might overlook something. But no, many AP libraries, including <a href="https://github.com/facebookresearch/Detectron/blob/cbb0236dfdc17790658c146837215d2728e6fadd/detectron/datasets/voc_eval.py#L56" rel="nofollow noopener">Mask R-CNN</a> and Pascal VOC devkit have the issue that a single very simple example in validation set will turn the metric totally useless. This is how I calculate AP:</p>
<ol>
<li>calculate precision and recall for each image, store the pairs in <code>R</code>
</li>
<li>iterate through all <code>n</code> unique recall values <code>r</code> in <code>R</code> and, for each value <code>r</code>, find max precision from <code>R</code> where <code>recall &gt;= r</code>(maximum precision where recall is greater or equal than <code>r</code>) (<a href="https://github.com/rafaelpadilla/Object-Detection-Metrics#interpolating-all-points" rel="nofollow noopener">source</a>)</li>
<li>multiply each max precision value by it’s proportion, i.e., <code>next_recall - current_recall</code>. As recall values are between 0 and 1  this will give a proportion to approximate area under the curve.</li>
<li>sum all results from last step</li>
</ol>
<p>Now this works fine and gives reasonable results if we assume that <strong>precision mostly drops when recall grows</strong>. But this is not always a realistic assumption. Given one easy image where precision=1 and recall=1 then everything blows up and AP value for the whole dataset will be 1 (best possible result) even if the algorithm fails on all other examples. This is because each recall gets assigned <em>maximum precision where recall is greater or equal than r</em>. Failing example:</p>
<pre><code>R[recall___] = [0.7, 0.91, 1]
R[precision] = [0.11, 0.10, 1.0]
AP = 1.0
</code></pre>
<p>As I understand, Average Precision should be an approximation of area under the curve of precision-recall plot, which this clearly does not achieve. But as all implementation are giving me the same results then I wonder if:</p>
<ol>
<li>Am I missing something here?</li>
<li>Can we have better and more accurate metric just by redefining <em>maximum precision only where recall is between r and r+1</em>?</li>
<li>And finally, how are so many object detection articles using AP and most never point out how they calculated it?</li>
</ol>
<p>I have also tested this on offical Pascal VOC MATLAB code from their devkit (MATLAB code):</p>
<pre><code class="lang-auto">function ap = VOCap(rec,prec)

mrec=[0 ; rec ; 1];
mpre=[0 ; prec ; 0];
for i=numel(mpre)-1:-1:1
    mpre(i)=max(mpre(i),mpre(i+1));
end
i=find(mrec(2:end)~=mrec(1:end-1))+1;
ap=sum((mrec(i)-mrec(i-1)).*mpre(i));
</code></pre>
<p>There is also Pytorch <a href="https://tnt.readthedocs.io/en/latest/source/torchnet.meter.html#apmeter" rel="nofollow noopener">TNT average precision metric</a> - yet a different one, looks like it defines AP for single validation example, not for the dataset as the inputs are <code>output</code> and <code>target</code> (making it hard to use for object detection where you have to calculate IOU and cannot use direct model output).</p>
<p>Or perhaps someone can just explain how you calculate <em>average precision</em> for the whole dataset in object detection? Just use mean of all precision values?</p>

STARCY: <p><a class="mention" href="/u/martinr">@martinr</a> That is a very smart observation you came across. I have a similar problem that I’ve been grappling with. Did you find a solution to this problem? How did you resolve this please?</p>