SlowGuess/NAACL_2022_OCR huggingface dataset:
Mean Average Precision by Queries (MAPQ). Different from MAP, MAPQ averages AP over last queries instead of sessions. Suppose  $\mathcal{Q}$  is the set of unique last queries, and  $S(q), q \in \mathcal{Q}$  is the set of sessions whose last queries are  $q$ , then the average precision for one query  $q$  is
$$
\frac {1}{\sum_ {i = 1} ^ {k} \operatorname {r e l} (k)} \sum_ {k = 1} ^ {N} \min  \left(1, \frac {\sum_ {r _ {s} \leq k} \operatorname {r e l} (k)}{k}\right) \tag {6}
$$
then we sum over all queries to obtain MAPQ@N.
Mean Reciprocal Rank by Queries (MRRQ). MRRQ averages MRR over session last queries instead of sessions.
$$
M R R Q @ N = \frac {1}{| \mathcal {Q} |} \sum_ {q \in \mathcal {Q}} \max  _ {s \in S (q)} \left(r _ {s}\right) \tag {7}
$$
Recall. Recall@N calculates the percentage of sessions whose relevant items were retrieved among the top N predictions.