\pdfoutput=1

\documentclass[11pt,a4paper]{article}
\usepackage[hyperref]{acl2020}
\usepackage{times}
\usepackage{url}
\usepackage{placeins}
\usepackage{nicefrac}
\usepackage{latexsym}
\usepackage{multirow}
\usepackage{float}
\usepackage{booktabs}
\usepackage{graphicx}
\renewcommand{\UrlFont}{\ttfamily\small}
\usepackage{pgfplots}
\pgfplotsset{compat=1.8}
\usetikzlibrary{patterns}
\usepackage{tikzsymbols}
\usepackage{graphicx}
\usepackage{fdsymbol}

\pgfplotsset{compat=1.8,
    /pgfplots/xbar legend/.style={
    /pgfplots/legend image code/.code={%
       \draw[##1,/tikz/.cd,yshift=-0.25em]
        (0cm,0cm) rectangle (3pt,0.8em);},
   },
}
\usepackage{caption}
\captionsetup{skip=6pt}
\usepackage{microtype}

\aclfinalcopy %
\newcommand\BibTeX{B\textsc{ib}\TeX}

\title{What Are People Asking About COVID-19? \\ A Question Classification Dataset}

\author{
  Jerry Wei$^\spadesuit$ $\hspace{1.5mm}$ 
  Chengyu Huang$^\vardiamondsuit$ $\hspace{1.5mm}$
  Soroush Vosoughi$^\varheartsuit$ $\hspace{1.5mm}$ 
  Jason Wei$^\varheartsuit$ \\
  $^\spadesuit$ProtagoLabs $\hspace{1mm}$
  $^\vardiamondsuit$International Monetary Fund $\hspace{1mm}$
  $^\varheartsuit$Dartmouth College\\
  $\texttt{jerry.weng.wei@protagolabs.com}$\\
  $\texttt{huangchengyu24@gmail.com}$\\
  $\texttt{\{soroush,jason.20\}@dartmouth.edu}$\\
  }
\begin{document}
\maketitle
\begin{abstract}
We present \textsc{Covid-Q}, a set of 1,690 questions about COVID-19 from 13 sources, which we annotate into 15 question categories and 207 question clusters.
The most common questions in our dataset asked about transmission, prevention, and societal effects of COVID, and we found that many questions that appeared in multiple sources were not answered by any FAQ websites of reputable organizations such as the CDC and FDA.
We post our dataset publicly at \url{https://github.com/JerryWei03/COVID-Q}.

For classifying questions into 15 categories, a BERT baseline scored 58.1\% accuracy when trained on 20 examples per category, and for a question clustering task, a BERT + triplet loss baseline achieved 49.5\% accuracy. 
We hope \textsc{Covid-Q} can help either for direct use in developing applied systems or as a domain-specific resource for model evaluation.

\end{abstract}
\vspace{-2mm}
\section{Introduction}
\vspace{-2mm}
A major challenge during fast-developing pandemics such as COVID-19 is keeping people updated with the latest and most relevant information.
Since the beginning of COVID, several websites have created frequently asked questions (FAQ) pages that they regularly update. 
But even so, users might struggle to find their questions on FAQ pages, and many questions remain unanswered. 
In this paper, we ask---what are people really asking about COVID, and how can we use NLP to better understand questions and retrieve relevant content?

\begin{figure}[ht]
\begin{tikzpicture}
\centering
\begin{axis}[
    legend style={font=\tiny},
    xbar,
    xmin=0,
    xmax=250,
    width=0.34\textwidth, 
    height=9cm, 
    ytick style={draw=none},
    xtick style={draw=none},
    xticklabel=\empty,
    xlabel={Unique Questions},
    xlabel shift = -3 mm,
    xlabel style = {font=\small},
    ytick = {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16},
    yticklabels = { Other (6),
                    Symptoms (7),
                    Having COVID (9),
                    Nomenclature (5),
                    Testing (9),
                    Comparison (10),
                    Individual Response (12),
                    Economic Effects (11),
                    Speculation (9),
                    Treatment (12),
                    Origin (10),
                    Reporting (16),
                    Societal Response (22),
                    Prevention (20),
                    Societal Effects (23),
                    Transmission (27)
                    },
    ticklabel style={font=\small},
    nodes near coords, 
    nodes near coords align={horizontal},
    every node near coord/.append style={font=\small},
    ]
\addplot+ [
    style={fill=cyan, bar shift=0pt, draw=black, postaction={pattern=grid}},
    ] 
    coordinates {   
                    (188,16)
                    (100,15)
                    (81,14)
                    (79,13)
                    (68,12)
                    (67,11)
                    (51,10)
                    (50,9)
                    (49,8)
                    (47,7)
                    (45,6)
                    (42,5)
                    (36,4)
                    (36,3)
                    (26,2)
                    (20,1)
                };
\end{axis}
\end{tikzpicture}

\caption{Question categories in \textsc{Covid-Q}, with number of question clusters per category in parentheses. 
}
\label{fig:categories}
\vspace{-6mm}
\end{figure}
We present \textsc{Covid-Q}, a dataset of 1,690 questions about COVID from 13 online sources.
We annotate \textsc{Covid-Q} by classifying questions into 15 general \emph{question categories}\footnote{We do not count the ``other" category.} (see Figure \ref{fig:categories}) and by grouping questions into \textit{question clusters}, for which all questions in a cluster ask the same thing and can be answered by the same answer, for a total of 207 clusters.
Throughout $\S$\ref{dataset_collection}, we analyze the distribution of \textsc{Covid-Q} in terms of question category, cluster, and source. 

\textsc{Covid-Q} facilitates several question understanding tasks.
First, the question categories can be used for a vanilla text classification task to determine the general category of information a question is asking about.
Second, the question clusters can be used for retrieval question answering (since the cluster annotations indicate questions of same intent), where given a new question, a system aims to find a question in an existing database that asks the same thing and returns the corresponding answer \cite{romeo-etal-2016-neural,Sakata2019}.
We provide baselines for these two tasks in $\S$\ref{sec:category_task} and $\S$\ref{sec:class_task}. 
In addition to directly aiding the development of potential applied systems, \textsc{Covid-Q} could also serve as a domain-specific resource for evaluating NLP models trained on COVID data. 

\begin{table*}[ht]
    \centering
    \small
    \begin{tabular}{l | c c c | c | c}
        \toprule
        & \multicolumn{3}{c|}{Questions} & & \\
        Source & Total & Multi-q-cluster & Single-q-cluster & Answers & Questions Removed\\
        \midrule
        Quora & 675 & 501 (74.2$\%$) & 174 (25.8$\%$) & 0 & 374\\
        Google Search & 173 & 161 (93.1$\%$) & 12 (6.9$\%$) & 0 & 174\\
        github.com/deepset-ai/COVID-QA & 124 & 55 (44.4$\%$) & 69 (55.6$\%$) & 124 & 71\\
        Yahoo Search & 94 & 87 (92.6$\%$) & 7 (7.4$\%$) & 0 & 34\\
        $^*$Center for Disease Control & 92 & 51 (55.4$\%$) & 41 (44.6$\%$) & 92 & 1\\
        Bing Search & 68 & 65 (95.6$\%$) & 3 (4.4$\%$) & 0 & 29\\
        $^*$Cable News Network & 64 & 48 (75.0$\%$) & 16 (25.0$\%$) & 64 & 1 \\
        $^*$Food and Drug Administration & 57 & 33 (57.9$\%$) & 24 (42.1$\%$) & 57 & 3\\
        Yahoo Answers & 28 & 13 (46.4$\%$) & 15 (53.6$\%$)& 0 & 23\\
        $^*$Illinois Department of Public Health & 20 & 18 (90.0$\%$) & 2 (10.0$\%$) & 20 & 0\\
        $^*$United Nations & 19 & 18 (94.7$\%$) & 1 (5.3$\%$) & 19 & 6\\
        $^*$Washington DC Area Television Station & 16 & 15 (93.8$\%$) & 1 (6.2$\%$) & 16 & 0\\
        $^*$Johns Hopkins University & 11 & 10 (90.9$\%$) & 1  (9.1$\%$) & 11 & 1\\
        \midrule
        Author Generated & 249 & 249 (100.0$\%$) & 0 (0.0$\%$) & 0 & 0\\
        \midrule
        Total & 1,690 & 1,324 (78.3$\%$) & 366 (21.7$\%$) & 403 & 717\\ 
        \bottomrule
    \end{tabular}
    \caption{Distribution of questions in \textsc{Covid-Q} by source. 
    The reported number of questions excludes vague and nonsensical questions that were removed. 
    Multi-q-cluster: number of questions that belonged to a question cluster with at least two questions;
    Single-q-cluster: number of questions that belonged to a question cluster with only a single question (no other question in the dataset asked the same thing).
    $^*$ denotes FAQ page sources. 
    }
    \label{tab:dataset_table}
\end{table*}


\section{Dataset Collection and Annotation}
\label{dataset_collection}
\vspace{0.5em} \noindent \textbf{Data collection.}
In May 2020, we scraped questions about COVID from thirteen sources: seven official FAQ websites from recognized organizations such as the Center for Disease Control (CDC) and the Food and Drug Administration (FDA), and six crowd-based sources such as Quora and Yahoo Answers.
Table \ref{tab:dataset_table} shows the distribution of collected questions from each source. 
We also post the original scraped websites for each source.

\vspace{0.5em} \noindent \textbf{Data cleaning.}
We performed several pre-processing steps to remove unrelated, low-quality, and nonsensical questions.
First, we deleted questions unrelated to COVID and vague questions with too many interpretations (e.g., ``Why COVID?"). 
Second, we removed location-specific and time-specific versions of questions (e.g., ``COVID deaths in New York"), since these questions do not contribute linguistic novelty (you could replace ``New York" with any state, for example). 
Questions that only targeted one location or time, however, were not removed---for instance, ``Was China responsible for COVID?" was not removed because no questions asked about any other country being responsible for the pandemic.
\begingroup
\setlength{\tabcolsep}{3pt}
\begin{table}[th]
    \small
    \centering
    \begin{tabular}{l l}
         \toprule
         \multirow{3}{*}{\shortstack[l]{Question Cluster \\ $[\#$Questions$]$ \\ (Category) }} & \\
         & \\
         & \multicolumn{1}{c}{Example Questions}\\
         \midrule
         Pandemic Duration & ``Will COVID ever go away?"\\
         $[$28$]$ & ``Will COVID end soon?"\\
         (Speculation) & ``When COVID will end?"\\
         \midrule
         Demographics: General & ``Who is at higher risk?"\\
         $[$26$]$ & ``Are kids more at risk?"\\
         (Transmission) & ``Who is COVID killing?"\\
         \midrule
         Survivability: Surfaces & ``Does COVID live on surfaces?"\\
         $[$24$]$ & ``Can COVID live on paper?"\\
         (Transmission) & ``Can COVID live on objects?"\\
         \bottomrule
    \end{tabular}
    \caption{Most common question clusters in \textsc{Covid-Q}.}
    \vspace{-3.5mm}
    \label{Table:FAQs}
\end{table}
\endgroup
Finally, to minimize occurrences of questions that trivially differ, we removed all punctuation  and replaced synonymous ways of saying COVID, such as ``coronavirus," and ``COVID-19" with ``covid."
Table \ref{tab:dataset_table} also shows the number of removed questions for each source.
\vspace{0.5em} \noindent \textbf{Data annotation.}
We first annotated our dataset by grouping questions that asked the same thing together into question clusters. 
The first author manually compared each question with existing clusters and questions, using the definition that two questions belong in the same cluster if they have the same answer.
In other words, two questions matched to the same question cluster if and only if they could be answered with a common answer.
As every new example in our dataset is checked against all existing question clusters, including clusters with only one question, the time complexity for annotating our dataset is $O(n^2)$, where $n$ is the number of questions.

After all questions were grouped into question clusters, the first author gave each question cluster with at least two questions a name summarizing the questions in that cluster, and each question cluster was assigned to one of 15 question categories (as shown in Figure 1), which were conceived during a thorough discussion with the last author. 
In Table \ref{Table:FAQs}, we show the question clusters with the most questions, along with their assigned question categories and some example questions.
Figure \ref{fig:histogram} shows the distribution of question clusters.

\begin{figure}[h]
\begin{tikzpicture}
\centering
    \begin{axis}[
        area style,
        width=0.5\textwidth, 
        height=4.5cm, 
        xlabel={Questions per Question cluster},
        ylabel={Question clusters},
        xlabel shift = -1.5 mm,
        xtick style={font=\small},
        ytick style={font=\small},
        label style={font=\small},
        ticklabel style = {font=\small}
        ]
    \addplot+[ybar interval,mark=no] plot coordinates { 
        (2, 86)
        (3, 30)
        (4, 24)
        (5, 12)
        (6, 10)
        (7, 5)
        (8, 8)
        (9, 6)
        (10, 3)
        (11, 5)
        (12, 3)
        (13, 5)
        (14, 2)
        (16, 2)
        (18, 1)
        (23, 1)
        (24, 1)
        (26, 1)
        (29, 1)
    };
    \end{axis}
\end{tikzpicture}
\caption{
Number of questions per question cluster for clusters with at least two questions. All questions in a question cluster asked roughly the same thing.
120 question clusters had at least 3 questions per cluster, 66 clusters had at least 5 questions per cluster, and 22 clusters had at least 10 questions per cluster.
}
    \vspace{-3.5mm}
\label{fig:histogram}
\end{figure}

\vspace{0.5em} \noindent \textbf{Annotation quality.} We ran the dataset through multiple annotators to improve the quality of our annotations. 
First, the last author confirmed all clusters in the dataset, highlighting any questions that might need to be relabeled and discussing them with the first author. 
Of the 1,245 questions belonging to question clusters with at least two questions, 131 questions were highlighted and 67 labels were modified.
For a second pass, an external annotator similarly read through the question cluster labels, for which 31 questions were highlighted and 15 labels were modified.
Most modifications involved separating a single question cluster that was too broad into several more specific clusters. 

For another round of validation, we showed three questions from each of the 89 question clusters with $N_{cluster} \geq 4$ to three Mechanical Turk workers, who were asked to select the correct question cluster from five choices. 
The majority vote from the three workers agreed with our ground-truth question-cluster labels 93.3\% of the time. 
The three workers unanimously agreed on 58.1\% of the questions, for which 99.4\% of these unanimous labels agreed with our ground-truth label.
Workers were paid $\$0.07$ per question.

Finally, it is possible that some questions could fit in several categories---of 207 clusters, 40 arguably mapped to two or more categories, most frequently the transmission and prevention categories.
As this annotation involves some degree of subjectivity, we post formal definitions of each question category with our dataset to make these distinctions more transparent.
\vspace{0.5em} \noindent \textbf{Single-question clusters.} 
Interestingly, we observe that for the CDC and FDA frequently asked questions websites, a sizable fraction of questions (44.6\% for CDC and 42.1\% for FDA) did not ask the same thing as questions from any other source (and therefore formed \textit{single-question clusters}), suggesting that these sources might want adjust the questions on their websites to question clusters that were seen frequently in search engines such as Google or Bing.
Moreover, 54.2\% of question clusters that had questions from at least two non-official sources went unanswered by an official source.
In the Supplementary Materials, Table \ref{tab:missing_faq} shows examples of these questions, and conversely, Table \ref{tab:unmatched_questions} shows CDC and FDA questions that did not belong to the same cluster as any other question.

\section{Question Understanding Tasks}
\label{sec:q_class}
\vspace{-1mm}
We provide baselines for two tasks: \textit{question-category classification}, where each question belongs to one of 15 categories, and \textit{question clustering}, where questions asking the same thing belong to the same cluster.

As our dataset is small when split into training and test sets, we manually generate an additional \textit{author-generated} evaluation set of $249$ questions. 
For these questions, the first author wrote new questions for question clusters with 4 or 5 questions per cluster until those clusters had 6 questions.
These questions were checked in the same fashion as the real questions.
For clarity, we only refer to them in $\S$\ref{sec:category_task} unless explicitly stated. 

\subsection{Question-Category Classification}
\label{sec:category_task}
The \textit{question-category classification} task assigns each question to one of 15 categories shown in Figure 1. 
For the train-test split, we randomly choose 20 questions per category for training (as the smallest category has 26 questions), with the remaining questions going into the test set (see Table \ref{tab:datasetsplit_category_class}).


\begin{table}[h]
    \centering
    \small
    \begin{tabular}{l c}
        \toprule
        Question Categories & 15 \\
        Training Questions per Category & 20\\
        Training Questions & 300 \\
        Test Questions (Real) & 668 \\
        Test Questions (Generated) & 238 \\
        \bottomrule
    \end{tabular}
    \caption{Data split for \textit{question-category classification}.}
    \vspace{-3mm}
    \label{tab:datasetsplit_category_class}
\end{table}



We run simple BERT \cite{devlin-etal-2019-bert} feature-extraction baselines with question representations obtained by average-pooling.
For this task, we use two models: (1) SVM and (2) cosine-similarity based $k$-nearest neighbor classification ($k$-NN) with $k=1$. 
As shown in Table \ref{tab:category_classification}, the SVM marginally outperforms $k$-NN on both the real and generated evaluation sets. 
Since our dataset is small, we also include results from using data augmentation \cite{wei-zou-2019-eda}.
Figure \ref{fig:heatmap} (Supplementary Materials) shows the confusion matrix for BERT-feat:~SVM + augmentation for this task.

\begingroup
\begin{table}[h]
\setlength{\tabcolsep}{7pt}
    \small 
    \centering
    \begin{tabular}{l | c c}
        \toprule
         Model & Real Q & Generated Q \\
         \midrule
         BERT-feat: $k$-NN & 47.8 & 52.1\\
         \hspace{2mm} + augmentation & 47.3 & 52.5\\
         \midrule
         BERT-feat: SVM & 52.2 & 53.4\\
         \hspace{2mm} + augmentation & 58.1 & 58.8\\
         \bottomrule
    \end{tabular}
    \caption{Performance of BERT baselines (accuracy in \%) on \textit{question-category classification} with 15 categories and 20 training examples per category.}
    \vspace{-4mm}
    \label{tab:category_classification}
\end{table}
\endgroup

\subsection{Question Clustering} 
\label{sec:class_task}
Of a more granular nature, the \textit{question clustering} task asks, given a database of known questions, whether a new question asks the same thing as an existing question in the database or whether it is a novel question.
To simulate a potential applied setting as much as possible, we use all questions clusters in our dataset, including clusters containing only a single question.
As shown in Table \ref{tab:datasetsplit_qclass}, we make a 70\%--30\% train--test split by class.\footnote{For clusters with two questions, one question went into the training set and one into the test set. 70\% of single-question clusters went into the training set and 30\% into the test set.}

\begin{table}[h]
    \centering
    \small
    \begin{tabular}{l c}
        \toprule
        Training Questions & 920\\
        Training Clusters & 460\\
        Test Questions & 437\\
        Test Clusters & 320\\
        Test Questions from multi-q-clusters & 323\\ 
        Test Questions from single-q-clusters & 114\\
        \bottomrule
    \end{tabular}
    \caption{Data split for \textit{question clustering}.}
    \vspace{-1mm}
    \label{tab:datasetsplit_qclass}
\end{table}

In addition to the $k$-NN baseline from $\S$\ref{sec:category_task}, we also evaluate a simple model that uses a triplet loss function to train a two-layer neural net on BERT features, a method introduced for facial recognition \cite{facenet} and now used in NLP for few-shot learning \cite{yu-etal-2018-diverse} and answer selection \cite{kumar-etal-2019-improving}.
\begingroup
\begin{table}[ht]
\setlength{\tabcolsep}{5pt}
    \small 
    \centering
    \begin{tabular}{l | c c}
        \toprule
          & \multicolumn{2}{c}{Accuracy (\%)} \\
         Model & Top-1 & Top-5 \\
         \midrule
         BERT-feat: $k$-NN & 39.6 & 58.8\\
         \hspace{2mm}+ augmentation & 39.6 & 59.0\\
         \midrule
         BERT-feat: triplet loss & 47.7 & 66.9 \\
         \hspace{2mm}+ augmentation & 49.5 & 69.4 \\
         \bottomrule
    \end{tabular}
    \caption{Performance of BERT baselines on \textit{question clustering} involving 207 clusters.}
    \vspace{-3mm}
    \label{tab:baseline_class}
\end{table}
\endgroup
For evaluation, we compute a single accuracy metric that requires a question to be either correctly matched to a cluster in the database or to be correctly identified as a novel question.
Our baseline models use thresholding to determine whether questions were in the database or novel. 
Table \ref{tab:baseline_class} shows the accuracy from the best threshold for both these models, and Supplementary Figure \ref{fig:clustering} shows their accuracies for different thresholds.
\section{Discussion}

\textbf{Use cases.} We imagine several use cases for \textsc{Covid-q}.
Our question clusters could help train and evaluate retrieval-QA systems, such as \url{covid.deepset.ai} or \url{covid19.dialogue.co}, which, given a new question, aim to retrieve the corresponding QA pair in an existing database. 
Another relevant context is query understanding, as clusters identify queries of the same intent, and categories identify queries asking about the same topic.
Finally, \textsc{Covid-q} could be used broadly to evaluate COVID-specific models---our baseline (Huggingface's \texttt{bert-base-uncased}) does not even have \textit{COVID} in the vocabulary, and so we suspect that models pre-trained on scientific or COVID-specific data will outperform our baseline.
More related areas include COVID-related query expansion, suggestion, and rewriting.
\vspace{0.5em} \noindent \textbf{Limitations.}
Our dataset was collected in May 2020, and we see it as a snapshot in time of questions asked up until then.
As the COVID situation further develops, a host of new questions will arise, and the content of these new questions will potentially not be covered by any existing clusters in our dataset.
The question categories, on the other hand, are more likely to remain static (i.e., new questions would likely map to an existing category), but the current way that we came up with the categories might be considered subjective---we leave that determination to the reader (refer to Table 9 or the raw dataset on Github). 
Finally, although the distribution of questions per cluster is highly skewed (Figure \ref{fig:histogram}), we still provide them at least as a reference for applied scenarios where it would be useful to know the number of queries asking the same thing (and perhaps how many answers are needed to answer the majority of questions asked). 
\bibliography{acl2020}
\bibliographystyle{acl_natbib}
\newpage
\section{Supplementary Materials}

\subsection{Question Clustering Thresholds}
For the question clustering task, our models used simple thresholding to determine whether a question matched an existing cluster in the database or was novel. 
That is, if the similarity between a question and its most similar question in the database was lower than some threshold, then the model predicted that it was a novel question.
Figure \ref{fig:clustering} shows the accuracy of the $k$-NN and triplet loss models at different thresholds.
\begin{figure}[ht]
\small 
\centering

\hspace{13mm} BERT-feat: $k$-NN

\begin{tikzpicture}
	\begin{axis}[
	    xlabel=Threshold,
	    ylabel=Accuracy,
height=5cm,
width=7cm,
	]
	\addplot coordinates {
(0.6859, 0.4691)
(0.7007, 0.4714)
(0.7097, 0.4714)
(0.7133, 0.4691)
(0.7166, 0.4691)
(0.7220, 0.4691)
(0.7257, 0.4691)
(0.7301, 0.4737)
(0.7325, 0.4783)
(0.7347, 0.4805)
(0.7365, 0.4805)
(0.7384, 0.4805)
(0.7395, 0.4828)
(0.7407, 0.4805)
(0.7427, 0.4828)
(0.7465, 0.4828)
(0.7480, 0.4851)
(0.7492, 0.4851)
(0.7505, 0.4851)
(0.7515, 0.4874)
(0.7522, 0.4897)
(0.7543, 0.4897)
(0.7561, 0.4920)
(0.7576, 0.4920)
(0.7584, 0.4943)
(0.7600, 0.4943)
(0.7608, 0.4966)
(0.7625, 0.4989)
(0.7632, 0.4989)
(0.7645, 0.5034)
(0.7655, 0.5057)
(0.7661, 0.5057)
(0.7668, 0.5034)
(0.7676, 0.5034)
(0.7682, 0.5080)
(0.7688, 0.5103)
(0.7695, 0.5103)
(0.7699, 0.5103)
(0.7702, 0.5126)
(0.7709, 0.5172)
(0.7713, 0.5172)
(0.7718, 0.5195)
(0.7723, 0.5195)
(0.7727, 0.5195)
(0.7733, 0.5217)
(0.7737, 0.5217)
(0.7743, 0.5217)
(0.7749, 0.5217)
(0.7753, 0.5263)
(0.7756, 0.5263)
(0.7759, 0.5286)
(0.7760, 0.5286)
(0.7765, 0.5286)
(0.7771, 0.5286)
(0.7776, 0.5332)
(0.7778, 0.5355)
(0.7780, 0.5378)
(0.7787, 0.5378)
(0.7792, 0.5378)
(0.7796, 0.5400)
(0.7798, 0.5423)
(0.7805, 0.5400)
(0.7808, 0.5400)
(0.7813, 0.5400)
(0.7815, 0.5400)
(0.7818, 0.5423)
(0.7821, 0.5446)
(0.7822, 0.5446)
(0.7827, 0.5446)
(0.7832, 0.5492)
(0.7834, 0.5492)
(0.7844, 0.5492)
(0.7849, 0.5515)
(0.7854, 0.5515)
(0.7860, 0.5492)
(0.7863, 0.5538)
(0.7866, 0.5538)
(0.7867, 0.5538)
(0.7869, 0.5538)
(0.7870, 0.5538)
(0.7875, 0.5515)
(0.7876, 0.5515)
(0.7879, 0.5515)
(0.7881, 0.5538)
(0.7884, 0.5538)
(0.7886, 0.5561)
(0.7891, 0.5561)
(0.7894, 0.5561)
(0.7897, 0.5561)
(0.7899, 0.5584)
(0.7902, 0.5584)
(0.7905, 0.5584)
(0.7909, 0.5584)
(0.7913, 0.5584)
(0.7917, 0.5584)
(0.7922, 0.5584)
(0.7925, 0.5584)
(0.7927, 0.5584)
(0.7930, 0.5584)
(0.7934, 0.5584)
(0.7938, 0.5629)
(0.7940, 0.5629)
(0.7942, 0.5675)
(0.7946, 0.5675)
(0.7948, 0.5675)
(0.7949, 0.5675)
(0.7952, 0.5629)
(0.7954, 0.5629)
(0.7955, 0.5606)
(0.7957, 0.5606)
(0.7962, 0.5606)
(0.7964, 0.5584)
(0.7966, 0.5561)
(0.7969, 0.5584)
(0.7972, 0.5584)
(0.7975, 0.5584)
(0.7976, 0.5606)
(0.7978, 0.5606)
(0.7979, 0.5606)
(0.7981, 0.5629)
(0.7982, 0.5629)
(0.7984, 0.5629)
(0.7990, 0.5629)
(0.7992, 0.5629)
(0.7994, 0.5652)
(0.7998, 0.5675)
(0.8000, 0.5721)
(0.8002, 0.5721)
(0.8004, 0.5721)
(0.8008, 0.5721)
(0.8009, 0.5721)
(0.8010, 0.5744)
(0.8014, 0.5767)
(0.8015, 0.5789)
(0.8017, 0.5789)
(0.8019, 0.5789)
(0.8020, 0.5767)
(0.8021, 0.5767)
(0.8024, 0.5767)
(0.8026, 0.5789)
(0.8031, 0.5789)
(0.8033, 0.5767)
(0.8035, 0.5767)
(0.8036, 0.5767)
(0.8038, 0.5767)
(0.8039, 0.5767)
(0.8042, 0.5767)
(0.8044, 0.5767)
(0.8048, 0.5789)
(0.8049, 0.5789)
(0.8052, 0.5812)
(0.8054, 0.5812)
(0.8058, 0.5812)
(0.8059, 0.5835)
(0.8061, 0.5835)
(0.8063, 0.5835)
(0.8065, 0.5858)
(0.8066, 0.5858)
(0.8070, 0.5858)
(0.8072, 0.5858)
(0.8075, 0.5858)
(0.8076, 0.5812)
(0.8078, 0.5789)
(0.8081, 0.5789)
(0.8082, 0.5789)
(0.8086, 0.5789)
(0.8087, 0.5789)
(0.8090, 0.5789)
(0.8094, 0.5789)
(0.8096, 0.5812)
(0.8099, 0.5812)
(0.8100, 0.5812)
(0.8103, 0.5789)
(0.8105, 0.5789)
(0.8108, 0.5767)
(0.8109, 0.5767)
(0.8112, 0.5789)
(0.8114, 0.5812)
(0.8116, 0.5812)
(0.8118, 0.5812)
(0.8120, 0.5789)
(0.8124, 0.5789)
(0.8127, 0.5789)
(0.8128, 0.5835)
(0.8130, 0.5858)
(0.8131, 0.5858)
(0.8132, 0.5858)
(0.8134, 0.5858)
(0.8138, 0.5835)
(0.8139, 0.5835)
(0.8142, 0.5835)
(0.8144, 0.5812)
(0.8145, 0.5812)
(0.8147, 0.5812)
(0.8149, 0.5812)
(0.8150, 0.5812)
(0.8152, 0.5812)
(0.8153, 0.5789)
(0.8155, 0.5789)
(0.8155, 0.5789)
(0.8157, 0.5812)
(0.8160, 0.5835)
(0.8163, 0.5812)
(0.8164, 0.5835)
(0.8167, 0.5812)
(0.8169, 0.5835)
(0.8171, 0.5835)
(0.8173, 0.5835)
(0.8173, 0.5835)
(0.8175, 0.5835)
(0.8176, 0.5835)
(0.8178, 0.5835)
(0.8179, 0.5835)
(0.8182, 0.5835)
(0.8185, 0.5812)
(0.8186, 0.5812)
(0.8188, 0.5835)
(0.8189, 0.5835)
(0.8191, 0.5812)
(0.8192, 0.5812)
(0.8196, 0.5767)
(0.8196, 0.5767)
(0.8198, 0.5789)
(0.8201, 0.5789)
(0.8202, 0.5812)
(0.8203, 0.5789)
(0.8205, 0.5789)
(0.8207, 0.5812)
(0.8208, 0.5812)
(0.8213, 0.5789)
(0.8215, 0.5789)
(0.8220, 0.5812)
(0.8222, 0.5767)
(0.8224, 0.5767)
(0.8226, 0.5767)
(0.8228, 0.5767)
(0.8228, 0.5789)
(0.8231, 0.5789)
(0.8232, 0.5767)
(0.8234, 0.5789)
(0.8235, 0.5767)
(0.8237, 0.5744)
(0.8239, 0.5767)
(0.8241, 0.5744)
(0.8244, 0.5744)
(0.8247, 0.5744)
(0.8250, 0.5744)
(0.8253, 0.5744)
(0.8254, 0.5721)
(0.8256, 0.5721)
(0.8259, 0.5675)
(0.8260, 0.5629)
(0.8262, 0.5629)
(0.8263, 0.5629)
(0.8266, 0.5606)
(0.8268, 0.5629)
(0.8269, 0.5584)
(0.8272, 0.5584)
(0.8275, 0.5584)
(0.8276, 0.5561)
(0.8279, 0.5606)
(0.8286, 0.5606)
(0.8289, 0.5629)
(0.8291, 0.5629)
(0.8292, 0.5629)
(0.8294, 0.5606)
(0.8296, 0.5584)
(0.8297, 0.5584)
(0.8299, 0.5606)
(0.8301, 0.5629)
(0.8303, 0.5606)
(0.8305, 0.5629)
(0.8307, 0.5629)
(0.8308, 0.5652)
(0.8310, 0.5629)
(0.8313, 0.5606)
(0.8317, 0.5606)
(0.8319, 0.5584)
(0.8322, 0.5584)
(0.8325, 0.5584)
(0.8328, 0.5584)
(0.8329, 0.5584)
(0.8330, 0.5584)
(0.8334, 0.5584)
(0.8335, 0.5584)
(0.8337, 0.5606)
(0.8338, 0.5629)
(0.8343, 0.5629)
(0.8344, 0.5629)
(0.8347, 0.5629)
(0.8353, 0.5629)
(0.8355, 0.5652)
(0.8356, 0.5675)
(0.8360, 0.5675)
(0.8361, 0.5675)
(0.8364, 0.5675)
(0.8366, 0.5675)
(0.8369, 0.5675)
(0.8370, 0.5675)
(0.8372, 0.5652)
(0.8374, 0.5629)
(0.8377, 0.5629)
(0.8378, 0.5629)
(0.8381, 0.5606)
(0.8384, 0.5606)
(0.8386, 0.5629)
(0.8389, 0.5629)
(0.8392, 0.5629)
(0.8393, 0.5606)
(0.8397, 0.5606)
(0.8400, 0.5584)
(0.8402, 0.5584)
(0.8403, 0.5561)
(0.8406, 0.5538)
(0.8408, 0.5515)
(0.8411, 0.5515)
(0.8412, 0.5538)
(0.8414, 0.5538)
(0.8416, 0.5538)
(0.8417, 0.5515)
(0.8421, 0.5492)
(0.8426, 0.5515)
(0.8431, 0.5515)
(0.8436, 0.5515)
(0.8438, 0.5515)
(0.8443, 0.5515)
(0.8445, 0.5515)
(0.8448, 0.5492)
(0.8449, 0.5492)
(0.8451, 0.5492)
(0.8454, 0.5492)
(0.8458, 0.5469)
(0.8462, 0.5446)
(0.8469, 0.5423)
(0.8473, 0.5423)
(0.8474, 0.5400)
(0.8479, 0.5400)
(0.8483, 0.5400)
(0.8485, 0.5400)
(0.8488, 0.5400)
(0.8494, 0.5400)
(0.8497, 0.5400)
(0.8500, 0.5378)
(0.8503, 0.5355)
(0.8506, 0.5355)
(0.8513, 0.5355)
(0.8519, 0.5355)
(0.8527, 0.5332)
(0.8529, 0.5332)
(0.8531, 0.5332)
(0.8534, 0.5309)
(0.8538, 0.5309)
(0.8545, 0.5309)
(0.8549, 0.5309)
(0.8551, 0.5286)
(0.8554, 0.5263)
(0.8557, 0.5263)
(0.8565, 0.5263)
(0.8571, 0.5240)
(0.8575, 0.5217)
(0.8579, 0.5195)
(0.8582, 0.5149)
(0.8586, 0.5103)
(0.8592, 0.5057)
(0.8594, 0.5057)
(0.8597, 0.5057)
(0.8601, 0.5057)
(0.8601, 0.5034)
(0.8603, 0.5034)
(0.8607, 0.5034)
(0.8610, 0.5034)
(0.8612, 0.5011)
(0.8615, 0.5011)
(0.8619, 0.4989)
(0.8623, 0.4989)
(0.8627, 0.4989)
(0.8631, 0.4920)
(0.8636, 0.4897)
(0.8643, 0.4874)
(0.8650, 0.4874)
(0.8656, 0.4874)
(0.8664, 0.4874)
(0.8669, 0.4874)
(0.8674, 0.4874)
(0.8678, 0.4874)
(0.8688, 0.4851)
(0.8694, 0.4828)
(0.8700, 0.4828)
(0.8706, 0.4783)
(0.8714, 0.4783)
(0.8718, 0.4760)
(0.8721, 0.4714)
(0.8725, 0.4714)
(0.8741, 0.4691)
(0.8747, 0.4645)
(0.8755, 0.4622)
(0.8760, 0.4622)
(0.8765, 0.4554)
(0.8770, 0.4531)
(0.8776, 0.4485)
(0.8783, 0.4485)
(0.8788, 0.4462)
(0.8798, 0.4394)
(0.8806, 0.4371)
(0.8820, 0.4325)
(0.8841, 0.4302)
(0.8852, 0.4279)
(0.8867, 0.4256)
(0.8885, 0.4211)
(0.8901, 0.4188)
(0.8921, 0.4142)
(0.8936, 0.4119)
(0.8956, 0.4073)
(0.8962, 0.4027)
(0.8975, 0.3959)
(0.8991, 0.3890)
(0.9007, 0.3867)
(0.9021, 0.3844)
(0.9042, 0.3799)
(0.9077, 0.3753)
(0.9084, 0.3730)
(0.9103, 0.3684)
(0.9154, 0.3638)
(0.9161, 0.3547)
(0.9197, 0.3501)
(0.9218, 0.3478)
(0.9255, 0.3455)
(0.9271, 0.3410)
(0.9303, 0.3318)
(0.9335, 0.3249)
(0.9390, 0.3181)
(0.9429, 0.3112)
(0.9483, 0.3021)
(0.9544, 0.2975)
(0.9664, 0.2906)
(1.0000, 0.2792)
	};
	\addlegendentry{top-5}
\addplot coordinates {
        (0.7347, 0.3021)
        (0.7359, 0.3043)
        (0.7395, 0.3043)
        (0.7443, 0.3066)
        (0.7480, 0.3089)
        (0.7592, 0.3089)
        (0.7700, 0.3089)
        (0.7703, 0.3089)
        (0.7738, 0.3089)
        (0.7743, 0.3089)
        (0.7754, 0.3066)
        (0.7757, 0.3043)
        (0.7759, 0.3043)
        (0.7785, 0.3066)
        (0.7803, 0.3089)
        (0.7804, 0.3089)
        (0.7805, 0.3112)
        (0.7807, 0.3112)
        (0.7812, 0.3135)
        (0.7815, 0.3158)
        (0.7821, 0.3181)
        (0.7834, 0.3181)
        (0.7854, 0.3181)
        (0.7884, 0.3158)
        (0.7902, 0.3158)
        (0.7903, 0.3181)
        (0.7904, 0.3204)
        (0.7912, 0.3204)
        (0.7914, 0.3227)
        (0.7928, 0.3249)
        (0.7935, 0.3272)
        (0.7940, 0.3295)
        (0.7951, 0.3318)
        (0.7954, 0.3295)
        (0.7955, 0.3318)
        (0.7956, 0.3295)
        (0.7959, 0.3318)
        (0.7963, 0.3318)
        (0.7965, 0.3318)
        (0.7966, 0.3295)
        (0.7973, 0.3318)
        (0.7974, 0.3318)
        (0.7975, 0.3318)
        (0.7980, 0.3341)
        (0.8002, 0.3364)
        (0.8014, 0.3364)
        (0.8015, 0.3387)
        (0.8019, 0.3387)
        (0.8020, 0.3364)
        (0.8034, 0.3364)
        (0.8035, 0.3387)
        (0.8036, 0.3387)
        (0.8037, 0.3387)
        (0.8048, 0.3410)
        (0.8058, 0.3410)
        (0.8061, 0.3410)
        (0.8065, 0.3410)
        (0.8070, 0.3410)
        (0.8072, 0.3432)
        (0.8075, 0.3432)
        (0.8081, 0.3410)
        (0.8096, 0.3410)
        (0.8100, 0.3432)
        (0.8101, 0.3455)
        (0.8102, 0.3432)
        (0.8108, 0.3432)
        (0.8109, 0.3432)
        (0.8117, 0.3432)
        (0.8118, 0.3432)
        (0.8127, 0.3455)
        (0.8129, 0.3455)
        (0.8131, 0.3478)
        (0.8134, 0.3501)
        (0.8145, 0.3478)
        (0.8148, 0.3478)
        (0.8150, 0.3501)
        (0.8152, 0.3501)
        (0.8155, 0.3501)
        (0.8155, 0.3501)
        (0.8156, 0.3501)
        (0.8164, 0.3524)
        (0.8167, 0.3524)
        (0.8167, 0.3501)
        (0.8170, 0.3501)
        (0.8173, 0.3501)
        (0.8176, 0.3501)
        (0.8176, 0.3524)
        (0.8178, 0.3524)
        (0.8178, 0.3524)
        (0.8181, 0.3524)
        (0.8192, 0.3524)
        (0.8196, 0.3501)
        (0.8196, 0.3524)
        (0.8197, 0.3524)
        (0.8202, 0.3547)
        (0.8208, 0.3524)
        (0.8208, 0.3524)
        (0.8208, 0.3501)
        (0.8212, 0.3501)
        (0.8222, 0.3501)
        (0.8222, 0.3478)
        (0.8224, 0.3455)
        (0.8228, 0.3478)
        (0.8228, 0.3501)
        (0.8232, 0.3524)
        (0.8234, 0.3501)
        (0.8234, 0.3501)
        (0.8237, 0.3478)
        (0.8238, 0.3501)
        (0.8253, 0.3501)
        (0.8254, 0.3524)
        (0.8254, 0.3501)
        (0.8254, 0.3524)
        (0.8259, 0.3524)
        (0.8263, 0.3547)
        (0.8265, 0.3524)
        (0.8266, 0.3547)
        (0.8269, 0.3547)
        (0.8271, 0.3570)
        (0.8272, 0.3593)
        (0.8275, 0.3616)
        (0.8288, 0.3593)
        (0.8291, 0.3616)
        (0.8292, 0.3616)
        (0.8292, 0.3638)
        (0.8294, 0.3661)
        (0.8294, 0.3638)
        (0.8296, 0.3638)
        (0.8297, 0.3638)
        (0.8300, 0.3638)
        (0.8301, 0.3638)
        (0.8301, 0.3638)
        (0.8303, 0.3616)
        (0.8307, 0.3638)
        (0.8308, 0.3638)
        (0.8308, 0.3638)
        (0.8314, 0.3616)
        (0.8317, 0.3616)
        (0.8323, 0.3616)
        (0.8325, 0.3638)
        (0.8328, 0.3616)
        (0.8328, 0.3616)
        (0.8328, 0.3638)
        (0.8328, 0.3638)
        (0.8329, 0.3661)
        (0.8330, 0.3661)
        (0.8341, 0.3661)
        (0.8344, 0.3661)
        (0.8345, 0.3684)
        (0.8354, 0.3707)
        (0.8355, 0.3730)
        (0.8356, 0.3753)
        (0.8356, 0.3753)
        (0.8357, 0.3753)
        (0.8364, 0.3776)
        (0.8366, 0.3753)
        (0.8369, 0.3753)
        (0.8370, 0.3753)
        (0.8373, 0.3753)
        (0.8375, 0.3730)
        (0.8377, 0.3730)
        (0.8378, 0.3730)
        (0.8382, 0.3707)
        (0.8386, 0.3707)
        (0.8390, 0.3707)
        (0.8390, 0.3707)
        (0.8391, 0.3730)
        (0.8393, 0.3730)
        (0.8397, 0.3730)
        (0.8399, 0.3730)
        (0.8401, 0.3707)
        (0.8402, 0.3707)
        (0.8402, 0.3707)
        (0.8403, 0.3684)
        (0.8403, 0.3684)
        (0.8404, 0.3684)
        (0.8407, 0.3661)
        (0.8408, 0.3661)
        (0.8408, 0.3638)
        (0.8409, 0.3638)
        (0.8412, 0.3638)
        (0.8415, 0.3661)
        (0.8416, 0.3684)
        (0.8417, 0.3661)
        (0.8422, 0.3638)
        (0.8426, 0.3661)
        (0.8434, 0.3661)
        (0.8435, 0.3684)
        (0.8441, 0.3707)
        (0.8446, 0.3707)
        (0.8449, 0.3730)
        (0.8450, 0.3730)
        (0.8454, 0.3753)
        (0.8455, 0.3776)
        (0.8457, 0.3799)
        (0.8462, 0.3776)
        (0.8469, 0.3776)
        (0.8469, 0.3776)
        (0.8473, 0.3776)
        (0.8479, 0.3753)
        (0.8483, 0.3776)
        (0.8483, 0.3799)
        (0.8487, 0.3822)
        (0.8492, 0.3822)
        (0.8495, 0.3822)
        (0.8497, 0.3822)
        (0.8497, 0.3822)
        (0.8499, 0.3822)
        (0.8499, 0.3799)
        (0.8500, 0.3799)
        (0.8509, 0.3822)
        (0.8513, 0.3822)
        (0.8534, 0.3822)
        (0.8536, 0.3822)
        (0.8538, 0.3822)
        (0.8545, 0.3822)
        (0.8549, 0.3844)
        (0.8550, 0.3844)
        (0.8552, 0.3822)
        (0.8557, 0.3822)
        (0.8559, 0.3844)
        (0.8560, 0.3844)
        (0.8567, 0.3844)
        (0.8571, 0.3867)
        (0.8571, 0.3867)
        (0.8571, 0.3867)
        (0.8572, 0.3890)
        (0.8577, 0.3867)
        (0.8579, 0.3890)
        (0.8584, 0.3867)
        (0.8585, 0.3844)
        (0.8586, 0.3844)
        (0.8589, 0.3867)
        (0.8594, 0.3844)
        (0.8597, 0.3844)
        (0.8599, 0.3867)
        (0.8601, 0.3867)
        (0.8601, 0.3867)
        (0.8601, 0.3867)
        (0.8607, 0.3890)
        (0.8610, 0.3890)
        (0.8611, 0.3867)
        (0.8612, 0.3867)
        (0.8612, 0.3867)
        (0.8613, 0.3867)
        (0.8615, 0.3867)
        (0.8620, 0.3890)
        (0.8622, 0.3890)
        (0.8622, 0.3913)
        (0.8623, 0.3936)
        (0.8624, 0.3959)
        (0.8624, 0.3959)
        (0.8626, 0.3959)
        (0.8627, 0.3959)
        (0.8628, 0.3936)
        (0.8629, 0.3913)
        (0.8631, 0.3890)
        (0.8632, 0.3867)
        (0.8633, 0.3890)
        (0.8636, 0.3890)
        (0.8637, 0.3890)
        (0.8655, 0.3913)
        (0.8658, 0.3936)
        (0.8669, 0.3959)
        (0.8670, 0.3959)
        (0.8673, 0.3959)
        (0.8678, 0.3959)
        (0.8680, 0.3959)
        (0.8688, 0.3936)
        (0.8688, 0.3936)
        (0.8691, 0.3959)
        (0.8692, 0.3959)
        (0.8696, 0.3936)
        (0.8700, 0.3959)
        (0.8700, 0.3959)
        (0.8705, 0.3936)
        (0.8706, 0.3936)
        (0.8706, 0.3913)
        (0.8706, 0.3913)
        (0.8707, 0.3913)
        (0.8711, 0.3913)
        (0.8714, 0.3913)
        (0.8715, 0.3936)
        (0.8715, 0.3959)
        (0.8718, 0.3936)
        (0.8721, 0.3959)
        (0.8721, 0.3959)
        (0.8722, 0.3936)
        (0.8723, 0.3936)
        (0.8725, 0.3936)
        (0.8731, 0.3936)
        (0.8740, 0.3913)
        (0.8743, 0.3936)
        (0.8747, 0.3913)
        (0.8747, 0.3936)
        (0.8754, 0.3936)
        (0.8754, 0.3913)
        (0.8757, 0.3913)
        (0.8760, 0.3913)
        (0.8761, 0.3890)
        (0.8767, 0.3867)
        (0.8769, 0.3890)
        (0.8770, 0.3867)
        (0.8771, 0.3867)
        (0.8772, 0.3844)
        (0.8776, 0.3844)
        (0.8779, 0.3822)
        (0.8781, 0.3822)
        (0.8783, 0.3822)
        (0.8784, 0.3844)
        (0.8787, 0.3844)
        (0.8787, 0.3844)
        (0.8788, 0.3822)
        (0.8795, 0.3822)
        (0.8797, 0.3844)
        (0.8798, 0.3822)
        (0.8802, 0.3799)
        (0.8806, 0.3822)
        (0.8810, 0.3799)
        (0.8820, 0.3776)
        (0.8830, 0.3799)
        (0.8835, 0.3776)
        (0.8841, 0.3776)
        (0.8845, 0.3776)
        (0.8846, 0.3753)
        (0.8852, 0.3753)
        (0.8858, 0.3753)
        (0.8858, 0.3730)
        (0.8861, 0.3753)
        (0.8870, 0.3776)
        (0.8871, 0.3799)
        (0.8890, 0.3799)
        (0.8899, 0.3799)
        (0.8901, 0.3776)
        (0.8905, 0.3776)
        (0.8920, 0.3776)
        (0.8923, 0.3753)
        (0.8925, 0.3730)
        (0.8933, 0.3730)
        (0.8956, 0.3730)
        (0.8957, 0.3707)
        (0.8959, 0.3684)
        (0.8959, 0.3661)
        (0.8962, 0.3661)
        (0.8967, 0.3638)
        (0.8973, 0.3616)
        (0.8975, 0.3638)
        (0.8985, 0.3616)
        (0.8985, 0.3593)
        (0.8989, 0.3593)
        (0.8995, 0.3570)
        (0.9004, 0.3593)
        (0.9007, 0.3616)
        (0.9020, 0.3616)
        (0.9021, 0.3616)
        (0.9022, 0.3616)
        (0.9032, 0.3593)
        (0.9040, 0.3570)
        (0.9042, 0.3570)
        (0.9043, 0.3570)
        (0.9045, 0.3570)
        (0.9057, 0.3547)
        (0.9075, 0.3524)
        (0.9077, 0.3524)
        (0.9078, 0.3524)
        (0.9082, 0.3501)
        (0.9084, 0.3501)
        (0.9088, 0.3478)
        (0.9091, 0.3501)
        (0.9094, 0.3478)
        (0.9103, 0.3478)
        (0.9126, 0.3455)
        (0.9137, 0.3455)
        (0.9148, 0.3432)
        (0.9154, 0.3455)
        (0.9154, 0.3432)
        (0.9155, 0.3410)
        (0.9160, 0.3410)
        (0.9161, 0.3387)
        (0.9164, 0.3364)
        (0.9168, 0.3364)
        (0.9187, 0.3387)
        (0.9191, 0.3387)
        (0.9197, 0.3364)
        (0.9208, 0.3364)
        (0.9218, 0.3341)
        (0.9235, 0.3364)
        (0.9237, 0.3364)
        (0.9241, 0.3341)
        (0.9255, 0.3341)
        (0.9262, 0.3341)
        (0.9268, 0.3318)
        (0.9271, 0.3295)
        (0.9284, 0.3272)
        (0.9295, 0.3249)
        (0.9301, 0.3227)
        (0.9303, 0.3204)
        (0.9308, 0.3181)
        (0.9328, 0.3204)
        (0.9332, 0.3181)
        (0.9333, 0.3158)
        (0.9335, 0.3181)
        (0.9335, 0.3158)
        (0.9337, 0.3135)
        (0.9367, 0.3135)
        (0.9390, 0.3112)
        (0.9391, 0.3112)
        (0.9416, 0.3112)
        (0.9423, 0.3089)
        (0.9429, 0.3066)
        (0.9450, 0.3043)
        (0.9461, 0.3021)
        (0.9466, 0.2998)
        (0.9479, 0.2998)
        (0.9483, 0.2975)
        (0.9484, 0.2975)
        (0.9502, 0.2998)
        (0.9506, 0.2975)
        (0.9544, 0.2952)
        (0.9600, 0.2929)
        (0.9634, 0.2906)
        (0.9661, 0.2883)
        (0.9664, 0.2883)
        (0.9726, 0.2860)
        (0.9730, 0.2838)
        (0.9759, 0.2815)
        (0.9830, 0.2792)
        (1.0000, 0.2769)
	};
	\addlegendentry{top-1}
	\end{axis}
\end{tikzpicture}
\vspace{3mm}

\hspace{13mm} BERT-feat: triplet loss

\begin{tikzpicture}
	\begin{axis}[
	    xlabel=Threshold,
	    ylabel=Accuracy,
height=5cm,
width=7cm,
	]
	\addplot coordinates {
(0.3716, 0.4703)
(0.4218, 0.4703)
(0.4231, 0.4703)
(0.4297, 0.4703)
(0.4342, 0.4703)
(0.4368, 0.4703)
(0.4387, 0.4749)
(0.4395, 0.4772)
(0.4410, 0.4772)
(0.4416, 0.4795)
(0.4436, 0.4795)
(0.4456, 0.4863)
(0.4467, 0.4863)
(0.4485, 0.4863)
(0.4502, 0.4863)
(0.4518, 0.4863)
(0.4535, 0.4863)
(0.4544, 0.4863)
(0.4552, 0.4886)
(0.4564, 0.4886)
(0.4579, 0.4909)
(0.4584, 0.4932)
(0.4593, 0.4909)
(0.4603, 0.4863)
(0.4610, 0.4886)
(0.4614, 0.4909)
(0.4621, 0.4909)
(0.4624, 0.4932)
(0.4628, 0.4954)
(0.4633, 0.4977)
(0.4647, 0.4977)
(0.4654, 0.4977)
(0.4657, 0.5023)
(0.4662, 0.5023)
(0.4665, 0.5023)
(0.4672, 0.5023)
(0.4685, 0.5068)
(0.4689, 0.5068)
(0.4699, 0.5091)
(0.4702, 0.5091)
(0.4708, 0.5114)
(0.4712, 0.5114)
(0.4719, 0.5137)
(0.4721, 0.5183)
(0.4729, 0.5183)
(0.4731, 0.5183)
(0.4736, 0.5183)
(0.4741, 0.5183)
(0.4745, 0.5160)
(0.4751, 0.5160)
(0.4760, 0.5160)
(0.4762, 0.5183)
(0.4765, 0.5205)
(0.4768, 0.5205)
(0.4772, 0.5228)
(0.4783, 0.5251)
(0.4792, 0.5274)
(0.4798, 0.5274)
(0.4801, 0.5274)
(0.4808, 0.5320)
(0.4816, 0.5365)
(0.4821, 0.5388)
(0.4831, 0.5411)
(0.4837, 0.5434)
(0.4842, 0.5457)
(0.4846, 0.5457)
(0.4847, 0.5479)
(0.4851, 0.5479)
(0.4855, 0.5479)
(0.4858, 0.5479)
(0.4859, 0.5479)
(0.4863, 0.5502)
(0.4867, 0.5502)
(0.4871, 0.5502)
(0.4874, 0.5571)
(0.4876, 0.5594)
(0.4882, 0.5594)
(0.4885, 0.5616)
(0.4887, 0.5639)
(0.4889, 0.5662)
(0.4891, 0.5662)
(0.4894, 0.5662)
(0.4899, 0.5662)
(0.4905, 0.5639)
(0.4910, 0.5639)
(0.4912, 0.5639)
(0.4919, 0.5639)
(0.4923, 0.5639)
(0.4929, 0.5662)
(0.4931, 0.5662)
(0.4934, 0.5662)
(0.4940, 0.5662)
(0.4943, 0.5685)
(0.4945, 0.5685)
(0.4949, 0.5685)
(0.4954, 0.5708)
(0.4956, 0.5731)
(0.4959, 0.5731)
(0.4961, 0.5731)
(0.4964, 0.5753)
(0.4968, 0.5753)
(0.4970, 0.5753)
(0.4975, 0.5776)
(0.4978, 0.5776)
(0.4980, 0.5776)
(0.4985, 0.5799)
(0.4991, 0.5799)
(0.4995, 0.5799)
(0.5000, 0.5822)
(0.5004, 0.5845)
(0.5006, 0.5845)
(0.5012, 0.5890)
(0.5014, 0.5890)
(0.5015, 0.5890)
(0.5019, 0.5890)
(0.5022, 0.5890)
(0.5025, 0.5890)
(0.5028, 0.5913)
(0.5032, 0.5913)
(0.5036, 0.5913)
(0.5037, 0.5936)
(0.5040, 0.5936)
(0.5046, 0.5982)
(0.5051, 0.6005)
(0.5053, 0.6005)
(0.5055, 0.6005)
(0.5059, 0.6005)
(0.5062, 0.6027)
(0.5067, 0.6027)
(0.5069, 0.6027)
(0.5072, 0.6050)
(0.5078, 0.6050)
(0.5085, 0.6050)
(0.5090, 0.6027)
(0.5097, 0.6027)
(0.5102, 0.6050)
(0.5106, 0.6027)
(0.5110, 0.6027)
(0.5114, 0.6050)
(0.5117, 0.6096)
(0.5120, 0.6119)
(0.5122, 0.6142)
(0.5125, 0.6164)
(0.5129, 0.6164)
(0.5131, 0.6164)
(0.5133, 0.6164)
(0.5135, 0.6187)
(0.5138, 0.6164)
(0.5143, 0.6187)
(0.5146, 0.6210)
(0.5151, 0.6210)
(0.5153, 0.6210)
(0.5156, 0.6210)
(0.5166, 0.6210)
(0.5168, 0.6210)
(0.5175, 0.6210)
(0.5177, 0.6233)
(0.5182, 0.6233)
(0.5188, 0.6256)
(0.5191, 0.6256)
(0.5193, 0.6233)
(0.5196, 0.6279)
(0.5197, 0.6279)
(0.5200, 0.6279)
(0.5204, 0.6324)
(0.5208, 0.6324)
(0.5212, 0.6324)
(0.5216, 0.6324)
(0.5225, 0.6347)
(0.5227, 0.6370)
(0.5230, 0.6370)
(0.5233, 0.6370)
(0.5237, 0.6370)
(0.5238, 0.6370)
(0.5241, 0.6393)
(0.5247, 0.6393)
(0.5249, 0.6370)
(0.5251, 0.6370)
(0.5254, 0.6370)
(0.5258, 0.6370)
(0.5260, 0.6370)
(0.5264, 0.6370)
(0.5268, 0.6416)
(0.5273, 0.6416)
(0.5275, 0.6416)
(0.5278, 0.6461)
(0.5281, 0.6484)
(0.5285, 0.6461)
(0.5288, 0.6507)
(0.5293, 0.6507)
(0.5297, 0.6530)
(0.5301, 0.6530)
(0.5304, 0.6507)
(0.5310, 0.6507)
(0.5314, 0.6507)
(0.5316, 0.6507)
(0.5318, 0.6507)
(0.5319, 0.6507)
(0.5325, 0.6530)
(0.5328, 0.6553)
(0.5330, 0.6553)
(0.5332, 0.6575)
(0.5336, 0.6575)
(0.5338, 0.6575)
(0.5342, 0.6575)
(0.5345, 0.6575)
(0.5349, 0.6575)
(0.5353, 0.6575)
(0.5356, 0.6598)
(0.5359, 0.6621)
(0.5362, 0.6621)
(0.5364, 0.6621)
(0.5368, 0.6644)
(0.5374, 0.6621)
(0.5379, 0.6621)
(0.5381, 0.6621)
(0.5388, 0.6621)
(0.5392, 0.6667)
(0.5394, 0.6667)
(0.5396, 0.6644)
(0.5400, 0.6644)
(0.5403, 0.6644)
(0.5406, 0.6621)
(0.5410, 0.6621)
(0.5413, 0.6621)
(0.5418, 0.6621)
(0.5419, 0.6598)
(0.5422, 0.6598)
(0.5424, 0.6598)
(0.5432, 0.6598)
(0.5436, 0.6621)
(0.5441, 0.6621)
(0.5443, 0.6621)
(0.5445, 0.6644)
(0.5449, 0.6644)
(0.5451, 0.6644)
(0.5459, 0.6644)
(0.5461, 0.6644)
(0.5468, 0.6644)
(0.5471, 0.6644)
(0.5473, 0.6644)
(0.5475, 0.6667)
(0.5476, 0.6667)
(0.5478, 0.6667)
(0.5484, 0.6689)
(0.5487, 0.6689)
(0.5494, 0.6667)
(0.5501, 0.6667)
(0.5505, 0.6667)
(0.5509, 0.6667)
(0.5514, 0.6667)
(0.5517, 0.6667)
(0.5519, 0.6667)
(0.5524, 0.6667)
(0.5528, 0.6644)
(0.5528, 0.6644)
(0.5531, 0.6667)
(0.5538, 0.6667)
(0.5543, 0.6644)
(0.5546, 0.6667)
(0.5549, 0.6667)
(0.5551, 0.6667)
(0.5555, 0.6667)
(0.5559, 0.6667)
(0.5564, 0.6667)
(0.5573, 0.6667)
(0.5575, 0.6644)
(0.5581, 0.6644)
(0.5583, 0.6644)
(0.5585, 0.6644)
(0.5591, 0.6621)
(0.5598, 0.6621)
(0.5607, 0.6621)
(0.5610, 0.6621)
(0.5612, 0.6644)
(0.5618, 0.6644)
(0.5630, 0.6644)
(0.5635, 0.6621)
(0.5643, 0.6598)
(0.5651, 0.6598)
(0.5656, 0.6575)
(0.5660, 0.6553)
(0.5664, 0.6553)
(0.5671, 0.6530)
(0.5677, 0.6530)
(0.5681, 0.6530)
(0.5688, 0.6507)
(0.5691, 0.6484)
(0.5695, 0.6484)
(0.5700, 0.6484)
(0.5706, 0.6484)
(0.5709, 0.6461)
(0.5715, 0.6438)
(0.5718, 0.6438)
(0.5725, 0.6438)
(0.5730, 0.6416)
(0.5734, 0.6416)
(0.5741, 0.6416)
(0.5751, 0.6416)
(0.5758, 0.6416)
(0.5764, 0.6416)
(0.5768, 0.6416)
(0.5771, 0.6416)
(0.5773, 0.6416)
(0.5777, 0.6416)
(0.5783, 0.6416)
(0.5795, 0.6416)
(0.5798, 0.6416)
(0.5805, 0.6393)
(0.5810, 0.6393)
(0.5815, 0.6393)
(0.5832, 0.6393)
(0.5835, 0.6370)
(0.5844, 0.6370)
(0.5848, 0.6370)
(0.5852, 0.6347)
(0.5863, 0.6347)
(0.5865, 0.6347)
(0.5873, 0.6347)
(0.5880, 0.6347)
(0.5887, 0.6347)
(0.5891, 0.6347)
(0.5902, 0.6347)
(0.5904, 0.6301)
(0.5906, 0.6301)
(0.5915, 0.6301)
(0.5928, 0.6301)
(0.5931, 0.6256)
(0.5939, 0.6256)
(0.5942, 0.6233)
(0.5955, 0.6233)
(0.5962, 0.6210)
(0.5971, 0.6210)
(0.5979, 0.6210)
(0.5986, 0.6210)
(0.5994, 0.6187)
(0.6006, 0.6187)
(0.6013, 0.6187)
(0.6030, 0.6164)
(0.6040, 0.6164)
(0.6053, 0.6119)
(0.6062, 0.6119)
(0.6068, 0.6142)
(0.6078, 0.6142)
(0.6094, 0.6142)
(0.6102, 0.6142)
(0.6106, 0.6119)
(0.6124, 0.6119)
(0.6128, 0.6119)
(0.6143, 0.6119)
(0.6154, 0.6119)
(0.6157, 0.6096)
(0.6185, 0.6096)
(0.6193, 0.6096)
(0.6204, 0.6096)
(0.6214, 0.6096)
(0.6227, 0.6096)
(0.6241, 0.6073)
(0.6253, 0.6073)
(0.6263, 0.6073)
(0.6270, 0.6050)
(0.6285, 0.6050)
(0.6299, 0.6027)
(0.6310, 0.5982)
(0.6330, 0.5982)
(0.6347, 0.5982)
(0.6365, 0.5982)
(0.6370, 0.5936)
(0.6390, 0.5913)
(0.6400, 0.5890)
(0.6417, 0.5890)
(0.6434, 0.5845)
(0.6444, 0.5799)
(0.6455, 0.5776)
(0.6466, 0.5776)
(0.6475, 0.5776)
(0.6483, 0.5753)
(0.6491, 0.5753)
(0.6499, 0.5731)
(0.6510, 0.5708)
(0.6529, 0.5662)
(0.6547, 0.5639)
(0.6558, 0.5616)
(0.6588, 0.5571)
(0.6598, 0.5548)
(0.6625, 0.5548)
(0.6653, 0.5525)
(0.6681, 0.5502)
(0.6694, 0.5502)
(0.6709, 0.5434)
(0.6755, 0.5434)
(0.6779, 0.5388)
(0.6798, 0.5388)
(0.6829, 0.5342)
(0.6865, 0.5320)
(0.6891, 0.5320)
(0.6937, 0.5274)
(0.6964, 0.5251)
(0.6985, 0.5205)
(0.7003, 0.5137)
(0.7018, 0.5137)
(0.7037, 0.5091)
(0.7079, 0.5091)
(0.7120, 0.5091)
(0.7170, 0.5023)
(0.7239, 0.4977)
(0.7284, 0.4954)
(0.7317, 0.4886)
(0.7375, 0.4817)
(0.7405, 0.4749)
(0.7430, 0.4703)
(0.7471, 0.4635)
(0.7539, 0.4589)
(0.7592, 0.4543)
(0.7656, 0.4498)
(0.7711, 0.4452)
(0.7767, 0.4384)
(0.7894, 0.4315)
(0.8008, 0.4247)
(0.8047, 0.4178)
(0.8096, 0.4087)
(0.8153, 0.4018)
(0.8215, 0.3973)
(0.8257, 0.3881)
(0.8309, 0.3813)
(0.8407, 0.3767)
(0.8516, 0.3699)
(0.8570, 0.3584)
(0.8712, 0.3493)
(0.8896, 0.3425)
(0.9020, 0.3333)
(0.9073, 0.3265)
(0.9175, 0.3151)
(0.9276, 0.3059)
(0.9468, 0.2991)
(0.9515, 0.2877)
(1.0000, 0.2763)
(1.0000, 0.2603)
};
	\addlegendentry{top-5}
\addplot coordinates {
        (0.4541, 0.3311)
        (0.5117, 0.3311)
        (0.5119, 0.3333)
        (0.5129, 0.3333)
        (0.5145, 0.3356)
        (0.5216, 0.3379)
        (0.5230, 0.3402)
        (0.5273, 0.3402)
        (0.5318, 0.3425)
        (0.5333, 0.3425)
        (0.5345, 0.3425)
        (0.5346, 0.3425)
        (0.5351, 0.3447)
        (0.5359, 0.3470)
        (0.5379, 0.3493)
        (0.5379, 0.3516)
        (0.5381, 0.3516)
        (0.5384, 0.3516)
        (0.5441, 0.3539)
        (0.5443, 0.3539)
        (0.5448, 0.3539)
        (0.5461, 0.3539)
        (0.5471, 0.3539)
        (0.5474, 0.3562)
        (0.5475, 0.3562)
        (0.5478, 0.3562)
        (0.5501, 0.3562)
        (0.5503, 0.3562)
        (0.5520, 0.3584)
        (0.5524, 0.3584)
        (0.5526, 0.3562)
        (0.5527, 0.3562)
        (0.5533, 0.3562)
        (0.5541, 0.3562)
        (0.5548, 0.3584)
        (0.5555, 0.3584)
        (0.5591, 0.3607)
        (0.5605, 0.3630)
        (0.5607, 0.3653)
        (0.5610, 0.3676)
        (0.5613, 0.3699)
        (0.5625, 0.3699)
        (0.5630, 0.3699)
        (0.5634, 0.3721)
        (0.5636, 0.3721)
        (0.5643, 0.3744)
        (0.5652, 0.3767)
        (0.5652, 0.3767)
        (0.5679, 0.3767)
        (0.5688, 0.3767)
        (0.5698, 0.3790)
        (0.5716, 0.3813)
        (0.5717, 0.3836)
        (0.5721, 0.3836)
        (0.5722, 0.3836)
        (0.5727, 0.3813)
        (0.5730, 0.3813)
        (0.5730, 0.3836)
        (0.5734, 0.3858)
        (0.5749, 0.3858)
        (0.5751, 0.3881)
        (0.5761, 0.3881)
        (0.5764, 0.3881)
        (0.5764, 0.3904)
        (0.5767, 0.3904)
        (0.5770, 0.3904)
        (0.5771, 0.3927)
        (0.5775, 0.3927)
        (0.5777, 0.3950)
        (0.5794, 0.3973)
        (0.5795, 0.3973)
        (0.5795, 0.3973)
        (0.5802, 0.3973)
        (0.5810, 0.3995)
        (0.5811, 0.3995)
        (0.5815, 0.3995)
        (0.5817, 0.4018)
        (0.5817, 0.4041)
        (0.5834, 0.4041)
        (0.5835, 0.4041)
        (0.5844, 0.4041)
        (0.5848, 0.4041)
        (0.5851, 0.4018)
        (0.5852, 0.4018)
        (0.5865, 0.4018)
        (0.5865, 0.4041)
        (0.5879, 0.4064)
        (0.5880, 0.4087)
        (0.5884, 0.4087)
        (0.5890, 0.4087)
        (0.5900, 0.4110)
        (0.5902, 0.4110)
        (0.5904, 0.4110)
        (0.5906, 0.4110)
        (0.5915, 0.4132)
        (0.5927, 0.4132)
        (0.5931, 0.4132)
        (0.5934, 0.4155)
        (0.5939, 0.4155)
        (0.5940, 0.4132)
        (0.5941, 0.4155)
        (0.5942, 0.4178)
        (0.5947, 0.4178)
        (0.5959, 0.4178)
        (0.5962, 0.4155)
        (0.5968, 0.4178)
        (0.5971, 0.4178)
        (0.5976, 0.4178)
        (0.5978, 0.4178)
        (0.5983, 0.4201)
        (0.5985, 0.4201)
        (0.5987, 0.4178)
        (0.5992, 0.4201)
        (0.5994, 0.4224)
        (0.5995, 0.4224)
        (0.6009, 0.4224)
        (0.6013, 0.4224)
        (0.6015, 0.4224)
        (0.6016, 0.4201)
        (0.6034, 0.4201)
        (0.6046, 0.4178)
        (0.6068, 0.4178)
        (0.6071, 0.4178)
        (0.6074, 0.4178)
        (0.6080, 0.4201)
        (0.6082, 0.4201)
        (0.6088, 0.4201)
        (0.6094, 0.4201)
        (0.6095, 0.4201)
        (0.6102, 0.4224)
        (0.6103, 0.4224)
        (0.6106, 0.4224)
        (0.6109, 0.4224)
        (0.6113, 0.4247)
        (0.6121, 0.4269)
        (0.6124, 0.4292)
        (0.6128, 0.4292)
        (0.6134, 0.4315)
        (0.6137, 0.4338)
        (0.6143, 0.4361)
        (0.6144, 0.4361)
        (0.6148, 0.4384)
        (0.6152, 0.4406)
        (0.6155, 0.4406)
        (0.6156, 0.4406)
        (0.6187, 0.4406)
        (0.6189, 0.4406)
        (0.6193, 0.4429)
        (0.6197, 0.4452)
        (0.6200, 0.4475)
        (0.6204, 0.4498)
        (0.6209, 0.4521)
        (0.6212, 0.4521)
        (0.6213, 0.4543)
        (0.6216, 0.4543)
        (0.6220, 0.4543)
        (0.6227, 0.4566)
        (0.6227, 0.4589)
        (0.6243, 0.4589)
        (0.6263, 0.4589)
        (0.6264, 0.4589)
        (0.6264, 0.4589)
        (0.6269, 0.4589)
        (0.6275, 0.4612)
        (0.6276, 0.4612)
        (0.6295, 0.4589)
        (0.6299, 0.4566)
        (0.6315, 0.4566)
        (0.6324, 0.4566)
        (0.6330, 0.4589)
        (0.6357, 0.4589)
        (0.6365, 0.4566)
        (0.6366, 0.4589)
        (0.6368, 0.4566)
        (0.6372, 0.4566)
        (0.6387, 0.4589)
        (0.6400, 0.4566)
        (0.6410, 0.4566)
        (0.6413, 0.4566)
        (0.6417, 0.4566)
        (0.6417, 0.4543)
        (0.6431, 0.4543)
        (0.6437, 0.4521)
        (0.6437, 0.4498)
        (0.6437, 0.4498)
        (0.6439, 0.4498)
        (0.6445, 0.4521)
        (0.6445, 0.4521)
        (0.6449, 0.4521)
        (0.6455, 0.4543)
        (0.6455, 0.4521)
        (0.6458, 0.4521)
        (0.6459, 0.4543)
        (0.6466, 0.4566)
        (0.6467, 0.4566)
        (0.6468, 0.4589)
        (0.6474, 0.4589)
        (0.6475, 0.4612)
        (0.6476, 0.4612)
        (0.6487, 0.4635)
        (0.6490, 0.4658)
        (0.6491, 0.4658)
        (0.6493, 0.4658)
        (0.6498, 0.4680)
        (0.6499, 0.4658)
        (0.6502, 0.4680)
        (0.6505, 0.4680)
        (0.6507, 0.4680)
        (0.6510, 0.4658)
        (0.6518, 0.4635)
        (0.6529, 0.4612)
        (0.6534, 0.4612)
        (0.6536, 0.4612)
        (0.6537, 0.4635)
        (0.6547, 0.4635)
        (0.6554, 0.4612)
        (0.6554, 0.4612)
        (0.6558, 0.4612)
        (0.6559, 0.4589)
        (0.6569, 0.4566)
        (0.6582, 0.4566)
        (0.6588, 0.4566)
        (0.6590, 0.4566)
        (0.6593, 0.4589)
        (0.6595, 0.4612)
        (0.6607, 0.4612)
        (0.6610, 0.4612)
        (0.6613, 0.4612)
        (0.6625, 0.4635)
        (0.6626, 0.4612)
        (0.6642, 0.4635)
        (0.6653, 0.4635)
        (0.6658, 0.4658)
        (0.6658, 0.4680)
        (0.6670, 0.4703)
        (0.6670, 0.4680)
        (0.6694, 0.4680)
        (0.6695, 0.4658)
        (0.6706, 0.4635)
        (0.6709, 0.4612)
        (0.6722, 0.4635)
        (0.6744, 0.4658)
        (0.6748, 0.4680)
        (0.6761, 0.4658)
        (0.6777, 0.4658)
        (0.6779, 0.4658)
        (0.6788, 0.4658)
        (0.6796, 0.4680)
        (0.6798, 0.4703)
        (0.6801, 0.4703)
        (0.6808, 0.4726)
        (0.6821, 0.4749)
        (0.6829, 0.4726)
        (0.6833, 0.4749)
        (0.6851, 0.4749)
        (0.6865, 0.4772)
        (0.6866, 0.4772)
        (0.6873, 0.4772)
        (0.6880, 0.4772)
        (0.6882, 0.4772)
        (0.6891, 0.4772)
        (0.6892, 0.4749)
        (0.6900, 0.4726)
        (0.6912, 0.4726)
        (0.6934, 0.4726)
        (0.6937, 0.4749)
        (0.6940, 0.4749)
        (0.6954, 0.4749)
        (0.6964, 0.4726)
        (0.6975, 0.4726)
        (0.6975, 0.4703)
        (0.6985, 0.4703)
        (0.6994, 0.4703)
        (0.6999, 0.4680)
        (0.7000, 0.4658)
        (0.7003, 0.4635)
        (0.7009, 0.4635)
        (0.7010, 0.4635)
        (0.7023, 0.4612)
        (0.7031, 0.4612)
        (0.7032, 0.4612)
        (0.7037, 0.4589)
        (0.7043, 0.4612)
        (0.7055, 0.4635)
        (0.7066, 0.4658)
        (0.7104, 0.4658)
        (0.7120, 0.4658)
        (0.7153, 0.4635)
        (0.7155, 0.4612)
        (0.7164, 0.4589)
        (0.7170, 0.4589)
        (0.7177, 0.4589)
        (0.7187, 0.4566)
        (0.7214, 0.4566)
        (0.7216, 0.4589)
        (0.7239, 0.4566)
        (0.7248, 0.4543)
        (0.7266, 0.4543)
        (0.7268, 0.4543)
        (0.7284, 0.4566)
        (0.7287, 0.4543)
        (0.7292, 0.4521)
        (0.7309, 0.4521)
        (0.7313, 0.4521)
        (0.7317, 0.4498)
        (0.7336, 0.4521)
        (0.7340, 0.4543)
        (0.7361, 0.4521)
        (0.7374, 0.4498)
        (0.7375, 0.4475)
        (0.7396, 0.4452)
        (0.7404, 0.4475)
        (0.7404, 0.4452)
        (0.7405, 0.4429)
        (0.7412, 0.4429)
        (0.7417, 0.4406)
        (0.7430, 0.4384)
        (0.7437, 0.4361)
        (0.7447, 0.4361)
        (0.7454, 0.4338)
        (0.7467, 0.4338)
        (0.7471, 0.4315)
        (0.7497, 0.4338)
        (0.7510, 0.4338)
        (0.7524, 0.4315)
        (0.7539, 0.4315)
        (0.7554, 0.4315)
        (0.7581, 0.4292)
        (0.7583, 0.4269)
        (0.7600, 0.4269)
        (0.7607, 0.4269)
        (0.7617, 0.4247)
        (0.7655, 0.4224)
        (0.7661, 0.4247)
        (0.7692, 0.4224)
        (0.7708, 0.4201)
        (0.7711, 0.4224)
        (0.7730, 0.4224)
        (0.7732, 0.4201)
        (0.7759, 0.4178)
        (0.7767, 0.4155)
        (0.7767, 0.4178)
        (0.7797, 0.4155)
        (0.7798, 0.4132)
        (0.7827, 0.4132)
        (0.7859, 0.4132)
        (0.7894, 0.4110)
        (0.7897, 0.4087)
        (0.7947, 0.4064)
        (0.7952, 0.4064)
        (0.8006, 0.4041)
        (0.8017, 0.4018)
        (0.8018, 0.4018)
        (0.8043, 0.3995)
        (0.8047, 0.3973)
        (0.8058, 0.3950)
        (0.8080, 0.3927)
        (0.8089, 0.3904)
        (0.8095, 0.3881)
        (0.8096, 0.3881)
        (0.8122, 0.3858)
        (0.8130, 0.3858)
        (0.8146, 0.3836)
        (0.8148, 0.3813)
        (0.8153, 0.3813)
        (0.8157, 0.3790)
        (0.8187, 0.3790)
        (0.8215, 0.3767)
        (0.8220, 0.3744)
        (0.8223, 0.3721)
        (0.8253, 0.3699)
        (0.8256, 0.3676)
        (0.8257, 0.3699)
        (0.8267, 0.3721)
        (0.8272, 0.3699)
        (0.8282, 0.3699)
        (0.8308, 0.3676)
        (0.8333, 0.3653)
        (0.8343, 0.3653)
        (0.8409, 0.3630)
        (0.8477, 0.3607)
        (0.8516, 0.3584)
        (0.8518, 0.3562)
        (0.8545, 0.3539)
        (0.8549, 0.3516)
        (0.8562, 0.3493)
        (0.8570, 0.3470)
        (0.8586, 0.3447)
        (0.8624, 0.3425)
        (0.8667, 0.3402)
        (0.8687, 0.3379)
        (0.8770, 0.3356)
        (0.8795, 0.3379)
        (0.8832, 0.3356)
        (0.8881, 0.3333)
        (0.8896, 0.3356)
        (0.8926, 0.3333)
        (0.8956, 0.3311)
        (0.8999, 0.3311)
        (0.9012, 0.3288)
        (0.9040, 0.3265)
        (0.9043, 0.3265)
        (0.9059, 0.3242)
        (0.9073, 0.3219)
        (0.9099, 0.3196)
        (0.9134, 0.3174)
        (0.9162, 0.3151)
        (0.9162, 0.3128)
        (0.9175, 0.3105)
        (0.9181, 0.3082)
        (0.9186, 0.3059)
        (0.9187, 0.3059)
        (0.9201, 0.3037)
        (0.9276, 0.3014)
        (0.9312, 0.2991)
        (0.9344, 0.2968)
        (0.9367, 0.2968)
        (0.9431, 0.2945)
        (0.9468, 0.2968)
        (0.9474, 0.2945)
        (0.9477, 0.2922)
        (0.9485, 0.2900)
        (0.9514, 0.2877)
        (0.9515, 0.2854)
        (0.9517, 0.2831)
        (0.9521, 0.2808)
        (0.9628, 0.2785)
        (0.9665, 0.2763)
        (1.0000, 0.2740)
        (1.0000, 0.2671)
        (1.0000, 0.2603)
	};
	\addlegendentry{top-1}
	\end{axis}
\end{tikzpicture}

\caption{
Question clustering accuracy for $k$-NN and triplet loss models at different thresholds.
If a given test question had a similarity that was less than the threshold, then it was classified as a novel question (i.e., not in the database of known questions).
When the threshold was too high, performance dropped because too many questions were classified as novel.
When the threshold was too low, performance dropped because the model attempted to match too many test questions to existing clusters in the database.
}
\vspace{-4mm}
\label{fig:clustering}
\end{figure}

\subsection{Question-Category Classification Error Analysis} 
Figure \ref{fig:heatmap} shows the confusion matrix for our SVM classifier on the question-category classification task on the test set of real questions. 
Categories that were challenging to distinguish were \emph{Transmission} and \emph{Having COVID} (34\% error rate), and \emph{Having COVID} and \emph{Symptoms} (33\% error rate). 

\subsection{Further Dataset Details} 
\vspace{0.5em} \noindent \textbf{Question mismatches.}
Table \ref{tab:missing_faq} shows example questions from at least two non-official sources that went unanswered by an official source.
Table \ref{tab:unmatched_questions} shows example questions from the FDA and CDC FAQ websites that did not ask the same thing as any other questions in our dataset. 
\begin{table}[h]
    \centering
    \small
    \setlength{\tabcolsep}{2pt}
    \begin{tabular}{l c c}
        \toprule
        Question Cluster & $N_{cluster}$ & Example Questions \\
        \midrule
        \multirow{3}{*}{Number of Cases} & \multirow{3}{*}{21} & ``Are COVID cases dropping?"\\
        & & ``Have COVID cases peaked?"\\
        & & ``Are COVID cases decreasing?"\\
        \midrule
        \multirow{3}{*}{Mutation} & \multirow{3}{*}{19} & ``Has COVID mutated?"\\
        & & ``Did COVID mutate?"\\
        & & ``Will COVID mutate?"\\
        \midrule
        \multirow{3}{*}{Lab Theory} & \multirow{3}{*}{18} & ``Was COVID made in a lab?"\\
        & & ``Was COVID manufactured?"\\
        & & ``Did COVID start in a lab?"\\
        \bottomrule
    \end{tabular}
    \caption{Questions appearing in multiple sources that were unanswered by official FAQ websites.}
    \label{tab:missing_faq}
\end{table}


\noindent \textbf{Example questions.} Table \ref{tab:representative_examples} shows example questions from each of the 15 question categories.
\vspace{0.5em} \noindent \textbf{Corresponding answers.}
The FAQ websites from reputable sources (denoted with $^*$ in Table \ref{tab:dataset_table}) provide answers to their questions, and so we also provide them as an auxiliary resource. 
Using these answers, 23.8\% of question clusters have at least one corresponding answer. 
We caution against using these answers in applied settings, however, because information on COVID changes rapidly.

\vspace{0.5em} \noindent \textbf{Additional data collection details.} 
In terms of how questions about COVID were determined, for FAQ websites from official organizations, we considered all questions, and for Google, Bing, Yahoo, and Quora, we searched the keywords ``COVID" and ``coronavirus." 

As for synonymous ways of saying COVID, we considered ``SARS-COV-2," ``coronavirus," ``2019-nCOV," ``COVID-19," and ``COVID19."
\vspace{0.5em} \noindent \textbf{Other COVID-19 datasets.}
We encourage researchers
to also explore other COVID-19 datasets: tweets streamed since January 22 \cite{Chen2020COVID19TF}, 
location-tagged tweets in 65 languages \cite{AbdulMageed2020MegaCOVAB},
tweets of COVID symptoms \cite{Sarker2020SelfreportedCS},
a multi-lingual Twitter and Weibo dataset \cite{Gao2020NAISTCM},
an Instagram dataset \cite{Zarei2020AFI},
emotional responses to COVID \cite{Kleinberg2020MeasuringEI},
and annotated research abstracts \cite{Huang2020CODA19RA}. 
\begin{figure*}[ht]
    \centering
    \includegraphics{figures/heatmap.png}
    \caption{Confusion matrix for BERT-feat: SVM predictions on the question-category classification task.}
    \label{fig:heatmap}
\end{figure*}
\begin{table*}[hbtp]
    \centering
    \setlength{\tabcolsep}{1.5pt}
    \small
    \begin{tabular}{l | l}
         \toprule
         \multicolumn{2}{c}{Food and Drug Administration}\\
         \multicolumn{1}{c}{Question} & \multicolumn{1}{c}{Closest Matches from BERT} \\
         \midrule
         \multirow{3}{*}{\begin{minipage}{1.4in} ``Can I donate\\ convalescent plasma?" \end{minipage}} & ``Why is convalescent plasma being investigated to treat COVID?"\\
         & ``Can I make my own hand sanitizer?"\\
         & ``What are suggestions for things to do in the COVID quarantine?"\\
         \midrule
         \multirow{3}{*}{\begin{minipage}{1.4in} ``Where can I report websites selling fraudulent medical products?"\end{minipage}} & ``What kind of masks are recommended to protect healthcare workers from COVID exposure?"\\
         & ``Where can I get tested for COVID?"\\
         & ``How do testing kits for COVID detect the virus?"\\
         \toprule
         \multicolumn{2}{c}{Center for Disease Control}\\
         \multicolumn{1}{c}{Question} & \multicolumn{1}{c}{Closest Matches from BERT} \\
         \midrule
        \multirow{3}{*}{\begin{minipage}{1.30in} ``What is the difference\\ between cleaning and\\ disinfecting?"\end{minipage}} & ``How effective are alternative disinfection methods?"\\
         & ``Why has Trump stated that injecting disinfectant will kill COVID in a minute?"\\
         & ``Should I spray myself or my kids with disinfectant?"\\
         \midrule
         \multirow{3}{*}{\begin{minipage}{1.5in} ``How frequently should facilities be cleaned to reduce the potential spread of COVID?"\end{minipage}} & ``What is the survival rate of those infected by COVID who are put on a ventilator?"\\
         & ``What kind of masks are recommended to protect healthcare workers from COVID exposure?"\\
         & ``Will warm weather stop the outbreak of COVID?"\\
         \bottomrule
    \end{tabular}
    \caption{Questions from the Food and Drug Administration (FDA) and Center for Disease Control (CDC) FAQ websites that did not ask the same thing as any questions from other sources.}
    \label{tab:unmatched_questions}
\end{table*}
\begin{table*}[ht]
\centering
\small
    \begin{tabular}{l | l}
         \toprule
         Category & Example Questions\\
         \midrule
         \multirow{3}{*}{Transmission} & ``Can COVID spread through food?"\\
         & ``Can COVID spread through water?"\\
         & ``Is COVID airborne?"\\
         \midrule
         \multirow{3}{*}{Societal Effects} & ``In what way have people been affected by COVID?"\\
         & ``How will COVID change the world?"\\
         & ``Do you think there will be more racism during COVID?"\\
         \midrule
         \multirow{3}{*}{Prevention} & ``Should I wear a facemask?"\\
         & ``How can I prevent COVID?"\\
         & ``What disinfectants kill the COVID virus?"\\
         \midrule
         \multirow{3}{*}{Societal Response} & ``Have COVID checks been issued?"\\
         & ``What are the steps that a hospital should take after COVID outbreak?"\\
         & ``Are we blowing COVID out of proportion?"\\
         \midrule
         \multirow{3}{*}{Reporting} & ``Is COVID worse than we are being told?"\\
         & ``What is the COVID fatality rate?"\\
         & ``What is the most reliable COVID model right now?"\\
         \midrule
         \multirow{3}{*}{Origin} & ``Where did COVID originate?"\\
         & ``Did COVID start in a lab?"\\
         & ``Was COVID a bioweapon?"\\
         \midrule
         \multirow{3}{*}{Treatment} & ``What treatments are available for COVID?"\\
         & ``Should COVID patients be ventilated?"\\
         & ``Should I spray myself or my kids with disinfectant?"\\
         \midrule
         \multirow{3}{*}{Speculation} & ``Was COVID predicted?"\\
         & ``Will COVID return next year?"\\
         & ``How long will we be on lockdown for COVID?"\\
         \midrule
         \multirow{3}{*}{Economic Effects} & ``What is the impact of COVID on the global economy?"\\
         & ``What industries will never be the same because of COVID?"\\
         & ``Why are stock markets dipping in response to COVID?"\\
         \midrule
         \multirow{3}{*}{Individual Response} & ``How do I stay positive with COVID?"\\
         & ``What are suggestions for things to do in the COVID quarantine?"\\
         & ``Can I still travel?"\\
         \midrule
         \multirow{3}{*}{Comparison} & ``How are COVID and SARS-COV similar?"\\
         & ``How can I tell if I have the flu or COVID?"\\
         & ``How does COVID compare to other viruses?"\\
         \midrule
         \multirow{3}{*}{Testing} & ``How COVID test is done?"\\
         & ``Are COVID tests accurate?"\\
         & ``Should I be tested for COVID?"\\
         \midrule
         \multirow{3}{*}{Nomenclature} & ``Should COVID be capitalized?"\\
         & ``What COVID stands for?"\\
         & ``What is the genus of the SARS-COVID?"\\
         \midrule
         \multirow{3}{*}{Having COVID} & ``How long does it take to recover?"\\
         & ``How COVID attacks the body?"\\
         & ``How long is the incubation period for COVID?"\\
         \midrule
         \multirow{3}{*}{Symptoms} & ``What are the symptoms of COVID?"\\
         & ``Which COVID symptoms come first?"\\
         & ``Do COVID symptoms come on quickly?"\\
         \bottomrule
    \end{tabular}
    \caption{Sample questions from each of the 15 question categories.}
    \label{tab:representative_examples}
\end{table*}
\clearpage
\end{document}


Peer reviews:

content: confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct
rating: 5: Marginally below acceptance threshold
review: This paper presents a dataset of 1700 questions related to covid which
are hand labeled and divided into 15 general categories and 207
clusters of semantic equivalence. The value is potentially useful for
general question classification, for semantic similarity, with
particular application to reducing load on question answerers by
removing redundancy, The data set is on the small side and some
important details about how the data was collection are omitted. The
authors make a number of factual errors. All of these could be easily
corrected and the data set is a useful resource.


"we scraped questions about covid". how is a 'question about covid'
determined? Are keywords used? If so what keywords? Additionally how
were questions that had location/time-specific versions vs. questions
with only one version determined? There are number of ways this could
be done, some noisier than others, some more scalable than others.

What are all the 'synonymous ways of saying COVID?'

same answer -> same question cluster. "In which country is SARS-CoV-2
believed to have originated" and "Which country manufactures the most
face masks" have the same answer but are not the same question nor are
even related. Plenty of useful questions do not have an answer (yet);
how are these to be clustered?

"there are fewer ways to ask how long COVID will last than ways to
write a positive movie review" -- I would argue both are countably
infinite. 

The cluster classification task was oddly formed. This is ultimately a
sentence similarity task or a coreference/nil clustering task. One
could use the data to pose the binary question "are these two
sentences asking the same question?" One could also posit the far more
useful "is this a question that has already been asked [and if so
which one] or is this a novel question?" task of coref/nil
clustering. Static assignment to clusters seems wrong for that kind of
data. By specifically excluding clusters with a small number of
questions you specifically skirt the issue you would have to deal with
in real application of this data.




title: timely contribution, could be better positioned with regard to previous work

content: confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct
rating: 7: Good paper, accept
review: The paper presents a dataset of questions related to COVID-19; included are annotations mapping questions to specific categories and classes.

It's always good to see papers that contribute datasets to the research community, particularly if they require significant efforts in annotation as this one does.

I think that some changes are needed in terms of positioning of this paper to make clear the contributions and benefits. The other reviewer, I believe, also struggled a little with this. I suggest an explicit "who would use this data" and "how would this dataset be used" section. I think limiting it to just people who want to train an QA IR system is a mistake. First, not all question have answers (Table 1); second, it's not obvious how the classes and categories focus, which are a large part of the paper, is core to training a QA IR system. Instead, I think the classes and categories can be used a lot more in the query intent / query understanding area of research. Pitching the paper more broadly - and being explicit on areas - would help to clarify and strengthen the contributions.

On the methodology of annotation:
- The paper rests on the technical soundness of the manual annotation process so the authors should really focus on ensuring the methodology / rigour is clear - especially as all the annotation work was done by the authors themselves. 
- Categories were made up by the author. How subjective and context specific was this? What are the limitations?
- Seems like one question is mapped to one class and one category - what about overlapping questions? Surely there are questions that fall in multiple - and if there are not in this dataset then there will certainly be out in the wild. The paper, at leasts, needs to address this.
- Categories may remain static (i.e., new questions about covid-19 would likely map to existing categories). But classes will be ever evolving. For example, if a vaccine is developed then a whole host of new question classes will arise. Having these static list of classes, at this snapshot in time, risks them becoming out-of-date very quickly in a rapidly changing environment.
- Figure 2 - questions/class highly skewed. So is it actually worth having the classes? How valuable are they?
- Having an explicit limitation section at the end of the paper will really help with the above points.


Some changes in presentation could really elevate the paper:
- The definition and difference between categories and classes was not clear at the beginning of the paper. In the middle section 2, the definition finally becomes clear. A definition of both is really needed earlier in the intro. 
- I found “classes” not really the best term. What about question “topics” or even question “clusters” since they are essential groups of duplicate questions. When classes are introduced, there needs to be a better justification for their purpose. It comes clearer much later but I think is needed earlier.
- Matched vs. unmatched is confusing. It’s not that they were not matched, it’s just that they were classes with only one question, right? Can a better term found? Also Table 1 presents matched/unmatched without any reference / definition - only much later in the paper do you find out.
- At some point, the paper suddenly starts talking about “labels” - it’s not clear whether this is classes or categories or both - why not just use the actual terms.

On the classifier / experiments
- The contribution of the paper is the dataset and not the ML methods. As stated, in the response to the review, the performance is merely an indication of how well current methods do on this task. The fact that the performance is good or bad does not really change the papers worth. However, insights from the experiments are part of the contribution; and these could be improved: e.g., were certain categories much harder than others? where two categories often confused for each other? 

Overall, this paper provided a good dataset for the community. But it needs to be much more explicit and clear about how people might use the dataset rather than a generic reference to training QA IR system. Addressing the presentation issues and directly addressing a number of limitations will greatly improve the paper.

title: A good dataset paper - some clarification and statement of limitation

content: confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct
rating: 5: Marginally below acceptance threshold
review: This paper presents a corpus of 1690 documents about COVID-19 manually annotated with questions classes, along with basic question classification algorithms.

Pros
- in scope for the conference
- dataset is publicly available
- technical details are kept to a minimum
- accessible to a large audience

Cons
- no clear statement of objectives
- no clear statement of specific contribution
- no explicit annotation guidelines (making it difficult to reproduce this effort)
- ad hoc categories (not modeled after general types of questions)
- no justification for the selection of documents
- mediocre performance of the basic question classification algorithms
- no discussion (the so-called discussion section is merely a summary)
- the claim that this dataset can help train QA systems is unwarranted at this stage

Overall
The significance of this preliminary work is extremely limited

Other comments
- the presentation of figure 2 is unnecessarily confusing
title: Preliminary work, limited significance