RealSakaiLab at the TREC 2020 Health Misinformation TrackSijie TAO† and Tetsuya SAKAI†† Department of Computer Science and Engineering, Waseda University
3–4–1 Okubo, Shinjuku-ku, Tokyo 169–8555, Japan
E-mail: †tsjmailbox@ruri.waseda.jp, †tetsuyasakai@acm.org
Abstract In this paper, we describe our experiments conducted for the AdHoc Retrieval task of the TREC 2020
Health Misinformation Track. This task oﬀers a challenges to participants to design a ranking model that promotes
retrieval of both credible and correct health information. To address both relevance and credibility, we combined
several techniques to re-rank a BM25 baseline ranking. The results from a language identification model, a news
category classifier and a majority score calculation were used to modify the BM25 scores of the baseline ranking.

1 Introduction

2.1 Language Identification
Although the corpus used in this task contains documents

Incorrect information in a search engine results pagein diﬀerent languages, non-English documents are considered(SERP) can be detrimental. Previous research has shown

irrelevant in this track. Therefore, it is necessary to detectthat a bias towards misinformation in SERP can increase thethe language in which a document is prepared and filter thepossibility of the users making incorrect choices in a decision-

non-English documents from the ranking. A pre-trained lan-making task [2].

guage identification model using fastText [3] [4]1 is used toThe AdHoc Retrieval task of the TREC 2020 Health Mis-

identify the languages in which a document is prepared.

information Track aims to provide a venue for participants

2.2 News Category Classification

to design ranking models to retrieve information that is both

To rank more documents that contain health information,credible and correct. Participants are required to develop ap-

we trained a classifier to detect the categories of the docu-proaches to rank not only relevant but also credible and cor-ments’ content. As all documents are news articles, a news

rect documents over those with incorrect information. This

category classifier was trained on an external dataset named

year, the track focuses on the pandemic, and hence, the top-

News Category Dataset2. This dataset contains 200,000 news

ics are all related to COVID-19.

headlines and each headline is assigned to a corresponding

We submitted four runs to the track. One is a BM25 rank-category. There are 41 categories, and we chose four cate-ing as the baseline, and the other three runs are re-ranked

gories, politics, science & tech, wellness, and world news asbased on the BM25 ranking scores. Several techniques are“relevant categories”, that could contain health-related in-combined in the re-ranking runs. First, as non-English doc-formation about COVID-19. If a document is classified un-uments are considered irrelevant in this track, we utilized ader one of the relevant categories, then the result from thelanguage identification model to filter out documents thatclassifier will be used to boost the ranking of the document.

are not in English. Further, to boost the rankings of the2.3 Majority Score

documents that may contain health information, we trained

As the task requires participants to retrieve credible infor-

a news category classifier to detect documents that have rel-mation, we address the credibility of a document by calculat-evant contents. Next, the credibility of a document is mod-ing a “Majority Score”. This idea is based on a hypothesis

eled by calculating a majority score, which is based on its

that “the more similar a document is to others, the moresimilarities with other documents. The results from the lan-likely the document is credible”. For each topic, the baseline

guage identification model, the news category classifier, andBM25 ranking returns 1,000 documents. The Majority Score

the majority score calculation are used to modify the BM25

of a retrieved document is calculated as the average of thescores of the baseline ranking.

similarity values with the other 999 documents. For calculat-2 Submitted Runs
In this section, details of the submitted runs are discussed.

ing document similarity, the documents are first represented

1：https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin
2：https://www.kaggle.com/rmisra/news-category-dataset

Runs

cam map three nDCG

Compatibility

Compatibility(harmful only)(helpful only)

RSL BM25

0.139

0.318

0.048

0.217RSL BM25LC

0.139

0.331

0.067

0.239

RSL BM25LM

0.138

0.319

0.045

0.257

RSL BM25LMC

0.148

0.338

0.070

0.268

0.139
0.331
0.075
Table 1 Evaluation results

0.334

Medianas tf-idf vectors, and the similarity value of two documents

method can retrieve more credible and correct documents.

is defined as the cosine similarity of the document represen-

Although the combined run RSL BM25LMC slightly out-tations.

performs the other runs, the run RSL BM25LM does not

2.4 Runs

produce better results than the BM25 baseline run or theWe submitted four runs to the track. The first, RSL BM25

RSL BM25LC in terms of cam map three and nDCG. There-is a baseline BM25 ranking, automatically generated byfore, from the evaluation results, it is not clear whether re-3

Anserini [6] with default parameter settings.

ranking using majority score can promote retrieval of more

The second run is named RSL BM25LC, where L stands

credible and correct documents. Re-ranking using majority

for Language Identification and C stands for Category Clas-score does not work as expected because using tf-idf may notsification. The BM25 scores are set to 0 if the detected lan-be eﬀective in capturing document similarity.

guage is not English. For English documents, the BM25
scores are modified as the following equation if they are classified under a relevant category:
RSL BM 25LC(d) = BM 25(d) ∗ (1 + P (d))

4 Conclusions
In this paper, we described our participation in the TREC

(1)

2020 Health Misinformation Track. We used several techniques to build our runs: a language identification model,

where P (d) is the output of the news category classifier for aa news category classifier, and a majority score calculation

given document d, which is the probability of the document

to modify the BM25 baseline ranking. The evaluation re-being classified under a relevant category (See Section 2.2).

sults show that our method cannot significantly impact theThe third run RSL BM25LM (Language Identification andretrieval of correct and credible documents. In future work,

Majority Score) re-ranks documents by boosting the BM25

we hope to find alternative ways for better identification ofscores using the majority scores:

correct and credible documents to help users avoid making
erroneous decisions.

RSL BM 25LM (d) = BM 25(d)∗(1+M ajorityScore(d))(2)
The last run RSL BM25LMC is a run combining

References

RSL BM25LC and RSL BM25LM, where the scores of the[1] Clarke, C., Vtyurina, A. and Smucker, M.: Assessing top-k
preferences, ArXiv, Vol. abs/2007.11682 (2020).
[2] Ghenai, A., Smucker, M. and Clarke, C.: A ThinkAloud Study to Understand Factors Aﬀecting Online HealthSearch, Proceedings of the 2020 Conference on Human Information Interaction and Retrieval (2020).
[3] Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H.
and Mikolov, T.: FastText.zip: Compressing text classification models, arXiv preprint arXiv:1612.03651 (2016).
[4] Joulin, A., Grave, E., Bojanowski, P. and Mikolov, T.: Bagof Tricks for Eﬃcient Text Classification, arXiv preprint
arXiv:1607.01759 (2016).
[5] Lioma, C., Simonsen, J. and Larsen, B.: Evaluation Measures for Relevance and Credibility in Ranked Lists, Proceedings of the ACM SIGIR International Conference on
Theory of Information Retrieval (2017).
[6] Yang, P., Fang, H. and Lin, J.: Anserini: Reproducible
Ranking Baselines Using Lucene, ACM J. Data Inf. Qual.,Vol. 10, pp. 16:1–16:20 (2018).

documents are calculated as the sum of the scores from thetwo runs.

3 Result and Discussion
The results were assessed to judge whether the retrieved
documents are useful, credible, and correct. Table 1 showsthe evaluation results of our runs and the medians of thescores from all the participant teams. The evaluation measure cam map three is an extended version of the convex aggregating measure (CAM) [5]. The organizers designed thismeasure based on the original one considering all the threeaspects (usefulness, credibility, and correctness). Other measures such as nDCG and compatibility [1] were also used toevaluate the runs.
From the evaluation results, it is diﬃcult to say that our3：https://github.com/castorini/anserini