DOI: 10.5281/zenodo.4298975
title: Crowdsourcing Document Similarity Judgements

description: This is the data obtained from crowdsourcing tasks which ask workers to provide similarity metrics between pairs of documents. Each document, as well as each pair, has a unique ID. We provide crowd workers with the pairs through three different task variations: Variation 1: We showed workers 5 pairs of documents and, for each, asked them to rate their similarity in a 4-level Likert scale (None, Low, Medium, High), tell us a confidence level of how sure they were (from 0 to 4) and a written reason as to why they chose that similarity level. For quality reasons, two of the 5 pairs were golden-standards, which means we knew their ratings already and checked the workers' responses. They had to give the golden pair with the higher similarity a higher score than the other golden pair, otherwise, their answer would be rejected. Variation 2: We repeated variation 1 but with a slight alteration: instead of a Likert scale for the similarity score, we asked for a Magnitude Estimation, which is any number above 0. It could be 1, 0.0001, 1000, 42, as long as it was coherent, as in a more similar pair had a higher score than a less similar pair and vice-versa; Variation 3: We showed workers 5 rankings. Each ranking had a main document and 3 auxiliary documents to be compared against the main one. They also had to report a confidence score and give a short written reason, just like variation 1. The first ranking is a golden-standard, and we knew the values for the 3 pairs in it (the pairs were the main document paired with each of the 3 auxiliary documents), and they had to give the golden pair with the highest similarity a higher rank than the one with the lower similarity. The raw results from the tasks are recorded in the JSON file CrowdResults.json. For a description of its contents, please read the file CrowdResults_README.md. These raw annotations from the crowd were then parsed into the three CSVs you see, each corresponding to the aggregated results from one of the task variations. <em>final_scores_likert.csv</em> is the resulting scores for each pair using the variation 1 tasks; <em>pair_id </em>is a unique identifier for each pair; <em>similarity_alg </em>is the similarity assigned to the pair of documents from an automated similarity algorithm; <em>relation </em>is the type of relationship shown by the pair, where smaller values indicate more similar pairs; <em>similarity_crowd_simple_maj </em>stores the simple majority result from the crowd's annotations; <em>similarity_crowd_simple_mean </em>stores the mean of the crowd's annotations; <em>similarity_crowd_simple_median </em>stores the median of the crowd's annotations; <em>final_scores_magnitude.csv</em> is the resulting scores for each pair using the variation 2 tasks; <em>pair_id </em>is a unique identifier for each pair; <em>similarity_alg </em>is the similarity assigned to the pair of documents from an automated similarity algorithm; <em>relation </em>is the type of relationship shown by the pair, where smaller values indicate more similar pairs; <em>scaled_similarity_worker</em> is the magnitude score scaled based on worker's behaviours <em>scaled_similarity_worker_docset </em>is the magnitude score scaled based both on the worker's behaviour and on the pair <em>final_scores_ranking.csv</em> is the resulting scores for each pair using the variation 3 tasks; <em>pair_id </em>is a unique identifier for each pair; <em>similarity_alg </em>is the similarity assigned to the pair of documents from an automated similarity algorithm; <em>relation </em>is the type of relationship shown by the pair, where smaller values indicate more similar pairs; <em>mean_similarity</em> is the mean ranking from that value This dataset was built and used as part of the TheyBuyForYou project.