SlowGuess/NAACL_2022_OCR huggingface dataset:
Tetsuya Sakai and Ruihua Song. 2011. Evaluating diversified search results using per-intent graded relevance. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 1043-1052.
Ben Shneiderman, Don Byrd, and W Bruce Croft. 1997. Clarifying search: A user-interface framework for text searches.
Christian Stab and Iryna Gurevych. 2014. Identifying argumentative discourse structures in persuasive essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 46-56.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819.
Xiaojie Wang, Ji-Rong Wen, Zhicheng Dou, Tetsuya Sakai, and Rui Zhang. 2017. Search result diversity evaluation based on intent hierarchies. IEEE Transactions on Knowledge and Data Engineering, 30(1):156-169.
Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval, pages 55-64.
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013-2018, Lisbon, Portugal. Association for Computational Linguistics.
Yi Zhang, Zachary G. Ives, and Dan Roth. 2019. Evidence-based Trustworthiness. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL).
A Hypothesis Testing
Our null hypothesis is  $P \leq P_0$  and alternate hypothesis is  $P > P_0$ , where  $P_0 = 0.5$  and  $P$  is the percentage of responses of the survey that prefer our prototype.
We use the one-sided Z-Test as our hypothesis testing statistics. The z-score of a standard z-test can be calculated as
$$
Z = \frac {\bar {x} - \mu}{\frac {\sigma}{\sqrt {N}}}
$$
We perform Boostrapping resampling to get the sample distribution and standard deviation. Specifically, we randomly sample data points with replacement from the distribution of each question's results to a sample size of 300, and repeat this process 1000 times to compute an average mean and average standard deviation. We then regard these mean and standard deviation as the sample mean and sample standard deviation. We then use these statistics to calculate the z-score for each type of question and get  $z = 7.29$ , 9.57, 1.72, and 4.79. We show that users prefer our prototype over Google Search with significance levels 0.01, 0.01, 0.05, and 0.01 for questions on organization, comprehensiveness, informativeness, and general preference, respectively.
B List of Sources
See Table 3.
C List of Controversial Queries
See next pages.
D Screenshot of Demo Website and Human Study Interface
cbs.com
who.int
clevelandclinic.com
medicalnewstoday.com
noah-health.org
familydoctor.org
medlineplus.gov
mayoclinic.org
webmd.com
cebm.net
sciencenews.org
sciencemag.org
yalemedicine.org
nejm.org
Table 3: List of urls of the sources we include as information source for our prototype multi-perspective search engine.
Topic	Query
COVID-19	Should wearing masks be mandatory?Should we all get vaccinated?Is herd immunity for COVID-19 achievable?Should COVID-19 vaccines be mandatory?Will the Covid-19 pandemic have a lasting impact on society?
Race & Justice	Sshould the US ban assault weapons?Should police departments be defunded, if not abolished?Should the death penalty be allowed?Should the use of private prisons continue?Should the US continue to build a border wall at the US-Mexico border?
Immigration	Will immigrants take american jobs?Should the US end the Deferred Action for Childhood Arrivals (DACA) policy?Should the US decriminalize illegal border crossings?Should immigrants who entered the US illegally be denied a path to citizenship?Should America use a merit-based immigration system?
Health Care	Should we support medicare for all?Does racial inequality persists in health care coverage?Should abortion be legal?Do the prescription drug costs need to be lowered?Should imported prescription drugs be allowed in the U.S.?
Climate Change	Should the US rejoin the Paris climate agreement?Should the US adopt a climate change plan?Should the US expand fossil fuel extraction on public land?Should fighting climate change be a priority?Is climate crisis inevitable?
Policy	Should the US have withdrawn from the Open Skies Treaty?Should the tariffs imposed on China by president Trump be maintained?Should confederate statues be taken down?Should the federal government adopt Net Neutrality Rules?Should the US re-enter a nuclear deal with Iran?
Table 4: List of controversial topic and queries we include in our user study. The topics and queries are selected from the 2020 United States presidential debates. The list of topics are selected from https://2020election.procon.org/.
Figure 4: Screenshot 1 of our prototype's demo website.
should wearing masks be mandatory?
Government Agency
Should Wearing Masks Be Mandatory?
Perspective 1
prevention / spread
Yes, Wearing masks can cut the spread of other viruses, and ...
The British Medical Association, which represents doctors called for the continued use of masks. Masks can also help reduce virus spread from contagious people with no symptoms. Face coverings have already been...
BBC, NPR, The Washington Post
May 2020 - Nov 2021
Perspective 2
No, There are better ways to protect you, and ...
This is why experts say frequent hand washing with soap and water is far more effective. So it could protect you, but only in certain situations such as when you're in close quarters with others where someone infected might sneeze or cou...
May 12, 2020
Figure 5: Screenshot 2 of our prototype's demo website.
$\leftrightarrow$
Figure 6: Screenshot 3 of our prototype's demo website.
prediction / speed / News Article
Face masks may reduce the growth rate of infections.
04 May 2020 Richard Gray BBC
- The researchers estimate that face masks may reduce the growth rate of reported infections by around  $40\%$
- Unpublished study by scientists at Arizona State University found that if  $80\%$  of people wore only moderately effective masks, it could reduce the number of deaths in New York by  $17 - 45\%$  over a two month period.
- "We lower the chances of future surges and lockdowns which are economically and psychologically disruptive, and we increase the chance of eliminating the virus."
social implications / prevention / News Article
Schools might find universal masking necessary.
20 Jul 2021 Clare Lombardo NPR
- "Given what we know about low rates of in-school transmission when proper prevention measures are used, together with the availability of effective vaccines for those age 12 years and up," the AAP says, "the benefits of in-person school outweigh the risks in almost all circumstances."
- In its guidance, the AAP says it recommends universal masking because enforcing masking only for unvaccinated students, as well as tracking the vaccination status of students and staff, might be difficult for schools to do.
The CDC notes, however, that schools might find
political implications / statistics / News Article
Joe Biden is calling for everyone to wear a mask.
13 Aug 2020 Asma Khalid NPR
Joe Biden is calling for everyone in the United States to wear a mask, well into the fall.
- In remarks at a June event in Wilmington, Biden also said there should be better guidance from the administration around mask-wearing.
- "Every governor should mandate mandatory mask-wearing.
Relates to
prediction /  $\mu$
All teach wear ma
27 Jul 20:
CDC Urges V Places With
The nation's recommends students (ove regardless of
In addition, t staff and stuc they are vacc
Figure 7: Screenshot 4 of our prototype's demo website.
prevention / spread / News Article
Figure 8: Screenshot 5 of our prototype's demo website.
It could protect you, but only in certain situations.
12 May 2020 BBC
This is why experts say frequent hand washing with soap and water is far more effective.
So it could protect you, but only in certain situations such as when you're in close quarters with others where someone infected might sneeze or cough near your face.
- One is that a mask is not seen as reliable protection as other methods, given that current research shows the virus is spread by droplets and contact with contaminated surfaces.
$\leftarrow \rightarrow$
A Survey of Online Search
In this survey, we will show you the search results of a same query presented in two different structures. We will then ask you a few questions that help us examine which is better.
Please answer them honestly and truthfully.
Once you start the survey, you will see a query at the top. That is the question we searched. You will then see two list of search results returned by this question, one on the left, and one on the right. The two list of search results are exactly the same, meaning that they are the same webpages returned by searching for the same question. However, they are presented in different orders, using different titles, and providing different evidence.
For instance, the two articles below are referring to the same article. However, they have different titles, different evidence (the smaller black texts below the title), and are presented in their lists at different positions. We hope that you can help us examine which presentation of search results is better.
- Democrats' deal on prescription drugs would lower costs for seniors
Democratic lawmakers have reached a deal to lower prescription drug prices ... to Part D will provide real relief for seniors with the highest drug costs. ...
- Democrats' deal would lower costs for seniors.
- "Preventing prices from rising faster than inflation and adding a hard out-of-pocket cap to Part D will provide real relief for seniors with the highest drug costs."
- *Allowing Medicare to finally negotiate drug prices is a big win for
seniors.* CEO Jo Ann Jenkins said in a statement
- Democrats' deal on prescription drugs would lower costs for seniors
We will ask you questions that compare the two list of results, and also ask you to explain why do you think one is better. Please provide your reasons truthfully and patiently as we will use it to filter random or invalid responses.
There will be five pages in this batch of survey. Each page has five questions. You'll need to answer all of the questions in all five pages to complete this HIT. You will go to the next page after you submit your answers for each page, and you will see a HIT key after you finish the last page. You should then copy and paste the HIT key to the Amazon MTurk HIT page and submit the HIT so you can be compensated.
Copy and paste your Amazon Mechanical Turk Worker ID here and start your task! Make sure it matches exactly your WorkerID on Amazon MTurk so we can compensate you.
Figure 9: Screenshot 1 of the survey interface.
#1 should COVID-19 vaccines be mandatory?
System 1
- Opinion | Should Covid Vaccines Be Mandatory? - The New York ...
Could Americans really be required to get vaccinated? Who should mandate Covid vaccination? The case against vaccine mandates.
Factbox: Countries making COVID-19 vaccines mandatory | Reuters
Nov 12 (Reuters) - Governments have been making COVID-19 shots mandatory for health workers and other high-risk groups, pushed by a sharp upturn in ....
COVID-19 vaccines: Should they be compulsory in healthcare?
*Due to the severe health risks associated with COVID-19 and the fact that reinfection with COVID-19 is possible, [a] vaccine should be offered to you ....
It also made vaccinations obligatory for Paralympic athletes heading to Tokyo because unvaccinated members on the team could pose a health risk. In Tasmania, ....
- Sean Penn says Covid-19 vaccine should be mandatory
Sean Penn says Covid-19 vaccinations should be mandatory,
and is calling on the film industry to lead the way.
The political fight over COVID-19 vaccine mandates is deepening ...
Figure 10: Screenshot 2 of the survey interface.
*No entity in Texas can compel receipt of a COVID-19 vaccine by any individual, including an employee or a consumer, who objects to such vaccination for any ....
- Mandatory Vaccine? As COVID-19 Vaccine Nears, Employers ....
With promising news from three COVID-19 vaccine trials showing  $90\%$  to  $95\%$  efficacy, employers are now weighing whether they should simply encourage their ...
As Delta surges, U.S. military braces for mandatory COVID-19 ...
President Joe Biden could issue a waiver requiring troops to get vaccinated prior to formal vaccine approval by the U.S. Food and Drug Administration. Last week ....
Vaccine Mandates Rolled Out In New York City, California And For ...
City workers who are unvaccinated must wear a mask when indoors at work, ...Vocational nurse Eon Walk administers the Pfizer COVID-19 vaccine at a mobile ....
The left side and right side present the same articles in different structures, titles, and explanations. Read through both lists of articles and answer the following questions.
Q1: Which system has a better organization of results?
$\bigcirc$  System 1  $\bigcirc$  System 2
Q2: Which system shows you a more comprehensive view of the topic?
Q3: Which system's article titles are more informative (in the context of the topic)?
Q4: Which system do you prefer?
Q5: Why do you prefer it?
Please explain your reasons honestly. Note that you will not be compensated if you are submitting empty or random responses.
System 2
- Yes, Covid-19 vaccines should be mandatory, and ...
- Employers are allowed to require their employees to receive vaccinations.
Since the 2009 swine flu pandemic, the Equal Employment Opportunity Commission has held that employers are allowed to require their employees to receive certain vaccinations, provided they are offered reasonable accommodations based on religion or disability.
Since 1905, when the U.S. Supreme Court let stand a Massachusetts law that levied fines against people who refused smallox inoculation, the courts have routinely upheld the states' authority to enforce vaccination if necessary for public health.
o INDONESIA made inoculations mandatory.
- "INDONESIA made inoculations mandatory in February, warning that anyone who refused to be vaccinated could be fined or denied social assistance or government services.
- "URAINE in October made vaccinations compulsory for public sector employees and the public sector."
** AUSTALIA in late June made vaccinations mandatory for high-risk aged-care workers and employees in quarantine hotels.
Italy, France, and Greece have made vaccination man
In the U.K., the House of Commons recently approved legislation that would make COVID-19 vaccination compulsory for staff working in care homes in England unless they have a medical exemption. Italy, France, and Greece have already made vaccination mandatory for
- The Department of Veterans Affairs, some university hospitals, and some health systems have already made COVID-19 vaccination compulsory, though many of them allow exemptions on religious or philosophical grounds.
It made vaccinations obligatory.
It also made vaccinations obligatory for Paralympic athletes heading to Tokyo because unvaccinated members on the team could pose a health risk. Hungary's government has decided to make vaccinations mandatory for all students.
Australia decided in late June to make vaccinations mandatory for high-risk aged-care workers and employees in quarantine hotels.
- Sean Penn says Covid-19 vaccinations should be mandatory.
"I do think that vaccines should be mandatory," Penn said, stipulating that people with medical conditions that might make it detrimental to get vaccinated be exempt. Sean Penn says Covid-19 vaccinations should be mandatory, and is willing to work with the CDC to lead the way.
- He offered to help everyone get vaccinations through his nonprofit organization, Community Organized Relief Effort.
- Employers are weighing they should encourage their employees to get vaccinated.
With promising news from three COVID-19 vaccine trials showing  $90\%$  to  $95\%$  efficacy, employers are now weighing whether they should simply encourage their employees to get vaccinated or make it mandatory. "It's the right thing to do for society," he says, noting that the mandate comes not just doctors, nurses and medical students, but also people working in billing and other jobs that don't put them in contact with patients.
Employers have an obligation to get rid of known hazards in the workplace, he says, and COVID-19 has proved to be a hazard unlike any
The Pentagon appears poised to do - mandate vaccinations.
Vaccination rates are highest in the Navy, which suffered from a high-profile outbreak last year aboard an aircraft carrier. With the Delta variant surging, the Pentagon appears poised to do so, but has not so far - mandate vaccinations to safeguard against COVID-19.
- He wouldn't have imagined three weeks later he would die alone in a hospital.  
- The doctor's name was Mattson, wrote on a Facebook post, implying people to get vaccinated.
Vaccination is important for health care workers' own health.
They noted vaccination is important not only for health care workers' own health, but also for that of colleagues, families, residents of long-term care facilities and patients - particularly unvaccinated children and the immunocompromised who are vulnerable to COVID-19.
New Vaccine Mandates Are Coming For Government Employees And Health Care Workers
- The city's largest teachers union, the United Federation of Teachers, recruited teachers from all levels of school and school districts and testing "helped keep schools among the safest places in the city."
- No, The opposition to COVID-19 vaccines is growing, and ...
Vaccines are encouraged but should remain voluntary.
- The order notes that vaccines are "encouraged" for those who are eligible
- They have the idea that vaccines are essentially forcing them into doing
President Biden has focused on getting as many Americans as possible vaccinated against the coronavirus, most notably rolling out wide-reaching vaccine mandates for government employees and for businesses with more than 100 workers.