SlowGuess/NAACL_2021_OCR huggingface dataset:
We conduct elaborate human experiments for analyzing the performance of our models as well as the utility of the task. As mentioned, we perform an experiment to establish the need of such a task and how multimodal outputs provide enhanced understanding to the end user. Before that, however, we perform a human experiment to label the relevant image for each question in the test set as discussed in the section on human evaluation. The interface for the experiment is as shown in Fig. 5. For each HIT, we provide the turkers with 5 (Question, Answer, Passage) triplets and multiple choice options where they select the most relevant image corresponding to the question-answer pair. We demonstrate the what relevance means with the help of an example as shown. We also provide them with an option to select the option 'None of these' as in some cases, no image might be relevant. In order to ensure the quality of responses (to accept or reject turkers' responses), out of 5 questions, we insert random images in one random question. Ideally, a turker paying attention while providing responses is expected to select 'None of these' for the question. We find more than  $90\%$  acceptance ratio in the first event indicating the high quality annotation.
After creating the test set (over 3.5k examples), we randomly select 200 examples from the test set (ensuring there are at least 2 images in the selected examples) and provide a unimodal as well multimodal answer for the annotators to analyze in another experiment. As shown in Figure 6, we ask the annotators a set of overall 6 questions. We have already discussed the outcomes of the experiment in the main paper. We, here, highlight how we maintain the quality of the responses. In some random inputs to the annotator, we make text-image pair incompatible while in some cases we make the answer non-recoverable from the input passage. A turker paying appropriate attention to the task will be easily able to identify the answer - 'No' to the two additional questions given at the end. The answers to those two questions determine whether a particular HIT is accepted or rejected. Since, we provided reasonable amount for annotation, we find  $\zeta 95\%$  acceptance ratio indicating that the evaluation so performed is pure and can be reliable used to make conclusions.
In this section, we describe the dataset collection process and present some statistics about the dataset.
As already described in the main paper, we create our dataset by curating and subsampling a set of questions with images from MS-Marco and Natural Questions dataset. Fig. 7 shows the distribution of different types of tokens in the dataset. We have only retained those frequently occurring tokens (for both levels) which have more than  $5\%$  of the total frequency for their category for the simplicity of representation.
Filtering for MS-MARCO From the MS-MARCO dataset we filter out the entries which do not have a Wikipedia page as a source for the answer paragraph. Since, we are focusing on extractive multimodal outputs in this paper, we further eliminate all those question-answer pairs where the answer does not appear in the selected passages. Instead of eliminating answers without an exact match, we use edit distance to retain answers that include minor edits (e.g. removal of parenthesis) in our dataset.
Filtering for Natural Questions For the Natural Questions dataset all answers are guaranteed to be grounded in Wikipedia entries. We use the short
You will see 5 Question-Answer Pairs along with a contextual passage for each Question-Answer Pair for each task. For each such Triplet, you will see multiple options of images along with an option saying 'None of these'. You are supposed to select exactly one response from them on the basis of relevance of the image to the question-answer pair based on the the context of passage. In other words, given the passage and the question, select a specific image that can be used along with the textual answer to effectively answer the question. The passage is to provide context for the question that has been asked. Please look the following example to understand :
Question: What is a fast food restaurant?
Answer: A fast food restaurant, also known as a quick service restaurant (QSR) within the industry, is a specific type of restaurant that serves fast food cuisine
Passage: McDonald's Corporation is an American fast food restaurant company, founded in 1940 as a restaurant operated by Richard and Maurice McDonald, in San Bernardino, California, United States. A fast food restaurant or a quick service restaurant (QSR), is a specific type of restaurant that serves fast food cuisine. They rechristened their business as a hamburger stand, and later turned the company into a franchisee, with the Golden Arches logo being introduced in 1953.
Figure 5: Instructions provided to the human annotators for labelling the relevant image for the triplet
The correct choice in this case would be the first option since it shows a fast food restaurant image which goes well with the answer.
This v  t   a   i   t provides additional context for the Question-Answering.
For every Question-Passage-Answers, we would like your opinion on 4 directions:
1. Is Answer (A) relevant to the question? Answer No - if the Answer (A) is completely irrelevant to the question
2. Is Answer (B) relevant to the question? Answer No - if the Answer (B) is completely irrelevant to the question
3. Is the textual reponse (Answer A) independently sufficient to answer the question?
4. Does the image (Answer B) add cognitive value to the response? Answer No - if the image did not add any value toward understanding the response.
5. Is the answer recoverable easily from the input passage? Answer Yes - if the answer is present is some paraphrased form in the passage
6. Is the text-image pair compatible with each other? Answer No - if the image and text are completely unrelated to each other
Passage: Westchester County is a county in the U.S. state of New York. Westchester covers an area of 450 square miles (1,200 km 2), consisting of 48 municipalities. According to the 2010 Census, the county had a population of 949,113, estimated to have increased by  $2.5\%$  to 972,634 by 2012. Established in 1683, Westchester was named after the city of Chester, England.
Question: what county is larchmont in
Figure 6: Interface shown to the human annotators for the task of identifying the need of the multimodal output
Is Answer (A) relevant to the question?
Yes No
Yes No
Yes No
Yes No
Yes No
Yes No
Is Answer (B) relevant to the question?
Is the textual reponse (Answer A) independently sufficient to answer the question?
Does the image (Answer B) add cognitive value to the response?
Is the answer recoverable easily from the input passage?
Is the text-image pair compatible with each other?
answer provided by the authors as our target answer, and use the long answer along with distractor passages as the input to our model. However, to reduce the noise from NQ, we removed questions with a single-word answer and questions where the original Wikipedia article had no images.
Scraping images from Wikipedia: Our main motivation of using answers grounded in Wikipedia articles for our corpus was to exploit the structure of
such articles to scrape images and get proxy supervision. To this end we presuppose the title of the article provided in the url field of MS-MARCO with http://en.wikipedia.com/wiki/ to get the URL of the appropriate Wikipedia article. We use the BeautifulSoup package to find all objects of the img class from the HTML page and scrape the largest available resolution of the image (found from the srcset property). Further,
Figure 7: Starting Token Distribution for the train set
we only scrape images which are of the .png or .jpeg formats to avoid other media. Finally we only retain a maximum of 20 images from each page in order to avoid a large number of irrelevant images.
Providing proxy supervision: Our proxy supervision score broadly consists of a proximity score and a caption relevance score. To compute the former we determine the position of each image in the HTML source of the Wikipedia page by finding the paragraph directly below the image. In case the image does not have any text below it we consider the paragraph directly above it. We then compute the number of tokens between the first (or last) word of the found paragraph and the first word of the answer paragraph and normalize it. For the case of multiple images together in the HTML we assign the same proximity score to all of them.
To compute the caption relevance we use the thumbcaption attribute of the image from the HTML source. In case an image does not have this property in the HTML we consider the text below the image in place of the caption. We then compute the TF-IDF scores of the "caption" with the answer, query and answer passage to get the caption relevance scores.
Image distribution We also show the distribution of the number of input images per question in Fig. 8. Evidently more than  $80\%$  of the dataset has more than two images while a significant proportion (more than  $30\%$ ) has more than 6 images mak
ing the task fairly difficult. This has also been demonstrated by the large difference between the UNITER accuracy and MExBERT's accuracy.
Figure 8: Distribution of the of Input Images Per Question for the Train Dataset
We show below some randomly chosen samples from the dataset (which were also correctly chosen by our model MExBERT) to provide reader with an idea about the variety of inputs and input images. The question is shown at the top of the box while the input passage and the set of images have been shown inside the box. The red boundary over one box one of the images denote the image which was annotated as the selected image during annotation and was also predicted correctly by the MExBERT framework.
Question: What is Papyrus?
Input Passage:
Egyptians used papyrus for 4000 years until other plants and trees were used to make paper for economical reasons. Papyrus is still made, but normally only as a tourist attraction. Cyperus papyrus (papyrus sedge, paper reed, Indian matting plant, nile grass) is a species of aquatic flowering plant belonging to the sedge family cyperaceae. it is a tender herbaceous perennial, native to africa, and forms tall stands of reed-like swamp vegetation in shallow water. papyrus is made from a plant that grows on the banks of the nile river in egypt. the aquatic plant, cyperus papyrus, grows up to 15 feet (4.5 meters) high. its green, triangular stem has long, sharp leaves and flower clusters 10 to 20 inches (25 to 50 cms) long. these flowers bloom at the tip. 1 the climate of egypt and certain parts of mesopotamia preserved papyri in the ruins of ancient towns and cemeteries. 2 egyptians used papyrus for 4000 years until other plants and trees were used to make paper for economical reasons. 3 papyrus is still made, but normally only as a tourist attraction
Question: What is a Shillelagh?
An Irish word for a cudgel made of blackthorn, oak, or other hardwoods. usually slightly smaller than a walking stick or cane. good for quick repetitious beating of individuals. there in the village of shillelagh you'll find the namesake shillelagh sticks, stout cubs or cudgels made from the wood of oak or blackthorn. the wood is fashioned into walking sticks, clubs, cudgels, fighting sticks, staffs, and even good luck charms. a shillelagh , willow or blackthorn stick is a wooden walking stick and club or, cudgel typically made from a stout knotty stick with a large knob at the, top that is associated with ireland and irish. folklore
Question: What does fawn mean in Dogs?
terriers and hounds, tan dog with a black saddle and white markings (to any extent). trim, various-general term. a small amount of white on the chest, muzzle, toes and/or tail tip. trindle. various-general term. brindle tricolour (i.e. black with brindle points and white markings. a fawn great dane. fawn is a light yellowish tan colour. it is usually used in reference to clothing, soft furnishings and bedding, as well as to a dog 's coat colour. it occurs in varying shades, ranging between pale tan to pale fawn to dark deer-red. the first recorded use of fawn as a colour name in english was in 1789. this can be a bit of a barrier when it comes to working out the genetics of particular breeds, so to make things easier, here's a list of some of the terms you'll find (either on breed standards or being used by breeders), and what they actually mean in terms of the genetics we've studied on this site
Question: What is Altitude training?
altitude training is the practice by some endurance athletes of training for several weeks at high altitude, preferably over 2,400 metres (8,000 ft) above sea level, though more commonly at intermediate altitudes due to the shortage of suitable high-altitude locations. altitude training works because of the difference in atmospheric pressure between sea level and high altitude. at sea level, air is denser and there are more molecules of gas per litre of air. altitude training works because of the difference in atmospheric pressure between sea level and high altitude. at sea level, air is denser and there are more molecules of gas per litre of air. regardless of altitude, air is composed of  $21\%$  oxygen and  $78\%$  nitrogen. altitude training works because of the difference in atmospheric pressure between sea level and high altitude. at sea level, air is denser and there are more molecules of gas per litre of air. altitude training can be simulated through use of an altitude simulation tent, altitude simulation room, or mask-based hypoxicator system where the barometric pressure is kept the same, but the oxygen content is reduced which also reduces the partial pressure of oxygen. altitude training works because of the difference in atmospheric pressure between sea level and high altitude.
Question: What is chiropractic treatment
. chiropractic is a health care profession dedicated to the non-surgical treatment of disorders of the nervous system and/or musculoskeletal system. generally, chiropractors maintain a unique focus on spinal manipulation and treatment of surrounding structures.iew chiropracticvideos. chiropractic is a health care profession dedicated to the non-surgical treatment of disorders of the nervous system and/or musculoskeletal system. generally, chiropractors maintain a unique focus on spinal manipulation and treatment of surrounding structures. chiropractic is a form of alternative medicine that focuses on diagnosis and treatment of mechanical disorders of the musculoskeletal system, especially the spine, under the belief that these disorders affect general health via the nervous system.he specific focus of chiropractic practice is chiropractic subluxation. traditional chiropractic assumes that a vertebral subluxation or spinal joint dysfunction interferes with the body's function and its innate intelligence. spinal adjustment/manipulation is a core treatment in chiropractic care, but it is not synonymous with chiropractic. chiropractors commonly use other treatments in addition to spinal manipulation, and other health care providers (e.g., physical therapists or some osteopathic physicians) may use spinal manipulation.top.ands-on therapy-especially adjustment of the spine-is central to chiropractic care. chiropractic is based on the notion that the relationship between the body's structure (primarily that of the spine)
the three gorges dam area is rich in archaeological and cultural heritage. many different cultures have inhabited the areas that are now underwater, including the daxi (circa 5000-3200 b.c.e), which are earliest neolithic culture in the region, and its successors, the chujialing (circa, the itaipu dam opened in 1984 in south america as the largest, producing 14,000 mw but was surpassed in 2008 by the three gorges dam in china at 22,500 mw. hydroelectricity would eventually supply some countries, including norway, democratic republic of the congo, paraguay and brazil, with over  $85\%$  of their electricity. construction on the three gorges dam was completed in 2008. the dam stands 185m high and 2,309m wide, making it the world's largest hydro plant, well ahead of brazil's 12,600mw itaipu installation."
Question: What caused august 6 1945
In this aug. 6, 1945, file photo, smoke rises around 20,000 feet above hiroshima, japan, after the first atomic bomb was dropped. on two days in august 1945, u.s. planes dropped two atomic bombs, one on hiroshima, one on nagasaki, the first and only time nuclear weapons have been used. ligh-angle view of a section of the city of hiroshima after the us atomic bombing on august 6, 1945. (photo by keystone/getty images). sacred trees stand bare and broken near fallen tombstones at the temple of kokutaiji, following the us atomic bombing of hiroshima, japan on august 6, 1945. the united states dropped atomic bombs on the japanese cities of hiroshima and nagasaki in august 1945, during the final stage of the second world war. the two bombings, which killed at least 129,000 people, remain the only use of nuclear weapons for warfare in history.he united states dropped atomic bombs on the japanese cities of hiroshima and nagasaki in august 1945, during the final stage of the second world war. the u.s. attacked japan on august 6, 1945 using a gigantic, atomic bomb, codename "little boy", that was equivalent to 20,000 tons of TNT. the bomb was dropped in hiroshima and destroyed the city, killing thousands of civilians.he u.s. attacked japan on august 6, 1945 using a gigantic, atomic bomb, codename "little boy", that was equivalent to 20,000 tons of TNT. the bomb was dropped in hiroshima and destroyed the city, killing thousands of civilians
Question: What does the three gorges dam produce?