DataHammer/scimrc huggingface dataset:
Question: What does the paper explore?
Evidence: Understanding what a question is asking is one of the first steps that humans use to work towards an answer. In the context of question answering, question classification allows automated systems to intelligently target their inference systems to domain-specific solvers capable of addressing specific kinds of questions and problem solving methods with high confidence and answer accuracy BIBREF0 , BIBREF1 .
To date, question classification has primarily been studied in the context of open-domain TREC questions BIBREF2 , with smaller recent datasets available in the biomedical BIBREF3 , BIBREF4 and education BIBREF5 domains. The open-domain TREC question corpus is a set of 5,952 short factoid questions paired with a taxonomy developed by Li and Roth BIBREF6 that includes 6 coarse answer types (such as entities, locations, and numbers), and 50 fine-grained types (e.g. specific kinds of entities, such as animals or vehicles). While a wide variety of syntactic, semantic, and other features and classification methods have been applied to this task, culminating in near-perfect classification performance BIBREF7 , recent work has demonstrated that QC methods developed on TREC questions generally fail to transfer to datasets with more complex questions such as those in the biomedical domain BIBREF3 , likely due in part to the simplicity and syntactic regularity of the questions, and the ability for simpler term-frequency models to achieve near-ceiling performance BIBREF8 . In this work we explore question classification in the context of multiple choice science exams. Standardized science exams have been proposed as a challenge task for question answering BIBREF9 , as most questions contain a variety of challenging inference problems BIBREF10 , BIBREF11 , require detailed scientific and common-sense knowledge to answer and explain the reasoning behind those answers BIBREF12 , and questions are often embedded in complex examples or other distractors. Question classification taxonomies and annotation are difficult and expensive to generate, and because of the unavailability of this data, to date most models for science questions use one or a small number of generic solvers that perform little or no question decomposition BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 . Our long-term interest is in developing methods that intelligently target their inferences to generate both correct answers and compelling human-readable explanations for the reasoning behind those answers. The lack of targeted solving – using the same methods for inferring answers to spatial questions about planetary motion, chemical questions about photosynthesis, and electrical questions about circuit continuity – is a substantial barrier to increasing performance (see Figure FIGREF1 ).
To address this need for developing methods of targetted inference, this work makes the following contributions:
In this work we explore question classification in the context of multiple choice science exams.
Answer: Question classification in the context of multiple choice science exams.
Question: What is the best way to cook a steak?
Answer: Because tastes differs, there are no best way to cook a steak.

Here are some optimized ways to cook steak.

Fastest way to cook a steak - cast iron skillet.  Pan sear room temperature steak in a host cast iron skillet; sear the steak on both sides; baste with melted butter / herbs until done to the desired doneness; rest/cut/serve.

Most consistent way to cook a steak - sou vide followed by flame torch the exterior.  Season steak on both sides / edges; put into vacuum bag, vacuum bag, put under sous vide water bath at 130F for 1-4 hours; take it out of the bag, pat dry; use flame torch to char both sides; rest/cut/serve.

Most scientific way to cook a steak - cold sear.  Put steak into non-sticking pan on high for 2 minutes; flip once and cook for another 2 minutes; reduce heat to medium low, flip and cook for another 2 minutes; flip and cook for another 2 minutes; keep on flipping / cooking for 2 minutes until reaching desired doneness; season; rest/cut/serve.
Question: How are the potentially relevant text fragments identified?
Evidence: This step consists of generating a query out of the claim and querying a search engine (here, we experiment with Google and Bing) in order to retrieve supporting documents. Rather than querying the search engine with the full claim (as on average, a claim is two sentences long), we generate a shorter query following the lessons highlighted in BIBREF0 .
We rank the words by means of tf-idf. We compute the idf values on a 2015 Wikipedia dump and the English Gigaword. BIBREF0 suggested that a good way to perform high-quality search is to only consider the verbs, the nouns and the adjectives in the claim; thus, we exclude all words in the claim that belong to other parts of speech. Moreover, claims often contain named entities (e.g., names of persons, locations, and organizations); hence, we augment the initial query with all the named entities from the claim's text. We use IBM's AlchemyAPI to identify named entities. Ultimately, we generate queries of 5–10 tokens, which we execute against a search engine. We then collect the snippets and the URLs in the results, skipping any result that points to a domain that is considered unreliable. Finally, if our query has returned no results, we iteratively relax it by dropping the final tokens one at a time.
This step consists of generating a query out of the claim and querying a search engine (here, we experiment with Google and Bing) in order to retrieve supporting documents. Rather than querying the search engine with the full claim (as on average, a claim is two sentences long), we generate a shorter query following the lessons highlighted in BIBREF0 .
We rank the words by means of tf-idf. We compute the idf values on a 2015 Wikipedia dump and the English Gigaword. BIBREF0 suggested that a good way to perform high-quality search is to only consider the verbs, the nouns and the adjectives in the claim; thus, we exclude all words in the claim that belong to other parts of speech. Moreover, claims often contain named entities (e.g., names of persons, locations, and organizations); hence, we augment the initial query with all the named entities from the claim's text. We use IBM's AlchemyAPI to identify named entities. Ultimately, we generate queries of 5–10 tokens, which we execute against a search engine. We then collect the snippets and the URLs in the results, skipping any result that points to a domain that is considered unreliable.
Answer:  Generate a query out of the claim and querying a search engine, rank the words by means of TF-IDF, use IBM's AlchemyAPI to identify named entities, generate queries of 5–10 tokens, which execute against a search engine, and collect the snippets and the URLs in the results, skipping any result that points to a domain that is considered unreliable.
Question: Which of these are religions and which are countries: Christianity, Judaism, Norway, Canada, Islam, Egypt, China, and Iran.
Answer: The countries in this list are Norway, Canada, Egypt, China and Iran. The religions are Christian, Judaism, and Islam.
Question: What are some horror movies from 1987?
Answer: There were many horror movies that were released in 1987. Here is a list of a few of them:

Evil Dead II
Hellraiser
Opera
Predator
Bad Taste
A Nightmare on Elm Street 3: Dream Warriors
Question: Are the original documents pre-processed？
Evidence: We evaluate the corpus using Named Entity Recognition (NER), i. e., the task of finding mentions of concepts of interest in unstructured text. We focus on detecting cases, conditions, factors, findings and modifiers in case reports (cf. Section SECREF6). We approach this as a sequence labeling problem. Four systems were developed to offer comparable robust baselines.
The original documents are pre-processed (sentence splitting and tokenization with ScispaCy). We do not perform stop word removal or lower-casing of the tokens. The BIO labeling scheme is used to capture the order of tokens belonging to the same entity type and enable span-level detection of entities. Detection of nested and/or discontinuous entities is not supported. The annotated corpus is randomized and split in five folds using scikit-learn BIBREF9. Each fold has a train, test and dev split with the test split defined as .15% of the train split. This ensures comparability between the presented systems.
The original documents are pre-processed (sentence splitting and tokenization with ScispaCy[ https://github.eom/allenai/scispacy/releases/tag/v0.2.2 (Neumann et al., 2019) and the en_core_sci_md model.]).
Answer: Yes.
Question: How does the BPE segment words?
Evidence: To enable open-vocabulary translation, we segment words via byte-pair encoding (BPE) BIBREF1 . BPE, originally devised as a compression algorithm BIBREF6 , is adapted to word segmentation as follows:
First, each word in the training vocabulary is represented as a sequence of characters, plus an end-of-word symbol. All characters are added to the symbol vocabulary. Then, the most frequent symbol pair is identified, and all its occurrences are merged, producing a new symbol that is added to the vocabulary. The previous step is repeated until a set number of merge operations have been learned.
BPE starts from a character-level segmentation, but as we increase the number of merge operations, it becomes more and more different from a pure character-level model in that frequent character sequences, and even full words, are encoded as a single symbol. This allows for a trade-off between the size of the model vocabulary and the length of training sequences. The ordered list of merge operations, learned on the training set, can be applied to any text to segment words into subword units that are in-vocabulary in respect to the training set (except for unseen characters).
To increase consistency in the segmentation of the source and target text, we combine the source and target side of the training set for learning BPE. For each language pair, we learn 89500 merge operations.
To enable open-vocabulary translation, we segment words via byte-pair encoding (BPE)5 (Sennrich et al., 2016b). BPE, originally devised as a compression algorithm (Gage, 1994), is adapted to word segmentation as follows: First, each word in the training vocabulary is represented as a sequence of characters, plus an end-of-word symbol. All characters are added to the symbol vocabulary. Then, the most frequent symbol pair is identified, and all its occurrences are merged, producing a new symbol that is added to the vocabulary. The previous step is repeated until a set number of merge operations have been learned.
Answer: First, each word in the training vocabulary is represented as a sequence of characters, plus an end-of-word symbol. All characters are added to the symbol vocabulary. Then, the most frequent symbol pair is identified, and all its occurrences are merged, producing a new symbol that is added to the vocabulary. The previous step is repeated until a set number of merge operations have been learned.
Question: Given this paragraph, who was the first person to run the Western States 100 in under 24 hours?
Evidence: The Western States Trail Ride was first completed on foot by seven soldiers from Fort Riley, Kansas during the 1972 event, proving that the mountain course could be marched in less than two days. In 1974 Gordy Ainsleigh was the first to run it in under 24 hours. Ainsleigh had finished the Western States Trail Ride (Tevis Cup) in 1971 and 1972 on horseback, but in 1973 his new horse was pulled with lameness at the 29-mile checkpoint. In 1974, with the inspiration and encouragement of Drucilla Barner, the first woman to win the Tevis Cup and Secretary of the Western States Trail Foundation, Gordy joined the horses of the Western States Trail Ride to see if he could complete the course on foot in under 24 hours. Twenty-three hours and forty-two minutes later Gordy arrived in Auburn, proving that a runner could cover the Western States trail in a day. However, he did not run 100 miles. The course was at most 89 miles until 1980 when it was extended to 93.5 miles and in 1985 finally to 100 miles. Running 100 miles in a day was nothing new. About 1,000 runners worldwide had run or walked 100 miles in a day prior to 1974 on tracks, roads, and trails.
Answer: In 1974 Gordy Ainsleigh was the first to run it in under 24 hours.
Question: Identify which instrument is string or woodwind: Auto-harp, Saxophone
Answer: Auto-harp is string, Saxophone is woodwind.
Question: List high quality binge worthy tv shows
Answer: Breaking Bad, Sons of Anarchy, Homeland, The Sopranos, Mind Hunter, Shameless
Question: Name some of the movies where Johnson acted in and the years in which the movies were released.
Evidence: Johnson entered Hollywood and the film industry, becoming a star initially through his wrestling popularity and noted work ethic. Over his acting career, he became one of the highest paid and most successful actors in Hollywood. He began his acting career on television while wrestling. In his first television acting job, in 1999, he played his own father in an episode of That '70s Show called "That Wrestling Show". Nearly a year later, he appeared in the Star Trek: Voyager episode "Tsunkatse" as an alien wrestler who fought popular character Seven of Nine. While Johnson was away from WWE, the company continued to sell "The Rock" merchandise, and he continued to be featured prominently in the opening montages of their television shows.

Johnson began his theatrical career in The Mummy Returns (2001), The Scorpion King (2002), The Rundown (2003), and Walking Tall (2004). He played a supporting role in Be Cool (2005) - most notable for the meta element of playing a bodyguard that wants to become an actor - and was the primary antagonist in Doom (2005). He also had roles in Gridiron Gang (2006), Reno 911!: Miami (2007), and Southland Tales (2006). He played a cocky famous American football player in The Game Plan (2007) and Agent 23 in Get Smart (2008). He presented the Academy Award for Best Visual Effects at the 80th Academy Awards.
Answer: The Mummy Returns (2001), The Scorpion King (2002), The Rundown (2003), Walking Tall (2004), Be Cool (2005), Doom (2005), Gridiron Gang (2006), Reno 911!: Miami (2007),  Southland Tales (2006), The Game Plan (2007) and Get Smart (2008).
Question: From the following passage, extract 3 common phenomena related to electricity.
Evidence: Electricity is the set of physical phenomena associated with the presence and motion of matter that has a property of electric charge. Electricity is related to magnetism, both being part of the phenomenon of electromagnetism, as described by Maxwell's equations. Various common phenomena are related to electricity, including lightning, static electricity, electric heating, electric discharges and many others.
The presence of either a positive or negative electric charge produces an electric field. The movement of electric charges is an electric current and produces a magnetic field. In most applications, a force acts on a charge with a magnitude given by Coulomb's law. Electric potential is typically measured in volts.
Electricity is at the heart of many modern technologies, being used for:
Electric power where electric current is used to energise equipment;
Electronics which deals with electrical circuits that involve active electrical components such as vacuum tubes, transistors, diodes and integrated circuits, and associated passive interconnection technologies.
Electrical phenomena have been studied since antiquity, though progress in theoretical understanding remained slow until the 17th and 18th centuries. The theory of electromagnetism was developed in the 19th century, and by the end of that century electricity was being put to industrial and residential use by electrical engineers. The rapid expansion in electrical technology at this time transformed industry and society, becoming a driving force for the Second Industrial Revolution. Electricity's extraordinary versatility means it can be put to an almost limitless set of applications which include transport, heating, lighting, communications, and computation. Electrical power is now the backbone of modern industrial society.
Answer: 3 common phenomena related to electricity are: 
1. Lightning
2. Static electricity 
3. Electric heating
Question: In a bingo game, which number is represented by the phrase “two little ducks”?
Answer: 22
Question: What is the histroy behind electoral votes?
Answer: The Founding Fathers established the Electoral College in the Constitution, in part, as a compromise between the election of the President by a vote in Congress and election of the President by a popular vote of qualified citizens.
Question: What was BERT originally trained to do?
Evidence: BERT stands for "Bidirectional Encoder Representations from Transformers" BIBREF1 is a contextual word embedding model. Given a sentence as an input, contextual embedding for the words are returned. The BERT model was designed so it can be fine tuned for 11 different tasks BIBREF1, including question answering tasks. For a question answering task, question and paragraph (context) are given as an input. A BERT standard is that question text and paragraph text are separated by a separator [Sep]. BERT question-answering fine tuning involves adding softmax layer. Softmax layer takes contextual word embeddings from BERT as input and learns to identity answer span present in the paragraph (context). This process is represented in Figure FIGREF4.
BERT was originally trained to perform tasks such as language model creation using masked words and next-sentence-prediction. In other words BERT weights are learned such that context is used in building the representation of the word, not just as a loss function to help learn a context-independent representation. For detailed understanding of BERT Architecture, please refer to the original BERT paper BIBREF1.
A ‘word embedding’ is a learned representation. It is represented in the form of vector where words that have the same meaning have a similar vector representation. Consider a word embedding model 'word2vec' BIBREF6 trained on a corpus. Word embeddings generated from the model are context independent that is, word embeddings are returned regardless of where the words appear in a sentence and regardless of e.g. the sentiment of the sentence. However, contextual word embedding models like BERT also takes context of the word into consideration.
BERT was originally trained to perform tasks such as language model creation using masked words and next-sentence-prediction.
Answer: To perform tasks such as language model creation using masked words and next-sentence-prediction.
Question: What intrinsic evaluation metrics are used?
Evidence: For all these tests we computed a coverage metric. Our embeddings do not necessarily contain information for all the words contained in each of these tests. So, for all tests, we compute a coverage metric that measures the fraction of the gold-standard pairs that could actually be tested using the different embeddings produced. Then, for all the test pairs actually covered, we obtain the success metrics for each of the 3 tests by computing the ratio of pairs we were able to correctly classified as i) being distinct (cosine INLINEFORM0 0.7 or 0.8), ii) belonging to the same class (cosine INLINEFORM1 0.7 or 0.8), and iii) being equivalent (cosine INLINEFORM2 0.85 or 0.95).
For all these tests we computed a coverage metric. Our embeddings do not necessarily contain information for all the words contained in each of these tests. So, for all tests, we compute a coverage metric that measures the fraction of the gold-standard pairs that could actually be tested using the different embeddings produced.
Then, for all the test pairs actually covered, we obtain the success metrics for each of the 3 tests by computing the ratio of pairs we were able to correctly classified as i) being distinct (cosine INLINEFORM0 0.7 or 0.8), ii) belonging to the same class (cosine INLINEFORM1 0.7 or 0.8), and iii) being equivalent (cosine INLINEFORM2 0.85 or 0.95).
Answer: The answers are shown as follows:
* coverage metric
* being distinct (cosine INLINEFORM0 0.7 or 0.8)
* belonging to the same class (cosine INLINEFORM1 0.7 or 0.8)
* being equivalent (cosine INLINEFORM2 0.85 or 0.95)
Question: Which of the following are commonly used as boys names: Adam, Shirley, Hunter, Mary, Susan, Stephanie, John, Buster, Matthew, James, Johan, Sean, Steven, Ada
Answer: Adam, Hunter, John, Buster, Matthew, James, Johan, Sean, Steven
Question: Do they use pretrained word vectors for dialogue context embedding?
Evidence: The utterance is concatenated with a special symbol marking the end of the sequence. We initialize our word embeddings using 300-dimensional GloVe BIBREF30 and then fine-tune them during training.
Answer: Yes.
Question: List the 7 most recent US Speakers of the House (newest to oldest).
Answer: Kevin McCarthy
Nancy Pelosi
Paul Ryan
John Boehner
Nancy Pelosi
Dennis Hastert
Newt Gingrich
Question: What are the top 5 best movies ever made?
Answer: The top 5 best movies ever made are: 
1. The Godfather (1972) 
2. The Dark Knight (2008) 
3. The Matrix (1999) 
4. Star Wars (1977) 
5. Jurassic Park (1993)
However, this largely depends on one's taste. There are some who claim that Weekend at Bernie's is the best movie ever made.
Question: Tell me if the following music pieces are considered classical music, jazz music or pop music? The magic flute, Kind of blue, Wish you were here, Bach's Saint John Passion, Take five, We are the champions.
Answer: The music pieces The magic flute, Bach's Saint John Passion are considered classical music. The pieces Kind of blue and Take five are considered jazz. music and the pieces Wish you were here and We are the champions are considered pop music.
Question: What are the best hikes near Portland Oregon.
Answer: There are many hikes near Portland Oregon, especially on the Columbia River Gorge. The Columbia River Gorge National Scenic Area is the best place to experience the beauty of the Pacific Northwest. The most popular hike in the gorge is Angel’s Rest. This hike is challenging but will only take you a few hours and the payoff is beautiful views of the gorge. Another very popular hike is at Multnomah Falls. This incredible waterfall is over 600 feet tall and makes for some impressive pictures from the bottom or top. You can make this hike as easy or difficult as you want. A lesser known but still incredible hike in the gorge is Hamilton Mountain. You will be treated to some incredible eastern gorge views if you continue on from the peak to the subsequent mountain saddle. When in doubt use the Alltrails app to find hikes in any area your travels take you.
Question: How many planets make up the Solar System?
Answer: There are 8 planets in the solar system. Mercury is the smallest and closest to the Sun. Venus is the second closest to the Sun. Earth, our home planet, is the third planet closest to the Sun followed by Mars. After the asteroid belt is Jupiter, the fifth and biggest planet in the  Solar System. Saturn, Uranus and Neptune are the last planets in the Solar System.

Originally the Solar System was said to have 9 planets until 2006 when the term "planet" was redefined to exclude "dwarf planets". This results in the exclusion of the ninth and farthest away planet, Pluto, reducing the count of planets in the Solar System to current count of 8.
Question: Which is the best place to Kayak in Austin?
Answer: Lady Bird Lake in Austin downtown is revered to be the best place to go Kayaking. Paddlers can choose to head west from the Rowing Dock to take in nature views. But a small paddle down the river to the east will greet paddlers to magnificent downtown skyline views.
Question: What countries made up Yugoslavia?
Answer: Yugoslavia was made up of 7 present day countries. These countries include Bosnia and Herzegovina, Croatia, Montenegro, North Macedonia, Serbia, Kosovo, and Slovenia.