DataHammer/scimrc huggingface dataset:
Question: How does the BERT model work?
Evidence: BERT, the Bidirectional Encoder Representations from Transformers, consists of several transformer encoder layers that enable the model to extract very deep language features on both token-level and sentence-level. Each transformer encoder contains multi-head self-attention layers that provide ability to learn multiple attention feature of each word from their bidirectional context. The transformer and its self-attention mechanism are proposed by BIBREF8. This self-attention mechanism can be interpreted as a key-value mapping given query. By given the embedding vector for token input, the query ($Q$), key ($K$) and value ($V$) are produced by the projection from each three parameter matrices where $W^Q \in \mathbb {R}^{d_{{\rm model}} \times d_{k}}, W^K \in \mathbb {R}^{d_{\rm model} \times d_{k}}$ and $W^V \in \mathbb {R}^{d_{\rm model} \times d_{v}}$. The self-attention BIBREF8 is formally represented as:
The $ d_k = d_v = d_{\rm model} = 1024$ in BERT large version and 768 in BERT base version. Once model can extract attention feature, we can extend one self-attention into multi-head self-attention, this extension makes sub-space features can be extracted in same time by this multi-head configuration. Overall, the multi-attention mechanism is adopt for each transformer encoder, and several of encoder layer will be stacked together to form a deep transformer encoder.
For the model input, BERT allow us take one sentence as input sequence or two sentences together as one input sequence, and the maximum length of input sequence is 512. The way that BERT was designed is for giving model the sentence-level and token-level understanding. In two sentences case, a special token ([SEP]) will be inserted between two sentences. In addition, the first input token is also a special token ([CLS]), and its corresponding ouput will be vector place for classification during fine-tuning. The outputs of the last encoder layer corresponding to each input token can be treated as word representations for each token, and the word representation of the first token ([CLS]) will be consider as classification (output) representation for further fine-tuning tasks. In BERT, this vector is denoted as $C \in \mathbb {R}^{d_{\rm model}} $, and a classification layer is denoted as $ W \in \mathbb {R}^{K \times d_{\rm model}}$, where $K$ is number of classification labels. Finally, the prediction $P$ of BERT is represented as $P = {\rm softmax}(CW^T)$.
BERT, the Bidirectional Encoder Representations from Transformers, consists of several transformer encoder layers that enable the model to extract very deep language features on both token-level and sentence-level.
Answer: BERT, the Bidirectional Encoder Representations from Transformers, consists of several transformer encoder layers that enable the model to extract very deep language features on both token-level and sentence-level.
Question: How to generate the task-specific semantic composition function?
Evidence: Deep learning models have been widely used in many natural language processing (NLP) tasks. A major challenge is how to design and learn the semantic composition function while modeling a text sequence. The typical composition models involve sequential BIBREF0 , BIBREF1 , convolutional BIBREF2 , BIBREF3 , BIBREF4 and syntactic BIBREF5 , BIBREF6 , BIBREF7 compositional models.
In spite of their success, these models have two major limitations. First, they usually use a shared composition function for all kinds of semantic compositions, even though the compositions have different characteristics in nature. For example, the composition of the adjective and the noun differs significantly from the composition of the verb and the noun. Second, different composition functions are learned from scratch in different tasks. However, given a certain natural language, its composition functions should be the same (on meta-knowledge level at least), even if the tasks are different.
To address these problems, we need to design a dynamic composition function which can vary with different positions and contexts in a sequence, and share it across the different tasks. To share some meta-knowledge of composition function, we can adopt the multi-task learning BIBREF8 . However, the sharing scheme of most neural multi-task learning methods is feature-level sharing, where a subspace of the feature space is shared across all the tasks. Although these sharing schemes are successfully used in various NLP tasks BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , they are not suitable to share the composition function.
In this paper, inspired by recent work on dynamic parameter generation BIBREF15 , BIBREF16 , BIBREF17 , we propose a function-level sharing scheme for multi-task learning, in which a shared meta-network is used to learn the meta-knowledge of semantic composition among the different tasks. The task-specific semantic composition function is generated by the meta-network. Then the task-specific composition function is used to obtain the task-specific representation of a text sequence. The difference between two sharing schemes is shown in Figure 1 . Specifically, we use two LSTMs as meta and basic (task-specific) network respectively. The meta LSTM is shared for all the tasks. The parameters of the basic LSTM are generated based on the current context by the meta LSTM, therefore the composition function is not only task-specific but also position-specific. The whole network is differentiable with respect to the model parameters and can be trained end-to-end.
We demonstrate the effectiveness of our architectures on two kinds of NLP tasks: text classification and sequence tagging. Experimental results show that jointly learning of multiple related tasks can improve the performance of each task relative to learning them independently.
Our contributions are of three-folds:
The task-specific semantic composition function is generated by the meta-network.
Answer: The task-specific semantic composition function is generated by the meta-network.
Question: Who created Craigslist?
Evidence: Craigslist (stylized as craigslist) is a privately-held American company operating a classified advertisements website with sections devoted to jobs, housing, for sale, items wanted, services, community service, gigs, résumés, and discussion forums.

Craig Newmark began the service in 1995 as an email distribution list to friends, featuring local events in the San Francisco Bay Area. It became a web-based service in 1996 and expanded into other classified categories. It started expanding to other U.S. and Canadian cities in 2000, and now covers 70 countries.
Answer: Craig Newmark created Craigslist in 1995. It started as an email distribution list to friends that featured local events in the San Francisco Bay Area.
Question: Jill, a 60-year-old personal training client, wants to improve her hip flexibility. Jill complains that her low back aches after prolonged sitting. Suggest two ways she could minimize this discomfort.
Answer: First, it is possible that Jill may have poor posture, which creates muscular fatigue, joint compression, and stresses the discs that cushion her vertebrae. She can focus on having better posture while sitting, keeping her spine straight, hips level, shoulders level, and head erect and straight. She should also balance her weight equally on both feet. If she needs to sit for extended periods of time, she can change her position often; she should periodically walk around and gently perform stretches for her muscles to relieve tension. Secondly, outside of things she could do while sitting, she can consider performing exercises for lower back care. For example, she can perform exercises such as pelvic tilts, trunk flex, partial sit-ups, or even engage in things like yoga or pilates to better strengthen her lower back as well as improve flexibility. Both of these will help reduce her lower back aches.
Question: Why should people consider going on holiday in Henley on Thames, UK?
Answer: Henley on Thames is a village in England 45 minutes west of London by train. Whether you're an avid walker, ale enthusiast, or history buff this small sanctuary on the Thames will surely hold something for you. Touted as one of the oldest market towns in England, Henley has a lively high street and village center. The storefronts include cafes, restaurants, clothing boutiques, art galleries, and antique shops - many are locally owned small businesses. Be sure to check out the award winning butcher, Gabriel Machin, located right on the town square. 

A stroll along the Thames is the perfect after lunch activity. If you head west along the village side of the river, you'll pass the River and Rowing museum. This is an excellent museum showcasing the history of the significance of the Thames in this region, as well as a Wind and the Willows exhibit for children. You can also choose to cross the bridge and walk the historic regatta path along the Thames heading east. This is where university rowing teams congregate every year for the annual regatta in July; this brings in thousands of visitors to the village during the summer months. 

You won't struggle for choice of the towns many historic pubs and restaurants. Enjoy a pint outside on the river deck at the Angel on the Bridge before making you way to one of the towns many restaurant and food options.
Question: Eminem's 8 Mile is named after a road in which city?
Answer: Detroit.
Question: Tell me whether these are alcoholic drinks or non-alcoholic drinks: soda, wine, milk, beer, fruit juice, sparkling water, vodka, tequila
Answer: Soda: non-alcoholic
Wine: alcoholic
Milk: non-alcoholic
Beer: alcoholic
Fruit juice: non-alcoholic
Sparkling water: non-alcoholic
Vodka: alcoholic
Tequila: alcoholic
Question: Why are drums so important to music, and what this appeal to have live drums in the modern world where drum beats can easily be sampled, and/or perfectly mimicked  by software?
Answer: Music is often broken down into fundamental components using this model of understanding: Melody, Harmony, and Rhythm.  Of these three it is true rhythm is the easiest for computers to do a near perfect job of imitating.  Advancements in technology from the first drum machines that appeared in the 1970s, to widely used beat machines of the 1990's, to software music composition suites of today means you now have a lot of control at youf finger tips to create rhythm. You can now create drum beats that account for tempo variations, pitch variations, changes in dynamics, how close the drum pattern stays on beat, the type of hit the beat should be using, what the tension on the drum heads should sound like, how much rebound against the drum head is present, velocity of the stick, the simulated room size of where the drum kit is located,  how much resonance is vibrating in the drum kit itself, tonal variance of the wood shells used to create the drum beat and many more.   These advancements make software drumbeats realistic, and indistinguishable from their human counterparts yet the role of the drummer in modern music could not be more important.  Humans love live music, and very few commercially successful live musical acts do not use a live drummer.  Even rap and hip hop--two musical genres famous for ushering in the area of "sampled" drum beats, and "sampled song loops"-- often will have a live drummer on stage despite the ease of which the music can be broadcast loudly without requiring one.   The reason is simple: live drums and the musicians who perform using them are both visually and sonically exciting.  There are genres where a drummer is not provided such as "DJ Couture" and "Electronic Dance music" (or EDM) but they are the exceptions and not the rule. From the earliest humans banging rocks and sticks together, to the modern drums of today the future of the live drummer in music has never been brighter.
Question: How do I travel from SF to Aspen ski resorts?
Answer: During winter season, United airline operates direct flight from SFO to ASE a few times a week. ASE is the local airport of Aspen. Once you arrive at ASE, most hotels and resorts offer complimentary shuttle to and from the local airport. Direct flight usually ends around the first week of April, so if that's not an option, you can either take a 1-stop flight from SFO to ASE, or fly into Denver then drive to Aspen. The drive is about 4.5 hours if the road condition is okay. If cost is a concern, flying to Denver and use rental car is usually cheaper.
Question: 5 best mailing sites
Answer: google
msn'
reddif
yahoo
outlook
Question: What are some ways I can improve my general health during my work day?
Answer: There are a number of things you can do to improve your general health during your working day. You can use a standing desk or a desk treadmill. You may be able to go for a walk while on a phone or conference call. It is also good to try and fit in some time for activities such as visiting the gym, doing yoga or meditating.
Question: What can I do at the airport?
Answer: When you get to the airport there are a lot of things you can do.  You can eat a meal, stand at the window of the terminal and watch planes land and take off, or play.  You can also sit and read a good book, meditate, drink alcohol,  exercise by walking between different terminals, or get to know someone new sitting next you.  If none of these things interest you then you can also work on your smartphone or computer, watch a movie on them, browse pictures and images, or play game.
Question: Given this paragraph, who wrote "August?"
Evidence: August (2001), is the first novel by author Gerard Woodward. It was shortlisted for Whitbread Book Award (2001).
Answer: Gerard Woodward
Question: What is the common point shared by previous approaches？
Evidence: Neural machine translation (NMT) emerged in the last few years as a very successful paradigm BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . While NMT is generally more fluent than previous statistical systems, adequacy is still a major concern BIBREF4 : common mistakes include dropping source words and repeating words in the generated translation.
Previous work has attempted to mitigate this problem in various ways. BIBREF5 incorporate coverage and length penalties during beam search—a simple yet limited solution, since it only affects the scores of translation hypotheses that are already in the beam. Other approaches involve architectural changes: providing coverage vectors to track the attention history BIBREF6 , BIBREF7 , using gating architectures and adaptive attention to control the amount of source context provided BIBREF8 , BIBREF9 , or adding a reconstruction loss BIBREF10 . BIBREF11 also use the notion of fertility implicitly in their proposed model. Their “fertility conditioned decoder” uses a coverage vector and an “extract gate” which are incorporated in the decoding recurrent unit, increasing the number of parameters.
In this paper, we propose a different solution that does not change the overall architecture, but only the attention transformation. Namely, we replace the traditional softmax by other recently proposed transformations that either promote attention sparsity BIBREF12 or upper bound the amount of attention a word can receive BIBREF13 . The bounds are determined by the fertility values of the source words. While these transformations have given encouraging results in various NLP problems, they have never been applied to NMT, to the best of our knowledge. Furthermore, we combine these two ideas and propose a novel attention transformation, constrained sparsemax, which produces both sparse and bounded attention weights, yielding a compact and interpretable set of alignments. While being in-between soft and hard alignments (Figure FIGREF20 ), the constrained sparsemax transformation is end-to-end differentiable, hence amenable for training with gradient backpropagation.
To sum up, our contributions are as follows:
Wu et al. (2016) incorporate coverage and length penalties during beam search—a simple yet limited solution, since it only affects the scores of translation hypothe-ses that are already in the beam.Other approaches involve architectural changes: provid-ing coverage vectors to track the attention history (Mi et al., 2016; Tu et al., 2016), using gating architectures and adaptive attention to control the amount of source context provided (Tu et al., 2017a; Li and Zhu, 2017), or adding a reconstruction loss (Tu et al., 2017b). Feng et al. (2016) also use the notion of fertility implicitly in their proposed model. Their fertility conditioned decoder uses a coverage vector and an extract gate which are incorporated in the decoding recurrent unit, in-creasing the number of parameters.
Answer: They all tried to slove the problem by amending the architecture。
Question: What did Wu et al. do to remit the inadequacy of NMT?
Evidence: Neural machine translation (NMT) emerged in the last few years as a very successful paradigm BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . While NMT is generally more fluent than previous statistical systems, adequacy is still a major concern BIBREF4 : common mistakes include dropping source words and repeating words in the generated translation.
Previous work has attempted to mitigate this problem in various ways. BIBREF5 incorporate coverage and length penalties during beam search—a simple yet limited solution, since it only affects the scores of translation hypotheses that are already in the beam. Other approaches involve architectural changes: providing coverage vectors to track the attention history BIBREF6 , BIBREF7 , using gating architectures and adaptive attention to control the amount of source context provided BIBREF8 , BIBREF9 , or adding a reconstruction loss BIBREF10 . BIBREF11 also use the notion of fertility implicitly in their proposed model. Their “fertility conditioned decoder” uses a coverage vector and an “extract gate” which are incorporated in the decoding recurrent unit, increasing the number of parameters.
In this paper, we propose a different solution that does not change the overall architecture, but only the attention transformation. Namely, we replace the traditional softmax by other recently proposed transformations that either promote attention sparsity BIBREF12 or upper bound the amount of attention a word can receive BIBREF13 . The bounds are determined by the fertility values of the source words. While these transformations have given encouraging results in various NLP problems, they have never been applied to NMT, to the best of our knowledge. Furthermore, we combine these two ideas and propose a novel attention transformation, constrained sparsemax, which produces both sparse and bounded attention weights, yielding a compact and interpretable set of alignments. While being in-between soft and hard alignments (Figure FIGREF20 ), the constrained sparsemax transformation is end-to-end differentiable, hence amenable for training with gradient backpropagation.
To sum up, our contributions are as follows:
Wu et al. (2016) incorporate coverage and length penalties during beam search—a simple yet limited solution, since it only affects the scores of translation hypothe-ses that are already in the beam.
Answer: They incorporated coverage and length penalties during beam search.
Question: What happened to Captain Bligh after the Mutiny on the Bounty?
Answer: The mutineers put then-Lieutenant Bligh and eighteen other crew into a small, open boat and set them adrift in the middle of the South Pacific ocean.  Bligh and his men had just a small amount of food and water.  Bligh used his memory of navigation charts to plan a course westward to the Dutch colony of Timor.  

They initially tried stopping at an island to gather supplies.  The island was inhabited, however, and they were attacked.  One crew member was killed and the rest barely escaped.  From then on they dared not stop at any island until reaching the Great Barrier Reef of Australia (then called New Holland by the Europeans).

In a remarkable achievement of seamanship, Bligh and his men sailed and rowed their open boat for 41 days over a distance of 3618 miles, in sometimes harsh weather conditions.  Food and drink had to be strictly rationed, and the men suffered greatly from hunger and thirst.  Except for the death of the crewman at the hands of the islanders, all the remaining crew reached Timor alive.  Sadly, some men died of disease contracted at the colony town of Batavia or on the final return voyage to Europe.  Of the original nineteen men set adrift by the Bounty mutineers, twelve returned alive to England.

Bligh continued his career in the British Navy.  He was assigned a new ship and once again sailed to Tahiti take breadfruit to British colonies in the Caribbean.  This time, the mission was completed successfully.  Bligh commanded additional ships, including fighting in several major naval battles.  In 1805, Bligh was appointed Governor of New South Wales.  He ended his British Naval career as a Vice Admiral.  William Bligh died in London on Dec 12, 1817.
Question: How is the CoLA grammatically annotated?
Evidence: The effectiveness and ubiquity of pretrained sentence embeddings for natural language understanding has grown dramatically in recent years. Recent sentence encoders like OpenAI's Generative Pretrained Transformer BIBREF3 and BERT BIBREF2 achieve the state of the art on the GLUE benchmark BIBREF4 . Among the GLUE tasks, these state-of-the-art systems make their greatest gains on the acceptability task with the Corpus of Linguistic Acceptability BIBREF0 . CoLA contains example sentences from linguistics publications labeled by experts for grammatical acceptability, and written to show subtle grammatical features. Because minimal syntactic differences can separate acceptable sentences from unacceptable ones (What did Bo write a book about? / *What was a book about written by Bo?), and acceptability classifiers are more reliable when trained on GPT and BERT than on recurrent models, it stands to reason that GPT and BERT have better implicit knowledge of syntactic features relevant to acceptability.
Among the GLUE tasks, these state-of-the-art systems make their greatest gains on the acceptability task with the Corpus of Linguistic Acceptability BIBREF0 . CoLA contains example sentences from linguistics publications labeled by experts for grammatical acceptability, and written to show subtle grammatical features. Because minimal syntactic differences can separate acceptable sentences from unacceptable ones (What did Bo write a book about? / *What was a book about written by Bo?), and acceptability classifiers are more reliable when trained on GPT and BERT than on recurrent models, it stands to reason that GPT and BERT have better implicit knowledge of syntactic features relevant to acceptability.
Answer: The answers are shown as follows:
* labeled by experts
Question: Give me a bulleted list of ways to get around New York City.
Answer: The following is a bulleted list of ways to get around New York City:
- The New York City Subway
- A taxi cab
- Walking
- A rideshare car service such as Uber or Lyft
- A rideshare moped service such as Revel
- New York City's public buses
- New York City's ferry service
- A rideshare bicycle such as Citibike
- Driving your personal vehicle
- Riding your personal bicycle
- Rollerblading or roller-skating
- Taking a pedicab
- Taking a scooter
Question: Who won the 1988 Superbowl
Answer: Washington Redskins