Sample of responses from different models on SQuAD-dev-test set §3.2. "Q:" is the Question, "R:" is the Response, "B." stands for baseline and "(o)" stands for oracle answer spans.
Model	Q:what fixed set of factors determine the actions of a determinis-tic Turing machine	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: fixed set of rules	✓	✗	-	b
QuAC B.	R: a deterministic Turing machine is the most basic Turing ma-chine , which uses a fixed set of rules to determine its future ac-tions	✓	✓	✓	e
STs+BERT B.	R: rules determine the actions	✓	✓	✓	e
PGN-Pre with SS+	R: fixed set of rules determine the actions	✓	✓	✓	e
GPT-2 with SS+	R: fixed set of rules fixed set of factors determine the actions	✓	✓	✗	d
GPT-2-Pre with SS+	R: fixed set of rules determine the actions	✓	✓	✓	e
D-GPT with SS+	R: fixed set of rules determine the actions	✓	✓	✓	e
D-GPT with SS+ (o)	R: rules determine the actions	✓	✓	✓	e
Model	Q:why was polonia relegated from the country 's top flight in 2013 ?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: because of their disastrous financial situation	✓	✗	-	b
QuAC B.	R: polonia was relegated from the country 's top flight in 2013 because of their disastrous financial situation	✓	✓	✓	e
LGRs+BERT B.	R: polonia was relegated from the country 's top flight because the disastrous financial situation	✓	✓	✗	d
PGN-Pre with SS+	R: polonia was relegated disastrous financial situation	✓	✓	✗	d
GPT-2 with SS+	R: polonia was relegated disastrous financial situation	✓	✓	✗	d
GPT-2-Pre with SS+	R: polonia was relegated disastrous financial situation	✓	✓	✗	d
D-GPT with SS+	R: polonia was relegated from the country's top flight for disas-trous financial situation	✓	✓	✓	e
D-GPT with SS+ (o)	R: polonia was relegated from the country's top flight for disas-trous financial situation	✓	✓	✓	e
Model	Q:how was sadat rewarded by the islamists for his attempts to bring egypt into modern times and civilization ?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: released Islamists from prison	✗	✗	-	a
QuAC B.	R: egyptian president anwar sadat - whose policies included opening egypt to western investment ( infitah ) ; transferring egypt 's allegiance from the soviet union to the united states	✗	✓	-	c
LGRs+BERT B.	R: sadat was rewarded by the islamists for his attempts to bring egypt assassinated	✓	✓	✗	d
PGN-Pre with SS+	R: sadat was rewarded piety and charity	✗	✓	-	c
GPT-2 with SS+	R: it was rewarded piety and charity	✗	✓	-	c
GPT-2-Pre with SS+	R: sadat was rewarded by piety and charity	✗	✓	-	c
D-GPT with SS+	R: sadat was rewarded piety and charity	✗	✓	-	c
D-GPT with SS+ (o)	R: sadat was assassinated	✓	✓	✓	e
Model	Q:from what type of materials must liquid oxygen be separated?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: combustible materials	✓	X	-	b
QuAC B.	R: both liquid and solid o n2 are clear substances with a light sky-blue color caused by absorption in the red	X	✓	-	c
STs+BERT B.	R: combustible materials	✓	X	-	b
PGN-Pre with SS+	R: it must be separated from combustible	✓	✓	✓	e
GPT-2 with SS+	R: it must be separated from combustible	✓	✓	✓	e
GPT-2-Pre with SS+	R: they must be separated from combustible	✓	✓	✓	e
D-GPT with SS+	R: liquid oxygen must be separated from combustible	✓	✓	✓	e
D-GPT with SS+ (o)	R: liquid oxygen must be separated from combustible materials	✓	✓	✓	e
Model	Q:on what type of exponentiation does the diffie - hellman key exchange depend on ?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: modular exponentiation	✓	X	-	b
QuAC B.	R: several public-key cryptography algorithms , such as rsa and the diffie - hellman key exchange	X	X	-	a
STs+BERT B.	R: modular exponentiation	✓	X	-	b
PGN-Pre with SS+	R: it depends on modular	✓	✓	✓	e
GPT-2 with SS+	R: it depends on modular	✓	✓	✓	e
GPT-2-Pre with SS+	R: it depends on the modular	✓	✓	✓	e
D-GPT with SS+	R: it depends on modular	✓	✓	✓	e
D-GPT with SS+ (o)	R: it depends on modular exponentiation	✓	✓	✓	e
Model	Q:what was sadat seeking by releasing islamists from prison ?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: political support in his struggle against leftists	✓	✓	✓	e
QuAC B.	R: egyptian president anwar sadat - whose policies included opening egypt to western investment ( infitah ) ; transferring egypt 's allegiance from the soviet union to the united states	X	✓	-	c
STs+BERT B.	R: sadat was seeking political support	✓	✓	✓	e
PGN-Pre with SS+	R: it was making peace with israel	X	✓	-	c
GPT-2 with SS+	R: it was making peace with israel	X	✓	-	c
GPT-2-Pre with SS+	R: sadat seeking was making peace with israel	X	✓	-	c
D-GPT with SS+	R: sadat was making peace with israel	X	✓	-	c
D-GPT with SS+ (o)	R: sadat was seeking political support	✓	✓	✓	e
Model	Q:how did the better jacksonville plan generate money ?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: authorized a half-penny sales tax	✓	X	-	b
QuAC B.	R: when a consolidation referendum was held in 1967 , voters approved the plan . on october 1 , 1968 , the governments merged to create the consolidated city of jacksonville	X	✓	-	c
STs+BERT B.	R: it generated money authorized a half-penny sales tax	✓	✓	X	d
PGN-Pre with SS+	R: it generated money half-penny sales tax	✓	✓	X	d
GPT-2 with SS+	R: it generated money half-penny sales tax	✓	✓	✗	d
GPT-2-Pre with SS+	R: it generated half-penny sales tax money	✓	✓	✓	e
D-GPT with SS+	R: it generated money for half-penny sales tax	✓	✓	✓	e
D-GPT with SS+(o)	R: it generated money authorized a half-penny sales tax	✓	✓	✗	d
Model	Q:how many items is the university library home to ?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: over two million	✓	✗	-	b
QuAC B.	R: another important library – the university library, founded in 1816 , is home to over two million items	✓	✓	✓	e
STs+BERT B.	R: it to is over two million	✓	✓	✗	d
PGN-Pre with SS+	R: it is home to over two million	✓	✓	✓	e
GPT-2 with SS+	R: it is home to over two million	✓	✓	✓	e
GPT-2-Pre with SS+	R: it is home to over two million	✓	✓	✓	e
D-GPT with SS+	R: it to is over two million	✓	✓	✗	d
D-GPT with SS+(o)	R: it to is over two million	✓	✓	✗	d
Model	Q:which sea was oil discovered in ?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: North Sea	✓	✗	-	b
QuAC B.	R: ” it ’ s scotland ’ s oil ” campaign of the scottish national party ( snp )	✗	✗	-	a
LGRs+BERT B.	R: oil was discovered in north	✓	✓	✗	d
PGN-Pre with SS+	R: oil was discovered in the north sea	✓	✓	✓	e
GPT-2 with SS+	R: oil was discovered in the north sea	✓	✓	✓	e
GPT-2-Pre with SS+	R: it was discovered in the north sea	✓	✓	✓	e
D-GPT with SS+	R: it was discovered in the north sea	✓	✓	✓	e
D-GPT with SS+(o)	R: oil was discovered in north	✓	✓	✗	d
Model	Q:where are jersey and guernsey	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: Channel Islands	✓	✗	-	b
QuAC B.	R: the customary law of normandy was developed between the 10th and 13th centuries and survives today through the legal systems of jersey and guernsey in the channel islands	✓	✓	✓	e
LGRs+BERT B.	R: they are in channel islands	✓	✓	✓	e
PGN-Pre with SS+	R: they are in the channel islands	✓	✓	✓	e
GPT-2 with SS+	R: they are on the channel islands	✓	✓	✓	e
GPT-2-Pre with SS+	R: they are on the channel islands	✓	✓	✓	e
D-GPT with SS+	R: they are in the channel islands	✓	✓	✓	e
D-GPT with SS+(o)	R: they are in channel islands	✓	✓	✓	e
Model	Q:near chur , which direction does the rhine turn ?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: north	✓	✗	-	b
QuAC B.	R: near tamins-reichenau the anterior rhine and the posterior rhine join and form the rhine	X	✓	-	c
LGRs+BERT B.	R: it turns north	✓	✓	✓	e
PGN-Pre with SS+	R: it turns north	✓	✓	✓	e
GPT-2 with SS+	R: it turns north	✓	✓	✓	e
GPT-2-Pre with SS+	R: it turns to the north	✓	✓	✓	e
D-GPT with SS+	R: it turns north	✓	✓	✓	e
D-GPT with SS+ (o)	R: it turns north	✓	✓	✓	e
Model	Q: what kind of contract is given when the contractor is given a performance specification and must undertake the project from design to construction, while adhering to the performance specifications?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: design build" contract	✓	X	-	b
QuAC B.	R: the modern trend in design is toward integration of previously separated specialties, especially among large firms	X	✓	-	c
LGRs+BERT B.	R: a "design build" contract is given	✓	✓	✓	e
PGN-Pre with SS+	R: design build is given	✓	✓	✓	e
GPT-2 with SS+	R: the design build is given	✓	✓	✓	e
GPT-2-Pre with SS+	R: design build is given a performance specification and must undertake the project	✓	✓	✓	e
D-GPT with SS+	R: design build is given	✓	✓	✓	e
D-GPT with SS+ (o)	R: the "design build" contract is given	✓	✓	✓	e
Model	Q: how many protestants live in france today?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: Approximately one million	✓	X	-	b
QuAC B.	R: approximately one million protestants in modern france repre-sent some 2% of its population	✓	✓	✓	e
LGRs+BERT B.	R: one million live in france today	✓	✓	✓	e
PGN-Pre with SS+	R: one million live in france today	✓	✓	✓	e
GPT-2 with SS+	R: one million live in france today	✓	✓	✓	e
GPT-2-Pre with SS+	R: one million live in france today	✓	✓	✓	e
D-GPT with SS+	R: one million live in france today	✓	✓	✓	e
D-GPT with SS+ (o)	R: one million live in france today	✓	✓	✓	e
Model	Q: what is raghuram rajan's career?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: Central Banking economist	✓	X	-	b
QuAC B.	R: central banking economist raghuram rajan argues that " systemic economic inequalities	✓	X	-	b
LGRs+BERT B.	R: he is economist	✓	✓	X	d
PGN-Pre with SS+	R: it is central banking economist	✓	✓	✓	e
GPT-2 with SS+	R: it is central banking economist	✓	✓	✓	e
GPT-2-Pre with SS+	R: it is central banking economist	✓	✓	✓	e
D-GPT with SS+	R: it is central banking economist	✓	✓	✓	e
D-GPT with SS+ (o)	R: he is economist	✓	✓	X	d
Model	Q:what type of steam engines produced most power up to the early 20th century?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: Reciprocating piston type steam engines	✓	X	-	b
QuAC B.	R: reciprocating piston type steam engines remained the dominant source of power until the early 20th century, when advances in the design of electric motors and internal combustion engines	✓	✓	✓	e
LGRs+BERT B.	R: reciprocating piston produced most power up	✓	✓	X	d
PGN-Pre with SS+	R: reciprocating piston type produced most power up	✓	✓	X	d
GPT-2 with SS+	R: reciprocating piston type produced most power up	✓	✓	X	d
GPT-2-Pre with SS+	R: the reciprocating piston type produced most power up to the early 20th century	✓	✓	✓	e
D-GPT with SS+	R: reciprocating piston type produced most power up to the early 20th century	✓	✓	✓	e
D-GPT with SS+(o)	R: reciprocating piston produced most power up to the early 20th century	✓	✓	✓	e
Model	Q:where did france win a war in the 1950 's	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: Algeria	✓	X	-	b
QuAC B.	R: france fought and lost a bitter war in vietnam in the 1950s	X	✓	-	c
LGRs+BERT B.	R: france won a war in the 1950 's algeria	✓	✓	✓	e
PGN-Pre with SS+	R: france won a war in vietnam	X	✓	-	c
GPT-2 with SS+	R: france won a war in vietnam	X	✓	-	c
GPT-2-Pre with SS+	R: france won a war in vietnam	X	✓	-	c
D-GPT with SS+	R: france won a war in vietnam	X	✓	-	c
D-GPT with SS+(o)	R: france won a war in algeria	✓	✓	✓	e
Model	Q:who did the ottoman empire ally with in ww i?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: Germany	✓	X	-	b
QuAC B.	R: the ottoman empire gradually declined into the late nineteenth century . the empire allied with germany	✓	✓	✓	e
LGRs+BERT B.	R: germany did the ottoman empire ally with in ww i	✓	✓	X	d
PGN-Pre with SS+	R: it separated with germany	X	✓	-	c
GPT-2 with SS+	R: it allyed with germany	✓	✓	✓	e
GPT-2-Pre with SS+	R: it allyed with germany	✓	✓	✓	e
D-GPT with SS+	R: it allied germany	✓	✓	X	d
D-GPT with SS+(o)	R: it allied germany	✓	✓	X	d
Model	Q:when was ambulatory care pharmacy approved as its own certification ?	correctness	complete-sentence	grammaticality	majority option
CoQA B.	R: In 2011	✓	X	-	b
QuAC B.	R: in 2011 the board of pharmaceutical specialties approved ambulatory care pharmacy practice as a separate board certification	✓	✓	✓	e
LGRs+BERT B.	R: it was approved in 2011	✓	✓	✓	e
PGN-Pre with SS+	R: it was approved in 2011	✓	✓	✓	e
GPT-2 with SS+	R: it was approved in 2011	✓	✓	✓	e
GPT-2-Pre with SS+	R: it was approved in 2011	✓	✓	✓	e
D-GPT with SS+	R: it was approved in 2011	✓	✓	✓	e
D-GPT with SS+ (o)	R: it was approved in 2011	✓	✓	✓	e
Model	Q:when did ARPnet and sita become operational	correctness	complete-sentence	grammaticality	majority option
Q:how much did saudi arabia spend on spreading wahhabism?	correctness	complete-sentence	grammaticality	majority option
✓	✓	✓	e
CoQA B.	R: over 100 billion dollars	✓	✓	-	b
QuAC B.	R: saudi arabia spent over 100 billion dollars in the ensuing decades for helping spread its fundamentalist interpretation of is-lam	✓	✓	✓	e
LGRs+BERT B.	R: saudi arabia spent over 100 billion dollars	✓	✓	✓	e
PGN-Pre with SS+	R: saudi arabia spent over 100 billion dollars	✓	✓	✓	e
GPT-2 with SS+	R: saudi arabia spent over 100 billion dollars	✓	✓	✓	e
GPT-2-Pre with SS+	R: saudi arabia spent over 100 billion dollars	✓	✓	✓	e
D-GPT with SS+	R: saudi arabia spent over 100 billion dollars	✓	✓	✓	e
D-GPT with SS+ (o)	R: saudi arabia spent over 100 billion dollars	✓	✓	✓	e