Label: duplicate  $\equiv$ , or non-duplicate  $\neq$ ; INV: same pred. (INV) after removals/ additions
Test TYPE and Description	Failure Rate	Example Test cases & expected behavior
RoB
Vocab.	MFT: Modifiers changes question intent	78.4	78.0	{ Is Mark Wright a photographer? | Is Mark Wright an accredited photographer? } ≠
Taxonomy	MFT: Synonyms in simple templates	22.8	39.2	{ How can I become more vocal? | How can I become more outspoken? } =
INV: Replace words with synonyms in real pairs	13.1	12.7	{ Is it necessary to follow a religion?
{ is it necessary to follow an organized → organised religion? } INV
MFT: More X = Less antonym(X)	69.4	100.0	{ How can I become more optimistic? | How can I become less pessimistic? } =
Robust.	INV: Swap one character with its neighbor (typ)	18.2	12.0	{ Why am I getting → gettnig lazy? | Why are we so lazy? } INV
DIR: Paraphrase of question should be duplicate	69.0	25.0	{ Can I gain weight from not eating enough?
{ Can I → Do you think I can gain weight from not eating enough? } =
NER	INV: Change the same name in both questions	11.8	9.4	{ Why isn't Hillary Clinton → Nicole Perez in jail?
{ Is Hillary Clinton → Nicole Perez going to go to jail? } INV
DIR: Change names in one question, expect ≠	35.1	30.1	{ What does India think of Donald Trump?
{ What India thinks about Donald Trump → John Green? } ≠
DIR: Keep first word and entities of a question, fill in the gaps with RoBERTa; expect ≠	30.0	32.8	{ Will it be difficult to get a US Visa if Donald Trump gets elected?
{ Will the US accept Donald Trump? } ≠
Temporal	MFT: Is ≠ used to be, non-duplicate	61.8	96.8	{ Is Jordan Perry an advisor? | Did Jordan Perry use to be an advisor? } ≠
MFT: before ≠ after, non-duplicate	98.0	34.4	{ Is it unhealthy to eat after 10pm? | Is it unhealthy to eat before 10pm? } ≠
MFT: before becoming ≠ after becoming	100.0	0.0	{ What was Danielle Bennett's life before becoming an agent?
{ What was Danielle Bennett's life after becoming an agent? } ≠
Negation	MFT: simple negation, non-duplicate	18.6	0.0	{ How can I become a person who is not biased? | How can I become a biased person? } ≠
MFT: negation of antonym, should be duplicate	81.6	88.6	{ How can I become a positive person? | How can I become a person who is not negative? } ≠
Coref	MFT: Simple coreference: he ≠ she	79.0	96.6	{ If Joshua and Chloe were alone, do you think he would reject her?
{ If Joshua and Chloe were alone, do you think she would reject him? } ≠
MFT: Simple resolved coreference, his and her	99.6	100.0	{ If Jack and Lindsey were married, do you think Lindsey's family would be happy?
{ If Jack and Lindsey were married, do you think his family would be happy? } ≠
SRL	MFT: Order is irrelevant for comparisons	99.6	100.0	{ Are tigers heavier than insects? | What is heavier, insects or tigers? } =
MFT: Orders is irrelevant in symmetric relations	81.8	100.0	{ Is Nicole related to Heather? | Is Heather related to Nicole? } =
MFT: Order is relevant for asymmetric relations	71.4	100.0	{ Is Sean hurting Ethan? | Is Ethan hurting Sean? } ≠
MFT: Active / passive swap, same semantics	65.8	98.6	{ Does Anna love Benjamin? | Is Benjamin loved by Anna? } =
MFT: Active / passive swap, different semantics	97.4	100.0	{ Does Danielle support Alyssa? | Is Danielle supported by Alyssa? } ≠
Logic	INV: Symmetry: pred(a, b) = pred(b, a)	4.4	2.2	{ (q1, q2) | (q2, q1) } INV
DIR: Implications, eg. (a=b) ∧ (a=c)⇒(b=c)	9.7	8.5	no example