SlowGuess/NAACL_2022_OCR huggingface dataset:
WordSwapChangeLocation, WordSwapChangeName, WordSwapChangeNumber, WordSwapContract, WordSwapExtend, WordSwapHomoglyphSwap, WordSwapMaskedLM, WordSwapQWERTY, WordSwapNeighboringCharacterSwap, WordSwapRandomCharacterDeletion, WordSwapRandomCharacterInsertion, WordSwapRandomCharacterSubstitution, RandomSwap, and WordSwapWordNet.
Test Transformation Name	Failure Rate (%)	Dataset Size Reduction (%)
D	D+COVER	D+MNCOVER	ΔD+MNCOVER	D+COVER	D+MNCOVER
Change first name in one of the questions	63.00	100.0059.40	100.0064.49	37.001.49	98.20	98.20
add one typo	19.40	28.5723.17	29.4123.52	10.014.12	88.00	88.00
Product of paraphrases(q1) * paraphrases(q2)	95.00	100.00100.00	100.00100.00	5.005.00	99.00	99.00
Replace synonyms in real pairs	8.37	13.337.89	12.508.81	4.130.44	73.31	73.31
Symmetry: f(a, b) = f(b, a)	6.00	8.334.83	10.004.61	4.00-1.39	82.00	82.00
Testing implications	15.07	15.257.90	15.257.46	0.19-7.60	99.29	99.29
same adjectives, different people v3	100.00	100.00100.00	100.00100.00	0.000.00	81.82	81.82
same adjectives, different people	100.00	100.00100.00	100.00100.00	0.000.00	81.48	81.48
Change same location in both questions	5.00	0.000.00	0.000.00	-5.00-5.00	96.60	96.60
Average Improvement	-	5.96-0.96	6.15-0.33	6.15-0.33	88.86	88.86
Table 6: Failure Rate(%) obtained using BERT model on the original dataset  $D$ , the dataset filtered using COVER coverage (D+COVER columns) and the dataset filtered with MNCOVER coverage (D+MNCOVER columns) from the QQP Suite. We report both the max failure rate as well as the mean in the subscript across 10 thresholds of coverage. Rows are sorted regarding the failure rate difference between the dataset filtered using MNCOVER and the original dataset (column  $\Delta_{D + \mathrm{MNCOVER}}$ ). We use 200 samples in this case.