Labels: positive, negative, or neutral; INV: same pred. (INV) after removals/ additions; DIR: sentiment should not decrease  $(\uparrow)$  or increase  $(\downarrow)$
Test TYPE and Description	Failure Rate (%)	Example test cases & expected behavior
G	a	RoB
Vocab.+POS	MFT: Short sentences with neutral adjectives and nouns	0.0	7.6	4.8	94.6	81.8	The company is Australian. neutralThat is a private aircraft. neutral
MFT: Short sentences with sentiment-laden adjectives	4.0	15.0	2.8	0.0	0.2	That cabin crew is extraordinary. posI despised that aircraft. neg
INV: Replace neutral words with other neutral words	9.4	16.2	12.4	10.2	10.2	@Virgin should I be concerned that → when I'm about to fly ... INV@united the → our nightmare continues... INV
DIR: Add positive phrases, fails if sent. goes down by > 0.1	12.6	12.4	1.4	0.2	10.2	@SouthwestAir Great trip on 2672 yesterday... You are extraordinary. ↑@AmericanAir AA45 ... JFK to LAS. You are brilliant. ↑
DIR: Add negative phrases, fails if sent. goes up by > 0.1	0.8	34.6	5.0	0.0	13.2	@USAirways your service sucks. You are lame. ↓@JetBlue all day. I abhor you. ↓
Robust.	INV: Add randomly generated URLs and handles to tweets	9.6	13.4	24.8	11.4	7.4	@JetBlue that selfie was extreme. @pi9QDK INV@united stuck because staff took a break? Not happy 1K.... https://t.co/PWK1jb INV
INV: Swap one character with its neighbor (typ)	5.6	10.2	10.4	5.2	3.8	@JetBlue → @JeBlue I cri INV@SouthwestAir no thanks → thakns INV
NER	INV: Switching locations should not change predictions	7.0	20.8	14.8	7.6	6.4	@JetBlue I want you guys to be the first to fly to # Cuba → Canada... INV@VirginAmerica I miss the #nerdbird in San Jose → Denver INV
INV: Switching person names should not change predictions	2.4	15.1	9.1	6.6	2.4	...Airport agents were horrendous. Sharon → Erin was your saviour INV@united 8602947, Jon → Sean at http://t.co/58tuTgli0D, thanks. INV
Temporal	MFT: Sentiment change over time, present should prevail	41.0	36.6	42.2	18.8	11.0	I used to hate this airline, although now I like it. posIn the past I thought this airline was perfect, now I think it is creepy. neg
Negation	MFT: Negated negative should be positive or neutral	18.8	54.2	29.4	13.2	2.6	The food is not poor. pos or neutralIt isn't a lousy customer service. pos or neutral
MFT: Negated neutral should still be neutral	40.4	39.6	74.2	98.4	95.4	This aircraft is not private. neutralThis is not an international flight. neutral
MFT: Negation of negative at the end, should be pos. or neut.	100.0	90.4	100.0	84.8	7.2	I thought the plane would be awful, but it wasn't. pos or neutralI thought I would dislike that plane, but I didn't. pos or neutral
MFT: Negated positive with neutral content in the middle	98.4	100.0	100.0	74.0	30.2	I wouldn't say, given it's a Tuesday, that this pilot was great. negI don't think, given my history with airplanes, that this is an amazing staff. neg
SRL	MFT: Author sentiment is more important than of others	45.4	62.4	68.0	38.8	30.0	Some people think you are excellent, but I think you are nasty. negSome people hate you, but I think you are exceptional. pos
MFT: Parsing sentiment in (question, “yes”) form	9.0	57.6	20.8	3.6	3.0	Do I think that airline was exceptional? Yes. negDo I think that is an awkward customer service? Yes. neg
MFT: Parsing sentiment in (question, “no”) form	96.8	90.8	81.6	55.4	54.8	Do I think the pilot was fantastic? No. negDo I think this company is bad? No. pos or neutral