SlowGuess/NAACL_2021_OCR huggingface dataset:
To ensure the grammar correctness of the heuristically generated logical negation events, we add appropriate auxiliary verbs (e.g., do, does, did, is, was, can, could, would, should, may, might) in accordance with the tenses (e.g., present, past, future) of the original events. Since NLTK's POS parser fails to recognize some of the verbs that have both noun and verb usage (e.g., "waters" the plant, "supports" her argument), we curate a list of dual-used words and map them manually. Also, while converting the original events to their logical negation counterparts, we revise grammar mistakes from ATOMIC and exclude awkward expressions as much as possible. In addition, to make the negation forms sound more natural, we replace the modifier "some" by "any" during conversion (e.g., "PersonX buys some shoes" is converted to "PersonX doesn't buy any shoes"). For the minority of compound events with clauses or complex sentence structures, we disregard them for the purpose of ensuring the data quality.
For semi-logical negation events, we curate a list of semi-logical negation cues besides not from various sources $^{5}$  (Council et al., 2010; Hossain et al., 2020; Kim et al., 2019) and categorize them into four types including affixes, single-word cues, multi-word cues and negative verbs (Table 1). We identify appropriate rules to insert each semi-logical negation cue in simple base events from ATOMIC consisting of a subject and a predicate. We apply the rules to original events from ATOMIC and randomly select at least 200 automatically generated semi-logical negation events per each negation cue for manual screening by the first
author to avoid misplacement of negation cues and awkward expressions. In the end, we were able to identify 5,019 high quality semi-logical negation events originating from ATOMIC.
As a final quality control step of the constructed logical and semi-logical events, after obtaining the crowdsourced inferences for each event, we remove all events that annotators comment as "unclear," "doesn't make sense" or "grammatically wrong."
Crowdsourcing of Commonsense Contradiction Events For collecting commonsense contradiction events, we present an original ATOMIC event to the annotators and ask them to formulate corresponding opposite events. We exclude ATOMIC events with placeholders representing generic objects) to capture semantic and pragmatic subtlety. In the MTurk task, we present annotators detailed instructions of formulating the opposite events (e.g., avoid using negative words as much as possible, use complete sentences, follow grammar rules) and concrete examples as references. Figure 2 shows details of the MTurk task. Although we explicitly instruct annotators to avoid using negation cues, there are still some exceptions. Therefore, after the compilation of all commonsense contradiction events, we remove ones that contain any explicit negation cues to make sure the categorization is clean.
Crowdsourcing of ANION Event Inferences For the collection of ANION event inferences, we adopt the MTurk templates used by the original ATOMIC data collection. Similarly to logical and semi-logical events, we remove all inferences of events that annotators comment as "unclear," "doesn't make sense" or "grammatically wrong."
A.2 Training Details of COMET Models
Input A knowledge tuple  $\{h,r,t\}$  is represented as a concatenated sequence with tokens of each element in the tuple:  $X = \{X^{h},X^{r},X^{t}\}$  where  $X^{h} = \{x_{0}^{h},\dots,x_{|h|}^{h}\}$  are the tokens comprising the event,  $X^{r} = \{x_{0}^{r},\dots,x_{|r|}^{r}\}$  as tokens comprising the relation, and  $X^{t} = \{x_{0}^{t},\dots,x_{|t|}^{t}\}$  are the tokens comprising the commonsense inference.
Initialization Similar to Bosselut et al. (2020), we initialize the trained parameters of COMET to the 345M parameter GPT2 model (GPT2-M) from Radford et al. (2019). Special tokens that
4https://www.nltk.org
<sup>5</sup>https://dictionary.cambridge.org/us/
grammar/british-grammar/negation_2
$^{6}$ https://homes.cs.washington.edu/
~msap/atomic/mTurkFiles/
represent relation types (e.g., xIntent) are added to the vocabulary and initialized via sampling from the normal distribution.
Hyperparameters Following Bosselut et al. (2019), we use a dropout rate of 0.1 and GeLU (Hendrycks and Gimpel, 2020) units as activation functions. During training, we use the Adam optimizer (Kingma and Ba, 2017) with a batch size of 64. For COMET models trained on different subsets of the ATOMIC and ANION datasets, we adopt a maximum learning rate of 6.25e-5 with a warmup period of 0.002 times of the total number of minibatches customized for each model, which decays linearly until finishing training.
We train different COMET models for different subsets of the full on original data (ATOMIC), original and logical negation data (ATOMIC + ANION-L), original and semi-logical negation data (ATOMIC + ANION-S), original and commonsense contradiction data (ATOMIC + ANION-C), and the overall dataset (ATOMIC + ANION), for 21k, 25K, 24K, 24K and 29K minibatches respectively, and apply early stopping for all models. The rest of the hyperparameters are the same as those of GPT2-M in Radford et al. (2019) implemented via the publicly available HuggingFace API<sup>7</sup>.
All models are fine-tuned and evaluated on a single NVIDIA QUADRO RTX 8000 GPU for six to twelve hours depending on the complexity of the experimental setup.
A.3 Training Details of Negation Discriminator
Input As input to the discriminator model, we design sentence patterns that express relation types in natural language and fill out the patterned sentences with events and conditions before encoding them (e.g., "PersonX addresses a talk. As a result, PersonX wants to convince others"). Relations and their corresponding patterned sentences are listed in Table 12. Adopting patterned sentences is found to be a more effective approach than concatenating components in knowledge tuples from the pilot study.
Loss Function The negation discriminator is trained to minimize the binary cross-entropy loss:
$$
\mathcal {L} _ {D} = y \cdot \log P (y) + (1 - y) \cdot \log (1 - P (y)) \tag {2}
$$
Relation	Patterned sentences
xIntent	{h}. Because PersonX wanted {t}.
xNeed	{h}. Before, PersonX needed {t}.
xAttr	{h}. PersonX is seen as {t}.
xWant	{h}. As a result, PersonX wants {t}.
oWant	{h}. As a result, others want {t}.
xEffect	{h}. As a result, PersonX then {t}.
oEffect	{h}. As a result, others then {t}.
xReact	{h}. As a result, PersonX feels {t}.
oReact	{h}. As a result, others feel {t}.
Table 12: Patterned sentences representing relation types in ATOMIC, used to construct inputs for training negation discriminators.
where  $y$  is the label for an input (i.e., logically valid or invalid).
Hyperparameters Parameters are initialized with the trained weights of the RoBERTa-base model in Liu et al. (2019). During training, we use the Adam optimizer (Kingma and Ba, 2017) and train the model with a batch size of 64. We adopt a maximum learning rate of 4.5e-5 with a warmup period of 10 minibatches. We trained L, S, C, LSC discriminators, for 25K, 14K, 21K and 6K minibatches respectively, and apply early stopping for all models. We use a probability threshold of 0.7 to determine whether an input knowledge tuples to the discriminator is plausible based on pilot study on the development sets. The rest of the hyperparameters are the same as those of RoBERTa-base (Liu et al., 2019) implemented via the publicly available HuggingFace API<sup>8</sup>.
All models are fine-tuned and evaluated on a single NVIDIA QUADRO RTX 8000 GPU for four to six hours depending on the different experimental setups.
A.4 Statistical Significance Testing
To compare  $\mathrm{P}@\{\# \text{valid}\}$  for the all and valid sets, we use a Permutation Test with 1,000 permutations to test for statistical significance. For multiple comparisons, we use the Bonferroni method (Haynes, 2013) to correct significance thresholds.
A.5 Quality Check for the Human Evaluation
We conduct comprehensive pre- and post-evaluation screening on the users and the tasks being completed to ensure the objectivity and high quality of the evaluations. Besides qualifying users
<sup>8</sup>https://huggingface.co/transformers/
<sup>9</sup>http://rasbt.github.io/mlxtend/
<sup>7</sup>https://huggingface.co/transformers/
Eval	ATOMIC	ANION-L	ANION-S	ANION-C
Trn	Dis	all	valid	ipv%	all	valid	ipv%	all	valid	ipv%	all	valid	ipv%
ATOMIC	L	55.69	55.65	-0.07	39.46	**46.39	17.55	37.13	37.48	0.96	46.92	46.83	-0.20
S	55.93	56.18	0.44	37.85	**41.93	10.78	39.29	**44.58	13.47	47.32	47.68	0.75
C	56.94	57.26	0.57	36.43	37.54	3.03	37.72	39.03	3.45	48.26	48.79	1.09
LSC	58.30	59.07	1.32	39.45	**45.59	15.57	38.55	**44.93	16.56	48.81	*51.44	5.40
ATOMIC+ ANION-L	L	58.62	58.72	0.16	46.05	**51.19	11.16	42.42	42.89	1.11	47.98	47.97	-0.02
S	58.93	59.31	0.64	45.90	**49.00	6.77	44.22	**47.59	7.64	48.10	48.69	1.24
C	59.63	60.07	0.73	45.88	46.23	0.76	43.40	43.74	0.79	48.81	49.84	2.12
LSC	60.83	62.49	2.74	45.96	**50.19	9.20	44.61	**48.30	8.27	49.73	*51.97	4.51
ATOMIC+ ANION-S	L	56.37	56.35	-0.05	44.77	**51.76	15.60	45.58	45.87	0.63	46.24	46.29	0.11
S	56.60	56.66	0.11	44.39	**47.42	6.83	46.07	**48.32	4.89	46.62	47.17	1.19
C	57.46	57.60	0.23	44.46	45.39	2.07	45.81	47.15	2.93	47.38	48.83	3.06
LSC	58.74	*60.39	2.81	44.94	**49.88	10.98	46.08	**48.67	5.62	48.56	**51.22	5.46
ATOMIC+ ANION-C	L	52.72	52.73	0.02	43.45	**49.62	14.20	41.83	41.88	0.12	48.93	48.97	0.07
S	52.93	53.33	0.76	42.66	**46.40	8.75	42.57	**46.40	8.98	49.18	49.49	0.62
C	53.70	54.07	0.69	42.83	43.26	1.00	42.25	42.70	1.07	49.30	*50.97	3.38
LSC	54.38	55.74	2.49	44.17	**48.84	10.58	42.37	**46.22	9.10	50.07	**52.80	5.46
ATOMIC+ ANION	L	54.16	54.20	0.08	46.54	**50.71	8.98	46.90	47.14	0.51	50.80	50.94	0.28
S	54.49	54.64	0.28	46.26	**48.36	4.54	47.73	**50.42	5.65	51.29	51.52	0.45
C	55.03	55.71	1.23	46.15	46.16	0.03	47.47	48.20	1.55	51.28	52.65	2.67
LSC	55.68	**57.58	3.41	46.39	**49.85	7.45	47.53	**50.62	6.50	51.83	*53.91	4.02
Table 13: For generations of COMET models trained on different subsets of ATOMIC and ANION, the Precision @ {# valid} scores of the all and valid sets determined by L, S, C and LSC discriminators with respect to the original and negation evaluation sets. The single (*) and double asterisks  $(^{**})$  indicate significance at  $p < 0.05$  and  $p < 0.01$  respectively. iprv% is the percentage improvement of the valid set over the all set.
Eval	ATOMIC	ANION-L	ANION-S	ANION-C
Trn	Dis	all	valid	iprv%	all	valid	iprv%	all	valid	iprv%	all	valid	iprv%
ATOMIC	L	59.41	59.65	0.40	44.92	**49.95	11.20	39.47	39.94	1.21	50.77	50.91	0.27
S	59.48	60.14	1.12	42.88	**46.24	7.83	45.27	**49.25	8.81	51.22	51.84	1.21
C	59.89	60.89	1.66	39.20	40.28	2.75	40.07	41.40	3.32	51.77	52.88	2.15
LSC	61.37	63.12	2.85	46.00	**50.34	9.44	46.15	**50.23	8.85	53.29	55.24	3.65
ATOMIC+ ANION-L	L	61.33	61.57	0.39	51.47	**56.16	9.11	45.73	46.04	0.68	51.40	51.57	0.33
S	61.13	62.05	1.51	50.12	**53.40	6.54	50.84	**54.09	6.40	51.89	52.91	1.96
C	61.48	62.96	2.40	48.23	49.06	1.72	46.42	46.99	1.22	52.31	53.67	2.61
LSC	63.66	*65.85	3.44	51.90	**56.26	8.40	51.12	**54.59	6.78	53.97	56.15	4.04
ATOMIC+ ANION-S	L	60.25	60.65	0.67	48.11	**54.45	13.18	45.97	46.35	0.81	50.82	50.89	0.15
S	60.23	60.85	1.03	46.48	**49.14	5.72	47.58	**50.29	5.70	51.11	52.00	1.74
C	60.43	61.28	1.40	44.61	46.31	3.80	46.21	**48.78	5.58	51.72	53.25	2.95
LSC	62.22	*64.44	3.58	47.63	**51.12	7.32	48.24	*50.70	5.11	53.51	*56.04	4.74
ATOMIC+ ANION-C	L	54.36	54.80	0.81	46.25	**51.57	11.51	42.81	43.13	0.76	50.71	50.81	0.20
S	54.50	55.75	2.29	45.59	*48.11	5.53	45.40	**48.50	6.83	51.00	51.78	1.52
C	54.43	55.50	1.97	42.61	43.26	1.53	43.11	44.13	2.38	51.44	*53.46	3.93
LSC	55.68	*57.91	4.01	47.11	**51.25	8.80	45.75	**49.03	7.18	52.44	**55.68	6.18
ATOMIC+ ANION	L	56.63	57.11	0.85	50.39	**54.52	8.20	47.92	48.27	0.73	53.11	53.41	0.56
S	56.53	57.42	1.57	48.92	**52.07	6.44	49.21	**52.51	6.72	53.10	53.67	1.09
C	56.40	57.64	2.21	47.96	48.30	0.70	48.16	50.00	3.82	53.90	55.48	2.94
LSC	58.27	60.53	3.87	50.25	**54.27	8.02	49.85	**53.09	6.50	54.50	**57.66	5.79
Table 14: For generations of COMET models trained on different subsets of ATOMIC and ANION, the Precision @ 3 scores of the all and valid sets determined by L, S, C and LSC discriminators with respect to the original and negation evaluation sets. The single (*) and double asterisks  $(^{**})$  indicate significance at  $p < 0.05$  and  $p < 0.01$  respectively. iprv% is the percentage improvement of the valid set over the all set.
Event + Rel	Generation	V	P
X does not skate around xAttr	athletic careless lazy uncoordinated unskilled	× × ✓	× ×
to be alone to be left alone to avoid Y to sit to wait	✓ ✓ ✓	✓ ✓ ✓
to calm down to watch a movie to have been provoked to not be angry to be calm	× × ✓	× × ✓
X does not look angry xNeed	to save money to get a job to pay rent to move in to get a new apartment	✓ ✓ × × ×	✓ × × ×
to hire X to fire X to hire someone else to accept the job to hire them	× × × ×	✓ × × ×
X is not offered the job oWant	satisfied hungry satiated full guilty	× × ✓ × × ×	× × × ×
relieved sad satisfied grateful angry	✓ × × × ×	✓ × × × ×
X does not buy a snack xReact	gains knowledge becomes lazy gets bored becomes ignorant cries	× × ✓ × × ×	× × × ×
xEffect	✓ × × ×	✓ × × ×
X does not learn new things	becomes confused does not think about X Y thinks about X Y is not remembered	× × ✓ × × ×	× × × ×
cries	✓ × × ×	✓ × × ×
during pilot batches, we double check to remove evaluation tasks that are not carefully conducted (e.g., tasks done by users that select all/no options for all hundreds of tasks that they perform). Figure 3 shows a snippet of the human evaluation MTurk
Table 15: Randomly selected generations of the original COMET model regarding logical negation events in ANION-L. The top 5 options are classified as either valid or invalid by the LSC discriminator.  $\mathbf{V}$  indicates whether an option is classified as valid by the LSC discriminator.  $\mathbf{P}$  indicates whether an option is plausible judging by humans.
Event + Rel	Generation	V	P
X hardly ever increases X's knowledge xAttr	intelligent determined studious lazy dedicated	✘ ✘ ✘ ✘	✘ ✘ ✘ ✘
X avoids skating around xIntent	to have fun to be safe to stay home to stay in shape to get fit	✘ ✘ ✓✓ ✘	✘ ✘ ✓✓ ✘
X not at all wants to learn karate xNeed	learn karate to not like it to avoid it to be lazy to find a teacher	✘ ✘ ✓✓ ✘	✘ ✘ ✓✓ ✘
X refuses to hear a scary noise xWant	to run away to go to sleep to be safe to keep quiet to avoid the noise	✘ ✘ ✓✓ ✘	✘ ✘ ✓✓ ✘
X never brings Y into conflicts oWant	to avoid X to be left alone to thank X to fight back to avoid conflict	✘ ✘ ✓✓ ✘	✘ ✘ ✓✓ ✘
X scarcely gets sunburned xReact	burned hurt sick sad satisfied	✘ ✘ ✘ ✘	✘ ✘ ✘
X under no circumstances forgets Y's wallet oReact	upset sad angry thankful grateful	✘ ✘ ✘ ✘	✘ ✘ ✘
X has trouble with advertising X's business xEffect	loses money loses clients gets fired gets sued cries	✓ ✓ ✓✓ ✘	✓ ✓ ✓✓ ✘
X fails to make it through the day oEffect	loses a friend worries about X worried want them to do better cries	✓ ✓ ✓✓ ✘	✓ ✓ ✓✓ ✘
Table 16: Randomly selected generations of the original COMET model regarding semi-logical negation events from ANION-S. The top 5 options are classified as either valid or invalid by the LSC discriminator. V indicates whether an option is classified as valid by the LSC discriminator. P indicates whether an option is plausible judging by humans.
Event + Rel	Generation	V	P
X keeps the old one	determined	×	×
careful	×	×
loyal	✓	✓
tired	×	×
xAttr	caring	×	×
X pays twice the retail price for a rare item xIntent	to own something rare	✓	✓
to buy something rare	✓	✓
to purchase something rare	✓	✓
to obtain something	×	×
to acquire something	×	×
X is a complete loner xNeed	to avoid people	✓	✓
to be lonely	✓	✓
to start a relationship	×	×
to stay quiet	✓	✓
to get into trouble	×	×
X is miserable playing xWant	to get better	✓	✓
to take a break	×	✓
to go home	×	×
to go to sleep	×	×
to cry	✓	✓
X is picky about everything oWant	to ignore X	✓	✓
to avoid X	✓	✓
to talk to X	×	×
to help X	×	×
to make X feel better	✓	×
X resigns himself xReact	relieved	✓	✓
relaxed	✓	×
satisfied	×	×
accomplished	×	×
sad	✓	✓
X gives away X's laptop	grateful	✓	✓
thankful	✓	✓
upset	×	×
oReact	sad	×	×
surprised	✓	✓
X goes home	relaxes	✓	✓
goes to sleep	✓	✓
is greeted by family	×	✓
xEffect	gets rest	✓	✓
gets tired	×	×
X puts Y out of mind	has a better day	×	×
becomes sad	✓	✓
cries	✓	✓
oEffect	becomes grateful towards X	×	×
feels better	×	×
Table 17: Randomly selected generations of the original COMET model regarding commonsense contradiction events from ANION-C. The top 5 options are classified as either valid or invalid by the LSC discriminator.  $\mathbf{V}$  indicates whether an option is classified as valid by the LSC discriminator.  $\mathbf{P}$  indicates whether an option is plausible judging by humans.
Instructions (click to collapse/expand)
a. Step 1: read a short event sentence.
○ Note that the names of specific people have been replaced by generic words (e.g., "PersonX", "PersonY").
b. Step 2: given this event, you are asked to formulate TWO (up to FOUR) corresponding OPPOSITE events.
○ There might be multiple grammatically correct ways of expressing your interpretation of the OPPOSITE events. Please express them in natural language (i.e., in a way that you normally talk), and make sure your OPPOSITE events are in complete sentences.
○ Please don't trivially negate. Try not to use negative words directly, including but not limited to: no, not, nothing, no one, none, nobody, nowhere, neither, nor, never, lack of, unless you feel necessary.
○ Changing one of the characters (e.g., from "PersonX" to "PersonY"), or one of the objects (e.g., from "mother" to "father" or from "cat" to "dog") are not the goals of the OPPOSITE event, unless you think they are appropriate.
○ Please do not add unnecessary/unrelated additional details to the OPPOSITE event.
Examples (click to collapse/expand)
Event	${Event}
Given this event, can you formulate corresponding OPPOSITE events? Make sure they are in complete sentences.
Opposite Event 1 (REQUIRED)
Opposite Event 2 (REQUIRED)
Opposite Event 3 (OPTIONAL)
Opposite Event 4 (OPTIONAL)
Figure 2: Snippets of the annotation task used to collect commonsense contradiction events.
${title}
Full Instructions (Expand/Collapse)
You will read a sentence fragment depicting an event, and be asked to  $\S$  (task).
Events are short phrases possibly involving participants. The names of specific people have been replaced by generic words (e.g. PersonX, PersonY, PersonZ). PersonX is always the subject of the event.
{$instruction}
Notes on the events: some of the events may be figurative, and should not be taken literally
(e.g., "PersonX kills two birds with one stone" does *not* make PersonX "murderous")
Examples (Expand/Collapse)
{$examples}
Event ${event}
${question}
\$note
Figure 3: Snippet of the human evaluation task used to evaluate model generated tail inferences.
Optional Feedback: Thanks for filling out the questions above! If something about the hit was unclear, please leave a comment in the box below. We would like to make this HIT easier for future workers, so we really appreciate feedback though it is optional.