SlowGuess/NAACL_2022_OCR huggingface dataset:
Malay Pramanick, Ashim Gupta, and Pabitra Mitra. 2018. An LSTM-CRF based approach to token-level metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, pages 67-75, New Orleans, Louisiana. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI.
Ekaterina Shutova. 2010. Automatic metaphor interpretation as a paraphrasing task. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 1029-1037, Los Angeles, California. Association for Computational Linguistics.
Ekaterina Shutova. 2011. Computational approaches to figurative language.
Gerard J Steen, Alleta G Dorst, Berenike Herrmann, Anna A Kaal, Tina Krennmayr, and Trynetje Pasma. 2010. A Method for Linguistic Metaphor Identification: From MIP to MIPVU. John Benjamins.
Kevin Stowe and Martha Palmer. 2018. Leveraging syntactic constructions for metaphor identification. In Proceedings of the Workshop on Figurative Language Processing, pages 17-26, New Orleans, Louisiana. Association for Computational Linguistics.
Chang Su, Shuman Huang, and Yijiang Chen. 2017. Automatic detection and interpretation of nominal metaphor based on the theory of meaning. Neurocomputing, 219:300-311.
John Sweller. 2006. Discussion of 'emerging topics in cognitive load research: Using learner and information characteristics in the design of powerful learning environments'. Applied Cognitive Psychology - APPL COGNITIVE PSYCHOL, 20:353-357.
Xiaoyu Tong, Ekaterina Shutova, and Martha Lewis. 2021. Recent advances in neural metaphor processing: A linguistic, cognitive and social perspective. In NAACL 2021.
Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 248–258, Baltimore, Maryland. Association for Computational Linguistics.
P. Wolff and Dedre Gentner. 2000. Evidence for role-neutral initial processing of metaphors. Journal of experimental psychology. Learning, memory, and cognition, 26 2:529-41.
A Crowdsourcing Details
We crowdsourced metaphorical expressions and their interpretations through Amazon Mechanical Turk. Workers were recruited from the United States and were limited to those who had a  $>98\%$  approval rating on the platform, and who had also completed more than 1000 Human Intelligence Tasks (HITs). Data collection was split into two stages: in the first stage, 1458 train examples, and all the dev and test examples were collected. In the second stage, the remaining 6558 training examples were collected. We identified some workers who created especially good examples in the first stage, and recruited them back for more examples in the second stage. Workers were paid $0.33 for each pair of sentences and were asked to generate 3 pairs at a time. An author of this paper wrote an initial pilot set of sentences, and timed themselves while writing some sentences. They found that each pair took around 1 minute to write, though this varied (less creative examples took less time, while more creative examples took more time). This extrapolates to an hourly rate of 19.80 USD, which is above the minimum wage in all US states, where workers were located.
Our HIT task was structured as follows: At the top of the page, the workers are shown the following instructions: "Your task is to generate three pairs of sentences with opposite or very different meanings, both of which contain rare/creative metaphors, which means metaphors that would not appear often in text on the internet, books, social media or news sites, but that can still be easily understood by people. For each metaphor, you should also provide a literal (non-metaphorical) sentence with the same meaning." Then, we display one example of a valid sentence pair. There is a button that opens a modal with more detailed instructions and some more valid/invalid examples for reference. Below that, we display three random words, which workers are encouraged to use in their sentences if they get stuck. Finally, we display three sets of 5 text fields for workers to fill in: one for the start phrase, two for each metaphorical phrase, and two for each literal interpretation. As the user types in each start phrase, we prepend a copy of their phrase before the corresponding metaphor fields in the UI using some embedded JavaScript, which we found helped reduce confusion and resulted in less improperly formatted responses.
We launched many batches of these HITs until
we had collected the desired quantity of data. Then, we converted the form responses into sentence pairs and validated each pair by hand before adding it to our dataset.
B Invalid Examples
Figurative language examples collected from crowdworkers were excluded if they (a) did not make sense given the meaning and the metaphorical expression, (b) had grammar or spelling errors that rendered them unintelligible, or (c) did not follow the format specified by the task template.
Examples are given below:
1. Do not make sense given the meaning and the metaphorical expression
Paired sentences	Possible answers
He was resourceful like toilet paper
He was resourceful like a mess	He was very resourceful.
He wasn’t resourceful at all
The night was as long as a spool of thread
The night was as long as a winding road	The night is long
The night dragged on
the concert of the lesson is a main and a major
the concert of the lesson features	we concert everyone
we concert our loved one
Table 7: Examples that were rejected due to being nonsensical.
2. Significant grammar or spelling errors
Paired sentences	Possible answers
fallten data are very much trusted	fallten are nice
fallten data are very valuable	flatten are safe
CAR IS BIRD FEATHEAR	CAR SITE IS ROUGH
CAR IS COTTON	CAR SITE IS HARD
Inflation is as natural as Minnesota rainfall in June	Inflation is perfectly natural
Inflation is as natural as Minnesota snowfall in June	Patient is in a natural result of other things
Table 8: Examples that were rejected due to having significant spelling or grammar errors.
3. Do not follow format
Paired sentences	Possible answers
This attack is as weak as a feather	The attack is useless
This attack is as weak as a breeze	The attack doesn't work
My car motor is dusty like old cave	Car motor is very rusty
My car motor is dusty like abandon building	car motor is very dusty
the writer is stuck between a rock And another hard place	He is just stuck doesn't have a choice
the writer is stuck between a rock And a pebble	The writer can get over the pebble
Table 9: Examples that were rejected due to not following the specified format.
Efforts were made to ensure that the final dataset contains no offensive content or personally identifiable information. WorkerID and other potentially personally identifying information were not included.
C Backward accuracies
Model	Zero-shot	Fine-tuned (L)
GPT-2	52.18	52.00
GPT-neo 1.3B	54.36	63.44
GPT-3 Curie	58.46	74.83
D Paired accuracies
Table 10: Zero-shot and finetuned backward autoregressive model accuracies on the test set
Model	Accuracy (pairs correct)
GPT-2 zero-shot	6.63
GPT-2 finetuned	5.06
GPT-neo zero-shot	10.3
GPT-neo finetuned	10.3
GPT-3 Curie zero-shot	17.4
GPT-3 Curie finetuned	50.0
BERT finetuned	70.6
RoBERTa finetuned	80.4
Human	89.7
E Accuracy breakdown by Part-of-Speech
E.1 Subject
Table 11: Accuracy for models on the test set, counted in terms of pairs of sentences in which both are correct  $(\%)$ . Results are from one run.
Part of speech	Accuracy	Frequency
NN	0.8569	538
PRP	0.8526	156
PRP$ NN	0.9	110
NN NN	0.8889	63
DT NN	0.8182	44
NN NN NN	0.9375	32
JJ NN	0.9167	12
E.2 Relation
Table 12: Accuracy breakdown and frequency of parts of speech in metaphor subjects. Only part-of-speech patterns with greater than 10 occurrences are shown.
Part of speech	Accuracy	Frequency
VBZ NN IN	0.8421	152
VBD RB JJ IN	0.8904	146
VBZ RB JJ IN	0.8889	99
VBZ	0.8352	91
VBD NN IN	0.8806	67
VBD	0.9180	61
VBN IN	0.9545	22
NN IN	0.8636	22
VBD JJ IN	0.9048	21
NNS IN	0.8889	18
VBD IN	0.8462	13
VBZ IN	1.0	13
VBD RB VBN IN	0.8182	11
E.3 Object
Table 13: Accuracy breakdown and frequency of parts of speech in metaphor relations. Only part-of-speech patterns with greater than 10 occurrences are shown.
Part of speech	Accuracy	Frequency
NN	0.8788	429
NN NN	0.8992	129
JJ NN	0.8352	91
NN IN NN	0.8372	43
JJ NN NN	0.8710	31
NN NN NN	0.9130	23
VBG NN	0.9545	22
NN IN JJ NN	0.6154	13
PRP$ NN	1.0	11
JJ	0.6364	11
NN IN NN NN	0.8182	11
Table 14: Accuracy breakdown and frequency of parts of speech in metaphor objects. Only part-of-speech patterns with greater than 10 occurrences are shown.
F Accuracy breakdown by hypernyms
F.1 Subject
Synset	Accuracy	Frequency
adult.n.01	0.8736	182
male.n.02	0.8684	152
woman.n.01	0.7391	46
female.n.02	0.9130	46
show.n.03	0.875	24
product.n.02	0.8636	22
motor_vehicle.n.01	0.9048	21
activity.n.01	0.8421	19
emotion.n.01	0.6667	18
publication.n.01	0.8333	18
feline.n.01	0.9375	16
being.n.01	0.7143	14
performer.n.01	0.8333	12
canine.n.02	12
body_covering.n.01	0.8333	12
vessel.n.03	0.8333	12
sound.n.01	1.0	12
domesticAnimal.n.01	0.9167	12
person.n.01	0.8	10
scheme.n.01	0.9	10
contestant.n.01	1.0	10
F.2 Object
Table 15: Accuracy breakdown and frequency of Word-Net hypernyms in metaphor subjects. Only hypernyms with 10 or greater occurrences are shown.
Synset	Accuracy	Frequency
time_period.n.01	0.85	20
natural_object.n.01	0.8947	19
person.n.01	0.8824	17
large_integer.n.01	0.9286	14
adult.n.01	1.0	14
solid.n.01	0.9167	13
male.n.02	1.0	13
child.n.02	0.8333	12
body_of_water.n.01	0.75	12
body_coverage.n.01	0.8333	12
digit.n.01	0.9167	12
region.n.01	0.8182	11
beverage.n.01	0.8182	11
juvenile.n.01	0.8182	11
container.n.01	0.9	10
rodent.n.01	0.9	10
feline.n.01	0.7	10
building.n.01	0.8	10
time_unit.n.01	0.9	10
travel.v.01	0.7	10
Table 16: Accuracy breakdown and frequency of Word-Net hypernyms in metaphor objects. Only hypernyms with 10 or greater occurrences are shown.
G Generation examples
Generation examples can be found in Table 17.
Startphrase	Completion	Label
His temper was a marshmallow. That is to say,	he was very easily angered, and once he was, he was very hard to cool down	Incorrect
He's got the swimming ability of a pack of dolphins. That is to say,	he's got none	Incorrect
The villain is as beautiful as the Joker. That is to say,	he's not	Correct
The child has the energy of a sloth. That is to say,	he is lazy	Correct
The girl moved as fast as a turtle. That is to say,	she moved at a turtle's pace	Literal
The sadness of her death was a flea. That is to say,	It was a flea that was a sadness	Literal
Table 17: Examples of completions generated by GPT-3 Davinci.