Abstract: We conduct two experiments to study the effect of context on metaphor paraphrase aptness judgments. The first is an AMT crowd source task in which speakers rank metaphor paraphrase candidate sentence pairs in short document contexts for paraphrase aptness. In the second we train a composite DNN to predict these human judgments, first in binary classifier mode, and then as gradient ratings. We found that for both mean human judgments and our DNN's predictions, adding document context compresses the aptness scores towards the center of the scale, raising low out of context ratings and decreasing high out of context scores. We offer a provisional explanation for this compression effect.

Introduction: A metaphor is a way of forcing the normal boundaries of a word's meaning in order to better express an experience, a concept or an idea. To a native speaker's ear some metaphors sound more conventional (like the usage of the words ear and sound in this sentence), others more original. This is not the only dimension along which to judge a metaphor. One of the most important qualities of a metaphor is its appropriateness, its aptness: how good is a metaphor for conveying a given experience or concept. While a metaphor's degree of conventionality can be measured through probabilistic methods, like language models, it is harder to represent its aptness. BIBREF0 define aptness as “the extent to which a comparison captures important features of the topic". It is possible to express an opinion about some metaphors' and similes' aptness (at least to a degree) without previously knowing what they are trying to convey, or the context in which they appear. For example, we don't need a particular context or frame of reference to construe the simile She was screaming like a turtle as strange, and less apt for expressing the quality of a scream than She was screaming like a banshee. In this case, the reason why the simile in the second sentence works best is intuitive. A salient characteristic of a banshee is a powerful scream. Turtles are not known for screaming, and so it is harder to define the quality of a scream through such a comparison, except as a form of irony. Other cases are more complicated to decide upon. The simile crying like a fire in the sun (It's All Over Now, Baby Blue, Bob Dylan) is powerfully apt for many readers, but simply odd for others. Fire and sun are not known to cry in any way. But at the same time the simile can capture the association we draw between something strong and intense in other senses - vision, touch, etc. - and a loud cry. Nonetheless, most metaphors and similes need some kind of context, or external reference point to be interpreted. The sentence The old lady had a heart of stone is apt if the old lady is cruel or indifferent, but it is inappropriate as a description of a situation in which the old lady is kind and caring. We assume that, to an average reader's sensibility, the sentence models the situation in a satisfactory way only in the first case. This is the approach to metaphor aptness that we assume in this paper. Following BIBREF3 , we treat a metaphor as apt in relation to a literal expression that it paraphrases. If the metaphor is judged to be a good paraphrase, then it closely expresses the core information of the literal sentence through its metaphorical shift. We refer to the prediction of readers' judgments on the aptness candidates for the literal paraphrase of a metaphor as the metaphor paraphrase aptness task (MPAT). BIBREF3 address the MPAT by using Amazon Mechanical Turk (AMT) to obtain crowd sourced annotations of metaphor-paraphrase candidate pairs. They train a composite Deep Neural Network (DNN) on a portion of their annotated corpus, and test it on the remaining part. Testing involves using the DNN as a binary classifier on paraphrase candidates. They derive predictions of gradient paraphrase aptness for their test set, and assess them by Pearson coefficient correlation to the mean judgments of their crowd sourced annotation of this set. Both training and testing are done independently of any document context for the metaphorical sentence and its literal paraphrase candidates. In this paper we study the role of context on readers' judgments concerning the aptness of metaphor paraphrase candidates. We look at the accuracy of BIBREF3 's DNN when trained and tested on contextually embedded metaphor-paraphrase pairs for the MPAT. In Section SECREF2 we describe an AMT experiment in which annotators judge metaphors and paraphrases embodied in small document contexts, and in Section SECREF3 we discuss the results of this experiment. In Section SECREF4 we describe our MPAT modeling experiment, and in Section SECREF5 we discuss the results of this experiment. Section SECREF6 briefly surveys some related work. In Section SECREF7 we draw conclusions from our study, and we indicate directions for future work in this area.

Question: What document context was added?
Answer: Preceding and following sentence of each metaphor and paraphrase are added as document context