SlowGuess/ACL_2020_OCR huggingface dataset:
Example (5) illustrates a rare phenomenon: an explicit temporal interval (i.e., two months) indicating the possession duration. Thus, annotators chose medium duration. Regarding co-possession, the company loaning the car was clearly a copossessor of the loaner car while the author of the blog borrowed the car, so annotators chose yes.
Finally, Example (6) exemplifies a long pos
8337
Table 4: Annotation examples on selected possessions identified in tweets consisting of text and an image. The possessors are the authors of the tweets, and the possessee are concrete objects in their tweets.
session with co-possession. The context is a law enforcement operation in which They (the police) kept the possessee (car). The duration of the possession is explicit (a year), and during that time my father was still the owner. Thus, annotators chose long and yes for duration and co-possession.
Table 4 presents annotation examples using possession relations identified in tweets consisting of text and images. We do not describe these examples in detail as they are self-explanatory.
In order to predict possession duration and co-possession, we experiment with Logistic Regression and a neural network ensemble including a text component and two image components. Each possession relation becomes an instance, and we create stratified training (80%) and test (20%) sets. We also reserve 20% of the training as validation set. More specifically, we build two classifiers: one for possession duration (short, medium, or long) and one for co-possession (yes or no).
Logistic Regression. We use the implementation by scikit-learn (Pedregosa et al., 2011), and use
bag-of-words features for the sentence at hand. Specifically, we use binary flags indicating word presence, and additional flags to indicate the word corresponding to the possessor and possessee.
Neural Network. The network architecture is similar to the one in our previous work (Chinnappa et al., 2019). It includes a text component and an image component (Table 4). The latter component is disabled if no image is available.
The text component is an LSTM that takes as input the sentence (or tweet) containing the possessee. Words are represented with the concatenation of their 300-dimensional GloVe embedding (Pennington et al., 2014) and an additional embedding indicating whether a token is the possessor, possesee, or neither. We train the additional embeddings from scratch with the rest of the network.
The image component uses two pretrained neural networks. First, we concatenate to the softmax output layer the weights from the average pooling layer (second to last layer) of InceptionNet (Szegedy et al., 2015). Second, we obtain the top 5 tags from the Google Cloud Vision API and incorporate them as an additional textual input.
8338
Figure 4: Neural network architecture to predict possession duration and co-possession. We include a text component (above dotted line) and two image components (below dotted line). Note that the top 5 tags from the Vision API become a textual input, and we use pretrained word embeddings and an LSTM for them.
Majority Baseline	Log. Regression	LSTMword embeddings	LSTM+addtl. embeds.
P	R	F	P	R	F	P	R	F	P	R	F
short	0.00	0.00	0.00	1.00	0.35	0.52	0.88	0.35	0.50	0.75	0.60	0.67
medium	0.00	0.00	0.00	0.68	0.35	0.46	0.64	0.21	0.32	0.68	0.49	0.57
long	0.73	1.00	0.84	0.81	0.97	0.88	0.78	0.96	0.86	0.94	0.90	0.82
W. Avg.	0.53	0.73	0.61	0.80	0.80	0.77	0.76	0.73	0.77	0.82	0.83	0.82
yes	0.00	0.00	0.00	0.62	0.58	0.60	0.75	0.58	0.65	0.72	0.62	0.67
no	0.56	1.00	0.72	0.69	0.72	0.70	0.72	0.85	0.78	0.73	0.82	0.77
W. Avg	0.31	0.56	0.40	0.66	0.66	0.66	0.73	0.73	0.72	0.73	0.73	0.73
Table 5: Results obtained with possession relations identified from text (OntoNotes and weblogs). Addtl. embeddings refers to the embeddings indicating whether a token is the possessor, the possesee, or neither one.
More specifically, we use GloVe embeddings and an LSTM to process the additional textual input. Note that individual tags identified in the image are sometimes multiple tokens (e.g., coffee mug), so an LSTM is a good choice.
We use the implementation by Keras (Chollet et al., 2015) with TensorFlow backend (Abadi et al., 2015). More specifically, we use the Adam optimizer (Kingma and Ba, 2014) and categorical cross entropy as a loss function. We use batch size 32 for up to 200 epochs, but stop earlier if there is no improvements in the validation for 5 epochs.
Table 5 presents the results with instances including only text. Regarding possession duration, the majority baseline (always long) obtains 0.61 F-measure. The second baseline, Logistic Regression, obtains 0.77 F-measure. These results are strong, however, Logistic Regression is biased towards the most common label (long, Table 1), and performs poorly with the other labels (short and medium). In fact, Logistic Regression outperforms LSTM+addtlembeds. with long, but the weighted F-measure is lower (0.77 vs. 0.82). Re
garding co-possession, we observe a similar trend, but the LSTM performs similar with both labels.
LSTM and Additional Embeddings. Table 5 presents results obtained with the LSTM using (a) only the word embeddings and (b) incorporating the additional embeddings for the possessor and possesee. The LSTM with only word embeddings obtains worse results predicting possession durations (0.77 vs. 0.82 weighted F-measure), and virtually the same results predicting co-possessions (0.72 vs. 0.73 weighted F-measure). These results lead to the conclusion that the specific possessor and possesee along with context are important to determine how long a possession holds true. On the other hand, determining whether there are concurrent co-possessors does not benefit from the specific possessor and possesee (i.e., events and other information contained in the sentence are sufficient).
Table 6 presents results with the tweets (all of them include both text and images). The results indicate that the text is vital to determine possession duration and co-possession, and that the image components do not bring any improvements. Logistic Regression obtains best results for both pos
8339
Majority Baseline	Log. Regression	LSTM+addtl. embeds	only image comp.	text + image
P	R	F	P	R	F	P	R	F	P	R	F	P	R	F
short medium long W.Avg.	.00	.00	.00	.00	.00	.00	.50	.11	.18	.00	.00	.00	.12	.10	.11
.00	.00	.00	.61	.57	.59	.58	.45	.50	.39	.29	.33	.52	.30	.38
.58	1.00	.73	.70	.78	.74	.66	.80	.73	.57	.71	.63	.64	.83	.72
.33	.58	.42	.64	.67	.65	.62	.64	.62	.48	.52	.49	.57	.59	.56
yes no W.Avg.	.00	.00	.00	.41	.29	.34	.37	.32	.34	.30	.14	.19	.33	.15	.21
.73	1.00	.84	.76	.85	.80	.76	.79	.78	.73	.88	.80	.74	.89	.81
.53	.73	.62	.67	.70	.68	.65	.67	.66	.61	.68	.63	.63	.69	.64
Table 6: Results obtained with possession relations identified from text and image (tweets).
session duration and co-possession, and obtains similar results than the text component of the neural network (LSTM $_{+\text{addtl\_embeddings}}$ ): 0.65 vs. 0.62 F-measure (duration) and 0.68 vs 0.66 F-measure (co-possession). While including the image component slightly decreases the results predicting co-possession (0.66 vs. 0.64 F-measure), it heavily decreases results predicting possession duration (0.62 vs. 0.56 F-measure). We attribute these unexpected results to the nature of the tasks. Image tags provide high-level information about the possessee (e.g., cup), and determining possession durations and co-possession requires fine-grained information about the possessee (e.g., reusable, disposable) as well as knowledge about the events that connect the possessor and possessee.
Standard relation extraction does not provide information about for how long relations hold true or whether relations are one-to-one or one-to-many. In this paper, we tackle both problems and determine possession durations and co-possessions. Possessions are ubiquitous yet understudied from a computational perspective. From a theoretical perspective, they include having control over something (e.g. flying a plane, impounding a vehicle, eating ice cream) thus most objects are actually possesees of one or more possessors. Additionally, as just exemplified, many possessions can be extracted even if prototypical possession verbs (e.g., have, buy, acquire) are missing.
We have presented new annotations on top of existing corpora. Regarding durations, we collect lower and upper bounds in order to derive sound duration intervals. The resulting three intervals obtain substantial agreement (0.63 Cohen's  $\kappa$ ). Regarding co-possessions, we obtain slightly better agreement (0.65 Cohen's  $\kappa$ ). We have also presented baseline models and a neural network
architecture to solve both tasks. Beyond word embeddings, the LSTM benefits from additional embeddings indicating the tokens that are the possessor and possessee. Information extracted from the image, however, is not helpful.
While the work presented here targets possession relations, we believe that a similar approach could be used to to determine for how long any semantic relation holds true.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaojiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org.
Omri Abend and Ari Rappoport. 2017. The state of the art in semantic representation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 77-89, Vancouver, Canada. Association for Computational Linguistics.
James F Allen and George Ferguson. 1994. Actions and events in interval temporal logic. Journal of logic and computation, 4(5):531-579.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages
8340
344-354, Beijing, China. Association for Computational Linguistics.
Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Comput. Linguist., 34(4):555-596.
Adriana Badulescu and Dan Moldovan. 2009. A semantic scattering model for the automatic interpretation of english genitives. NLE.
Carmen Banea and Rada Mihalcea. 2018. Possession identification in text. Natural Language Engineering, 24(4):589610.
Austin Blodgett and Nathan Schneider. 2018. Semantic Supersenses for English Possessives. In LREC.
Dhivya Chinnappa and Eduardo Blanco. 2018. Mining possessions: Existence, type and temporal anchors. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 496-505, New Orleans, Louisiana, USA. Association for Computational Linguistics.
Dhivya Chinnappa, Srikala Murugan, and Eduardo Blanco. 2019. Extracting possessions from social media: Images complement language. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 663-672, Hong Kong, China. Association for Computational Linguistics.
François Chollet et al. 2015. Keras. https://github.com/fchollel/keras.
Heng Ji, Ralph Grishman, and Hoa Dang. 2011. Overview of the TAC2011 knowledge base population track. In TAC 2011 Proceedings Papers.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Parisa Kordjamshidi, Archna Bhatia, James Puste-jovsky, and Marie-Francine Moens. 2018. Proceedings of the first international workshop on spatial language understanding. New Orleans. Association for Computational Linguistics.
David McClosky and Christopher D. Manning. 2012. Learning constraints for consistent timeline extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 873-882, Jeju Island, Korea. Association for Computational Linguistics.
Bill McDowell, Nathanael Chambers, Alexander Ororbia II, and David Reitter. 2017. Event ordering with a generalized model for sieve prediction ranking. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1:
Long Papers), pages 843-853, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Preslav I. Nakov and Marti A. Hearst. 2013. Semantic interpretation of noun compounds using verbal and other paraphrases. ACM Trans. Speech Lang. Process., 10(3):13:1-13:51.
Feng Pan, Ritu Mulkar-Mehta, and Jerry R. Hobbs. 2011. Annotating and learning event durations in text. Computational Linguistics, 37(4):727-752.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9.
Shiao Wei Tham. 2004. Representing Possessive Predication: Semantic Dimensions and Pragmatic Bases. Ph.D. thesis, Stanford University.
Stephen Tratz and Eduard Hovy. 2010. A taxonomy, dataset, and classifier for automatic noun compound interpretation. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL '10, pages 678-687, Stroudsburg, PA, USA. Association for Computational Linguistics.
Stephen Tratz and Eduard H. Hovy. 2013. Automatic interpretation of the english possessive. In ACL (1), pages 372-381. The Association for Computer Linguistics.
8341