Venue: NAACL 2018

# Visually Guided Spatial Relation Extraction from Text

 Taher Rahgooy\({}^{*}\) Umar Manzoor\({}^{*}\) Parisa Kordjamshidi\({}^{\dagger}\)

\({}^{*}\)Tulane University, Computer Science Department, New Orleans, LA, USA

\({}^{\dagger}\)Florida Institute for Human and Machine Cognition (IHMC), Pensacola, FL, USA

{trahgooy,umanzoor,pkord jam}@tulane.edu

###### Abstract

Extraction of spatial relations from sentences with complex/nesting relationships is very challenging as often needs resolving inherent semantic ambiguities. We seek help from visual modality to fill the information gap in the text modality and resolve spatial semantic ambiguities. We use various recent vision and language datasets and techniques to train inter-modality alignment models, visual relationship classifiers and propose a novel global inference model to integrate these components into our structured output prediction model for spatial role and relation extraction. Our global inference model enables us to utilize the visual and geometric relationships between objects and improves the state-of-art results of spatial information extraction from text.

## 1 Introduction

Significant progress has been made in spatial language understanding by mapping natural language text to spatial ontologies [25]. The research results show that spatial entities can be extracted with a good accuracy, however, spatial relation extraction is still challenging [26, 30]. Particularly, when the sentences convey more than one relationship, finding the right links between the spatial objects and spatial prepositions becomes difficult. For example, the spatial meaning of _There is a car in front of the house on the left_, can be interpreted in different ways: _(A car in front of the house) on the left_ or _A car in front of (the house on the left_). This issue is related to the well-known _prepositional phrase attachments_ (pp-attachments) syntactic ambiguity which is problematic for our goal of spatial semantic extraction too. The previous research shows some of these ambiguities can be resolved by simultaneously reasoning from the associated image [47, 89, 100]. Consider the scene in Figure 1, we can easily resolve the ambiguity and choose the correct interpretation with the help of the associated image.

Although we do not directly tackle the task of pp-attachment here, resolving this issue will help our task to find the accurate link between the spatial prepositions (i.e. _spatial indicators_) and spatial objects/roles (_trajectory_ and _landmark_). The spatial semantic links can go beyond the syntactic links/attachments, therefore merely fixing the preposition attachments is not sufficient for our task. We exploit the image to find the right preposition that describes the relationships between the spatial roles, for example _on the left_ can be a relationship between _the house_ and implicit landmark _picture_ as well as _a car_ and implicit landmark _picture_. There are many recent works on combining vision and language for domains such as image captioning [101], visual image retrieval [111, 102], visual question answering [100, 101], activity recognition [101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 222, 233, 234, 235, 236, 237, 238, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 282, 284, 286, 287, 288, 289, 291, 285, 286, 288, 289, 292, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 323, 335, 336, 337, 338, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 411, 422, 435, 436, 447, 448, 453, 454, 455, 456, 457, 458, 459, 460, 461, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 525, 512, 525, 536, 547, 558, 565, 579, 580, 511, 539, 525, 547, 581, 595, 596, 597, 598, 599, 600, 61, 626, 63, 64, 650, 66, 671, 68, 690, 626, 64, 691, 601, 63, 651, 66, 672, 68, 692, 693, 603, 604, 605, 606, 61, 694, 61, 607, 61, 695, 626, 61, 696, 63, 61, 697, 64, 626, 64, 651, 66, 673, 68, 698, 609, 61, 607, 626, 61, 627, 63, 63, 64, 651, 674, 68, 699, 61, 64, 66, 68, 690, 63, 64, 69, 65, 66, 675, 68, 69, 69, 700, 63, 607, 64, 69, 66, 676, 68, 69, 71, 72, 73, 74, 75, 76, 77, 78

visual models, therefore, such models need to be trained on external datasets and later incorporated in our multi-modal setting, 2) Aligning text entities with image entities is a complex and challenging task itself. Each modality in isolation represents spatial relations imperfectly, however, each one can reflect different types of spatial relation better than the other. If we can handle the mentioned challenges and combine the two modalities then vision modality fills the information gap of the text modality and improves the information extraction.

To overcome the above challenges, we 1) trained two visual models namely word-segment alignment, trained on ImageClef Referring Expression Dataset1 to connect the two modalities, and preposition classifier, trained on Visual Genome dataset Krishna et al. (2017) to help in link disambiguation, and 2) generated a unified graph, based on both image and text data and proposed a global machine learning model to exploit the information from the companion images.

Footnote 1: [http://tamaraberg.com/referitgame/](http://tamaraberg.com/referitgame/)

The contribution of this paper includes a) exploiting the visual information to solve the SpRL task and improving the state-of-the-art results significantly b) forming a global inference model that imposes the consistency constraints on the decisions made based on the two modalities c) exploiting external vision and language datasets to inject external knowledge into our models d) augmenting an existing dataset which is annotated by spatial semantics with the image segment alignments, this dataset will help the evaluation of the existing methods for combining vision and language for fine-grained spatial semantic extractions.

## 2 Model Description

Given a piece of text, \(S\), here a sentence - split into number of phrases, and an accompanying image, \(I\) segmented into number of segments represented by bounding boxes, the goal is to identify the textual phrases that have spatial roles and detect the relationships between them. The spatial roles included in this task are defined as:

* **Spatial indicators (sp):** these are triggers indicating the existence of spatial information in a sentence;
* **Trajectorys (tr):** these are the entities whose location are described;
* **Landmarks (lm):** these are the reference objects for describing the location of the trajectors.

In the textual description of Figure 1, the location of _car (trajectory)_ has been described with respect to _house (landmark)_ using the preposition _in front of_. Furthermore, spatial relationships and their types are defined as follows:

* **Spatial relations:** these indicate a link between the three above mentioned roles (\(sp.tr.lm\)), forming spatial triplets.
* **Coarse-grained relation types:** these indicate the coarse-grained type of relations in terms of spatial calculi formalisms including _region_, _direction_, and _distance_ types.
* **Fine-grained relation types:** these indicate the fine-grained type of relations in terms of each specific spatial calculi formalism. _Region connection calculi (RCC8)_ types (e.g. disconnected (DC), externally connected (EC), etc.), a closed set of directional relations (e.g. left, right), and an open set of distal relations (e.g. close, far) are defined for regional, directional, and distal relationships respectively.

For example, given the sentence and its accompanying image in Figure 1, the goal is to identify the spatial relations, \({\langle}[\text{{A car}}]_{\text{tr}}\), [_in front of_]sp, [_the house_]lm\({\rangle}\) and \({\langle}[\textit{the house}]_{\text{tr}}\), [_on the left_]sp, [_None_]lm\({\rangle}\) and also determine their coarse-grained types (_direction_ and _direction_ respectively) and fine-grained types (_front_ and _left_ respectively).

We formulate this problem as a structured output prediction problem. Given a set of input-out pairs as training examples, \(E=\{(x^{i},y^{j})\in\mathcal{X}\times\mathcal{Y}:i=1..N\}\), a scoring function \(g(x,y;W)=\langle W,\phi(x,y)\rangle\) will be learned. Where \(W\) is the weight vector and \(\langle,\rangle\) is dot product between two vectors. This function is a linear discriminant function defined over combined feature representation of inputs and outputs denoted by \(\phi(x,y)\)

Figure 1: A captioned scene from CLEF IAPR TC-12 dataset: _There is a car in front of the house on the left_.

 

(Tsochantaridis et al., 2005). However, for this work we use a piece-wise training model in which independent models are trained per concepts in the output and the predictions are done based on global inference (Punyakanok et al., 2005).

We construct a graph using the phrases \(\{p_{1},...,p_{n}\}\) and bounding boxes \(\{b_{1}...b_{m}\}\), and link these nodes to make composed concepts (like relations, roles, etc.). We associate a classifier to each concept in the graph and encode the domain knowledge as global constraints over these concepts. We use global reasoning by imposing these constraints over various (node and edge) classifiers to produce the final outputs. The input of our structured output prediction model is the aforementioned graph and the output is the concepts assigned to the nodes and edges of this graph. In the following we describe the information that we use from each modality (i.e. text and image) and from inter-modality relationships and describe the relevant classifiers, features and constraints.

### Text

We use binary classifiers to identify spatial roles and relations. The spatial roles of trajectory, landmark and spatial indicator are denoted by \(tr\), \(lm\), and \(sp\). \(sp.tr.lm\), \(sp.tr.lm.\gamma\), and \(sp.tr.lm.\lambda\) denote spatial relations, coarse-grained relation types, and fine-grained relation types respectively. Additionally, we denote candidate fine-grained types related to coarse-grained type \(\gamma\) by \(\Lambda_{\gamma}\).

**Features:** We use phrase-based features \(\phi_{phrase}(p_{i})\) for role classifiers in which \(p_{i}\) is the identifier of \(i^{\text{th}}\) phrase in the sentence which include several linguistically motivated features such as lexical form of the words in the phrases, lemmas, pos-tags, etc. In addition, motivated by features used in (Roberts and Harabagiu, 2012) and (Kordjamshidi et al., 2017a), we use a combination of phrase-based features like concatenation of headwords of the roles, concatenation of their pos, and other relative features such as distance between roles, dependency relations, sub-categorization, etc., to represent the relations and this is referred to as \(\phi_{triplet}^{text}(p_{i},p_{j},p_{k})\).

**Constraints:** The constraints over spatial concepts expressed in text are as follows,

\begin{tabular}{l l} \(\sum_{i}\sum_{k}sp_{i}tr_{j}lm_{k}=sp_{i}\) & Each \(pr\) candidate at \\  & local should appear in \\  & one relation \\ \end{tabular}

\begin{tabular}{l l} \(\sum_{j}tr_{j}\geq sp_{i}\) & For each \(sp\) we should \\  & have at-least one \(tr\) \\ \(\sum_{k}lm_{k}\geq sp_{i}\) & For each \(sp\) we should \\  & have at-least one \(lm\). Including null \\  & landmarks \\ \(sp_{i}tr_{j}lm_{k}\gamma\leq sp_{i}tr_{j}lm_{k}\) & is-a constraints between relations and \\  & coarse-grained types \\ \(sp_{i}tr_{j}lm_{k}\underset{\lambda\in\Lambda_{\gamma}}{\leq}sp_{i}tr_{j}lm_{ k}\sqrt{}\) & is-a constraints between coarse-grained \\  & and corresponding \\  & fine-grained types. \\ \end{tabular}

### Image

In the image modality, we have two types of classifiers, 1) for localization of an object in the image given a referring expression, and 2) for extraction of spatial relations, called _Word-Segment Alignment_ and _Preposition Classifier_ respectively.

**Word-Segment alignment:** motivated by (Schlangen et al., 2016), we trained a set of binary object localization classifiers to link words and image segments. These per word classifiers are trained using _ImageClef Referring Expression Dataset_.

**Proposition classifier:** is a multi-class classifier that takes two bounding boxes and returns the spatial relation (preposition here) between them. This classifier is trained on a subset of visual genome dataset (Krishna et al., 2017) described in section 3.1, this classifier provides the external knowledge from visual resources and help in disambiguation of ambiguous links (i.e. finding the correct link between spatial preposition and spatial roles).

**Features:** A deep convolutional neural network, "GoogLeNet" is used to extract features for bounding boxes that are used by _Word-Segment Alignment_, for details see (Schlangen et al., 2016).

For the _Preposition Classifier_ we use bounding box features \(\phi_{box}(b)=[l,x_{min},y_{min},w_{b},h_{b}]\) where \(l\) is the label of the box, (\(x_{min}\), \(y_{min}\)) is the top-left point of the box, \(w_{b}\), \(h_{b}\) are the width and height of the box respectively. In addition we use pair features \(\phi_{pair}^{visual}(b_{i},b_{j})\) including, label of each box, distance between the center of the two boxes, a vector from the center of first box to the center of second box, aspect ratio of each box, word to vector representation of each box's label, the normalized area of each box, intersection, union, intersection over union of the 

two boxes, and four directional (above, below, left and right) features calculated with reference to the two boxes. Box and pair features are adopted from Ramisa et al. (2015).

### Inter-Modality

An essential part of having a global inference over multimodal data is to have the connections between the two modalities. _Word-Segment Alignment_ classifier is used to align the headword of each phrase \(p\) in the sentence to its corresponding bounding box \(b\) in the image and this alignment is denoted by \(p\to b\). A binary feature \(isAligned\) (that indicates if the phrase is connected to an object in the image) is added to the features of classifiers in the text side.

**Constraints:** Given two bounding boxes \(b_{1}\) and \(b_{2}\) we say that the preposition \(\alpha\) is supported by the image and write \(iSup^{\alpha}_{b_{1}b_{2}}\) if \(\alpha\) is ranked among top \(N\) prepositions according to _Preposition Classifier_ scores. Using this indicator we define the following inter-modality constraint.

\begin{tabular}{l l} \(iSup^{\alpha}_{b_{1}b_{2}}\leq sp_{it}r_{j}lm_{k}\) & For aligned pairs, the visual relation should support the textual relation \\ \end{tabular}

### Global Reasoning

We obtain the output of each classifier in the model holistically by global reasoning that is by considering global correlations among classifiers, when calculating outputs. This goal is achieved by optimizing an objective function that is the summation of classifiers' discriminant functions,

\[\sum_{i\in C_{sp}}\langle W_{sp},\phi_{sp_{i}}\rangle.sp_{i}+\sum_ {i\in C_{tr}}\langle W_{tr},\phi_{tri_{i}}\rangle.tr_{i}+\] \[\sum_{i\in C_{lm}}\langle W_{lm},\phi_{lm_{i}}\rangle.lm_{i}+\] \[\sum_{i\in C_{sp}}\sum_{j\in C_{tr}}\sum_{k\in C_{lm}}\langle W_{ sptrlm},\phi_{sp_{i}tr_{j}lm_{k}}\rangle.sp_{i}tr_{j}lm_{k}+\] \[\sum_{\gamma\in I}\sum_{i\in C_{lp}}\sum_{j\in C_{tr}}\sum_{k\in C _{lm}}\langle W_{sptlrm},\phi_{sp_{i}tr_{j}lm_{k}}\rangle.sp_{i}tr_{j}lm_{k}\gamma+\] \[\sum_{\lambda\in N}\sum_{i\in C_{Grv}}\sum_{j\in C_{frv}}\sum_{k \in C_{lm}}\langle W_{sprtlm},\phi_{sp_{ir}tr_{j}lm_{k}}\rangle.sp_{i}tr_{j} lm_{k}\lambda+\] \[\sum_{\alpha\in P*rep}\sum_{(i,j)\in C_{Supp}}\langle W_{iSup^{ \alpha}_{ij}},\phi_{iSup^{\alpha}_{ij}}\rangle.iSup^{\alpha}_{ij}.\]

Each classifier is shown as a binary variable (e.g. \(tr_{i}\) for trajector classifier). \(\Lambda\), \(\Gamma\), \(Prep\) are the candidates for fine-grained relations, coarse-grained relations, and prepositions from text respectively. \(C_{l}\) denotes the candidates for label \(l\).

The following model variations are designed using combination of text and image modalities for experimentation.

**Baseline Model (BM):** Independent classifiers are trained only on the textual features described in Section 2.1. This is a learning only model and each classifier makes independent predictions.

**Baseline + Constraints (BM+C):** The output of the classifiers obtained from the \(BM\) model are adjusted by global inference over textual constraints defined in Section 2.1.

**Ground-truth alignments (GT):** This setting is very similar to the \(BM+C\) model except the \(isAligned\) feature (see Section 2.3) added to consider the ground-truth alignments.

**Alignment Classifier (AC):** Similar to the GT model, but instead of ground-truth information we use _Word-Segment Alignment_ classifier to align bounding boxes with the phrases in the sentence.

**GT + Preposition (GT+P):** In this setting, ground-truth alignments alongside _Preposition classifier_ is used to enforce all constraints in the global inference over the two modalities.

**AC + Preposition (AC+P):** Same as GT+P model but with _Word-Segment Alignment_ classifiers instead of ground-truth alignments.

## 3 Experimental Setup

We report the experimental results of our model and compare it with the state-of-the-art Kordjamshidi et al. (2017a) model, referred here as _M0_ model. A role prediction is considered correct if there is a phrase overlap between the ground-truth and predicted roles and each relation is counted as correct when all three arguments are correct. All the base classifiers described in Section 2.1 are sparse perceptrons. We use _Saul_Kordjamshidi et al. (2015, 2016) to implement the models and solve the global inference of Section 2.4. The code is publicly available here. 2

Footnote 2: [https://github.com/HetML/SpRL/tree/paper2](https://github.com/HetML/SpRL/tree/paper2)

### DataSets

**CLEF 2017 mSpRL dataset:** This dataset is a subset of IAPR TC-123 Benchmark which is annotated for the SpRL task Kordjamshidi et al. (2017c, 2012). It contains \(613\) images with descriptions including \(1,213\) sentences. The standard split of the dataset contains \(761\) training and \(939\) testing spatial relations Kordjamshidi et al. (2017b). Furthermore, we 

to this dataset to align phrases in the text with the segments of the related images using brat tool.4 The alignments are used only for evaluations and are publicly available. 5

Footnote 3: [http://brt.nlplab.org/](http://brt.nlplab.org/)

Footnote 4: [http://brt.nlplab.org/](http://brt.nlplab.org/)

**Visual Genome dataset (VG):** Visual Genome dataset has seven main components Krishna et al. (2017), one of them is relationships component which contains the relationships (prepositions) between two bounding boxes. The dataset contains 108077 images and the relationships component contains 2316104 relation instances. We used a subset the relationships that correspond to the most frequent prepositions in CLEF dataset. We used 80% for training (811661 instances) and 20% for testing (202916 instances).

**ReferItGame Dataset:** It contains \(120,000\) referring expressions and covers \(99.5\) percent of the regions of SAIAPRTC-12 dataset which is a segmented and annotated version of the IAPR TC-12 dataset Kazemzadeh et al. (2014).

### Experimental Results

Word-Segment Alignment:We implemented and trained classifiers per words as described in Section 2.2 for the most frequent words in ReferItGame dataset using Schlangen et al. (2016) approach. We evaluated the trained model on both ReferitGame and CLEF testset, and obtained 64% and 45% accuracy respectively. This trained model is used to align words and segments in CLEF dataset. The end-to-end evaluation results show that the models trained by this external dataset are helpful though those are not highly accurate for every referring word.

**Preposition Classifier:** As described in Section 2.2, these are trained on a subset of Visual Genome dataset described in Section 3.1 and evaluated on both Visual Genome and CLEF test sets. Table 0(a) shows five best prepositions result whereas the result for other prepositions is less than 20% F1.

**Spatial Relations:** The experimental results in Table 0(b) show that our baseline model (BM) is as good as the state-of-the-art model (M0). Incorporating \(isAligned\) feature (in _GT_ and _AC_ models) further improves the results because having the phrases visualized in the image increases the confidence scores of the spatial role and relation classifiers and leads to a higher recall. The global inference over constraints in _BM+C_ significantly improves the performance of _BM_ (about 5% F1). _GT+P_ results show that inter-modality constraints help in improving the results (about 2% F1) which indicates some of the visual relations successfully confirmed and boosted their corresponding relations in the text modality. However, this improvement is limited which is expected considering the low performance of _Preposition Classifier_. The _GT+P_ results indicate the significance of the visual information in our model when the correct alignments are provided. The alignment classifiers in the _AC_ model also slightly improve the _BM+C_. However, as it is visible in _AC+P_ results, when we have both noisy alignments and noisy visual relations the results drop slightly compared to _AC_.

## 4 Conclusion

Our global inference model exploits visual modality classifiers including object localization by referring expressions and spatial relation classifiers between visual objects, as well as classifiers that extract spatial roles and relation from text. The global inference imposes consistency over the two modalities and identifies the spatial relations in text in accordance with their counterparts in the image. The experimental results show the effectiveness of the visual information in resolving the ambiguity of spatial semantics of text. There is still a large room to improve the modality alignments and relation extraction from images to obtain better gains from visual information.

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline  & \multicolumn{3}{c|}{Visual Genome} & \multicolumn{3}{c|}{CLEF} \\ \hline  & P & R & F1 & P & R & F1 \\ \hline \hline above & 47.24 & 21.59 & 29.63 & 87.50 & 22.58 & 35.90 \\ behind & 65.02 & 22.65 & 33.56 & 80.00 & 22.22 & 34.78 \\ in & 80.99 & 54.96 & 65.48 & 83.80 & 79.87 & 81.79 \\ in front of & 31.65 & 6.67 & 11.02 & 80.00 & 16.33 & 27.12 \\ on & 79.76 & 95.75 & 87.03 & 38.39 & 91.49 & 54.09 \\ \hline \end{tabular}
\end{table}
Table 1: Experimental results, where P and R denote precision and recall respectively.

 



## References

* Christie et al. (2016) Gordon Christie, Ankit Laddha, Aishwarya Agrawal, Stanislaw Antol, Yash Goyal, Kevin Kochersberger, and Dhruv Batra. 2016. Resolving language and vision ambiguities together: Joint segmentation & prepositional attachment resolution in captioned scenes. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. pages 1493-1503.
* Delecraz et al. (2017) Sebastien Delecraz, Alexis Nasr, Frederic Bechet, and Benoit Favre. 2017. Correcting prepositional phrase attachments using multimodal corpora. In _Proceedings of the 15th International Conference on Parsing Technologies_. pages 72-77.
* Faghri et al. (2017) Fartash Faghri, David J. Fleet, Ryan Kiros, and Sanja Fidler. 2017. VSE++: improved visual-semantic embeddings. _CoRR_ abs/1707.05612. [http://arxiv.org/abs/1707.05612](http://arxiv.org/abs/1707.05612).
* Gupta and Malik (2015) Saurabh Gupta and Jitendra Malik. 2015. Visual semantic role labeling. _CoRR_ abs/1505.04474. [http://arxiv.org/abs/1505.04474](http://arxiv.org/abs/1505.04474).
* Haldekar et al. (2017) Mandar Haldekar, Ashwinkumar Ganesan, and Tim Oates. 2017. Identifying spatial relations in images using convolutional neural networks. In _Neural Networks (IJCNN), 2017 International Joint Conference on_. IEEE, pages 3593-3600.
* Hu et al. (2016) Ronghang Hu, Huzhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natural language object retrieval. In _Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on_. IEEE, pages 4555-4564.
* Karpathy and Fei-Fei (2017) Andrej Karpathy and Li Fei-Fei. 2017. Deep visual-semantic alignments for generating image descriptions. _IEEE Trans. Pattern Anal. Mach. Intell._ 39(4):664-676. [https://doi.org/10.1109/TPAMI.2016.2598339](https://doi.org/10.1109/TPAMI.2016.2598339).
* Kazemzadeh et al. (2014) Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. 2014. Referit game: Referring to objects in photographs of natural scenes. In _EMNLP_.
* Kordjamshidi et al. (2015) P. Kordjamshidi, D. Roth, and H. Wu. 2015. Saul: Towards declarative learning based programming. In _Proc. of the International Joint Conference on Artificial Intelligence (IJCAI)_. [http://cogcomp.cs.illinois.edu/papers/KordjamshidiRoWul5.pdf](http://cogcomp.cs.illinois.edu/papers/KordjamshidiRoWul5.pdf).
* Kordjamshidi et al. (2012) Parisa Kordjamshidi, Steven Bethard, and Marie-Francine Moens. 2012. SemEval-2012 task 3: Spatial role labeling. In _Proceedings of the First Joint Conference on Lexical and Computational Semantics: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval)_. volume 2, pages 365-373.
* Kordjamshidi et al. (2016) Parisa Kordjamshidi, Daniel Khashabi, Christos Christodoulopoulos, Bhargav Mangipudi, Sameer Singh, and Dan Roth. 2016. Better call saul: Flexible programming for learning and inference in nlp. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. pages 3030-3040.
* Kordjamshidi and Moens (2015) Parisa Kordjamshidi and Marie-Francine Moens. 2015. Global machine learning for spatial ontology population. _Web Semantics: Science, Services and Agents on the World Wide Web_ 30:3-21.
* Kordjamshidi et al. (2017a) Parisa Kordjamshidi, Taher Rahgooy, and Umar Manzoor. 2017a. Spatial language understanding with multimodal graphs using declarative learning based programming. In _Proceedings of the 2nd Workshop on Structured Prediction for Natural Language Processing_. pages 33-43.
* Kordjamshidi et al. (2017b) Parisa Kordjamshidi, Taher Rahgooy, Marie-Francine Moens, James Pustejovsky, Umar Manzoor, and Kirk Roberts. 2017b. Clf 2017: Multimodal spatial role labeling (msprl) task overview. In _International Conference of the Cross-Language Evaluation Forum for European Languages_. Springer, pages 367-376.
* Transactions on Speech and Language Processing_ 8:1-36.
* Kordjamshidi et al. (2017c) Parisa Kordjamshidi, Martijn van Otterlo, and Mariet Francine Moens. 2017c. Spatial role labeling annotation scheme. In N. Ide James Pustejovsky, editor, _Handbook of Linguistic Annotation_, Springer Verlag.
* Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. _International Journal of Computer Vision_ 123(1):32-73.
* Liao et al. (2017) Wentong Liao, Shuai Lin, Bodo Rosenhahn, and Michael Ying Yang. 2017. Natural language guided visual relationship detection. _CoRR_ abs/1711.06032. [http://arxiv.org/abs/1711.06032](http://arxiv.org/abs/1711.06032).
* Lu et al. (2016) Cewu Lu, Runjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In _European Conference on Computer Vision_. Springer, pages 852-869.
* Seyre et al. (2017) Julia Seyre, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2017. Weakly-supervised learning of visual relations. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. pages 5179-5188

V. Punyakanok, D. Roth, W. Tau Yih, and D. Zimak (2005)Learning and inference over constrained output. In IJCAI'05, pp. 1124-1129. External Links: Link Cited by: SS3.
* J. Pustejovsky, P. Kordjamshidi, M. Moens, A. Levine, S. Dworman, and Z. Yocum (2015)Semeval-2015 task 8: spaceeval. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), Vol. , pp. 884-894. External Links: Document, ISSN 1763-640X, Link Cited by: SS3.
* A. Ramisa, J. Wang, Y. Lu, E. Dellandrea, F. Moreno-Noguer, and R. Gaizauskas (2015)Combining geometric, textual and visual features for predicting prepositions in image descriptions. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, H. Wallerstein, C. Cortes, T. Larochelle, B. Durrett, and G. Garnett (Eds.), pp. 214-220. External Links: Link, ISBN 978-3-3