Semi-supervised multimodal coreference resolution in image narrations
Authors: Arushi Goel
Basura Fernando
Frank Keller
Hakan Bilen
Venue: EMNLP 2023 Main (2023)
Keywords: Coreference Resolution
Vision Language Understanding
Narrative Grounding
Semi-Supervised Learning

Abstract: In this paper, we study multimodal coreference resolution, specifically where a longer descriptive text, i.e., a narration is paired with an image. This poses significant challenges due to fine-grained image-text alignment, inherent ambiguity present in narrative language, and unavailability of large annotated training sets. To tackle these challenges, we present a data efficient semi-supervised approach that utilizes image-narration pairs to resolve coreferences and narrative grounding in a multimodal context. Our approach incorporates losses for both labeled and unlabeled data within a cross-modal framework. Our evaluation shows that the proposed approach outperforms strong baselines both quantitatively and qualitatively, for the tasks of coreference resolution and narrative grounding.

Semi-supervised Multimodal Coreference Resolution in Image Narrations
Arushi Goel', Basura Fernando', Frank Keller', and Hakan Bilen'
'School of Informatics, University of Edinburgh, UK ?CFAR, IHPC, A ASTAR,, Singapore
Abstract
we cas see two smoll are sitting white color mat in that one boy is crying and he is wearing yellow color t-shirt and color shori. The other buy weeming color hu is holling some object his On the head the boy we cun see white color sticker and on their t-shars can see designs
In this paper, we study multimodal corefer- ence resolution, specifically where a longer de- scriptive text, f.e., narration is paired with an image. This poses significant challenges due to fine-grained image-text alignment, in- herent ambiguity present in narrative language, and unavailability of large annotated training present
oolor and
Figure 1: Example image- narration pair from the Coref- erenced Image Narratives dataset (Goel et al., 2022). Phrases marked in the same color corefer to the same entity which are also grounded in the image. We do not show singletons for brevity.
sets. To tackle these challenges, a data efficient semi- supervise approach that utilizes image-narration pairs resolve corefer- ences and narrative grounding in multimodal context. Our approach incorporates losses for both labeled and unlabeled data within a cross- modal framework. Our evaluation shows that the proposed approach outperforms strong base- lines both quantitatively and qualitatively, for the tasks of coreference resolution and narrative grounding.
identifying fine-grained visual details (e.g., disam- biguating them by recognizing the action 'crying', spotting 'white color t-shirt and cream color short' or white color sticker on the head'), ) and captur- ing long-range dependency across sentences (e.g.. two small boys and their).
MCR has recently gained increasing attention, with several notable studies (Ramanathan al., 2014; Huang et al., 2018; Cui al., 2021; Parcal- abescu et al., 2021; Das et al., 2017; Guo et al., 2022; Goel et al., 2022; Hong et al., 2023). How- ever, many of them focus on images with sim- ple short sentences, such as 'A woman is driving motorcycle. Is she wearing helmet?' (Das et al., 2017: Parcalabescu et al., 2021), or are lim- ited to identifying movie characters or people (Ra- manathan et al., 2014: Cui al., 2021). More recently, Goel al. (2022) introduced challeng- ing and unconstrained MCR problem (see Figure 1) including a dataset, Coreferenced Image Narratives (CIN), with both people and objects as referents with long textual descriptions (narrations). As man- ually annotating large dataset with coreferenc- ing and grounding labels is expensive, the authors provide annotations only for evaluation purposes. They also propose a weakly supervised method that learns to jointly ground mentions in images and use them as anchors along with prior linguistic rules (Lee et al., 201 1) to group coreferring men-
1 Introduction
In linguistic processing, coreference resolution is a standard task that aims to identify referring expressions such as noun phrases and pronouns that refer to the same entity. It is fundamental to many standard problems including question an- swering (K (Kwiatkowski et al., 2019; Das et al., 2017), sentiment analysis (Cambria et al.. 2017; Medhat et al., 2014), summarization (Gupta and Lehal, 2010: Shi et al., 2021) and machine transla- tion (Lopez, 2008; Bahdanau et al., 2014; Wu et al., 2016). In this work, we focus on a multimodal coreference resolution (MCR) scenario where the coreferences occur in a narration paired with an image and also link to an image region as shown in Figure 1. Here resolving coreferences is chal- lenging, mentions referring different entities can be very similar when encoded by language model, e.g., one boy, the other boy, the boy. Hence it demands a tne-grained understanding of each modality and as well as across them. In particular, it requires simultaneously grounding instances by

---

tions from only image and narration pairs without the annotations. The method has multiple short- comings: (1) weakly supervised grounding fails to disambiguate multiple instances of the same object class, boy (one boy, the other boy), (2) language rules such as exact match of phrases are either too strict or too generic, e.g., pronoun match, linking pronouns to one antecedent (one boy, he, he, his) and, (3) they require an additional modality, mouse traces learn coreferences which can be expensive obtain.
cific regions in an image. Recently, consider- able efforts have been dedicated to developing datasets that can effectively address this intricate task. Parcalabescu et al. (2021) introduced the VALSE dataset, which encompasses various coref- erence scenarios. However, this dataset focuses on the downstream task of visual question answer- ing without evaluating coreference resolution grounding. Hence, evaluate our method on CIN dataset (Goel al., 2022) that contains coreference chains and grounding annotations. Another ap- proach to MCR datasets involves linking people's s names mentioned in the text to corresponding im- ages and resolving pronouns that connect to those specific names (Ramanathan et al., 2014; Cui et al., 2021; Hong et al., 2023). However, our main fo- cus is to resolve coreferences in a generic scenario (with visual complexity) unlike the others that are either limited to only people names/characters (Ra- manathan et al., 2014; Cui et al., 2021; Hong et al., 2023) or have simple sentences (Das et al., 2017; Parcalabescu et al., 2021).
Motivated by these limitations, we argue that it difficult to successfully resolve coreferences from only image-narration pairs in cases where multiple instances of the same object category are present; this situation is common coincides with a men- tion in the narration. Since full manual annota- tions of coreference chains and bounding boxes is expensive, we propose to resolve coreferences and ground mentions in a semi-supervised setting where only a few data samples are labeled. Our ap- proach involves a customized multi-modal fusion model that combines image region features and mention features from narrations through cross- attention (Vaswani et al., 2017; Li al., 2021). We investigate different task-specific losses for training on labeled and unlabeled data, and show that naively combining training on the labeled and pseudo-labeled data suffers from severe overfit- ting (Arazo et al., 2020). Hence, we propose robust loss function and thresholding-busee train- ing scheme to effectively learn from the unlabeled set. This novel approach results in consistent per- formance improvements with the inclusion of unla- beled data during training.
Vision-language learning. Existing work on vi- sion and language understanding employs either pre-trained object detector features (He et al., 2017; Ren et al., 2015) as an image encoder, VIT (Doso- vitskiy et al., 2020) serman, 2014) combined with transformer-hasee text encoder (Devlin et al.. 2018). To model cross- modal interaction between the image and text en- coders, UNITER (Chen et al., 2020), ALBEF (Li et al., 2021) and VIn (Zhang et al., 2021b) em- ploy a multimodal encoder. They are pre-trained on large-scale image-caption pairs such as COCO (Lin et al., 2014), Conceptual captions (Sharma et al., 2018; Changpinyo et al., 2021), Visual Genome (Krishna et al., 2017). The pre-training objectives are implemented with image-text contrastive loss, masked language modeling, and image-text match- ing loss. Our method is inspired by these architec- tures and is trained using a set of self-supervised and task-based objectives in a semi-supervised learning fashion.
CNN (Simonyan and Zis-
Our main contributions are (1) vision-language framework for MCR trained on a small labeled and unlabeled dataset, (2) novel task-specific losses (on both labeled and pseudo-labeled data) for learning joint multi-modal embeddings for coref- erence resolution while simultancously improving narrative grounding, (3) extensive evaluation of our proposed method on the CIN dataset and ablation studies to validate our design choices, showing con- sistent performance gains compared to baselines coreference resolution and narrative grounding.
Semi-supervised learning. There is a large body of work in semi-supervised learning (Zhai et al., 2019; Van Engelen and Hoos, 2020; Ouali et al., 2020). These methods typically exploit unlabeled data via either pseudo-labeling with small amounts of labeled data (Lee al., 2013; Arazo et al., 2020; Rizve et al., 2021; Sohn et al., 2020; Zhang et al., 2021al or by enforcing consistency regularization
2 Related work
Multimodal coreference resolution. MCR in- volves comprehending the contextual information in language and establishing connections with spe-

---

(Berthelot al., 2019; Abuduweill et al., 2021) on the unlabeled data to produce consistent predictions over various perturbations of the same input by applying several augmentation strategies (Zhang et al., 2017; Cubuk et al., 2018, 2020). Our method draws inspiration from pseudo-labeling literature and uses robust loss function and thresholding to counter overfitting to pseudo-labels.
CR
BBR
GD
MLM
PCR
PGD
Food Forward
Cross Atteutina
Muttinedal
Profice pecudo- labels for unlaheted
Self Attention
N
3 Method
Visual
Text Eiccoder
3.1 Task Overview
Encoder
Data
Narrations
Dosa
Our goal is (1) to group mentions (i.e., referen- tial words phrases) in the narration that core- fer the same entity and, (2) to ground each mention to region in image. Formally, let {m1, m2......... M denote narration with N mentions for an image with / regions where I = {r1, r2..... }
Figure 2: Illustration of model architecture and training methodology. The pre-extracted image regions are fed into the visual encoder. the narrations are fed into the text encoder and both modalities are fused using multimodal encoder. The model is optimized using self supervisedd objectives (in grey) and specialized task- based losses both the labeled data (in yellow boxes) and the pseudo-labeled data (in green boxes).
}. We wish to learn an embed- ding function / that takes in an image and its narration N, parsed to contain a set of mentions, and outputs a score for a mention pair (m, m'):
representations fe(m) [i Ewem fe(w)
f(m)- f(m') (fmm)
where, |m| indicates the mention length in words, (1)
and the embeddings fe(m) have the same dimen- sionality as the visual features.
The mention pair m and m' corefers if the score in Equation (1) is high, otherwise they do not. For grounding of the mention region also learn another function g that out- puts score for the mention m being located region r in image T. Next, we describe in detail our methodology to learn the two functions f and
Next, the multi-moda encoder f fuses the visual features from the visual encoder f+(v,, with the mention features from the text encoder f((m). Sim- ilar to the cross-modal architectures (Li et al., 2021; Zhang et al., 2021b), the embeddings from the text encoder are first encoded using self-attention lay- ers (Vaswani et al., 2017). Then, a multi-head cross attention module integrates the textual and visual features. In the cross-attention module, the self-attended mention embeddings (m) treated as the query, while the image representa- tions fo (u()) are treated as keys and values. The attention weights between the mention m and the region r are given as:
the image
3.2 Model Architecture
Figure 2, we illustrate our model architecture. Each image is parsed into set of regions through a pre-trained object detector (Ren et al., 2015), where each region is represented by d-dimensional joint embedding v, Rd including its visual, se- mantic and spatial features. In particular, the visual encoder fo is instantiated as transformer block that takes in a joint feature embedding e, for the object region and outputs a D dimensional em- bedding, i.e., fu(w)) d D
exp( fe(m) .f((te)) vd
g(m,r) Erer exp( (dlm)_.full,p (2)
Va
Furthermore, we encode the words in each nar- ration N using tokenizer (Devlin et al., 2018) to get set of tokens for the words is the vocabulary size. The text encoder fi which is also transformer block that takes in the word token w and outputs a D dimensional embedding, ie., fe(w): : V + D The mention embeddings are computed by averaging its corresponding word
where the softmax is computed over the image regions for each mention. This attention matrix (or the grounding function) from the multi multi-head cross attention learns fine-grained mention to re- gion alignment scores. Finally, the vision-aware mention embedding is represented as:
RVV where V
f(m) = ((m,r).f((v))
(3)

---

where, f(m) RD. This weighted embedding is then passed to a feed-forward module (Li al., 2021) with an MLP and layer normalization. All the transformer encoders/blocks are based on the architecture proposed by (Li et al., 2021). It is im- portant to note that compared to Goel et al. (2022), our model fuses vision and text features with multimodal encoder, unlike theirs.
training set Ds. For the ground-truth bounding box bm for a mention m we compute the intersection over union (loU) between this bounding-box and the R pre-extracted image regions. This is crucial because we don't 't have the exact region-mention match for the detections from the object detector. Following this, we get the binary alignment func- tion h(m,r), which is 1 for the mention and the detected image region if the region r' has the maximum IoU overlap with the ground-truth bounding box bm, and 0 otherwise. Once we have the ground-truth alignment h(m, r), we compute the cross-entropy loss as:
3.3 Semi-supervised Learning
Concretely, aim learn the parameters of the modules fe- Je and given a training dataset 'D with [D] samples of image-narration pairs. Specifi- cally, we use small labeled set D, = { y4 ) Del where 2; = N is the image-narration input (P(m) } is the la- bel for the input pair. particular, the label for each mention m the narration is given as: P(m) and A Amm), the set of positive and negative men- tions respectively for the mention m and bm. the bounding-bol box coordinates of the region correspond- ing to the mention m.
Lgd = Σ E ((mrr))lo((((mmr) (6)
mEN rET
pair and yi
(S3) Bounding box regression loss (BBR) We fur- ther propose to add additional supervision to fine the object proposals from the detector for a mention. For each mention m, the ground-truth bounding box localization is represented as bm = (I, y; wc h). To learn refinements, we predict the box deltas from the model as 6m = (6x, oy 6w, 5%) for each mention m. We then take the highest scor- ing region for given mention 772 as:
Duc to the unavailability of a large labeled train- ing set, we leverage the unlabeled data D. =D\ (D, where, Du [x:) |Pul with only image-narration pairs inputs. Our overall training objective is the joint loss function as follows:
Tin arg arg max g(m, r).
rE/
Our goal is to learn a transformation that maps a proposed box I'm to ground-truth box dn We then apply the smooth-L1 loss following Ren et al. (2015), denoted as Lobr- Further details about this loss are given in the appendix.
Σ
1 [D, (,(,)))++ + Σ (x(r)
RED [Dal Cu(x)
(4)
(D,))DD
where, Lg is the supervised loss and Lu is the un- supervised loss. First, we discuss how to formulate bask-based supervised losses on the dataset D.- (S1) Coreference loss (CR) Specifically, we pro- pose to learn the similarity between the mention embeddings using a supervised contrastive loss (Khosla et al., 2020) which is defined as:
Next, we discuss how to train on the unlabeled subset of the dataset by generating pscudo- for the coreference and grounding tasks.
(U1) Pseudo coreference loss (PCR) Given the un- labeled dataset Da,, we compute the pseudo corefer- ring pairs for the mentions in N. More specifically, compute aseudo-positive p(m) and pseudo- negatives Á(m) for mention m by computing the cosine similarity between the embeddings as in Equation (1). For each mention m, if the similarity with another mention m' is greater than a threshold then we label it as positive otherwise a negative. Finally, we compute the triplet loss as:
Per Σ mEN
-1
Σ
P(m))
pEP(m)
(5)
log crp(s(m). (p)/T)
A(EA(m) (xp(f fmm)
where T is the temperature. This loss helps to clus- ter embeddings for coreferring mentions together and push the embeddings of non-referrants away from each other.
Σ max((1ff(m - [P(m))
Σ
f(6)112
(S2) Grounding loss (GD) To align the mention m and region r, we use the grounding function g de- fined in Equation (2). In particular, we first define the ground-trute truth binary alignment on the labeled
MEN
pc p(P)
(8)
-l1((m)
Σ
1(a)12
A((m) dEA(m)

---

where is the margin, ((m is the embeddings for the query mention m. [Prm)] Epep(m) (p) is the mean of embeddings of the pseudo-positing
masked and are replaced by special masking to- ken. The model needs predict the mask token based on the unmasked words. This task is trained with a cross-entropy loss, Lmim-
labels P(m) and [A(m)]] Eaci(m) f(a) is the mean
Hence, our overall training objective in Equa- tion (4) is a combination of specialized task losses the labeled training set D, (Lir, Lyd and Libr) and the unlabeled training set Du (Cper and ppdd) and global pre-training objectives on the entire training dataset 'D (Este and mmmm).
of embeddings of the pseudo-negative labels A(m).
The key intuition behind using the mean in triplet loss formulation is reduce overfitting to the noise in the pseudo labels. This works bet- ter in practice compared to the contrastive loss formulation in Equation (5) or mining a random positive/negatiee label for the standard tripict loss, especially when dealing with pseudo labels.
3,4 Inference
To obtain the coreference scores, we form chains by measuring the cosine similarity between the mentions as described in Equation (1), considering the pairs with similarity higher than a predefined threshold positives. When evaluating narrative grounding. we extract the cross-attention scores from the last layer of the multimodal encoder. For each mention, we identify the region with the high- est softmax score as the positively referred region.
(U2) Pseudo grounding loss (PGD) Furthermore, we compute the pseudo grounding loss on the un- labeled training dataset. Specifically, we impute the pseudo-labels from the grounding function, g(m,r). We only consider samples whose ground- ing score is greater than a confidence threshold t, which is set to 0.9 in our experiments. The high threshold value ensures that we consider only con- fident samples in the unlabeled set and eliminates learning from noisy samples. We denote this label after binary thresholding as h(m,r). The pseudo grounding alignment loss is:
Experiments
Datasets. We evaluate our proposed method on the CIN dataset (Goel et al., 2022) that consists of 1000 test and 880 validation image-narration pairs from the Flickr30k split of the Localized Nar- ratives dataset (Pont-Tuset et al.. 2020) annotated with coreference chains and bounding boxes. We use the test split of the CIN dataset to report the performance on CR and narrative grounding. The annotations from the validation split are used the small labeled set during training. The unla- beled dataset is the F'lickr30k training subset of the Localized Narratives dataset, which consists of 50k image-narration pairs but is not annotated with bounding boxes or coreference chains.
Cpgd Σ
mEN PEI
Apart from the above mentioned task-based losses, we combine the standard image-text pre- training losses (Vaswani et al., 2017: Li et al., 2021). These losses help to learn better unimodal repre- sentations before fusion.
(U3) Image-Texx contrastive loss (ITC) Following Goel et al. (2022), we incorporate the contrastive loss to align the image and narration pairs to learn better representations before fusion. This loss is defined as:
Implementation details. For the image regions, we extract bounding box regions, visual features and object class labels using the Faster-RCNN ob- ject detector (Ren et al., 2015) as in Goel et al. (2022). We use a 4-layer transformer architecture for the text encoder and the multi-modal encoder similar to the ALBEF (Li et al., 2021) framework. The weights of the transformer encoders are initial- ized with the first four layers of BERT (Devlin al., 2018). The visual encoder is a stack of two trans- former encoder layers. Each transformer encoder layer includes multi-head self-attention layer and an FFN. There are two heads the multi-heas at- tention layer, and two FC layers followed by ReLU
Lite Σ log exp(f.(u,)f((m) MEN
Σ (xpeh((((
(10)
where f( is the mention-region match- ing score from the visual and text representations before fusing in the multi-modal encoder and v, are the raw features for the highest scoring region for mention
(U4) Masked language modeling loss (MLM) To fine-tune the pretrained BERT model (Devlin et al., 2018) on the image-narration data, we also use the pre-trained task of masked language modeling. particular, the input word tokens are randomly

---

Modality
MUC
B²
CEAF
CoNLL
Method
Text
Image
R
P
F1
R
P
F1
R
P
F1
FI
Neural Coref (Lee et al., 2017)**
0.11
0.17
0.13
longdoe (Toshniwal et al., 2021)"
7.79
8:43
7.24
62.27
76.10
67.69
48.77
84.95
61.02
45.3
VisuaBBRT (Su et al,, 2019)"
18.17
6.08
8.06
09.01
36.08
41.03
21:25
57.10
28.67
25,92
al., 2020y'
16.92
28.
72.78
38.91
(Zhang al.,, 202
62.32
61.30
42.88
53.69
41.58
(Wang al., 2020)
al., 2022)
24.87
13.30
12.55
56.05
86.20
67.05
50.67
Ours
31.11
35.25
31.86
70.63
87.85
78.06
63.99
93.44
75.47
61.79
Table Coreference resolution results on the CIN dataset (Goel al., 2022) from our proposed method and other state-of-the-ar unimodal and multi-moda baselines. indicates the use of predicted mentions, while the other results rely on ground-truth mentions during inference. means zero-shot performance.
activation layers in the FFN. Training details are in the appendix.
wal et al., 2021), but drop in performance on the average CoNLL FI scores. These results show the inability of these models to effectively find sin- gletons, hence leading poor performance in the precision scores. Moreover, we can conclude that the vision and language pre-trained models fail to generalize for MCR.
Evaluation. We report results for coreference reso- lution and narrative grounding. For the former, we use the standard CoNLL Fl score which is the aver- age of three coreference- based metrics: MUC, B and CEAFF For the latter, we follow Goel et al. (2022) and report the grounding accuracy for both noun phrases and pronouns. More precisely, if the overlap between the ground-truth truth box and the pre- dicted box is greater than 0.5, then is considered to be a correct prediction.
We also compare to two weakly supervised meth- ods that trained the CIN dataset, MAF (Wang et al., 2020) and WS-MCR (Goel al., 2022). Goel et al. (2022) present results on the MAF model as a baseline and their proposed method, WS-MCR. MAF is a weakly supervised grounding method trained with ITC that is evalu- ated for CR and WS-MCR (Goel et al., 2022) learns grounding and CR combining the ITC loss and prior linguistic rules. Both of these methods improve significantly in MUC scores com- pared to other zero-shot unimodal and multi-modal baselines.
5 Results and Discussion
5.1 Coreference Resolution
Table reports the coreference resolution perfor- mance on the CIN dataset (Goel al., 2022) for our method and the baselines. Further details about the baselines are given in the appendix. The text- based baselines Neural Coref (Lee et al., 2017) and longdoe (Toshniwal et al., 2021) are evaluated in a zero-shot way on the task. Their low CoNLL Fl scores indicate the incapability of the model to generalize
Finally, we compare with the text-only variant (without any images) of our method. This method improves over the baselines scores. The significant gains in performance of our final method, with both text and image, combined with label supervision shows the importance carefully tuning the model with a small amount labeled data and large amounts of pseudo-labeled data.
the CoNLL Fl
domains which is in line with what has been evaluated extensively in the coref- erence literature (Toshniwal et al., 2021; Xia and Van Durme, 2021; Gandhi et al., 2023).
We further compare to strong multi-modal base- lines by directly evaluating the VLMs in zero- shot way the CIN dataset. Interestingly, all three methods: VisualBERT (Su al., 2019). UNITER (Chen et al., 2020) and VinVL (Zhang et al., 2021b) perform better MUC and 3 com- pared to the text- based baseline, longdoc (Toshni-
5.2 Narrative Grounding
In Table 2, we present a comprehensive comparison between the baselines and our proposed approach on the task narrative grounding. This task is both challenging and crucial, as it evaluates the precise

---

alignment between image regions and phrases in textual data. Notably, our proposed method goes beyond the traditional alignment noun phrases and also addresses the grounding pronouns, which is vital for multimodal coreference resolution. We measure noun phrase grounding, pronoun ground- ing, and overall accuracy to measure performance (Goel et al., 2022).
Data type
% Samples
CoNLL FI
20%
60.04
Labeled
50%
61.24
20%
56.82
Unlabeled
50%
59.11
Table 3:
CR performance by changing the amount of labeled and unlabeled data during training.
Remarkably, our proposed method exhibits supe- rior performance compared to weakly supervised baselines, showing a margin of improvement if approximately 2% and 2.5% noun phrase and pronoun grounding accuracy, respectively. Further- more, when compared to our unsupervised base- line, namely "Ours (ITC MLM)", the inclusion of labeled and pseudo-labeled data yields significant performance boost of approximately 6%. These re- sults demonstrate the significance of training with both grounding alignment and coreference resolu- tion loss, highlighting the mutual benefits derived from this approach.
reinforces the ability of our proposed method to leverage and effectively learn from pseudo-labeled data. Detailed results in the appendix.
Impact of different loss functions. In Table 4, wc assess the performance of coreference resolu- tion by incorporating various losses proposed in Section 3. Throughout the training process, the model consistently integrates the self-supervised objectives of ITC and MLM, see first row of Table 4. Integrating the supervised contrastive corefer- ence resolution loss, CR, in addition to ITC and MLM. results in a significant performance drop. Due to the limited availability of labeled data, the model struggles to effectively generalize for coreference resolution, leading overfitting and consequently lower Fl scores. However, by progressively incor- porating the bounding box regression loss, BBR, and the grounding alignment loss GD, we get much stronger training signal even with small labeled set. This multi-task training objective contributes to an improvement of approximately 1.5% the CoNLL Fl score.
Method
Noun Phrases
Pronouns
Overall
MAF (Wang et al., 2020)
21.60
18.31
20.91
WS-MCR (Goel et al., 2022)
30.27
25.96
29.36
Ours (ITC + MLM)
27. 44
22.77
26.45
Ours (Full)
32,58
28.45
31.71
Table 2:
Comparison of narrative grounding perfor- mance on the CIN dataset (Goel et al., 2022).
5.3 Ablation Study
Subsequently, investigate the impact of incor- porating loss on pseudo-labeled data. By introduc- ing the pseudo coreference loss, denoted as PCR, we observe a remarkable improvement of approx- imatcly 2% in the CoNLL Fl scores. This result highlights the significance of leveraging pseudo clusters and underscores the effectiveness of our proposed robust triplet loss, which computes the triplet loss using the mean of positive and negative embeddings. Notably, this approach successfully incorporates pseudo-labeled data without leading to overfitting while achieving substantial perfor- mance gains. Consequently, our final proposed method, which integrates the pseudo grounding loss, PGD, exhibits the most superior overall perfor- mance, validating the potency of pseudo-labels for both coreference resolution and grounding.
Varying labeled and unlabeled data. We study the impact of labeled data on the learning process, allowing us to showcase the strengths of our ap- proach. In Table 3, we measure the model's perfor- mance on CoNLL F1 scores at different proportions of labeled data (20% and 50%). Remarkably, de- spite the limited amount of labeled data samples, the model demonstrates consistently high perfor- mance without any significant drop. This highlights the exceptional ability of our model to effectively learn from a small labeled set, without relying on a large number of annotated training samples. Furthermore, to validate the efficacy of our pro- posed method, we also investigate the influence of unlabeled data samples during training. Following the same data split in the supervised experiments, we observe the changes in performance indicated by row 2 in Table 3. As the quantity of unlabeled samples increases, the model exhibits enhanced coreference resolution performance. This result
Choice of coreference resolution loss. In Table 5, we examine the impact of different types of coref- erence resolution losses. We present a compari- son of the following loss combinations: (1) Binary

---

Loses
MUC:
IF
CHAF
CONLE
(OR(D))
POXU
LIQUU)
(U1)
(RI)
F1
x
X
23.81
25.83
25.12
69.32
85.87
76.41
61.00
19.09
72.05
57.19
22.70
20.23
80.03
55.52
x
/
x
23.86
24.52
22.51
69.31
84.15
75.67
59.50
89.15
70,80
56.26
27.65
29.04
26.66
69.93
85.43
76.61
60.92
90.61
72.36
58.51
30.66
32.82
30.31
70.70
86.09
77.33
62.64
92.92
74.27
60.64
31.11
35.25
31.86
70.63
87.85
78.06
63,99
93.44
75.47
61.79
Table 4: Ablation study on our proposed method with the combination the proposed losses.
MUR
82
CHARA
CONUE
10
in image see people sitting and the trant man is steeping and amma
R2F
DC1
31.55
23.57
00.50
74447
57.68
10001
55100
standing and the background blary..
12
23.08
72.40
64.1
812
23.46
77.45
14.55
people sitting an and the frost man sleeping and man stoddng and the backgound is blurty.
anage
Table 5: Performance comparison with the choice of coreference resolution loss on labeled dataset D, and the unlabeled dataset D...
Figure Visualization for grounding and coreference resolution. The colored boxes in image correspond to the mentions with the same color the sentence.
cross-entropy loss (BCE) applied both D3 and Du. (2) Supervised contrastive loss (CR) applicd to both D. and Du. and (3) Supervised contrastive loss (CR) on D. and random triplct mining loss (RTC) on D..
5,4 Qualitative Results
In Figure 3, qualitatively visualize the perfor- mance of our method and compare it with the weakly supervised baseline from Goel et al. (2022). Our model correctly separates the mentions the front man and the the man both during CR and grounding, whereas the WS-MCR (Goel et al., 2022) method incorrectly assigns the mention the man to the the front man and grounds it incorrectly too (denoted by the blue dotted line). Hence, our method can effectively learn disambiguate the instances based on the visual details which is also helpful for coreference resolution.
We observed significant performance drop when training with the BCE loss, compared to uti- lizing the supervised contrastive loss. We hypothe- size that the supervised contrastive loss provides a better clustering of mentions by contrasting them in the embedding space directly than the binary cross-entropy loss. Consequently, the embeddings become more robust for CR, contributing to im- proved performance.
Interestingly, when applying the supervised con- trastive loss to Da (row 2), we observed a drop in performance. Our hypothesis is that the contrastive loss tends to overfit in the presence of noisy pseudo labels, leading to degradation in performance. In contrast, our pscudo triplet loss formulation PCR is softer in penalizing noisy pseudo labels. This allows the model to gradually adapt and become more resilient to such noise, resulting in more effi- cient clustering of mentions. We also compare to another ablation where instead of taking the mean of the embeddings for sreudo-positivy labels and pscudo-negative labels, sample random pos- itive and negative label (results in row abbrevi- ated as RTC. Randomly sampling the labels gener- alizes better than the other ablations but the mean cluster embeddings outperforms than randomly se- lecting samples.
Conclusion
In conclusion, this paper addresses the challenging task of multimodal coreference resolution where an image is accompanied by a longer descriptive text. We propose a data efficient semi-supervised approach that incorporates task-based losses for both labeled and unlabeled data, operating within cross-modal framework. Our method achieves remarkable results for CR and narrative grounding tasks on the CIN dataset, showcasing its effective- ness handling the complexities of MCR. the future, we plan to investigate how the power of pre-training combined with semi sumi-supereised fine- tuning can be fully utilized for the task of MCR.

---

Limitations
This research/project is supported by the Na- tional Research Foundation, Singapore, under its NRF Fellowship (Award NRE-NRFF14-2022. 0001). Thanks to Nikita Moghe and Matt Grenan- der for discussions and constructive feedback.
Here, we outline limitations that are important con- siderations for future work.
First, the current model's performance in coref- erence resolution and grounding is limited by the use of a pre-trained object detector. The detectors pretrained for object detection task have limited object category vocabulary and lack in fine-gr tne-grained properties including adjectives, human actions and the open vocabulary found in narrations. This forces the model to rely on a predetermined of regions and object classes, preventing it from directly learning region coordinates for a mention on an image. To improve performance, we envi- sion the development of an end-to-end approach that climinates this reliance on pre-defined regions. Second, our model currently depends ground- truth mentions resolve coreferences and ground them. In the future, one promising direction would be detect mentions simultaneously with corefer- ence resolution and grounding. This would signif- icantly improve the applicability of our proposed method and reduce dependence on off-the-shelf mention detectors or ground-truth annotations.
References
Abulikemu Abuduweili, Xingjian Li, Humphrey Shi, Cheng-Zhong Xu, and Dejing Dou. 2021. Adaptive consistency regularization for semi- supervssd trans- learning. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6923-6932.
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. 2020. Pseudo- labeling and confirmation bias in deep semi- supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning align and translate. arXiv preprint arXiv:1409.0473.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raf- fel. 2019. Mixmatch: A holistic approach semi- supervised learning. Advances neural information processing systems, 32.
Ethics Statement
All datasets used this work have been previ- ously released. The use of the CIN dataset (Goel et al., 2022) in our paper is consistent with their intended use. The detail of the dataset is descri- bied in Goel et al. (2022). Multimodal datasets frequently include social biases, and expect the models trained them reflect the biases in these datasets. important note that multimodal models have both beneficial and harmful applica- tions. Beneficial applications include advanced image and video retrieval, visual description sys- tems to assist the visually impaired, and user in- terfaces that enhance interaction with smart home devices. However, harmful applications, such as non-consensual surveillance fine-tuning models retrieve inappropriate content, must be carefully addressed and mitigated.
Erik Cambria, Dipankar Das, Sivaji Bandyopadhyay. Antonio Feraco, al. 2017. A practical guide to sentiment analysis.
Soravit Changpinyo, Piyush Sharma. Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web- scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 3558-3565
Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference computer vision, pages 104-120. Springer.
Ekin D Cubuk, Barret Zoph, Dandelion Mane. Vijay Vasudevan, and Quoc Le. 2018. Autoaugment: Learning augmentation policies from data. arXiv preprimi arXiv:1805.0955
Acknowledgements
AG is supported by the Armeane Choksi Schol- arship and HB is supported by the EPSRC pro- gramme grant Visual AI EP/T028572/1 and HB and FK are supported by Edinburgh Labora- tory for Integrated Artificial Intelligence (ELIAT).
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. 2020. Randaugment: Practical au- tomated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition work- shops, pages 702-703.

---

Yuging Cui, Apoorv Khandelwal, Yoav Artzi, Noah Snavely, and Hadar Averbuch-Elor. 2021. Who's waldo? linking people across text and images. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 1374-1384.
De-An Huang, Shyamal Buch, Lucio Dery, Animesh Garg, Li Fei-Fei, and Juan Carlos Niebles. 2018. Finding" it"; Weakly-supervised reference-aware vi- sual grounding in instructional videos. In Proceed- ings of the IEEE Conference on Compuler Vision and Pattern Recognition, pages 5948-5957.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the LEEE conference on computer vision and pattern recognition, pages 326-335.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. Advances in neural information processing systems, 33:18661-18673.
Jacob Devlin, Ming-Wa Mei Chang, Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. Xrviv preprint
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. International journal of computer vision. 123(1):32-
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
73.
Tom Kwiatkowski, Jennimaria Palomaki. Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lec, al. 2019. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Nupoor Gandhi, Anjalie Field, and Emma Strubell. 2023. Annotating mentions alone enables efficient domain adaptation for coreference resolution. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 10543-10558, Toronto, Canada. Association for Computational Linguistics.
Dong-Hyun Lee et al. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning, ICML, volume 3. page 896.
Ross Girshick. 2015. Fast r-enn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448.
Heeyoung Lee, Yves Peirsman. Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Ju- rafsky. 2011. I. Stanford's 's multi-pass sieve coreference resolution system the con1l-2011 shared task. In Proceedings of the fifteenth conference on computa- tional nutural language learning: Shared task, pages 28 -4.
Arushi Goel, Basura Fernando, Frank Keller, and Hakan Bilen. 2022. Who are you referring to? weakly supervised coreference resolution with multimodal grounding. arXiv preprint arXiv:2211. (tt::2211.1156..
Danfeng Guo, Arpit Gupta, Sanchit Agarwal, Jiun- Yu Kao, Shuyang Gao, Arijit Biswas, Chien-Wat Wei Lin, Tagyoung Chung, and Mohit Bansal. 2022. Gravi- bert: Graphical Disual-linguistee representations for multimodal coreference resolution. In Proceedings of the 29th International Conference on Computa- tional Linguistics, pages 285-297.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettle- moyer. 2017. End-to-end neural coreference resolu- tion. arXiv preprint
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS.
Vishal Gupta and Gurpreet Singh Lehal. 2010. A survey of text summarization extractive techniques. Jour- nal of emerging technologies in web intelligence, 2(3):258-268.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European confer- ence on computer vision, pages 740-755. Springer.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages
Adam Lopez. 2008 Statistical machine translation. ACM Computing Surveys (CSUR), 40(3):1-99
Xudong Hong, Asad Sayeed, Khushboo Mehra, Vera Demherg, and Bernt Schiele. 2023. Visual writ- ing prompts: Character-grounded story generation with curated image sequences. arXiv preprint arXiv:2301.08577
Ilya Loshchilov and Frank Hutter. 2017.
Decou- pled weight decay regularization. arXiv preprint 171

---

Walaa Medhat, Ahmed Hassan, and Hoda Korashy, 2014. Sentiment analysis algorithms and applica- tions: A survey. Ain Shams engineering journal, 104):1093-1113.
with consistency and confidence. Advances in neural information processing systems, 33:596-608 608.
Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VI-bert: Pre-training of generic visual-linguiste representations. arXiv preprini arXiv: 7908.08880.00
Yassine Ouali, Céline Hudelot, and Myriam Tami. 2020. An overview of deep semi-supervised learning. arXiv preprint
Shubham Toshniwal, Patrick Xia, Sam Wiseman, Karen Livescu, and Kevin Gimpel. 2021. On general- ization in coreference resolution, arXiv preprint arXiv:2109.09667.
Letitia Parcalabescu, Michele Cafagna, Lilitta Murad- jan, Anette Frank, lacer Calixto, and Albert Gatt. 2021. Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena. arXiv preprint 1rX.v:21
Jesper E Van Engelen and Holger Hoos. 2020. A sur- vey -uperviie learning. Machine learning. 109(2):373 440.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo. Radu Soricut, and Vittorio Ferrari. 2020. Connecting vision and language with localized narratives. In European conference on computer vision, pages 647- 664. Springer.
Qinxin Wang, Hao Tan, Sheng Shen, Michael W Ma- honey, and Zhewei Yao. 2020. Maf: Multimodal alignment framework for weakly-supervised phrase grounding. arXiv preprint arXiv:2010.05379
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi. Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, al. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. arXiv preprint 1609.::609.088 081400
Vignesh Ramanathan, Armand Joulin, Percy Liang, and Li Fei-Fei. 2014. Linking people in videos with "their" names using coreference resolution. In Euro- pean conference on computer vision, pages 95-110. Springer.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.
Patrick Xia and Benjamin Van Durme. 2021. Moving on from OntoNotes: Coreference resolution model transfer. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing. pages 5241-5256, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. 2021. defense of pseudo-labeling: An uncertainty- pseudo-label selection framework for semi-supervised learning.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov. and Lucas Beyer. 2019. S41: Self-supervised semi-supervised learning. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 1476-1485.
arXiv preprint arXiv:2101.06322
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2556-2565.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. 2021a. Flexmatch: Boosting semi- supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34:18408- 18419.
Tian Shi. Yaser Keneshloo, Naren Ramakrishnan, and Chandan K Reddy. 2021. Neural abstractive text summarization with pruuence-to-sqqenne models. ACM Transactions on Data Science, 2(1):1-37.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint 171.v:1710.09411
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint Xrviv: 140v: 1556.
Pengchuan Zhang, Xiujun Li. Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang. Yejin Choi, and Jian- feng Gao. 2021b. Vinvl: Revisiting visual represen- tations in vasion-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying semi-supervised learning

---

Appendix
C Training details
A Baselines
The whole architecture is trained end-to-end with the AdamW (Loshchilov and Hutter, 2017) opti- mizer. The initial learning rate of the model is le-5. The learning rate is gradually warmed up for epochs with a unit multiplier and then de- cayed following a step scheduler with step size of 10 epochs and gamma of 0.95. We use a batch size of eight and weight decay of 0.01. The model is trained for 30 epochs and we choose the best per- forming model based the test set. The model is trained on 4 V100 GPUs with data parallelism. All code and models will be made available at https: / thub. wwt.:/ooo/////////ee
We consider the following baselines to fairly com- pare and evaluate our proposed method:
(a) Text-only CR: For all these methods, we di- rectly evaluate the coreference chains using the narration only without the image. (1) Neural-Coref (Lee et al., 2017): This method is trained end-to- end using a neural network on a large corpus of wikipedia data to detect mentions and coreferences. For this baseline, we use the predicted mentions (2) longdoc (Tosh- niwal et al., );21): This is a strong transformer based method using Longformer-Large as the back- bone for coreference resolution, trained on multiple datasets. We use the gold mentions to predict coref- erence chains for this model.
instead of the gold mentions.
D Further Ablations
Pre-trained weights from ALBEF (Li et al., 2021). In Table 6, we show CR results by replac- ing the text-encoder and the multi-modal encoder in our model from the ALBEF (Li et al., 2021) framework with 6 transformer blocks in each en- coders. Moreover, we use the pre-traiend weights from ALBEF (Li et al., 2021) and then fine-tune using our semi-supervised training strategies. De- spite strong pre-training from ALBEF, the model does not fine-tune tunetune&ae and transfer well to the task of MCR compared to our method which is initialized with BERT weights.
(b) Multi-modal CR: We evaluate strong vision and langauge models for the task on coreference resolution on the CIN dataset. (1) VisualBERT (Su et al., 2019), UNITER (Chen et al., 2020), VInVI. (Zhang el al., 2021b): All these three baselines are strong vision language models trained on image- variety
caption data and shows improvement's of downstream tasks such VQA, NLVR etc. To test it for CR, we compute the cosine similarity for the multi-modal mention embeddings in a zero- shot way. (2) MAF (Wang el al., 2020): MAF is a weakly supervised phrase grounding method, orig- inally trained on the Flickr30k-EEttine
Varying the threshold for pseudo-grounding loss. In Table 7, we compare the results with differ- ent grounding thresholds for the pseudo grounding loss. Including all the pseudo predictions (thresh- old of 0.0) leads to significant drop in perfor- mance showing the importance of thresholding- based training strategy. Furthermore, as presented in the results the threshold of 0.9 works the best compared to 0.5 and 0.7.
(Plummer et al., 2015). Goel et al. (2022) train this model on narrations data and evaluate it for CR. (3) WS- MCR (Goel et al., 2022): This is a strong weakly supervised method for multimodal coreference olution on the CIN dataset (Goel al., 2022). In this method, they train a vision and text encoder with an image-text contrastive loss and weak prior linguistic rules as a regularizer.
Varying the amount of labeled and unlabeled data. In Table 8 and Table 9, we present detailed results from the discussion in Section 5.3 on differ- ent CR metrics by varying the amount of labeled and unlabeled data.
B Bounding box regression loss (BBR)
We define the smmoth-L1 loss (Ren et al., 2015) for the bounding box transformation as follows:
Libr [ 0.5(bm (bm b/, if (bm Ky,l <
B
E Qualitative Results
1mm 1m - 0.5 B, otherwise
(11)
In Figure 4 and Figure 5, present detailed qual- itative visualizations. Figure 4 shows the corefer- ence chains predicted by our proposed method and the baseline, WS-MCR (Goel et al., 2022). The bascline model misses the instances his to re- late to the the man (row 2, column 2) which is
where / is set to following previous work (Gir- shick, 2015; Ren et al., 2015), 6'm are the trans- formed bounding box coordinates after applying the delta transformation 6m on the maximum re- gion proposal r' similar to (Girshick, 2015).

---

Method
MUC
B
CEAF
CoNLL
R
F1
R
F1
R
P
FI
F1
Weights from ALBEF (1. al., 2021)
31,70
25.97
26.47
70.78
78.35
73.77
54.58
89.02
66.94
55.73
Ours (Weights from BERT (Devlin 2018))
31.11
35.25
31.86
70.63
87.85
78.06
63.99
93.44
75.47
61.79
Table 6: CR performance with ALBEF (Li et al., 2021) as pre-trained weights.
Grounding
MUC
B
CEAF
CoNLL
threshold
R
R
P
F1
P
F1
FI
0.0
28.20
22.47
23.08
70.10
76.29
72.40
52.32
87.22
64.71
$3.40
0.5
31.18
30.04
70.80
82.67
75.76
59.68
92.18
71.77
58.82
0.7
30.48
33.34
30.58
70.63
86.74
77.58
63.30
93.22
74.85
61.01
0.9
31.11
35.25
31.86
70.63
87.85
78.06
63.99
93.44
75.47
61.79
Table Performance of our proposed method by varying grounding threshold f include samples above this threshold in Equation (9).
MUC
82
CEAFE
CONLL
2022). Compared to the baseline, our method suc- cessfully grounds entrance door, some other people and two women. Hence our method clearly exhibits strong coreference resolution and grounding capa- bilities compared to previous work.
F1
R
F1
Fi
20%
26.40
31.15
27.26
69.3
58.26
77.75
64.25
92.02
75.11
60.04
34.13
29.1
75.53
73.47
Table 8: CR performance by varying the number of labels in the labeled dataset.
MUC
CEAF,
CoNLL
F1
R
P
F1
Fi
30%
30.6
27 09
23
70.84
79.36
74.23
56.49
00.79
69.04
5683
99%
30.87
79.00 97
28
83
70.25
63.58
76.23
60.24
90.02
73.27
59.11
75.47
Table 9: CR performance by varying the number of labels in the unlabeled dataset.
correctly clustered by our method. Moreover, WS- MCR (Goel et al., 2022) cannot relate big orange color building the building (row2, column 4) like our method. This highlights that our method is able to learn fine-grained correlations between the image regions and text to effectively resolve such ambiguities. Despite significant advantages, our method still fails to resolve cases like some other people by clustering them into the same chain. We believe that the model needs more complex visual understanding (localize different instance of some other people) and contextual knowledge from text (people standing on road people standing on footpath) for these specific cases.
In Figure 5, we show grounding of the corre- sponding mentions on the image from our proposed
method and the baseline WS-MCR (Goel et al.,

---

Ground Trush
mmag we rond, thasing man worting color shirt and black
wo color cloth
the foolpath we see two women sining and tikinng with each other, and in that
not
hand, and the man
pulked.
woman helding bubs, and we also
alko wrnningg orangy
people abss by holding
some other people alo standing fastpaill, and the background
block color belt amond chowl.
color paat.
big orange color building with white color glass windows, and can also saz building.
door
WS-MCR (Goed st. 2002)
the
insque we mund, iuan wostnng past.
soc dansing
we his hand, and the man
gro
bilking
yoate other people abso standing by holding flags
also caring orangs black rolor amonnd
foregeth, background glass onange also entrance
buildies.
Oues
image see dancing us read, westing white color pait.
white red and his and the color black color around chos
parket, we torse athir people also sanding on by holcing flags
sitting aiking witl each otees is that
the feuipath two women
woman helding buln,
also
danding
fosepath, the background
relor huildiigg with white color windows, also
building
Figure 4: Coreference resolution for image-narration pair. We break down the full narration as individual sentences in the columns for simplicity. The rows from top to bottom show ground-truth annotations, predictions from the WS-MCR method (Goel et al., 2022) and Ours. The mentions the same color form a part of the coreference chain.

---

two women are sitting and talking with each other. and in that one woman is holding a baby, and we can also see some other people are also standing on the footpath, and in the buckground we can see big orange color building with white color gloss windows, and we can alko see the entrunce door of the building.
Solnc ooh peop
Ours
on the footpath,
on the footpath we can see two women are sitting and talking with each other, and in that one woman is holding a baby, and we can also see some other people are also standing on the footpath, and in the bac kerogrd we can see a big orange color building with white color glass windows, and we can also see the entrance door of the building.
1311
WS-MCR (Goel ef al., 2022)
Lel
willi cole coler gloss windons
on the footpath we can seC tWo women are sitting and talking with each other, and in that one woman is holding baby, and we can also see some other people are also standing on the footpath, and in the background we can see big orange solor duiiding with white color glass windows, and we can also see the entrance door of the building.
COnPII 2007
af SIV IAY 2011
Eostpallls
Ground Truth
AIT gleso anteni
ailli cooo glos sirdloss
Figure 5: Visualization of Grounding and CR performance. Zoom in for better visualization of bounding boxes.


Reviews:
[{"review_id": "EJsMGJs5i7", "reviewer": "Reviewer_xjtT", "summary": "", "questions": "* Why doesn’t the grounding model use the ground-truth bounding boxes during training, or at least parts of training (L287-289) for g? This could be viewed as some kind of teacher forcing and should be an objective if it is available?\n* L421: Which layers of BERT are used for initialization? BERT has 12 layers but the text encoder only has 4.\n* Without ground truth mention boundaries, Lee et al., 2017 (which is failing to detect mentions) isn’t a fair comparison - I’d suggest removing it from the table altogether as it is more distracting/confusing (feel free to leave a note in the appendix that it was tried but had low scores, possibly because there were no gold mentions).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments which we respond to below.\n\n> The task/dataset is still new/private (I can't find it online) and so there aren't really any established baselines on this dataset. Therefore, strong baselines are quite important.\n\nPrior to the submission deadline, we contacted the authors for a copy of the dataset which is now available at https://github.com/VICO-UoE/CIN along with the accepted ICCV’23 preprint. We agree that this is still a new task and dataset. Hence we included new uni-modal (longdoc) and multi-modal (VisualBERT, UNITER, ALBEF) baselines over the original work as discussed in Table 1.\n\n> In particular, a pipelined approach of fine-tuning a neural coref model on the narrations (as opposed to using them out of the box) since the proposed model also gets to fine-tune on that data.\n\nAs suggested, we now provide a new neural coref baseline for a fair comparison that fine-tunes the Longformer-large model (Toshniwal et al. 2021) only on the labeled set of the CIN dataset. Its weights are initialized with the pre-trained model on other known 8 coref datasets including OntoNotes, PreCo. Surprisingly, it is nontrivial to finetune this large model without overfitting into the small labeled set of the CIN. The CoNLL F1 score for the fine-tuned model is 42.41 which is slightly lower than the score of 45.31 from the pre-trained model (Table 1, 2nd row). This also emphasizes the need for learning from larger unlabeled data. We will add these results to the paper and discuss them.\n\n> then integrating into ALBEF (or even this paper's model) would be a fairer comparison + stronger starting point for CIN.\n\nA shortcoming of finetuning coref models such as Longformer-large that are trained only on text data is their inability to provide associations with the visual input. Hence, finetuning Longformer-large yields lower performance (42.41) than even our self-supervised model with a F1 of 57.19 (Table 4, row 1) which is learned originally on multi-modal dataset.\n\n> Why doesn’t the grounding model use the ground-truth bounding boxes during training, or at least parts of training (L287-289) for g? This could be viewed as some kind of teacher forcing and should be an objective if it is available.\n\nThanks for the valuable suggestion. Due to the short rebuttal time and required computation time, we leave this as future work where we would also like to take this a step further by jointly learning the bounding box prediction for mentions along with the coref.\n\n> L421: Which layers of BERT are used for initialization? BERT has 12 layers but the text encoder only has 4.\n\nAfter careful experimentation, we find that initializing with the first 4 layers of BERT works best. Thanks for raising this and we will clarify this in the final version. \n\n> Without ground truth mention boundaries, Lee et al., 2017 (which is failing to detect mentions) isn’t a fair comparison - I’d suggest removing it from the table altogether as it is more distracting/confusing (feel free to leave a note in the appendix that it was tried but had low scores, possibly because there were no gold mentions).\n\nThanks, we will move this result to the appendix and replace it with a footnote in the final version. \n\n> L588: “The supervised contrastive loss effectives promotes…more distinctive embeddings for the clusters” – is this proven, or just a hypothesis that this is what is happening? This claim feels a bit strong here when the whole system is more complex than just CR loss, but I could be misunderstanding the phrasing in this section.\n\nThanks for pointing this out. This is a hypothesis and we will replace L588 as follows: We hypothesize that the supervised contrastive loss provides a better clustering of mentions by contrasting them in the embedding space directly than the binary cross-entropy loss. \n\n> Eq(1) is just cosine similarity right? It doesn't need an equation and you can save space.\n\nThanks, we will edit this. \n\n\n> Grammar and presentation improvements \n\nThanks for the suggestions, we will update these.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper proposes a new semi-supervised model for vision-and-text for the task for coreference grounding and shows that it outperforms some off-the-shelf several baselines or previously reported work.\n* They thoroughly study the model, along with ablations and additional experiments/analysis to justify the complexity of the model.", "reasons_to_reject": "* The task/dataset is still new/private (I can't find it online) and so there aren't really any established baselines on this dataset. Therefore, strong baselines are quite important. In particular, a pipelined approach of fine-tuning a neural coref model on the narrations (as opposed to using them out of the box) and then integrating into ALBEF (or even this paper's model) would be a fairer comparison + stronger starting point for CIN, since the proposed model also gets to fine-tune on that data.", "questions_to_authors": "* Why doesn’t the grounding model use the ground-truth bounding boxes during training, or at least parts of training (L287-289) for g? This could be viewed as some kind of teacher forcing and should be an objective if it is available?\n* L421: Which layers of BERT are used for initialization? BERT has 12 layers but the text encoder only has 4.\n* Without ground truth mention boundaries, Lee et al., 2017 (which is failing to detect mentions) isn’t a fair comparison - I’d suggest removing it from the table altogether as it is more distracting/confusing (feel free to leave a note in the appendix that it was tried but had low scores, possibly because there were no gold mentions).", "paper_topic_and_main_contributions": "The paper proposes a model for visual coreference given image and narrations. Their semi-supervised model uses a fused text-and-vision transformer with multiple supervised and unsupervised objectives so that they can train on a small set of labeled data and a much larger set of unlabeled data. They demonstrate the effectiveness of this model on the Coreferenced Image Narratives (CIN) dataset for coreference resolution (of the narrations) and narrative grounding (match text cluster to image). Across multiple ablations, they show the effect of each of the training objectives and the pseudo-labels that are created for the unsupervised objectives.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "L452: Yang et al., 2022 is more focused on coref in LLMs, which doesn’t seem like the right citation for the model scale in this work/longdoc; \n[Moving on from OntoNotes: Coreference Resolution Model Transfer](https://aclanthology.org/2021.emnlp-main.425) (Xia & Van Durme, EMNLP 2021) and [Annotating Mentions Alone Enables Efficient Domain Adaptation for Coreference Resolution](https://aclanthology.org/2023.acl-long.588) (Gandhi et al., ACL 2023) would be better citations if you want more.", "Typos_Grammar_Style_and_Presentation_Improvements": "* L237: “where, softmax” -> “where the softmax”\n* L376: This should be “Devlin et al., 2018” (or no citation since it is mentioned in the sentence already). It should definitely not be “Vaswani et al., 2017.”\n* **Please make Table 2 and Table 3 bigger**. Also it is confusing why 50% labeled is not scoring the same as 50% unlabeled? What am I not understanding here/do they not add up to 1?\n* **And all the tables really should be bigger/more readable.** You can mess with the hspace between columns if it means the font for the numbers can be bigger. Or consider moving full results (with MUC/B^3/CEAF) into the appendix and only report CoNLL F1 in the main body (except for Table 1: the subscores are quite interesting there!)\n* L588: “The supervised contrastive loss effectives promotes…more distinctive embeddings for the clusters” – is this proven, or just a hypothesis that this is what is happening? This claim feels a bit strong here when the whole system is more complex than just CR loss, but I could be misunderstanding the phrasing in this section.\n* Eq(1) is just cosine similarity right? It doesn't need an equation and you can save space.\n* Similarly, all the softmax equations can be \\softmax(...) instead of a big fraction (as long as it's clear what the softmax is over)"}}, {"review_id": "0sNX2WULm1", "reviewer": "Reviewer_X5uF", "summary": "", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and acknowledgment of the effectiveness of our method, clear writing, and extensive experiments. We will address any further concerns that may be raised during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper has the following merits:\n1. The paper is solid in terms of methodology. The framework is convincing in theory and the experiments also support their claim.\n2.  It is easy to follow, researchers can implement with scratch by simply reading the paper and the instructions in appendix.\n3. The authors carefully present their experiment sections with numbers, analysis and qualitative examples. ", "reasons_to_reject": "I don't have any particular reasons to reject the paper as I am leaning to give a thumb-up for this paper.", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "In this paper, the authors proposed a framework that tackles coreference resolution across vision (image) and text (narration or caption) as well as within text. This is a semi-supervised framework. The images and texts are fed into visual encoder and text encoder respectively. The text embeddings are fed into a self-attention module which is followed by a cross attention layer that consumes visual embeddings. With an additional feed forward layer, the attention modules form a multimodal encoder. The MM encoders are trained through self learning technique for masked language model (MLM) and image-text contrastive loss. Moreover, with the information from labeled data, the MM encoders are updated with coreference loss, grounding loss and bounding box regression loss; while the unlabeled data provides pseudo coreference loss and pseudo grounding loss. The authors also conduct excessive experiments to compare against the SOTA baselines as well as the ablation settings with varied labeled and unlabeled data and different combinations of loss functions.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "wTRsNG38kO", "reviewer": "Reviewer_B6pm", "summary": "", "questions": "A) What is the reason for using gold data for text mentions but extracting the image regions automatically? Does the image region depend on the gold data or do predicted bounding boxes also work? In general, I think the usage of gold mentions needs to be made explicit early in the text.\n\nB) Is there a reason you did not use LEA as an evaluation metric? While it is not as popular, it is in theory much superior to the other measures, hence I would love to see it used more.\n\nC) Is U1 only calculated for the text narrations?\n\nD) U1: What exactly does h_m refer to?\n\nE) U1: How are the mentions for this loss generated, just any random text span?\n\nF) U1: How is the m embedding for initial positive/negative sampling generated?\n\nG) Line 395 onwards, does this imply a coreference chain can be associated with multiple image regions?\n\nH) Previous work has shown that even just training on domain-specific mentions can yield large improvements (https://aclanthology.org/2023.acl-long.588/). Wouldn't it make sense to test existing coreference models not just in a zero-shot setting but also in some sort of few-shot setup? (This should not be considered a missing reference, as it is a very recent publication)\n\nI)  \"Moreover, it validates\n453 the need for multi-modal models for the task of\n454 multimodal coreference resolution.\" Why is this the case? The low f1 to me just indicates that models are very domain specific.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments which we respond to below.\n\n> Some of the architecture choices are not motivated well, and relatively little investigation into the reason for the positive impact of individual loss functions is provided. This is a problem as improvements brought about by the loss functions are the paper's main contribution. \n\n\nWe would like to emphasize that we comprehensively evaluate the importance of adding each loss term in Table 4, the choice of loss for coreference resolution in Table 5 and a discussion about the importance of losses in Section 5.3. Moroever, we also study the impact of adding unlabeled data during training in Table 3. \n\n\n\n\n> Perhaps this is a lack of experience on my side as I am not very familiar with multi-modal approaches, but the PCR loss does not make sense to me despite the given intuitive explanation. From my understanding, it just incorporates the models existing understanding of distance into the training. The paper clearly shows it still performing well but lacks an exploration of why. My question would be if it essentially acts as encoding a cosine-distance based coreference resolution policy. A few minor changes in the paper may potentially be able to clear this up (see questions).\n\n\nThe PCR loss computes the coreference loss over the unlabeled data. To compute this loss, the model treats its own coreference predictions as pseudo-labels. Hence, to prevent overfitting of the model to the wrong predictions, at each iteration, we only consider highly confident positive and negative predictions (coreference links) from the model in a triplet loss formulation (Eq. 8). This process goes through several iterations, each time adding updated pseudo-labels hence improving the performance of the model. This self-learning paradigm has been shown to be extremely effective in literature as the selection of unlabeled samples with high-confidence predictions moves decision boundaries to low-density regions [1,2].  We will revise the explanation in the paper. \n\n[1]Oymak, Samet, and Talha Cihad Gulcu. \"Statistical and algorithmic insights for semi-supervised learning with self-training.\" arXiv preprint arXiv:2006.11006 (2020).\n\n[2]Rizve, Mamshad Nayeem, Kevin Duarte, Yogesh S. Rawat, and Mubarak Shah. \"In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning.\" arXiv preprint arXiv:2101.06329 (2021).\n\n\n> The authors do not validate their text coreference against very recent models e.g. the word-level coreference model, with the Toshniwal et al model specifically targeting long documents at the expense of accuracy. As such the coreference approach explored here seems to effectively ignore a wide body of coreference work. To be fair this seems to also be the case for the CIN dataset paper (Goel et al, 2022).\n\n\nAs suggested, we now provide a new neural coref baseline for a fair comparison that fine-tunes the Longformer-large model (Toshniwal et al. 2021) only on the labeled set of the CIN dataset. Its weights are initialized with the pre-trained model on other known 8 coref datasets including OntoNotes, PreCo. Surprisingly, it is nontrivial to finetune this large model without overfitting into the small labeled set of the CIN. The CoNLL F1 score for the fine-tuned model is 42.41 which is slightly lower than the score of 45.31 from the pre-trained model (Table 1, 2nd row). This also emphasizes the need for learning from larger unlabeled data. We will add these results to the paper and discuss them.\n\n\n> It is not clear at first sight which methods are trained and which are evaluated in a zero-shot setting. While this is easy to fix I believe that, in the current state, the results are slightly misleading.\n\n\nThanks for pointing this out. We will address this in the final version.\n\n\n> What is the reason for using gold data for text mentions but extracting the image regions automatically? Does the image region depend on the gold data or do predicted bounding boxes also work? In general, I think the usage of gold mentions needs to be made explicit early in the text.\n\n\nAlong with coreference resolution, our objective also involves grounding i.e. predicting a bounding box for a mention. To tackle this, we adopt a two-stage approach. In the first stage, we employ an object detector to extract bounding boxes. Subsequently, the second stage involves learning the matching process with respect to these extracted boxes. These predicted bounding boxes act as inputs during both the training and inference phases. While directly using the ground-truth boxes is possible it is not scalable as they are expensive to obtain, are missing for the unlabeled part of the dataset, and is orthogonal to the objective of learning grounding. In the future, as an extension, we plan to jointly learn the bounding box prediction for mentions along with coref.\n\nFor coreference resolution, during the training process, we utilize mentions extracted from an off-the-shelf NLP parser (https://spacy.io/) as outlined in Goel et al. (2022). However, during the inference stage, we opt for gold mentions, aligning with established practices in previous coreference research. As outlined in the limitations, we will extend our work to predict mentions along with coreference chains. We will ensure that these distinctions are explicitly clarified in the text.\n\n\n> Is there a reason you did not use LEA as an evaluation metric? While it is not as popular, it is in theory much superior to the other measures, hence I would love to see it used more.\n\n\nThanks for the suggestion. We use the evaluation metrics commonly used in the coref literature and as proposed in the original work. As suggested, we present new results on the LEA metric for both the supervised and semi-supervised setup. The results are in the table below: \n\n\n| Ours | CoNLL F1 |  LEA-P  |  LEA-R  |  LEA-F1  |\n| --- | ----------- |  ---  |  ---  |  ---  |\n| Supervised | 58.51 |  42.07  |  60.66  |  49.68  |\n| Final | 61.79 |  58.72  |  70.72  |  64.17  |\n\n\nOur final semi-supervised method surpasses in the LEA metric by significant margins compared to the carefully designed supervised training. We will add these results in the final version.\n\n\n> Is U1 only calculated for the text narrations?\n\nU1 is only calculated for text mentions, where the embeddings for the mentions are the output from the multimodal encoder as given in Eq 3. \n\n\n> U1: What exactly does h_m refer to?\n\nThanks for pointing this out. It is indeed a typo and h_m should be f(m) which is the embeddings from the multi-modal encoder for the query mention m. \n\n\n> U1: How are the mentions for this loss generated, just any random text span?\n\nDuring training, we use the mentions generated from an off-the-shelf NLP parser (https://spacy.io/) following Goel et al. (2022). We will make it clear in the implementation details. \n\n\n> U1: How is the m embedding for initial positive/negative sampling generated?\n\nThe embeddings for the positive and negative mentions are generated using the output from the multimodal encoder as given in Eq 3, similar to the query embedding f(m). \n\n\n> Line 395 onwards, does this imply a coreference chain can be associated with multiple image regions?\n\nA coreference chain always corresponds to a single image region except where there are multiple bounding boxes for the mentions, eg. “three men”, in which case there could be three bounding boxes. We will clarify this in the text. \n\n\n> Previous work has shown that even just training on domain-specific mentions can yield large improvements (https://aclanthology.org/2023.acl-long.588/). Wouldn't it make sense to test existing coreference models not just in a zero-shot setting but also in some sort of few-shot setup? (This should not be considered a missing reference, as it is a very recent publication)\n\nThanks for the reference. Due to the short rebuttal period, we were unable to setup an experiment for the given paper. This is however an interesting direction and we will consider this in future work as this might lead to further improvements when combined with our training regime. \n\n\n> \"Moreover, it validates 453 the need for multi-modal models for the task of 454 multimodal coreference resolution.\" Why is this the case? The low f1 to me just indicates that models are very domain specific.\n\nWe agree that the low F1 is due to the domain specificity of the models. However, we argue that these models require a large labeled dataset to close the domain gap as also corroborated by our new experiment on Longformer-large (as discussed above). As shown in the experiments adding text and image modalities (Table 1, last row) in a semi-supervised framework mitigates this issue in a low data regime, especially when compared to the model with only text modality (Table 1, 2nd last row). Hence, we conclude that our model can be efficiently trained on the combined labeled and unlabeled data in a multi-modal setup.\n\n\n> Grammar and presentation improvements \n\nThanks for the suggestions, we will update these.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper produces state-of-the-art results on a widely applicable task, it is relevant to a wide range of multimodal tasks. It shows clear improvements over reported SotA results. The paper shows a performance increase when adding the image and attributes this to the poor singleton identification of text-only models. Qualitative evaluation is used to showcase and explain the models' strengths.", "reasons_to_reject": "Some of the architecture choices are not motivated well, and relatively little investigation into the reason for the positive impact of individual loss functions is provided. This is a problem as improvements brought about by the loss functions are the paper's main contribution.\nPerhaps this is a lack of experience on my side as I am not very familiar with multi-modal approaches, but the PCR loss does not make sense to me despite the given intuitive explanation. From my understanding, it just incorporates the models existing understanding of distance into the training. The paper clearly shows it still performing well but lacks an exploration of why. My question would be if it essentially acts as encoding a cosine-distance based coreference resolution policy. A few minor changes in the paper may potentially be able to clear this up (see questions).\n\nThe authors do not validate their text coreference against very recent models e.g. the word-level coreference model, with the Toshniwal et al model specifically targeting long documents at the expense of accuracy.  As such the coreference approach explored here seems to effectively ignore a wide body of coreference work. To be fair this seems to also be the case for the CIN dataset paper (Goel et al, 2022).\n\nIt is not clear at first sight which methods are trained and which are evaluated in a zero-shot setting. While this is easy to fix I believe that, in the current state, the results are slightly misleading.\n\nFor me, there were a few cases in which lack of detail impacted my understanding (e.g. PCR loss but also 395-397)", "questions_to_authors": "A) What is the reason for using gold data for text mentions but extracting the image regions automatically? Does the image region depend on the gold data or do predicted bounding boxes also work? In general, I think the usage of gold mentions needs to be made explicit early in the text.\n\nB) Is there a reason you did not use LEA as an evaluation metric? While it is not as popular, it is in theory much superior to the other measures, hence I would love to see it used more.\n\nC) Is U1 only calculated for the text narrations?\n\nD) U1: What exactly does h_m refer to?\n\nE) U1: How are the mentions for this loss generated, just any random text span?\n\nF) U1: How is the m embedding for initial positive/negative sampling generated?\n\nG) Line 395 onwards, does this imply a coreference chain can be associated with multiple image regions?\n\nH) Previous work has shown that even just training on domain-specific mentions can yield large improvements (https://aclanthology.org/2023.acl-long.588/). Wouldn't it make sense to test existing coreference models not just in a zero-shot setting but also in some sort of few-shot setup? (This should not be considered a missing reference, as it is a very recent publication)\n\nI)  \"Moreover, it validates\n453 the need for multi-modal models for the task of\n454 multimodal coreference resolution.\" Why is this the case? The low f1 to me just indicates that models are very domain specific.", "paper_topic_and_main_contributions": "The paper introduces a semi-supervised cross-modal coreference resolution model. In addition to linking mentions in a text, the mention clusters are also connected to regions in images that are described in the respective text. The training makes use of a variety of loss functions some supervised, some unsupervised. The presented approach archives state-of-the-art scores for the task on the CIN dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please consider using booktabs for your tables (especially using \\cmidrule for grouping columns, e.g. P, R, and F1 for any of the metrics), the interrupted vertical lines are a bit visually distracting for me, please consider removing them. In my opinion, this would help the visual presentation of the paper immensely.\n\n345: is \"impute\" the correct verb here?\n\n\"Table\" as in \"Table 3\" (e.g. line 535) are typically capitalized, same for section.\n\nPlease add a label indicating unsupervised losses in Table 4, it is hard to keep the abbreviations in mind at all times."}}]

Author rebuttal:

Meta-review: All reviewers felt that the work was both sound and exciting. In particular, there was general agreement that the proposed model, and thorough experiments, show a convincing improvement over strong results from previous work. Reviewers had a few points about baselines and clarity of motivation, which the author response addressed satisfactorily.
Decision: Accept-Main