Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization
Authors: Cennet Oguz
Pascal Denis
Emmanuel Vincent
Simon Ostermann
Josef van Genabith
Venue: EMNLP 2023 Main (2023)
Keywords: Anaphora Resolution
Object Localization

Abstract: In multimodal understanding tasks, visual and linguistic ambiguities can arise. Visual ambiguity can occur when visual objects require a model to ground a referring expression in a video without strong supervision, while linguistic ambiguity can occur from changes in entities in action flows. As an example from the cooking domain, "oil" mixed with "salt" and "pepper" could later be referred to as a "mixture". Without a clear visual-linguistic alignment, we cannot know which among several objects shown is referred to by the language expression “mixture”, and without resolved antecedents, we cannot pinpoint what the mixture is. We define this chicken-and-egg problem as Visual-linguistic Ambiguity. In this paper, we present Find2Find, a joint anaphora resolution and object localization dataset targeting the problem of \emph{visual-linguistic ambiguity}, consisting of 500 anaphora-annotated recipes with corresponding videos. We present experimental results of a novel end-to-end joint multitask learning framework for Find2Find that fuses visual and textual information and shows improvements both for anaphora resolution and object localization with one joint model in multitask learning, as compared to a strong single-task baseline.

Find2Find:
Multitask Learning for Anaphora Resolution and Object Localization
Cennet Oguz', Pascal Denis?, Emmanuel Vincent
Simon Ostermann', and Josef van Genabith
German Research Center for Artificial Intelligence (DFKI), Saarland Informatics
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000 Lille, France Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France [cenett. oguz, s simon, w.w..oserennn.. josef, van ran_genabithh @dfki. de ppscaal.. denis, emmanuel. vincent (iiriii..............eee.eeeeiie fr
Abstract
In multimodal understanding tasks, visual and linguistic ambiguities can arise. Visual ambi- guity can occur when visual objects require a model to ground referring expression in a video without strong supervision, while lin- guistic ambiguity can occur from changes in entities in action flows. As example from the cooking domain, "oil" mixed with "salt" and "pepper" could later be referred to "mix- ture". Without a clear risual-linguisti align- ment, we cannot know which among several objects shown is referred to by the language expression "mixture", and without resolved an- tecedents, we cannot pinpoint what the mixture is. We define this chicken-and-egs problem as Hisual-linguiss we present Find2Find, a joint anaphora resolu- tion and object localization dataset targeting the problem of Aisual-linguise
add bociima
Kast Bapas
bxiokongla bellad
Figure 1: Examples of visual and linguistic ambigui-
ties. Figure 1) represents the visual ambiguity related which specific pan (in Figure la) is referenced with the phrase the pan because many pans occur on the stove. Figure 2) shows the linguistic ambiguity with the use of the pronoun them (in Figure 2b).
ambiguity. In this paper,
ambiguity, consist- ing of 500 anaphora-ami aanootae- recipes with cor- responding videos. We present experimental re- sults a novel end-to-end joint multitask learn- ing framework for Find2Find that fuses visual and textual information and shows improve- ments both for anaphora resolution and object localization as compared strong single-task baseline.
joint text-video embeddings in multimodal under- standing (Zhou et al., 2018b; Miech et al., 2019). They often contain narration explaining the vi- sual content of the corresponding time frames in the video (Malmaud et al., 2015; Alayrac al., 2016; Zhou et al., 2018b; Miech et al., 2019; Zhukov et al., 2019).
Instructional videos often contain visual and lin- guistic ambiguities that can be easily resolved by humans but pose two unique key challenges for automatic text-image processing systems. The first challenge is visual ambiguity, occurring when it is necessary to ground a referring expression in an image with ambiguous visual referents. In Figure (Ib), it is not clear from the picture which pan is referred to with the noun phrase the pan without a correct bounding box annotation. The second key challenge is linguistic ambiguity, instantiated for example by the use of anaphoric pronouns (Figure 1, (2b) null pronouns (Figure 2).
1 Introduction
Deep neural networks have achieved enormous suc- cess in various language and computer vision tasks, such as multimodal understanding using video-text and image-text data (Malmaud et al., 2015; Alayrac et al., 2016; Zhou et al., 20;8;; Miech et al., 2019; Zhukov et al., 2019). However, many current sys- tems require a large number of accurate annota- tions, including image-level labels, location-leve labels (bounding boxes and key points), and pixel- level labels.
A specific type of video data with naturally oc- curring semi-aligned texts are narrated instructional videos. Such videos are available in large quantities (e.g. on YouTube) and often chosen for learning
In this work, we focus on modeling cases where both ambiguities are intertwined, a phenomenon we refer to as visual- linguistii ambiguity. Figure

---

add choppet entore and comeett to
garli paz.
An example to display how visual- lisuul-iiguuisii ambiguity occurs with a zero anaphor. The zero anaphor [o] refers to two previous instructions as shown. The entities are aligned to the object with the arrows and the color codes.
Figure 2:
provides a motivating example: To find which pan in (lb) is denoted by the textual span the pan, we need to find its antecedent in the preceding text. The visual object localization of the textual antecedent the hot pan in (Ia) then includes sup- plementary visual information about the correct pan: The referent is the hot pan with beef in it. In Figure 1, To find the correct location of the object referred as them (2b), we first need find the textual antecedent to understand what them refers to. When identifying pierogi as antecedent, we can use the pierogi for the visual object localization of them.
(Oguz et al., 2022). Our new data set contains 500 recipes with annotated anaphora and associated ob- ject localization. Together with the new data set, we propose the new task of multimodal resolution of Visual-Lingiistie Ambiguities. The task provides unique opportunity for models to fuse text and vision information for solving anaphora resolution and object localization the same time. Second, we present a new multitask learning system' ! for modeling the two tasks of anaphora resolution and object localization jointly, using a fusion of visual and textual data. Our experiments show that infor- mation from each of the tasks mutually benefits performance on the other task. Our idea is based the fact that in both tasks, the goal is to extract mentions from given text: what connects object localization and anaphora resolution is that in vi- sual object localization, a corresponding language expression needs to be found, and anaphora res- olution, accurate spans that resolve the anaphoric relations between the anaphor and the antecedents need to be identified.
An even complex case is null pronouns (Figure 2): To do object localization and anaphora resolution, the first requirement is the detection of the null pronoun, which then needs to be resolved and located in the image. located visual object of null pronoun then assists in finding the textual antecedent, and a resolved null pronoun helps to apply object localization.
Our guiding idea this work is that anaphora resolution, the task of connecting linguistic expres- sions such as the anaphor (i.e., the repeated refer- ence) and its antecedent (i.e., the previous mention in the document). (Poesio et al., 2018; Fang et al., 2022: Oguz et al., 2022), and object localization, the task of identifying the location of one more objects in image and drawing bounding boxes around their visual space (Tompson al., 2015; Zhou et al., 2016; Choe et al., 2020), can jointly help to resolve iisual-linguisee ambiguitics. To test this we propose a multitask learning neural model for jointly resolving visual- -ingal-linguistic ambi- guity.
Related Work
2.1 Anaphora Resolution
Anaphora resolution (Poesio et al., 2018: Fang et al., 2022: Oguz et al., 2022) is the process of resolving the relations between an anaphor (i.c., a reference expression) and its antecedent (i.c., the previous mention of the same entity). The rela- tions between anaphor and antecedent mostly ap- pear in two different forms: coreference and bridg- ing. Coreference resolution (Clark and Manning, 2016a; Lee et al., 2017) the task of finding the lin- guistic expressions that refer the same real-world entities in a document, whereas bridging resolution (Yu and Poesio, 2020; Kobayashi et al., 2022) fo-
Our contributions are two-fold: First, we present a new ! (Section 4), Find2Find, for the joint evaluation of anaphora resolution and object local- ization based on an extension of Chop& Change
http://giitih.commmqguuCComeeeee

---

cuses on entities with an associative relation that does not express the same entity but relates (e.g., a car and its engine). Most previous work and datasets (Yu et al., 2022) tackle coreference resolution and bridging resolution separately. An exception is Fang et al. (2021), Fang et al. (2022), and Oguz et al. (2022), which focus on documents with rich anaphoric relations for anaphora anno- tation and resolution of coreference and bridging with end-to-end neural networks (Lee al., 2017). Anaphora resolution is composed of two sub- tasks: mention detection and antecedent selection. A typical neural-based method for anaphora olution starts initially with neural-based mention- ranking modeling (Clark and Manning, 2016a), us- ing distributional features of entities (Clark and Manning, 2016b) with predefined mentions. Lee et al. (2017) combine mention detection and an- tecedent selection in an end-to-end neural learning system. Yu and Poesio (2020) propose multi-task learning system for coreference and bridging reso- lution with an end-to-end learning approach (Lee al., 2017). Here, coreference and bridging reso- lution models learn the mention detection with the objective of coreference/bridging resolution: the span is resolved then is a mention.
sweet potato
a person is showing a sweet potato and one big tomato
(c)
Figure 3: Different examples of object localization. The dashed lines in (b) and (c) represent the occurrence of annotation for only the test data whereas the straight lines (a) indicate that annotation is present for train and test.
Multiple Instance Learning (MIL). MIL (Di- etterich et al., 1997) is a learning strategy that adresses the essence of the incomplete annota- tion problem, where only coarse-grained labels are available for learning. MIL has been used effec- tively for weakly-supervised learning in several computer vision works including object tracking (Babenko et al., 2010), object localization (Gok- berk Cinbis et al., 2014), image classification (Wu et al., 2015). Huang et al. (2018) extend MIL ref- erence awareness for visual grounding of instruc- tional videos. We extend MIL for object localiza- tion with anaphoric information to avoid the issue of ambiguous language references such it, them (Zhou al., 2a; Huang et al., 2018).
Various feature sets are used for anaphora reso- lution along with contextualized language features (Joshi et al., 2019, 2020), e.g., the token length of spans (Clark and Manning, 2016b), context- dependent boundary representations with a head- finding attention mechanism over the span (T.ee et al., 2017), distance features of the anaphor and the antecedent based word distance (Clark and Manning, 2016a) and sentence distance (Oguz al., 2022) where additionally visual features are used for anaphora resolution recipes for cook- ing videos. Our work similar in spirit: however, unlike Oguz et al. (2022) we combine anaphora resolution with object localization tasks from com- puter vision in a joint multitask learning model to benefit both leveraging visual features of entities for anaphora resolution.
Weakly Supervised Object Localization The most common way to do object localization is supervised learning, which uses object-level cat- egories with bounding boxes, e.g., Figure 3 a. However, for a data-greedy neural learning sys- tem, object-level bounding box annotation is time- consuming and expensive. Weakly supervised ob- ject localization approaches avoid this problem and focus learning object localization with image- level object labels (e.g., Figure b) under the MII. paradigm (Deselaers et al., 2012; Prest et al., 2012; Gokberk Cinbis et al., 2014; Oquab et al., 2015).
2.2 Object Localization
Object localization is the task of identifying the location of one or more objects in image, e.g. Figure (a.b.c). Object localization has been stud- ied in computer vision for a long time with vari- ous learning methods (Gokberk Cinbis et al., 2014; Zhou et al., 2018 8a; Huang et al., 2018).

---

Object localization with the MIL approach aims match object labels with object bounding boxes (e.g., Figure b). Studies on object localization with image descriptions rather than object labels take weak supervision further (Karpathy and Fei- Fei, 2015; Zhou et al., 201 8a; Huang et al., 2018), in Figure c. For example, Huang et al. (2018) propose extend object localization to localization based on context-dependent referring expressions, and (Kuo al., 2022) offers model for object localization by using a wide range referring expressions, localization or detection queries for zero, one, or multiple objects. Another challenge for object localization with image de- scriptions is the automatic extraction of object la- bels from image descriptions. Zhou et al. (201 8a) extract object labels manually for training and test- ing object localization models whereas Huang et al. (2018) apply the pre-trained Stanford CoreNLP parser (Manning et al., 2014) for entity detection.
Train
Test
Entity
9,316
2,842
Null Pronoun
1,002
282
Pronoun
314
129
Noun Phrases
8,000
2,431
Instruction 4,633
1,422
Recipe
400
100
Table I: Annotated Data Statistics
3.1 Anaphora Resolution
The task anaphora resolution to assign each gold span (anaphor) C₁ where e E (.i.1,...., T4,1) of each instruction I to one or more gold spans (an- tecedent) Y/ E (1.,1 Ii-1, c1,1,-.., .. 1, dummy antecedent tions I1,..., 4,-1 and all preceding gold spans e1,1 01,1 , 1,-1.n from the previous instructions 1,,..., /-1- For nominal span in Figure 1b, the anaphor span the pan refers the antecedent a hot pan in a previous instruction. For a null pronoums example in Figure 2, the null pronouns & refers to two previous instructions as the antecedents be- cause the null pronoun does not point to any entity and it is also not new entity for the recipe, it is instead produced by the previous instructions.
all preceding instruc-
To date, object localization studies have not ex- plored learning from to learn of mention extrac- tion from image descriptions, instead extracting the mentions by using parsing methods (Kiddon et al., 2015; Huang et al., 2017, 2018) or using the predefined mentions list (Zhou et al., 2018s (0) before the learning process. Thus, we claim object local- ization and anaphora resolution share subtask of entity extraction like the mention detection process in anaphora resolution explained in Section 2.1
The selection of dummy as antecedent indi- cates that the anaphor is an incorrect sequence of consecutive words or singular entity without an antecedent.
3 Task
3.2 Object Localization
A recipe consists of instructions I where cooking instruction I; (e.g. add chopped onions and car- rots to the pan) consists of nominal or null spans and one verbal predicate where 1. A span x; of I might be an incorrect consecutive fragment add chopped
An image V E HxH is represented as a bag of ten regions (e.g., the visualization of region proposals of the given positive frame in Figure 4), if fiw "1, -. with suitable side lengths of Il and
hxw
window patches
and where w W. Given a span H; and region propos- als ti hxw ,.., ",, hxw is to identify whether or not the region proposal belongs to the object of interest, i.e., the text x;; when 2; is gold span (e.g., null pronoun [Ø] or nominal spans) of the corresponding instruction 7,- Therefore, the task is to link the text span x; to a corresponding visual region proposal vy where is gold span fer, ei,,) } of the instruction. We use the dot-product attention between the span s; and the region vú for ranking the visual-semante matching.
gold span e, e.g., noun phrase chopped onions, a pronoun it, or null pronoun [db as in mix [o]. Null pronouns are extremely common in recipe instructions (Kiddon et al., 2015; Huang et al., 2017). In our approach, we also have video clip that contains the visual content of the instruc- tion 1, with the action and the entities included in the process. Following Zhou al. (2018a); Huang al. (2018), we evenly divide each video clip into three equal parts and randomly sample one image (one frame) Vú from each of the three sub-clips to capture the temporal changes of entities.
the task object localization

---

with
spin wirghed reples
FENN Txcaliation
ReginCUPP
Auericc
Stack
reghn pupposh
Weally Sepervioed Obfert Tocalintien (WSOF)
pobdic
ITIN Angdars Black
SpeciERT
Merion Dewectes
ipa vockes
the PeIEpEoOOL.O Doc
Presiin
Monticn Estraction Regiosstafiee
Figure 4: The architecture of the multitask learning framework of anaphora resolution and object localization.
4 Data
vided textual descriptions (i.e., instructions). After deep examination of the Findlt dataset, we find that 30 videos of Findlt could be used with our annotated recipes. Since only 30 videos produce an insufficient test set for the evaluation of object localization, annotated 70 more videos from the recipes annotated for anaphora resolution. We extract the frames of the instruction clips by using the temporal boundaries. We obtain video subsets consecutive frames of the instruction to acquire the state changes of entities in time. We pick one frame with a clear visual content of each entity from the 3 subsets. Thus, cach entity is rep- resented in maximum of three frames for state changes. For evaluation, object with bounding box on the selected frames for each entity. total, we have 5,688 images for 100 recipes annotated for our object localization test set. Note that we do not annotate bounding boxes for the training data as we train our models in weakly supervised setting.
Anaphora Resolution Data. The YouCookII dataset (Zhou et al., 2018a,b) includes manually provided descriptions (i.e., instructions) of ac- tions with the corresponding temporal boundaries (i.e. start and end timestamps) 2,000 cooking videos. The videos provide visual input of the cor- responding objects to observe changes of the ob- jects clearly. Oguz al. (2022) use the YouCookll dataset to propose multimodal anaphora resolu- tion dataset, Chop&Change, with a novel annota- tion schema address the state change of entities in cooking instructions of recipe videos. While the Chop&Change unnotation schema captures three anaphoric relations (coreference, near-idente ide-iidnn,ty, and bridging), in our work here we concentrate on anaphora resolution ignoring relations and focusing on finding the antecedent. Table shows that the Chop&Change dataset includes 264 training recipe documents and 89 test documents in total. For our work here, we increase the number of annotated recipes to 400 for train and 100 for test recipes by using the Chop&Change annotation schema. All annotated recipes are associated with respective videos. The structure of annotation is explained with an example in Appendix A.
annotate the visual
Methodology
5.1 Model
In this section, we explain the details of our anaphora resolution and object localization models. Additionally, we formulate the multitask learning approach of joint anaphora resolution and object localization (see Figure 4). In order to analyze a given text, it is important resolve referring ex- pressions. Thus, text-based anaphora resolution is an important method identifying the antecedent
Object Localization Data. To construct our test set we examine Huang al. (2018) who present a study on reference-aware visual grounding and provide an object localization dataset, Findlt, of 62 IouCookII videos for the given entities in the pro-

---

of anaphora. However, here we also have object lo- calization, providing bounding box for the visual content of language references. The tasks of object localization and anaphora resolution share informa- tion via the mention extraction and representation part of the model. Therefore. our mention repre- sentations are trained with anaphora resolution and object localization over the mention extraction and representation. The implementation details can be seen in Appendix B.
descriptions of pictures make frequent references to objects in the pictures and their attributes. How- ever, the references are not always clearly defined. Particularly in a video, the use of pronouns and el- lipses are extremely common (Kiddon et al., 2015; Huang et al., 2017). Our object localization model follows a Weakly Supervised Object Localization (WSOL) strategy (Huang et al., 2018; Zhou et al., 201 8a; Choe et al., 2020): Only full image descrip- tions (Figure 3 (c)) are used for localization instead of a specific object-level label (as in bounding box and specific label pairs for each object in a picture like in Figure 3 (a)).
As implied in the explanation of both tasks, be- fore resolving references in the previous context or on the image, referring expressions and language references in the text need to be identified, task known as Mention Extraction. In the next Section, we first describe our approach to representing men- tions.
Following the object region and text ranking ap- proach, we formulate the task of WSOL as map- ping the frame region e; to the text span z;. We use positive and negative frames where positive frames come from the video clip V of instruction 1 whereas negative frames are drawn from other videos without shared entities. We define pos; to be a region vector set that includes the region propos- als Axw ..... ... neg, to contain the region proposals Axw .... '10 hxw from the negative image, e.g., the negative and positive frames in Figure 4.
5.1.1 Mention Extraction and Representation We consider all continuous token sequences with up to L words as potential mention span and compute the corresponding span score. We use SpanBERT (Joshi et al., 2020) as a state-of-art rep- resentation for coreference resolution. We capture the linguistic dependencies between anaphor and antecedent in a recipe document by exploiting self- attention: we define SPANBERT(us) (...... WT) to be the SpanBERT representation of recipe, where to is the first token and WT refers to the last token of the recipe. SpanBERT captures the long-range dependencies between the antecedent and anaphor in the recipes. A span 2; consists of zero or more to- kens of instruction 4. We use the verb as pointer for null pronouns. For example, mix is the token of the null pronoun & in the instruction mix [o] (Fig- ure 2). The vector representation I: of a given span 24 is obtained by concatenating the contextualized SpanBERT word vectors of its boundary tokens and its width feature:
Axw from the positive image V,, and
The aim of WSOL to produce a scoring func- tion to maximize the joint probability of positive frame regions V/ E pos; and minimize the joint probability of negative regions r; C neg; with the text span I;. We concatenate the mention repre- sentation vector 9i of span I; and the visual ob- ject region vector r; of the region & to obtain the WSOL input, [gt, ri], effectively fusing tex- tual and visual information in one input. We then prepare positive (FFNN r;) = 1) and negative (FFNN r:) = 0) samples to train a model with iitention-basedd deep MIL (Ilse et al., 2018). Posi- tive examples depict exactly the action in question. whereas negative examples correspond to one of four special cases as described below:
9: ( "E 6(i]
d(i) WIDTH(END(i)- (T)RT(i)).
0 Ft Vr,
START(() and END( represent the starting and ending token indexes for 9s, respectively. d(i) is
2: & (ei Cin), Vr:
FFNN rt)
0 (cr es, neg,
the width feature of the span where WIDTH(() is the embedding function of the predefined bins of [1, 3, 4, 8, 16) as defined by Clark and Manning 6b01
xi E (e....... 6 ri E pos;
Our localization model uses the span representation 9s that is extracted by the mention extraction and ten positive, i.c., pos,, and ten negatives, i.c., neg, region representation vectors r; to learn the best region from pos; for the given span gi- Thus, our mention detection model learns the span vector
5.1.2 Weakly Supervised Object Localization Following prior work (Karpathy and Fei-Fei, 2015; Huang et al., 2017, 2018), we observe that sentence

---

also based on the object localization objective (see
6.1.2 Frame Regions of Instructions
also Figure 4). We define the label of FFNN rz)
Image regions can be proposed by either off-the- shelf object localizers, e.g., region proposal net- works (Girshick al., 2014; Ren al., 2015), or dense sliding windows (c.g., random regions). Re- gion proposal networks and dense sliding window methods neglect language semantics. Therefore, we leverage the state-of-art method RegionCIIIP (Zhong et al., 2022) which uses a CLIP model (Rad- ford et al., 2021) match image regions with tem- plate captions align these region-text pairs in the feature space. We select the first 20 region pro- posals of each entity with the highest objectness scores.
as when the span z; a gold span {G_,1.... Cin) and the region vector r; represent the positive re- gions pos;-
5.1.3 Anaphora Resolution
Following Lee et al. (2017) and Oguz et al. (2022), we implement our anaphora resolution system as an end-to-end system with mention detection but now extended by object localization. For anaphora resolution, the representation of span pair 9ıj is obtained by concatenating the two span embed- dings [gt. Uj] and their clement-wise multiplication, g: 9j. among others:
6.2 Experiments
gij - 1gt Mj. 95 95s &diet (1..))
6.2.1 Anaphora Resolution
where the feature vector (dist(i, is the distance DISTANCE(START()) (T)RT(()) between the in- dex of the instruction span and span j. DIS- TANCH(-) is an embedding function of the prede- fined bins [1,2,3... 3.., 30) as in Oguz et al. (2022). We use softmax (FFtNax(FNN(ggg)) to score the resolu- tion for anaphor and antecedent gj pairs.
Candidate and Gold Spans. Without any prun- ing, we consider all continuous token sequences (Clark and Manning. 2016b; Lee et al., 2017) potential spans for anaphor/antecedent candidates for the training and testing phases. Additionally, we consider gold spans for the training and testing phases in order to investigate the performance anaphora resolution models without mention detce- tion noise.
5.2 Evaluation
Following Hou et al. (2018) and Yu and Poesio (2020), we assess the performance our end-to- end anaphora resolution with the F1-score where precision is the result of dividing the number of correctly predicted pairs by the total number of predicted pairs and recall is computed by dividing the number of correctly predicted pairs by the total number of gold pairs.
With and without Object Localization. To understand the effect of object localization on anaphora resolution we examine the anaphora reso- lution results with and without object localization. Here, we remove the object localization model and data from the training and testing phases of our anaphora resolution model.
6.2.2 Object Localization
To evaluate object localization, we follow prior work (Fukui et al., 2016; Rohrbach et al., 2016; Huang et al., 2018) and compute accuracy as the ratio of phrases for which the predicted bounding box overlaps with the ground-truth by more than 0.5 Intersection-over-Uiie Union (IoU).
With and without Anaphora Resolution. We investigate the object localization performance with and without anaphora resolution to understand the impact of anaphora resolution. We perform four different experiments for object localization.
Random (Huang et al., 2018): Since we use the object region proposal with the highest objectness scores with the given ten region proposals, we first examine random selection with the highest object- ness score to show the complexity of object local- ization.
6 Experimental Setup
6.1 Input
6.1.1 Cooking Instructions.
To encode the recipes use spanBERT (Joshi al., 2020), transformer model designed to better represent and predict spans of text. We use the concatenation of the boundary tokens to represent each span (Clark and Manning, 2016a,b; Lee et al., 2017; Kobayashi et al., 2022).
Deep Visual-Semantic Alignment (DVSA) (Karpathy and Fei-Fei, 2015): This weakly supervised visual grounding method based on image-based regions and given gold and candidate

---

(d)
pour the dressing to jar
spread to another piece of bread
place [8] in plate
rinse [0]
The examples of object localization results of random with the green bounding box, DVSA with the blue bounding box, and our multitask learning method with the red bounding box. The task is the object localization of the gold mention bold font the instructions.
Figure 5:
Methods
Nominal
Null
lution and object localization for both candidate and gold spans. The difference between the re- sults of candidate and gold spans demonstrates that the mention extraction model propagates errors to anaphora resolution and object localization (the sequential structure of model see Figure 4). For example, we have the candidate spans such as all n-grams words (bigrams words such as cook the, the bacon, bacon fat) of an instruction cook the bacon fat with the same visual features with bacon and fat in a pan. Thus, an incorrect mention of mention detection directly causes an error in object localization and anaphora resolution.
Random
13.98
16.66
14.07
DVSA w Gold Mentions
19.90
19.90
AR Cand. Mentions
21.02
24.46
20.79
AR Gold Mentions
21.17
25.98
22.36
Table 2:
The Top-1 results of object localization with gold and candidate mentions. The column group named Nominal shows the results object localization for nominal phrases, Null depicts the results of null pro- nouns, and All refers to the full dataset.
mentions without anaphoric information, which uses multiple-instame learning. Thus, we consider DVSA as a baseline for weakly supervised object localization to our method without anaphora resolution. We examine the results of DVSA with gold and candidate mentions.
7.2 Weakly Supervised Object Localization
Overall, we observe an improvement in the perfor- mance of object localization with multitask learn- ing shown in Table 2. Table 2 shows that our joint multitask learning method outperforms DVSA even with candidate mention. Note that DVSA is not the component that is responsible for extracting the mentions for object localization. Thus, we do not analyze object localization for null pronouns and candidate mentions because DVSA ranks the lan- guage expression and region pairs and does not apply mention extraction. However, we attached the DVSA method for ranking the gold nominal mentions with the given region proposals.
AR-Cand. Mentions We train and test the ob- ject localization model with all continuous tokens (Clark and Manning, 2016b; Lee al., 2017) as potential spans for anaphor/antecedeet candidates. Candidate spans also increase the noise in the ob- ject label set because we consider possible spans as labels of the objects.
AR-Gold Mentions We show the results of ob- ject localization when we use gold mentions and anaphor information for training and testing.
The results of object localization with null pro- nouns clearly demonstrate the benefits of anaphora resolution for object localization. For example, our model localizes the singular mention jar in Fig- ure (a), and the anaphor another piece of bread in Figure (b) better than other methods. Thanks to the mention detection and representation of our multitask learning approach, our object localization model is capable of localization of null pronouns, e.g., Figure 5 (c,d) as our model learns to represent the null pronouns the mention detection process as a mention. Thus, the results of null pronouns
7 Results and Discussion
7.1 Overview
We investigate the anaphora resolution and object localization results of gold and candidate spans comparing the F1-scores and Top-1 scores with multitask learning using both tasks and single tasks. Overall, our results in Table demonstrate that replacing single-task learning with our multitask joint learning approach improves anaphora reso-

---

Nominal Anaphora Res.
Zero Anaphora Res.
Anaphora Res.
Methods
Precision
Recall
Fl-score
Precision
Recall
Fl Fl-score
Precision
Recall
Fl-score
w/o Object Loc.
Cand. Mentions
54.76
46.65
50.38
73.38
68.76
71.00
63.03
54.06
58.20
Gold Mentions
58.76
52.25
55.31
75.38
71.18
73.22
64.16
58.15
61.01
w Object Loc.
Cand. Mentions
52.03
50.49
51.25
77.68
69.97
73.63
62.46
56.19
59.16
Gold Mentions
58.24
55,43
56.80
80.10
76.02
78.01
64.92
61.93
63.39
Table 3: Results of the anaphora resolution with and without object localization for gold and candidate mentions. We show the results for the full test datasets Anaphora Res. columns, for only null pronouns 7ero Anaphora Res., and the resolution results of anaphoric mentions for all nominal phrases in Nominal Anaphora Res. part.
clearly evidence the contribution of anaphora reso- lution for object localization.
dataset for object localization and anaphora reso- lution with cooking recipes to improve the perfor- mance of both tasks by exploiting their commonali- ties. We implement a model for the joint learning of anaphora resolution and object localization, fusing visual and textual information, and show empiri- cally that multitask learning paradígm mutually improves both tasks. Especially, the results of zero anaphora resolution indicate that object localiza- tion helps to avoid linguistic ambiguity of null pro- nouns. Our joint multitask learning approach does not apply the temporal features of evolving visual entities even though the temporal features of tex- tual recipes (Oguz et al., 2022) are included due to instruction order. In future work, claim the temporal encoding of visual objects improve the results of joint anaphora resolution and object localization.
7.3 Anaphora Resolution
Multitask learning of anaphora resolution and object localization increases the performance of anaphora resolution. Table 3 shows >> 2% improve- ments for anaphora resolution with gold mentions and more than 1% for candidate mentions with combined training and testing. When nominal and investigated separately, the re- sults of nominal anaphora demonstrate an improve- ment on the results of gold mentions when object localization is included. Additionally, the results of zero anaphora show significant improvements with multitask learning for gold and candidate mentions.
zero anaphora
Thus, we observe a big part of the improvement for the combined experiments with object localization comes from the zero anaphora resolution samples. The benefit of object localization on anaphora reso- lution is also seen in candidate nominal mentions; it is however not as significant as gold nominal mentions, as the errors of mention detection for candidate spans directly cause localization as well anaphora resolution errors.
Acknowledgements
We would like to thank Gokul Srinivasagan and David Meier for helping with the annotation of the object localization, Alina Leippert for help- ing with the annotation the anaphora resolution. This research was funded by the joint IMPRESS (0 (011S20076) project between the French National Institute for Research in Digital Science and Tech- nology (Inria) and the German Research Center for Artificial Intelligence (DFKI).
The most common error in anaphora resolution is find the closest antecedent in the entity chain. For example, for tomato-> -> tomato, the first tomato is the antecedent of it, and tecedent of the second tomato. However, the model fails to select the first tomato as the antecedent for the second one.
the an-
Limitations
The first and most important limitation is the pre-trained region proposal network (RegionCLIP (Zhong et al., 2022)). Our object localization is highly dependent on the quality of the region pro- posals. Therefore, better region proposal net- work delivers more improvements for our multitask learning.
8 Conclusion and Future Work
In this work, we study the problem of visual- linguistic ambiguity multimodal data and pro- pose the novel task of joint anaphora resolution and object localization, We create the Find2Find

---

References
2016. Multimodal compact bilinear pooling for vi- sual question answering and visual grounding. arXiv preprint arXiv: (eee::60660000 078407787.
Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien, 2016. Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4575-4583.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Ji- tendra Malik. 2014. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587.
Boris Babenko, Ming- Ming-Hsuan Yang, and Serge Belongie. 2010. Robust object tracking with online multiple instance learning. IEEE transactions on pattern anal- ysis and machine intelligence, 33(8):661-116)
Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid. 2014. Multi-fold mil training for weakly supervised object localization. the IEEE conference on computer vision and pattern recognition, pages 9409-2416
Proceedings of
Junsuk Choe, Seong Joon Oh, Seungho Lee, Sanghyuk Chun, Zeynep Akata, and Hyumjung Shim. 2020. Evaluating weakly supervised object localization methods right. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR).
Yufang Hou, Katja Markert, and Michael Strube. 2018. Unrestricted bridging resolution. Computational Lin- guistics, 44(2):237-284
De-An Huang, Shyamal Buch, Lucio Dery, Animesh Garg, Li Fei-Fei, and Juan Carlos Niebles. 2018. Finding "it": Weakly visual grounding in instructional videos. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR).
Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking corefer- ence models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, pages 2256-22666662222222222 Austin, Texas. Association for Computational Linguistics.
reference-aware
De-An Huang, Joseph J Lim, Li Fei-Fei, and Juan Car- los Niebles. 2017. Unsupervised reference resolution in instructional videos. In Pro- ceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 2183-2192.
Kevin Clark and Christopher D. Manning. 2016b. Im- proving coreference resolution by learning entity- level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 643-653, Berlin, Germany. Association for Computational Linguistics.
lingul-iin
Maximilian slse, Jakub Tomczak, and Max Welling. 2018. Attention-bass based deep multiple instance learn- ing. In International conference on machine learning. pages 2127-2136. PMLR.
Thomas Deselaers, Bogdan Alexe, and Vittorio Ferrari. 2012. Weakly supervised localization and learning with generic knowledge. International journal of computer vision, 100:275-293.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span- BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Thomas G Dietterich, Richard H I Lathrop, and Tomás Lozano-Pérez, 1997. Solving the multiple instance problem with axis-parallel rectangles. Artificial intel- ligence, 89(1-2):31 1-71.
Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference reso- lution: Baselines and amalysis. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCLP),, pages 5803-5808, Hong Kong. China. Association for Computational Linguistics.
Biaoyan Fang. Timothy Baldwin, and Karin Verspoor. 2022. What does it take to bake a cake? the RecipeRef corpus and anaphora resolution in pro- cedural text. In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 3481-3495, Dublin, Ireland. Association for Computational Lin- guistics.
Andrej Karpathy and Fei-Fei. 2015. Deep visual- semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128- 3137.
Biaoyan Fang, Christian Druckenbrodt, Saber A Akhondi, Jiayuan He, Timothy Baldwin, and Karin Verspoor. 2021. ChEMU-ref: corpus for model- ing anaphora resolution in the chemical domain. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1362-1375, Online. Association for Computational Linguistics.
Chloé Kiddon, Ganesa Thandavam Ponnuraj. Luke Zettlemoyer, and Yejin Choi, 2015. Mise en place: Unsupervised interpretation of instructional recipes. In Proceedings of the 2015 Conference Empiri- cal Methods in Natural Language Processing, pages 982-992.
Akira Fukui, Dong Huk Park, Daylen Yang. Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach.

---

Diederik P Kingma and Jimmy Ba. 2014, Adam: method for stochastic optimization. arXiv preprint arXiv:1412 1412.6980.
First Workshop on Computational Models of Refer- ence, Anaphora and Coreference, pages 11-22. New Orleans, Louisiana. Association for Computational Linguistics.
Hideo Kobayashi, Yufang Hou, and Vincent Ng. 2022. Constrained multi-tase task learning for bridging resolu- tion, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pages 759-770, Dublin, Ire- land. Association for Computational Linguistics.
Alessandro Prest, Christian Leistner, Javier Civera, Cordelia Schmid, and Vittorio Ferrari. 2012. Learn- ing object class detectors from weakly annotated video. In 2012 IEEE Conference on computer vision and pattern recognition, pages 3282-3289 IEEE
Weicheng Kuo, Fred Bertsch, Wei Li, AJ Piergio- vanni, Mohammad Saffar, and Anelia Angelova. Generalized localization with natural language queries. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, Octo- ber 23-27, 2022, Proceedings, Part XXXVI, pages 502 520. Springer.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
2022. Findit:
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015, Faster r-cmm: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettle- moyer. 2017. End-to-end neural coreference reso- lution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. 2016. Ground- ing of textual phrases in images by reconstruction. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 817-834. Springer.
Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, and Kevin Mur- phy. 2015. What's cookin"? ? interpreting cooking videos using text, speech and vision. arXiv preprint arXiv:1503.01555
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. 2015. Efficient ob- ject localization using convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Christopher D Manning. Mihai Surdeanu, John Bauer, Jenny Rose Finkel. Steven Bethard, and David Mc- Closky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguis- tics: system demonstrations, pages 55-60.
Jiajun Wu, Yinan Yu, Chang Huang, and Kai Yu. 2015. Deep multiple instance learning for image classifica- tion and auto-annotation. In Proceedings of the IEEE conference computer vision and pattern recogni- rion, pages 3460-3469.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning text-video embed- ding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages
Juntao Yu, Sopan Khosla, Ramesh Manuvinakurike,
Lori Levin, Vincent Ng. Massimo Poesio, Michael Struhe, and Carolyn Rosé, 2022. The CODI-CRAC 2022 shared task on anaphora, bridging. and dis- course deixis in dialogue. In Proceedings of the CODI-CRAC 2022 Shared Task on Anaphora, Bridg- ing, and Discourse Deixis in Dialogue, pages 1-14, Gyeongju, Republic of Korea. Association for Com- putational Linguistics.
Cennet Oguz, Ivana Kruijff-Korbyovw Emmanuel Vin- cent, Pascal Denis, and Josef van Genabith. 2022. Chop and change: Anaphora resolution in instruc- tional cooking videos. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022. pages 364-374, Online only. Association for Compu- tational Linguistics.
Maxime Oquab. Léon Bottou, Ivan Laptev, and Josef' Sivie. 2015. Is object localization for weekkly- supervised learning with convolutional neural net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 685- 694.
Juntao Yu and Massimo Poesio. 2020. Multitask learning- based neural bridging reference resolution. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3534-3546, Barcelona, Spain (Online). International Committee Computational Linguistics.
Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina. Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolu- tion with the ARRAU corpus. In Proceedings of the
Yiwu Zhong, Jianwei Yang. Pengehuan Zhang, Chun- yuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. 2022. Regionclip: Region- language-image pretrain- ing. In Proceedings of the IEEE/CVF Conference

---

on Computer Vision and Pattern Recognition, pages 16793-16803.
refers the onions of the third instruction: the onions + the onion the onions. Most of the er- rors of anaphora resolution for recipes occur when the model predicts the onions of the third instruc- tion as the antecedent of the anaphor the onions the sixth instruction. Additionally, Figure 6 shows the null pronouns in the eighth instruction and the antecedents such the fifth, sixth, and seventh instructions.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In Proceed- ings of the IEEE Conference on Computer Vision and Pallern Recognition (CVPR).
Luowei Zhou, Nathan Louis, and Jason Corso. 2018 80. Weakly- video object grounding from text by loss weighting and object interaction. In BMVC.
B Implementation Details
Luowei Zhou, Chenliang Xu, Jason Corso. 2018b. Towards automatic learning of procedures from web instructional videos. In AAAI.
We use Adam (Kingma and Ba, 2014) for optimiza- tion and learning rate of 0.001. We clip gradients element-wise at 5 and use 0.3 dropouts for regular- ization. We use Negative log-likelihood as a loss function for the both task of anaphora resolution and object localization.
Dimitri Zhukov, Jean-Baptiste Alayrae, Ramazan Gok- berk Cinbis, David Fouhey, Ivan Laptev, and Josef' Sivic. 2019. Cross-task weakly supervised learn- ing from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 3537-3545.
boll potatoes and siveel potstoes for 5-7 minules
heat some all a pan
cook
further the
shred
the potatoes and sweet potatoes to
bowi
add
7.
edd garlic salt
black pepper
and chili powder
mix
shape
party
10. fry pollies pan with same oil
Figure 6: An example the annotation of anaphora resolution for cooking instructions. The grey boxes rep- resent the singular mentions without any anaphoric rela- tions. The beginning of the arrows shows the anaphora whereas the ends point the antecedent.
Anaphora Annotation
Anaphora resolution is challenging task for in- structional languages because of temporally evolv- ing entities (Oguz et al., 2022). For example, Fig- ure demonstrates how the onions of the sixth instruction step refers to the onions of the fourth instruction and the onions the fourth instruction


Reviews:
[{"review_id": "3oI5QB7Toy", "reviewer": "Reviewer_RhkM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your feedback. We will provide a more comprehensive analysis of results.\n\n### Reasons To Reject #1 and #2:\n\n> Because of the page limitation for the original submission, even though we had experimented with both FasterRCNN [2] and RegionCLIP [1], we had only included results with RegionCLIP as these showed state-of-the-art results for zero-shot object detection and open-vocabulary object detection [1]. We will use the additional page for a comprehensive analysis of results, including the experiments with FasterRCNN. We observed that RegionCLIP gives better region proposals compared to FasterRCNN.\n\n[1] Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., ... & Gao, J. (2022). Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16793-16803).\n\n[2] Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem of vision and language ambiguities is important for multimodal learning research. The task and dataset proposed in this paper can benefit related research a lot.\n2. This paper proposes a novel method to jointly model the anaphora resolution and object localization tasks. The experiments show the effectiveness of the proposed method. The authors also discuss the benefit of the tasks to each other, which can be a good reference for future research.\n3. This paper is well-written and well-organized. The demonstration of the motivation and proposed dataset and methods are clear.", "reasons_to_reject": "1. The experiment part should include more baseline methods to show the effectiveness of proposed method. It can implement some simple baselines based on existing vision-language models.\n2. The experiment part should evaluate more vision-language models on the proposed task to give a more comprehensive analysis.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focus on solving the vision and language ambiguities in multimodal learning tasks. It proposes a new task of multimodal resolution of vision and language ambiguities together with a dataset containing anaphora resolution and object localization tasks. This paper also proposes a new multitask learning system for modeling the two tasks jointly and shows that the two tasks can benefit each other.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IEHp5QTAbQ", "reviewer": "Reviewer_b6CE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your careful and comprehensive feedback.\n\n### Reasons To Reject #1:\n\n> More data is always better. One of the contributions of the paper is that we increase the original Chop&Change data set from 264 to 500 videos. However, annotations for video captions, anaphora resolution, and object localization are expensive. Even though 500 videos may still not sound that much, this provides us with 14,532 image-caption pairs (3 images from 4,633 caption-video pairs) to train our object localization model. To test the model we have 5,688 images. While statistical significance testing for object localization is difficult and we are not aware of any work yet that provides a statistical significance test for object localization, we will add statistical significance results for anaphora resolution obtained via the Approximate Randomization Test (usually used for coreference resolution as well). The $p$-value between the best and the second-best anaphora resolution results varies from 0.03 to 0.04 across the runs, hence our results are significant at the $p<0.05$ level.\n\n### Reasons To Reject #2.1:\n\n> There is a temporal relation between **mix oil with salt and pepper** and **the mixture** and this temporal relation should be resolved to understand what **the mixture** is. But, we still cannot know which **mixture** is pointed out in the video. In image 1a in Figure 1, we resolve that **the pan** is the **the hot pan** thanks to temporal relations by using anaphora resolution. However, we cannot say which pan of the image (1b) is **the pan**. This causes the chicken-and-egg problem which describes the co-dependence between the two tasks and appears in the absence of a clear visual-linguistic alignment (please see lines 009-015).  \n\n### Reasons To Reject #2.2:\n\n> Let us assume we have a step like **add the onion pieces**. We face an incorrect consecutive fragment if the model detects **the onion** as a mention instead of  **the onion pieces**. This causes error propagation to the tasks of anaphora resolution and object localization.  Additionally, we agree that talking about the issue of incorrect spans at the beginning of the task description might be confusing for the readers. We will carry the part of incorrect spans from section 3. Task to section 5.1.1 Mention Extraction and Representation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem introduced is interesting and important. Formulating the ambiguities in the multi-modal space is important and this work takes a good step in this direction.\n2. The dataset introduced will help researchers study the problem further and help the research take flight.\n3. The approach proposed in the paper is also a solid baseline for this problem and shows good performance on the dataset.", "reasons_to_reject": "1. Although the results show significant gains, the dataset is small and statistical significance is not provided. Thus, we cannot be sure if the approach would generalize to other datasets and if they are significant on this dataset.\n2. Minor points\n    1. In the abstract, the authors refer to the “visual-linguistic ambiguity” as chicken-and-egg problem. How is this chicken-and-egg? Oil mixed with salt and pepper becomes a mixture and there is a temporal relation among them. \n    2. In line 227, the authors say the span might be an incorrect consecutive fragment. What do they mean by “incorrect” here?\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new task of visual-linguistic ambiguity, provide a small dataset containing 500 cooking videos with instruction and anaphora annotations. They also propose an end-to-end multitask learning framework for solving anaphora resolution as well as object localization.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CAIJrQKvd1", "reviewer": "Reviewer_pwhj", "summary": "", "questions": "It is meaningful and exciting work with solid experiments. Here are my questions:\nA. Can you improve the writing to prevent confusion reading the paper specified in the Typos Grammar Style And Presentation Improvements section?\nB. Can you elaborate more on the data annotation details? Is there any quality check process?\nC. Can you elaborate more on how this novel dataset and task can potentially help improve the visual-textual understanding abilities of other multi-modal domains?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Questions A:\n\n> Many thanks for your careful and comprehensive feedback. We will carefully revise the writing of the paper to prevent possible confusion on the part of readers. \n\n### Questions B:\nWe have two different annotations in this study:\n\n#### Annotation for anaphora resolution:\n\nWe follow the annotation guidelines of  Oguz et al. (2022) and show an example of anaphora annotation in Appendix A Figure 6.  We selected at least five videos for training and at least 1 video for testing each of the 89 different recipes. This helps us to increase the object variations of visual and language data in our experiments. \n\n1. Boil **potatoes** and **sweet potatoes** for **5-7 minutes**\n2. Heat **some oil** in **a pan**\n3. Cook **onions** in **it**\n4. Give a further chop to **onions**\n5. Shred **the potatoes** and **the sweet potatoes** in **a bowl**\n6. Add **the onion pieces**\n7. …\n\nThe bold entities are our mentions. A mention can be either an anaphoric mention or a singular mention (i.e., a common technical term for the mentions without anaphoric relation).  For example, the spans **some oil**, **5-7 minutes**, and **a bowl**  are singular mentions, The other mentions are anaphoric mentions. The mention ‘it’ in step 3 refers to **a pan** in step 2; the mention **onions** in step 4 refers to  **onions** in step 3; the mention **potatoes** in step 5 refers to  **the potatoes** in step 1,  the mention **the sweet potatoes** in step 5 refers to  **sweet potatoes** in step 1;   The mention **the onion pieces** in step 6 refers to **onions** in step 4. \n\n\nWe do not apply any clustering of the mentions. For example the mentions **the onion pieces** in step 6, **onions** in step 4, and **onions** in step 3 do not get in one cluster. The mention of **the onion pieces** in step 6 refers to **onions** in step 4, and the **onions** in step 4 refers to **onions** in step 3. This is because the onions we see in step 6 are the onions chopped in step 4 and the onions in step 4 are the onions cooked in step 3.\n\n#### Annotation for object localization:\n\nEach mention is connected to the object bounding boxes of images with mention IDs.  For example, the ID of mention ‘it’ in step 3 is E10, the ID of the bounding box of the corresponding image is also E10. We obtain 3 video subsets of consecutive frames of the instruction to acquire the state changes of entities in time. We pick one frame with a clear visual content of each entity from the 3 subsets. Thus, each entity is represented in a maximum of three frames for state changes. Then we annotate each frame of an object with a bounding box around the object.\n\n### Questions C:\n\nAll instructional videos share a universal property, and that is that substances and objects in them undergo change, including chemical, physical, part-whole, and other changes. This makes such data extremely challenging for NLP, as both textual reference to objects and the objects themselves continuously change during the temporal progression of the video. What we present in this paper with cooking videos is highly relevant for industrial process videos, construction video, videos of scientific experiments, etc, and at the cutting edge of multimodal NPL towards mining such multimodal data. To date, there is very little research targeting this challenging setting and we see our research as an important contribution to advancing the field.\n\nAn instruction example for the woodworking domain\n1. put 3 equal size planks on the floor\n2. screw them together\n3. cut 4 pieces of wood to equal size\n4. glue the legs to the tabletop\n5. paint the table\n\n+ 3 equal size planks, the floor are sungular mentions (no antecedent)\n+ them in step 2 refers  3 equal size planks in step 1\n+ the legs in step 4 refers  4 pieces of wood  in step 3\n+ the tabletop in step 4 refers screw them together step 1\n+ the table in step 5 refers glue the legs to the tabletop  step 4\n\nLet’s assume that we aim to find the object location of ‘them’ in step 2. We first need to resolve ‘them’ to find which object we should localize for the given image.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel Dataset: The creation of the \"Find2Find\" dataset is a significant contribution. It addresses a gap in resources for studying visual-linguistic ambiguity, and provides a valuable resource for future research in this area.\n2. Innovative Approach: The paper proposes a novel joint task of anaphora resolution and object localization. This new perspective and approach can inspire and inform subsequent research in multimodal understanding tasks.\n3. Empirical Evidence: The authors provide empirical evidence to support their approach. This validation strengthens the credibility of the proposed method and demonstrates its practical applicability.\n4. Benchmark: The proposed model and the empirical results can serve as a benchmark for future studies aiming to solve similar problems in the realm of multimodal understanding tasks.\n", "reasons_to_reject": "I do not have much reason to reject it, but just a little concerned about its generalization ability. The \"Find2Find\" dataset and the proposed model are focused on the cooking domain. The effectiveness of the proposed method in other domains or more diverse contexts remains unexplored. The results and insights, thus, may not be generalizable.\n", "questions_to_authors": "It is meaningful and exciting work with solid experiments. Here are my questions:\nA. Can you improve the writing to prevent confusion reading the paper specified in the Typos Grammar Style And Presentation Improvements section?\nB. Can you elaborate more on the data annotation details? Is there any quality check process?\nC. Can you elaborate more on how this novel dataset and task can potentially help improve the visual-textual understanding abilities of other multi-modal domains?", "paper_topic_and_main_contributions": "Paper Topic:\n\nThis paper addresses the challenge of visual-linguistic ambiguity in multimodal understanding tasks. Specifically, it tackles the issues arising from visual objects requiring the model to ground a referring expression in a video without strong supervision (visual ambiguity), and ambiguities that occur from changes in entities in action flows (linguistic ambiguity). To explore and resolve this chicken-and-egg problem, the authors propose a novel task of joint anaphora resolution and object localization.\n\nMain Contributions:\n\n1. Dataset Creation: The authors contribute a novel dataset, \"Find2Find\", designed specifically for the task of joint anaphora resolution and object localization. This dataset includes 500 anaphora-annotated recipes with corresponding videos, offering a rich resource for the study of visual-linguistic ambiguity.\n\n2. Model Implementation: The paper introduces an innovative end-to-end multitask learning framework for handling the aforementioned dataset. This model is capable of jointly learning anaphora resolution and object localization, thereby fusing visual and textual information to address the problem of visual-linguistic ambiguity.\n\n3. Empirical Validation: The authors provide experimental results demonstrating the effectiveness of their approach. The results reveal that the multitask learning paradigm mutually improves both anaphora resolution and object localization tasks. Moreover, the study shows that object localization can assist in avoiding linguistic ambiguity of null pronouns, thereby underlining the efficacy of the proposed method.\n\nIn sum, this paper contributes a new data resource and a practical approach to address the challenging problem of visual-linguistic ambiguity, offering advancements for the field of multimodal understanding tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Clarity and Explanation of Symbols: The paper does not adequately explain the different symbols used in formulas. This lack of clarity can make it challenging for readers to understand the proposed method, limiting the potential for other researchers to build upon this work.\n\n2. Formatting Errors: There are formatting errors, such as the one in line 247, which may hinder a smooth reading experience and could potentially cause confusion or misunderstanding.\n\n3. Confusing Expressions: There are instances of confusing expressions or lack of clarity in the writing, notably on lines 262, 337, and 406. This lack of clear expression can obscure the meaning and impede understanding of the paper's content.\n\n4. Reference to Non-Existent Appendix: The paper refers to an Appendix 9 in line 294, but this appendix does not appear to exist. This could lead to confusion for readers and might detract from the credibility of the paper."}}]

Author rebuttal:

Meta-review: This paper addresses an important yet previously overlooked issue of visual-linguistic ambiguity in multimodal understanding tasks. The authors have conducted thorough research, from task design, dataset construction, baseline method development, to empirical validation. This lays a solid foundation for subsequent research in the area. While there are some areas in the manuscript that could be improved, we look forward to an engaging camera-ready version that incorporates feedback from the reviews and discussion.
Decision: Accept-Main