Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment
Year: 2023

Abstract:
Despite recent progress towards scaling up multimodal vision-language models, these models are still known to struggle on compositional generalization benchmarks such as Winoground. We find that a critical component lacking from current vision-language models is relation-level alignment: the ability to match directional semantic relations in text (e.g., ‘mug in grass’) with spatial relationships in the image (e.g., the position of the mug relative to the grass). To tackle this problem, we show that relation alignment can be enforced by encouraging the language attention from ‘mug’ to ‘grass’ (capturing the semantic relation ‘in’) to match the visual attention from the mug to the grass (capturing the corresponding physical relation). Tokens and their corresponding objects are softly identified using a weighted mean of cross-modal attention. We prove that this notion of soft cross-modal equivalence is equivalent to enforcing congruence between vision and language attention matrices under a ‘change of basis’ provided by the cross-modal attention matrix. Intuitively, our approach projects visual attention into the language attention space to calculate its divergence from the actual language attention, and vice versa. We apply our Cross-modal Attention Congruence Regularization (CACR) loss to fine-tune UNITER and improve its Winoground Group score by 5.75 points.

heading: 1 Introduction
text: Compositionality is the ability to combine meanings of constituents according to structured rules. Recent work shows that Vision-Language Models (VLMs) fail to construct compositional representations and generally ignore syntactic & structural information [Thrush et al., 2022, Milewski et al., 2022, Liang et al., 2022]. Winoground [Thrush et al., 2022] is a vision-language compositionality task that tests a VLM’s ability to match syntactic permutations of text with their visual interpreta-
tions, for example correctly matching “grass in mug” and “mug in grass” to their corresponding images. Winoground finds that all recent stateof-the-art VLMs perform below chance levels on this compositionality task. Contemporaneously, Milewski et al. [2022] probe for structural knowledge in VLMs, finding that they encode significantly less linguistic syntax than Language Models (LMs) and virtually no visual structure. Recently, Yuksekgonul et al. [2022] built a large dataset confirming that VLMs treat images as a ‘bag of objects’ and don’t adequately represent visuo-linguistic relations.
Since models must determine whether the compositional structure of an image matches that of the caption, it’s important for the model to learn to cross-modally align intra-modal relations. That
5444
is, if the relation from ‘mug’ to ‘grass’ is ‘in-ness’, the model should recognize when the equivalent physical relation holds between a mug and grass in the image, and representationally align these relations such that an image-text matching head may more easily determine whether the relations are cross-modally equivalent. In simpler terms, the compositional structure of input for each modality should be represented such that they can be crossmodally matched even for difficult examples like Winoground.
Unfortunately, there has been less highly influential work on relation alignment between vision & language, and Thrush et al. [2022] did not benchmark any such models. In this work, we begin exploration of these relation alignment approaches by tentatively grouping them into 3 categories:
1. Structural Data: training a model on data that explicitly captures relational structure
2. Structural Model: infusing an inductive bias into the architecture of the model that enables more compositional representations
3. Structural Training: modifying the objective function or imposing a parameter constraint to encourage relation alignment
Since Structural Data approaches require complex annotations and Structural Model approaches are often incompatible with large transformers, we identify Structural Training as a promising avenue for providing compositional inductive biases to VLMs due to their architecture-agnostic compatibility and computational scalability.
In this work, we propose a Structural Training approach for relation alignment that uses the crossmodal attention matrix as a change of basis1 to the opposite modality, which we then compare to the original modality to calculate a divergence loss, effectively measuring cross-modal congruence between intra-modal attentions.
We show how our approach, Cross-modal Attention Congruence Regularization (CACR), generalizes previous Structural Training work on crossmodal attention regularization (IAIS [Ren et al., 2021]) by taking into account all possible entity alignments and computationally simplifying relation alignment. The CACR regularization term can easily be dropped into most transformerbased Vision-Language model objectives with no
1not defined in a strict linear algebraic sense
added data and minimal computational overhead, to encourage relation alignment during training. Finally, we show that CACRbase improves on IAISbase—where IAISlarge holds the current stateof-the-art on Winoground.

heading: 2 Related Work
text: Below, we categorize several relation alignment approaches following the framework in Sec. 1.
1. Structural Data [Wu et al., 2019, Zhang and Peng, 2019, Yu et al., 2021, Cui et al., 2021, Wan et al., 2021, Khan et al., 2022]
2. Structural Model [Andreas et al., 2016, Guo et al., 2019, Hong et al., 2021, Zhang, 2022, Wang et al., 2022b, Kim et al., 2022, Wang et al., 2022a]
3. Structural Training [Ren et al., 2021, Yang et al., 2021a,b, Xue et al., 2021]
While some of these works introduce ideas from multiple of these categories, we group them by their core contribution. For example, ROSITA proposes a graphical data pre-training approach, and a selfsupervised objective to accompany it; we consider it a Structural Data approach since the training objective ultimately is just a necessity for the data being provided.
Unfortunately, many of these works do not provide publicly available code or pre-trained checkpoints, so we were unable to complete an exhaustive analysis of the compositional performance of these relation alignment approaches. Due to the added complexity of Structural Model approaches, we leave exploration of their compositional abilities to future work.
Regardless, we chose one exemplar for both Structural Data (ROSITA) and Structural Training (IAIS) that made their pre-trained image-text matching checkpoints available; we generated their scores on Winoground, which have not previously been calculated. In Tab. 1, we present these two relation alignment models’ Winoground scores alongside a few entity alignment and global alignment models.
Notice that global alignment approaches tend to perform the lowest on Winoground, even when scaled considerably. Entity alignment approaches perform intermediately and OSCAR+ specifically held the state-of-the-art prior to our benchmarking of these relation alignment models. Of the
two relation alignment approaches we benchmark, IAIS beats out OSCAR+ and achieves a new stateof-the-art on Winoground. But ROSITA, despite providing structural data to encourage cross-modal relation alignment, underperforms OSCAR+. We attribute this partly to the improved visual features OSCAR+ has access to as a result of VinVL, but further comparison of IAIS and ROSITA is explored in our recent work.
Based on these past results and analysis, we choose to further explore structural training approaches to relation alignment. In other words, our research question becomes: How can we infuse the vision-language model’s training objective with an implicit structural prior that encourages cross-modal alignment of relations?

heading: 3 Cross-modal Attention Congruence Regularization
text: To attempt a solution to this question, we begin by noting that attention activations encode some degree of relational information. Attention values in transformers may be seen as an informational gating mechanism that implicitly encode how representations are composed [Abnar and Zuidema, 2020]. For example, past work in language has shown how syntax trees may be extracted [Mareček and Rosa, 2019] from attention across layers and used to guide attention [Bai et al., 2021, Li et al., 2020] for improved compositionality. In this section, we extend this intuition to the multimodal domain by proposing to use the cross-modal attentions, which as a change-of-basis matrix encode a
transformation from one modality’s compositional structure to the opposite modality’s, to encourage cross-modal relation alignment.

heading: 3.1 Relation Alignment Using Attention
text: In specific, we focus on the self-attention matrix S computed in a transformer by
S = QK⊤ = (XWQ)(XWK)⊤ (1)
Then, some row i in S corresponds to a distribution over columns j0, ..., jn where Si,j tells us how much of the previous layer’s entity representation j we want to infuse into the current layer’s entity representation i, intuitively their compositional relation. Since X is a series of visual and linguistic tokens, we can segment S into four submatrices for intra- and cross-modal relations [Bugliarello et al., 2021]. Denote the intra-modal attention submatrices in the last multimodal encoder layer as SV V (vision to vision) and SLL (language to language); the cross-modal attention matrices as SV L (vision to language) and SLV (language to vision).
S = ( SLL SLV SV L SV V ) (2)
If an image and caption have the same underlying compositional structure, the entities that crossmodally correspond to each other should bear similar intra-modal compositional structure. That is, a word w should attend to other words (in SLL) in a similar way that its visual object counterpart o attends to other objects (in SV V ). Furthermore, we can use the cross-modal matrices (SLV and SV L) to identify entities that cross-modally correspond as they will generally attend to each other [Aflalo et al., 2022]. Unfortunately, since representations are heavily contextualized by the final layer, clear bijective correspondences between words and objects may not always be identified using an argmax over the cross-modal attention matrix as Ren et al. [2021] attempts. Deeper analysis of when their model, IAIS, fails to identify cross-modal bijective correspondences is provided in Sec. 5.

heading: 3.2 Attention Congruence
text: We opt to use the cross-modal matrices (SLV and SV L) as a whole to ‘change basis’ to the opposite modality, with which we can then calculate ‘congruence’ with the original modality. However, we use ‘change of basis’ and ‘congruence’ loosely since the cross-modal matrices are not guaranteed
to be square and thus do not satisfy strict linear algebraic definitions. We formulate SV V in the language basis as SLV SV V S⊤LV , which we then encourage to be similar to SLL.
Under the hood, this says that for each ai→j ∈ SLL, we can use row vectors SLV,i and S⊤LV,j to calculate a weighted sum a∗i→j over SV V . If we were to do this for all i, j, we would construct a matrix of the same shape as SLL where each entry is a∗i→j , i.e. an approximation of the visual correspondent of the relation ai→j taking into account all the possible cross-modal alignments of i and j. Since this computation intuitively makes a lot of sense and may more easily be compared to previous approaches, we choose to illustrate it in Fig. 3. However, since this computation is relatively expensive, we instead use the SLV SV V S⊤LV formulation which produces the same matrix of a∗i→j values but with considerably fewer operations. This also enables us to view the operation as a ‘changeof-basis’ to the opposite modality and the CACR loss as encouraging a sense of cross-modal ‘congruence’.
Specifically, we align the original SLL with the language-basis SV V matrix using LCACR-L:
LCACR-L = m-KL(σ(SLV SV V S⊤LV ), σ(SLL)). (3)
We apply a softmax to normalize both matrices since SLV SV V S⊤LV will generally be larger in scale due to summation. Additionally, m-KL(·) [Ren et al., 2021] is a symmetric matrix-based KullbackLeibler Divergence (m-KL) which measures the distance between two matrices S and S′:
m-KL(S, S′) = N∑
i
KL(Si||S′i) + KL(S′i||Si),
(4) where (·)i stands for the ith row-vector in the matrix.
Similarly, we have LCACR-V:
LCACR-V = m-KL(σ(SV LSLLS⊤V L), σ(SV V )), (5)
Combining LCACR-V and LCACR-L, we present our LCACR objective, an attention activation regularizer for cross-modal relation alignment:
LCACR = LCACR-V + LCACR-L. (6)
When the vision inputs and the language inputs have the same sequence length and SV L, SLV are invertible, then SV V and SV LSLLS⊤V L (as well as SLL and SLV SV V S⊤LV ) can become strictly congruent. In this case, SV LSLLS⊤V L can be interpreted as the language view of SV V . Aligning
SV LSLLS ⊤ V L and SV V leads to cross-modal relation alignment. It is similar for SLV SV V S⊤LV and SLL. In the general case where the vision inputs and the language inputs may have different sequence lengths, the two forms are not linear algebraically congruent but the relevant intuition still holds.

heading: 3.3 Hard and Soft Cross-modal Equivalence
text: In this section, we show that CACR can be interpreted as leveraging cross-modal soft equivalences, where IAIS [Ren et al., 2021] uses hard bijective equivalences. In their approach, each element in the intra-modal attention matrix is aligned with a single counterpart in the opposite modality. This is built upon a strict assumption that there exists a one-to-one mapping (provided by an argmax over the cross-modal attention) from SLL to SV V and vice versa, which is unsatisfied in practical cases. CACR may be seen as a soft cross-modal equivalence method which instead uses the whole SLV (or SV L) to implicitly build an ‘equivalence weighting’ which is then used to compute a weighted mean over SV V (or SLL). We illustrate and compare hard cross-modal equivalence and our soft cross-modal equivalence in Figure 3, taking the language-side alignment as an example.
We note that IAIS could be seen as a special case of soft cross-modal equivalence by forcing the cross-modal attention map to be a one-hot matrix, i.e., taking the argmax of the attention matrix as the index of the cross-modal counterpart. We show in Section 5 that IAIS can have inferior performance when a clear bijective cross-modal correspondence isn’t available.
In Alg. 1, we show the pseudo-code of the soft cross-modal equivalence method for calculating the vision-side loss. SCACR−V can be computed similarly. Computing the hard and soft cross-modal equivalence is computationally complex and difficult to be parallelized due to indexing operations. For practical applications, we sought to simplify this soft cross-modal equivalence algorithm to a mathematical equivalent that would improve computational tractability. From here, we arrive at CACR, which is a closed-form formulation of soft cross-modal equivalence which utilizes only differentiable matrix multiplications. Therefore, our CACR is more computationally efficient and easier to be parallelized than soft cross-modal equivalence.
Algorithm 1 Soft Cross-modal Equivalence (V) Require: SLL ∈ N ×N,SV L ∈ N ×M,SV V ∈
M ×M 1: L ← 0 2: for i, j ∈ SV V do 3: W ← SV L[i] · S⊤V L[j] ▷ soft weighting 4: a∗i→j ←W ◦ SLL ▷ element-wise
weighted mean 5: L = L+ m-KL(a∗i→j , SV V [i, j]) 6: end for 7: return L

heading: 3.4 Proof of Equivalence Between CACR and Soft Cross-modal Equivalence
text: Computing the hard (IAIS) and soft cross-modal equivalence is computationally complex and difficult to parallelize due to indexing operations. However, CACR loss is mathematically equivalent to soft cross-modal equivalence but can be computed efficiently. We take CACRV for illustration of this equivalence, but CACRL can be proved in the same way.
Beginning with the visual-basis form of SLL in CACR, the attention at index [i, j] in SV LSLLS⊤V L is
(SV LSLLS ⊤ V L)[i, j] = ∑NL
p ∑NL k avi→lkalp→lkavj→lp
= ∑NL
p ∑NL k SV L[i, k]SV L[j, p]︸ ︷︷ ︸
soft weighting
SLL[p, k]
(7) where avi→lj stands for the attention from the i-th visual token to the j-th linguistic token, NL is the total number of language tokens and NV is the total number of the visual tokens. Comparing Eq.7 and Alg.1, we observe that the summation we arrive at above is equivalent to the content of the for-loop (line 3-5). Thus, although of seemingly different linear algebraic form, CACR generalizes IAIS by way of its equivalence to the Soft Crossmodal Equivalence formulation presented above.

heading: 4 Results
text: How does CACR compare to other vision-language models in its compositional ability?
We fine-tuned CACR on Flickr30k [Young et al., 2014] for 5000 epochs using PyTorch [Paszke et al., 2019] with a train-validation-test split of 90-5-5.
The training batch size is 4 and 31 negative samples are provided for every individual positive sample in a standard image-text matching training setup. We use a learning rate of 5 × 10−5, the AdamW optimizer [Loshchilov and Hutter, 2017], and introduce LCACR with an exponential warmup schedule. Training was completed on a node with 4 NVIDIA GTX 1080 Ti’s, each with 11 GB of memory.
In Tab. 2, we present our approach’s scores alongside a few other models. Since we use CACR to fine-tune UNITER, we include scores for the two baseline UNITER sizes. We also include scores for IAISbase which is also built on UNITER.
The fact that CACRbase outperforms IAISbase suggests that, with adequate computational resources, CACRlarge could similarly outperform IAISlarge, potentially achieving a new state-of-theart on Winoground. Furthermore, its performance compared to UNITERlarge is impressive consider-
ing that CACRbase is approximately half its size in parameters. Despite our resource constraints, we were able to train a partly frozen (first 12 layers) version of CACRlarge, which achieves IAISlarge levels of performance on Winoground (text: 37.5, image: 18.75, group: 15.75) with just 20% of the training time.
Finally, we report Flickr30k retrieval scores in Tab. 3 to verify that we are not somehow overfitting to Winoground. Though CACR takes some minor losses to its retrieval scores, this may be attributed to imperfect hyperparameters, suggesting that CACR’s performance on Winoground could be even higher with adequate hyperparameter tuning. It’s also important to remember here that we’re only training on Flickr30k, so this isn’t a case of our model overfitting to Winoground and ‘forgetting’ its true image-text matching ability. Rather, it shows that the hyperparameters that we
adapted from IAIS need to be modified to more perfectly train CACR on Flickr30k, which would then carry over to compositional improvements on Winoground.

heading: 5 Analysis
text: Why does CACR’s soft cross-modal equivalence approach outperform hard cross-modal equivalence?

heading: 5.1 Qualitative
text: Hard cross-modal equivalence, implemented by IAIS, assumes that cross-modal submatrices can be used to find a singular equivalent of an entity in the opposite modality. Specifically, if i∗ = argmax(SLV [i]) then SLL[i] should correspond to SV V [i
∗]. In simple terms, IAIS says the following: if word A attends most to object A and word B attends most to object B, then word A should attend to word B in a similar way that object A attends to object B. Underlying IAIS is the hard assumption that argmaxing over the cross-modal attention submatrix is an effective means of identifying the opposite modality equivalent of an entity. However, we show in this section that this is often not the case.
Given the argmaxes for rows in the SLV submatrix, we can identify the bounding box that each token maximally attends to, which IAIS assumes is its visual equivalent. In Fig. 4a, we visualize an example where ‘clouds’ maximally attends (green) to the ground, which would prevent IAIS from identifying the correct cross-modal equivalence. ‘Turbines’ (Fig. 4b), on the other hand, maximally attends to a bounding box that better matches our intuition. It is qualitatively clear from the several examples displayed that the argmax assumption often fails to identify the correct cross-modal equivalence. Since words may attend to several visual tokens for different reasons, we shouldn’t assume that the cross-modal argmax provides us with a clear bijective correspondence.
Instead, the cross-modal matrices should be seen as providing useful high-level information about what visual entities are relevant to a word, and vice versa (as intuitively demonstrated by [Ilinykh and Dobnik, 2022]). We can certainly gain useful information about cross-modal correspondences using it, but it isn’t as simple as using an argmax, due to words having multiple referents and entity representations being intermixed. Instead, our soft
cross-modal equivalence approach takes all the possible cross-modal alignments into account with a weighted sum.
To illustrate how the soft approach accounts for critical cross-modal alignment information, we present a few Winoground examples with UNITER’s cross-modal attention activations in Fig. 4 and 5. We use UNITER since this is the baseline model from which attentional information is bootstrapped to calculate cross-modal alignments. For example, in Fig. 5c, using the representation for the bounding box covering the mug’s handle may not adequately capture the visual referent of ‘mug’ and therefore disrupt our ability to calculate the visual-basis relation between ‘mug’ and ‘grass’ if restricted by an argmax.

heading: 5.2 Quantitative
text: In the absence of annotations, we attempted a quantitative measurement of whether overlap in argmaxes (several words attending to one bounding box or vice versa) as quantified by the Shannon Entropy of argmax indices inversely correlates with soft Winoground score. Intuitively, if an example has more like a one-to-one mapping between text and image, the entropy of its cross-modal argmaxes should be higher as each token will attend to a different box, which would suggest that the model is
better aligning entities. However, we found no significant correlation with Winoground score, which we attribute to the fact that high entropy on its own doesn’t mean correct entity alignment.
Rather, high entropy in argmax indices could still be produced by a bad representation if ‘mug’ attends to the grass & ‘grass’ attends to the mug; conversely, low entropy could be produced by a good representation for an example like ‘fire truck’ where two tokens refer to a single object. Quantitative exploration of cross-modal attention is difficult without annotations and we leave this task to future work to explore in a multimodal compositionality context.
As a general takeaway, while the cross-modal argmax assumption of IAIS does hold in some cases and may be more meaningful during the course of IAIS training, it is clearly quite a strict assumption that could suffer if an entity attends to several cross-modal entities or there are no corresponding cross-modal entities. Furthermore, since IAIS is only active in the final self-attention layer, all the token representations are intermixed and therefore don’t necessarily have a one-to-one correspondence with our intuitive notions of what they should be—the word ‘turbine’ may not solely represent the traditional meaning of that word but per-
haps the entire scene that includes the turbines, clouds, and ground.
We hypothesize that by removing the hard argmax assumption, our approach better accounts for varying cross-modal entity equivalences and thus enables stronger relation alignment. By also calculating alignment between all pairs of source and target modality entities, CACR should considerably improve sample efficiency, which is important considering that the final layer S matrix of the converged IAIS model is largely flat. Therefore it’s important to backpropagate as much alignment knowledge over the course of training as possible, which CACR’s soft equivalence weighting implicitly enables.

heading: 6 Conclusion
text: In this work, we identified that a key factor holding back models from vision-language representational compositionality is cross-modal relation alignment. We categorized recent compositional inductive bias approaches into 3 categories: Structural Model, Structural Data, and Structural Training, showing that a previous Structural Training model (IAIS) achieves state-of-the-art performance on Winoground. We then identified a potential key weakness in IAIS, its hard argmax assumption, and developed a soft cross-modal equivalence approach to address it. Having linear algebraically simplified this approach, we arrived at CACR, an auxiliary loss that encourages cross-modal congruence of intra-modal attention. CACR improves on IAIS’ performance on Winoground, and even outperforms a UNITER model nearly twice as large.
As computational scaling becomes more widespread, it’s necessary to develop compositional inductive biases that do not require complex annotated data or exotic model architectures. Our work illustrates how taking advantage of the transformer’s own attentional structure can improve the quality of fine-grained vision-language representations, opening the avenue for large scale approaches to visually-grounded compositionality.

heading: 7 Limitations, Risks & Ethics
text: Though CACR shows significant gains in compositional performance, results are limited in their exploration of only one pre-trained model and compositionality dataset. A significant risk of models is their tendency to be biased by distributions in their training data; vision-language models are
not free from this flaw, but we see our work as teaching VLMs to learn better structured representations rather than memorizing spurious correlations in data. We remain far from solving the vision-language compositionality problem, so biases must continue to be actively mitigated.

heading: Acknowledgements
text: This material is based upon work partially supported by National Science Foundation awards 1722822 and 1750439, and National Institutes of Health awards R01MH125740, R01MH132225, R01MH096951 and R21MH130767. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred.

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: Quantitative exploration of cross-modal attention is difficult without annotations and we leave this task to future work to explore in a multimodal compositionality context. As a general takeaway, while the cross-modal argmax assumption of IAIS does hold in some cases and may be more meaningful during the course of IAIS training, it is clearly quite a strict assumption that could suffer if an entity attends to several cross-modal entities or there are no corresponding cross-modal entities. Furthermore, since IAIS is only active in the final self-attention layer, all the token representations are intermixed and therefore don’t necessarily have a one-to-one correspondence with our intuitive notions of what they should be—the word ‘turbine’ may not solely represent the traditional meaning of that word but per-
haps the entire scene that includes the turbines, clouds, and ground. We hypothesize that by removing the hard argmax assumption, our approach better accounts for varying cross-modal entity equivalences and thus enables stronger relation alignment. By also calculating alignment between all pairs of source and target modality entities, CACR should considerably improve sample efficiency, which is important considering that the final layer S matrix of the converged IAIS model is largely flat. Therefore it’s important to backpropagate as much alignment knowledge over the course of training as possible, which CACR’s soft equivalence weighting implicitly enables