ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning
Year: 2023

Abstract:
Two-Tower Vision-Language (VL) models have shown promising improvements on various downstream VL tasks. Although the most advanced work improves performance by building bridges between encoders, it suffers from ineffective layer-by-layer utilization of uni-modal representations and cannot flexibly exploit different levels of uni-modal semantic knowledge. In this work, we propose ManagerTower, a novel VL model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels. The managers introduced in each cross-modal layer can adaptively aggregate uni-modal semantic knowledge to facilitate more comprehensive cross-modal alignment and fusion. ManagerTower outperforms previous strong baselines both with and without Vision-Language Pre-training (VLP). With only 4M VLP data, ManagerTower achieves superior performances on various downstream VL tasks, especially 79.15% accuracy on VQAv2 Test-Std, 86.56% IR@1 and 95.64% TR@1 on Flickr30K. Code and checkpoints are available at https://github.com/ LooperXX/ManagerTower.

heading: 1 Introduction
text: In recent years, there has been a growing interest in the field of Vision-Language (VL) representation learning due to the development of VisionLanguage Pre-training (VLP) techniques. VLP aims to learn transferable multi-modal knowledge from large-scale image-text pairs, which can further improve the performance of various downstream VL tasks, such as visual question answering (Goyal et al., 2017), visual entailment (Xie et al., 2019), visual reasoning (Suhr et al., 2019), and image-text retrieval (Young et al., 2014).
Visual and textual modalities in VL models are typically processed by uni-modal encoders and subsequently fused in a cross-modal encoder. This
∗Contribution during internship at Microsoft. †Contact Person
general architecture can be referred to as the TwoTower architecture. METER (Dou et al., 2022) and BridgeTower (Xu et al., 2022) are two representative Two-Tower VL models. METER uses CLIPViT (Radford et al., 2021) and RoBERTa (Liu et al., 2019b) as pre-trained uni-modal encoders, but it ignores different levels of uni-modal semantic knowledge in them and only feeds the last-layer outputs of each uni-modal encoder into the cross-modal encoder. In an effort to address this issue, as illustrated in Figure 1(a), BridgeTower connects multiple top uni-modal layers with each cross-modal layer in a layer-by-layer fashion to exploit unimodal semantic knowledge at different levels.
In this work, we build upon the research of BridgeTower and advance it in two aspects. Specifically, we address the limitations of BridgeTower: (i) its layer-by-layer utilization of different unimodal layer representations is ineffective. Each cross-modal layer can only utilize an artificiallyconnected uni-modal layer representation, thus restricting the exploitation of different levels of unimodal semantic knowledge. (ii) the number of cross-modal layers is tied to the number of uni-
14507
modal layer representations it used, thus limiting its scalability and capability. For example, increasing the number of uni-modal layer representations used requires a corresponding increase in the number of cross-modal layers. This leads to an increase in the number of parameters and computation cost, while does not always result in performance improvements as demonstrated by Xu et al. (2022).
As shown in Figure 1(b), we propose a novel VL model architecture, ManagerTower, that aggregates multi-layer uni-modal representations via managers in each cross-modal layer. Each manager takes multi-layer uni-modal representations as the insights of pre-trained uni-modal experts at different levels, and then adaptively aggregates them to facilitate more comprehensive cross-modal alignment and fusion. More concretely, inspired by the linear combination of layers (Wang et al., 2019) method, we adapt it as the Static Aggregation of Experts (SAE) manager and then remove redundant information to design the Static Aggregation of Uni-modal Experts (SAUE) manager, which focuses on aggregating uni-modal semantic knowledge. We further propose the Adaptive Aggregation of Uni-modal Experts (AAUE) manager to adaptively aggregate multi-layer uni-modal representations for each token in different cross-modal layers. Moreover, in principle, managers can be easily integrated into any cross-modal encoders and work well with any uni-modal encoders, making ManagerTower scalable and flexible.
We first explore the feasibility of various designs of managers by evaluating and analyzing the performance on VQAv2 and Flickr30K datasets. Then, we pre-train ManagerTower with commonly used 4M VLP data and evaluate it on various downstream VL tasks. With the same pre-training and fine-tuning settings and uni-modal backbones as previous strong baselines such as METER and BridgeTower, ManagerTower achieves superior performances on various downstream VL tasks, especially 79.15% accuracy on VQAv2 Test-Std, 86.56% IR@1 and 95.64% TR@1 on Flickr30K. It outperforms not only many base-size models pre-trained on 4M data but also some models pretrained on more data and/or with larger size.

heading: 2 Preliminary
text: In this work, for a fair comparison with METER and BridgeTower, we use the same cross-modal encoder and pre-trained uni-modal encoders.

heading: 2.1 Visual Encoder
text: CLIP-ViT, the visual encoder of CLIP (Radford et al., 2021), has been widely used in VL models (Shen et al., 2021; Dou et al., 2022). It reshapes each input image into a flattened patch sequence and prepends a [class] token to the sequence. After a linear projection, position embeddings are added to the sequence to get the input visual representation V0. The ` th visual layer representation can be computed as: V`=EncoderV` (V`−1), `= 1 . . . LV, where ` is the layer index and LV is the number of layers of the visual encoder.

heading: 2.2 Textual Encoder
text: RoBERTa (Liu et al., 2019b) is widely used in the field of VL (Dou et al., 2022; Li et al., 2022b) due to its robust performance. It tokenizes the input text with the byte-level Byte-Pair Encoding (BPE) (Sennrich et al., 2016; Radford et al., 2019) and adds [
] and [
] tokens to the start and end of the sequence, respectively. Then, it applies word embeddings and positional embeddings to the tokenized sequence to get the input textual representation T0. Similarly, the ` th textual layer representation can be computed as: T`=Encoder T ` (T`−1), `=1 . . . LT, where LT is the number of layers of the textual encoder.

heading: 2.3 Cross-Modal Encoder
text: We adopt the transformer encoder (Vaswani et al., 2017) with the co-attention mechanism as the cross-modal encoder (Lu et al., 2019). For each cross-modal layer, each modality has a multi-head self-attention (MSA) block, a multi-head crossattention (MCA) block, and a feed-forward (FFN) block. The MCA block allows the visual part of the cross-modal encoder to attend to the textual part and vice versa. Each cross-modal layer is denoted as EncoderC` , `=1 . . . LC, where LC is the number of cross-modal layers. For brevity, the ` th cross-modal layer computes as:
C̃V` = C V `−1, (1) C̃T` = C T `−1, (2)
CV` ,C T ` = Encoder C ` (C̃ V ` , C̃ T ` ), (3)
where CV` ,C T ` are the output representations of the visual and textual part at the ` th layer, C̃V` , C̃ T ` are inputs of each part. CV0 ,C T 0 are initialized with the last-layer representations from uni-modal encoders: CV0 =VLVWV,C T 0 =TLTWT, where WV,WT
are linear cross-modal projections. In this work, we use the same default setting as BridgeTower for a fair comparison: LV=LT=12, LC=6, and only top N=6 uni-modal layer representations are used.

heading: 2.4 Utilization of Uni-Modal Experts
text: Different layers of uni-modal encoders encoding different levels of semantic information are well demonstrated in vision (Dosovitskiy et al., 2020; Raghu et al., 2021; Naseer et al., 2021) and language (Peters et al., 2018b; Liu et al., 2019a; Jawahar et al., 2019). According to Dosovitskiy et al. (2020) and Raghu et al. (2021), lower layers of ViT tend to attend both locally and globally, while higher layers primarily focus on global information. Similarly, Jawahar et al. (2019) found that the intermediate layers of BERT (Devlin et al., 2019) encode a hierarchy of linguistic information, with surface features at the bottom, syntactic features in the middle, and semantic features at the top.
In the field of VL, some works have explored the usage of pre-trained multi-layer uni-modal representations (Dou et al., 2022; Xu et al., 2022). They simply feed the weighted sum of uni-modal layer representations into the first cross-modal layer, or layer-by-layer exploit multiple top uni-modal layer representations in each cross-modal layer. In this work, we take each layer of the pre-trained unimodal encoder as a uni-modal expert, and the output representation of each layer as the insight of the uni-modal expert into the current input.

heading: 3 Manager Design
text: Figure 2 depicts the overall framework of ManagerTower. It introduces managers in each cross-modal
layer to adaptively aggregate the insights of pretrained uni-modal experts at different levels. In the subsequent subsections, we will elaborate on the detailed design schema for the three types of managers, and conclude with the cross-modal encoder with our well-designed managers.1

heading: 3.1 Static Aggregation of Experts
text: The effectiveness of layer fusion in learning comprehensive representations has been well demonstrated in machine translation (Wang et al., 2018, 2019; Wei et al., 2020). Motivated by this, we decide to apply this technique in the context of VL. As a preliminary approach, we choose to utilize the linear combination of layers method (Wang et al., 2019), which is a simple yet effective way to aggregate the representations of previous layers through the use of learned weights in each encoder layer.
A natural idea is to adapt it to aggregate unimodal and cross-modal output representations of all previous layers. We name it the Static Aggregation of Experts (SAE) manager. The calculation of the ` th visual manager is:
MV` (V7, . . . ,V12,CV1 , . . . ,CV`−1) = (4) `−1∑
i=1
WV,`i+6 LN(CVi )+ 6∑
i=1
WV,`i LN(Vi+6),
whereMV` denotes the manager for the visual part of the ` th cross-modal layer, WV,`∈R(6+`−1)×D is a learnable parameter matrix, denotes the element-wise product operation and LN(·) denotes Layer Normalization (Ba et al., 2016). The
1More details on pre-training objectives and downstream fine-tuning are described in Appendix A.
softmax with a learnable temperature is used to normalize WV,`. We then omit the superscript V,` of W for brevity. The learned aggregation weight W is initialized with 16+`−1 on average in order to assign equal weights to the output representation of all previous layers.
However, directly applying SAE to VL models is non-trivial, since it does not bring a desired performance improvement compared to BridgeTower but led to a significant performance decrease. We posit that this decrease may be due to the average initialization of W not being suitable for cross-modal and pre-trained uni-modal output representations as they have different scales. To investigate this hypothesis, we propose dividing the parameter matrix W into uni-modal and cross-modal parts and initializing them with 16 and 1 `−1 , respectively,
2 and also learn the softmax temperature separately. The experimental result yield a significant improvement compared to the direct application of SAE, but a limited improvement compared to BridgeTower. These observations provide a compelling argument for re-examining how to aggregate multi-layer pretrained uni-modal representations.

heading: 3.2 Static Aggregation of Uni-Modal Experts
text: Since Equation (4) can be divided into uni-modal and cross-modal parts, by computing the cosine similarity of aggregated uni-modal/cross-modal representations between every two consecutive textual/visual managers, we further analyze the insights aggregated by different SAE managers.
As shown in Figure 3, for SAE managers, the uni-modal similarity is always similar to 1, while the cross-modal similarity increases with depth and gets closer to 1. This indicates that, the uni-modal representations aggregated by different SAE managers are almost identical, and the aggregated crossmodal representations get similar with depth.
We hypothesize that, since different SAE managers provide similar aggregated uni-modal representations to each cross-modal layer, output representation of more preceding cross-modal layers may bring redundant information to confuse the managers. This leads to aggregated cross-modal representations converging to indistinguishable vectors as the depth increases.
Hence, we propose focusing on aggregating the insights of pre-trained uni-modal experts and keep-
2We also try some different initialization methods: one, progressive, exponential moving average, BridgeTower-like, etc., but the results are similar to or lower than the average.
ing only the output representation of the previous cross-modal layer. We name it the Static Aggregation of Uni-modal Experts (SAUE) manager. The calculation of the ` th visual manager becomes:
MV` (V7, . . . ,V12,CV`−1) = WC LN(CV`−1)+ 6∑
i=1
Wi LN (Vi+6), (5)
where W∈R6×D and WC ∈R1×D are learnable parameter matrices and initialized with 16 and 1 on average, respectively. The softmax with a learnable temperature only normalizes W.
The significant improvement compared to BridgeTower empirically support our hypothesis. Moreover, in Figure 3, the cross-modal similarity of SAUE decreases with depth, which indicates that comprehensive and distinguishable cross-modal representations are learned as depth increases.

heading: 3.3 Adaptive Aggregation of Uni-Modal Experts
text: Although the SAUE manager achieves a significant performance improvement, it still has two limitations: (i) W, the learned aggregation weight of uni-modal expert insights, is almost identical between managers in different cross-modal layers, as shown in Figure 3 & 7, which is inconsistent with the intuition that the need for uni-modal semantic knowledge varies among cross-modal layers; (ii) in the inference phase, managers in different crossmodal layers use the same aggregation weight of uni-modal expert insights for all tokens in different samples, which does not match the intuition that the need for uni-modal semantic knowledge varies among tokens and samples.
To address the above limitations, we propose the Adaptive Aggregation of Uni-Modal Experts (AAUE) manager. During training and inference phases, AAUE managers can adaptively exploit different levels of uni-modal semantic knowledge from pre-trained uni-modal experts, for different tokens in different samples. Take the visual AAUE manager for example, the calculation of the ` th visual manager becomes:
MV` (V7, . . . ,V12,CV`−1)= WC LN(CV`−1)+ 6∑
i=1
WA,i LN (Vi+6), (6)
WA=softmax(LN(C V `−1)×WM+ ), (7)
where WM ∈ RD×6 is a linear projection layer. The generated aggregation weights WA∈R6×L×D can adaptively aggregate uni-modal representations of each token from different levels of pre-trained uni-modal experts. The softmax has a learnable temperature and ∼ N (0, 1
62 ) is a Gaussian noise
for exploration of aggregation (Xue et al., 2022). Furthermore, to better help managers to exploit uni-modal semantic knowledge for the current cross-modal layer, we propose replacing the visual query CV`−1 in Equation (7) with the cross-modal fused query CA(CV`−1,C T `−1) to further improve performance, where CA is a cross-attention mechanism. We visualize WA in Section 4.4.

heading: 3.4 Cross-Modal Encoder with Managers
text: Since the 1st cross-modal layer lacks the output representations of the previous cross-modal layer as the query, we introduce the SAUE managers in the 1st cross-modal layer and the AAUE managers in the subsequent cross-modal layers. Hence, Equation (1) & (2) of the 1st cross-modal layer with SAUE managers becomes:
C̃V1 =MV1 (V7, . . . ,V12), (8) C̃T1 =MT1 (T7, . . . ,T12). (9)
For the 2nd and subsequent cross-modal layers with AAUE managers:
C̃V` =MV` (V7, . . . ,V12,CV`−1,CT`−1), (10) C̃T` =MT` (T7, . . . ,T12,CT`−1,CV`−1), (11)
where we omit the modality type and layer index embeddings added to uni-modal layer representations V,T in the above equations for simplicity.
Figure 4 shows adaptive aggregation of the insights of pre-trained visual experts in AAUE manages, which is the uni-modal (right) part of Equation (6). As for SAUE managers, they directly broadcast the learned weights W∈R6×D to WA and then aggregate the insights.

heading: 4 Experiments

heading: 4.1 Implementation Details
text: ManagerTower consists of a pre-trained textual encoder, RoBERTaBASE with 124M parameters, a pre-trained visual encoder, CLIP-ViT B-224/16 with 86M parameters, and a randomly-initialized 6-layer cross-modal encoder with managers which has 113M+12M parameters. The detailed setting of the cross-modal encoder is the same as BridgeTower. The maximum length of the text sequence is set to 50, and the image patch size is 16× 16. We use an image resolution of 384 × 384 for Flickr30K and 576×576 for VQAv2 for a fair comparison with BridgeTower. AdamW (Loshchilov and Hutter, 2019) optimizer with a base learning rate of 2e−5 and warmup ratio of 0.1 is used.

heading: 4.2 Investigation and Analysis
text: In this section, we investigate various designs of managers and evaluate the performance by directly fine-tuning on VQAv2 and Flickr30K without VLP. Experimental settings are the same as BridgeTower for a fair comparison. Note that uni-modal encoders are initialized with their pre-trained weights.

heading: 4.2.1 Type of Manager
text: We first investigate the performance of different types of managers and different queries. Take the
visual manager for example, based on the top N=6 visual layer representations V ∈ RN×L×D from CLIP-ViT, different managers provide the aggregation weights that can be broadcast to WA for aggregating the insights of pre-trained visual experts. From the perspective of aggregation weights WA, the SAE and SAUE managers are static sentencelevel managers that share the same aggregation weights for all tokens in different samples. Correspondingly, the AAUE manager is an adaptive token-level manager that adaptively generates different aggregation weights for different tokens in different samples. Besides, we also implement Equation (7) with commonly used cross-attention and concat-attention mechanisms for comparison.
Results are shown in Table 1. By focusing on aggregating the insights of pre-trained uni-modal experts, the SAUE manager outperforms the SAE manager on both datasets. Furthermore, with the help of the cross-modal fused query, the AAUE manager achieves substantially better performance than other managers. This demonstrates the effectiveness of adaptive token-level aggregation with the cross-modal fused query compared to static sentence-level aggregation. Notably, the crossmodal fused query incorporates output representations of both visual and textual parts of the previous cross-modal layer, which can better help managers to correctly aggregate uni-modal semantic knowledge required by the current cross-modal layer.

heading: 4.2.2 Number of Cross-Modal Layers
text: We compare ManagerTower to BridgeTower with different numbers of cross-modal layers in Table 2 to further evaluate the effectiveness of ManagerTower. Regardless of the number of cross-modal layers, ManagerTower consistently and signifi-
cantly outperforms BridgeTower on both datasets. More interestingly, the performance of ManagerTower with LC = 3 (76.04) is even better than that of BridgeTower with LC=6 (75.91). Unlike BridgeTower, the number of uni-modal layer representations used N in ManagerTower is not tied to the number of cross-modal layers LC and can be flexibly adjusted. We fix N=6 as the default setting. Therefore, ManagerTower actually uses the same number of uni-modal layer representations as BridgeTower, but achieves even better performance using half the number of cross-modal layers. This further demonstrates the flexibility and effectiveness of ManagerTower to adaptively aggregate uni-modal semantic knowledge, compared to layerby-layer exploitation in BridgeTower.

heading: 4.2.3 Number of Uni-Modal Experts.
text: We further investigate the effect of varying N in ManagerTower with LC = 3. As shown in Figure 5, there exist two interesting observations: (i) ManagerTower (LC=3,N=3) is still better than BridgeTower (LC=3,N=3). This indicates that when the same number of uni-modal layer representations are introduced, ManagerTower allows more effective aggregation of uni-modal semantic knowledge, thus facilitating cross-modal alignment and fusion in each cross-modal layer. (ii) the performance of ManagerTower first increases
gradually, but decreases after N> 6. We assume that lower-layer uni-modal representations may not help ManagerTower learn cross-modal fusion and also increases the computational cost, which is also consistent with the observation in Xu et al. (2022).

heading: 4.3 Comparison with Previous Arts
text: Pre-train Settings. We pre-train ManagerTower with two standard VLP objectives, masked language modeling (MLM) and image-text matching (ITM), on the commonly used 4M public data: Conceptual Captions (CC) (Sharma et al., 2018), SBU Captions (Ordonez et al., 2011), MSCOCO Captions (Chen et al., 2015), and Visual Genome (VG) (Krishna et al., 2017). The pre-train settings are the same as BridgeTower and METER for a fair comparison. ManagerTower is pre-trained for 100k steps with a batch size of 4096 and a learning rate of 1e−5. The image resolution for VLP is 288× 288 and only center-crop (Radford et al., 2021) is used without any data augmentation.
Main Results. Table 3 shows the performance of ManagerTower compared with other previous works on various downstream VL tasks. ManagerTower achieves superior performances on these datasets with only 4M VLP data. With the same pre-training and fine-tuning settings and uni-modal backbones as previous strong baselines METER and BridgeTower, ManagerTower significantly improves performances on various downstream VL tasks, especially 79.15% accuracy on VQAv2
Test-Std, 86.56% IR@1 and 95.64% TR@1 on Flickr30K. This further demonstrates that with all other factors fixed, compared to BridgeTower that introduces bridges to METER, ManagerTower allows more effective aggregation of multi-layer unimodal representations via well-designed managers. Managers can adaptively aggregate more accurate uni-modal semantic knowledge to facilitate comprehensive cross-modal alignment and fusion in each cross-modal layer. Notably, ManagerTower not only outperforms many base-size models pretrained on 4M data, but also surpasses some models pre-trained on more data and/or with larger size.

heading: 4.4 Visualization of Aggregation Weights
text: We delve into managers by visualizing the average aggregation weights they generate for each cross-modal layer over all samples in VQAv2 Valid in Figure 6. For each row, the first column shows the learned aggregation weights of SAUE managers. The other five columns show the aggregation weights generated by AAUE managers and share the Y-axis to provide easy horizontal comparison.
Interestingly, the aggregation weight distributions provided by managers are completely different from the one-hot distributions specified in BridgeTower, and there are two distinct trends: (i) For SAUE managers in the 1st cross-modal layer, vertically: textual manager exhibits increasing and then decreasing weights, most favoring T10, unlike T12 and T7 used in METER and BridgeTower,
0.14
0.16
0.18
T ex
tu al
W ei
gh ts 1
0.0
0.2
0.4 2
0.0
0.2
0.4 3
0.0
0.2
0.4 4
0.0
0.2
0.4 5
0.0
0.2
0.4 6
7 8 9 10 11 12
Uni-Modal Expert Index
0.10
0.15
0.20
V is
u al
W ei
gh ts 1
7 8 9 10 11 12
Uni-Modal Expert Index
0.0
0.2
0.4
0.6 2
7 8 9 10 11 12
Uni-Modal Expert Index
0.0
0.2
0.4
0.6 3
7 8 9 10 11 12
Uni-Modal Expert Index
0.0
0.2
0.4
0.6 4
7 8 9 10 11 12
Uni-Modal Expert Index
0.0
0.2
0.4
0.6 5
7 8 9 10 11 12
Uni-Modal Expert Index
0.0
0.2
0.4
0.6 6
respectively; visual manager exhibits increasing weights, most favoring V12, the same as METER and BridgeTower. (ii) For AAUE managers in the 2nd to 6th cross-modal layers, horizontally: whether textual or visual managers, they exhibit diverse aggregation weight distributions in different layers.
Overall, comparing the aggregation weight distributions horizontally and vertically, ManagerTower learns diverse distributions in different cross-modal layers. This provides strong evidence that the introduced managers can adaptively aggregate unimodal semantic knowledge for comprehensively cross-modal representation learning.

heading: 4.5 Intuitive Comparison Between BT&MT
text: We provide brief illustrations in Figure 8 to intuitively compare BridgeTower (BT) and ManagerTower (MT) with different type of managers.
BT vs. MT with SAUE Managers. In Table 2 & 5, we provide the performance comparison between BridgeTower and ManagerTower.3 In fact, BridgeTower can be seen as an approximate special case of ManagerTower with SAUE managers if we replace the learned weights W in each manager with layer-by-layer one-hot distributions4 used in BridgeTower. However, as shown in Figure 7, the aggregation weight of textual and visual SAUE managers share a similar progressive trend
3The re-implemented BridgeTower obtained higher experimental results than the original paper due to the better fine-tuning settings we used for all experiments in Section 4.2.
4It means that, for each cross-modal layer, only one unimodal expert is activated at a time in the bottom-up direction.
across cross-modal layers, which is completely different from the distributions in BridgeTower. This allows ManagerTower with SAUE managers to achieve significant performance gains (from 75.91 to 76.55) compared to BridgeTower. Besides, the similar trend of aggregation weights is consistent with the observations in Figure 3, that is, the cosine similarity of aggregated uni-modal representations between managers is always similar to 1.
SAUE Manager vs. AAUE Manager. When we compare Figure 6 & 7, their respective aggregation weight distributions are completely different. This further demonstrates that compared with SAUE managers, AAUE managers can adaptively generates different aggregation weights for different tokens in different samples. Interestingly, the first column of two figures both comes from the SAUE managers, but the distributions are still clearly different. We presume that high-layer AAUE managers may help low-layer SAUE managers rectify their management of experts.
We also provide the visualizations of aggregation weights of SAE and AAUE managers without VLP in Figure 9 & 10. Comparing the visualization of three types of managers without VLP, we can find that (i) the learned aggregation weights of SAE and SAUE managers are still a little close to the average initialization we used and they all share a similar progressive trend across cross-modal layers; (ii) for each AAUE manager, its generated aggregation weights vary significantly across 6 uni-modal experts; comparing different cross-modal layers,
the distribution of aggregation weights generated by the AAUE manager is also very different.

heading: 5 Related Work
text: Vision-Language Models. Although VL models differ in model architecture, most of them use uni-
modal encoders to extract visual and textual representations, and then fuse them in a cross-modal encoder, which can be unified into the Two-Tower architecture (Lu et al., 2019; Su et al., 2020; Chen et al., 2020; Li et al., 2020a,b; Zhou et al., 2020; Kim et al., 2021; Radford et al., 2021; Jia et al., 2021; Li et al., 2021a,b, 2022a; Dou et al., 2022; Wang et al., 2021a,b, 2022a,b; Yu et al., 2022). As a representative model, METER (Dou et al., 2022) adopts pre-trained uni-modal encoders and feeds their last-layer representations into the cross-modal encoder. BridgeTower (Xu et al., 2022) proposes building layer-by-layer connections between the top uni-modal layers and each cross-modal layer to utilize different uni-modal layer representations. However, they still cannot provide adaptive and effective aggregation of multi-layer pre-trained unimodal representations in each cross-modal layer.
Multi-Layer Representation Aggregation. The effectiveness of layer representation aggregation in learning comprehensive representations has been well demonstrated in vision (Lin et al., 2017; Huang et al., 2017; Yu et al., 2018; Xie et al., 2021) and language (Peters et al., 2018a; Wang et al., 2018, 2019; Wei et al., 2020). Recent VL models also explore utilization of multi-layer uni-modal representations for better cross-modal representation learning. METER feeds the weighted sum of uni-modal representations into the first crossmodal layer. BridgeTower introduces bridges into METER so that different uni-modal layer representation are fed layer by layer into each cross-modal layer. In this work, ManagerTower explores adaptive and effective aggregation of multi-layer unimodal representations via well-designed managers.

heading: 6 Conclusion
text: We propose ManagerTower, a novel VL model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels via the introduced managers in each cross-modal layer. The feasibility of various designs of managers is well explored, and the effectiveness of ManagerTower on various downstream VL tasks is well demonstrated. More comprehensive cross-modal alignment and fusion in each cross-modal layer is achieved by adaptive aggregation of different levels of uni-modal semantic knowledge. We hope that our work can inspire more research on how to better exploit multi-layer pre-trained uni-modal representations for cross-modal representation learning.
Limitations
In this work, we propose managers that allow adaptive aggregation of uni-modal layer representations in each cross-modal layer. Inevitably, AAUE managers significantly improve performance which slightly increasing the computational budget, as we detailed discussed in Appendix C. This needs to be further optimized in the future. Analysis and optimization are also needed for the other types of managers as shown in Appendix D. Moreover, as shown in Figure 5, the performance of ManagerTower first increases gradually with the number of uni-modal representations, but then stops increasing and even decreases when the number of uni-modal representations exceeds 6. How to obtain better ManagerTower performance using a lower computational budget while utilizing more insights of uni-modal experts, especially when scaling the model, e.g., 24-layer CLIP-ViT L-224/16 and 24-layer RoBERTaLARGE, is a question worth further exploration. For example, designing reasonable sparse activation functions for managers in ManagerTower, instead of simple top-N or top-p sampling (which did not work well in our preliminary experiments).

heading: Acknowledgements
text: This work was supported by the National Key R&D Program of China via grant 2020AAA0106501 and the National Natural Science Foundation of China (NSFC) via grant 62236004 and 61976072.

heading: B Switch Visual and Textual Backbones
text: We experiment with different pre-trained visual and textual backbones as uni-modal encoders to further investigate the impact on performance of the managers of ManagerTower compared to the bridges of BridgeTower. As shown in Table 5, regardless of the visual and textual backbones we apply, ManagerTower significantly and consistently outperforms BridgeTower on both datasets. This further proves the effectiveness and generalization of our proposed ManagerTower architecture and managers, which can provide adaptive and effective aggregation of multi-layer uni-modal representations for vision-language representation learning.

heading: C Computational Budget
text: Table 6 shows the computational budget and downstream task performance without VLP for
BridgeTower and ManagerTower, including the number of parameters, the number of FLoatingPoint operations (FLOPs)5. We measure the average inference time of processing 1 VQA instance over 10K runs on 1 NVIDIA TITAN V GPU. The sequence length is 50, and the image resolution is 384× 384. Compared with BridgeTower (1st row), ManagerTower (4th row) uses an acceptable additional computational budget (3.69% parameters, 4.22% FLOPs, and 3.77ms inference time) and achieves significant performance improvements of 0.74% and 3.1% on VQAv2 and Flickr30K, respectively. We further analyze other well-performed variants of ManagerTower in the 2nd and 3rd rows. It is worth noting that the two variants share a similar computational budget as BridgeTower, but achieve better performance. This not only demonstrates the efficiency and effectiveness of our ManagerTower architecture, but also reminds us that the cross-modal fused query via the cross-attention mechanism is the main reason for the additional computational budget of ManagerTower (4th row), as it is the only difference between the 3rd and 4th row models. This inspires us to explore a more efficient method to fuse CV`−1 and C T `−1 to get the cross-modal fused query in the future.

heading: D Details on Cross-Attention and Concat-Attention Managers
text: Cross-Attention Managers. We implement the standard cross-attention mechanism (Vaswani et al., 2017) and reduce the linear projection layer for value to save computational budget.6 Take the visual manager for example, it takes CV`−1 ∈ RL×D as the query, and the first token of multi-layer unimodal representations, i.e., V[:, 0] ∈ RN×D, as the key. Hence, the shape of generated aggregation weights is N× L, which can be broadcast to
5We use Facebook Research’s fvcore to calculate FLOPs. 6The calculation of cross-modal fused query also uses this
simplified version of the cross-attention mechanism.
the aggregation weights WA∈RN×L×D. The following calculation is the same as AAUE managers in Figure 4. The results in Table 1 show a significant decrease compared to other managers on Flickr30K. We leave the detailed analysis of this phenomenon to the future work.
Concat-Attention Managers. Take the visual manager as an example, it broadcasts CV`−1 ∈ RL×D to RN×L×D, and concatenates it with V ∈ RN×L×D along the last dimension as the concatenated query. It then directly projects the query to WA∈RN×L×D. The following calculation is the same as AAUE managers in Figure 4. In fact, this type of manager is different from all other managers from the perspectives of the generated aggregation weights. Although its aggregation weights delve into the feature dimension of CV`−1 and V, the substantially increased number of parameters and computational cost do not result in a significant performance gain, making it impractical and inefficient. More efficient variants of this type of manager should be investigated in the future.

heading: E Detailed Comparison with Previous Arts
text: Due to the space limitations, we omit some baselines and details in Table 3. Here we provide more details on the comparison with previous arts in Table 7.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations.
7 A2. Did you discuss any potential risks of your work? This paper proposes a novel Vision-Language model architecture, ManagerTower, which may not contains potential malicious or unintended harmful effects and uses.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section Abstract, and Section 1. Introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2. Preliminary, and Section 3. Manager Design.
3 B1. Did you cite the creators of artifacts you used? Section 2. Preliminary, and Section 3. Manager Design.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? All the pre-train data and pre-trained uni-modal backbones are publicly available, we just use them for pre-training and initialization, and we don’t repackage or release them. Our code and pre-trained models will be released, and the name of the license for each asset will be stated in our code repo.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Our code and pre-trained models will be released, and the name of the license for each asset will be stated in our code repo.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? All the data we used are publicly available and used safely by previous works.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We will provide the documentation of our code and pre-trained models in our code repo.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section Appendix A. Implementation Details.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section Appendix C. Computational Budget.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4. Experiments.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4. Experiments.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4. Experiments.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave the detailed analysis of this phenomenon to the future work. Concat-Attention Managers. Take the visual manager as an example, it broadcasts CV`−1 ∈ RL×D to RN×L×D, and concatenates it with V ∈ RN×L×D along the last dimension as the concatenated query. It then directly projects the query to WA∈RN×L×D. The following calculation is the same as AAUE managers in Figure 4. In fact, this type of manager is different from all other managers from the perspectives of the generated aggregation weights. Although its aggregation weights delve into the feature dimension of CV`−1 and V, the substantially increased number of parameters and computational cost do not result in a significant performance gain, making it impractical and inefficient. More efficient variants of this type of manager should be investigated in the future