TAVT:Towards Transferable Audio-Visual Text Generation
Year: 2023

Abstract:
Audio-visual text generation aims to understand multi-modality contents and translate them into texts. Although various transfer learning techniques of text generation have been proposed, they focused on uni-modal analysis (e.g., text-to-text, visual-to-text) and lack consideration of multi-modal content and cross-modal relation. Motivated by the fact that humans can recognize the timbre of the same low-level concepts (e.g., footstep, rainfall, and laughing), even in different visual conditions, we aim to mitigate the domain discrepancies by audiovisual correlation. In this paper, we propose a novel Transferable Audio-Visual Text Generation framework, named TAVT, which consists of two key components: Audio-Visual MetaMapper (AVMM) and Dual Counterfactual Contrastive Learning (DCCL). (1) AVMM first introduces a universal auditory semantic space and drifts the domain-invariant low-level concepts into visual prefixes. Then the reconstructbased learning encourages the AVMM to learn “which pixels belong to the same sound” and achieve audio-enhanced visual prefix. The welltrained AVMM can be further applied to unimodal setting. (2) Furthermore, DCCL leverages the destructive counterfactual transformations to provide cross-modal constraints for AVMM from the perspective of feature distribution and text generation. (3) The experimental results show that TAVT outperforms the stateof-the-art methods across multiple domains (cross-datasets, cross-categories) and various modal settings (uni-modal, multi-modal).

heading: 1 Introduction
text: Audio-visual text generation bridges the gap between perception (visual and auditory) and communication (via language), and is hence becoming an increasingly important goal for artificial agents. Uni-modal text generation tasks like machine translation (Wang et al., 2022; Jin et al., 2022b; Yin
∗ Equal contribution. † Corresponding author
et al., 2022), and image caption (Chen et al., 2017; Tewel et al., 2021; Hu et al., 2022) have already flourished as a result of the large-scale pre-training and huge model capacity. However, for audiovisual text generation tasks, data annotation is more arduous (temporal structure) and expensive (requires monitors and speakers) than uni-modal text generation. Moreover, despite the effectiveness, existing works (Iashin and Rahtu, 2020a; Le et al., 2020; Hori et al., 2021) inevitably suffer severe degradation due to varying construction conditions in different domains.
In this paper, to break through the constraint, we propose a novel task, named transferable audiovisual text generation. The main challenge of this task is the multi-modal domain shifts caused by varying conditions, like the visual style and audio energy. One common approach to handle the domain shift is domain-alignment-based transfer learning. However, existing works (Sun and Saenko, 2016; Rozantsev et al., 2018; Ding et al., 2022) focus on the uni-modal analysis, which is insufficient due to the lack of consideration of crossmodal relations.
We observe that while audio and visual are often correlated in natural events and jointly affect human perception, they have different characteristics. As shown in Figure 1, the fact Timbre is an
14983
intrinsic property of the object leads to sounds of the same concept (“speaking” or “cutting”) being similar across domains where the appearance like background, perspective, and style is significantly different. Based on this phenomenon, domain invariant low-level concepts can be extracted from the visual with the supervision of the audio, which is pervasive, reliable, and cheaper in contrast to expensive human annotation.
Grounded on the above discussions, we propose an Audio-Visual Meta-Mapper network (AVMM). The key idea of AVMM is to use a universal auditory semantic space to align low-level concepts across different visual domains. In particular, we introduce a visual prefix that serves as a multimodal bridge between the visual and the audio. Then, we reconstruct audio features from the universal auditory semantic space and produce the audio-enhanced visual prefix. Essentially, the accuracy of reconstructed audio provides a constraint for AVMM to learn the latent visual-textual alignment. The reconstruct-based paradigm has another windfall, allowing AVMM to transfer to both multimodal and uni-modal settings.
While the reconstruct-based paradigm implicitly learns the visual-textual alignment, we propose Dual Counterfactual Contrastive Learning (DCCL) to directly optimize the visual-audio alignment score and promote the robustness of reconstructed audio. We introduce distribution-based contrastive learning to further improve the accuracy of reconstructed audio and dependency-based contrastive learning with token-wise diversity-aware weights to provide modality-aware constraint from the perspective of text generation. Then, we apply the above module and a base audio-visual text generation network to the meta-learning framework, named TAVT, to empower AVMM with the ability to accrue knowledge across domains, which would assist in building internal multimodal representations broadly suitable for many domains. Our main contributions are as follows:
• We are the first one to study the transferable audio-visual text generation task. • We introduce the audio-visual meta-mapping network that aligns domain-invariant lowlevel concepts between visual and a universal auditory semantic space. • Experimental results on both cross-dataset and cross-category benchmarks demonstrate the effectiveness of our models.

heading: 2 Related Work
text: Audio-Visual Learning. In the past few years, there have been several works that focus on audiovisual learning. (Arandjelovic and Zisserman, 2018) used a two-stream neural network to find the most similar visual area to the current audio clip. Some works (Hu et al., 2019, 2020) employed contrastive learning to match the visual and audio components. Recently, (Liu et al., 2021; Cheng et al., 2023) proposed a framework for cross-modal representation learning with a discrete embedding space that was shared amongst different modalities and promoted model interpretability. The above approaches focus on the correlation of audio-visual pairs. While we aim to align the visual of different domains with a universal auditory semantic space and enhance multi-modal transfer learning. Audio-Visual Text Generation. The mainstream audio-visual text generation task is video captioning, which has attracted many researchers (Venugopalan et al., 2015; Le et al., 2020; Ye et al., 2022; Jin et al., 2022a). (Hao et al., 2018) proposed multimodal feature fusion strategies to integrate audio information into models. (Guo et al., 2019; Iashin and Rahtu, 2020a) proposed an attention mechanism to combine the visual and audio features for better knowledge representation. (Tian et al., 2019) introduced an audio-visual controller to manipulate the parameters and generate diverse modalityaware captions. (Rahman et al., 2019) utilized the idea of cycle consistency to build a model with visual and audio inputs. (Iashin and Rahtu, 2020b) encoded the feature representation of audio and speech for a specific event proposal and produces a caption. Compared to our work, none of the above address the problem of domain shift and suffer severe degradation when deployed to a low-resource domain. Furthermore, they all take audio as supplementary information for visuals while we attempt to utilize the audio-visual correlation to minimize the domain discrepancy.

heading: 3 Methods
text: We aim to train a model that can learn and quickly adapt to new multimodal domains with limited labeled data under the meta-learning setting. In the next sections, we will first define the audio-visual meta-learning setting, then explain our architecture, and finally describe how it is used during training and inference time.

heading: 3.1 Problem Formulation
text: We define a meta-training stage, where the model is trained on the meta-trainDmeta−train partition and a separate meta-test stage where the performance is measured on the Dmeta−test partition. For the support set, there are k samples chosen for each of the N − 1 randomly sampled domains, and for the query set, there are m samples from the remaining domain, with m > k. The set Di is defined as Di = (v i 1, a i 1, t i 1), ...(v i k, a i k, t i k), where v i j is the visual feature, aij represents the audio features, and tij is the output text, i.e., a caption to the video.

heading: 3.2 Model Architecture
text: The architecture that we present in this paper is modular and consists of three components: the meta-mapper, an audio-visual encoder, and a language model as illustrated in Figure 2. Audio-Visual Meta-Mapper Network. Intuitively speaking, low-level visual concepts in different domains often share similar sounds, e.g., footsteps, laughing, and rain. Therefore, we propose an audiovisual meta-mapping network (AVMM) to map different visuals across domains into a universal auditory semantic space and as well as addressing shifts in the semantic distribution.
We first introduce the universal auditory semantic space which has audio clusters learned from Flickr(Thomee et al., 2015). In particular, we cut the audio into units with fixed time length T and run the clustering algorithm to find the k-centres of all audio features as the audio clusters. We define the audio clusters as M = {m1,m2, ...,mk}. These audio clusters could serve as a bridge between the audio and visual and assist in learning quickly new domains by observing only limited labeled examples.
To map the visual features into the latent space
of the audio clusters, inspired by prompt learning(Lester et al., 2021), we introduce a set of l learnable tokens [p1, p2, ..., pl]. These tokens are called visual prefix for the audio clusters. Particularly, considering that the audio and visual are aligned in temporal, we also cut the frame sequence into l clips with time length T and obtain the embedding [c1, c2, ..., cl] by pooling the features of each clip. Then, we apply a mapper with self-attention (SA) on the ci to obtain the visual prefix pi as follows:
ci = 1 |T | ∑
t∈T vt
pi = SA(ci)
(1)
Then, we develop a reconstructor to reproduce the audio features from the visual prefix, and the accuracy of reconstructed audio provides a constraint for the self-attention layer to retrieve cross-domain invariance from the visual features ci, and accumulate it into pi. To achieve this, the reconstructor learns to combine audio clusters from M to reconstruct dynamic audio features Arec = { a1rec, a 2 rec, ..., a l rec } . The predicted audio features airec is a weighted sum of audio clusters m1,m2, ...,mk, where the weights are predicted
by applying a linear layer ϕ to pi followed by a softmax function to normalize weights, as follows:
airec = K∑
k=1
wkmk,
where, wk = SoftMax(ϕk(pi))
(2)
During meta-training, we apply the mean square error (MSE) loss Lrec on the reconstructed audio Arec against the ground-truth audioA to update the parameters in the meta-mapper network and shared across all domains in Dmeta−train. Audio-Visual Encoder. Formally, given a sequence of video frames, we first extract a sequence of the frame features V = {v1, v2, . . . , vm} and the audio features A = {a1, a2, . . . , am}. Then, we prepend this visual prefix to the frame features, yielding the following sequence V ′ = [p1, ..., pl, v1, ..., vm]. After that, we adopt a selfattention module to learn visual representation f ′ v ∈ Rd and audio representation f ′ a ∈ Rd as follows: f ′
= MHA(f, f, f),where, f ∈ {V′ , A} and MHA(·) denotes multi-head attention (Vaswani et al., 2017). Then we apply audio-visual cross-attention to identify attention across two different kinds of feature fields. For simplicity, we formulate this stage as:
xt=AV−Encoder(fi, fj , fj),where, i, j∈{v, a} (3) where xt ∈ {xav, xva}. Illustratively, the details of the encoder are provided in Appendix A. Language Model Generator. As opposed to the original Transformer’s decoder, we introduced an α to evaluate the contribution of different modalities (audio and visual) to each word. At time step t, αt is computed by measuring the relevance between the cross-attention of each modality and the previous words Y = {y1, y2, ..., yt−1} as follows:
αt = σ (Wt [Y,MHA(Y, xt, xt)] + bt) (4)
where [·, ·] indicates concatenation, σ is the sigmoid activation and Wt is a 2d× d weight matrix. The decoder outputs caption Ỹ is defined as:
Ỹ=αav·MHA(Y,xav, xav)+αva·MHA(Y,xva, xva) (5)
With αt, the model can provide interpretability for the audio-visual fusion strategy.

heading: 3.3 Counterfactual Contrastive Learning
text: Although the reconstruction-based paradigm provides a constraint for AVMM, it cannot directly
optimize the visual-audio alignment scores. Therefore, we propose a Dual Counterfactual Contrastive Learning (DCLL) which constructs fine-grained supervision signals from counterfactual results to directly optimize the visual-textual alignment without relying on the quality of randomly-selected negative samples. Distribution-based Contrastive Learning. Concretely, we take the reconstructed audio cues Arec as positive samples A+ and inverse the audio clusters M and weight matrix wk pairings to construct counterfactual audio cues as negative samples A−. Then, we illustrate the contrastive learning method with the causal triplet (A,A+, A−). Intuitively, we construct distribution-based contrastive learning as follows
Ldis = − log ( e(s(A,A +)/τ)
∑n i=1 e (s(A,Ai)/τ)
) (6)
where s (p, q) = pT q/ ∥p∥ ∥q∥ denotes the dot product between l2 normalized p and q; τ is the temperature parameter. The distribution-based contrastive learning further improves the accuracy and robustness of reconstructed audio. Dependency-based Contrastive Learning. For the audio-visual text generation task, there exists a modality imbalance in natural language tokens as different tokens depend on different modalities and the reconstructed audio should also show similar dependence for different tokens. We consider the dependency-based contrastive loss to maintain consistency in the distribution of scores for the original and positive samples. First, the (A,A+, A−) paired with the V are fed into the audio-visual encoder to generate the joint embeddings of them. Then, we compute the dependence score ψ (V,A) = αav/αva of original sample as the anchor r, the score ψ (V,A+) of the factual sample as the positive r+ and the score ψ (V,A−) of the counterfactual sample as the negative r−. Concretely, the contrastive loss is formulated as follows:
Ldep = − log ( e(s(r,r +)/τ)
∑n i=1 e (s(r,ri)/τ)
) (7)
The dependency-based contrastive learning considers the different performances of audio-visual in text generation to further provide cross-modal constraints on the mapper. Token-Wise Modality-Aware Weights. In order to identify some vague concepts like talking and
singing, we devise the token-wise modality-aware weights to encourage the model to use the corresponding modality in the text generation process. We obtain the association of each word with audio modality and visual modality as follows:
W ima = 1
N
N∑
i=1
( αiav αiva ) (8)
Where W ima indicates the i-th word’s weights and N is the data sample size. We apply the weight Wma on the cross-entropy loss, as follows:
Lcap=− n∑
t=1
Wma ∗ log(wt = yt|y1:t−1) (9)
where log(wt = yt|y1:t−1) denotes the probability of predicting word wt given the previously generated y1:t−1.

heading: 3.4 Meta-Training and Inference
text: The holistic training procedure is shown in Algorithm 1. Here, for simplicity, we assume that our full model is defined as a function fθ, which receives the visual features v and audio features a as input and produces y as output. The loss function, optimized per domain during training, is as follow:
L = Lcap + Lrec + λLdis + µLdep, (10)
where the hyper-parameter λ and µ seek a trade-off between the two counterfactual-contrastive learning losses (details about the hyper-parameter can be found in the supplementary materials). Meta-Training To meta-train the model, we randomly select K − 1 specific domains in D as support set Ds, and the remaining domain as a query set Dq. When adapting to a new domain Di, the trainable meta parameters θ become domainspecific parameters, namely θi. These domainspecific parameters are computed with N gradientstep updates, similar as in MAML(Finn et al., 2017), with the following rule for one gradient update: θ
′ i = θ−α∇θLDi(fθ). This is referred as the
inner-loop update, where α is the hyperparameter for the step size. Next, the model meta-parameters θ are optimized for the performance of f
θ ′ i , using the query set Dq samples and the domain-specific parameters θ
′ i as initialization of the model:
min ∑ LDi(fθ′i) = ∑ LDi(fθ−α∇θLDi (fθ))(11)
which is called outer-loop optimization. The metaoptimization across all domains Di is performed
Algorithm 1: Transferable Audio-Visual Text Generation
Input: k source domains D={D1, ..., Dk} Output: Model parameters θ 1 while not done do 2 Randomly select (k − 1)Ds ∼ D, and the remaining domain as Dq ; 3 Sample Bi ∼ Ds ; 4 foreach Bi do 5 θ
′ i = θ − α∇θLBi(fθ)
6 end 7 Sample Bq ∼ Dq ; 8 θ ′ q = θ − α∇θLBq(fθ);
9 Meta-optimization; 10 θ=θ−β∇θ
∑K−1 i=1 (LDi(fθ′i)+LDq(fθ′q))
11 end 12 Meta-Test; 13 if audio in Dtarget then 14 θ = θ − α∇θL(fθ); 15 else
/* Frozen the AVMM */ 16 θ = θ − α∇θLcap(fθ); 17 end
using stochastic gradient descent update rule, as follows: θ ← θ−β∇θ ∑LDi(fθ′i), where β is the step size hyperparameter. Meta-Test In the meta-test stage, we consider a new domain Dtarget, which also has a support set Ds for fast adaptation by fine-tuning the model meta-parameters θ to a given task, and a query set Dq to evaluate the model on this domain. Note that in audio-absent datasets like MSVD, we can freeze the parameters of the audio-visual meta-mapper network and utilize the reconstructed audio features to boost the performance.

heading: 4 Experiment

heading: 4.1 Datasets and Metrics.
text: Datasets For transferable audio-visual text generation, we design two benchmark datasets. A cross-domain benchmark was constructed based on MSR-VTT(Xu et al., 2016) containing 20 categories as well as multimodal audio and video streams. We divided a new 10-domain dataset based on MSR-VTT categories as shown in Table 1 (more detailed information about the dataset can be found in Appendix B). For cross-dataset benchmark, we use MSVD(Chen and Dolan, 2011)
which consists of 1,970 video clips collected from YouTube, and MSR-VTT† which consists of the whole Dmeta−test as target domains. Metrics. We evaluate the methods across all four commonly used metrics for video captioning: BLEU-n (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), ROUGE-L (ROUGE, 2004), CIDEr (Vedantam et al., 2015). We follow the standard practice to use the Microsoft COCO evaluation server (Chen et al., 2015).

heading: 4.2 Train Details.
text: Feature Extraction. For the visual feature, we use the ResNet-101 (He et al., 2016) model pre-trained on ImageNet as the backbone feature extractor for frames. The frames are used as the input of the CNN without re-sizing or cropping. The audio features are extracted by the VGGish(Hershey et al., 2017). For the sentences, to simplify the implementation, we use a joint vocabulary containing words in both the source domain and the target domain. And words appearing less than 3 times are replaced with a special token. Proposed Method Settings. The hidden size is 1024 for all the multi-head attention mechanisms. The numbers of heads and attention blocks are 8. For meta-training, we adopt Adam with a fixed inner learning rate of 0.0001 and outer learning rate of 0.001. We train the source domains with a batch size of 32. For the meta-test, we use the beam-search method with a beam width of 5 to generate the predicted sentences during testing. We train TAVT on an NVIDIA GeForce RTX 2080, for TAVT each epoch takes around 4 hours.

heading: 4.3 Performance Evaluation
text: Compared Models. To the best of our knowledge, there is no work investigating transferable audio-visual text generation. So we first choose the state-of-the-art video caption approaches based on two different approaches: (1)The RNN-based models: RecNet (Wang et al., 2018), AVAF(Guo et al.,
2019), MARN(Pei et al., 2019), AVIC(Tian et al., 2019), SGN(Ryu et al., 2021) and SHAN(Deng et al., 2022). (2) The Transformer-based models Att-TVT(Chen et al., 2018) and SBAT (Jin et al., 2020). Then, for a fair comparison, these methods are all trained on Dmeta−train with the same meta-learning framework as TAVT and tested on the target domain. Evaluation Results. In Table 2 we report the performance of our method in comparison with the aforementioned competitors on the cross-datasets benchmark. As it can be observed: (1) Our method outperforms all compared methods on all metrics by a large margin on MSR-VTT† and MSVD. (2) In particular, AVIC and Att-TVT focus on designing complex multimodal fusion strategies to learn visual-audio representations, but leave the audio invariance unexploited. TAVT uses audio as a supervisory signal to align visual information in different domains, focusing on transferring the invariant in audio to visual prefix. Therefore, our model outperforms them by a significant margin (+1.5%∼9.6% of CIDEr on MSR-VTT†). (3) Note that in the MSVD which has only a visual stream, we freeze the parameters of AVMM and use the reconstructed audio instead the real audio i.e., TAVT(Arec). TAVT still performs well (+13.5% of CIDEr on MSVD), which indicates that the metamapper network has accumulated domain-sharing knowledge through meta-training. In other words, the frozen meta-mapper network can produce discriminative visual prefix and reconstruct informative audio features even in the absence of real audio supervision.
In Table 3 we also report the performance on cross-categories benchmark. TAVT outperforms all other methods on five domains, which indicates that our method generalizes well. In particular, for some low-resource domains with only a few labeled data such as kids and beauty, other methods suffer from severe performance degradation, while TAVT outperforms them by a large margin (+3.2% on kids and +4.5% on beauty).

heading: 4.4 Ablation Studies
text: To analyze the effect of different components, we conduct ablation studies on the MSR-VTT† dataset. The following variants of our method are evaluated: Effectiveness of audio-visual meta-mapper. To evaluate the advantage of AVMM and audio features. We first remove both AVMM and other sub-
modules which are related to audio and only use the visual information as the lower bound ( w/o. audio). Then, we give the results without AVMM (w/o. AVMM). We also report the result that retains AVMM but uses the reconstructed audio instead of real audio (w/o. real audio).
Table 4 shows that while audio contains information that is complementary to visual and can improve performance somewhat ( w/o. audio vs w/o. meta-mapper), the more important for transfer learning is the cross-domain invariance which contained in audio and provided the supervised signal to align visual in different domains (w/o. metamapper vs TAVT). In addition, the reconstructed audio can ultimately exhibit an upper bound on performance close to that achieved using real audio, demonstrating the accuracy and validity of the meta mapper network (w/o. real audio vs TAVT). Effectiveness of different modules. The results
in Table 5 illustrate that the constraints provided by the accuracy of the reconstructed audio features are effective and critical (w/o. Lrec). The counterfactual thinking is helpful and can further improve the accuracy by +0.8% on CIDEr (w/o. Ldis). Moreover, optimizing the cross-modal relationship between audio and visual from a text generation perspective can further improve the performance of the model (w/o. Ldep). In addition, w/o. MAML denotes the model without meta-learning. We can observe that TAVT performs significantly better than w/o. MAML. Effectiveness of token-wise modality-aware weight. To investigate how token-wise modalityaware weight improves the performance from the perspective of linguistics, we visualize some token’s weight and the text generation process in Appendix E. The result in Table 5 shows that tokenwise modality-aware weights can further improve the performance of TAVT by optimizing the association of text and audio-visual modalities.
The performance in low-resource domain. To quantitatively verify the performance in the low resource domain, we compare TAVT and previous methods under different sizes of labeled video as shown in Figure 5. We observe that TAVT consistently outperforms the other methods and can achieve close to full performance with only 40% of the training data, while other methods require about 70% of the training data. Hyper-Parameter Analysis. To seek the tradeoff between the DCCL, we introduce the hyperparameter λ and µ. Figure 6 shows that the model achieves the best performance when λ=1e-4 and µ=1e-2, suggesting that proper hyper-parameters are crucial to achieve good performance.

heading: 4.5 Qualitative Analysis
text: To qualitatively verify the effectiveness of our TAVT, we display the results of TAVT in the multimodal and uni-modal settings. As shown in Figure 4, TAVT can accurately describe low-level concepts such as “chirping”, “rainy” and “piano” with
the help of audio-enhance visual prefix. When transferred to a uni-modal domain where missing audio, the AVMM can capture the correlation audio-visual correlation and reconstruct audio features that are semantically relevant to the visual.

heading: 5 Conclusion
text: In this paper, we first study the task of transferable audio-visual text generation tasks. To mitigate multimodal content domain shift, we observe that low-level visual concepts have similar sounds in different domains and propose a novel framework TAVT with two technical contributions. The first one is the audio-visual meta-mapper, which can transfer the domain-invariant concept information within the universal auditory semantic space into the visual prefix. Moreover, the well-trained audio-visual meta-mapper can also reconstruct semantic audio features in audio-absent mode. We then apply dual counterfactual contrastive learning to directly optimize the visual-audio alignment. Extensive experiments on both cross-datasets and cross-domains benchmarks verify the effectiveness of our model. Furthermore, our TAVT framework can be transferred to other text generation tasks e.g., video QA in a plug-and-play fashion.

heading: 6 Limitation
text: We identify a few limitations of the current work. Our approach still suffers from biases in the training data and may produce incorrect output or lead to an inaccurate understanding of multi-modal content. And a large-scale audio-visual pre-trained model is a promising direction toward more advanced and cheaper approaches for transfer learning, which we leave for future study.

heading: 7 Ethics Statement
text: We adopt the widely-used datasets that were produced by previous researchers and followed all relevant legal and ethical guidelines for their acquisition and use. Besides, we recognize the potential influence of our technique. When deployed our approach will have to record, store and process video and audio information related to human activities, which will have privacy implications for some application domains. We are committed to conducting our research ethically and ensuring that our research is beneficial. We hope our work can inspire more investigations for transfer learning on multi-modal tasks and wish our framework can serve as a solid baseline for further research.

heading: 8 Acknowledge
text: This work was supported in part by the National Key R&D Program of China under Grant No.2022ZD0162000, National Natural Science Foundation of China under Grant No.62222211, Grant No.61836002 and Grant No.62072397, and Yiwise.

heading: A Encoder
text: For the visual input V ′ = [p1, ..., pl, v1, ..., vm] and audio input A = [a1, ..., am], we first adapt the self-attention layer SelfAtt to learn the visual representation and audio representation as follows:
f ′ v = SelfAtt(V ′ , V ′ , V ′ ) (12) f ′ a = SelfAtt(A ′ , A ′ , A ′ ) (13)
The SelfAtt layer contains multi-head attention which can be calculated by multiple single heads:
MHA(F, F, F ) = Concat(h1, h2, ..., hh), (14)
hi = ATT(FW Q i , FW K i , FW V i ) (15)
where, WQi ,W K i ,W V i ∈ Rd d h and W 1 ∈ Rd×d. hi denotes the i-th head and h is the number of heads. ATT represents scaled dot-product attention as ATT(Q,K, V ) = softmax(Q·K
T √ dk
). Then we apply audio-visual cross-attention to identify attention across two different kinds of feature fields as follows:
xav = CrossAtt(f ′ v, f ′ a, f ′ a) (16) xva = CrossAtt(f ′ a, f ′ v, f ′ v) (17)
where CrossAtt layer is similar to SelfAtt contains multi-head attention.

heading: B Details about Datasets
text: To more adequately validate the effectiveness of our proposed approach, we first regrouped the 20 categories of MSR-VTT and reorganization a new 10-domain dataset based on the MSR-VTT (Xu et al., 2016) categories information as shown in Table 6. The number of videos regrouped 10 domains is shown in Figure 7. We use five of them as source
domains and the remaining five domains (“Animation”, “Music”, “Animal”, “Kids” and “Beauty”) as target domains. And a random sampling strategy is applied in the target domain for dataset split.

heading: C Experiment Results
text: C.1 Baseline Setting
To the best of our knowledge, there is no work investigating transferable audio-visual text generation, thus we choose the state-of-the-art video caption approach and use MAML on top of it to construct the baseline. We compare the performance of the TAVT with that of state-of-the-art methods based on two different approaches: (1)The RNN-based models: RecNet (Wang et al., 2018) which adds a reconstructor to reconstruct the visual features from the generated caption, AVAF(Guo et al., 2019) combined different multimodal fusion methods and attention mechanism, MARN(Pei et al., 2019) which equips with a memory consisting of words and corresponding visual contexts, AVIC(Tian et al., 2019) introduce audio-visual controller to balance the importance between audio and
visual modalities, SGN(Ryu et al., 2021) which encode a video into semantic groups and SHAN(Deng et al., 2022) which use syntax-guided hierarchical attention to integrate visual and sentence-context features. (2) The Transformer-based models AttTVT(Chen et al., 2018) which fuses the modalities from video and text with attention mechanism, SBAT (Jin et al., 2020) which uses boundary-aware pooling operation to reduce the redundancy.
C.2 Performance on Cross-Categories Benchmark
To further validate the generalizability of the proposed method, we also conduct experiments on five reorganized Dmeta−test domains. From the results in Table 7,8 and 9, we observe that our method outperforms other methods in all five target domains, which indicates that our method has good generalization and can compensate for the performance degradation caused by limited label data in low-resource domains such as kids and beauty.
C.3 Hyper-Parameter Analysis
We evaluate 11 different values τ from 0.01 to 1.0 on MSRVTT† and report the results in Figure 8. It shows that the performance achieves the best when τ is set to 0.4 and becomes poor when τ is too small or too large. This result suggests that a proper τ value is crucial to achieving good performance.

heading: D Analysis of Universal Auditory Semantic Space
text: D.1 Construction Details
Audio Clusters. We design our model by introducing a universal auditory semantic space consisting of a set of audio clusters. We are interested in natural environmental sounds. We download videos from videos on Flickr(Thomee et al., 2015) and extract their sounds. We downloaded over 750,000 videos from Flickr, which provides over a year (377 days) of continuous audio. The only pre-processing we do on the sound is to extract the spectrogram from the video files and subtract the mean. We extract spectrograms for approximately five seconds of audio and obtain the audio clip features by VGGish and select conv4_1 as the extraction layer. Selection of Clustering Algorithm. To obtain the audio cluster, we apply K-means (MacQueen et al., 1967) to the extracted audio features. Specifically, K-Means require a manual setting of cluster number values. Thus, we experimented with different cluster numbers. From the results shown in Table 10, we can see that when setting the cluster number as 100 achieve better results on our extracted audio features from Flickr.
D.2 Visualization of Audio Clusters.
In Figure 9, we visualize the semantic space. We can see that there are a large number of low-level concepts shared across different domains. The sounds of these low-level concepts are similar but have significant visual differences. This provides a guarantee for the effectiveness of our approach.

heading: E Analysis of Token-Wise Modality-Aware Weights
text: Modality Imbalance. There exists a modality imbalance in natural language tokens as different tokens depend on different modalities. For example, some nouns of objects like “shirts”, “lamps” and “flowers” are only visually related, but some objects like “guitar” and “alarm clock” can make sounds that are auditory related. Moreover, the verbs which indicate the low-level concepts like “talking”, “hit” and “kick” are obviously auditory related. And we visualize the word clouds in Figure 10. Visualization of Text generation. To investigate how the token-wise modality aware weight improves the performance from the perspective of linguistics, we visualize the text generation process. In Figure 11, we present the modality dependency scores of each word. Firstly, we can find that words have different dependencies on different modalities in the process of generation, which proves the existence of modal imbalance. For example, some quantifiers (a, group) and some nouns (children) rely more on the visual modality, while guitar and sing rely more on the audio modality. There are also some prepositions and conjunctions that often rely on the context text for a generation. Secondly, comparing w/o. ma and TAVT, we can find that token-wise modality-aware weights can optimize the dependence of different words on different modalities, encouraging the model to use the correct modality to generate words, which can help identify some vague concepts such as talking and singing.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6
3 A2. Did you discuss any potential risks of your work? Section 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.2
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: And a large-scale audio-visual pre-trained model is a promising direction toward more advanced and cheaper approaches for transfer learning, which we leave for future study