MoS$^2$: Mixture of Scale and Shift Experts for Text-Only Video Captioning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 08:57:56

Abstract: Video captioning is a challenging task and typically requires video-text paired data for training. 
However, manually annotating coherent textual descriptions for videos is laborious and time-consuming. 
To address this problem, we propose to utilize solely text data to enhance video captioning models.
Drawing inspiration from the exceptional text generation capabilities demonstrated by large language models (LLMs), we aim to leverage these models to generate high-quality and high-diversity video captions for the target domain.
Specifically, we prompt GPT-4 with few-shot target-domain captions to generate a limited set of plausible video captions. Subsequently, we continue to prompt GPT-4 with the generated captions to acquire large-scale captions. 
To fully exploit the generated captions, we propose a Mixture of Scale and Shift experts (MoS$^2$) for efficient adaptation of pre-trained image captioning models for video captioning. 
MoS$^2$ estimates a probability distribution over a collection of experts by a lightweight routing network, determining the allocation of tokens to appropriate experts.
This dynamic adjustment mechanism allows for specific responses to input features, thereby enhancing the model's ability to handle data variations. 
Our approach not only customizes model responses to input variations, effectively addressing the distribution shift between synthetic and actual captions but also significantly reduces the number of learnable parameters, allowing for more efficient adaptations.
With only text data, we achieve superior performance and significantly narrow the performance gap between zero-shot and fine-tuned models. Our method boosts video captioning performance with the synthetic text data, thus substantially alleviating the dependence on paired and large-scale real data of the target domain. The code will be publicly available.

Main paper: 
 related works: 
 Text-Only Captioning. With the popularity of CLIP [35] , recent researches [8, 19, 24, 31, 39, 41, 44, 56] attempt to improve image captioning by leveraging the aligned latent space. ZeroCap [41] aims to generate descriptions with the highest CLIP similarity to the image by combining CLIP with large language models. Despite the correlated CLIP text and visual latent space, the modality gap still hinders the generation of high-quality captions. DeCap [24] learns a text decoder on text-only data and projects visual embedding into CLIP text embedding space to reduce the modality gap, achieving excellent zero-shot image captioning performance. Nevertheless, these methods rely on pre-trained contrastive models, and the inherent discrepancy between generative and contrastive paradigms could potentially constrain performance. In response, our work seeks to enhance video captioning using pre-trained image captioning models. Furthermore, we utilize GPT-4 to generate domain-specific and high-quality corpora, aiming to bolster textonly captioning capabilities.Parameter-Efficient Learning. With the increasing model sizes, fully fine-tuning all parameters becomes increasingly cumbersome. To reduce the computational burden, researchers explore alternative parameter-efficient learning methods that only train a few parameters [14, 15, 21, 25, 40, 55] . One pioneering approach [14] inserts lightweight adapters into transformer layers, achieving comparable performance with full fine-tuning. Prompt Tuning [21, 25, 58] appends a group of learnable prefix tokens to language models to guide downstream tasks, while keeping the entire model frozen. In this paper, we investigate representative parameter-efficient methods for video captioning and introduce a novel mixture of scale and shift architecture tuning methods, which improves the model's flexibility and performance while preserving a minimal number of trainable parameters.Mixture of Experts (MoE) models, characterized by their scalable capacity and enhanced performance, have gained prominence in the development of large language models [18] . These models leverage a trainable gating mechanism [38] , which dynamically allocates inputs to specific experts, enabling sparse activation to augment capacity without significantly increasing computational demands. Nevertheless, achieving optimal load balance among the experts remains a challenge. To address this, [20] introduces a novel automatic load balancing strategy by incorporating a loss term that promotes more uniform activation of experts. Furthermore, [30] introduced an importance term to mitigate imbalances caused by the self-reinforcing effect of expert utilization. Building on these foundational advancements, our work introduces a novel architecture, the Mixture of Scale and Shift, which significantly enhances video captioning capabilities with while minimizing the increase in trainable parameters. 
method: 
 In this section, we present our refined text-only video captioning pipeline. In Section 3.1, we introduce the text-only captioning setting. Subsequently, Section 3.2 details our innovative parameterefficient tuning method, which allows the model to dynamically adjust its parameters in response to varying inputs. Section 3.3 discusses the training loss, while Section 3.4 describes the inference pipeline for video captioning. The overall pipeline is illustrated in Figure 1 .Text-Only Video Captioning Pipeline. The development of visual captioning systems typically commences with the curation of a paired dataset. Unfortunately, the manual annotation of such datasets is both time-consuming and laborious. Curating video captioning datasets is even more arduous than image captioning datasets, as annotators must invest additional time to comprehend the dynamic and complex content of videos.Recent research attempts text-only captioning, aiming to reduce the reliance on paired datasets. Text-only captioning seeks to enhance caption performance in specific domains using solely textual data. This task offers two configurations based on text data availability: full-text and few-shot. In the full-text scenario, all relevant textual data from the target domain are employed to optimize the text decoder, thereby maximizing performance within that domain. Conversely, the few-shot setting, characterized by limited text availability, does not support comprehensive model fine-tuning. Previous text-only captioning methods mainly focus on the full-text configuration. Nonetheless, full-text configuration is usually impractical, primarily because caption datasets are typically manually annotated, which is both labor-intensive and timeconsuming. Consequently, producing high-quality in-domain captions for target domains remains a challenging endeavor.In response, our study explores an alternative setting, better suited for real-world applications. We aim to enhance captioning performance by leveraging minimal text samples from the target domain. We attempt to leverage LLMs' instruction-following and contextual reasoning capabilities to produce captions that are not only high in quality but also diverse and well-aligned with the target domain. We employ LLMs to exploit their instruction-following and contextual reasoning capabilities, aiming to produce captions that are not only high-quality but also closely aligned with the target domain. Subsequently, we employ the synthesized data to fine-tune the text decoder, enhancing the model's captioning performance within on the target domain.Inspired by the remarkable instructionfollowing capability of large language models (LLMs), we propose leveraging ChatGPT or language GPT-4 [1] to synthesize in-domain captions, thereby reducing annotation costs. Specifically, we randomly select 10 captions from the downstream dataset and construct a "few-shot" instruction-following prompt, as illustrated in This prompt encourages the LLMs to generate captions that align with the provided sample captions. However, the text synthesized in this way has a high percentage of duplicates. To mitigate this, we incorporate the generated text back into the prompt. Specifically, we randomly sample captions from both the selected downstream and synthesized captions and then prompt ChatGPT or GPT-4 [1] with the novel prompt. This self-instructed technique injects randomness into the prompt, effectively reducing text repetition. In preliminary experiments, we compare the performance of Chat-GPT and GPT-4 [1] and find that GPT-4 consistently provides higher quality captions, consistent with [29] . Our methodology enables the synthesis of in-domain captions without the need to access visual data.TTo match the distribution of a target dataset, SSF [26] incorporates scale and shift factors to modulate deep features extracted by a pre-trained model. Specifically, modifies an input token 𝒙 ∈ R 𝑑 by scaling and shifting as follows:EQUATIONwhere 𝜸 ∈ R 𝑑 and 𝜷 ∈ R 𝑑 represent learnable affine transform parameters. ⊙ denotes the dot product, and 𝒛 signifies the adjusted token. However, their scale and shift factors are input-independent, potentially constraining the model's expressive capacity. To address this limitation, we propose an adaptive learning mechanism for scale and shift parameters that tailors to the specific inputs:EQUATION)where 𝜸 𝒙 and 𝜷 𝒙 are input-dependent scale and shift factors, allowing for a more flexible and responsive model adaptation.A straightforward method for determining input-dependent scale and shift factors involves employing an MLP to estimate these factors. However, such an approach introduces a significant number of learnable parameters, which can lead to unstable training and suboptimal performance, especially in scenarios with limited training data. Moreover, there is a distribution shift between synthetic and actual captions, further increasing learning difficulty. To mitigate these challenges, we propose a novel approach that constructs a sparse Mixture of Scale and Shift expert model (MoS 2 ). We define a collection of discrete scale and shift experts, coupled with a lightweight routing network tasked with predicting a probability distribution among these experts. Subsequently, the top-k experts with the highest probabilities are aggregated to formulate the input-dependent scale and shift factors. The MoS 2 architecture aims to reduce the number of parameters to be updated and simplify the estimation of scale and shift factors. As opposed to SSF [26] , our MoS 2 model can adaptively adjust to varying inputs and enhance the model's capacity, thereby improving performance in downstream tasks. The architecture of MoS 2 is depicted in Fig. 2 .The mixture of scale components comprises a routing network, denoted as 𝐺, in conjunction with 𝑛 scale experts, represented as 𝚪 = {𝜸 1 , 𝜸 2 , . . . , 𝜸 𝒏 }, 𝜸 𝒊 ∈ R 𝑑 . We derived the input-dependent scale factor 𝜸 𝒙 from the weighted summation of adaptively selected scale experts. Formally, for the input token 𝒙, the routing network generates an 𝑛-dimensional probability distribution 𝑔(𝒙) across the scale experts:EQUATIONwhere 𝑾 𝒈 ∈ R 𝑑 ×𝑛 signifies the trainable weight of the router, ⊗ denotes matrix multiplication, and 𝝐 refers to Gumbel noise [17] , facilitating weighted sampling from a distribution. It is flexible to choose the appropriate routing network and we choose a single linear layer for simplicity. Subsequently, we identify the top-𝑘 scale experts with the highest probabilities and normalize the probability distribution as follows:EQUATIONwhere 𝑔 𝑛 (𝒙) denotes the normalized distribution, 𝑘 signifies the number of experts chosen per token, and the Norm function adjusts the output by dividing each element by their cumulative sum. The input-adaptive scale factor is then computed via a weighted summation of the selected experts:EQUATION)In a parallel manner, we define a collection of shift experts, 𝑩 = {𝜷 1 , 𝜷 2 , . . . , 𝜷 𝑛 }, and estimate their probability distribution 𝑔 ′ (𝒙) via a shift router, described as:EQUATION)where 𝑾 ′ 𝒈 denotes the trainable weight of the shift router. Subsequently, we identify top-𝑘 shift experts with the highest probabilities and normalize the probability distribution:EQUATION)The input-dependent shift vector is derived through a weighted aggregation of these selected shift experts:EQUATION)Upon determining the input-dependent scale, 𝜸 𝒙 , and shift, 𝜷 𝒙 , we proceed to adjust the hidden representations accordingly as prescribed in Eq. ( 2). It is noteworthy that our approach could converge to SSF [26] if the router matrices 𝑾 𝒈 and 𝑾 ′ 𝒈 are optimized to exhibit repetitive column patterns. We allocate distinct parameters to the routers of scale and shift to the representation capacity of the model. Moreover, by increasing 𝑛 while maintaining 𝑘 constant, we are able to enhance the adaptability of the model without significantly affecting its computational demand. It operates independently per token and is orthogonal to the attention mechanism. During adaptation, we freeze the pre-trained weights, with updates applied exclusively to the MoS 2 parameters. The selective adaptability of MoS 2 enables the model to dynamically adjust its parameters in response to specific inputs, substantially improving the adaptability for downstream applications.T 1 T 2 + +Load Balancing Loss. Our MoS 2 , characterized as a lightweight Mixture of Expert (MoE) architecture, confronts a load-balancing challenge in the absence of regularization mechanisms. Specifically, unregularized routing networks are prone to allocating disproportionate weights to a small subset of experts, regardless of the input. This imbalance is self-reinforcing [30] , with preferred experts receiving enhanced training, thereby increasing their likelihood of selection in subsequent routing decisions. To counteract this imbalance, we introduce a load-balancing loss function designed that penalizes the over-reliance on any single expert. The importance of an expert is quantified as the mean routing probability assigned to that expert across a batch. Our objective is to ensure a balanced utilization of experts by penalizing those with highimportance scores. For a batch consisting of 𝑇 tokens, represented as 𝑿 = {𝒙 1 , 𝑥 2 , . . . , 𝒙 𝑇 }, the load balancing loss, L 𝑙𝑏 , is formulated as follows:EQUATION)where 𝑛 denotes the number of experts, p𝑖 represents the batchaveraged routing probability for expert 𝑖, 𝑝 (𝒙) ∈ R 𝑛 signifies the router probability for token 𝒙, and 𝑞 𝑖 indicates the proportion of tokens dispatched to expert 𝑖, being the indicator function. The load balancing loss ensures that the training process remains balanced, fostering an equitable distribution of learning across all experts. The final load balance loss is computed by averaging across all mixture of scale and shift modules, which we omit it for simplicity. Through the load balancing loss, the training process is calibrated to ensure a balanced, equitable distribution of learning opportunities across all experts, thereby mitigating the risk of over-reliance on a limited subset of the available experts.In light of the substantial costs and effort required for manually annotating paired datasets, our study aims to enhance video captioning through an innovative approach that relies solely on text data. We incorporated our designed MoS 2 layer into the text decoder ℎ, positioned after layer normalization, multi-head attention, and pointwise feed-forward networks. The MoS 2 layer is fine-tuned on our synthesized in-domain captions. Specifically, we employ casual language modeling [36] to train MoS 2 , which involves predicting the current text token conditioned on the preceding tokens. Formally, we minimize the following loss:EQUATION)where 𝜌 represents the parameters of the MoS 2 layer. The overall training loss incorporates both the causal language modeling loss, L 𝑐 , and the load balancing term for MoS 2 , L 𝑏 , is expressed as:EQUATION)with 𝛼 representing the coefficient for the load balancing term. Our methodology, leveraging text-only training data, significantly reduces reliance on paired datasets. The text-only training paradigm, coupled with the incorporation of the MoS 2 layer, offers a promising avenue for advancing video captioning capabilities, without the necessity for extensive manual data annotation. After training, we can combine the visual encoder with the fine-tuned text decoder for video captioning.We integrate our text-only training pipeline and the MoS 2 architecture into BLIP-2 [22] model for video captioning. BLIP-2 [22] connects a frozen visual encoder and a large language model by a lightweight perceiver-based [16] transformer, Q-Former, which facilitates the integration of image encoders with large language models, thereby bootstrapping vision-to-language generative modeling. For video inputs containing multiple frames, we simply concatenate the frame-level visual representations before processing with the Q-Former.  
experiments 4.1 experimental setup: 
 We start from BLIP-2 [22] , a pre-trained image captioning model equipped with ViT-G from EVA-CLIP [9] and OPT-2.7B [59] . Further improvement can be achieved by augmenting the model with a more powerful visual encoder and textual decoder. We evaluate our approach on three popular video captioning benchmarks, including MSRVTT [49] , MSVD [5] , and VATEX [45] . We evaluate the quality of generated captions using four standard captioning evaluation metrics, including BLEU-4 (B4) [33] , METEOR (M) [4] , ROUGE-L (R) [27] , and CIDEr (C) [42] . Additional experimental details are provided in the appendix.Table 2 , Table 3 and Table 4 presents the video captioning results in both few-shot and full-text settings. Our method significantly surpasses the previous state-of-the-art in the few-shot setting, demonstrated by CIDEr score enhancements of +13.6, +11.3, and +3.6 on the MSVD, MSRVTT, and VATEX datasets, respectively. Additionally, compared to the zero-shot outcomes of BLIP-2, our approach leveraging few-shot in-domain text synthesis shows remarkable improvements, with increases in CIDEr scores of +55.4, +10.5, and +10.4 across these datasets. Notably, our few-shot result outperforms the fine-tuning result of Uni-Perceiver [63] on the MSVD dataset. Specifically, we attained a CIDEr score of 142.5, surpassing the score of 131.0 achieved by Uni-Perceiver [63] , which is pretrained on extensive image-text and video-text pairs. These results highlight the effectiveness of our novel few-shot in-domain text synthesis and a mixture of scale and shift tuning methods. Additionally, the minor discrepancies observed between our few-shot and full-text results suggest that our text generation technique successfully captures the critical elements of the original datasets, underscoring the robustness of our methods. In the full-text setting, our approach outperforms the state-of-the-art method DeCap [24] , by a margin of +26.3 on MSRVTT, recording a CIDEr score of 61.1 against 34.8. These findings suggest that our parameter-efficient tuning approach adeptly adapts pre-trained image captioning models for video captioning tasks, effectively narrowing the performance gap between zero-shot and fine-tuning results.We conduct comprehensive ablation analysis on the MSR-VTT [49] dataset with our synthesized corpus. We use mean of BLEU-4 [33] , METEOR [4] , ROUGE-L [27] , and CIDEr [42] as the primary evaluation metric for robustness.Parameter Efficiency. We conduct a comparative analysis of the state-of-the-art parameter-efficient tuning methods, as detailed in Table 5 . Our proposed tuning method, MoS 2 , demonstrates superior performance across all metrics while maintaining competitive trainable parameters, at a reduced parameter count of 10.5M. This denotes a notable advancement in learning efficiency, achieving a delicate balance between parameter count and performance. The SSF [26] method introduces input-independent scale and shift factors, requiring a minimal number of trainable parameters (0.2M), and demonstrates significant parameter efficiency by delivering commendable performance. Conversely, BitFit [55] , which adjusts only the bias term, fails to achieve a performance increase. The LLMaM-Adapter [58] method, which prepends learnable adaptation prompts to word tokens, exhibits enhanced performance with a slightly increased parameter count, underscoring the efficacy of prompts for fine-tuning Transformer models. The above methods are characterized by a limited number of trainable parameters, wherein only trainable vectors are employed. LoRA introduces trainable rank decomposition matrices into each layer of the Transformer architecture, resulting in a considerable increase in the parameter count. Adapter [14] utilizes 52.6M parameters and achieves the second-best result. Full fine-tuning, with the most trainable parameters, does not achieve the best performance, implying that no need to update so many parameters. Similarly, the Adapter incorporates MLP, further increasing the total number of trainable parameters to 52.6 million, achieving the second-best performance. Notably, full fine-tuning, which utilizes the highest number of trainable parameters, does not yield the best results, suggesting that updating an extensive array of parameters is not necessary for optimal performance.The quantity of expert models plays a crucial role in MoE-based models. We attempt to increase the number of scale and shift expert models, as shown in Table 6 . Our experimental findings indicate a direct correlation between an increase in the number of expert models and enhanced model performance. Nonetheless, considering the parameter efficiency, we ultimately select 32 expert models, opting not to expand further.Top-𝑘 Routing. Presented in Table 7 , we investigate the effect of varying the number of expert models (𝑘) selected by each token. Our findings indicate that increasing 𝑘 does not improve performance. Therefore, we set 𝑘 = 2 for subsequent analyses.Load Balance Loss Coefficient. As load balancing significantly influences the performance of MoE models, we incorporate a load balance loss term into our method. As demonstrated in Table 8 , an appropriate load balance loss coefficient, specifically set at 𝛼 = 0.1, notably improves performance by +3.3. This finding highlights the Different Components. We conduct ablation experiments on the two expert models, specifically analyzing the effects of removing the mixture of scale or the mixture of shift components. As indicated in Table 9 , omitting the mixture of shift results in a significant performance decrease of -5, whereas eliminating the mixture of scale leads to a marginal reduction of -0.2. We hypothesize that the more pronounced effect of omitting the mixture of shift could be attributed to its incorporation as residuals to the hidden states, which likely stabilizes the gradient and simplifies the training process. In contrast, the mixture of scale directly multiplies the hidden states, potentially introducing greater variability in model training.Insertion Layers. We investigate the distribution of MoS 2 layers within Transformer models to determine their optimal placement. Our findings, presented in Table 10 , indicate that incorporating MoS 2 into the middle eight layers of the Transformer significantly enhances performance relative to other configurations such as the initial or final eight layers, or an interleaved arrangement. Additionally, we observed that an increase in the total number of layers does not consistently improve performance; rather, the most effective enhancement is achieved through the middle eight layers.We analyze the placement of components within the Transformer block. As demonstrated in Table 11 , removing MoS 2 subsequent to layer normalization substantially degrades performance, more so than its removal after the feed-forward network (FFN). Conversely, eliminating MoS 2 subsequent to the attention layers has a negligible impact on performance; nevertheless, we maintain MoS 2 in this position by default.We evaluate the impact of increasing the number of synthetic text on model performance, as depicted in Fig. 3 . Our results indicate a consistent enhancement in model efficacy correlating with increased text quantities, suggesting potential for further gains. This trend demonstrates that the synthetic texts generated by our approach effectively encapsulate critical aspects of the target domain knowledge. 
conclusion: 
 This work presents a refined text-only training pipeline that significantly enhances the usability and performance of domain-specific captioning. By leveraging GPT-4 to create high-quality, domainspecific captions, we successfully reduce dependency on extensive paired data, thereby reducing associated annotation costs and labor. We also introduce the Mixture of Scale and Shift Experts (MoS 2 ) architecture, which effectively mitigates distribution shifts between synthetic and real captions, thereby enhancing model flexibility. MoS 2 outperforms state-of-the-art methods in both few-shot and full-text video captioning across three established benchmarks. Remarkably, MoS 2 achieves near full-text model performance with only minimal real training examples (10-shot), demonstrating the viability of using large language models (LLMs) to generate training data and underscoring the potential of parameter-efficient learning techniques in domain-specific applications.  
: 
 have demonstrated that large-scale pre-training can significantly enhance zero-shot captioning performance. Despite notable advancements in zero-shot captioning methods [22, 24, 46] , the performance of zero-shot captioning methods is largely lower compared to models [6, 13, 43, 48] fine-tuned with targetdomain data. However, adapting vision-language models for each upcoming new domain is cumbersome, which requires fine-tuning the models with domain-specific datasets containing visual-text pairs. Besides expensive computational costs for models fine-tuning, building large-scale training datasets for captioning is also timeconsuming and labor-intensive.To improve the captioning performance on the target domain with lower training costs, some recent researches [11, 24, 34, 52] attempt to fine-tune the text decoder of the vision-language models using only textual data. DeCap [24] first projects visual and text embeddings into the CLIP space [35] and optimizes only the text decoder on a dedicated text corpus for image captioning. Subsequent studies [11, 34, 52, 57] further enhance the efficacy of textonly training by employing contrastive learning based models [35] . Current text-only training captioning works only focus on image captioning. These works extract textual training data from webscraped image-text pairs or corpora tailored for specific target domains, potentially limiting the effectiveness and broad applications. Although only textual descriptions (i.e., video captions) are required for training, we believe extracting high-quality text data for video captioning is even more challenging.To cope with the issue of data collection, this work investigates video captioning from a different perspective. Different from prior works [6, 13, 28, 43, 48] fine-tune models with large-scale videotext pairs of the target domain, we aim to improve the video captioning performance on the target domain with only a few (e.g., 10-shot) textual training examples of the target domain. Inspired by prior image captioning works [62] , we try to generate a domainspecific text-only corpus to achieve text-only training. Since LLMs (e.g., GPT-4 [1] ) have superior instruction-following and contextual reasoning capabilities, we design few-shot instruction-following prompts request GPT-4 [1] to generate captions that closely align with the target domain. Compared to crawling text descriptions from the web [10] , there is no need to manually craft complex filtering rules. In addition, the captions generated by LLMs are more similar to human-like styles, of higher-quality, free from grammatical errors and meaningless symbols. LLMs offer precise control over the text generation process through prompt engineering, which facilitates adherence to specific guidelines or themes.Currrent pre-trained captioning models [2, 54? ] typically incorporate large vision encoders [9, 35] and employ LLMs [58, 59] as the text decoders. Although only the text decoder is fine-tuned in our text-only video captioning framework, optimizing all parameters within a large-scale text decoder remains costly. To further improve the training efficiency and preserve the foundational knowledge learned from the large-scale pre-training, we further explore parameter-efficient transfer learning, where a minimal set of parameters within the text decoder is updated.Existing parameter-efficient fine-tuning techniques, though effective, typically assume minor differences between training and target domains. However, our approach, which prompts LLMs to generate domain-specific text, still faces challenges stemming from the distribution shift between synthetic and actual captions. To counter this, we develop a novel architecture, the Mixture of Scale and Shift Experts (MoS 2 ), which incorporates input-dependent adaptability into the model's parameters. A lightweight routing network estimates a probability distribution over a collection of experts, determining the allocation of tokens to appropriate experts. This dynamic adjustment mechanism allows for specific responses to input features, thereby enhancing the model's ability to handle data variations. Unlike traditional Mixture-of-Experts (MoE) networks that employ dense feed-forward networks with a high parameter count, our method simplifies the expert components to scale or shift factors, which significantly reduces the number of parameters, aligning with our goal of efficient training. Consequently, adaptive scale and shift factors are applied to individual tokens, enhancing flexibility and overall performance. By employing MoS 2 , our approach not only customizes model responses to input variations, effectively addressing the distribution shift between synthetic and actual captions but also significantly reduces the number of learnable parameters, allowing for more efficient adaptations. Following the fine-tuning of the text decoder, we integrate it with the associate vision encoder for enhanced video captioning.This paper presents a parameter-efficient text-only video captioning method trained with synthetic target-domain data. Extensive experiments compared to state-of-the-art methods validate the superiority of our proposed text-only video captioning method.The key contributions of this paper are summarized as follows:• We conduct text-only training for video captioning for the first time and validate its effectiveness. We design in-context prompts to guide GPT-4 in generating diverse, high-quality domain-specific captions, which streamlines the construction of domain-specific corpora, minimizing reliance on paired data and associated costs. • We propose a novel mixture of scale and shift architecture to address the distribution shift between synthetic and actual captions, which not only enhances the flexibility and performance of the model but also significantly reduces the number of parameters requiring updates. • On three commonly used benchmarks, MoS  
 
 Appendix:  


Decision: The paper introduces a text-only training pipeline for video captioning, reducing dependency on paired data. After author rebuttal, most of the reviewers acknowledged that the motivation is clear and the paper includes sufficient experiments. However, there are typos and formatting errors, and some expressions are not precise. Overall, we would like to recommend Accept (Poster) of the paper.

Reviews: review: paper_summary: paper_summary: The author proposed a text-only training pipeline for video captioning that reduces the dependency on paired data. They also introduce the Mixture of Scale and Shift Experts to mitigate distribution shifts between synthetic and real captions.
strength_weakness: strength_weakness: 1. The authors conduct text-only training for video captioning which not only is parameter-efficient but also minimizes reliance on paired data
2. The authors design a mixture of scale and shift architecture to address the distribution shift between synthetic and actual captions
3. The method achieves great performance on three established benchmarks
4. The motivation is clear and the idea is easy to follow
limitations: limitations: 1. Poor presentation. Line 330 and line 333 seem to be duplicated. Line 110: captioning models [2, 54? ]
2. Could the synthetic data used by the authors also improve DeCap's performance?
paper_summary: paper_summary: The paper introduces a novel method for text-only video captioning, named MoS2. The authors leverage a large language model to create high-quality and high-diversity captions in the target domain for training. The proposed MoS2, a Mixture of Scale and Shift experts, allows a large-scale pre-training model for parameter-efficient fine-tuning. It significantly outperforms existing methods in three video captioning datasets.
strength_weakness: strength_weakness: 1.	The proposed MoS2 effectively addresses the gap between generated data and target data. It enhances the training efficiency of LMM in domain transfer learning.
2.	The paper is well-written and clearly illustrated.
3.	The ablation study in the paper is comprehensive.
limitations: limitations: 1.	The authors could consider adding some visualization results (e.g. case study, caption distribution before and after MoS2 adjustment).
2.	The authors should correct the typos and formatting errors in the paper.
         e.g.
         * line109 "[2, 54? ]"
         * line381 "TTo match"
paper_summary: paper_summary: Video captioning is a complex task that generally needs video-text paired data for training, but manually creating coherent text descriptions for videos is a labor-intensive and time-consuming process. The paper proposes a video captioning pipeline that begins with data expansion using GPT-4, designs a PEFT method called MoS2 for text-only training, and does video captioning inference. However, there are still some issues in the paper that need to be addressed.
strength_weakness: strength_weakness: 1. The pipeline proposed in this paper is particularly clear and easy to follow, which makes it accessible even to those who may not be experts in the field. This straightforward approach ensures that researchers can understand and apply the methods discussed.
2.	The paper writing is commendable for its clarity and straightforwardness. The authors use simple and direct language, making complex concepts more understandable.
3. The paper conducts sufficient experiments to verify the effectiveness of the proposed method for video captioning.
limitations: limitations: 1. The proposed PEFT method, MoS², is used to fine-tune a strong text decoder. I suggest comparing it directly with common PEFT methods like LoRA or adapters across additional NLP tasks, such as text generation.
2. I question the relevance of this paper to video captioning. In fact, BLIP2 is inherently suited for image captioning, not video captioning. I recommend testing it on image captioning tasks. Moreover, in the original BLIP2, a 224x224 resolution image corresponds to one cls token and 256 patch tokens, which interact with 32 learnable tokens. However, if tokens from different frames are merely stacked together in video captioning, it could neglect essential frame-wise temporal modeling.
3. Line 157 in the paper "We conduct text-only training for video captioning for the first time and validate its effectiveness" is not precise. Text-only methods such as DeCap[1] and MultiCap[2] have previously conducted experiments for video captioning on the MSR-VTT and VATEX datasets,  indicating that this approach is not novel.
4. On line 382, "TTo match" should be corrected to "To match" to maintain the professional quality of the manuscript.

Reference:

 [1] Li, Wei, et al. "DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training." The Eleventh International Conference on Learning Representations. 2022.

   [2] Yang, Bang, et al. "MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning." The 61st Annual Meeting Of The Association For Computational Linguistics. 2023.