Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations
Year: 2023

Abstract:
Due to the huge amount of parameters, finetuning of pretrained language models (PLMs) is prone to overfitting in the low resource scenarios. In this work, we present a novel method that operates on the hidden representations of a PLM to reduce overfitting. During fine-tuning, our method inserts random autoencoders between the hidden layers of a PLM, which transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. The autoencoders are plugged out after fine-tuning, so our method does not add extra parameters or increase computation cost during inference. Our method demonstrates promising performance improvement across a wide range of sequenceand token-level low-resource NLP tasks. Our code is available at https://github.com/DAMONLP-SG/MVCR.

heading: 1 Introduction
text: Fine-tuning pretrained language models (PLMs) (Devlin et al., 2019; Conneau and Lample, 2019; Liu et al., 2020) provides an efficient way to transfer knowledge gained from large scale text corpora to downstream NLP tasks, which has achieved state-of-the-art performance on a wide range of tasks (Yang et al., 2019; Yamada et al., 2020; Chi et al., 2021; Sun et al., 2021; Zhang et al., 2021b,a; Chia et al., 2022; Tan et al., 2022; Zhou et al., 2022b). However, most of the PLMs are designed for general purpose representation learning (Liu et al., 2019; Conneau et al., 2020), so the learned representations unavoidably contain abundant features irrelevant to the downstream tasks. Moreover, the PLMs typically possess a huge amount of parameters (often 100+ millions) (Brown et al., 2020;
∗Equal contribution, order decided by coin flip. Linlin Liu and Xingxuan Li are under the Joint Ph.D. Program between Alibaba and Nanyang Technological University.
†Corresponding author.
Min et al., 2021), which makes them more expressive compared with simple models, and hence more vulnerable to overfitting noise or irrelevant features during fine-tuning, especially in the low-resource scenarios.
There has been a long line of research on devising methods to prevent large neural models from overfitting. The most common ones can be roughly grouped into three main categories: data augmentation (DeVries and Taylor, 2017; Ding et al., 2020; Liu et al., 2021; Feng et al., 2021; Xu et al., 2023; Zhou et al., 2022c), parameter/activation regularization (Krogh and Hertz, 1991; Srivastava et al., 2014; Lee et al., 2020; Li et al., 2020) and label smoothing (Szegedy et al., 2016; Yuan et al., 2020), which, from bottom to top, operates on data samples, model parameters/activations and data labels, respectively.
Data augmentation methods, such as backtranslation (Sennrich et al., 2016) and masked prediction (Bari et al., 2021), are usually designed based on our prior knowledge about the data. Though simple, many of them have proved to be quite effective. Activation (hidden representation) regularization methods are typically orthogonal to other methods and can be used together to improve model robustness from different aspects. However, since the neural models are often treated as a black box, the features encoded in hidden layers are often less interpretable. Therefore, it is more challenging to apply similar augmentation techniques to the hidden representations of a neural model.
Prior studies (Yosinski et al., 2015; Allen-Zhu and Li, 2023; Fu et al., 2022) observe that neural models trained with different regularization or initialization can capture different features of the same input for prediction. Inspired by this finding, in this work we propose a novel method for hidden representation augmentation. Specifically, we insert a set of randomly initialized autoencoders (AEs) (Rumelhart et al., 1985; Baldi, 2012) between the
4799
layers of a PLM, and use them to capture different features from the original representations, and then transform them into Multi-View Compressed Representations (MVCR) to improve robustness during fine-tuning on target tasks. Given a hidden representation, an AE first encodes it into a compressed representation of smaller dimension d, and then decodes it back to the original dimension. The compressed representation can capture the main variance of the data. Therefore, with a set of AEs of varying d, if we select a random one to transform the hidden representations in each fine-tuning step, the same or similar input is compressed with varying compression dimensions. And the upperlevel PLM layers will be fed with more diverse and compressed representations for learning, which illustrates the “multi-view” concept. We also propose a tailored hierarchical AE to further increase representation diversity. Crucially, after fine-tuning the PLM with the AEs, the AEs can be plugged out, so they do not add any extra parameters or computation during inference.
We have designed a toy experiment to help illustrate our idea in a more intuitive way. We add uniform random Gaussian noise to the MNIST (LeCun et al., 1998) digits and train autoencoders with different compression ratios to reconstruct the noisy input images. As shown in Fig. 1, the compression dimension d controls the amount of information to be preserved in the latent space. With a small d, the AE removes most of the background noise
and preserves mostly the crucial shape information about the digits. In Fig. 1b, part of the shape information is also discarded due to high compression ratio. In the extreme case, when d = 1, the AE would discard almost all of the information from input. Thus, when a small d is used to compress a PLM’s hidden representations during fine-tuning, the reconstructed representations will help: (i) reduce overfitting to noise or task irrelevant features since the high compression ratio can help remove noise; (ii) force the PLM to utilize different relevant features to prevent it from becoming overconfident about a certain subset of them. Since the AE-reconstructed representation may preserve little information when d is small (Fig. 1b), the upper layers are forced to extract only relevant features from the limited information. Besides, the shortcut learning problem (Geirhos et al., 2020) is often caused by learning features that fail to generalize, for example relying on the grasses in background to predict sheep. Our method may force the model to use the compressed representation without grass features. Therefore, it is potentially helpful to mitigate shortcut learning. As shown in Fig. 1(d), with a larger d, most information about the digits can be reconstructed, and noises also start to appear in the background. Hence, AEs with varying d can transform a PLM’s hidden representations into different views to increase diversity.
We conduct extensive experiments to verify the effectiveness of MVCR. Compared to many strong baseline methods, MVCR demonstrates consistent performance improvement across a wide range of sequence- and token-level tasks in low-resource adaptation scenarios. We also present abundant ablation studies to justify the design. In summary, our main contributions are:
• Propose a novel method to improve low-resource fine-tuning, which leverages AEs to transform representations of a PLM into multi-view compressed representations to reduce overfitting.
• Design an effective hierarchical variant of the AE to introduce more diversity in fine-tuning.
• Present a plug-in and plug-out fine-tuning approach tailored for our method, which does not add extra parameter or computation during inference.
• Conduct extensive experiments to verify the effectiveness of our method, and run ablation studies to justify our design.

heading: 2 Related Work
text: Overfitting is a long-standing problem in large neural model training, which has attracted broad interest from the research communities (Sarle et al., 1996; Hawkins, 2004; Salman and Liu, 2019; Santos and Papa, 2022; Hu et al., 2023; He et al., 2021b). To better capture the massive information from large-scale text corpora during pretraining, the PLMs are often over-parameterized, which makes them prone to overfitting. The commonly used methods to reduce overfitting can be grouped into three main categories: data augmentation, parameter regularization, and label smoothing.
Data augmentation methods (DeVries and Taylor, 2017; Liu et al., 2021; Feng et al., 2021) are usually applied to increase training sample diversity. Most of the widely used methods, such as synonym replacement (Karimi et al., 2021), masked prediction (Bari et al., 2021) and back-translation (Sennrich et al., 2016), fall into this category. Label smoothing methods (Szegedy et al., 2016; Yuan et al., 2020) are applied to the data labels to prevent overconfidence and to encourage smoother decision boundaries. Some hybrid methods, like MixUp (Zhang et al., 2018; Verma et al., 2019), are proposed to manipulate both data sample and label, or hidden representation and label. Most parameter/hidden representation regularization methods are orthogonal to the methods discussed above, so they can be used as effective complements.
Neural models are often treated as a black box, so it is more challenging to design efficient parameter or hidden representation regularization methods. Existing methods mainly focus on reducing model expressiveness or adding noise to hidden representations. Weight decay (Krogh and Hertz, 1991) enforces L2 norm of the parameters to reduce model expressiveness. Dropout (Srivastava et al., 2014) randomly replaces elements in hidden representations with 0, which is believed to add more diversity and prevent overconfidence on certain features. Inspired by dropout, Mixout (Lee et al., 2020) stochastically mixes the current and initial model parameters during training. Mahabadi et al. (2021) leverage variational information bottleneck (VIB) (Alemi et al., 2017) to help models to learn more concise and task relevant features. However, VIB is limited to regularize last layer sequencelevel representations only, while our method can be applied to any layer, and also supports token-level tasks like NER and POS tagging.

heading: 3 Methodology
text: We first formulate using neural networks over the hidden representation of different layers of deep learning architectures as effective augmentation modules (§3.1) and then devise a novel hierarchical autodencoder (HAE) to increase stochasticity and diversity (§3.2). We utilise our novel HAE as a compression module for hidden representations within PLMs, and introduce our method MultiView Compressed Representation (MVCR) (§3.3). We finally discuss the training, inference, and optimization of MVCR. An overview of MVCR is presented in Fig. 2.

heading: 3.1 Generalizing Neural Networks as Effective Augmentation Modules
text: Data augmentation (Simard et al., 1998) aims at increasing the diversity of training data while preserving quality for more generalized training (Shorten and Khoshgoftaar, 2019). It can be formalized by the Vicinal Risk Minimization principle (Chapelle et al., 2001), which aims to enlarge the support of the training distribution by generating new data points from a vicinity distribution around each training example (Zhang et al., 2018). We conjecture that shallow neural networks can be used between the layers of large-scale PLMs to construct such vicinity distribution in the latent space to facilitate diversity and generalizability.
We consider a neural network g(·), and denote its forward pass F (·) for an input x with F (x) = g(x). We denote a set of M such networks with G = {g1(·), . . . , gM (·)}, where each candidate network gi(·) outputs a different "view" of a given input. We treat G as a stochastic network and define a stochastic forward pass FS(·,G), where a candidate network gi(·) is randomly chosen from the pool G in each step, enabling diversity due to different non-linear transformations. Formally, for an input x, we obtain the output o of network gi(·) using FS as,
o = FS(x,G) = gi(x), i ∈ {1, . . . ,M} (1)
For a chosen candidate network gi, the output o represents a network dependent “view” of input x.
We now formalize using g(·) over the hidden representation of large-scale PLMs for effective generalization. Let f(·) denote any general transformerbased PLM containing N hidden layers with fn(·) being the n-th layer and hn being the activations at that layer for n ∈ {1, . . . , N}, and let h0 denote
the input embeddings. We consider a set of layers L ⊂ {1, . . . , N} where we insert our stochastic network G for augmenting the hidden representations during the forward pass. To this end, we substitute the standard forward pass of the PLM f(·) during the training phase with the stochastic forward pass FS(·,G). Formally,
hn = { FS(fn(hn−1),Gn), n ∈ L fn(hn−1), n /∈ L
(2)
We now devise a novel hierarchical autoencoder network which can be effectively used for augmenting hidden representations of large-scale PLMs.

heading: 3.2 Stochastic Hierarchical Autoencoder
text: Autoencoders (AEs) are a special kind of neural networks where the input dimension is the same as the output dimension (Rumelhart et al., 1985). For an input x ∈ Rd, we define a simple autoencoder AEd,d̂(·) with compression dimension d̂ as a sequential combination of a feed-forward downprojection layer Dd,d̂(·) and an up-projection layer Ud̂,d(·). Given input x, the output o ∈ Rd of the autoencoder can be represented as:
o = AEd,d̂(x) = U(D(x)) (3)
Hierarchical Autoencoder (HAE) We extend a standard autoencoder to a hierarchical autoencoder and present its overview in Fig. 2 (Hierarchical AE). We do this by inserting a sub-autoencoder AEd̂,d′(·) with compression dimension d′ within an autoencoder AEd,d̂(·), where d′ < d̂. For AEd̂,d′(·), the down- and up-projection layers are denoted with D′
d̂,d′ (·) and U ′ d′,d̂ (·), respectively.
We obtain output o ∈ Rd for hierarchical autodencoder HAEd,d̂,d′(·) as,
o=HAEd,d̂,d′(x)=U(AEd̂,d′(D(x)))
=U(U ′(D′(D(x)))) (4)
Note that, HAEs are different from the AEs having multiple encoder and decoder layers, since it also enforces reconstruction loss on the outputs of D(x) and U ′(D′(D(x))) as expressed in Eq. 4. Thus HAEs can compress representations step by step, and provides flexibility to reconstruct the inputs with or without the sub-autoencoders. By sharing the outer layer parameters of a series of inner AEs, HAE can introduce more diversity without adding significant parameters and training overhead.
Stochastic Hierarchical Autoencoder We use a stochastic set of sub-autoencoders Ed̂ within an HAE to formulate a stochastic hierarchical autoencoder, where d̂ is the input dimension of the subautoencoders. While performing the forward pass of the stochastic hierarchical autoencoder, we randomly choose one sub-autoencoder AEd̂,di ∈ Ed̂ within the autoencoder AEd,d̂. We set the compression dimension di = d̂/2. Hence, for an input x ∈ Rd, the output o ∈ Rd of a stochastic hierarchical autoencoder HAES
d,d̂ is given by
o = HAESd,d̂(x)
=
{ U(AEd̂,di(D(x))), z > 0.3
U(D(x)), z ≤ 0.3 (5)
where z is uniformly sampled from the range [0, 1], and AEd̂,di ∈ Ed̂ is randomly selected in each step.
So for 30% of the time, we do not use the subautoencoders. These randomness introduces more diversity to the generated views to reduce overfitting. For the stochastic HAE, we only compute the reconstruction loss between x and o, since this also implicitly minimizes the distance between AEd̂,di(D(x)) and D(x) in Eq. 5. See §A.2 for detailed explanation. In the following sections, we use HAE to denote stochastic hierarchical autoencoder. We also name the hyper-parameter d̂ as HAE compression dimension.

heading: 3.3 Multi-View Compressed Representation
text: Autoencoders can represent data effectively as compressed vectors that capture the main variability in the data distribution (Rumelhart et al., 1985). We leverage this capability of autoencoders within our proposed stochastic networks to formulate MultiView Compressed Representation (MVCR).
We give an illustration of MVCR in Fig. 2. We insert multiple stochastic HAEs (§3.2) between multiple layers of a PLM (shown for one layer in the figure). Following §3.1, we consider the Transformer-based PLM f(·) with hidden dimension d and N layers. At layer n, we denote the set of stochastic HAEs with HAES,n = {HAES,n
d,d̂1 ,HAES,n d,d̂2 , . . . ,HAES,n d,d̂M }, where M is
total the number of HAEs. To prevent discarding useful information in the compression process and for more stable training, we only use them for 50% of the times. We modify the forward pass of f(·) with MVCR, denoted by FMVCR(·,HAES,n) to obtain the output hn for an input hn−1 as
hn = F MVCR(hn−1,HAE S,n)
=
{ FS(fn(hn−1),HAE
S,n), z ≤ 0.5 fn(hn−1), z > 0.5
(6)
where z is uniformly sampled from the range [0, 1]. Following Eq. 2, we finally define MVCR(·,HAES) for a layer set L of a PLM to compute the output hn using stochastic HAEs as
hn = MVCR(hn−1,HAE S)
=
{ FMVCR(fn(hn−1),H
S,n), n ∈ L fn(hn−1), n /∈ L
(7)
Note that MVCR can be used either at layer-level or token-level. At layer-level, all of the hidden (token) representations in the same layer are augmented with the same randomly selected HAE in each training step. At token-level, each token representation
can select a random HAE to use. We show that token-level MVCR performs better than layer-level on account of more diversity and stochasticity (§5) and choose this as the default setup.
Network Optimization We use two losses for optimizing MVCR. Firstly, our method do not make change to the task layer, so the original taskspecific loss Ltask is used to update the PLM, task layer, and HAE parameters. We use a small learning rate αtask to minimize Ltask. Secondly, to increase training stability, we also apply reconstruction loss LMSE to the output of HAEs to ensure the augmented representations are projected to the space not too far from the original one. At layer n, we have
LMSE = 1 M × L
M∑
m=1
L∑
i=1
(hin −HAEm(hin))2 (8)
where M is the number of HAEs on each layer, and L is the PLM input sequence length (in tokens). We use a larger learning rate αMSE to optimize LMSE since the HAEs are randomly initialized. This reconstruction loss not only increases training stability, but also allows us to plug-out the HAEs during inference, since it ensures that the generated views are close to the original hidden representation.

heading: 4 Experiments
text: In this section, we present the experiments designed to evaluate our method. We first describe the baselines and training details, followed by the evaluation on both the sequence- and token-level tasks across six (6) different datasets.
Baselines We compare our method with several widely used parameter and hidden representation regularization methods, including Dropout (Srivastava et al., 2014), Weight Decay (WD) (Krogh and Hertz, 1991) and Gaussian Noise (GN) (Kingma and Welling, 2013).1 Some more recent methods are also included for more comprehensive comparison, such as Adapter (Houlsby et al., 2019), Mixout (Lee et al., 2020), as well as the variational information bottleneck (VIB) based method proposed by Mahabadi et al. (2021). 2 More information
1We use the same setting as (Mahabadi et al., 2021) to produce these baseline results.
2Some of our reproduced VIB results are lower than the results reported in VIB paper though we use the exact same code and hyper-parameters released by the authors. To ensure a fair comparison, we report the average result of the same three random seeds for all experiments.
about these baselines can be found in §A.3. In this paper, we focus on regularization methods for hidden representation. Thus, we do not compare with data augmentation methods which improve model performance from a different aspect.
Training Details Same as Mahabadi et al. (2021), we fine-tune BERTbase (Devlin et al., 2019) for the sequence-level tasks. The token-level tasks are multilingual, so we tune XLM-Rbase (Conneau et al., 2020) on them. We use MVCRl to denote our methods, where l is the PLM layer that we insert HAEs. For example in MVCR1, the HAEs are inserted after the 1st transformer layer. On each layer, we
insert three HAEs with compressed representation dimensions 128, 256 and 512, respectively. All HAEs are discarded during inference. For each experiment, we report the average result of 3 runs.
For each dataset, we randomly sample 100, 200, 500 and 1000 instances from the training set to simulate the low-resource scenarios. The same amount of instances are also sampled from the dev sets for model selection. The full test set is used for evaluation. More details can be found in §A.5.

heading: 4.1 Results of Sequence-Level Tasks
text: For the sequence-level tasks, we experiment with two natural language inference (NLI) benchmarks, namely SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018), and two text classification tasks, namely IMDb (Maas et al., 2011) and Yelp (Zhang et al., 2015). Since MNLI has matched and mismatched versions, we report the results on each of them separately.
We present the experimental results for the NLI tasks in Table 1, and text classification tasks in Table 2. MVCR1 and MVCR12 are our methods that insert HAEs after the 1st and 12th transformer layer, respectively.3 We have the following observations: (a) MVCR1 and MVCR12 consistently outperform all the baselines in the low-resource settings, demonstrating the effectiveness of our method. (b) In very low-resource settings such as with 100 and 200 training samples, MVCR1 often performs better than MVCR12. This indicates that bottom-layer hidden representation augmentation is more efficient than top-layer regularization on extreme low-resource data. We attribute this to the fact that the bottom-layer augmentation impacts more parameters on the top layers. (c) We observe that MVCR outperforms strong baselines, such as VIB. We believe this is due to the fact that MVCR acts as the best-of-both-worlds. Specifically, it acts as a data augmentation module promoting diversity while also acts as a lossy compression module that randomly discards features to prevent overfitting.

heading: 4.2 Results of Token-Level Tasks
text: For token-level tasks, we evaluate our methods on WikiAnn (Pan et al., 2017) for NER and Universal Dependencies v2.5 (Nivre et al., 2020) for POS tagging. Both datasets are multilingual, so we conduct experiments in the zero-shot cross-lingual setting,
3More details about the hyper-parameter search space for insertion layers can be found in §A.6.
where all models are fine-tuned on English first and then evaluated on the other languages directly.
As shown in Table 3, our methods are also proven to be useful on the token-level tasks. MVCR2,12 is our method that inserts HAEs after both the 2nd and 12th transformer layers. Comparing the results of MVCR2 and MVCR2,12, we can see adding one more layer of HAEs indeed leads to consistent performance improvement on WikiAnn. However, MVCR2 performs better on the POS tagging task. We also observe that Mixout generally does not perform well when the number of training samples is very small. Detailed analysis about HAE insertion layers is presented in §5.

heading: 5 Analysis and Ablation Studies
text: To better understand the improvements obtained by MVCR, we conduct in-depth analysis and ablation studies. We experiment with both sequence- and token-level tasks by sampling 500 instances from 3 datasets, namely MNLI, IMDb and WikiAnn.
• Insertion Layer(s) of MVCR Based on (Clark et al., 2019), each layer in BERT captures different
type of information, such as surface, syntactic or semantic information. Same as above, we fix the dimensions of the three HAEs on each layer to 128, 256 and 512. To comprehensively analyse the impact of adding MVCR after different layers, we select two layers for bottom, middle and top layers. More specifically, {1, 2} for bottom, {6, 7} for middle and {11, 12} for top layers.
As shown in Fig. 3, adding MVCR after layer 1 or 12 achieves the best performance than the other layers. This result can be explained in the way of different augmentation methods (Feng et al., 2021). By adding MVCR after layer 1, the module acts as a data generator which creates multi-view representation inputs to all the upper layers. While adding MVCR after layer 12, which is more relevant to the downstream tasks, the module focuses on preventing the task layer from overfitting (Zhou et al., 2022a). Apart from one layer, we also experiment with adding MVCR to a combination of two layers (Table 4 and Table 5). For token-level task, adding MVCR to both layer 2 and 12 performs even better than one layer. However, it does not help improve the performance on sequence-level tasks. Similar to the β parameter in VIB (Mahabadi et al., 2021) which controls the amount of random noise added to the representation, the number of layers in MVCR controls the trade-off between adding more variety to the hidden representation and keeping more information from the original representation. And the optimal trade-off point can be different for sequence- and token-level tasks.
• Number of HAEs in MVCR We analyse the impact of the number of HAEs in MVCR on the diversity of augmenting hidden representations. We fix the compression dimension of HAE’s outer en-
coder to 256, and only insert HAEs to the bottom layers, layer 1 for MNLI and IMDb, and layer 2 for WikiAnn. As shown in Fig. 4, the performance improves with an increasing number of HAEs, which indicates that adding more HAEs leads to more diversity and better generalization. However, the additional performance gain is marginal after three HAEs. This is probably because without various compressed dimensions, the variety only comes from different initialization of HAEs which can only improve the performance to a limited extent. Hence, we fix the number of HAEs to three for other experiments, including the main experiments.
• Diversity of HAE Dimensions The compression dimensions of HAEs control the amount of information to be passed to upper PLM layers, so using HAEs of varying dimensions may help generate more diverse views during training. To analyse the impact of the compression dimension diversity, we run experiments of three types of combinations: “aaa”, “aab” and “abc”, which contains one, two and three unique dimensions respectively. We compute the average performance of the HAEs with dimensions {32,32,32} to {512,512,512} for “aaa”. We sample the dimensions for “aab” and “abc” since there are too many possible combinations.4 As we can see from Fig. 5, “abc” consistently outperforms “aab”, while “aab” consistently outperforms “aaa”, which indicates increasing compression dimension diversity can help further improve model generalization.
• HAE vs. AE vs. VAE HAEs in MVCR serve as a bottleneck to generate diverse compressed views of the original hidden representations. There are many other possible alternatives for HAE, so we replace HAE with the vanilla AE and variational autoencoder (VAE) (Kingma and Welling, 2014) for comparison. The results in Fig. 6 show that
4See §A.7 for more details about the combinations.
HAE consistently outperforms AE and VAE.
• Token-Level vs. Layer-Level MVCR In our method, the selection of random HAE can be on token-level or layer-level. For token-level MVCR, each token in the layer randomly selects an HAE from the pool, with the parameters shared within the same layer. Compared with layer-level MVCR, token-level MVCR adds more variety to the model, leading to better results as observed in Fig. 7.
• More Results and Analysis We conduct experiments to compare the training overhead of MVCR
with other baselines. MVCR has slightly longer training time yet shorter time for convergence. We also experiment with inference with or without MVCR, which results in very close performance. Both experiments can be found in §A.7.

heading: 6 Conclusions
text: In this work, we have proposed a novel method to improve low-resource fine-tuning robustness via hidden representation augmentation. We insert a set of autoencoders between the layers of a pretrained language model (PLM). The layers are randomly selected during fine-tuning to generate more diverse compressed representations to prevent the top PLM layers from overfitting. A tailored stochastic hierarchical autoencoder is also proposed to help add more diversity to the augmented representations. The inserted modules are discarded after training, so our method does not add extra parameter or computation cost during inference. Our method has demonstrated consistent performance improvement on a wide range of NLP tasks.
Limitations
We focus on augmenting the hidden representations of a PLM. Thus most of our baselines, such as dropout (Srivastava et al., 2014) and variational information bottleneck methods (Mahabadi et al., 2021), do not require unlabeled data. For a fair comparison, we assume that the unlabeled data is not available. Therefore, only the limited labeled training set are used to train the autoencoders in our experiments. However, such unlabeled generalor in-domain data (e.g., Wikipedia text) are easy to obtain in practice, and can be used to pre-train the autoencoders with unsupervised language modeling tasks, which may help further improve the performance. We leave it for future work.
Ethical Impact
Deep learning has demonstrated encouraging performance on a wide range of tasks during the past few years. However, neural models are data hungry, which usually requires a large amount of training data to achieve reasonable performance. It is expensive and time consuming to annotate a large amount of data. Pretrained language models (PLMs) (Devlin et al., 2019; Conneau and Lample, 2019; Liu et al., 2020) have been proven to be useful to transfer knowledge from massive unlabeled text to downstream tasks, but they are also
prone to overfitting during fine-tuning due to overparameterization. In this work, we propose a novel method to help improve model robustness in the low-resource scenarios, which is part of the attempt to reduce neural model reliance on the labeled data, and hence reduce annotation cost. Our method has also demonstrated promising performance improvement on cross-lingual NLP tasks, which is also an attempt to break the language barrier and allow a larger amount of population to benefit from the advance of NLP techniques.

heading: Acknowledgements
text: This research is supported, in part, by Alibaba Group through Alibaba Innovative Research (AIR) Program and Alibaba-NTU Singapore Joint Research Institute (JRI), Nanyang Technological University, Singapore.

heading: A Appendix
text: A.1 Dataset Usage
SNLI and Universal Dependencies v2.5 are under Attribution-ShareAlike 4.0 International license, which is free to share and adapt. MNLI is freely available for typical machine learning uses, and may be modified and redistributed. IMDb and Yelp are available for limited non-commercial usage. WikiAnn is under ODC-By license for research usage.
A.2 Stochastic Hierarchical Autoencoder Implicit Reconstruction Loss
The stochastic hierarchical autoencoder implicitly minimizes the reconstruction loss of its subautoencoder inputs and outputs. We can rewrite Eq. 5 as o = U(x′), where
x′ =
{ AEd̂,di(D(x)), z > 0.3
D(x), z ≤ 0.3 (9)
Therefore, minimizing the distance between x and o also enforces the generated x′ to be as close as possible even when x is fed into different neural modules. That implicitly minimizes the distances between AEd̂,di(D(x)) and D(x). Let x′′ = D(x), so it turns out that the distance between AEd̂,di(x
′′) and x′′ is also minimized. Therefore, we can omit the explicit reconstruction loss between AEd̂,di(x
′′) and x′′ during model training, which helps reduce the complexity of our method.
A.3 Baselines
In this section, we describe more details about our baseline methods.
Dropout Dropout (Srivastava et al., 2014) regularizes the model by randomly ignoring layer outputs with probability p during training. It has been widely used in almost all PLMs (Devlin et al., 2019; Liu et al., 2019; He et al., 2021a) to prevent overfitting. We use dropout at all layers and choose the best performance from p ∈ {0.25, 0.5, 0.75} as this baseline.
Weight Decay (WD) WD (Krogh and Hertz, 1991) is a widely used regularization technique by adding a term λ2∥w∥2 in the loss function to penalise the size of the weight vector. We adopt a variation of WD and replace the term with λ2∥w−w0∥2, which is more tailored to the fine-tuning of PLMs (Lee et al., 2020). We choose the best performance from λ ∈ {0.25, 0.5, 0.75} as this baseline.
Gaussian Noise (GN) Variational Auto-Encoder (VAE) (Kingma and Welling, 2013) shows that adding GN to the hidden layer can improve the model training. We add a small amount of GN, 0.002 · N (0, I), to the hidden output of each layer in PLM during training as this baseline.
Adapter Adapter (Houlsby et al., 2019) is an encoder-decoder module for parameter-efficient transfer-learning in NLP. It freezes the PLM parameters, and only adjust the parameters of the light-weight modules inserted between transformer layers during fine-tuning, which can also be viewed as a regularization method. So we include it in the baselines as a reference.
Mixout Motivated by dropout, mixout (Lee et al., 2020) stochastically mixes the parameters of two models, which aims to minimize the deviation of the two. During training, we replace layer outputs with the corresponding values from the initial PLM checkpoint with probability p. We choose the best performance from p ∈ {10−1, 10−2, 10−3} as this baseline.
Variational Information Bottleneck (VIB) Mahabadi et al. (2021) leverages VIB (Alemi et al., 2017) to suppress irrelevant features when finetuning the model on target tasks. It compresses the sentence representation x generated by PLMs into a smaller dimension representation z with mean µ(x) and also introduces more diversity through GN with variance Var(x).
A.4 Layer-level MVCR
The default setting of MVCR is token-level selection, which means at each training step each token in the same layer randomly selects a different HAE from the pool where the weights are shared. A variation of token-level MVCR is layer-level MVCR (Fig. 8), where all tokens in the same layer randomly selects the same HAE from the pool.
A.5 Training Details
We use the same hyper-parameters as (Mahabadi et al., 2021) and (Hu et al., 2020) in the experiments. The model parameters are optimized with Adam (Kingma and Ba, 2014). We use learning rate 2e-5 (αtask in §3.3) to tune all parameters to minimize the task-specific losses, and use 2e-3 (αMSE in §3.3) to tune the HAE parameters to minimize the reconstruction losses. For all low-resource experiments, we train the model for 100 epochs with batch size 32. In the first 20 epochs, we freeze the parameters of the PLM and the classifier, and pretrain the HAEs only with the reconstruction loss described in Eq. 8. Then all parameters are tuned in the remaining 80 epochs. All the baseline models are fine-tuned for 80 epochs for a fair comparison with MVCR. Similarly, for full data setting, we train our model for 25 epochs, including 5-epoch HAE pretraining and 20-epoch all-parameter finetuning. Similarly, we train the baseline models for 20 epochs. The number of training epochs are sufficiently large for both baseline and our methods to converge. All experiments are conducted on the Nvidia Tesla v100 16GB GPUs, and each lowresource experiment takes about 20 to 40 minutes to complete.
A.6 Hyper-Parameter Search
Most of the hyper-parameters we use in the downstream tasks are same as (Mahabadi et al., 2021) and (Hu et al., 2020). Therefore, we only need to decide the hyper-parameters specific to our methods. To reduce the complexity, we only sample 500 samples from the MNLI, IMDb and WikiAnn datasets for hyper-parameters search. Instead of using a different set of hyper-parameters for each dataset, we use the set of hyper-parameters for all
of the sentence-level tasks and another set for the token-level tasks. To determine the HAE insertion layers, we conduct hyper-parameter search from 5 choices {{1}, {2}, {12}, {1,12}, {2,12}} with the sampled data, and decide to use {{1}, {12}} for all of the sentence-level tasks, and use {{2}, {2,12}} for all of the token-level tasks. We also conduct more comprehensive analysis of the insertion layers in §5.
A.7 More Results and Analysis
Number of HAEs in MVCR Table 6 shows the results of adding different number of HAEs to MVCR.
Diversity of HAE Dimensions We conduct extensive experiments to study the impacts of different types of dimension combinations, including “aaa”, “aab” and “abc”. Results are presented in Table 7. On avergage, "dim-abc" outperforms the other two types, while "dim-aab" outperforms "dim-aaa". Furthermore, as shown in the first row of Table 7, the results are close for “aaa” ranging from {32,32,32} to {512,512,512}. As such, we choose {64, 64, x}, {256, 256, x} for the analysis of “aab” and {128,256,x} for “abc”.
HAE vs. AE vs. VAE We replace HAE in MVCR with AE and VAE and compare the results
in Table 8. The results for IMDb is also plotted in Fig. 9.
Token- vs Layer-level MVCR We compare the performance of Token- and Layer-level MVCR, and report detailed results in Table 9. The results for IMDb is also plotted in Fig. 10.
Training Overhead We compare the training runtime and converge time on MNLI with 500 training data in Table 10. The total runtime of 80 epochs
for MVCR is only 3.07% more than the average runtime of all baselines. Moreover, MVCR converges faster than other baselines.
Inference With or Without MVCR We compare the results of inference with or without MVCR on MNLI and IMDb. The results in Table 11 shows that the performances are similar.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethical Impact

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section A.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section A.1
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Discussed in the original paper of the datasets.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Discussed in the original paper of the datasets.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 4, Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section A.4, Section A.6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section A.5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We leave it for future work. Ethical Impact
Deep learning has demonstrated encouraging performance on a wide range of tasks during the past few years. However, neural models are data hungry, which usually requires a large amount of training data to achieve reasonable performance. It is expensive and time consuming to annotate a large amount of data. Pretrained language models (PLMs) (Devlin et al. , 2019; Conneau and Lample, 2019; Liu et al. , 2020) have been proven to be useful to transfer knowledge from massive unlabeled text to downstream tasks, but they are also
prone to overfitting during fine-tuning due to overparameterization. In this work, we propose a novel method to help improve model robustness in the low-resource scenarios, which is part of the attempt to reduce neural model reliance on the labeled data, and hence reduce annotation cost. Our method has also demonstrated promising performance improvement on cross-lingual NLP tasks, which is also an attempt to break the language barrier and allow a larger amount of population to benefit from the advance of NLP techniques