Then the importance value $\alpha _i$ for hidden-state $H_i$ is calculated by: where $W^T \in \mathbb {R}^{4d} $ and $b\in \mathbb {R}$ are trainable parameters. Let $S$ represent the computation scores for all hidden-states. $S$ is computed as follows: It should be noted that $\sum _{i=0}^{l}S_i = 1$ where $S_i$ is the weight of hidden-state $i$ when computing the representation. Subsequently, we obtain the input sequence's new representation $A$ by: With the same shape as the output of Transformer-based encoder's final layer, HIRE's output $A$ is expected to contain the additional useful information from the encoder's hidden-states which is helpful for a better understanding of the input text and we call it complementary representation.

Here $H_0$ is the initial embedding of input text, which is the input of the encoder's first layer but is updated during training and $H_j \in \mathbb {R}^{n\times d}$ is the hidden-state of the encoder at the output of layer $j$. For the sake of simplicity, we call them all hidden-state afterwards. For each hidden-state of encoder, we use the same 2-layer Bidirectional Gated Recurrent Unit (GRU)  to summarize it. Instead of taking the whole output of GRU as the representation of the hidden state, we concatenate GRU's each layer and each direction's final state together. In this way, we manage to summarize the hidden-state into a fixed-sized vector. Hence, we obtain $U \in \mathbb {R}^{(l+1) \times 4d}$ with $U_i$ the summarized vector of $H_i$: where $0 \le i \le l$.

shows diverse combinations of different layers' outputs of the pre-trained BERT result in distinct performance on CoNNL-2003 Named Entity Recognition (NER) task .  points out for pre-trained language models, including Transformer, the most transferable contextualized representations of input text tend to occur in the middle layers, while the top layers specialize for language modeling. Therefore, the onefold use of last layer's output may restrict the power of the pre-trained representation. In this paper, we introduce RTRHI: Refined Transformer Representation with Hidden Information based on the fine-tuning approach with Transformer-based model, which leverages the hidden information in the Transformer's hiddens layer to refine the language representation. Our approch consists of two main additional components: HIdden Representation Extractor (HIRE) dynamically learns a complementary representation which contains the information that the final layer's output fails to capture. We put 2-layer bidirectional GRU beside the encoder to summarize the output of each layer into a single vector which will be used to compute the contribution score. Fusion layer integrates the hidden information extracted by the HIRE with Transformer final layer's output through two steps of different functionalities, leading to a refined contextualized language representation.

Model and Method ::: Hidden Representation Extractor	Since Transformer-based encoder normally has many identical layers stacked together, for example, $\text{BERT}_{\text{LARGE}}$ and $\text{XLNet}_{\text{LARGE}}$ all contain 24 layers of the identical structure, the output of the final layer may not be the most perfect candidate to fully represent the information contained in the input text. Trying to solve this problem, we introduce an HIdden Representation Extractor (HIRE) beside the encoder to draw from the hidden states the information that the output of the last layer fails to capture. Since each layer's hidden states don't carry the information of same importance to represent a certain input sequence, we adopt a mechanism which can compute the importance dynamically. We name the importance as contribution score. The input to the HIRE is $\lbrace H_0, \dots , H_j, \dots , H_l\rbrace $ where $ 0 < j \le l$ and $l$ represents the number of layers in the encoder.

Conclusion	In this paper, we have introduced RTRHI, a novel approach that refines language representation by leveraging the Transformer-based model's hidden layers. Specifically, an HIdden Representation Extractor is used to dynamically generate complementary imformation which will be incorporated with preliminary representation in the Fusion Layer. The experimental results demonstrate the effectiveness of refined language representation for natural language understanding. The analysis highlights the distinct contribution of each layer's output for diverse task and different example. We expect future work could be conducted in the following domains: (1) explore sparse version of Hidden Representation Extractor for more effective computation and less memory usage; (2) incorporating extra knowledge information  or structured semantic information  with current language representation in the fusion layer during fine-tuning; (3) integrate multi-tasks training  or knowledge distillation ,  into our model.

Model and Method ::: Fusion Layer	This layer fuses the information contained in the output of Tansformed-based encoder and the one extracted from encoders' hidden states by HIRE. Given the preliminary representation $R$, instead of letting it flow directly into task-specfic output layer, we combine it together with the complementary representation $A$ to yeild $M$, which we define by: where $\circ $ is elementwise multiplication (Hadamard Product) and $[;]$ is concatenation across the last dimension. Later, two-layer bidirectional GRU, with the output size of $d$ for each direction, is used to fully fuse the information contained in the preliminary representation and the additional useful information included in the complementary representation. We concatenate the outputs of the GPUs in two dimensions together, and we hence obtain the final contextualized representation $F$ of input text: The use of GRUs enables the complete interaction between the two different kinds of information mentioned before. Therefore, $F$ is expected to be a refined universal representation of input text.

Model and Method ::: Transformer-based encoder layer	Transformer-based encoder layer is responsible for encoding input text into a sequence of high-dimensional vectors, which is consided as the contextualized representation of the input sequence. Let {$w_1,\dots , w_n$} represent a sequence of $n$ words of input text, we use a Transformer-based encoder to encode the input sequence, thereby to obtain its universal contextualized representation $R \in \mathbb {R}^{n\times d}$: where $d$ is the hidden size of the encoder. It should be noted that $R$ is the output of Transformer-based encoder's last layer which has the same length as the input text. We call it preliminary representation in this paper to distinguish it with the one that we introduce in section SECREF6. Here, we omit a rather extensive formulations of Transformer and refer readers to ,  and  for more details.

Analysis	We compare the contribution score's distribution of different NLU tasks. For each task, we run our best single model over the development set and the results are calculated by averaging the values across all the examples within each dataset. The results are showed in Figure FIGREF27. From the top to the bottom of the headmap, the results are placed in the following order: single-sentence tasks, similarity and paraphrase tasks and natural language inference tasks. From figure FIGREF27, we find that the distribution differs among the different tasks, which demonstrates RTRHI's dynamic ability to adapt for distinct task when computing the complementary representation. The most important contribution occurs below the final layer for all the tasks except MRPC and RTE. All layers have a close contribution for MRPC and RTE task. Figure FIGREF28 presents the distribution of contribution scores over different layers for each example of SST-2 dataset. The number on the ordinate axis denotes the index of the example. We observe that even though there are subtle differences among these example, they follow certain same patterns when calculating the complementary representation, for example, layer 21 and 22 contribute the most for almost all the examples and also the layers around them.

Deepening Hidden Representations from Pre-trained Language Models for Natural Language Understanding	Transformer-based pre-trained language models have proven to be effective for learning contextualized language representation. However, current approaches only take advantage of the output of the encoder's final layer when fine-tuning the downstream tasks. We argue that only taking single layer's output restricts the power of pre-trained representation. Thus we deepen the representation learned by the model by fusing the hidden representation in terms of an explicit HIdden Representation Extractor (HIRE), which automatically absorbs the complementary representation with respect to the output from the final layer. Utilizing RoBERTa as the backbone encoder, our proposed improvement over the pre-trained models is shown effective on multiple natural language understanding tasks and help our model rival with the state-of-the-art models on the GLUE benchmark.

But the figure shows also that for some examples, all layers contribute almost equally.

Optimization: We use Adam optimizer  with $\beta _1=0.9$, $\beta _2=0.98$ and $\epsilon = 10^{-6}$ and the learning rate is selected amongst {5e-6, 1e-5, 2e-5, 3e-5} with a warmup rate ranging from 0.06 to 0.25 depending on the nature of the task. The number of training epochs ranges from 4 to 10 with the early stop and the batch size is selected amongst {16, 32, 48}. In addition to that, we clip the gradient norm within 1 to prevent exploding gradients problem occuring in the recurrent neural networks in our model. Regularization: We employ two types of regularization methods during training. We apply dropout  of rate 0.1 to all layers in the Transformer-based encoder and GRUs in the HIRE and fusion layer. We additionally adopt L2 weight decay of 0.1 during training.

Experiments ::: Main results	Table TABREF22 compares our method RTRHI with a list of Transformer-based models on the development set. To obtain a direct and fair comparison with our baseline model RoBERTa, following the original paper , we fine-tune RTRHI separately for each of the GLUE tasks, using only task-specific training data. The single-model results for each task are reported. We run our model with five different random seeds but the same hyperparameters and take the median value. Due to the problematic nature of WNLI dataset, we exclude its results in this table. The results shows that RTRHI consistently outperforms RoBERTa on 4 of the GLUE task development sets, with an improvement of 1.6 points, 0.4 pionts, 0.4/0.2 points, 0.3 points on CoLA, SST-2, MNLI and QNLI respectively. And on the QQP and RTE task, our model get the same result as RoBERTa. It should be noted that the improvement is entirely attributed to the introduction of HIdden Representation Extractor and Fusion Layer in our model.

Related Work	Transformer-based language models take Transformer  as their model architecture, but pre-trained with different objectives or language corpus. OpenAI GPT  is the first model which introduced Transformer architecture into unsupervised pre-training. The model is pre-trained on 12-layer left-to-right Transformer with BooksCorpus  dataset. But instead of using a left-to-right architecture like GPT, BERT  adopts Masked LM objective when pre-training, which enables the representation to incorporate context from both direction. The next sentence prediction (NSP) objective is also used by BERT to better understand the relationship between two sentences. The training procedure is conducted on a combination of BooksCorpus plus English Wikipedia. XLNet , as a generalized autoregressive language model, uses a permutation language modeling objective during pre-training on the other hand. In addition to BooksCorpus and English Wikipedia, it also uses Giga5, ClueWeb 2012-B and Common Crawl for pre-training. Trained with dynamic masking, large mini-batches and a larger byte-level BPE, full-sentences without NSP, RoBERTa  improves BERT's performance on the downstream tasks.

Experiments ::: Implementation	Our implementation of RTRHI is based on the PyTorch implementation of Transformer . Preprocessing: Following , we adopt GPT-2  tokenizer with a Byte-Pair Encoding (BPE) vocabulary of subword units size 50K. We format the input sequence in the following way. Given single sequence $X$, we add 
 token at the begining and 
 token at the end: 
X
. For a pair of sequences $(X, Y)$, we additionally use 
 to separate these two sequences: 
X
Y
. Model configurations: We use RoBERTa-Large as the Transformer-based encoder and load the pre-training weights of RoBERTa . Like BERT-Large, RoBERTa-Large model contains 24 Transformer-blocks, with the hidden size being 1024 and the number of self-attention heads being 16 , .

The pre-training corpora includes BooksCorpus, CC-News, Openwebtext and Stories. By fine-tuning on the downstream tasks in a supervised manner, these powerful Transformer-based models all push the state-of-the-art results on the various NLP tasks to a new level. Recent works have proposed new methods for fine-tuning the downstream tasks, including multi-task learning , adversarial training  or incorporating semantic information into language representation .

Introduction	Language representation is essential to the understanding of text. Recently, pre-training language models based on Transformer  such as GPT , BERT , XLNet , and RoBERTa  have been shown to be effective for learning contextualized language representation. These models have since continued to achieve new state-of-the-art results on a variety of natural processing tasks. They include question answering , , natural language inference , , named entity recognition , sentiment analysis  and semantic textual similarity , . Normally, Transformer-based models are pre-trained on large-scale unlabed corpus in a unspervised manner, and then fine-turned on the downstream tasks through introducing task-specific output layer. When fine-tuning on the supervised downstream tasks, the models pass directly the output of Transformer encoder's final layer, which is consided as the contextualized representation of input text, to the task-specific layer. However, due to the numerous layers (i.e., Transformer blocks) and considerable depth of these pre-training models, we argue that the output of the last layer may not always be the best representation of the input text during the fine-tuning for downstream task.

Model and Method ::: Output layer	The output layer is task-specific, which means we can adopt HIRE and fusion layer to other downstream tasks by only changing the output layer, such as question answering. GLUE benchmark contains two types of tasks: 1. classification; 2. regression. For classification tasks, given the input text's contextualized representation $F$, following , we take the first row $C \in \mathbb {R}^{2d}$ of $F$ corresponding to the first input token ([CLS]) as the aggregate representation. Let $m$ be the number of labels in the datasets, we pass $C$ through a feed-forward network(FFN): with $W_1 \in \mathbb {R}^{2d\times d}$, $W_2 \in \mathbb {R}^{d\times m}$, $b_1\in \mathbb {R}^{d}$ and $b_2\in \mathbb {R}^{m}$ the only parameters that we introduce in output layer.

Taking advantage of the robustness of RoBERTa by using it as the Transformer-based encoder of RTRHI, we conduct experiments on GLUE benchmark , which consists of nine Natural Language Understanding (NLU) tasks. RTRHI outperforms our baseline model RoBERTa on 5/9 of them and advances the state-of-the-art on SST-2 dataset. Even though we don't make any modification to the encoder's internal architecture or redefine the pre-training procedure with different objectives or datasets, we still get the comparable performance with other state-of-the-art models on the GLUE leaderboard. These results highlight RTRHI's excellent ability to refine Transformed-based model's language representation.

Model and Method ::: Training	For classification tasks, the training loss to be minimized is defined by the Cross-Entropy: where $\theta $ is the set of all parameters in the model, $N$ is the number of examples in the dataset, $p_{i,c}$ is the predicted probability of class $c$ for example $i$ and $y$ is the binary indicator defined as below: For regression tasks, we define the training loss by mean squared error (MSE): where $Q_i$ is the predicted value for example $i$ and $y_i$ is the ground truth value for example $i$ and $N$, $\theta $ are same as the ones in equation DISPLAY_FORM19.

Experiments ::: Dataset	We conducted the experiments on the General Language Understanding Evaluation (GLUE) benchmark  to evaluate our method's performance. GLUE is a collection of 9 diverse datasets for training, evaluating, and analyzing natural language understanding models. Three different tasks are presented in GLUE benchmark according to the original paper: Single-sentence tasks: The Corpus of Linguistic Acceptability (CoLA)  requires the model to determine whether a sentence is grammatically acceptable; the Stanford Sentiment Treebank (SST-2)  is to predict the sentiment of movie reviews with label of positive or negative.  Similarity and paraphrase tasks: Similarity and paraphrase tasks are to predict whether each pair of sentences captures a paraphrase/semantic equivalence relationship. The Microsoft Research Paraphrase Corpus (MRPC) , the Quora Question Pairs (QQP)  and the Semantic Textual Similarity Benchmark (STS-B)  are presented in this category.  Natural Language Inference (NLI) tasks: Natural language inference is the task of determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”.

GLUE benchmark contains the following tasks: the Multi-Genre Natural Language Inference Corpus (MNLI) , the Stanford Question Answering Dataset (QNLI) , the Recognizing Textual Entailment (RTE)  and the Winograd Schema Challenge (WNLI) . Four official metrics are adopted to evaluate the model performance: Matthews correlation , accuracy, F1 score, Pearson and Spearman correlation coefficients. More details will be presented in section SECREF25.

Table TABREF26 presents the results of RTRHI and other models on the test set that have been submitted to the GLUE leaderboard. Following , we fine-tune STS-B and MRPC starting from the MNLI single-task model. Given the simplicity between RTE, WNLI and MNLI, and the large-scale nature of MNLI dataset (393k), we also initialize RTRHI with the weights of MNLI single-task model before fine-tuning on RTE and WNLI. We submitted the ensemble-model results to the leaderboard. The results show that RTRHI still boosts the strong RoBERTa baseline model on the test set. To be specific, RTRHI outperforms RoBERTa over CoLA, SST-2, MRPC, SST-B, MNLI-mm with an improvement of 0.8 points, 0.4 points, 0.7/0.9 points, 0.2/0.1 points and 0.2 points respectively. In the meantime, RTRHI gets the same results as RoBERTa on QQP and WNLI.

By category, RTRHI has better performance than RoBERTa on the single sentence tasks, similarity and paraphrase tasks. It's worth noting that our model obtains state-of-art results on SST-2 dataset, with a score of 97.1. The results is quite promising since HIRE does not make any modification with the encoder internal architecture  or redefine the pre-training procedure  and we still get the comparable results with them.

Finally, the probability distribution of predicted label is computed as: For regression task, we obtain $Q$ in the same manner with $m = 1$, and take $Q$ as the predicted value.

Question: Do they use all the hidden layer representations?
Answer: Yes