SlowGuess/NAACL_2021_OCR huggingface dataset:
We applied the mixed precision training to all models to accelerate the training speed. The baseline model took around 15 hours to finish training on 8 Nvidia V100 GPU, while ours took around 30 hours. During the training phase, our model had an
Figure 2: Representation similarity evolvement during the learning course, NAT (or AT) denotes the baseline of representation similarity between two baselines NAT (or AT) models with different initialization seeds. NAT-AT denotes the representation similarity between NAT and AT encoder. Best viewed in color.
extra decoder of 25M parameters, but the parameter size (around 64M) was the same as the baseline model in inference.
A.3 Model Inference
In our model, the auxiliary AT decoder is employed to better capture the source sentence representation during the training phase. However, since our focus is on the NAT translation performance, the auxiliary AT decoder is not necessary during the inference. We simply ignored the AT decoder in our approach to keep the same translation speed as the baseline NAT models, leaving the space for future works to leverage the ignored AT decoder and benefit the translation.
A.4 Significance Tests
We use the compare-mt (Neubig et al., 2019) with 1000 re-samples to perform statistical significance tests for the MASK-PREDICT model output and our model output. The results show that our model significantly outperforms the baseline model in all language directions. All these experimental results confirm the effectiveness of our proposed method even with knowledge distillation. Besides, considering the results in Table 2, as a new knowledge transfer approach, our proposed multi-task learning method MULTI-TASK NAT can complement the classic knowledge distillation method, which is promising for future exploration.
B Probing Tasks
We conducted 10 probing tasks<sup>3</sup> to study what linguistic properties are captured by the encoder (Connieu and Kiela, 2018). A probing task is a classification problem that focuses on simple linguistic properties of sentences. 'SeLen' predicts the number of words in a sentence. 'WC' predicts which of the target words appear on the given sentence given its sentence embedding. 'TrDep' checks whether the encoder representation infers the hierarchical structure of sentences. In 'ToCo' task, it measures the sequence of top constituents immediately below the sentence node. 'BShif' predicts whether two consecutive tokens within the sentence have been inverted or not. 'Tense' asks for the tense of the main-clause verb. 'SubN' focuses on the number of the main clause's subject. 'ObjN' studies the number of the direct object of the main clause. In 'SoMo', some sentences are modified by replacing a random noun or verb with another one and the classifier should identify whether a sentence has been modified. 'CoIn' contains sentences made of two coordinate clauses. Half of the sentences have inverted the order of the clauses and the task is to tell whether a sentence is intact or modified.
We first extracted the sentence representations of input from the AT and NAT encoder, which were used to carry out our probing tasks. For both the AT model and NAT model, the mean of the encoder top layer representations was used as the sentence representation. The classifier we used in this work was a Multi-Layer Perceptron (MLP) with a hidden dimension of 200. We optimized the model using the Adam optimizer with a learning rate of 0.001. The batch size was set to 64 and we trained the model for 4 epochs. The 10-fold cross-validation was also employed to get the final performance.
C Representation Similarity
In this section, we adopt CCA (Morcos et al., 2018), a widely-utilized representation comparison metric (Saphra and Lopez, 2019; Voita et al., 2019), to calculate the encoder representation similarity throughout the training course. The CCA score ranges from 0 to 1, and a higher CCA score indicates both encoder representations contain more similar information.
In our experiments, we first extracted the sentence encoder representations for 100,000 training
examples (which is sufficient enough for CCA calculation) for each model. Then, we calculated the CCA score between the encoder representations of two models, where the representation was the mean of the top-layer encoder representations. In addition, to demonstrate the evolvement of the representation similarity during the learning course, we calculated the CCA score between two models for every 10,000 training steps.
Figure 2 shows the evolvement of representation similarity during the model training. We compute the CCA similarity between two NAT (or AT) models under different initialization seeds and take the similarity as the baseline (green and blue lines in Figure 2). Comparing to the baseline similarity of NAT (or AT), a lower similarity between AT and NAT representations (red line in Figure 2) indicates that AT and NAT encoders capture different source information.
<sup>3</sup>https://github.com/facebookresearch/SENTEval/tree/master/data/probing