Low Resource Languages	We pick languages with fewer than 500 training examples to investigate the performance of low-resource languages: Tamil (ta), Marathi (mr), Belarusian (be), Lithuanian (lt), Armenian (hy), Kazakh (kk). Table TABREF15 shows the performance of the models. While DBLP:journals/corr/abs-1904-09077 shows effective zero-shot crosslingual transfer from English to other high-resource languages, we show that cross-lingual transfer is even effective on low-resource languages when we train on all languages as mBERT is significantly better than BERT when we have fewer than 50 examples. In these cases, the mMiniBERT distilled from the multilingual mBERT yields results better than training individual BERT models. The gains becomes less significant when we have more training data. The multilingual baseline mMeta-LSTM does not do well on low-resource languages. On the contrary, mMiniBERT performs well and outperforms the state-of-the-art Meta-LSTM on the POS tagging task and on four out of size languages of the Morphology task.

Conclusion	We have described the benefits of multilingual models over models trained on a single language for a single task, and have shown that it is possible to resolve a major concern of deploying large BERT-based models by distilling our multilingual model into one that maintains the quality wins with performance fast enough to run on a single CPU. Our distilled model outperforms a multilingual version of a very strong baseline model, and for most languages yields comparable or better performance to a large BERT model.

Small and Practical Models	The results in Table TABREF1 make it clear that the BERT-based model for each task is a solid win over a Meta-LSTM model in both the per-language and multilingual settings. However, the number of parameters of the BERT model is very large (179M parameters), making deploying memory intensive and inference slow: 230ms on an Intel Xeon CPU. Our goal is to produce a model fast enough to run on a single CPU while maintaining the modeling capability of the large model on our tasks.

Multilingual Models	We compare per-language models trained on single language treebanks with multilingual models in Table TABREF1 and Table TABREF14 . In the experimental results we use a prefix INLINEFORM0 to denote the model is a single multilingual model. We compare Meta-LSTM, BERT, and MiniBERT. mBERT performs the best among all multilingual models. The smallest and fastest model, mMiniBERT, performs comparably to mBERT, and outperforms mMeta-LSTM, a state-of-the-art model for this task. When comparing with per-language models, the multilingual models have lower F1. DBLP:journals/corr/abs-1904-02099 shows similar results. Meta-LSTM, when trained in a multilingual fashion, has bigger drops than BERT in general. Most of the Meta-LSTM drop is due to the character-LSTM, which drops by more than 4 points F1.

Multilingual Models for Sequence Labeling	We discuss two core models for addressing sequence labeling problems and describe, for each, training them in a single-model multilingual setting: (1) the Meta-LSTM  , an extremely strong baseline for our tasks, and (2) a multilingual BERT-based model  .

Multilingual BERT	BERT is a transformer-based model  pretrained with a masked-LM task on millions of words of text. In this paper our BERT-based experiments make use of the cased multilingual BERT model available on GitHub and pretrained on 104 languages. Models fine-tuned on top of BERT models achieve state-of-the-art results on a variety of benchmark and real-world tasks. To train a multilingual BERT model for our sequence prediction tasks, we add a softmax layer on top of the the first wordpiece  of each token and finetune on data from all languages combined. During training, we concatenate examples from all treebanks and randomly shuffle the examples.

Distillation	For model distillation  , we extract sentences from Wikipedia in languages for which public multilingual is pretrained. For each sentence, we use the open-source BERT wordpiece tokenizer  ,  and compute cross-entropy loss for each wordpiece: INLINEFORM0  where INLINEFORM0 is the cross-entropy function, INLINEFORM1 is the softmax function, INLINEFORM2 is the BERT model's logit of the current wordpiece, INLINEFORM3 is the small BERT model's logits and INLINEFORM4 is a temperature hyperparameter, explained in Section SECREF11 . To train the distilled multilingual model mMiniBERT, we first use the distillation loss above to train the student from scratch using the teacher's logits on unlabeled data. Afterwards, we finetune the student model on the labeled data the teacher is trained on.

Meta-LSTM	The Meta-LSTM is the best-performing model of the CoNLL 2018 Shared Task  for universal part-of-speech tagging and morphological features. The model is composed of 3 LSTMs: a character-BiLSTM, a word-BiLSTM and a single joint BiLSTM which takes the output of the character and word-BiLSTMs as input. The entire model structure is referred to as Meta-LSTM. To set up multilingual Meta-LSTM training, we take the union of all the word embeddings from the bojanowski2017enriching embeddings model on Wikipedia in all languages. For out-of-vocabulary words, a special unknown token is used in place of the word. The model is then trained as usual with cross-entropy loss. The char-BiLSTM and word-biLSTM are first trained independently. And finally we train the entire Meta-LSTM.

Small and Practical BERT Models for Sequence Labeling	We propose a practical scheme to train a single multilingual sequence labeling model that yields state of the art results and is small and fast enough to run on a single CPU. Starting from a public multilingual BERT checkpoint, our final model is 6x smaller and 27x faster, and has higher accuracy than a state-of-the-art multilingual baseline. We show that our model especially outperforms on low-resource languages, and works on codemixed input text without being explicitly trained on codemixed examples. We showcase the effectiveness of our method by reporting on part-of-speech tagging and morphological prediction on 70 treebanks and 48 languages.

Small BERT structure	We use the vocab and wordpiece model included with the cased public multilingual model on GitHub. We use the BERT configuration of the public multilingual BERT with the following modifications for mMiniBERT. Hidden size = 256 Intermediate layer size = 1024 Num attention heads = 4 Layers = 3

Size and speed	We choose a three-layer BERT, we call MiniBERT, that has the same number of layers as the Meta-LSTM and has fewer embedding parameters and hidden units than both models. Table TABREF7 shows the parameters of each model. The Meta-LSTM has the largest number of parameters dominated by the large embeddings. BERT's parameters are mostly in the hidden units. The MiniBERT has the fewest total parameters. The inference-speed bottleneck for Meta-LSTM is the sequential character-LSTM-unrolling and for BERT is the large feedforward layers and attention computation that has time complexity quadratic to the sequence length. Table TABREF8 compares the model speeds. BERT is much slower than both MetaLSTM and MiniBERT on CPU. However, it is faster than Meta-LSTM on GPU due to the parallel computation of the transformer. The MiniBERT is significantly faster than the other models on both GPU and CPU.

Per-Language Results	We show per-language F1 results of each model in Table SECREF38 and Table SECREF38 . For per-language models, no models are trained for treebanks without tuning data, and metrics of those languages are not reported. All macro-averaged results reported exclude those languages. lccccc treebankBERTMeta-LSTMmBERT mMeta-LSTM mMiniBERT af_afribooms97.6297.6397.4993.1696.08 am_att3.285.63.16 ar_padt90.4690.5590.328990.06 ar_pud71.5968.9671.06 be_hse94.8191.0595.0287.5994.95 bg_btb99.0198.7798.7296.4398.19 ca_ancora98.8498.6298.7797.5798.45 cs_cac99.1799.4399.398.4698.48 cs_cltt87.4887.2587.6787.6287.53 cs_fictree98.6298.6398.2597.297.18 cs_pdt99.0699.

Codemixed Input	We use the Universal Dependencies' Hindi-English codemixed data set  to test the model's ability to label code-mixed data. This dataset is based on code-switching tweets of Hindi and English multilingual speakers. We use the Devanagari script provided by the data set as input tokens. In the Universal Dependency labeling guidelines, code-switched or foreign-word tokens are labeled as X along with other tokens that cannot be labeled. The trained model learns to partition the languages in a codemixed input by labeling tokens in one language with X, and tokens in the other language with any of the other POS tags. It turns out that the 2nd-most likely label is usually the correct label in this case; we evaluate on this label when the 1-best is X. Table TABREF25 shows that all multilingual models handle codemixed data reasonably well without supervised codemixed traininig data.

Data	We use universal part-of-speech tagging and morphology data from the The CoNLL 2018 Shared Task  ,  . For comparison simplicity, we remove the languages that the multilingual BERT public checkpoint is not pretrained on. For segmentation, we use a baseline segmenter (UDPipe v2.2) provided by the shared task organizer to segment raw text. We train and tune the models on gold-segmented data and apply the segmenter on the raw test of test data before applying our models. The part-of-speech tagging task has 17 labels for all languages. For morphology, we treat each morphological group as a class and union all classes as a output of 18334 labels.

The Importance of Distillation	To understand the importance of distillation in training mMiniBERT, we compare it to a model with the MiniBERT structure trained from scratch using only labeled multilingual data the teacher is trained on. Table TABREF37 shows that distillation plays an important role in closing the accuracy gap between teacher and student.

Training Hyperparameters	We use exactly the same hyperparameters as the public multilingual BERT for finetuning our models. We train the part-of-speech tagging task for 10 epochs and the morphology task for 50 epochs. For distillation, we use the following hyperparameters for all tasks. learning rate: 1e-4 temperature: 3 batch size: 256 num epochs: 24 We take the Wikipedia pretraining data as is and drop sentences with fewer than 10 characters.

79 sv_talbanken96.9296.5696.1390.2394.49 ta_ttb72.9171.0173.7546.970.22 te_mtg98.9698.9698.5498.6898.54 th_pud8.2708.43 tl_trg29.3128.6225.17 tr_imst89.59188.6373.2381.99 tr_pud23.7223.8423.46 uk_iu92.490.9892.6479.4988.79 ur_udtb82.2483.7282.6481.8982.48 vi_vtb83.748483.9383.5883.94 yo_ytb58.7886.8261.88 zh_cfl46.5543.5545.73 zh_gsd87.6488.3888.3187.0588.5 zh_hk66.3364.9766.23 zh_pud86.3583.686.14 Morphology F1 of all models.

28 tr_imst93.9694.0393.184.6491.8 tr_pud73.168.3672.47 uk_iu97.2996.697.289396.88 ur_udtb93.8393.8793.699393.05 vi_vtb77.6776.4277.4472.0177.06 yo_ytb43.4830.8534.59 zh_cfl49.8339.7749.42 zh_gsd87.685.785.9682.7686.08 zh_hk66.2957.8865.86 zh_pud83.373.382.95 POS tagging F1 of all models. lccccc treebankBERT F1Meta-LSTM F1mBERT F1mMeta-LSTM F1mMiniBERT F1 af_afribooms97.1197.3696.5388.9893.75 am_att32.3632.36 ar_padt88.2688.2487.7683.1485.34 ar_pud36.3334.2836.08 be_hse82.8374.0387.5259.1681.82 bg_btb97.

473.0265.36 pt_bosque96.2796.1687.0483.1385.72 pt_gsd97.295.3367.7276.0171.88 pt_pud52.0649.7950.95 ro_nonstandard88.5288.9186.8982.182.14 ro_rrt97.0297.2396.5893.294.85 ru_gsd88.8386.7381.4464.278.93 ru_pud37.9735.2637.49 ru_syntagrus97.0296.995.9991.9694.33 ru_taiga88.5684.0286.01 sa_ufal15.916.1416.33 sk_snk92.0689.6391.5868.2585.29 sl_ssj94.3993.7894.4182.6989.23 sl_sst88.4691.8978.2285.59 sr_set94.8394.7192.7973.5190.48 sv_lines89.5489.5588.6683.2786.4 sv_pud77.3973.9476.

0798.9998.2298.61 cs_pud97.1396.5397 da_ddt97.5997.4797.1892.3695.93 de_gsd94.8194.1794.5391.9493.82 de_pud88.7687.4288.7 el_gdt97.9797.497.9194.8797.16 en_ewt95.8295.4595.292.2494.19 en_gum96.2295.0294.7992.3394.24 en_lines97.2296.8195.7993.9695.25 en_partut96.1195.995.0293.2994.61 es_ancora98.8798.7898.1796.2797.8 es_gsd93.793.989.6590.6189.58 es_pud85.8786.185.71 et_edt97.2797.1797.0294.3295.64 eu_bdt96.296.195.5191.5394.15 fa_seraji97.5797.1797.1795.2996.

92 fi_ftb96.2696.1293.1587.2389.79 fi_pud95.5593.2395.01 fi_tdt96.8197.0293.991.5892.6 fr_gsd96.6296.4596.2395.3796.05 fr_partut96.189695.4394.3594.93 fr_pud90.7790.190.64 fr_sequoia96.7797.5997.0795.9196.75 fr_spoken97.5595.7896.190.0793.25 ga_idt91.9291.5590.8384.1685.72 gl_ctg96.9997.2196.592.8795.84 gl_treegal93.491.2891.9 he_htb82.7682.4982.6980.9381.93 hi_hdtb97.3197.3997.196.296.43 hi_pud86.4885.3385.68 hr_set97.7997.9497.4796.2497.2 hu_szeged96.5194.7195.9985.595.

47 hy_armtdp84.4286.6263.8286.98 id_gsd93.0693.3793.390.8193.35 id_pud63.5263.563.33 it_isdt98.3398.0698.2796.797.8 it_partut98.1298.1798.0996.9998.06 it_postwita95.6695.8695.694.1793.2 it_pud93.8492.7293.67 ja_gsd88.6388.7388.5487.0388.43 ja_modern41.5551.2621.61 ja_pud89.1587.9689.3 kk_ktb75.9361.781.3652.9180.06 ko_gsd95.9295.6490.386.3988.62 ko_kaist95.5695.4293.8687.4693.43 ko_pud41.9346.1131.96 la_ittb98.3498.4298.397.1897.65 la_perseus89.9183.8585.23 la_proiel96.3496.

3795.9792.0293.78 lt_hse88.8881.4390.0165.686.9 lv_lvtb94.7994.4793.7188.2591.3 mr_ufal77.4572.175.9265.4875.41 nl_alpino97.196.1697.3393.7896.19 nl_lassysmall95.5495.9295.7294.495.47 no_bokmaal989897.9595.2797.04 no_nynorsklia94.0888.2792.55 no_nynorsk97.9497.9297.6994.9196.59 pl_lfg98.798.598.3995.2197.48 pl_sz98.5697.9198.0594.7397.29 pt_bosque96.7496.7396.1695.5395.85 pt_gsd95.8395.4493.8493.0794.44 pt_pud89.4889.6689.29 ro_nonstandard94.6794.489492.0591.9 ro_rrt97.6397.

5297.4795.7896.71 ru_gsd92.2391.3990.8488.1390.14 ru_pud89.788.9289.52 ru_syntagrus98.398.6598.3297.1398.03 ru_taiga93.6292.7593.18 sa_ufal32.4729.5827.11 sk_snk97.0896.3296.9893.6196.35 sl_ssj97.0796.6896.8994.2495.58 sl_sst94.5190.3491.79 sr_set98.6398.3398.3194.7997.36 sv_lines97.2196.5996.9993.6495.57 sv_pud94.5292.0694.32 sv_talbanken98.0397.3497.7794.9196.76 ta_ttb75.7172.774.2861.5174.6 te_mtg94.2592.7293.4287.3293.42 th_pud2.372.731.54 tl_trg70.6928.6268.

5497.5897.4791.4195.4 ca_ancora98.3798.2198.2896.0497.67 cs_cac96.3396.4996.5488.1193.47 cs_cltt81.6179.8983.8678.8280.61 cs_fictree96.3996.494.0983.3787.59 cs_pdt97.1896.9197.1589.7794.63 cs_pud93.8887.4491.81 da_ddt97.2297.0895.6289.8294.08 de_gsd90.8490.5890.480.6988.99 de_pud30.4130.5530.4 el_gdt94.5793.9594.8387.692.07 en_gum96.879693.7990.1193.71 en_lines97.3296.6893.1187.4992.07 en_partut94.8895.3890.7679.9990.18 en_pud93.2591.2393.1 es_ancora98.4598.4297.695.

1797 es_gsd93.5293.7288.7289.2688.78 es_pud52.752.852.73 et_edt96.1496.1195.7890.5192.14 eu_bdt93.2792.5692.6776.7284.53 fa_seraji97.3597.2596.9193.8296.28 fi_ftb96.3496.4892.3277.8986.47 fi_pud93.5891.1291.65 fi_tdt95.0395.5890.9688.4487.48 fr_gsd96.0596.1194.6786.9794.51 fr_partut93.3292.9388.987.4887.05 fr_pud59.1557.558.94 fr_sequoia97.0997.1391.5485.2390.74 fr_spoken10010098.6280.6796.67 ga_idt82.281.7881.263.4466.82 gl_ctg98.9898.9595.2789.9895.1 gl_treegal80.0568.7375.

97 he_htb81.2780.8580.7976.8978.74 hi_hdtb93.3293.8592.9189.0990.65 hi_pud22.122.3722.03 hr_set91.9991.8591.2481.6287.81 hu_szeged93.6591.2892.9371.2587.36 hy_armtdp41.1354.4551.0836.5946.43 id_gsd94.849694.8591.6294.39 id_pud39.8342.7939.79 it_isdt97.797.8297.8795.4797.37 it_partut97.3597.7398.0196.3397.9 it_postwita95.6296.0595.0391.5293.17 it_pud57.8257.4157.6 ja_gsd90.2990.4590.2990.3990.41 ja_modern63.961.1763.99 ja_pud57.457.2657.27 kk_ktb64.625.5559.49 ko_gsd99.6299.

5599.498.9999.37 ko_kaist10010099.9499.2499.93 ko_pud38.3338.6638.27 la_ittb96.796.9497.1590.7893.91 la_perseus82.0964.7372.24 la_proiel90.8291.0191.5179.0883.99 lt_hse75.2169.6573.6142.5165.22 lv_lvtb88.6191.3488.179.1181.91 mr_ufal63.9559.1164.233.6354.01 nl_alpino96.2296.1396.5391.995.67 nl_lassysmall96.4696.0295.5592.1695.28 no_bokmaal96.8597.1396.4891.1795.31 no_nynorsklia94.2289.5691.08 no_nynorsk96.797.0496.4992.1294.79 pl_lfg95.8594.6884.9647.9984.56 pl_sz93.991.9371.

Tuning	For Meta-LSTM, we use the public repository's hyperparameters. Following devlin2019, we use a smaller learning rate of 3e-5 for fine-tuning and a larger learning rate of 1e-4 when training from scratch and during distillation. Training batch size is set to 16 for finetuning and 256 for distillation. For distillation, we try temperatures INLINEFORM0 and use the teacher-student accuracy for evaluation. We observe BERT is very confident on its predictions, and using a large temperature INLINEFORM1 to soften the distribution consistently yields the best result.

Question: How do they compress the model?
Answer: we extract sentences from Wikipedia in languages for which public multilingual is pretrained. For each sentence, we use the open-source BERT wordpiece tokenizer  ,  and compute cross-entropy loss for each wordpiece: INLINEFORM0