Edit model card

distilrubert-tiny-cased-conversational

Conversational DistilRuBERT-tiny (Russian, cased, 3‑layers, 264‑hidden, 12‑heads, 10.4M parameters) was trained on OpenSubtitles[1], Dirty, Pikabu, and a Social Media segment of Taiga corpus[2] (as Conversational RuBERT). It can be considered as tiny copy of Conversational DistilRuBERT-small.

Our DistilRuBERT-tiny is highly inspired by [3], [4] and architecture is very close to [5]. Namely, we use

  • MLM loss (between token labels and student output distribution)
  • MSE loss (between averaged student and teacher hidden states)

The key features are:

  • unlike most of distilled language models, we didn't use KL loss during pre-training
  • reduced vocabulary size (30K in tiny vs. 100K in base and small )
  • two separate inputs for student: tokens obtained using student tokenizer (for MLM) and teacher tokens greedily splitted by student tokens (for MSE)

Here is comparison between teacher model (Conversational RuBERT) and other distilled models.

Model name # params, M # vocab, K Mem., MB
rubert-base-cased-conversational 177.9 120 679
distilrubert-base-cased-conversational 135.5 120 517
distilrubert-small-cased-conversational 107.1 120 409
cointegrated/rubert-tiny 11.8 30 46
distilrubert-tiny-cased-conversational 10.4 31 41

DistilRuBERT-tiny was trained for about 100 hrs. on 7 nVIDIA Tesla P100-SXM2.0 16Gb.

We used PyTorchBenchmark from transformers to evaluate model's performance and compare it with other pre-trained language models for Russian. All tests were performed on Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz and nVIDIA Tesla P100-SXM2.0 16Gb.

Model name Batch size Seq len Time, s Mem, MB
CPU GPU CPU GPU
rubert-base-cased-conversational 1 512 0.147 0.014 897 1531
distilrubert-base-cased-conversational 1 512 0.083 0.006 766 1423
distilrubert-small-cased-conversational 1 512 0.03 0.002 600 1243
cointegrated/rubert-tiny 1 512 0.041 0.003 272 919
distilrubert-tiny-cased-conversational 1 512 0.023 0.003 206 855
rubert-base-cased-conversational 16 512 2.839 0.182 1499 2071
distilrubert-base-cased-conversational 16 512 1.065 0.055 2541 2927
distilrubert-small-cased-conversational 16 512 0.373 0.003 1360 1943
cointegrated/rubert-tiny 16 512 0.628 0.004 1293 2221
distilrubert-tiny-cased-conversational 16 512 0.219 0.003 633 1291

To evaluate model quality, we fine-tuned DistilRuBERT-tiny on classification (RuSentiment, ParaPhraser), NER and question answering data sets for Russian and obtained scores very similar to the Conversational DistilRuBERT-small.

Citation

If you found the model useful for your research, we are kindly ask to cite this paper:

@misc{https://doi.org/10.48550/arxiv.2205.02340,
  doi = {10.48550/ARXIV.2205.02340},
  
  url = {https://arxiv.org/abs/2205.02340},
  
  author = {Kolesnikova, Alina and Kuratov, Yuri and Konovalov, Vasily and Burtsev, Mikhail},
  
  keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
  
  title = {Knowledge Distillation of Russian Language Models with Reduction of Vocabulary},
  
  publisher = {arXiv},
  
  year = {2022},
  
  copyright = {arXiv.org perpetual, non-exclusive license}
}

[1]: P. Lison and J. Tiedemann, 2016, OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016)

[2]: Shavrina T., Shapovalova O. (2017) TO THE METHODOLOGY OF CORPUS CONSTRUCTION FOR MACHINE LEARNING: «TAIGA» SYNTAX TREE CORPUS AND PARSER. in proc. of “CORPORA2017”, international conference , Saint-Petersbourg, 2017.

[3]: Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.

[4]: https://github.com/huggingface/transformers/tree/master/examples/research_projects/distillation

[5]: https://habr.com/ru/post/562064/, https://huggingface.co/cointegrated/rubert-tiny

Downloads last month
13,011
Hosted inference API

Unable to determine this model’s pipeline type. Check the docs .