DistilmBERT training/distillation dataset
Author: mbant
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Thanks a lot for Distil**m**BERT (amongst everything else), is there any info on the dataset used in the distillation process? 

Both the dataset itself or the process used to obtain it would be greatly appreciated! 

Am I right to assume you used a similar (if not the same) data as the original [`multilingual-bert`](https://github.com/google-research/bert/blob/master/multilingual.md#details) with a processed dump of the biggest 104 wikipedia dumps?
Again, any pointer to the preprocessing steps would be great!

Comments:
By reading the [official docs](https://github.com/huggingface/transformers/tree/master/examples/distillation), I think that they have trained Distil**m**BERT . For what concern the pre-processing steps, there are no information about that (surely I'm interested in these steps too). It would be more useful and precise to specify:
- the dataset used for training the model --> on the **concatenation of Wikipedia** in 104 different languages. Is it correct guys?
- the pre-processing steps developed --> **no information** about this step
- when to use Bert('base-multilingual-cased') and when to use DistilMBert --> the latter one is twice as fast as the former one, as said in the official docs 
- the difference between Bert('base-multilingual-cased') and DistilMBert --> on a [Twitter account](https://twitter.com/BramVanroy/status/1203096204122435590), a HuggingFace's dev said the following statement: "_Distil-mBERT is just an instance of DistilBERT with multilingual weights_." A question related to this topic: "_pretrained with the supervision of bert-base-multilingual-cased_" means that they have initialized the weights of the DistilMBERT model with the ones of multi-lingual BERT model?

> ##  Questions & Help
> Thanks a lot for Distil**m**BERT (amongst everything else), is there any info on the dataset used in the distillation process?
> 
> Both the dataset itself or the process used to obtain it would be greatly appreciated!
> 
> Am I right to assume you used a similar (if not the same) data as the original [`multilingual-bert`](https://github.com/google-research/bert/blob/master/multilingual.md#details) with a processed dump of the biggest 104 wikipedia dumps?
> Again, any pointer to the preprocessing steps would be great!

Thanks for adding a bit of context, but I'm pretty sure that (given that Distil**m**BERT is just a DistilBERT with multilingual weights) the distillation procedure is pretty much the same used in the [paper](https://arxiv.org/abs/1910.01108), just using `bert-base-multilingual-cased` as teacher.

I was really just curious to know if they had used the 104 languages wiki dump for distillation as well and if either the data or the script used to obtain them are available somewhere :)

Yeah, you're right about the distillation procedure they've followed.

Hello @mbant 
Indeed we used the concatenation of 104 wikipedia. We only extract ~110M seqs among these dumps following the smoothing probability of 0.7 used in mBERT (see [here](https://github.com/google-research/bert/blob/master/multilingual.md#details)).
The pre-training distillation phase is then the same as described in the paper! The teacher is indeed `bert-base-multilingual-cased`.
Victor

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.