another paper also just last year which looked at pre training on jft which is sort of this even bigger data set sort of 300 million images and what they find is that more true more pre training data is not always better which is this sort of very closely held assumption in the community so in particular like they sort of try training from random initialization versus the entire data set and in a bunch of places the performances are actually really pretty comparable other criterion if you fix the number of iteration or training data