then once you fine tune you're probably able to push that representation through to the end because of your training signal but if you hadn't done the pre-training you wouldn't even have that higher level representation and then the signal I guess is not strong enough to back propagate through the whole model it would be very interesting if they investigate if they do this linear probe analysis again after they fine-tune the model that and to see if then still it is the intermediate layers that have the best representation or if now the best