the end or does the representation like in this curve does the good representation now shift towards the end and leave the lower layer with even more capacity to do some low level stuff yeah maybe they've done this I haven't seen it so and as you can see these Bert and autoregressive object if they sort of trade off so the birthda tends to do poorly in the linear probe setting but then it catches up during fine tuning in C for 10 almost being at the level of the autoregressive and in an image net actually outperforming it this this