How do you measure the degree of influence of pre-training on a fine-tuning?

jpcbertoldo: <h1>
<a name="context-1" class="anchor" href="#context-1"></a>Context</h1>
<p>I have a randomly initialized (deep learning) model A, and a pre-trained model Ap.<br>
The two are trained (or fine-tuned, more specifically for Ap) on the same target task* and I call them A’ and Ap’, respectively.</p>
<p><strong>p(.)</strong> is the performance of a model on the target task. Let’s admit that, in general, p(Ap’) &gt; p(A’).</p>
<p>*Assuming that either with everything identical (seeds, etc), or that I have enough seeds so I can rely on the average behavior.</p>
<h1>
<a name="what-i-am-looking-for-2" class="anchor" href="#what-i-am-looking-for-2"></a>What I am looking for</h1>
<p>A measure/metric of the robustness/dependency of my method with respect to the pre-training; i.e. something that tells me the degree of dependency of my method on the pre-training process.</p>
<p>In the end I want to repeat this process over many fine-tunning methods and compare them.</p>
<hr>
<h1>
<a name="edit-1-3" class="anchor" href="#edit-1-3"></a>Edit 1</h1>
<p>I would like have some conclusion that looks like: "the fine-tunning method <strong>X</strong> is more robust to pre-training than fine-tunning method <strong>Y</strong> based on the fact that even with a <em>worse</em>* pre-training it retains more performance.</p>
<p><strong>A bit more formally :</strong></p>
<ul>
<li>say that the training (or, again, fine-tuning) methods <code>X</code> and <code>Y</code> are being compared;</li>
<li>their respective performances after they are applied to the (raw, pre-trained) models are <code>(p(Ax), p(Apx))</code> and <code>(p(Ay), p(Apy))</code>;</li>
</ul>
<p>I’d like to make conclusions, based on comparisons between <code>(p(Ax), p(Apx))</code> and <code>(p(Ay), p(Apy))</code>, that <code>X</code> is more/less robust to the pre-training than <code>Y</code>.</p>
<p><strong>Ideally…</strong> given a method <code>X</code>, I’d like to generate a sequence <code>(p(Ax), p(Ap1x), ..., p(ApNx))</code> such that the pre-trained models <code>Ap1, ..., ApN</code> are increasingly better in their original (surrogate) task – and <code>A</code> (without pre-training) is a reference.  Then the sequence of performance on the target task <code>(p(Ax), p(Ap1x), ..., p(ApNx))</code> allows one to take conclusions about <code>X</code> [compared to another fine-tuning mehtod <code>Y</code>].</p>
<p><strong>A practical (optional) constraint</strong>: for the sake of reproducibility, I’d like to use publicly availabe pre-trained models so in a perfect scenario all the model <code>Ap1, ..., ApN</code> should be available or “<em>easily</em>” reproducible (i.e. can be programatically generated).</p>

tom: <p>In the ideal analysis, you would</p>
<ul>
<li>repeatedly randomly initialize your model,</li>
<li>do the pretraining on each,</li>
<li>train both the “raw” randomly initialized and the pre-trained model on your data and get performance metrics.</li>
</ul>
<p>This would give you a random sample of the performance metrics with and without pretraining, so you can take expectation and compute a bootstrapped confidence interval. (It might be interesting to also variate on the train/val split, but maybe this is optional.)</p>
<p>This costs quite a bit of computational power. The good news is that if your multiple fine-tuning methods start from the same type of pretrained model, you can amortize the pre-training cost over your experiments.</p>
<p>Best regards</p>
<p>Thomas</p>

jpcbertoldo: <p>Thank you for your reply <a class="mention" href="/u/tom">@tom</a>.<br>
I added a few details to the question after you wrote.</p>
<p>I first thought of something close to what you said, but what I’m concerned about the “A practical (optional) constraint” (above).</p>
<p>Are you aware of where one could find multiple pre-trained weights which were trained with the same procedure but different seeds?</p>

tom: <p>I am not sure there are, you could try to see if <a class="mention" href="/u/rwightman">@rwightman</a> did something in the timm context - he is probably one of the people training the most models.<br>
Both Ross (in timm) and TorchVision publish their training methods, so if you can afford the compute, you can make your own.</p>
<p>Best regards</p>
<p>Thomas</p>