Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the two-stage transfer learning approach of head tuning first (a generalization of linear probing where the head can be nonlinear), followed by finetuning both the backbone and continuing to finetune the head. Their work builds upon the setup of Kumar et al, but they claim to relax the assumption made in that work that the pretrained model is optimal for the downstream task. 

They decompose the training update to the features during finetuning and show theoretically that when the adaptation ‘energy’ increases (meaning that the predictions on the downstream task differ from the correct labels), the final representations (obtained after finetuning) change more. So, intuitively, controlling that energy is one way of controlling how much adaptation is performed on the features. One way to increase the energy is by doing less head probing before the fine-tuning phase (this way the head will be less suited to the downstream task at the start of the finetuning phase, so more adaptation occurs). Since the number of head probing epochs is difficult to tune, they propose label smoothing as an additional knob that can affect energy increase and also explore the effect of the capacity of the head. They show in several empirical scenarios results consistent with their intuitions and theoretical results. 

strength_and_weaknesses: Strengths
========
- To the best of my knowledge, this analysis of the influence of ‘energy’ in the head on the changes made to the features during FT is novel and it is interesting.

- The finding that adding label smoothing can make the HP-FT procedure more robust to the number of epochs for HP is an interesting one.

Weaknesses
==========
- I found that the paper has clarity issues (see below)

- Does label smoothing always help? Are there cases where this addition can degrade performance? (I see at least one such entry in Table 1, for Flowers dataset, Sup-C10). The authors don’t discuss the potential dangers of using it / weaknesses of this approach.

- I’m not sure what the practical take-aways are (see below)

clarity,_quality,_novelty_and_reproducibility: Clarity
=====

Fig 1, the authors mention that one of the ways the setup differs between their work and Kumar et al is that ‘information exchanges between the two parts of the network’. I don’t really understand this point. Doesn’t this information exchange always happen during finetuning? What is special here?

In Section 2.3, for the different degrees to which we want to change the feature extractor, in ‘strong’, what is meant by ‘the pretrained model overfits’? Not clear if this refers to overfitting to the upstream task or the downstream task, and how exactly this is measured.

Fig 1 caption: define X_{PT} and X_{DS} – I assume it’s ‘pretraining’ and ‘downstream’ respectively? But this should be stated.

Under Equation 1, in the equation z = Bx, I think this x should be X.

In Section 3.1, it’s not clear to me what the superscript 0 in z_t^{(0)} and x^{(0)} means. Is this the first dimension of the embedding or the first example in the batch or something else?

Figure 2: should explain the caption. I’m assuming this HP x means that the head probing was run for x epochs? Should clarify.

The title of Section 3.2 is “Initial value: two extreme cases”. I did not understand what this is referring to. Initial value of what? And what are the two extreme cases? Are they ‘strong’ and ‘tiny’? It’s confusing as the results shown and discussed in this section (e.g. in Figures 2 and 3) show FT results for various phases of HT tuning which if I understand correctly are the ‘intermediate’ cases?

Figure 4 caption says it’s clear that mild adaptation makes the features more separable. More separable than what? Looks like strong adaptation makes them even more separable. Also, this figure should clarify what is the task in this case. I assume we don’t always (i.e. for all tasks) expect greater separability with more adaptation. What does this depend on? Could we have the opposite phenomenon for other tasks?

I don’t understand Proposition 1. My understanding was that when the energy increases, the euclidean distance between original and final features also increases. But Proposition 1 states the opposite. Is this a mistake?

Figure 5: the names of the columns are hard to understand. I also don’t know what “1-gap” means. Why is the last column LP (linear probe?) instead of HP (head probe)? Not clear what exactly is the experimental setup here and what is the intended take-away.

The authors mention their analysis is also relevant for finding features that work well in a new, OOD domain. It would be useful to elaborate on this, as I feel if this is true it would add a lot of value to the paper. But I’m not currently able to see how it’s true.

Figure 6: the caption says HP but the x-axis of subplots says LP. which is it?

The tasks used in all tables should be described. All notations used in figures should be described. There were several other instances aside from the above where I wondered what exactly a symbol means.

Quality
======
In practice, it is not easy to judge to what degree we want to update the pretrained model for a new downstream task (‘strong’, ‘mild’ or ‘tiny’). This is due to often not having enough data or resources downstream to assess aspects like ‘the model overfits’, and in addition it is difficult to estimate task relatedness (e.g. how related or different is the downstream task to the upstream one) - this in and of itself is an open problem. So it’s not easy to tell whether the pretrained model is ‘reasonably good’ (for mild for instance) on the downstream task. Given this, it is not clear to me what is the recommendation given by the authors for how to modify the way in which we finetune models, since the recommendation seems to be conditional on somehow making this difficult choice.

While the paper proposes a set of practical tricks to alleviate this, like label smoothing, I’m not necessarily convinced that it always helps and never hurts (for all downstream tasks). For example, I see at least one entry in Table 1 where it can hurt performance, for the Flowers dataset, with Sup-C10. It feels like more analysis on the effect of label smoothing needs to be done before this recommendation can be safely made.

In Figure 3, it seems that the ‘end’ histogram always matches the e_y pretty well, regardless of the head used (random or after HT). This doesn’t really make a strong case for having to be careful about how we do HT / FT. Are there other cases where we can see bigger differences? Perhaps transferring to a downstream task that is “very different” from the upstream one?

While the analysis here is interesting, it’s not clear to me how exactly it relates to the ultimate goal: analyzing the effect of design choices of HP/FT on the actual downstream *performance*. As the authors also pointed out, analyzing the change on the features is an indirection and it’s less clear why it’s relevant / how to inform practical decision making.

Section 3.3 “backbone depth and head capacity” makes some intuitive observations which however aren’t too surprising, and also aren’t very actionable. Again, how can we assess whether low-level features (of the pretrained model) will be beneficial for a new downstream task while high-level features harmful? 

Novelty
======
The proposed particular analysis is novel to the best of my knowledge. 

Reproducibility
============
The procedure proposed is simple enough that I feel confident I could implement it. But I encourage the authors to provide all details about hyperparameters, task details etc, to aid in reproducibility, and make their code available.

summary_of_the_review: This paper analyzes the effect of different head designs on the adaptation that occurs in the learned features after the finetuning phase. Their analysis is novel to the best of my knowledge and some findings are interesting. However, I found that the writing and presentation can be improved (see clarity section above) and I’m also unsure about the usefulness of practical takeaways from the paper (see above detailed comments). For this reason, I’m doubtful if the size of the contribution meets the bar for acceptance.


===============================

After rebuttal: The new version of the paper is clearer and better structured. I particularly liked the User Guide section. I also like the discussion around difficulty in making these design choices in practice, the risks associated with some decisions (e.g. label smoothing, etc) and the inclusion of recommendations for making them based on increasingly-risky interventions. Based on this, I increased my score from a 5 to a 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors focus on the choice of task head in fine tuning, and how the task head controls feature adaptation and downstream model quality. 

The gradient was decomposed as the product between a direction and an energy, and analysis was done on the effect of the energy vs feature adaptation after FT step.

The paper's extensive experiments would have guidance to how to use a few techniques to get better model, such as early stopping at HP, and label smoothing at HP.
strength_and_weaknesses: Strength
1) Extensive study for numerical experiments.
2) The energy decomposition seems novel.
3) The problem of how to make pretraining / fine tuning work better is very core, and thus the paper may seem of great interest to people in the area.

Question
The overall writeup seems challenge for a regular non-DL-theory practitioner to follow, even with the Section 4 on real world problems. Can you please make the implication / take away message a little more explicit for model training of real-world problem?
clarity,_quality,_novelty_and_reproducibility: Novelty seems mostly on the theoretical area and significant. The take away messages from some sections seem to be aligned with the empirical result, such as a higher energy needs a longer feature adaptation. Was there some take away message that go beyond the general empirical impression? Please let me know if I do not understand well

Clarity is good but maybe challenge to follow if the reader does not work on DL theory.
summary_of_the_review: Recommend as accept but marginally above threshold. If the authors could help me to appreciate better the paper's empirical value, then it would be awesome
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies how the choice of task heads influence the pretrained features z’s adaptation and hence influence the downstream performance.  The authors explain feature adaptation by decomposing the learning dynamics and identify the energy and direction terms matter most. They find that the training accuracy and loss at the beginning of finetuning determines the energy for the feature’s adaptation. They also identify a trend in the effect of changes in this initial energy on the resulting features after fine-tuning, which helps to design an appropriate fine-tuning procedure. Finally they provide examples of how to enhance downstream performance in practice. They suggest paying more attention to how much and in what ways we want the features to adapt.

strength_and_weaknesses: Strength
- The paper is well structured and written.
- The paper characterizes the change of the backbone and head during finetuning, which provides a better understanding about the evolving process of feature adaptation during fine-tuning.

Weakness
- The authors made observations on the quadratic trend of $z_t^Tz_0$ and $\|z_0\|_2^2$, however, it is not clear how this trend can help adapt features on the downstream task.
- The authors claimed that the change of the NTK term is slow during FT without further examination. Is it possible that the NTK term also matters when the source and target data are different and the change could be large?
- How much to adapt may also depends on the dataset size, which were not considered.
- There are no recommendations for how to select the backbone depth or manipulate the head’s capacity. The selection of $\tau$ is also not clear.

clarity,_quality,_novelty_and_reproducibility: The paper is well written. The novelty of the paper lies at understanding the transfer learning performance by analyzing two key terms with different conditions. 
summary_of_the_review: The paper presents an analysis of the fine-tuning behavior with head probing in different settings. However, it is not clear whether the analysis can lead to a concrete approach or recommendation for controlling feature adaption.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the role of choice of task head, i.e. head tuning (HT), in the downstream performance of fine-tuning (FT) from a pre-trained model.
Recent work has suggested that it is a good idea to first perform head tune on a downstream task and then use that head for fine-tuning (so that features do not move too much), rather than fine-tuning from a randomly initialized head (where features move a lot).
This paper also considers an intermediate case of stopping early during head tuning (to mildly adapt features) and finds empirically that in many cases the optimal stopping criterion is somewhere in the middle.

The paper delves deeper into the amount of feature adaptation w.r.t. number of steps $\tau$ of HT performed initially, and find some interesting trends like norm of change in features $\|z\_t - z\_0\|$ is monotonically decreasing w.r.t. $\tau$ while the norm of the features $\|z\_t\|$ has a quadratic behavior. Theory on an overparametrized linear setting (NTK regime) that tries to explain these trends.
The analysis of 1 step of feature dynamics highlights an “energy” term that strongly determines the amount of feature adaptation: energy is low when initial HT on top of pre-trained features already has good accuracy and vice versa.


Using these findings and analysis, the paper identifies label smoothing during HT as a way to control the “energy” and allow feature adaptation. This often (but not always) performs comparably in practice to the optimal $\tau$ (number of HT steps) which is not easy to find.
strength_and_weaknesses: **Strengths**

- Overall the paper presents an interesting analysis of the role of task head in the fine-tuning performance. The following observations are interesting: the optimal stopping criterion is somewhere in the middle and that label-smoothing during head-tuning can help get closer to the performance of this optimal stopping criterion.

- The paper also performs deeper analysis into some of these phenomena, e.g. analyzing the level of feature adaptation as a function of number of initial HT steps $\tau$ or "energy". Many of the hypotheses are verified empirically even if not theoretically proved.


**Weaknesses**

- A lot of the analysis is hand wavy. For e.g. the effect of the task head on the FT features is studied by just looking at just 1 step of gradient descent. The notion of "energy" derived from this is also vague since it is technically a vector. Although empirically heuristics like the training accuracy or probability gap seem to be a good proxy for "energy". The theoretical result for the linear model uses $\alpha$ in $\alpha Y$ as a proxy for "energy". Despite all of these, the analysis seems somewhat meaningful.

- I think the presentation of the paper can be significantly improved. I found the paper hard to follow and it took me at least 2 passes to understand how the different points in the paper connect to each other. There is a constant switching from number of HT steps being important to the "energy" being crucial to feature adaptation being the key. Rather than presenting the analysis first, I think it might help to start with the empirical observation that the ideal choice of task head is to initially HT for some $\tau^*$ number of steps before FT. Then the goal of the entire study could be to understand this phenomenon and try to achieve this optimal performance in a simple way (e.g. label-smoothing is a solution). Without such a concrete goal, I was lost many times about the purpose of different sections.

Other comments/questions:
- If my interpretation is correct, it is a bit misleading to suggest in Proposition 1 (informal) that the result is about the effect of $\tau$, because the X-axis seems to be something different, $\alpha$ in $\alpha Y$
- "This kind of z can make the original overlapped features become separable after adapting to a similar manifold" on page 5. Could you please elaborate what this means?
- How was the label-smoothing coefficient of 0.9 picked? How much of an effect does this have?
- The discussion in Section 3.3 about head capacity seems to be hand-wavy and a distraction, since most of the results seem to use a linear head
clarity,_quality,_novelty_and_reproducibility: The observations about the effect of number of HT steps on FT is novel and relevant, to the best of my knowledge. There are many issues regarding clarity that are described above.
summary_of_the_review: Overall I find the contributions of this paper in understanding the role of task head in fine-tuning to be positive. However due to the presentation concerns I have only assigned a score of weak accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: As clearly stated in the title, the paper presents an extensive analysis of how to prepare (or train) a task head in a backbone + task head neural network configuration. The authors also make a lot of reference to the cited paper by Kumar et al., frequently pointing out the differences in their approach (2022). The authors analyze the behavior of features after model pre-training and during the head probing (HP) phase, in which only the parameters of the head are updated, and the fine-tuning (FT) of the entire network. The analysis is carried out both with a toy example and also with more general downstream real tasks. By decomposing the learning dynamics of the representation
extracted by the backbone, they find the keys are the energy and direction terms, which highly correlate with the accuracy at the beginning of the finetuning process, then they show how to manipulate the energy (e.g., stop the Head Probing stage earlier, label smoothing only in Head Probing, non-linear head, etc.) to enhance the downstream performance.
strength_and_weaknesses: strenghts:

The paper is well written and the problem is clearly stated multiple times
● Every consideration is presented both in a discussion and in a formal fashion
● Several comparisons are made with other similar analyses, outlining the differences in the proposed approach
● When realized (see the weaknesses), the experiments are extensive and well commented
● The topic of the paper is very relevant, and such an analysis, if done comprehensively, can be very useful, theoretically and practically, to many researchers in building better models and learning procedures

weaknesses

Sometimes the claims and observations of the authors seem a bit too excessive in comparison with the results actually obtained from the experiments. Some example are:
○ “Although we do not analyze the relationship between the change of zt and out-of-distribution performance – this adds too many “moving parts” to the current setting – our analysis is also relevant to finding features which will work in new, unseen domains” in Section 4. I think that out of distribution analysis is necessary to get a more thorough idea about the interaction between HP and FT, as stated in the highly mentioned Kumar et al., 2022
○ The paper lacks experiments regarding the relationship between backbone depth and head capacity. There are some examples in the appendix but they are too limited, considering that the use of non-linear and more complex heads (instead of the more basic linear probing) is one of the 3 practical suggestions given in the introduction
● The experimental section lacks some information (that can often be found in the appendix, but that I consider essential for the fruition of the paper):
○ A description, even short, of the used architectures (and how they are decomposed in backbone and head)
○ A small description of the considered configurations, for example of the Byol and MoCo unsupervised cases
○ In table 1 it is not clear to me what HP𝛕*-FT means. If it means the configuration with the best 𝛕, how 𝛕 was chosen?
● Some images (Figure 8 and 11) in the appendix are cited in the paper. Since these are quite useful perhaps it would be appropriate to include them somehow in the main paper
● I think the difference between the dataset used for pre-training and the dataset used for fine-tuning is a key element of such an analysis. This and what is meant by distance between tasks are elements that should be explored further in the main paper in my opinion.
clarity,_quality,_novelty_and_reproducibility: clarity:  although the subject matter is quite basic, some of the elements involved require in-depth theoretical knowledge, e.g. theory
behind the empirical neural tangent kernel (NTK). Experiments should have been better organized.
novelty: the approach is novel
originality: is original 
summary_of_the_review: The paper could have been definitely better with a stronger experimental section (see my comments above.)

POST REBUTTAL:
The paper is now satisfying, the User Guide section is useful, many explanations have been provided.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper analyzes how the choice of task head controls feature adaptation and hence influences the downstream performance.  By decomposing the feature's learning dynamics, they find the key aspect is the training accuracy and loss at the beginning of finetuning, which determines the "energy" available for the feature's adaptation.

Strength:
-  The analysis of the influence of ‘energy’ in the head on the changes made to the features during FT is novel and interesting.
-   Every consideration is presented both in a discussion and in a formal fashion 
- Several comparisons are made with other similar analyses, outlining the differences in the proposed approach


Weakness:
- in practice NTK is slower and assuming being in the NTK regime weakens the paper
- applicability and clear takeaways for practitioners can be improved
- initial submission had a number of clarity issues that are mostly solved during the rebuttal period.
- non-trivial trend and gap estimation: it is not clear how the "non-trivial trend" helps to design the proposed of \tau sweeping strategy. 
- It is not explained how optimal learning rate is selected for different tasks. It might indicate the necessity to search both learning rate


The paper was significantly changed during the rebuttal period and in response to the reviewers comments.
justification_for_why_not_higher_score: please see the meta review
justification_for_why_not_lower_score: There has been significant improvement during the rebuttal period.
Decision: Accept: poster