Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper studies regularization for neural network fine-tuning, motivated by limiting deviation of the final model from the initialization states. The provide a generalization bound that utilizes a novel Rademacher complexity term built on the layer weights and their deviation from the initial weights. This bound relates particularly to fine-tuning, since a part of the bound can be fixed to the pre-trained weights, providing an alternative regularization objective specific to fine-tuning. Using this objective, the authors provide several fine-tuning benchmark experiments and demonstrate competitive performance.

Strengths of the paper:
- Well written, easy to follows.
- Motivation for the algorithm stems directly from the analysis, as opposed to heuristic-style arguments that typically dominate the field of CV /deep learning research, especially for fine-tuning. Moreover, the generalization bounds are derived such that they lead to an optimization objective (as opposed to conventional approaches that typically have not led directly to an effective algorithm).
- The analysis appears to be general, without any particularly strong assumptions.
- Two different norms are considered with corresponding algorithms and experiments.
- Extensive ablations are performed on vision tasks.

Weaknesses:
- Only tested on computer vision benchmarks. If the paper claims this approach to be a general technique then it is necessary that the methods do well on other tasks (e.g., language), otherwise the experimental claims rely too much on the convolutional inductive biases.
- If the paper is in fact framed as a CV paper, then it is natural that a comparison be made with respect to prior (albeit heuristic) computer vision research, e.g., label-smoothing regularization, entropy regularization and so on.
- An empirical comparison of the tightness of the bounds is warranted given the deviation of this analysis from PAC-Bayesian (Neyshabur 2018) or spectral norms (Bartlett and Long).
review_title: Effective, simple and well-motivated, although lacks in comparisons with prior work

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes new regularization methods for fine-tuning deep neural networks based on matrix $\infty$-norm distance. The authors claim that their choice of matrix $\infty$-norm distance is more suitable than commonly used Frobenius norm distance (a.k.a., Euclidean distance) when measuring the distance in the parameter space of convolutional networks by a comparison of two generalization bounds. Moreover, the authors empirically show that enforcing a hard constraint on the weights by projected methods throughout the training process is more effective in regularizing neural networks than widely used strategy of adding a penalty term to the objective function.

Overall, the paper is well written and has a nice logical flow. The problem of finding a proper distance metric for fine-tuning is interesting, though I have a few concerns outlined below regarding their theoretical analysis of using generalization bound to guide the choice of distance metric, especially the proof of the theorems. 

Concerns:
1. The authors try to modify the peeling technique of prior work to prove two generalization bounds, i.e., Theorem 1 and Theorem 2. A key step in proving the two theorems is to prove Lemma 2 given in the Appendix. However, from the proof of Lemma 2, if I understand correctly, the second equality and the fourth equality seem to interchange the order of sum and supremum freely, i.e., $\sum_{j=1}^n v_j \sup_{W_{1:k}} …=\sup_{W_{1:k}} \sum_{j=1}^n v_j…$, which of course does not hold in general. It should be stated clearly on why the two equalities hold here.

2. The authors provide two generalization bounds for fine-tuning. The two bounds are almost the same except for the norm used. The authors then claim that a comparison of the two bounds suggests that matrix $\infty$-norm is more effective than Frobenius norm when measuring distance in weight space of neural networks just because matrix $\infty$-norm itself is independent of the feature map size. This is misleading in the sense that matrix $\infty$-norm and Frobenius norm are actually equivalent, i.e., for an arbitrary matrix, its matrix $\infty$-norm is not strictly smaller than its Frobenius norm and vice versa, and thus the two bounds are also equivalent and cannot be used to tell which norm is better. Therefore, I do not think that their choice of matrix $\infty$-norm  as the distance metric can be theoretically justified by comparing the two generalization bounds as in the paper, despite that empirical results show that their method performs well in practice.

3. In section 5.3, the authors hope to demonstrate the ability of the distance-based regularization methods to control model capacity by sweeping through a range of hyperparameter values and plotting the corresponding predictive performance. The authors claim that Figure 2 shows that the PGM methods behave as the theoretical analysis predicts and the penalty-based approaches are not able to influence the model capacity as much as the constraint based approaches. This statement is inaccurate in several ways. First, the symbol $\lambda_j$ in the third line is confusing. It seems to represent the hyperparameter for both the constraint based methods and penalty methods. However, $\lambda_j$ first appears in equation (5) where it represents the hyperparameter for penalty methods. Second, from Figure 2, as $c$ becomes larger and larger, there is only a very small drop of accuracy for the PGM methods. So, it does not lead to overfitting, and the PGM methods do not behave exactly as the generalization bound predicts. Third, small $c$ for PGM methods corresponds to large $c$ for penalty methods by the equivalence of constraint methods and penalty methods. Therefore, Figure 2 shows that the penalty-based approaches actually have the same influence on the model capacity as the constraint based methods.

Minor comments:
- From the proof of Theorem 2, the term $\sqrt{c}$ in the bound should be $c$. Therefore, the bounds in Theorem 1 and Theorem 2 exhibit the same dependence on the number of classes.

- I am a little confused by the sentence “In the case of the final classification layer, $W_L^0$ can be randomly initialized.” in 7th line of Section 3. Do you mean that $W_j^0$s ($j<L$) are pre-defined and fixed, but $W_L^0$ is random? However, when proving the upper bound for empirical Rademacher complexity, especially the last step where the rightmost term evaluates to zero, it seems that you assume that all these matrices $W_j^0$ ($1\leq j\leq L$) are fixed. It would be better if this can be clarified.

- In section 4 and Appendix E, to support the claim that projection based methods are better than penalty based methods, the authors state that penalty methods have weaker assurance on whether a constraint is being forced. However, Figure 1 shows that for ResNet101 model penalty-based method is actually more effective in enforcing the constraints in the sense that not only it successfully constraints weight distance to be less than $\gamma_j$, but also the number of weights which have small distance is larger. Therefore, more evidence might be needed to support their claim.

Some typos:

(1) In line 6 of Page 2, best way restrict -> best way to restrict

(2) In the last line of Page 5, change the $l^1$ distance-> change the MARS distance

(3) In the third line of the proof of Lemma 2, $\varphi_j$-> $\varphi$

(4) In the third formula of the proof of Theorem 2, $sqrt{2}$-> $\sqrt{2}$
review_title: Official Blind Review #3 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The work proposes a Rademacher type bound for the fine-tuned models based on the distance between the fine-tuned weights and the pre-trained weights. Since the distance term shows up in the upper bound on the generalization gap, the authors further propose to adopt it as the regularization term to boost the generalization performance of the model during the fine-tuning process. Some experiments are also done to show the effectiveness of the proposed regularization.

I am seriously concerned about the correctness of the Rademacher-type bound the authors have proposed. The bound does not seem correct to me.

The flaw comes from the function class F_* defined in section 3 of the draft. The function class F_*, by definition, depends on the pre-trained weights W_j^0. However, W_j^0 is not fixed, it is random! This is because W_j^0 depends on the data (W_j^0 is pre-trained using the data), which by the assumption of the draft, is random. As a consequence you cannot assume W_j^0 as fixed. The randomness of the hypothesis class F_* destroys almost all the derivations the authors are currently using in their proof. 

Another minor bug is the second term in the bound for theorem 1 seems to have some subscript issues. To me the product term related to B_j^\infty should go from i=1 to j instead of from j=1 to L. I may have missed something in this point but could the authors double check if the subscript of the B_j^\infty is correct? In particular the derivation from the second inequality to the third on page 13 of the appendix. 

The second issue is easy to fix. However the first issue seems like a fundamental flaw. I do not have a good way to handle it for now. 
review_title: the proof seems problematic

final_score: rating: 7: Good paper, accept
review_content: In this manuscript the authors derive a bound on the rademacher complexity of neural network models which can be written as a funciton of the MARS norms of the weights in the network. This motivates the authors to put a regularization on the MARS norm of the network weights instead of the more typical L2 norm. Here the authors implement this regularization as a hard bound on the weights, which they enforce by projecting the weights back on the allowed ball. They use their regularization for transfer learning of ResNet-101 and EfficientNetB0 from ImageNet onto the set of smaller image classification tasks. On these tasks, the projection methods and to a smaller degree the MARS based methods generalize better. 

Overall I vote for acceptance. This is an interesting contribution to the literature, providing both a theoretical insight and an experimental test that this theoretical insight is relevant for applications. However there is a certain disconnect between the theory and the experimental observations. Performance  benefits more from the projection methods than from the switch of norm although the switch of norm has a much stronger theoretical justification.

Pros: 
1) Well structured paper with interesting results
2) Theoretical results are well justified to be more helpful than existing bounds.
3) There is an empirical test that the switch in bound is helpful for practice.
4) Overall the generalization is actually improved.

Cons:
1) Empirically the less justified change has a larger impact, indicating that there might be another more important theoretical insight
2) The hyperparameter setting procedure remains opaque. The authors always talk about gamma_i/ lambda_i parameters changing the strength of regularization per layer, but only test how scaling all regularizations up or down affects performance. A description how the values were chosen is really necessary I think and some analysis to convince us that the worse performance of the regularization is not caused by a bad hyperparameter choice would definitely be a plus.
3) I think there is a bit of a missed opportunity here for the scaling over layers as the bound suggests an unequal weighting of the layer norms. I think directly regularization of the bound which would allow layers to compensate for each other or giving each layer an equal budget in terms of raising the bound would be interesting variants here.
review_title: Interesting for both theory and practice, although the connection between the two is a bit weak

Meta-review: This paper proposes constraints to be applied to the weights of a deep neural model during training. These constraints, motivated by an analysis of Rademacher complexity, are compared with other constraints and penalty approaches in transfer learning. The authors were able to build on the reviewers feedback to improve their paper on several points during the discussion phase, leading to a consensus for acceptance among reviewers. They also agreed to conduct experiments targeting stronger experimental results to compare all methods in the situation where they provide state-of-the-art results. This will make a useful contribution to the ICRL audience, and I recommend acceptance.

Decision: Accept (Poster)