Distance-Based Regularisation of Deep Networks for Fine-Tuning
Authors: Henry Gouk
Timothy Hospedales
Massimiliano Pontil
Venue: ICLR 2021 Poster (2021)
Keywords: Deep Learning
Transfer Learning
Statistical Learning Theory

Abstract: We investigate approaches to regularisation during fine-tuning of deep neural networks. First we provide a neural network generalisation bound based on Rademacher complexity that uses the distance the weights have moved from their initial values. This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks. Our bound is highly relevant for fine-tuning, because providing a network with a good initialisation based on transfer learning means that learning can modify the weights less, and hence achieve tighter generalisation. Inspired by this, we develop a simple yet effective fine-tuning algorithm that constrains the hypothesis class to a small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results. It outperforms both state of the art fine-tuning competitors, and penalty-based alternatives that we show do not directly constrain the radius of the search space.

Published as a conference paper at ICLR 2021
DISTANCE-BASED REGULARISATION OF DEEP NETWORKS FOR TUNING
Henry Gouk & Timothy M. Hospedales
Massimiliano Pontil
School of Informatics
CSML, Istituto Italiano di Teenologia & &Department of Computer Science, UCL mass mi 1 iano. pont 1iliitiiiii
University of Edinburgh
(henry. gouk, t. hospeda es
l @ed. ac. uk
ABSTRACT
We investigate approaches to regularisation during fine-tuning of deep neural networks. First we provide a neural network generalisation bound based on Rademacher complexity that uses the distance the weights have moved from their initial values. This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks. Our bound is highly relevant for fine-tuning, because providing a network with a good initialisation based transfer learning means that learning less, and hence achieve tighter generalisation. Inspired by this, we develop a simple yet effective fine-tuning algorithm that constrains the hypothesis class to small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning. Empirical evalu- ation shows that our algorithm works well, corroborating our theoretical results. It outperforms both state of the art fine-tuning competitors, and penalty- based alternatives that we show do not directly constrain the radius of the search space.
modify the weights
INTRODUCTION
The ImageNet Large Scale Visual Recognition Challenges have resulted in a number of neural network architectures that obtain high accuracy when trained on large datasets of labelled examples (He al.] 2016 Tan and Le 2019 Russakovsky et al. 2015). Although these models have been shown to achieve excellent performance in these benchmarks, in many real-world scenarios such volumes of data are not available and one must resort to fine-tuning trained on a large dataset, to initialise the weights for a model that will be trained on small dataset. The assumption being that the weights from the pre-trained model provide better initialisation than randomly generated weights. Approaches for fine-tuning are typically ad hoc, requiring one to experi- ment with many problem-deye -eeendet tricks, and often process that will work for one problem will not work for another. Transforming fine-tuning from an art into a well principled procedure is therefore an attractive prospect. This paper investigates, from both theoretical and empirical point of view, the impact of different regularisation strategies when fine-tuning a pre-ttrinne network for new task.
existing model: taking the weights from model
Existing fine-tuning regularisers focus on augmenting the cross entropy loss with terms that indirectly or directly penalise the distance the fine-tuned weights move from the pre-trained values. The intuition behind this seems sensible-the closer the fine-tuned weights are to the pre-trained weights, the less in- formation is forgotten about the source dataset- buseeet-uttttt is not obvious how this idea should be translated into an effective algorithm. One should expect that the choice of distance metric is quite important, but existing methods exclusively make use of Fuclidean distance (Li et al. 2019 2018) without any theo- retical or empirical justification regarding why that metric was chosen. These methods achieve only a small improvement in performance over standard fine-tuning, and it is reasonable to expect that using a metric more suited to the weight space of neural networks would lead to greater performance. More- over, while the of penalty terms to regularise neural networks is well established, the impact of using penalties vs constraints as regularisers has not been well studied in the context of deep learning.
In order to study the generalisation error of fine-tune models, derive new bounds the empirical Rademacher complexity of neural networks based on the distance the trained weights move from their initial values. In contrast to existing theory (e.g.. Neyshabur et al. (2018); Bartlett et al. (017);

---

Published as a conference paper at ICLR 2021
Long and Sedghi (019)), ) we do not resort to covering numbers make use of distributions over models to make these arguments. By deriving two bounds utilising different distance metrics, but proved with the same techniques, we are able to conduct controlled theoretical comparison of which metric one should use as the basis for fine-tuning regularisation scheme. Our findings show that metric based the maximum absolute row sum (MARS) matrix norm is a more suitable measure of distance in the parameter space of convolutional neural networks than Euclidean distance. Additionally, we challenge the notion that using penalty term encourage the fine-tuned weights to lie near the pre-trained values is the best way to restrict the effective hypothesis class. We demonstrate that the equivalence of penalty methods and constraint methods in the case of linear models (Oneto) et al.) 2016) does not translate to the context of deep learning. As result, using projected stochastic subgradient methods to constrain the distance the weights each layer can move from the initial settings can lead to improved performance.
Several regularisation methods are proposed, with the aim of both corroborating the theoretical analysis with empirical evidence, and improving the performance of fine-tuned networks. One of these approaches is a penalty-based method that regularises the distance from initialisation according to the MARS-bbaee distance metric. The other two techniques make of efficient projection functions to enforce constraints on the Euclidean and MARS distance between the pre-trained and fine-tuned weights throughout training. The experimental results demonstrate that projected subgra- dient methods improve performance over using penalty terms, and that the widely used Euclidean metric is typically not the best choice of metric to measure distances in network parameter space.
2 RELATED WORK
The idea of developing an algorithm to restrict the distance of weights from some unbiased set of reference weights has been explored in various forms to improve the performance of fine-tuned networks. Li et al. (2018) presented the ESSP regulariser, which consists of adding term to the objective function that penalises the squared g distance the trained weights from the initial weights. This is based applied to linear support vector machines (Yang et al. (007)). The subsequent work of et al. (2019) follows the intuition that the features produced by the fine-tuned network should not differ too much from the pre- trained features. They also use Euclidean distance, but to measure distance between feature vectors rather than weights. The idea is extended to incorporate an attention mechanism that weights the importance of each channel. The method is implemented by adding a penalty term to the standard objective function. In contrast to these approaches, we solve a constrained optimisation problem rather than adding a penalty, and we demonstrate that the MARS norm is more effective than the Euclidean norm when measuring distance in weight space.
idea initially made use of when performing domain adaptation, where was
Many recent meta-leaming algorithms also make use of idea that keeping fine-tuned weights close to their initial values is desirable. However, these approaches typically focus on developing methods for learning the initial weights, rather than working with pre-specified initial weights. The model-agnostic meta-learning approach (Finn et al.) (017)) does this by simulating few-shot learning tasks during the meta-learning phase in order to find good set of initial weights for neural network. Once the learned algorithm is deployed, it adapts to new few-shot learning tasks by fine-tuning the initial weights for small number of iterations. [enevi et al. (2018) proposes a modified penalty term for ridge regression where, instead of penalising the distance of the parameters from zero, they are regularised towards a bias vector. This bias vector is learned during the course of solving least squares problems on a collection of related tasks. Denevi et al. (2119) extend this approach to a fully online setting and more general family of linear models.
Previous work investigating the generalisation performance of neural network based on the distance the weights have travelled from their initial values has done so with the aim of explaining why existing methods for training models work well. [Bartlett et al.] (2017) present bound derived via covering numbers that shows the generalisation performance of fully connected networks is controlled by the distance the trained weights are from the initial weights. Their bound makes use of a metric that scales with the number of units in the network, which means if the result is extended to class of simple convolutional networks then the generalisation performance will scale with the resolution of the feature maps. A similar bound can also be proved through the use of PAC- Bay-BBaa analysis (Neyshabur et al. 2018). One can make use of different metrics and techniques for applying
2

---

Published as a conference paper at ICLR 2021
covering numbers to bounding generalisation that do not have the same implicit dependence on the number of units, but they instead depend directly on the number of weights in the network (Long and Sedghi 2 Neyshabur al. (2019) investigate the performance of two layer neural networks with ReI.U activation functions, demonstrating that as the size of the hidden layer increases, the Frobenius distance (i.e., the Frobenius norm of the difference between initial and trained weight matrices) shrinks, Inspired by this observation, they show that one can construct a bound on the Rademacher complexity of this class by using Euclidean distance between the initial weights and trained weights of each individual unit. Although they bound the Rademacher complexity directly, they still incur an explicit dependence on the size of the hidden layer, and their analysis is restricted fully connected networks with only single hidden layer.
In contrast to these previous studies, our focus is on designing an algorithm that will improve the performance of fine-tuned networks, rather than explaining the performance of the standard fine-tuning methods that are already widespread. Therefore, we do not aim infer what properties of existing methods enable networks to generalise well-we instead derive such properties and then develop algorithms that enforce them. Moreover, we put a particular emphasis on choosing a metric that is suitable for contemporary convolutional networks, and thus will not scale with the size of the feature maps, while also being easy to implement efficiently.
3 DISTANCE-BASED GENERALISATION BOUNDS
Throughout this section we will analyse a loss class consisting of feed-forward neural networks,
(() (Φr o --- Φn (z², where each (z) = (((,ā) is a layer with 1-Lipschitz activation
function, ş. Both the norm of the weight matrix of each layer and the distance of the fine-tuned weights from the pre-trained weights are bounded from above. Formally, we define
F. = y) + (y, fz)) [[W S B3, [W S. By, IIW Will 5
[
where j goes from to L, 1 is p=Lipschitz loss function with range of 1], and WTO is the pre-trained weight matrix for layer j. Each of the pre-trained weight matrices can be either random variables drawn from a distribution with support such that the constraints are always fulfilled, or they can be fixed (i.c., drawn from a Dirac delta distribution). The only other requirement is that they independent of the training data used for fine-tuning. From empirical perspective it is most useful to consider them non-randon quantities. We have used * as a placeholder for the norm used to measure the magnitude of weight matrices and the distance from the pre-trained weights. The primary focus is on using the MARS norm,
| max
3
to prove a bound on the empirical Rademacher complexity (Bartlett and Mendelson 2002) of Foo- Our main theoretical result, presented below, is obtained by modifying the "peeling" -style arguments typically used to directly prove bounds on the Rademacher complexity of neural network hypothesis classes (for examples, see Neyshabur et al. (2015); Golowich et al. (2018).. Our modification of the argument allows to rephrase the resulting theorem in terms of the distance the parameters can move from their initialisation during training.
Theorem 1. For all 8 E (0, 1), the expected loss of all models in Foo is, with probability 1 6, bounded by
1oog Σ₁ B
D 11 2B
E (()))) 1 1n "E(EE))
1og(2/0)
+3
Vm
2m
where is the number of training examples, is the number of classes, F, E Rdd and [ S C.o.
Crucially, when / is chosen carefully (e.g., the ramp loss), the expectation in Theorem is an upper bound for the expected classification error rate. The proof for Theorem [ can be found in the supplemental material. The two main terms in this theorem are (i) the product of bounds on the layer norms; and (ii) the summation, which is a bound on the distance the fine-tuned weights can be from the pre-trained weights. The first of these is primarily dependent on the weights obtained via pre-training, whereas (ii) can be controlled during the fine-tuning process. Moreover, one would expect that if better initial weights are selected via pre-treeee are-training, then the distance the final weights will be from the initial values will be smaller, thus leading to better generalisation. This is the motivation behind the regularisers we develop in Section
3

---

Published as a conference paper at ICLR 2021
An analogous bound is also derived for the Frobenius norm (i.e., FP) to facilitate theoretical comparison between the proposed regularisation method and SP,, an existing approach that relies on penalising Euclidean distance between the pre-trained and fine-tuned parameters. In order to avoid a direct dependence on the number of parameters (as accomplished in Theorem T it is necessary to restrict Fr only ReL.U activation functions.
Theorem 2. For all 8 (0, 1), the expected loss of all models in FF is, with probability 1 6, bounded by
2v2cpC2 Σ)=
D
L I1 1=1
2B
2 II
log(2/0)
E (((())))) (E.y)
((f(E)))
+31
Vm
2m
where mn is the number training examples, is the number classes, ||:il22 S C2. pp.) = (.)LU.),
and ny is the number of columns in Wj:
We can make several observations by comparing our two bounds with each other and previously published bounds: (i) in contrast to Theorem | Theorem incurs a significant dependence on the resolution of the intermediate feature maps due to the vn, factors; (ii) each term in the summations in our bounds can be at most one, whereas the corresponding terms in Bartlett et al. (0117) can be more than one due to the use of multiple types of norms; (iii) our bound from Theorem does not have a direct dependence on the number of weights, in contrast to the bound provided by Long and Sedghi 2019). More detailed comparisons can be found in the supplemental material.
FINE-TUNING WITH DISTANCE REGULARISATION
The analysis presented in Section suggests that the weights in fine-tuned models should be close to the pre-trained weights in order to achieve good generalisation performance. More specifically, the learning process should search only within a set of weights within predefined distance from the pre-trained weights. We discuss two strategies for accomplishing this: using projected subgradient methods to enforce a hard constraint, and augmenting the standard cross entropy objective with a term that penalises the distance between the pre-trained and fine-tuned weights. The method based on projection functions is attractive because it guarantees that the constraints will be fulfilled even if one uses heuristic optimisation method to train the network parameters. However, the penalty- proaches are more common in the literature, and convenient from an implementation point of view due to the ubiquity of automatic differentiation. Nevertheless, in contrast to the projection- based meth- ods, the techniques that use penalties have weaker assurances on whether constraint is actually being enforced. We discuss the drawbacks of this non-equivalence further in the supplemental material.
ap-
4.1 OPTIMISING WITH PROJECTIONS
One way enforce constraints on the weights of neural networks during training is variant of the projected stochastic subgradient method. This is similar to typical stochastic subgradient methods used when training neural networks, but has the additional step of applying projection operation after each weight update to ensure that the new weights lie inside the set of feasible parameters that satisfy the constraints. In the case of classic subgradient descent, in order to guarantee convergence towards a stationary point the projection function must perform a Euclidean projection,
. ()) arg min IIW W
W 2
(1)
s.t. (W)) < 0,
where W are the newly updated parameters that may violate the constraint, W are the projected parameters, and g()) is convex function specifying the constraint. Although the Euclidean projection is required for the classic projected subgradient method, other optimisation algorithms may require the projection to be performed with respect to a different metric. Looking at different optimisers as instantiations of mirror descent with different Bregman divergences is one way to determine the type of projection that should be performed. Unfortunately some of the most common optimisers used when
in deep learning, such as Adam (Kingma and Ba 2015). are not guaranteed to converge there are no constraints on the parameters being optimised (Reddi et al.) 2018)). This makes extending it to perform constrained optimisation purely heuristic endeavour, further compounded by the fact
4

---

Published as a conference paper at ICLR 2021
that it is also not clear which metric the projection should be performed with respect to. As such, the projections used in our approaches are performedd with respect to the norm that is most convenient from an efficiency point of view.
Rather than attempting to constrain the distance between the all pre-trained and fine-tuned weights in the network using single projection, constraints applied layer-wise basis. This makes optimisation more manageable and also allows practitioners to favour fine-tuning certain parts the network-e.gg if one is fine-tuning network pre-trained photos perform task on paintings where the same underlying classes are present, one might wish to allocate more fine-tuning capacity earlier layers in the network. The types of constraints that we wish enforce take the form
IW; W/ ||-
%/
where "i is a hyperparameter that corresponds to the maximum allowable distance between the pre-trained weights and the fine-tuned weights for layer j. With minor rearrangement, this yields constraint specification in the form required for Equation
9j (W W 7;) = |[, - W W 7j
where we have made the dependence on the pre-trained weights and the hyperparameter explicit. The resulting optimisation problem is
T2
Kyr (oz .... (())(
(2)
|[W S. SS (1..
The remainder of this section presents derivations for the projection functions, Fr and Tocs responding to this constraint specification when it instantiated with the Frobenius norm and the MARS norm, respectively. We provide pscudocode in the supplementary material that illustrates how these projections are integrated to the neural network fine-tuning procedure when using a variant of the stochastic subgradient method. We refer the Frobenius norm instantiation as e-PGM and the MARS norm version as MARS-PGAS where the PGM indicates the use of projection gradient methods.
4.1.1 CONSTRAINING FROBENIUS DISTANCE
When using the Frobenius distance, Equation I can be rewritten as
FF W, 7) arg min W
-
WW
s.t. I ( P- S O.
To simplify the problem, instead work translated version of the same parameter space where (0)) the origin. Setting f W - w0 and T W W())) the problem becomes
(T 7) arg min T 1 T
| < 0,
which is the Euclidean projection onto the 8² ball with radius and has the known closed form solution
n2 (T,7)=
7.
max 11 |
Expanding the definition of T and translating back into the correct parameter space yields the Frobenius distance projection function.
TF (W (0) w.,) = (0()) max (I, W| w
1
(W - (w()))
(3)
4.1.2 CONSTRAINING MARS DISTANCE
The constraint on the MARS distance can be equivalently expressed as a collection constraints on the e distance of each row in the weight matrix from the corresponding row in the pre-trained weight matrix. That is,
I w0 |1 57 +
||
( 59 Vi,
5

---

Published as a conference paper at ICLR 2021
where Wi is the ith row of W. One can then make use of the same translation trick used to derive the Frobenius distance projection function to change the el distance constraints to el norm constraints,
(C,, 7) arg min |1 - -
|1 7 - 0,
(0) where i' = प. ii The problem in Equation is Euclidean projection onto the e ball with radius 7, for which there is no known closed form solution. There exist algorithms to find the e projection in time linearly proportional to the dimensionality of the vector (Duchi al., 2008), but they are not amenable to implementation on graphics processing units due to the sequential nature of the computations involved. Instead, we apply projection that minimises the ll distance between the original point and its projection, subject to the projected point lying inside the !! ball with radius
m (f,,7)
max(1, 1
This projection, while not providing the closest feasible point measured Euclidean distance, still provides point that satisfies the constraints, but is trivial to implement efficiently. Finally, the projection function for the entire weight matrix given by applying row-wise, and translating back into the correct parameter space,
(0) wi
.7) 1
To (W(0),,,)))
7)>)=
(T (w wn .7) Wn
where W contains
4.2 PENALTY METHODS
One popular approach in the literature encourage model to not move too far from set of initial weights is to augment the loss function with penalty term. In our case, this would involve taking the standard objective for the problem hand (e.g., cross entropy the hinge loss), and adding penalty terms corresponding to each layer,
min Σ f(E)) + Σ ,,W,, - 11.. WiLL
(5)
1=1
where Xy are hyperparameters used balance the regularisation terms with the main loss function. Due to the subdifferentiability of the two norms considered in this paper, instantiations of Equation D can be trained via automatic differentiation and a variant of the stochastic subgradient method. For the Frobenius norm, we actually penalise the squared Frobenius norm, which recovers the EP-SP approach of et al. (2018). We refer to the instantiation that penalises the MARS distance as MARS-SP.
5 EXPERIMENTS
This section provides an empirical investigation into the predictive performance of the proposed methods relative to existing approaches for regularising fine-tuning, and also conducts experiments to demonstrate which properties of the novel algorithms are responsible for the change in performance. Two network architectures are used: ResNet-101 (He et al.) (016)), which representative of typical large neural network, and EfficientNetB0 (Tan Le 2019), leading architecture intended for use on mobile devices. Both networks are pre-trained on the 2012 ImageNet Large Scale Visual Recognition Challenge dataset (Russakovsky et al.) 2015). The Adam optimiser is used for all experiments (Kingma and Ba, 2015). Information regarding the datasets and hyperparameter optimisation procedure can be found in the supplemental material.
5.1 PREDICTIVE PERFORMANCE
The first of experiments performance comparison of the proposed methods and existing regularisation approaches for fine-tuning. The baselines considered are standard fine-tuning with no specialised regularisation, SP-SP eL 2018). DELTA (Li al.) 2019). and label smoothing (I.S)

---

Published as a conference paper at ICLR 2021
Results obtained with different regularisation approaches when fine-tuning ResNet-101 (top) and EfficientNetBO (bottom) models pre-trained on the ILSVRR-2002 subset of ImageNet. We report the mean + std. dev. of accuracy measured across five different random seeds.
Table 1:
Regularisation
Butterlly
Flowers
Publigg
Caltech
Avg.
None
51.81=0.87
70,02=0.166 16
76.68 1,068+1.077
84, 14.1900..4
75. 0.+
66. 15:00.55
75.32=0061
6.71
DELTA Tietal
2019
1.26
77.91+0.24
86.5740.27
82.2342.488
78.88+0.27
LS (Mallere el 2019
60.86600.65
76.52.0.23
10.49
90.50.10.31
84.38.0.44
68.08.10.81
10.0410.10
3.14
SP
60.16=1.32
76.5620.19
83.1110.27
86.2344.41
83.79=3.69
69:87:0:24
79.9420.17
MARS-SP
58,52=1222 123
69,54=0.36
75,90+
84.2240.:55
79.17=0.73
69,53=00.93
79.19=0.33
5,57
e PGM MARS-POM
69.61-0.15
77.9710.13
86.91100.66
90.47.10.33
84.14.10.81
70.97:10.71
78.4510.13
72.04=0.70
09.50=0.36
87.42:1041
89.23:11.36
88.75:0.288
70.23::0.53
79.11=0019
1.86
Regularisation
Aireraft
Butterfly
Flowers
Pots
PubFig
DTD
Caltech
Avg. Rank
None
54.58=0.65
69,43=00:77
77.43+0.14
84.8740.119
75.51+0083
64.98=0043
81.07:-0330
DELTA al 2019
70.61-0.18
79.61:0021
84.60+00.23
89.27 +0.22
88.62-0..54
71.37-00.35
81.53+-0.38
3.57
LS (aileret
71.5210.14
83.4210.70
84.95.0.555
77.05.1..31
65.53.10.26
83.1210.17
(-SP Liera 20x
69.26-0.26
78.88
86.61
89.79:10.8
84.04--0.98
71.12-0052
82.91+00.39
3,71
SP
66.96_0.49
17.7910.48
89.24110.00
85.33.1..59
69.41:10.49
82.10110244
PPPGM
70.87-0.33
81.81+0.224
86.95++0.17 17
89.33-+0.19
88 08.45
71.63++0.59
84, 94.30+=0.09
2.29
MARS-PGM
0.34
82.32:00.10
90.36.10.15
9..38:10.24
90.30:1.400
73.57:40.38
84.84
as formalised Müller et al. (2019). Once hyperparameters are obtained, each network architecture and regulariser combination is fine-tuned on both the training and validation folds of each dataset.
The fine-tuning process repeated five times with different random seeds to measure the robustness of each method to the composition of minibatches and initialisation of the final linear layer, which is trained from scratch. Test set accuracy is reported for the ResNet- 101 and EfficientNeBBO architectures in Table[]
Comparing the average ranks of the methods across different datasets (Demšar) 2006), the most salient trend is that the projection-basee methods exhibit a significant increase in accuracy over their penalty counterparts that use the same distance metrics. This suggests that, in the case fine-tuning, using a projection to enforce constraint on the weights throughout the training process is better regularisation strategy than adding a term to the objective function that penalises the deviations of weights from their pre-trained values. Additionally, looking further at the relative performance of the two conjection-based regularisers, we can see that the MARS distance variant is more often better choice than the Frobenius distance. This observation further supports the conclusions of the comparison of the bounds in Section 3
5.2 DISTANCE FROM INITIALISATION
To further investigate the relationship between the penalty and projection strategies for regularisation, we analyse the distances between the pre-trained and fine-tuned weights at a per-layer basis. Figure provides histograms indicating the distributions of per-layer MARS distances for the models without any regularisation, regularised with MARS-SP, trained with the MARS-PGM and also DELTA. All networks were trained on the Pets dataset with the same hyperparameters used by the models examined in Section 5.1 We observe three trends from these plots. Firstly, both the penalty and projection methods reduce the MARS distance between the pre-trained and fine-tuned parameters relative to the unregularised model. Secondly, a significant number the constraints enforced by the projection method activated-ii.c i.... many of the weight matrices lie on the boundary of the feasible set. In contrast, the penalty-based regulariser does not enforce similarly activated constraints. Finally, the results demonstrate that the DELTA method of Li et al. (2019) does not operate by implicitly regularising the same quantity, as its MARS distance histogram is longer tailed than the others.
5.3 CAPACITY CONTROI.
To demonstrate the ability of the distance-basee regularisation methods to control model capacity, we sweep through a range of hyperparameter values and plot the corresponding predictive performance. Hyperparameters were generated according to >; = =c² and 7 = eîj. where c is varied, and λÿ. 9 are the values found during the hyperparameter optimisation process. Plots of c versus the resulting accuracy on the pets dataset are given in Figure for both the ResNet101 and EfficientNetBO architec- tures. We can that the PGM methods behave as the theoretical analysis predicts: hyperparameter

---

Published as a conference paper at ICLR 2021
30
30-
10-
MARS
MARS
MARS
MARS
(a) None
(b) MARS-PGM
(c) MARS-SP
(d) DELTA
00
(e) None
(1) MARS- PARRS-PGM
(g) MARS-SP
(h) DELTA
Figure 1: Histograms MARS distances between pre-trained and pets dataset fine-tuned weights for the specified regularisation strategies. MARS-PGM, successfully constrains weight distances to be less than 7,, indicated by the dashed line. Top: ResNet101, Bottom: EfficientNe:BO.
Accuracy
Accuracy
50
DELTA 1--SP
50
DENTA
e P.roM
PPGM
MA
MARS-SP
MARS-PGM
MARS PGM
0
0
10
10
100
102
104
10-4
10-2
2
100
102
10
c
(a) ResNet101
(b) EfficientNeeBO
Figure 2: Sensitivity of the regularisation methods to the choice of hyperparameters. Measurements are taken on the pets dataset, and is factor applied to the hyperparameters found during tuning.
configurations that lead very small distances between pre-trained and fine-tuned weights result in underfitting, relaxing the hyperparameters too much Ieads to overfitting, and using the optimised hyperparameters (i.e., when achieves the best performance.
5.4 EMPIRICAL COMPARISON BOUNDS
We perform an empirical comparison of our two bounds. along with bound based on the spectral norm (Long and Sedghi 2019), to demonstrate the relative tightness. This is done by training neural networks on the MNIST dataset (LeCun et al. 1998). We use the architecture Scherr et al. (2010), which consists of a single convolutional layer with
9 filters and l 112 channels, followed by 5 x max pooling, and finally linear classifier layer. The network is trained for 15 epochs using the Adam optimiser (Kingma and Ba 2015), as training any further does not result in any performance increase. Following similar work that has performed empirical evaluation of neural network bounds, evaluate the bounds by computing the relevant norms and distances the trained network weights. 'hese quantities then used in place of the upper bounds (i.e., B and D?) used in the definition
8

---

Published as a conference paper at ICLR 2021
117
Fileiia: NARS
112
187
lea
Speczad
100
12P
107
107
Spece
Ppece
(a) -PPGM
(b) MARM-ARRPPEE
(c) Unregularised
(d) Unregularised Accuracy
Figure 3: An empirical comparison of the tightness of our bounds on the Rademacher complexity, and that of Long and Sedghi ( The (a) and (b) plots demonstrate the empirical value of the bounds as function of the regularisation parameters: the vertical axes correspond to the model complexities for each of the three measures considered, and the horizontal axes represents the regularisation strength for 22-PGM (a) and MARS-PGM (b). The plot in (c) shows how the model complexity measures change throughout the training of an unregularised model, and (d) shows the training and test set performance throughout training for the same unregularised model.
of the hypothesis classes considered by Theorems [nddd] In keeping with previous work that has performed this type of empirical comparison, we measure the distance from random initialisation, rather than fine-tuning from a pre-trained network (Bartlett et al. 2017 Neyshabur et al. (019). Figure shows how these three quantities vary for different choices of y (the same hyperparameter used for both layers) when using MARS-PGM and 22-PGM, and also how they differ through the process of training and unregularised model. We observe several trends: (i) like previous work in this area, all of the empirically evaluated Rademacher complexity bounds still too loose be useful for model selection and providing performance guarantees on the expected test set performance; (ii) the bounds based on the MARS norm are consistently tighter (by orders of magnitude) than the bounds based on the Frobenius and spectral norms; (iii) the empirical measurements of model complexity plateau even though the hyperparameters governing the worst-case capacity continue to increase- we suspect this is caused by implicit regularisation from early stopping. This implicit regularisation from early stopping also provides an explanation for why we observe only small degradation in performance in Figure when when the hyperparameters are set to very large values.
CONCLUSION
This paper investigates different regularisation methods for fine-tuni tne-ttning deep learning networks. To facilitate this, we provide two new bounds the generalisation performance of neural networks based on the distance of the final weights from their initial values. The discussion comparing these bounds suggests that the MARS distance is more appropriate metric in the parameter space of convolutional networks than Frobenius distance. Additionally, several new algorithms are presented that enable an experimental comparison between different regularisation strategies. The empirical results corroborate our theoretical investigation, demonstrating that constraining MARS distance more effective than constraining Euclidean distance. Crucially, also show that, in line with theoretical results, enforcing a hard constraint throughout the entire training process on the distances the parameters can move is far more effective than the widely used strategy of adding a penalty term to the objective function. Implementations of the methods used in this paper are available online
ACKNOWLEDGMENTS
This work was supported by the Engincering and Physical Sciences Research Council (EPSRC) Grant number EP/S000631/I; and the MOD University Defence Research Collaboration (UDRC) Signal Processing.

---

Published as a conference paper at ICLR 2021
REFERENCES
Peter I. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463 482, 2002.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neu- ral networks. Advances in Neural Information Processing Systems 30, pages 6240-6249, 2017.
James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D. Cox. Hyperopt: A Python library for model selection and hyperparameter optimization. Computational Science & Discovery, 8(1):01400) 1,00,, July 2015.
Tianshui Chen, Wenxi Wu, Yuefang Gao, Le Dong, Xiaonan Luo, and Liang Lin. Fine- Fine-grained representation learning and recognition by exploiting hierarchical semantic embedding. In Proceedings of the 26th ACM International Conference on Multimedia, pages 2023-2031, 2018.
M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Janez. Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan):1-30, 2006.
Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning to learn around common mean. In Advances in Neural Information Processing Systems 31, 2018.
Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-leat stochastic gradient descent with biased regularization. In Proceedings of the 36th International Conference on Machine Learning, 2019.
John Duchi, Shai Chalev-Shwartz Yoram Singer, and Tushar Chandra. Efficient projections onto the 1-ball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning, 2008
Chelsea Finn, Pieter Abbecl, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning, 2017.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-Independent Sample Complexity of Neural Networks. In Conference On Learning Theory, pages 297-299, July 2018.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--233 1998.
Michel Ledoux and Michel Talagrand. Probability in Banach Spaces Isoperimetry and Processes. Springer, 1991.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks. In International Conference on Learning Representations, 2019.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In Proceedings of the 35th International Conference on Machine Learning, 2018
Philip M Long and Hanie Sedghi. Size-free generalization bounds for convolutional neural networks. arXiv preprint arXiv: 1905. /2600, 2019.
10

---

Published as a conference paper at ICLR 2021
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, A. Vedaldi, Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- tine-grained visual classification of aircraft. Technical report, 2013.
Andreas Maurer. A vector-contraction inequality for rademacher complexities. In International Conference on Algorithmic Learning Theory, pages 3-17. Springer, 2016.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32, 2019.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference Learning Theory, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-Bayesian Approach to Spectrally-Normaizzed Margin Bounds for Neural Networks. In International Conference on Learning Representations, February 2018.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. To- wards understanding the role of prramttrmmtiinoioo in generalization of neural networks. In International Conference on Learning Representations, 2019.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of the 6th Indian Conference Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
Luca Oneto, Sandro Ridella, and Davide Anguita. Tikhonov, ivanov and morozov regularization for support vector machine learning. Machine Learning, 103(1):103-1366 2016.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2012.
Nicolas Pinto, Zak Stone, Todd Zickler, and David Cox. Scaling up biologically-inspree computer vision: A case study in unconstrained face recognition on facebook. In IEEE CVPR Workshops, pages 42. IEEE, 2011.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In International Conference on Learning Representations, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3): 211-252, December 2015.
Dominik Scherer, Andreas Müller, and Sven Behnke. Evaluation of pooling operations in convo- lutional architectures for object recognition. In International Conference on Artificial Neural Networks,pager 92-101. Springer, 2010.
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference Machine Learning. 2019.
Jun Yang, Rong Yan, and Alexander G Hauptmann. Adapting classifiers data with shifted distributions. In Proceedings the 7th IEEE International Conference on Data Mining Workshops, pages 69-76. IEEE, 2007.
11


Reviews:
[{"review_id": "TdQmSVH7w_w", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies regularization for neural network fine-tuning, motivated by limiting deviation of the final model from the initialization states. The provide a generalization bound that utilizes a novel Rademacher complexity term built on the layer weights and their deviation from the initial weights. This bound relates particularly to fine-tuning, since a part of the bound can be fixed to the pre-trained weights, providing an alternative regularization objective specific to fine-tuning. Using this objective, the authors provide several fine-tuning benchmark experiments and demonstrate competitive performance.\n\nStrengths of the paper:\n- Well written, easy to follows.\n- Motivation for the algorithm stems directly from the analysis, as opposed to heuristic-style arguments that typically dominate the field of CV /deep learning research, especially for fine-tuning. Moreover, the generalization bounds are derived such that they lead to an optimization objective (as opposed to conventional approaches that typically have not led directly to an effective algorithm).\n- The analysis appears to be general, without any particularly strong assumptions.\n- Two different norms are considered with corresponding algorithms and experiments.\n- Extensive ablations are performed on vision tasks.\n\nWeaknesses:\n- Only tested on computer vision benchmarks. If the paper claims this approach to be a general technique then it is necessary that the methods do well on other tasks (e.g., language), otherwise the experimental claims rely too much on the convolutional inductive biases.\n- If the paper is in fact framed as a CV paper, then it is natural that a comparison be made with respect to prior (albeit heuristic) computer vision research, e.g., label-smoothing regularization, entropy regularization and so on.\n- An empirical comparison of the tightness of the bounds is warranted given the deviation of this analysis from PAC-Bayesian (Neyshabur 2018) or spectral norms (Bartlett and Long).", "extra_scores": {}, "extra_text": {}}, {"review_id": "OJVhTLAicgv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this manuscript the authors derive a bound on the rademacher complexity of neural network models which can be written as a funciton of the MARS norms of the weights in the network. This motivates the authors to put a regularization on the MARS norm of the network weights instead of the more typical L2 norm. Here the authors implement this regularization as a hard bound on the weights, which they enforce by projecting the weights back on the allowed ball. They use their regularization for transfer learning of ResNet-101 and EfficientNetB0 from ImageNet onto the set of smaller image classification tasks. On these tasks, the projection methods and to a smaller degree the MARS based methods generalize better. \n\nOverall I vote for acceptance. This is an interesting contribution to the literature, providing both a theoretical insight and an experimental test that this theoretical insight is relevant for applications. However there is a certain disconnect between the theory and the experimental observations. Performance  benefits more from the projection methods than from the switch of norm although the switch of norm has a much stronger theoretical justification.\n\nPros: \n1) Well structured paper with interesting results\n2) Theoretical results are well justified to be more helpful than existing bounds.\n3) There is an empirical test that the switch in bound is helpful for practice.\n4) Overall the generalization is actually improved.\n\nCons:\n1) Empirically the less justified change has a larger impact, indicating that there might be another more important theoretical insight\n2) The hyperparameter setting procedure remains opaque. The authors always talk about gamma_i/ lambda_i parameters changing the strength of regularization per layer, but only test how scaling all regularizations up or down affects performance. A description how the values were chosen is really necessary I think and some analysis to convince us that the worse performance of the regularization is not caused by a bad hyperparameter choice would definitely be a plus.\n3) I think there is a bit of a missed opportunity here for the scaling over layers as the bound suggests an unequal weighting of the layer norms. I think directly regularization of the bound which would allow layers to compensate for each other or giving each layer an equal budget in terms of raising the bound would be interesting variants here.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hjv1qXuZX8s", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work proposes a Rademacher type bound for the fine-tuned models based on the distance between the fine-tuned weights and the pre-trained weights. Since the distance term shows up in the upper bound on the generalization gap, the authors further propose to adopt it as the regularization term to boost the generalization performance of the model during the fine-tuning process. Some experiments are also done to show the effectiveness of the proposed regularization.\n\nI am seriously concerned about the correctness of the Rademacher-type bound the authors have proposed. The bound does not seem correct to me.\n\nThe flaw comes from the function class F_* defined in section 3 of the draft. The function class F_*, by definition, depends on the pre-trained weights W_j^0. However, W_j^0 is not fixed, it is random! This is because W_j^0 depends on the data (W_j^0 is pre-trained using the data), which by the assumption of the draft, is random. As a consequence you cannot assume W_j^0 as fixed. The randomness of the hypothesis class F_* destroys almost all the derivations the authors are currently using in their proof. \n\nAnother minor bug is the second term in the bound for theorem 1 seems to have some subscript issues. To me the product term related to B_j^\\infty should go from i=1 to j instead of from j=1 to L. I may have missed something in this point but could the authors double check if the subscript of the B_j^\\infty is correct? In particular the derivation from the second inequality to the third on page 13 of the appendix. \n\nThe second issue is easy to fix. However the first issue seems like a fundamental flaw. I do not have a good way to handle it for now. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "GUlmylQMl3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes new regularization methods for fine-tuning deep neural networks based on matrix $\\infty$-norm distance. The authors claim that their choice of matrix $\\infty$-norm distance is more suitable than commonly used Frobenius norm distance (a.k.a., Euclidean distance) when measuring the distance in the parameter space of convolutional networks by a comparison of two generalization bounds. Moreover, the authors empirically show that enforcing a hard constraint on the weights by projected methods throughout the training process is more effective in regularizing neural networks than widely used strategy of adding a penalty term to the objective function.\n\nOverall, the paper is well written and has a nice logical flow. The problem of finding a proper distance metric for fine-tuning is interesting, though I have a few concerns outlined below regarding their theoretical analysis of using generalization bound to guide the choice of distance metric, especially the proof of the theorems. \n\nConcerns:\n1. The authors try to modify the peeling technique of prior work to prove two generalization bounds, i.e., Theorem 1 and Theorem 2. A key step in proving the two theorems is to prove Lemma 2 given in the Appendix. However, from the proof of Lemma 2, if I understand correctly, the second equality and the fourth equality seem to interchange the order of sum and supremum freely, i.e., $\\sum_{j=1}^n v_j \\sup_{W_{1:k}} …=\\sup_{W_{1:k}} \\sum_{j=1}^n v_j…$, which of course does not hold in general. It should be stated clearly on why the two equalities hold here.\n\n2. The authors provide two generalization bounds for fine-tuning. The two bounds are almost the same except for the norm used. The authors then claim that a comparison of the two bounds suggests that matrix $\\infty$-norm is more effective than Frobenius norm when measuring distance in weight space of neural networks just because matrix $\\infty$-norm itself is independent of the feature map size. This is misleading in the sense that matrix $\\infty$-norm and Frobenius norm are actually equivalent, i.e., for an arbitrary matrix, its matrix $\\infty$-norm is not strictly smaller than its Frobenius norm and vice versa, and thus the two bounds are also equivalent and cannot be used to tell which norm is better. Therefore, I do not think that their choice of matrix $\\infty$-norm  as the distance metric can be theoretically justified by comparing the two generalization bounds as in the paper, despite that empirical results show that their method performs well in practice.\n\n3. In section 5.3, the authors hope to demonstrate the ability of the distance-based regularization methods to control model capacity by sweeping through a range of hyperparameter values and plotting the corresponding predictive performance. The authors claim that Figure 2 shows that the PGM methods behave as the theoretical analysis predicts and the penalty-based approaches are not able to influence the model capacity as much as the constraint based approaches. This statement is inaccurate in several ways. First, the symbol $\\lambda_j$ in the third line is confusing. It seems to represent the hyperparameter for both the constraint based methods and penalty methods. However, $\\lambda_j$ first appears in equation (5) where it represents the hyperparameter for penalty methods. Second, from Figure 2, as $c$ becomes larger and larger, there is only a very small drop of accuracy for the PGM methods. So, it does not lead to overfitting, and the PGM methods do not behave exactly as the generalization bound predicts. Third, small $c$ for PGM methods corresponds to large $c$ for penalty methods by the equivalence of constraint methods and penalty methods. Therefore, Figure 2 shows that the penalty-based approaches actually have the same influence on the model capacity as the constraint based methods.\n\nMinor comments:\n- From the proof of Theorem 2, the term $\\sqrt{c}$ in the bound should be $c$. Therefore, the bounds in Theorem 1 and Theorem 2 exhibit the same dependence on the number of classes.\n\n- I am a little confused by the sentence “In the case of the final classification layer, $W_L^0$ can be randomly initialized.” in 7th line of Section 3. Do you mean that $W_j^0$s ($j<L$) are pre-defined and fixed, but $W_L^0$ is random? However, when proving the upper bound for empirical Rademacher complexity, especially the last step where the rightmost term evaluates to zero, it seems that you assume that all these matrices $W_j^0$ ($1\\leq j\\leq L$) are fixed. It would be better if this can be clarified.\n\n- In section 4 and Appendix E, to support the claim that projection based methods are better than penalty based methods, the authors state that penalty methods have weaker assurance on whether a constraint is being forced. However, Figure 1 shows that for ResNet101 model penalty-based method is actually more effective in enforcing the constraints in the sense that not only it successfully constraints weight distance to be less than $\\gamma_j$, but also the number of weights which have small distance is larger. Therefore, more evidence might be needed to support their claim.\n\nSome typos:\n\n(1) In line 6 of Page 2, best way restrict -> best way to restrict\n\n(2) In the last line of Page 5, change the $l^1$ distance-> change the MARS distance\n\n(3) In the third line of the proof of Lemma 2, $\\varphi_j$-> $\\varphi$\n\n(4) In the third formula of the proof of Theorem 2, $sqrt{2}$-> $\\sqrt{2}$", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes constraints to be applied to the weights of a deep neural model during training. These constraints, motivated by an analysis of Rademacher complexity, are compared with other constraints and penalty approaches in transfer learning. The authors were able to build on the reviewers feedback to improve their paper on several points during the discussion phase, leading to a consensus for acceptance among reviewers. They also agreed to conduct experiments targeting stronger experimental results to compare all methods in the situation where they provide state-of-the-art results. This will make a useful contribution to the ICRL audience, and I recommend acceptance.

Decision: Accept (Poster)