Interpretations of Domain Adaptations via Layer Variational Analysis
Authors: Huan-Hsin Tseng
Hsin-Yi Lin
Kuo-Hsuan Hung
Yu Tsao
Venue: ICLR 2023 poster (2023)
Keywords: deep learning theory
domain adaptation
transfer learning
variational analysis
knowledge transfer

Abstract: Transfer learning is known to perform efficiently in many applications empirically, yet limited literature reports the mechanism behind the scene. This study establishes both formal derivations and heuristic analysis to formulate the theory of transfer learning in deep learning. Our framework utilizing layer variational analysis proves that the success of transfer learning can be guaranteed with corresponding data conditions. Moreover, our theoretical calculation yields intuitive interpretations towards the knowledge transfer process. Subsequently, an alternative method for network-based transfer learning is derived. The method shows an increase in efficiency and accuracy for domain adaptation. It is particularly advantageous when new domain data is sufficiently sparse during adaptation. Numerical experiments over diverse tasks validated our theory and verified that our analytic expression achieved better performance in domain adaptation than the gradient descent method.

Published as a conference paper at ICLR 2023
INTERPRETATIONS OF DOMAIN ADAPTATIONS VIA LAYER VARIATIONAL ANALYSIS
Huan-Hsin Tseng, Hsin-Yi Lin, Kuo-Hsuan Hung, Yu Tsao Research Center for Information Technology Innovation, Academia Sinica, Taiwan (htseng, hylin, khhung, tsaa) seiti.sinn sinica... edu.ts tw
ABSTRACT
Transfer learning is known to perform efficiently in many applications empirically, yet limited literature reports the mechanism behind the scene. This study establishes both formal derivations and heuristic analysis to formulate the theory of transfer learning in deep learning. Our framework utilizing layer variational analysis proves that the success of transfer learning can be guaranteed with corresponding data conditions. Moreover, our theoretical calculation yields intuitive interpretations towards the knowledge transfer process. Subsequently, an alternative method for natwork-based transfer learning is derived. The method shows an increase efficiency and accuracy for domain adaptation. is particularly advantageous when new domain data is sufficiently sparse during adaptation. Numerical experiments over diverse tasks validated our theory and verified that our analytic expression achieved better performance domain adaptation than the gradient descent method.
INTRODUCTION
Transfer learning is technique applied to neural networks admitting rapid learning from one (source) domain to another domain, and it mimics human brains in terms of cognitive understanding. The concept of transfer learning has been considerably advantageous, and different frameworks have been formulated for various applications in different fields. For instance, it has been widely applied aidely in image classification (Quattoni et al.) 2008 Zhu al., 2 Hussain et al., 2018). object detection (Shi et al. 2016), and natural language processing (NLP) (Houlsby et al. 2019 Raffel et al.) (019). In addition to applications computer vision and NL.P, transferability is fundamentally and directly related to domain adaptation and adversarial learning (Luo et al. 2017 Cao al., 2018 Ganin et al., 2016). Another majority field adopting transfer learning is domain adaptation, which investigates transition problems between
close domains (Kouw & Loog 2018). A typical understanding is that transfer learning deals with a general problem where two domains can be rather distinct, allowing sample space and label space to differ. While domain adaptation is considered subfield in transfer learning where the sample/label spaces are fixed with only the probability distributions allowed to be varied. Several studies have investigated the transferability of network features or representations through experimentation, and discussed their relation to network structures YYsiinkki et al. 2014).
features, and parameter spaces (Neyshabur al. 2020 Gonthier al. 2020).
In general, all methods that improve the predictive performance of a target domain, using knowledge of source domain, are considered under the transfer learning category (Weiss et al. 2016 Tan et al. 2018). This work particularly focuses on network- based transfer-learning, which reters to a specific framework that reuses a pretrained network. This approach often referred to as finetuning, which has been shown powerful and widely applied with deep-learning models (Ge & Yu 2017] Guo et al.) 2019). Even with abundant successes in applications, the understanding of the network-basse transfer learning mechanism from a theoretical framework remains limited.
This paper presents a theoretical framework set out from aspects of functional variation analy- sis (Gelfand et al. 2000)) to rigorously discuss the mechanism of transfer learning. Under the framework, error estimates can be computed to support the foundation of transfer-learning, and an interpretation is provided to connect the theoretical derivations with transfer learning mechanism.
Our contributions can be summarized as follows: we formalize transfer learning in a rigorous setting and variational analysis to build up theoretical foundation for the empirical technique. A theorem is

---

Published as a conference paper at ICLR 2023
proved through layer variational analysis that under certain data similarity conditions, transferred net is guaranteed to transfer knowledge successfully. Moreover, a comprehensible interpretation is presented to understand the finetuning mechanism. Subsequently, the interpretation reveals that the reduction of non-trivial transfer learning loss can be represented by a linear regression form, which naturally leads to analytical (globally) optimal solutions. Experiments in domain adaptation were conducted and showed promising results, which validated our theoretical framework.
2 RELATED WORK
Transfer Learning Transfer learning based on deep learning has achieved great success, and finetuning pretrained model has been considered an influential approach for knowledge transfer (Guo) et al. 2019; Girshick et al. 2014|Long et al. (2155) Due to its importance, many studies attempt to understand the transferability from wide range perspectives, such as dependency on the base model (Kornblith et al.) (019), and the relation with features and parameters (Pan & Yang) 2009). Empirically, similarity could be a factor for successful knowledge transfer. The similarity between the pretrained and finetuned models was discussed in (Xuhong et al. 2018). Our work begins by setting up mathematical framework with data similarity defined, which leads to a new formulation with intuitive interpretations for transferred models.
Domain Adaptation Typically domain adaptations consider data distributions and deviations to search for mappings aligning domains. Early literature suggested that assuming data drawn from certain probabilities (Blitzer et al.] (2006) can be used to model and compensate for the domain mismatch. Some studies then looked for theoretical arguments when a successful adaptation can be yielded. Particularly, (Redko et al., 2020) estimated learning bounds on various statistical conditions and yielded theoretical guarantees under classical learning settings. Inspired by deep learning, feature extractions (Wang & Deng 2018) and efficient finetuning of networks (Patricia & Caputo [014]] Donahue et al. 2014 Li et al. 2019) become popular techniques for achieving domain adaptation investigated further to see what was being transferred and learned in (Wei et al. 2018). This will be close to our investigation on weights optimizations.
tasks. The finetuning of networks
3 MATHEMATICAL FRAMEWORK AND ERROR ESTIMATES
3.1 FRAMEWORK FOR NETWORK-BASED TRANSFER LEARNING
To formally address error estimates, we first formulate the framework and notations for consistency.
Definition 3.1 (Neural networks). An n-layer neural network R - R.. is a function of the form:
fn fn-1 o...
(1)
for each j 1.....,,,, f =1,,.. (2) = WW and an activation function o; R&:: > RR; with W, E L (R's ..B)), R and L(K, V) the collection of all linear maps between two linear spaces K
- Re is a layer composed by an affine function
The concept of transfer learning is formulated as follows:
Definition 3.2 (K-layer fixed transfer learning). Given one (large and diverse) dataset D
((rs, Me) X x })} and a corresponding n-layer network f In fa-1 o... o fi : X + y
trained by D under loss L(f), ), the k-layer fixed transfer learning finds new network X + y of the form,
g In In-1 0... gh+11
o....off f1.
(2)
fexed
under loss L(g) when new and similar dataset D = {G5,, W/) E X y} given. The first layers of f remain fixed and gj = og Aj are new layers with affine functions Åş to be adjusted (k j j < n).
The net f trained by the original data D is called the pretrained net and g trained by new data D called the transferred net or the finetuned net. Transfer learning has the empirical implication of
2

---

Published as a conference paper at ICLR 2023
"transferring" knowledge from domain to another by fixing the first few pretrained layers. One main goal of this study is to characterize the transferring process under layer finetuning.
We utilize the loss function to define the concept of "well-trained" net, in order to comprehend the transfer learning mechanism. The losses for the pretrained net and the transferred net are computed general label space y with norm Iby:
N
N
1
C(
-iB.. ((g) = E llf(x.)) = N
119(0.) T.11.
N
Definition 3.3 (Well-trained net). Given dataset D = { Mii E X }), network h is called well-trained under data within error Etmined > 0 if
N
C(h) Σ [1((z;) - 1:113 S Emiinedd
(4)
The definition then implies that well-pretrained net within error ppetrrined is to satisfy
((J) S pretrained
In certain domain adaptation applications, discussions are in terms of probability distributions (Redko al., 2020). Our framework is compatible with the usual probabilistic setting under the independent and identically distributed (i.i..)) consideration, frequently applied in practice. In this case, Eq. is equivalent to
[()- BO ((f(x),)).
L(g) = =B (o(())
where l is a norm (error) function.
One intuition for successful knowledge transfer that the previous knowledge base and the new domain share some common (or similar) features. The resemblance of two datasets D, D under transfer learning in this work is defined by:
Definition 3.4 (Data deviation). The data deviation Edata 2 of two datasets D and D defined as,
max min [[ Us) (155 [olx,))llx>0} S
Edatas
(i < (D) j S (DI)
(7)
Note that the first minimization in Eq. is recognized as performing sample alignment in the context of domain adaptation. In fact, for each S [D] there exists an index
js = arg min (Z,, T.) (Iz 1y) 1))xxx
(8)
such ji then corresponds to the closet sample in D to i. This step is usually referred sample alignment. Domain alignment is then to perform sample alignment over all samples of given domain. Consequently, this definition aligns two datasets first and then considers the largest distance among all aligned pairs. Without loss of generality, in the rest discussions the sample index is rearranged by Eq. (8] with j; renamed as for convenience.
3.2 TRANSFER LOsS BOUND VIA LAYER VARIATIONS
Neural networks of the form Eq. (T) with Lipschitz activation functions are Lipschitz continuous. We utilize Lipschitz constants control the network propagation and estimate the error of the last r-layer finetuned nets. The Lipschitz constant network h is denoted by Cn, in this paper. Let pretrained net f be composed of two parts:
f (fr ....o fn-r11) (f....... fi) F Fr
(9)
where FØ-r- fn-r o fi are layers be fixed and F fn o fn fn-rt1 the last r layers to be finetuned (1 < n). With only two essential definitions above, our main theorem addressing the validity of the transfer learning technique be derived:
3

---

Published as a conference paper at ICLR 2023
Theorem 3.5 (Finetuned loss bounded by pretrain loss and data), Let D. DD be two datasets with deviation ddtta and f = F Fn-r be a well pretrained network under D, then a net finetuning the last r layers of f with the form == (F + "F" o Fn-r has bounded loss on D,
((g)
Σ ll( Vell3 - prerraieed + C2 data:
(10)
where 5F is a functioon with C&F I datar C₁ C2 are two constants depending on CF.-.. CF and Cā = max; |
Sketch Proof. By (F + 5F) Fa-r and notations in Eq.
8 we have
9 (ã.) (F + 8FF Fx-r (*+) f(z,) + U1
(1I)
with 01 = 8F F Fn.r((Ts)
(T:) fā.. - f( We compute,
N
1 Σ [f(x;) T. + 1a113 3 ( + E + (112113).
N lg 31 N
(12)
with Eq. and the triangle inequality applied in the last inequality. Further, we estimate,
ee |13
2 (1 - s 2 Cr... C Cz dmma)
(13)
With Eq. 12 and condition C&F S dataa the result Eq. 10) can be yielded. For the complete proof.
Appendix 7.1
Theorem 3.5 depicts that the finetuned loss is bounded by the pretrained loss and the data difference in Eq. (TO). linking the relation of network functions funnttinn and data into one. This then explains our intuition why pele-pretrained net beforehand is necessary, and transfer learning is anticipated to be viable on datasets with certain similarities. Thus, with minimal conditions and definitions, Theorem 335 suceinctly addresses when transfer learning guarantees well-finetuned model under new dataset. In fact, the generalization error under finetuning is also bounded (Theorem) 7.1 Appendix).
Remark 3.6. For two very different datasets, one has Edtt O. Eq. (T)) then indicates such transfer learning process is generally more difficult as the upper bound becomes large, even with a small pretraining error pperrriiedd Typically in domain adaptation tasks, dana is not expected to be large.
Remark 3.7. Eq. (10) is regarded as an extension of Eq. (5) whenever + D. The two equations coincide if Eduta O where two domains D. D perfectly align.
4 INTERPRETATIONS & CHARACTERIZATTONS OF NETWORK-BASBD TRANSFER LEARNING
Inspired by the mathematical framework of Theorem 3.5 an intuitive interpretation of network-based transfer learning can be derived, which turn leads to a novel formulation for neural network finetuning. The derivation begins with the Layer Variational Analysis (LVA), where the effect of data variation and transmission can be addressed.
4.1 LAYER VARIATIONAL ANALYSIS (LVA)
To clarify the effect of one layer variation (r in Theorem 3.5). denote the latent variables 2q.
2 and the last finetuned layer IN from Eq. 2 by,
F (dra), z = (G.). In. = Jn + Ö5n
(14)
The target prediction error be related to the pretrain error via the following expansion.
(l)( Filly -
(z,)
(J,(E,) fn)))) (s(x)) yi) (v+ -%))l (15)
with six self-canceled-outt terms added for auxiliary purposes. Note the identities g(x.) gn (%.).
f2(2)) = fxx) are used and the second grouping can be rewritten by,
(1,) (z)) ((.))(z)) dzz + 0zf H()x)(z)) dz, + O(02))
(16)
4

---

Published as a conference paper at ICLR 2023
where 82, := (Z, - 24) corresponds the latent feature shift; (z)))(z) and (z) the Jacobian and the Hessian matrix of fn at 2, respectively. In principle, non-linear activation functions introduce high order dz; terms to Eq. (16), where by considering regression type problems (assuming fa purely an affine function) the finetune loss Eq. 155 is allowed to be simplified as,
N
[(g) = N
1
Σ (5fx(2)) 9:11
(17)
with 68: := yi - ys and the vector E y defined as
0: öz [(n)(=z)) 82, + (yi f(x)))
(18)
which is referred to transferal residue. Via Transmission of Layer Variations (Appendix 7.3 one may also write,
ōy; (x)) dz: + (ys [(x;))
(19)
4.2 LVA INTERPRETATIONS & INTUITIONS
The equivalent form Eq. 17 of the transfer loss L(g) = N E (g( )
13 under LVA renders
an intuitive interpretation for network-basse transfer learning. The resulting picture is particularly helpful for domain adaptations.
Transferal residue:
We justify the name transferal residue of Eq.
19
There are two pairs:
qi = dy; - J() dx, + (y: - f(x;)
(20)
pretrain crroi(
missmatch (A)
where the first pair (A) evaluates domain mismatch; the other measures the pretrain error. To see this, observe: (a) a perfect pretrain gives f(x;) = Y/ then (*) O, and (b) if two domains match perfectly D = D, 5x, = ōy; = 0 then (A) = O.
In such a perfect case, = 0 and there is nothing new for f to learn/adapt in D. On the contrary, if f ill-pretrained with |8z; |l. ||8y:|| large, becomes large too, indicating there is much to adapt in D. Therefore, the transferal residue characterizes the amount of new knowledge needed to be learned in the new domain, and hence the name. In short,
| + 0 (nothing to adapt),
||as|| + cc. (much to adapt)
Layer variations cancel transferal residue: Whenever the transferal residue is nonzero, the finetune net adjusts from f to react, as Eq. 177 assigns layer variation å. to neutralize q:- Minimizing the adaptation loss L(g) with 55n, we see that
no
f. = 0,
4 +
need
fn min Eq. adapt
17
adapt
In fact, the last adaptation step @ is analytically solvable 8 a linear functional with ||-Ily being To-norm. By Moore-Penrose pseudo-inverse, the minimum is achieved at
8.f =
q
(21)
with 2 = .e,,..., Fx) and y = (q......9N) .N)). short, layer variations tend to digest the knowledge discrepancy due to domain differences.
Knowledge transfer: As 4 reflects the amount of new knowledge left be digested in D, see a term (z,)(z;) automatically emerges trying to reduce the remainder in 9:. Especially, the Jacobian term appears with negative sign opposite to f(x)); is presented as self-correction (self-adaptation) to the new domain. The purpose of J (z))xx) serves to annihilate the new label
shift oy: Eq. (00) (e.g. Fig. and notice that the self-correctin not needed if D D, 8z, 0.
Since the pretrained f contains previous domain information, the term J (z;));;; indicates the old knowledge is carried over by f to dissolve the unknown data. Such quantification then
5

---

Published as a conference paper at ICLR 2023
y
tarpet y
8x
prediction
RG
gy
.....*
pretrain error f()
transferal revidue 8q
Figure 1: An illustration of transferal residue Eq. (20). In this example, a pretrained (animal classifier) with 1-hot label is adapted from D + D, where D is of dog images and D is of cat images. The self-correction term transferring the old knowledge attempts to correct predictions, Eq. 20
yields an interpretation and renders an intuitive view for network-basse transfer learning. The LVA formulation naturally admits types of domain transitions according to Eq. (20): (a) 8x; = -y; (b) 8x; 0, dy: 0, (c) 8x; 0, 8y; + 0 and (d) dæ; 0, dy; f O.
The analysis can be extended to multi-layer cases as well as Convolutional Neural Networks (CNN); only that the analytic computation soon becomes cumbersome. Therefore, only one-layer LVA finetuning is demonstrated in this work. Further discussions and calculations for LVA of two layers and CNNs can be found in Appendix 7.2. Although there is no explicit formula for multi-layer cases to obtain the optimal solution, iterative regression can be utilized recursively to obtain a sub- optimal result analytically without invoking gradient descent. The LVA formulation not only renders interesting perspectives in theory but also demonstrates successful knowledge adaptations in practice.
5 EXPERIMENTS
Three experiments in various fields were conducted to verify our theoretical formulation. The first task was simulated ID time series regression, the second was Speech Enhancement (SE) of real- world voice dataset; the third was an image deblurring task by extending our formula onto Convolutional Neural Networks (CNNs). The three tasks demonstrated the LVA had prompt adaptation ability under
new domains and confirmed the calculations Eq. 15 ~ (8). The code is available on Githut For
detailed implementations and additional results on adaptive image classifications, see Appendix 8
5.1 TIME SERIES (ID SIGNAL) REGRESSION
Goal: Train DNN predicting temporal signal DD and adapt it to another signal with noise D.
Dataset: Two signals are designed as follows, with D mimicking a noisy signal (a low-quality device)
= (t,, sin()r()))} D D= (4 0.05 (innst)))nn..))))) + 0.03 n::) }
(22)
1, 1,.... ., 2000) E [-1, equally distributed, 7(4) 0.4 f, + 1.3998, & ~ (V.55, 0.8). Ti 1). (u, is normal distribution of mean μ and variance U uniform distribution [].,
with (4 i
The pretrained model f consists of 3 fully-connected layers of 64 nodes with ReLu as activation functions. We refer to Appendix for more implementation details.
Implementation:
The adaptation results were compared between the conventional gradient descent (GD) method and LVA by Eq. (18). (2I). The transferred net from f was finetuned with GD for 12000 epochs, while LVA only requires one step to derive. After adaptation, LVA reached the lowest Lz-loss compared to GD of same finetuned layer numbers, see 2 ft. This verified the stability and validity of the proposed framework. In addition to obtaining the lowest loss, the actual signal
Result analysis:
https://gi hhtp://gggttbb..// com http:////wt./////aeeeeeeereeeeee
at onal -nnal -nn.lyaisnnngaggll ysis..iit

---

Published as a conference paper at ICLR 2023
prediction performance can also be observed improved via visualization Fig. 2 b)~(e) in the case of finetuning the last one and two layers. Notably, the finetuning loss of two layers indeed further decreased than that of one layer, which met the expectation the two-layer case had more free parameters to reduce loss. Comparing Fig. [b))). it showed that the 1-layer case LVA obtained more accurate signal predictions compared to GD's predictions. Fig. [d)(() showed the results of 2-layer finetune by GD and LVA. The 2-layer LVA formulation is derived in Appendix (72 Observing that the LVA formulation equips prompt adaptation ability, we further conduct two real-world applications.
45
40
Figure 2: (a) Datasets D and D given by Eq. (22). (b) 1-layer finetuning by GD. (c) I-layer LVA finetuning, (d) 2-layer finetuning by GD, (e) 2-layer LVA finetuning, (f) the training loss the finetuned net over epochs.
5.2 SPEECH ENHANCEMENT
To prove that the proposed method is effective real data, we conducted experiments an SE task.
Goal: Train a SE model to enhance (denoise) speech signals in a noisy environment D and adapt to another noisy environment D.
Dataset: Speech data pairs were prepared for the source domain DD (served as the training set) and target domain D (served as the adaptation set) as follows:
D [(zigk 3:)) N, udaat D ((Tsjkk B.)) No (ZijA= = yr + ej x ny: Tesk = 5t + Fy 7igy)
where Y/ denotes patet of clean speech (as a label), which is then corrupted with noise n& of type i to form noisy speech (patch) Tisk over an amplification of e, x exp (S determined by a signal-to-noise ratio (SNR,) given. N1 (resp. Ni) denotes the number of patches extracted from clean utterances in D (resp. D); N2 (resp. N₂) is the number of noise types contained in D (resp. D). An SE model f on 2> performed denoising such that (I(zk) yt- In this experiment, 8,000 utterances (corresponding to N1 = 1 112000 patches) were randomly excerpted from the Deep Noise Suppression Challenge (Reddy al., 2020) dataset; these clean utterances were further contaminated with domain noise types: ng E nhite-noise, Train, Sea, Aircraft-cabin, Airplane-takeoff], J. (thus N2 = 5), at four SNR levels: {-5, 0, 5, 10} (dB) to form the training set. These 1 12,000 noisy-clean pairs were used to train the pretrained SE model. We tested performance using different numbers of adaptation data with N₁ varying from 20 to 400 patches (Fig. 3 These adaptation data were contaminated by three target noise types: n. (Baby-ery, Bell, Siren} (thus N₂ = 3) at {-1, (dB) SNR. Note the speech contents, noise types, and SNRs were all mismatched in the training and adaptation sets.
A Bi-directional LSTM (BLSTM) (Chen al. 2015) model was used to construct the SE model f under L2-loss, consisting of one-layer BLSTM of 300 nodes and one fully- fully-connected layer for output. The transferred net g adapted from f by finetuning the last layer was performed by GD to compare with LVA in Eq. (2T). For data processing and network, details see Appendix [8.2]
Implementation:
Domain alignment: It was mentioned that the domain alignment ought to be applied prior to LVA. For real-world data, optimal transport (Villani 2209) (OT) was selected for implementation to match domain distributions. To contrast, LVA with and without OT were conducted.
Evaluation metric: SE performances are usually measured by evaluation metrics, L2-error (MSE), the perceptual evaluation of speech quality (PESQ) (Rix et al. 2001) and short-time objective intelligibiliiy (STOI) (Taal et al.] (001)) Each metric evaluates different aspects of speech: PESQ [-0.5, 4.5] evaluates speech quality; STOI E [0, 1] estimates speech intelliigbiility.
A patch is defined as temporal segment extracted from utterance. We used 128 magnitude spectral vectors form patch in practice.
7

---

Published as a conference paper at ICLR 2023
PESQ
STOI
LOSS
2.2
0.85
0.025
2.0
0.020
0.80
0.015
0.010
0.75
0.005
20
50
100
200
300
400
20
60
100
200
300
400
20
60
100
200
300
400
target samples
target samples
target samples
Figure 3: Performance of GD vs. LVA by different patch amount N1 of 'D for adaptation.
Result analysis: We excerpted another 100 utterances, contaminated with the target noise types: T4 E (Baby-cry, Bell, Siren} at 1-1,1} (dB) SNR levels, form the test set. Note that the speech contents were mismatched in the adaptation and test sets. Fig. showed the domain adaptation results from environment D D to compare GD with LVA. On this test set, the PESQ and STOI of the original noisy speech (without SE) were 1.704 and 0.808, respectively. The results in Fig. 3 showed consistent out-performance of LVA over GD under a different number of adaptation data in D (N₁ as the horizontal axis). Especially, the Lz-loss of LVA was notably less than that of GD, confirming that LVA indeed derived the globally optimal weights of transferred net. Meanwhile, it was observed that LVA significantly outperformed GD when the amount of target samples in D was considerably less. It was also noticed that the LVA equipped OT alignment (LVA-OT) achieves similar performance to LVA. An example of the enhanced spectrograms by different SE models: the Pretrained, GD, LVA, and LVA-OT are shown in Fig. with clean and unprocessed noisy spectrograms for comparison. Fig. 4 shows LVA recovers the speech components with clear structures than Pretrained and GD (see green rectangles). More extensive result analyses are provided in Appendix 8.2
innesed
Figure 4: Spectrograms of utterance (Baby-ery noise SNR -1 dB; N 400 for adaptation).
5.3 SUPER RESOLUTION FOR IMAGE DEBLURRING
Goal: Train a Super-Resolution Convolutional Neural Network (SRCNN, Fig. 5 to deblur images of domain 2 and adapt to another more blurred domain D.
Dong et al. 2016
low-resolution
finetuning CNN f.
Image
high ceserraattiion Image
feature extraction
non linear mapping
reconstruction
Figure 5: [SRCNN for image deblurring] Pretrain model: a sequence of CNN layers f In o.... .20 fi. Then last CNN layer fn is finetuned to adapt to more blurred images in D.
8

---

Published as a conference paper at ICLR 2023
Dataset: 2000 resoltesou images were randomly selected labels from the CUFED dataset (Wang et al. 2016) to train SRCNN. The source domain (x. y) E and the target do- main (i, 37 E D were set as follows,
{x3 blurred patches), Ee {x4, x6 blurred patches), y 5 (High- Res patches}
Fig- shows image pairs for adaptation. Another image dataset SET14 (Zeyde et al. 2010) was used to test adapted models. More image preparation details see Appendix [3.3
y: original
2: bicubic x3
2: bicubic x4
T: bicubic x6
Figure 6: Sample images (CUFED) of 'D and D.
Implementation: This experiment demonstrated that image related tasks can also be implemented by extending our LVA, Eq. (17), (8),, from fully-connected layers CNNs. Although the analytic formula Eq. 21 is only valid on a fully-connected layer of regression type, key observation admitting such extension is that CNN kernels can locally be regarded as a fully-connected layer, in that every receptive field is exactly covered by the kernels of a CNN. By proper arrangement, CNN kernel = (Cijy) can folded as fully-connected layer 2D matrix weight W (Wrm). As such, the LVA can be extended onto CNNs as finetune layers; detailed implementations see Appendix. In this experiment, an SRCNN model f (Fig. 5 was trained on source domain to perform deblur f(x) = y, and subsequently used GD and LVA to adapt f + on such that g y-
Result analysis: Fig. 7 showed the results of adapted models by GD and LVA. In Fig. 7 column (b) were more blurred inputs ž of D: column (c)(d)(e) were deblurred images be compared with the ground truths in column (a). Column (c)d) compared the two methods adapted with the same number of target domain samples (NV 256 patches) and LVA reached the highest PSNR scores. While more target domain samples were exclusively included for GD to N 16384 in column (e), the corresponding PSNRs were still not compatible to LVA adaptation. There were more extensive experiments conducted to contrast the adaptation outcome of GD and LVA, see Appendix 8.3
(a) ground truth
(b) bicubic x6
(c) LVA. (N=256)
GD (N-256)
GD (N=16384)
PSNR 36.58
PSNK: 19.78
PSNK: 25.45
37,08
PSNR
Figure 7: Image deblurring results of adapted SRCNNs testing data SET14.
CONCLUSIONS
This study aims to provide a theoretical framework for interpreting the mechanism under domain adaptation. We derived firm basis to ensure that a finetuned net can adapted to a new dataset. The objective of this study was achieved to address "why transfer learning might work?" and unravel the underlying process. Based on the LVA, novel formulation for finetuned nets was introduced and yielded meaningful interpretations of transfer learning. The formalism was further validated by domain adaptation applications in Speech Enhancement and Super Resolution. Various tasks were observed to reach the lowest Losloss under LVA, where the LVA is the theoretical limit for the gradient descent to approach. The LVA interpretations and successful experiments give this study an inspiring perspective view transfer learning.

---

Published as a conference paper at ICLR 2023
REFERENCES
John Blitzer, Ryan McDonald, and Fernando Pereira. Domain adaptation with structural correspon- dence learning. In Proceedings of the 2006 conference on empirical methods natural language processing, pp. 120-128, 2006.
Zhangjic Cao, Mingsheng Long, Jianmin Wang. and Michael I Jordan. Partial transfer learning with selective adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2724-2732, 2018.
Zhuo Chen, Shinji Watanabe, Hakan Erdogan, and John R Hershey. Speech enhancement and recognition using multi-task learning of long short-term memory recurrent neural networks. In Sixteenth Annual Conference of the International Speech Communication Association, 2015.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647-655. PMLR, 2014.
Chao Dong. Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2): 295-307, 2016. doi: 10.100//PAMMI.20552229225
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Dovain-adversarial training of neural networks. The journal of machine learning research, 17(1):209-2033) 2016.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1086-1095, 2017.
Izrail Moiseevitch Gelfand, Richard A Silverman, et al. Calculus of variations. Courier Corporation, 2000.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 180-587, 2014.
Nicolas Gonthier, Yann Gousseau, and Saïd Ladjal. An analysis of the transfer learning convolu- tional neural networks for artistic images. arXiv preprint arXiv:201 1.0227,, 2020.
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 4805-4814, 2019.
Xiang Hao, Xiangdong Su, Radu Horaud, and Xiaofei Li. Fullsubnet: A full-band and sub-band fusion model for real-time single-channel speech enhancement. In ICASSP 2021-2021 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 6633-6637. IEEE, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Mahbub Hussain, Jordan J Bird, and Diego R Faria. A study cnn transfer learning for image classification. In UK Workshop on computational Intelligence, pp. 191-202 Springer, 2018.
Simon Korbblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2661-2671, 2019.
Wouter M Kouw and Marco Loog. An introduction to domain adaptation and transfer learning. arXiv preprint arXiv: 1806, 2018.
10

---

Published as a conference paper at ICLR 2023
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Zeyu Chen, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. arXiv 2019.
preprint arXiv:1901.0922)
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference machine learning, pp. 97-105. PMLR, 2015.
Xugang Lu, Yu Tsao, Shigeki Matsuda, and Chiori Hori. Speech enhancement based deep denoising autoencoder. In Interspeech, volume 2013, pp. 436-440, 2013.
Zelun Luo, Yuliang Zou, Judy Hoffman, and Li Fei-Fei. Label efficient learning of transferable representations across domains and tasks. arXiv preprint arXiv:1112.00123,,
2017. 2017
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning?
arXiv preprint arXiv:2008. /667, 2020.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-135, 2009.
Novi Patricia and Barbara Caputo.
Learning learn, from transfer learning domain adaptation: unifying perspective. Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 1442-1449, 2014.
Ariadna Quattoni, Michael Collins, and Trevor Darrell. Transfer learning for image classification with sparse prototype representations. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2008.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv 10683, 2019.
Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The in- terspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. arXiv preprint arXiv:2005. 13981, 2020.
levgen Redko, Emilic Morvant, Amaury Habrard, Mare Sebban, and Younès Bennani. sur- vey on domain adaptation theory: learning bounds and theoretical guarantees. arXiv preprint DxXiv:2004.11829 2020.
A. W. Rix, J. G. Becrends, M. P. Hollier, and A. P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.0ICH37221), volume 2, pp. 749-752 vol.2, 2001. doi: 10. 10.109//CAASPP2200.94900
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Computer Vision-ECCV 2010: I 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pp. 213-226. Springer, 2010.
Kevin Scaman and Aladin Virmaux. Lipschitz regularity of deep neural networks: analysis and efficient estimation. arXiv preprint 1rXiv:1805.10965, 2018
Hoo-Chang Shin, Holger R Roth, Mingchen Gao, Le Lu, Ziyue Xu, Isabella Nogues, Jianhua Yao, Daniel Mollura, and Ronald M Summers. Deep convolutional neural networks for computer-aided detection: Cnn architectures. dataset characteristics and transfer learning. IEEE transactions medical imaging, 35(5): 185-1298, 2016.
C. H. Taal, R. C. Hendriks, R. Heusdens, and Jensen. An algorithm for intelligibility prediction of lime-frequency weighted noisy speech. IEEE Transactions on Audio. Speech, and Language Processing, 19(7):2125-2336 2011. doi: 10.1109/TASI.2001 1.21 200.1 14881.
11

---

Published as a conference paper at ICLR 2023
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pp. 270-279. Springer, 2018.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey, Neurocomputing, 312: 135-153, 2018.
Yufei Wang. Zhe Lin, Xiaohui Shen, Radomir Mech, Gavin Miller, and Garrison. W. Cottrell. Event- specific image importance. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Ying Wei, Yu Zhang, Junzhou Huang, and Qiang Yang. Transfer learning via learning to transfer: Supplementary material. In 35th International Conference Machine Learning, ICML 2018, volume 11, pp. 8069, 2018.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1-40, 2016.
LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference Machine Learning, pp. 2825-2834. PMLR, 2018
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? arXiv preprint arXiv:1411.1992 2014.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pp. 711-730. Springer, 2010.
Yin Zhu, Yuqiang Chen, Zhongqi Lu, Sinno Jialin Pan, Gui Gui-Rong Xue. Yong Yu, and Qiang Yang. Heterogeneous transfer learning for image classification. Twenty-fift aaai conference on artificial intelligence, 2011.
7 APPENDIX
7.1 PROOF OF THEOREM 3.6
Consider finetuning on the last layers in Eq. (2) with k + n. Denote Fn-r = fn-r fi and the last r layers F' fn ... fn of pretrained network and finetuned net G = In 0 gn-r+1. We also write the difference - F = SF. With the notations, we rewrite
g (
Fa (3:) (F + 6F) o F... (5.) f(z,)) + v1.
(A.23)
where U1 88 o (7.) + f(x.) fxx, and ja is the corresponding index such that Eq. 7 is satisfied. Therefore, one computes,
N 19(3.) - 0.1
llf(z,) v4 e
N
3
(A.24)
< (11((1)) - - + llls. 13 + +1122113)
@pretrainedd + =dmu + 11.1113),
2
where the last term is further estimated by the triangle inequality to yield
|l 11 s 2 216 Fn., + (l - f(z,, )13) c + C2 0² -
(A.25)
22 C 0z
12

---

Published as a conference paper at ICLR 2023
with Cs == max; || Note that without any additional assumption, the first term in Eq. (A.25) is of order 1. To obtain a small error 9, one sufficient condition is to have C&F = Edata- It is interesting to point out that the analysis relates the Lipschitz constant C&F of the network perturbation with the data similarity ddata-
Theorem 7.1 (Generalization emror bound). Given a test (held-out) set Dien {xx V/s sh
N' J==1 with Drest I [D, then a well-finetuned net g Theorem 3.5 has a generalization error bound.
Niss
Liea(1) (g)
lg(
1/ S Ci E + C2
(A.26)
Niest
where Edata the data deviation between D and Diest computed by Definition 3.4 C1. C2 are two constants with Cj depending on the Lipschitz constant Cg and L(g) the finetune loss in Theorem 3.5
Proof.
Niea
Lieat (ge
(g( - 1/50001 113
Niest
Nica
3
(119(1) g( + 11 - -Vil3 + (l9(3.)) ) -7 (3-)
(A.27)
Niest
3
3
(C2 Frest + clest) +
l1((3)) T3
Nust
N Nieat
7.2 PERTURBATION APPROXIMATION DERIVATION FOR 2-LAYER CASE
Similarly, let 8z, 2, and denote the finetuned nhhh layer and (n 1)th layer by
In = Jn + ō.. 9n-1 = fn-1 + 51m-1
(A.28)
The perturbation assumption allows the approximation of
(2) ~ fz)) + J( 62;,
(A.29)
where J(fn-1)(zz) is the Jacobian matrix of fa-1 at 2;- The network assumed to deviate from the pretrained fn by an affine function,
6fn-1(2;) W 82, + b,
(A.30)
for some weight Wn-1 and bias With Eq. (9 and Eq. A..3 we obtain
b (J(fn11(2z) + Wn-1) 82
(A.31)
With Eq. A.31
g (T.) (fn 8fr) 1
(A.32)
Sa-1 (z)) + fn (JJ(n-))(zz) + Wa-1) 82, + +b) + 5ʃa f
Note that the terms involving multiplications o.n and 8 are omitted as we only focus on the first order terms. Parallel to Eq. (15) for I-layer case, we approximate
ll( We ll = (g( (7+) - fa o fn-1(2;)) + (Sa fn-1(2;) - -y:) + (ys We) lb
(A.33)
and therefore, (5,) - Tilly 1l fn-1(2;) + fn (()(f-1)(2)) + W 112--+600 6)2 zi)) + f(x;) 9i - 5yb112.
(A.34)
13

---

Published as a conference paper at ICLR 2023
We that the 2-layer fine-tuning case involves three unknown (parameters), including ffn, b, and Wn-1. Although there no explicit formula to obtain the optimal solution, iterative regression can be utilized multiple times to obtain a sub-optimal result analytically without using gradient descent. In fact, one notices that b is ignored, the optimization of 8fn and W based on the I2-loss can be solved. Once f. and Wn- be solved again by regression. Similar approximations for general multi-layers cases would still be reliable as long as the features in the latent space close each other. That is to say, the features of the source and target domain share similarities. These intuition successfully adapt knowledge in experiments for both images and speech signals. The results can be found in Section 5
solved and fixed, b can be introduced back into the formula and can
intuition-motivated motivated formulations were shown
7.3 TRANSMISSION OF LAYER VARIATIONS
We show heuristically the propagation of layer variations in neural nets can be traced via data deviations. Given datasets D, DD with the indexing after sample alignment, define
8z; = T; Zis 8y: == yi y/
(A.35)
Let the data deviation be from Def. 3.4 and Eq. (A.35) can be decomposed by orders of s,
or, ōz² dxx:) +..,
8ū; 5yg (1) +5y
(A.36)
with o (10z, lxx) o (108) ( 1b)) = To obtain simple picture, first neglect the high order terms d. (2) dyt etc. In such case, ox:") (2) and σy, (1) can be simply denoted as 8x; and dyr. respectively, without confusion. By Eq. (14). finetuning the last layer Theorem revells that
N
1
C(g)
1| (f
o.......................... (T.) T7 113
N
(A.37)
1 N
(8 51 f(x.)
(() + + O(s?)
where (zx) is the Jacobian of x; and (ss²) contains terms like (r,,))),,, etc with (x)( the Hessian tensor of f at x;- Some effort converts Eq. (A.37) into a familiar form,
N Σ
C(g) = N
(E - 1
(A.38)
with
= 5yz J(f)(z)) 8o; + (ys - (xx))
(A.39)
Regard that Eq. (A.38) recovers Eq. (6)) via different routes and starting points, although the qi in Eq. (18) and A.39 slightly differ due to distinct variational conditions. This heuristic derivation then confirms that data deviations arouse network variations induce the alternative finetune form. Consequently, both derivations yield same results.
EXPERIMENTAL DETAILS
All code and data can be found at https:://ithub. com/ HHTseng///gg/// Layer- Sttpt--arraiiinnnl--aaa wwwiiiatiinnnlaae www.oaannaallssee git The norms |Ix and |ly in all experi- ments are set as Euclidean Lg -orms, unless otherwise specified. Some notable details are remarked here.
14

---

Published as a conference paper at ICLR 2023
8.1 1D SERIES REGRESSION
Dataset (Fig.
A ID series D for the pretrained f and another series D for transfer learning are given by
D {1i sin(T.:))) xs E [-1, N=2000
N=2000
(B.40)
D
(x; 1.0+0.05.. ((4) u4 + 0.03 x E 1-11,1]
with
(x,) = 0.4 1.3998,
ε ~ (V1.,, 0.8)
(normal distribution mean=1.5, variance=0.8),
N ~-r0 1-),11
(uniform distribution in [-1, 1))-
2.0
old data
new data
1.5
1.0
ylx)
0.0
-0.5
-1.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
x
Figure 8: The original data D and new data D for transfer learning.
Network Architecture The pretrained model f using D was 3 fully-connected-layrr network with 64 nodes at each layer and ReLu used as activation functions except the output layer. A finetuned model JGD using Gradient Descent (GD) retrained the last layer of f by D. f and JGD were trained under La-loss with ADAM optimizer learning rate 10-3 3 by 8,000 and 12,000 epochs, respectively. The finetuned model by the LVA method Iresado using required no training process but directly replaced the last layer of f by fn + (fn ofz) with 8fn given in Eq. 21).
Hardware
One NVIDIA V100 GPU (32 GB GPU memory) with 4 CPUs (128 GB CPU memory).
Runtime
Approximately 3 sec in the LVA method, 4 minutes in GD method.
8.2
SPEECH ENHANCEMENT
Dataset The utterances used in the experiment were excerpted from the Deep Noise Suppression Challenge (Reddy et al. 2020). We randomly selected 8000 clean utterances for training and 100 utterances for testing. Five noise types ( (White-noise, Train, Sea, Aircraft-cabin, Airplane-takeoff) were involved to form the training set (to estimate the pretrained SE model) and three noise types, (Baby-cry, Bell, Siren) was used to form the transfer learning and testing sets. For the training set, the 8000 clean utterances were equally divided and contaminated by the five noise types (thus, each noise type had 1600 noisy utterances) at four signal-to-nole ratio (SNR) levels, (-5, 0, 10] dB. For the testing set, 100 clean utterances were contaminated by the (Baby-cry. Bell, Siren} noises {-1,1} (dB) SNRs. For the adaptation set, we prepared 20-400 patches contaminated by three noise types, (Baby-ery, Bell, Siren), at (-1, dB SNRs.
15

---

Published as a conference paper at ICLR 2023
Noisy Speech
Enhanced Speech
Clean Spectrogram
STFT
iSTFT
MSE loss
Noisy phase
SE model
Enhanced Spectrogram
Noisy Spectrogram
Figure 9:
The flowchart of Deep Denoising Autoencoder (DDAE) for SE.
Implementation All speech utterances were recorded a 16kHz sampling rate. The speech signals were first transformed to spectral features (magnitude spectrograms) by short-time Fourier transform (STFT). The Hamming window with 512 points and a hop length of 256 points were applied to transform the speech waveforms into spectral features. To adequately shrink the magnitude range, a loglp function (loglp(x) == log(1 [1 + r)) was applied to every element in the spectral features. During training, noisy spectral vectors were treated as input the SE model to generate enhanced spectral vectors. The clean spectral vectors were treated as the reference, and the loss was computed based on the difference of enhanced and reference spectral vectors. Subsequently, the SE model was trained to minimize the difference. During inference, noisy spectral vectors were inputted to the trained SE model to generate enhanced ones, which were then converted into time-domain waveform with the preserved original noisy phase via inverse STFT.
Network Architecture The overall flowchart of the SE task is shown in Fig. In this study, we implemented SE systems using two neural network models to evaluate the proposed LVA. The first one is the deep denoising autoencoder (DDAE) model (Lu et al.) 2013), which serves as a simpler neural-network architecture. The other the Bi-directional LSTM (BLSTM) (Chen al. (015). which is relatively complicated compared to DDAE. For DDAE, the model is composed of fully-connected layers with 512 units. Activation functions were added to each layer except the last layer. The LeakyReLU with negative slope of 0.01 used the activation function. For BLSTM, the model consists a one-layer BLSTM of 300 nodes and one fully cally-connected layer. The training setting is the for the two SE models. The learning rate of Adam Optimizer was 10-3 After pretraining, only the last layer parameters were updated in the finetuning stage. According to Eq. (1 the pseudo inverse of latent features was used calculate 8f.
Hardware One NVIDIA V100 GPU (32 GB GPU memory) with 4 CPUs (128 GB CPU memory).
Runtime Approximately minutes in the LVA method and 25 minutes in GD method with 40 utterances.
8.2.1 ADAPTATION ON MISMATCHED SPEAKERS
In Sec. 5.2 we have confirmed the effectiveness of the proposed LVA on the noise type and SNR adaptation of the SE task, where the main speech distortion is from the background noise. In
16

---

Published as a conference paper at ICLR 2023
BELL SMR:-1
SKR
Kokay
Pretrain
2.0
CD
2.2
2.0
LXS
2.0
500
3.5
1.4
1.4
122
1.2
1.2
2.6
2.4
2.2
2.0
2.2
2.0
Scom
1.88
500
1.6
1.A
122
1.2
1.2
target samples
target samples
target samples
(a) PESQ score of DDAE.
SPEE SMR.-L
Nocke
0.35
0.90
GD
Protuin
0.85
Lix
0,50
5:00
0.75
0.75
0.75
0.70
0.65
0.85
0.00
0.60
BABY SNR.
SHR
SIREN SMR:1
0.90
0.85
0.80
0.60
0,80
9.75
0.70
0.65
0.65
0.00
0.80
samples
samples
target samples
(b) STOI score of DDAE.
17

---

Published as a conference paper at ICLR 2023
SMR
BELL
0.030
Kokay
0.025
0.05
Pretrain
0.025
LXS
LXS- DT
S00R
0.015
0.030
0.02
0.030
0.005
0.000
0.000
0.00
0.000
0.025
0.043
0.020
0.005
0.000
0.015
0.000
0.023
Score
0.533
0.010
0.030
0.015
0.012
0.000
0.003
0.000
0.002
0.500
target samples
targel Samplees
target samples
(c) Loss of DDAE.
SNR.
SELL SMR--1
SIREN Ski.
Nocee
Protoah
2.4
GD
Lix
Score
2.0
3.5
1.6
1.2
1.2
BABY
SELL
2.
2.4
2.6
2.0
2.4
2.2
Score
5005
1.0
1.6
1.4
300
target samples
samples
samples
(d) PESQ score of BLSTM.
18

---

Published as a conference paper at ICLR 2023
SMR:-1
Kokay
0.85
0.00
0.85
Pretrain
LXS
CD
LXVS
0.00
6.80
0.75
0.75
0.75
6.70
0.05
0.00
0.00
0.50
0.90
0.00
0.90
0.65
0.85
0.80
0.65
0.60
0.60
0.50
larget samples
target samples
target
(c) STOI score of BLSTM.
SIREN
0.030
0.025
0.05
Lix
Nocke
GD
Protoin
0,020
0.015
Score
0.033
0.010
0.02
0.030
0.005
0.000
0.000
BASY
BELL
SPEN SNR
0.025
0.34
0.000
0.020
0.015
0.015
Score
0.02
0,000
0.025
0.020
0.000
+00
target samples
samples
samples
(I) Loss of BLSTM.
19

---

Published as a conference paper at ICLR 2023
this section, we further investigate the achievable performance of LVA on SE where speakers mismatched in the source and target domains.
First, excluded the utterances of the speakers used in the training set and regenerated the adaptation and the test set by randomly sampling 200 utterances (100 for the adaptation set and 100 for the test set) from the remaining Deep Noise Suppression Challenge dataset. The experimental setup was same as Sec. with testing speakers not given in the training set. We note that since the speakers did match, an OT process was performed align speakers. In Table [ we denote LVA-OT as LVA for shorthand. The comparison of BLSTM model adaptation using GD and LVA is shown in Table !
The table demonstrates that both GD and LVA improve the PESQ and STOI score after adaptation, while LVA consistently yields better performance over different noise types and SNR levels.
Table 1: Performance of finetuned models on mismatched speakers.
noisy (no SE
Pretrain (no finetune)
Finetuned
Finetuned by LVA
by GD
noise type SNR
PESQ
STOI
PESQ
STOI
PESQ
STOI
PESQ
STOI
Babyery
1.40
0.74
1.58
0.78
1.58
0.79
1.88
0.81
Babycry
-1
1.52
0.77
1.69
0.80
1.71
0.82
1.97
0.83
Bell
-1
1.96
0.84
1.97
0.84
2.18
0.86
2.44
0.89
Bell
-1
2.07
0.86
2.09
0.86
2.29
0.87
2.55
0.90
Siren
-1
1.56
0.81
1.74
0.82
1.81
0.84
2.03
0.84
Siren
+1
1.64
0.83
1.84
0.84
1.97
0.85
2.08
0.85
Average
1.69
0.81
1.82
0.82
1.92
0.84
2.16
0.85
8.2.2 ADAPTATION USING FULLSUBNET
In this extended experiment, we utilize a FullSubNet (Hao et al. (202]) pretrained network to perform SE domain adaptation tasks. The FullSubNet is state-of-the-art SE model reaching the performance of PESQ: 2.89 and STOI: 0.96 on the DNS- coaalchlee..e.
This implementation serves examine the effect of pretrained network. To compare with previous results of the BLSTM model in Sec. 8.2, same source domain data and the target domain set are used for adaptation. The FullSubNet receives input of spectrum amplitude and outputs complex Ideal Ratio Mask (cIRM) to compare with cIRM labels under L2-loss. The architecture of the FullSubNet composed of full-band and sub-band model. Within each model, two stacked LSTM layers and one fully connected layer are included. The LSTMs contain 512 hidden units and 384 hidden units in the full-band model and the sub-band model, respectively. The detailed structures be found in (Hao et al. 2021).
pretrained FullSubNet was obtained from the official Githut where the last linear layer was finetuned to adapt to the target set. The finetuned networks were subsequently evaluated on an adaptation test set to measure PESQ and STOI, where the results are shown in Table 2 The comparison shows that finetuning from pretrained FullSubNet achieves enhanced baseline scores on the target domain (see Table 2 [no finetune]). While adaptation under the gradient descent significantly improves the pretrained network, the proposed LVA method outperforms the traditional gradient descent method terms PESQ and STOI. The results are consistent with the previous SE experiments to confirm that the proposed LVA promptly reaches better adaptation.
8.3
SUPER RESOLUTION FOR IMAGE DEBLURRING
The LVA formulation Eq. following observation shown Fig. II where the left-hand side depicts the usual convolution operation in CNNs. Given a CNN, denote the kernels by (Cijas) with (i, ax 8) E Tn La2 Cin x Cou and L4, To2 the kemel size (e.g., 3 x 3), Cin input channels and Cou output channels. is noticed
CNN extension
19 be extended to CNN layers with the
20

---

Published as a conference paper at ICLR 2023
Table 2: Performance of finetuned models from pretrained FullSubNet.
noisy
Pretrain
Finetuned by GD
Finetuned
(no SE)
(no finetune)
by LVA
noise type SNR
PESQ
STOI
PESQ
STOI
PESQ
STOI
PESQ
STOI
Babycry
-1
1.08
0.75
1.88
0.91
2.37
0.89
2.61
0.91
Babyery
1.09
0.77
2.05
0.92
2.63
0.91
2.75
0.92
Bell
-1
1.06
0.84
2.10
0.94
2.92
0.93
3.00
0.94
Bell
1.08
0.86
2.29
0.95
3,05
0.94
3.11
0.95
Siren
1.17
0.81
2.58
0.94
2.92
0.93
3.02
0.94
Siren
1.18
0.83
2.80
0.95
3.06
0.94
3.16
0.95
Average
1.11
0.81
2.28
0.94
2.82
0.94
2.94
0.94
low-tesolution
finetuning CNN fn
image
high-resolution image
feature extraction
non-linear mapping
reconstruction
Figure 10: SRCNN for image deblurring. A pretrained model composed by sequence of CNN layers Jn f2 fI, where the last CNN layer fn was finetuned to fit more blurred images D.
that CNN kernels Cijog cover a patch the input image one at a time, denoted by Tijo then each covered patch Vijo has the same size Cijo. for fixed B. Consequently, each output pixel value (ks, E, () is computed by,
VkLB = Σ Cijas
Dija
(B.41)
ijaa
where the underscore remarks indices are be summed over. recall that a fully-connected layer of weights [W.n) calculates the output vector = (UI...., Um....) from an input vector (u,,..., .).. ......... by,
Um Σ W mn Un
(B.42)
then we noticed mathematically there NO distinction between Eq. (B.4l and Eq. (B.22) as index has the role of ija and m has the character of kt3. In this view. CNN kernlls Cijag can be transformed to 2D matrix W mm by index conversion function 77, and image patches (ija can be transformed into 1-D vector u, by another index function ç, see Fig. (RHS).
Thus, we completed the conversion of CNN to fully-conneeted-lee layer and naturally Eq. follow. There can also be possible extensions to other locally connected networks in the future scope.
(88
Datasets We used the CUration of Flickr Events Dataset (CUFED) Wang et al. 2016 to train SRCNN (Scaman & Virmaux 2018) for image deblurring available at https::: acsweb uesd. edu/ http//yyw17////ee/ eveett curat lon. htmll We randomly selected 2000 images from the CUFED and randomly crop 20 patches of size 33 33 from each image. Three pairs of training &
21

---

Published as a conference paper at ICLR 2023
CNN korne Gjad
equavalent matrix =[C[ooo))
potch Aya (R.0)
nattened Irput (
futeree 4
convolved output Yepp
Figure 11: (LHS) The masked patch Tija is regarded as fully-connected under CNN kernels, Eq. (B.4T). (RHS) By flattening the input and the convolved output into ID vectors, a CNN is equivalent to fully-connected layer Eq. B.42).
finetune combination D, D were considered, see Fig. 12
D1 {(1,, vc) yi {original patches), E { x3 blurred patches] N=10K
:=1
(B,43)
D1 {(T,, [k) g: {original patches), 3 E: { x6 blurred patches) }
N=
i=1
D2 = { (at, w:) 1% E {original patches), 2; E { [x3 blurred patches) N=40K
i=1
(B.44)
D2 = { (*,, 91) 1 E {original patches}, 34 E { [ x6 blurred patches) N=40K
i=1
and
D3 { ys) ye E {original patches), { x3, x4 blurred patches] } N=
(B.45)
D3 ( Th) 0 E (original patches], T. E { x blurred patches) N=40K i=1
Another image dataset SET14 (Zeyde et al. 2010) for testing finetuned models.
The color images were first transformed to a different color space, YCbCr, to apply the SRCNN model on a single luminance channel. Our SRCNN, Fig. 10, receives D and DD of low resolution image patches to output high resolution ones of the same dimension. A pretrained model composed by a sequence of CNN layers = Sn ..... /2 Si was pretrained by D under L2-loss. Then the last CNN layer f. was finetuned to adapt to more blurred images D) Two finetuning
Implementation
methods by GD and LVA were compared. In particular, the conversion Eq. B..4 + Eq. B.42 was used to admit LVA solutions.
Multiple pretrained architectures f were implemented, of which were
f fi o o
(3-layer CNNs)
= f1 o 12 f4 Ss. = f fa
(5-layer CNNs)
(B.46)
O
(7-layer CNNs)
Network Architecture We adopted the fast End-to-End Super Resolution model similar to the original architecture of SRCNN (Dong et al. 2016) with the insertion of a batch-normalizatie layer right after each ReLu activation function. The training batch size for f was set 128, and the testing batch size for gradient descent finetuning as 1.
The network architecture of pretrained models corresponding to Eq. (B.66):
1. 3-layer-CNN of kernel size: (9, 5, 5) with channel size: (1, 1).
22

---

Published as a conference paper at ICLR 2023
Original
Bicubic x3
Bicubic x4
Bicubic x6
Original
Bicubic x3
Bicubic x4
Bicubic x6
Original
Bicubic x3
Bicubic x4
Bicubic x6
Figure 12: Samples of CUFED for training and finetuning (bicubic(x3, x4, x6)) for
Eq. BB.43 b B.43
2. S-lyyerCCNN of kernel size: 5, 5, 5, 5) channel size: 32,
3. 7-layer-CNN of kernel size: (9, 5, 5, 5, 5, 5, 5), channel size: (1,32, 32, 32, 1).
where the first and the last number in the channel size is always to indicate the input and output luminance channel.
23

---

Published as a conference paper at ICLR 2023
shows the deblurring results by finetuned models of GD and the LVA, where our method reached the highest PSNR with the least finetuning samples N = 256. This outcome indicated the LVA method is beneficial for few-shot learning, especially when adaptation data is few.
More extensive results Fig- 13
More extensive experiments conducted observe performance. The following results com- pared the finetune loss and PSNR with different combinations of training & finetune data in Eq. (B.43 (B.45) and multiple pretrained architectures of f in Eq. [B46
In Fig. (a) (f), the horizontal axis of each subfigure is the patch number and the title of each subfigure indicated the transition from the old data D to new data D by D - e.g. {x3, x4) - x6.
Overall, the following facts were observed in Fig. (a)~(ff:
1. The loss is roughly inverse- propooooppoo to PSNR score. Increasing CNN layers can han- dle more difficult tasks, such {x3} + {x4, x6} compared to {x3] - { {x6] and { (44, x6] + { [33 since {x3] + { {x4, x6} was asked to adapt to more new data than it was originally trained.
2. In all figures, the LVA method had stable performance over GD in both loss and PSNR even when sample sizes (r-axis) largely decreased. We can see ups and downs in orange bars, but not blue bars.
3. Obviously in Fig. (a), GD showed difficulty in learning {x3) + {x4, x6} and {x3} - { {x6] than {x3, x4) + { [x6], but the LVA method remained roughly the same (stable) as well retaining its best performance over GD.
4. Overall, GD method improvement was not consistent with the data number, likely due to the local minimum occurring in neural networks. Conversely in the LVA method, the performance (stably) proportional to data samples. Therefore, one could easily derive acceptable results with few finetune data.
Fig. (a) ~ (f) then concludes that the LVA method has better performance and is more efficient than GD in terms of adaptation data amount considered.
Hardware One NVIDIA V100 GPU (32 GB GPU memory) with 4 CPUs (128 GB CPU memory).
Runtime Approximately minutes in the LVA method and minutes in the GD method with 256 patches (1,000 epochs).
8.4
ADDITIONAL EXPERIMENTS ON DOMAIN ADAPTIVE IMAGE CLASSIFICATIONS
We conduct two additional experiments on images to investigate domain adaptation ability using (1) Office-31 Saenko et al. 2010) and (2) MNIST USPS.
Office-31: Table shows the results of domain transitions within the 3 domains of Office-31:
Amazon (A), DSLR (D) and Webcam ( (W), resulting in 6 possible transitions to classify 31 items. On each individual source domain (A, D, W), the pretrained model was trained for 30 epochs. Subsequently, it was finetuned onto target domains for another 30 epochs. The proposed LVA obtained higher adaptation accuracy in most domain transitions, confirming that LVA can promptly adapt to new tasks.
Table 3: Office-31: adaptation accuracy between 3 domains.
Domain
A-+D
A-W
D-+A
D->W
W W
W-+D
pretrain
77.4 %
67.1 %
54.2 %
80.7 %
59.4 %
92.9 %
GD
78.7
71.0
58.7
86.5
60.6
94.2 %
LVA
84.5 %
79.4 %
71.0 %
87.7 %
70.3 %
93.6 %
24

---

Figure 13: Results image deblurring by SRCNN, Column (b): testing input images from SET14 14; column (c): results of LVA model finetuned with 256 patches; column (d): results of GD model finetuned with 256 patches; column (e): results of GD model finetumed with 16,384 patches. The Published as a conference paper at ICLR 2023
(a) Ground truth
(b) Bicubic x6
(c) LVA (N=256) PSNR: 36.58
(d) GD (N=256) PSNR: 19.78
(e) GD (N=16384)
PSNR: 25.45
Ground truth
Bicubic x6
PSNR: 37.98
PSNR: 20.64
PSNR: 23.81
Ground truth
Bicubic x6
PSNR: 36.80
PSNR: 20.41
PSNR: 23.70
corresponding PSNR shown in the figures indicated LVA method reached the best PSNR with the least finetuning samples.
In this experiment, the pretrained model was trained on MNIST (source domain) with 98.94% accuracy by 20 epochs. While adapting to the target domain USPS, the accuracy of the pretrained model dropped rapidly to 65.32%. Subsequently, gradient descent (GD) and the proposed LVA were deployed to enhance the target domain results. Table shows the adaptation accuracy attained at different finetune epochs. It was observed that LVA consistently outperformed GD and reached high accuracy in just a few epochs. These two additional experiments on image classifications again verify that the proposed LVA is capable of real-world DA tasks; the adaptability is general.
MNIST + USPS:
Table 4: MNIST + USPS (classification accuracy)
Finetune method 10 epochs
50 epochs
90 epochs
120 epochs
pretrain
65.32%
GD
74.64% 86.90%
80.12%
82.06% 90.83%
82.46% 90.98%
LVA
90.13%
25

---

Published as a conference paper at ICLR 2023
PSNR
PSNR (x4,x6)
(x3.x4)
1.0
10
target
target samples
target
target samples
(a) PSNR SRCNN.
LOSS X3 x4
LOSS:
LOSS (x4,x6)
LOSS (x3.x4) X6
0.028
0.000
0.007
0.006
0.037
0.003
0.0%6
0.005
0.035
0.003
0.001
0.032
0.002
0.031
500
1500
1000
19.00
500
750
1000
1000
1900
target samples
target samples
target samples
target samples
(b) Loss of SRCNN.
26

---

Published as a conference paper at ICLR 2023
3-ayer loss: >
3-layer loss: x3 x6)
3-lyyer loss: x4) X6
0.020
0.008
0.020
0.015
0.010
0.004
0.005
0.005
0.002
256
64
128
256
(c) Loss: finetuning fa of fs f2 fi patch number (z-axis)
3-ayer
PSNR x6)
3-layer (X3,
40
40
40
peaudo
peauto
pasudo
CO
CD
GD
30
30
30
25
20
15
15
15
64
128
256
(d) PSNR: finetuning of fa fa f1
5-layer loss;
5-layer (x4, xe)
5- S-layer (x3, x4)
0.018
0.016
0.016
peeudo
powuto
poeuuo
0.014
0.014
0.014
GD
0.012
0.012
0.012
0.010
0,010
0.010
0.004
0.004
0.004
0.002
0.002
0.002
0.000
0.000
0.000
16
128
258
128
258
(e) Loss: finetuning fo of f = fa o fa fa fa
S-layer PSNR
5-laye PSNR: x3 XB)
5-layer PSNR: (K3, x4) XB
35
30
30
30
25
25
20
15
15
18
32
64
128
256
8
16
32
64
128
258
16
32
64
128
286
(1) PSNN: finetuning fa of f fo f4 fa
f2
f1
27

---

Published as a conference paper at ICLR 2023
7-layar
7-layer loss: |s4,
7-layer
0.008
0.008
0.008
posudo
ppoudo
pscudo
0.007
0.007
0.007
0.006
0.006
0.008
0.004
0.004
0.004
(g) Loss: finetuning of fr
fo
fa
f2 fi
T-layer PSNR: x3
7-layer PSNR: (X4, X6)
7-layer PSNR: [x3, x4) xB
40.0
40.0
40.0
paoudo
pseuto
paeuco
37.5
GD
37.5
37.5
35.0
35.0
35.0
32.5
30.0
30.0
30.0
27.5
27.5
27.5
25.0
25.0
25.0
22.5
22.5
22.5
20.0
20.0
20.0
16
32
84
120
255
8
16
32
64
128
256
16
32
64
128
298
(h) PSNR: finetuning fr
fr fo
fo
fi
28


Reviews:
[{"review_id": "vb1dJyl5QQK", "reviewer": "Reviewer_xrcZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a theory framework to enable better finetuning by only adding a Lipschitz function.", "summary_of_the_review": "The proposed framework allows us to finetune the model by only adding a residual but the residual can be different to compute and may not even be applicable for many cases.", "strength_and_weaknesses": "pros:\n\n1. It is interesting to see we can only add a Lipschitz function $\\delta_f$ to finetune on different datasets.\n\n\nCons:\n\n1. The framework requires sample alignment first. To me, the sample alignment can be a very huge cost and may not be applicable in most finetuning cases. For example,  given a pre-trained ImageNet classifier and a new dataset such as Office31, we have to align samples first. It can be challenging to measure the difference and thus align successfully. If we can align them, we may just use the aligned image and labels to train the model. So we don't need to finetune.\n\n2. The experiment results are not enough and I would like to see quantitative comparisons.  In particular, since the paper title suggests the domain adaptation, I would like to see how can we benefit from the proposed framework on popular domain adaptation datasets, e.g., DomainNet, Office31.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ur9cjnTQeK", "reviewer": "Reviewer_2KJc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new aspect to look at transfer learning. The basic idea is to allow each layer $f_n$ to change a little bit ($\\delta f_n$). This gives bounds on losses on the new dataset ($L(g)$) under a few assumptions. Based on this framework, $L(g)$ can be re-written as a least-square problem, and $\\delta f_n$ can be optimized for the new dataset. The paper conducts experiments on a synthetic dataset, a speech enhancement task, and image de-blurring/super-resolution. Results show the proposed method outperforms GD. ", "summary_of_the_review": "Based on the questions and concerns above, I think the paper has some problems in theory and space for improvement in experiments. Therefore I think this version cannot be accepted. I will make the final decision based on authors' response and revised version. \n\n----------\n\nAfter rebuttal:\n\nThe authors improved the experiments and I found them good, so I'll increase my score to 6. I think the higher-order analysis can be further improved, especially for more standard neural networks with Lipschitz activations. ", "strength_and_weaknesses": "**Strengths**\n\n- The paper has a good overview of the literature.\n- The method of layer-wise optimization (from the $\\delta f_n$ point of view) is novel. Eq (17)-(18) gives new insights to transfer learning.\n- Experiments are conducted on multiple domains with consistent improvement over the GD baseline.\n\n**Weaknesses**\n\n- The paper does not explicitly define norms such as $\\\\| \\cdot \\\\|_{\\\\mathcal{Y}}$.\n- The data deviation assumption might be strong in practice.\n- Thm 3.6 is not clear and does not provide much insight. Do you mean $\\exists \\delta_f$ or $\\forall \\delta f$? Does it need to satisfy the constraint that $f+\\delta f$ is a network layer? Also, simply by setting $\\delta f==0$ should yield eq (10); in that case, why do you have this theorem. \n- Eq (16): it seems to be $\\approx$ instead of $=$ because $f_n$ is an affine transformation followed by activation. This might also introduce further error terms in the following equations, so I would like to see analysis on that. E.g., a higher-order analysis.\n- Around eq (21): it should be $\\min$ eq (17) instead of (19). That $\\delta f_n$ is linear seems infeasible due to non-linear activation, so I think mentioning the pseudo-inverse is unnecessary.\n- Experiments on speech enhancement: \n  - Do you use mismatched speakers of clean speech between training and adaption sets? \n  - Why is the fmax=6k given that training data are 16kHz? \n  - It would be interesting to test on the no-reverb test set as it provides different noise types and part of them can be used as the adaptation set. \n  - The baseline is sort of weak. More recent denoisers can achieve test pesq > 2.8 or even 3.0 (see e.g. FullSubNet, Demucs, CleanUNet, ...). I think it's feasible to take the pretrained models (all of these models have) and conduct adaptation to new noise types and languages in DNS-2021/2022. If it comes out with good results then the paper can have more impact. \n- Experiments on image super-resolution/de-blurring: the major concern is that there is no quantitative results, so it is hard to evaluate the methods fairly. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3F-CbLDFcP", "reviewer": "Reviewer_hjUS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a theoretical analysis of network-based transfer learning using layer variational analysis. Their analysis is claimed to prove that the success of transfer learning is guaranteed with certain data conditions. Based on their analysis, they also propose an alternative method for network-based transfer learning, which demonstrates better efficiency and accuracy for domain adaptation.", "summary_of_the_review": "I recommend rejection of this paper because of the following:\nTheir theoretical analysis does not capture several important aspects of transfer learning (see Weaknesses section) and their analysis has a few potential technical issues (see Quality section). The proposed method based on their analysis may not be seen as entirely new to me because similar ideas do exist in meta-learning literature (see Weaknesses section).", "strength_and_weaknesses": "**Strength:**\n\n1. Most technical analysis is correct and their empirical study is also sufficient.\n2. The proposed approach which directly solves the linear functional on top of the pre-trained feature extraction show promising results on downstream tasks\n\n**Weaknesses:**\n\n1. Their analysis does not capture many important aspects of network-based transfer learning. Firstly, they only discuss the training loss but do not consider the generalization ability of models on test data. But achieving good training loss does not necessarily mean the learning is successful. Secondly, SGD optimization is normally used to train neural networks, their analysis does not consider the complexity of optimization. And their variational analysis in function space does not capture the optimization procedure in parameter space.\n2. While the proposed method based on their analysis is interesting in the transfer learning setting, the idea is not entirely new. Similar ideas already exist in the meta-learning literature. ([Meta-Learning with Differentiable Closed-Form Solvers](https://arxiv.org/abs/1805.08136), [Meta-Learning with Differentiable Convex Optimization](https://openaccess.thecvf.com/content_CVPR_2019/papers/Lee_Meta-Learning_With_Differentiable_Convex_Optimization_CVPR_2019_paper.pdf), [Meta-Learning Priors for Efficient Online Bayesian Regression](https://arxiv.org/abs/1807.08912)). I am not sure if it is the first time the idea is applied to transfer learning. Moreover, although I can see the connection between the proposed method and their theoretical analysis, I don't think the theoretical part is absolutely necessary to come to their proposed approach. They could simply talk about their method without the layer variational analysis.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster