Learning Representation from Neural Fisher Kernel with Low-rank Approximation
Authors: Ruixiang ZHANG
Shuangfei Zhai
Etai Littwin
Joshua M. Susskind
Venue: ICLR 2022 Poster (2022)

Abstract: In this paper, we study the representation of neural networks from the view of kernels. We first define the Neural Fisher Kernel (NFK), which is the Fisher Kernel applied to neural networks. We show that NFK can be computed for both supervised and unsupervised learning models, which can serve as a unified tool for representation extraction. Furthermore, we show that practical NFKs exhibit low-rank structures. We then propose an efficient algorithm that computes a low-rank approximation of NFK, which scales to large datasets and networks. We show that the low-rank approximation of NFKs derived from unsupervised generative models and supervised learning models gives rise to high-quality compact representations of data, achieving competitive results on a variety of machine learning tasks.

Published as a conference paper at ICLR 2022
LEARNING REPRESENTATION FROM NEURAL FISHER KERNEL WITH LOW-RANK APPROXIMATION
Ruixiang Zhang
Shuangfei Zhai, Etai Littwin, Josh Susskind
Mila, Université de Montréal
Apple Inc.
ruixiang. zhang@umont ca
{szhai, elittwin, j @appslee.indppppeeccom
ABSTRACT
In this paper, we study the representation of neural networks from the view of kernels. We first define the Neural Fisher Kernel (NFK), which is the Fisher Kernel (Jaakkola and Haussler 1998) applied to neural networks. We show that NFK can be computed for both supervised and unsupervised learning models, which can serve as unified tool for representation extraction. Furthermore, we show that practical NFKs exhibit low-rank structures. We then propose an efficient algorithm that computes a low rank approximation of NFK, which scales large datasets and networks. We show that the low-rank approximation of NFKs derived from unsupervised generative models and supervised learning models gives rise high-quality compact representations of data, achieving competitive results on a variety of machine learning tasks.
INTRODUCTION
Modern deep learning systems rely on finding good representations of data. For supervised learn- ing models with feed forward neural networks, representations can naturally be equated with the activations of each layer. Empirically, the community has developed a set of effective heuristics for representation extraction given trained network. For example, ResNets (He al. 2016) trained on Imagenet classification yield intermediate layer representations that can benefit downstream tasks such as object detection and semantic segmentation. The logits layer of a trained neural network also captures rich correlations across classes which can be distilled to a weaker model (Knowledge Distillation) (Hinton et al. 2015).
Despite empirical prevalence of using intermediate layer activations as data representation, it is far from being the optimal approach to representation extraction. For supervised learning models, remains a manual procedure that relies on trial and error to select the optimal layer from a pre-trained model to facilitate transfer learning. Similar observations also apply to unsupervised leurning models including GANs (Goodfellow et al., 2014)) VAEs (Kingma and Welling 2014), as evident from recent studies (Chen et al.] 2020a) that the quality of representation in generative models heavily depends extract activations features. Furthermore, although that GANs and VAFs are known to be able to generate high-quality samples from the data distribution, there no strong evidence that they encode explicit layerwise representations to similar quality as in supervised learning models, which implies that there does not exist a natural way to explicitly extract a representation from intermediate layer activations in unsupervisedly pre-trained generative models. Additionally, layer activations alone do not suffice to reach the full power of learned representations hidden neural network models, shown in recent works (Mu et al., 2020) that incorporating additional gradients-based features into representation leads to substantial improvement over solely using activations-bant features.
on the choice of layer from which
In light of these constraints, we are interested in the question: is there principled method for representation extraction beyond layer activations? this work, we turn to the kernel view of neural networks. Recently, initiated by the Neural Tangent Kernel (NTK) (Jacot et al. 2018) work, there have been growing interests in the kernel interpretation of neural networks. It was shown that neural networks in the infinite width regime Our key intuition is that, the kernel machine induced by the neural network provides a powerful and principled way of investigating the non-linear feature transformation in neural networks using the
reduced kernel regression with the induced NTK.

---

Published as a conference paper at ICLR 2022
linear feature space of the kernel. Kernel machines provide drastically different representations than layer activations, where the knowledge of a neural network is instantiated by the induced kernel function over data points.
In this work, we propose to make use of the linear feature space of the kernel, associated with the pre-trained neural network model, the data representation of interest. To this end, we made novel contributions on both theoretical and empirical side, as summarized below.
We propose Neural Fisher Kernel (NFK) as a unified and principled kernel formulation for neural networks models in both supervised learning and unsupervised learning settings.
We introduce a highly efficient and scalable algorithm for low- low-rank kernel approximation of NFK, which allows us to obtain compact yet informative feature embedding the data representation.
We validate the effectiveness of proposed approach from NFK in unsupervised learning, semi- supervised learning and supervised learning settings, showing that our method enjoys superior sample efficiency and representation quality.
2 PRELIMINARY AND RELATED WORKS
In this section, we present technical background and formalize the motivation. We start by intro- ducing the notion of data representation from the perspective kernel methods, then introduce the connections between neural network models and kernel methods.
Notations.
Throughout this paper, we consider dataset with N data examples D ((xx,y:)] we use p(x) to denote the probability density function for the data distribution and use Pdata (x) to denote the empirical data distribution from D.
Kernel Methods. Kernel methods (Hofmann et al. 2008) have long been staple of practical machine learning. At their core, kernel method relies on a kernel function which acts as a similarity function between different data examples in some feature space. Here consider positive definite kernels KC :' x X + I over a metric space X' which defines a reproducing kernel Hilbert space H of function from X' to R, along with a mapping function y X' H. such that the kernel function can be decomposed into the inner product K(xx, z) = (f(x),, p(z)). Kernel methods aim to find a predictive linear function f(x) (f, ((x)) h in H. which gives label output prediction for each data point x E X. The kernel maps each data example x E X to a linear feature space (xx). which is
the data representation of interest. Given dataset D, the predictive model function is typically
estimated via Kernel Ridge Regression (KRR), f argmin N Σ (f (x.) y:)² 2 + X 1l
Neural Networks and Kernel Methods. A long line of works (Neal 1996 Williams
1996 Roux
and Bengio 2007| Hazan and Jaakkola 2015 Lee et al. 2018 de G. Matthews al.
2018 Jacot
et al.]] Chen and Xu [021]] Geifman et al. 2020 Belkin et al. 2018 Ghorbani et al., 2020).
have studied that many kernel formulations can be associated to neural networks, while most of them correspond to neural network where being fixed kernels (e.g. Laplace kernel, Gaussian kernel) or only the last layer is trained, e.g., Conjugate Kernel (CK) (Daniely et al. 2016), also called as NNGP kernel (Lee et al.) 2018). On the other hand, Neural Tangent Kernel (NTK) (Jacot et al. 2018) is a fundamentally different formulation corresponding to training the entire infinitely wide neural network models. Let (0; x) denote a neural network function with parameters O, then the empirical NTK is defined as Kunkk (x, 2) = (Ve Vel z)). (Jacot et al., 2018 Lee et al. 2018) showed that under the so-called NTK parametrization and other proper assumptios, the function f(;; 0) learned by training the neural network model with gradient descent is equivalent to the function estimated via ridgeless KRR using Kutk as the kernel. For wiith neural networks, by taking first-order Taylor expansion of funnction f around the 0, kernel regression under Knntl can be seen as linearized neural network model at parameter 0, suggesting that pre-trained neural network models can also be studied and approximated from the perspective of kernel methods.
Fisher Kernel. The Fisher Kernel (FK) first introduced in the seminal work (Jaakkola and Haussler 1998 Given a probabilistic generative model pe (z), the Fisher kernel is defined as: Kitsher (x, Z)
Ve log pe (x) T Ve log Pe Po(z) = U I U2 where Ux = Ve log Px(x) is the so-called Fisher score and I is the Fisher Information Matrix (FIM) defined the covariance of
Ex-po(x) Vo log Po (x)Vo Vo log po Then the Fisher vector is defined the Fisher score: =
as Vx= - log Vo I- --- U One can utilize the Fisher Score mapping from the
2

---

Published as a conference paper at ICLR 2022
pe(x) (x(()))
po(x) x Bo(x)
Kij Kut (x)2)) Vyloggo(x)) 'I" "V Vylogg((x))
K
Xg
Xg
Pro trained model
Energy based model
Neural Fisher Kernel
Low-rank kemel approximation
Data representation
Overview of our proposed approach. Given pre-trained neural network model, which can be either an unsupervised generative model Po(x) (e.g. GANs, VAEs). or a supervised learning model po (ylx), we aim to extract compact yet informative representation from it. By reinterpreting various families of models as energy- beregy models (EBM), we introduce Neural Fisher Kernel (NFK) Katk principled and unified kernel formulation for neural network models (Section. [.1]. We introduce a highly efficient and scalable kernel approximation algorithm (Section. (32) to obtain the low-dimensional feature embedding Cx2 which serves the extracted data representation from NFK.
Figure I:
data space X' to parameter space θ and obtain representations that are linearized. As proposed in (Jaakkola and Haussler [998| Perronnin and Dance (000)), the Fisher vector Vxx can be used as the feature representation derived from probabilistic generative models, which was shown to be superior to hand-crafted visual descriptors in a variety of computer vision tasks.
Generative Models In this work, consider variety of representative deep generative mod- els. including generative adversarial networks (GANs) (Goodfellow et al.] 2014), variational auto- encoders (VAEs) (Kingma and Welling 2014), as well we normalizing flow models (Dinh al. 2013) and auto-regressive models (van den Oord et al. 2016). Please refer to (Salakhutdinov 2014 for more technical details on generative models.
3 LEARNING REPRESENTATION FROM NEURAL FISHER KERNEL
We aim to propose a general and efficient method for extracting high- high-quality representation from ire-trained neural network models. As formalized in previous section, we can describe the outline of our proposed approach as: given a pre-trained neural network model f(x 0) (either unsupervised generative model p(x; 0) or supervised learning model ply x; 0)), with pre-trained weights O, we adopt the kernel formulation K, induced by model f(x; 0) and make use of the associated lincar feature embedding (xx) of the kernel K2 as the feature representation of data We present an overview introduction to illustrate our approach in Figure.
At this point, however, there exist both theoretical difficulties and practical challenges which impede a straightforward application of our proposed approach. On the theoretical side, the NTK theory is only developed in supervised learning setting, and its extension to unsupervised learning is not established yet. Though Fisher kernel is immediately applicable in unsupervised learning setting, deriving Fisher vector from supervised learning model p(y x;; 0) can be tricky, which needs the log- log-density estimation of marginal distribution (x from p(y x; O). Note that it is drastically different problem from previous works (Achille
2019) where Fisher kernel applied the joint distribution over p(x, y). On the practical efficiecyy side, the dimensionality of the feature space associated with NTK or FK is same as the number of model parameters |0), which poses unmanageably high time and space complexity when it comes to modern large-scale neural network models. Additionally, the size of the NTK scales quadratically with the number of classes in multi-class supervised learning setting, which gives rise to more efficiency concerns.
To address the kernel formulation issue, we propose Neural Fisher Kernel (NFK) Sec. 3. as unified kernel for both supervised and unsupervised learning models. To tackle the efficiency challenge, we investigate the structural properties of the proposed NFK and propose a highly scalable low-rank kernel approximation algorithm in Sec. 3.2 to extract compact low-dimensional feature representation from NFK.
3.1 NEURAL FISHER KERNEL
In this section, we propose Neural Fisher Kernel (NFK) as a principled and general kernel formulation for neural network models. The key intuition is that we can extend classical Fisher kernel theory to unify the procedure of deriving Fisher vector from supervised learning models and unsupervised learning models by using Energy-based Model (EBM) formulation.
3

---

Published as a conference paper at ICLR 2022
1.0
htp comv
0.35
0.3
explainec veriance ratio
0.90
0.8
0.85
0.70
model random model
3000
105
16a 10
10
10*
compoments
Figure 2: Left: The spectrum structure of NFKs from a CNN (green) and a MLP (red), trained on MNIST binary classification task. The NFK of CNN concentrates on fewer eigen-modes compared to the MLP. Right: The low-rankness of the NFK on DCGAN trained on MNIST. For trained model, the first 100 principle components of the Fisher Vector matrix explain 99.5% of all variances. An untrained model with the same architecture on the other hand, demonstrates much lower degree of low-rankness.
3.1.1 UNSUPERVISED NFK
We consider unsupervised probabilistic generative models pe (x) = p(x: 0) here. Our proposed NFK formulation can be applied to all generative models with tractable evaluation (or approximation) of Ve log (x)).
GANs. We consider the EBM formulation of GANs (Dai al., 2017 Zhai al., 2019 Che et al. 2020 Given pre-trained GAN model, we use D(x; 0) to denote the output of the discriminator D, and G(h) to denote the output of generator G given latent code h p(h). As a brief recap, GANs can be interpreted as an implementation of EBM training with a variational distribution, where
we have the energy- energy-function Exx; 0) - (x; 0). Please refer to (Zhai al.) 2019 Che al. 2020) for more details. Thus have the unnormalized density function (x(x) xx =(xx())) given by
the GAN model. Following (Zhai et al. 2019), we can then derive the Fisher kernel Kunkk and Fisher
vector from standard GANs as shown below:
Knnx xx 2) (Vx, Vi)
Vx = (diag(I)-
Ux Vo 0) Enyp(h) DG
where x, E I= En-p(h) [UaN) dcth) Note that we use diagonal approximation of FIM throughout this work for the consideration of scalability.
VAEs. Given a VAE model pre-trained via maximizing the variational lower-bound ELBO
(x)BO = (ggh)))) log p(x.h)
q[hix)] we can approximate the marginal log-likelihood log pe (0) by
evaluating (ELBO(x) (x) via Monte-Carlo estimations or importance sampling techniques (Burda et 2016). Thus we have our NFK formulation as
Knnx z) (Vx, Vi)) Ux ~ VØLELBO(X)
Vx (diag(I) t)Ux
where x, z E X.I = Ex-po(x) [UxU]]]
Flow-based Models, Auto-Regressiy Models. For generative models with explicit exact data
density modeling Pe (x), we can simply apply the classical Fisher kernel formulation in Sec.
3.1.2 SUPERVISED NPK
In the supervised learning setting, we consider conditional probabilistic models po x) p(y | x; 0). In particular, we focus classification problems where the conditional prob- ability is parameterized by a softmax function over the logits output fxx 0): po x) exp(f*(x: 0))/ Σy cxp(fx (x: 0)), where y is discrete label and f² 0) denotes y-th logit output. We then borrow the idea from JEM (Grathwohl et al. 2020) and write out joint ergy function term over yx.y) as Ex yields exactly the same conditional probability, at the same time leading to free energy function:
-xx(x; 0). is easy to see that joint energy
4

---

Published as a conference paper at ICLR 2022
Algorithm 1 Baseline method: compute low-rank NFK feature emhedding
Input dataset D; pre-train NN model f(x; 0); NFK feature dimensionality k; test data input x" Output low-rank NFK feature embedding enn:(x))
2: compute kernel Gram matrix = VV
1: compute Fisher vector for all data
V = [Vx] IERRRRA Nx
E R N
3: compute truncated eigen-decomposition K = (dd "T, & E
4: kernel function evaluations between x" and all data examples K((x*, X) [K(x*, x1)ره==
5: obtain ey (x*) E IR via Eq. 5 and Eg.
E(x; 0) = - log Ey exp(f* 0)). It essentaall reframes a conditional distribution over y given
x to an induced unconditional distribution over x, while maintaining the same conditional probability p x). This allows to write out the NFK formulation as:
Kxnx(x, z) = (V2i Vie) Vx= = (Iiag(I) +jUx
Ux = Σ poly x)Vaf - Ex Exexpo()) Σ poly x) Vo
(3)
where I = (x) [U-U and po (x) is the normalized density corresponding to the free energy Eg. which could be sampled from via Markov chain Monte Carlo (MCMC) algorithm. In this work, we use empirical data distribution as practical approximation.
3.2 NFK WITH LOW-RANK A APPROXIMATION
Fisher vector Vx is the linear feature embedding pxx) given by NFK Kofk (x, z) (Vx, Va) for neural network model fx; 0). However, straightforward application of NFK by using Vz as feature representation suffers from scalability issue, since Vx E R shares same dimensionality as the number of parameters (0. with that in mind that | can be tremendously large considering the scale of modern neural networks, it is unfortunately infeasible to directly leverage V as feature representation.
Low-Rank Structure of NFK. Motivated by the Manifold Hypothesis of Data that it is widely believed that real world high dimensional data lives low dimensional manifold (Roweis and Saul 2000: Rifai et al. 221 la b). we investigate the structure of NFKs and present empirical evidence that NFKs of good models have low-rank spectral structure. Firstly, we start by examining supervised learning models. We study the spectrum structure of the empirical NFK of trained neural networks with different architectures. We trained a LeNet-5 (LeCun et al. network by minimizing binary cross entropy loss, and then compute the eigen-decomposition of the NFK Gram matrix. We show the explained ratio plot in Figure 2 We see that the spectrum of
1998 CNN and a 3-layer MLP
CNN NTK concentrates on fewer large eigenvalues, thus exhibiting a lower effective-rank structure compared to the MLP, which can be explained by the fact that CNN has better model inductive bias for image data domain. For unsupervised learning models, we trained small unconditional DCGAN (Radford et al., 2016) model on MNIST dataset. We compare the results of fully trained model against a randomly initialized model in Fig. Fig.E (note the logarithm scale of the x-axsi). Remarkably, the trained model demonstrates an extreme degree of low-rankness that top 100 principle components explain over 99.5% of the overall variance, where 100 is two orders of magnitude smaller than both number of examples and number of parameters in the discriminator. We include more experimental results and discussions in appendix due to the space constraints.
Efficient Low-Rank Approximation of NFK. The theoretical insights and empirical evidence presented above hint at a natural solution to address the challenge of digh-iimensionality of V E 0 we can turn to seek a low-rank approximation to the NFK. According to the Mer- cer's theorem (Mercer 1909). for positive definite kernel K(x, 2) = (p(x), pz()) we have X(x, 2) = Ein (x) x, E X. where {(>,, ox)} are the cigenvalues and eigenfunc- tions of the kernel IC, with respect to the integral operator J p(z)/C(xx 2) (z) dz = (x). The linear feature embedding representation (xx) can thus be constructed from the orthonormal eigen- basis {Ax,, xb)} as xxx) = VA1án (x), V2202(x) [.]
(VA,o,(x) i= 1,..., xo. To obtain a low-rank approximation, we only keep top-k largest eigen-hasis {(),, Φi)) ordered by corresponding eigenvalues A; to form the low-rank k-dimensional feature embedding e(x) RR k < N, k << 0
e(x) = (Virgi()) (o02(x).... vinda(x)] = =[VTrc((x))
i= :=1,...,, 1....,k
(4)
5

---

Published as a conference paper at ICLR 2022
Algorithm 2 Our proposed method: compute low-rank NFK feature embedding
1: V ed (aiiag(2)) P P E R using power iteration methods via JVP/VJP evaluations
2: compute K (x, X) p as Kddaag E P. via JVP evaluation
3: obtain enik (x*) E R via Eq. 5 and Eq.
By applying our proposed NFK formulation Kufk to pre-trained neural network model ['(; 0), we can obtain compact low-dimensional feature representation enfk (x) E IR* in this way. We call the low-rank NFK feature embedding.
We then illustrate how to estimate the eigenvalues and eigenfunctions NFK Kn from data, Given dataset D, the Gram matrix K E IR of kernel I. is defined as K(x;, Xj) = K(x,,))) We
use X = xx | to denote the matrix of data examples, and use (X( E RN to denote the concatenated vector of evaluating i-th eigenfunction @s at all data examples. Then by performing cigen-decomposition of the Gram matrix K = Ødia ag (A) the i-th eigenvector &; E I and eigenvalue A, can be seen unbiased estimation of the i-th eigenfunction o; and eigenvalue À of the kernel IC, evaluated at training data examples X, @ (X) x V A = A, Based on these estimations, we can thus approximate the cigenfunction o via the integral operator by Monte-Carlo estimation with empirical data distribution,
N N
Xx0x(x)= / p(z))C(x, 2)0:(2) dz se Eramin K(x, X5)o Exx,
1
R
xxx,,,)
(5)
Given new test data example x", we can now approximate the eigenfunction evaluation (x(x)) by the projection of kernel function evaluation results centered on training data examples K((x,, X)
[C(x*,,))];" x onto the i-th cigenvector 4 of kernel Gram matrix K. We adopt this method as the baseline approach for low-rank approximation, and present the baseline algorithm description in Alg.
However, due to the fact that it demands explicit computation and manipulation of the Fisher vector matrix V E Nx and the Gram matrix K E IR N in Alg. | straightforward application of the baseline approach, as well as other off-the-shelf classical kernel approximation (Williams and Seeger 2000 Rahimi and Recht 2007) and SVD methods (Halko et al. (01T). are practically infeasible to scale to larger-scale machine learning settings, where both the number of data examples N and the number of model parameters 00 can be extremely large.
To tackle the posed scalability issue, we propose a novel highly efficient and scalable algorithm for computing low-rank approximation of NFK. Given dataset 'D and model f(x; 0), We aim compute the truncated SVD of the Fisher vector matrix V Φdi ag (EEP ,P R Based on the idea of power methods (Golub and Van der Vorst 2000 Bathe 1971) for finding leading top eigenvectors, we
VV
start from random vector Vo and iteratively construct the sequence V ||VVTV]]' By leveraging
the special structure of V that it can be obtained from the Jacobian matrix Je (X) E N up to element-win wise linear transformation under the NFK formulation in Sec. 3 we can decompose each iterative step into a Jacobian Vector Product (JVP) and Vector Jacobian Product (VJP). With techniques, we can evaluate both JVP and VJP efficiently, which only requires the same order of computational costs of one vanilla backward-pase and forward-pass of neural networks respectively. With computed truncated SVD results, we can approximate the projection term in Eq. 5 by K. (x, X) +; V क; R Vzdiag(E,) which is again in the JVP form so that we can pre-compute and store the truncated SVD results and evaluate the eigenfunction of any test data example via one efficient JVP forward-pass. We describe our proposed algorithm briefly in Alg. 2
modern
EXPERIMENTS
In this section, we evaluate NFK in the following settings. We first evaluate the proposed low-rank kernel approximation algorithm (Sec. 3.2).. in terms of both approximation accuracy and running time efficiency. Next, we evaluate NFK on various representation learning tasks both supervised, semi-supervised and unsupervised learning settings.

---

Published as a conference paper at ICLR 2022
OIFARO Discriminator, Single
Single
Lusilite
Loddinc
432
102
19
2'
21
21
2.5
21
25
24
23
234
#548A oramplea
#dila examples
WRL--60-2
130
Approcimation enoor
12
122
compo www..talinnddnn..
1.0
60-
0.6
20
21
232
242
2.30
215
215
#3883 043/Faee
Figure 3: Top row: Running time efficiency evaluation for truncated SVD algorithm on single GPU. We vary the number of data examples used, shown in z-axis. y-axis denotes the wall-clock running time (in seconds). Red crosses mark the cases when it is no longer possible for the baseline method to obtain the results in an affordable waiting time and memory consumption. Bottom left: Running time costs with different number of GPUs used in our distributed SVD implementation. Bottom right: Approximation errors (in blue) of our proposed implementation for each eigenmode (in descending order of eigenvalues), v.s. the explained variance (in red). Best viewed in color.
4.1 QUALITY AND EFFICIENCY OF LOW-RANK NFK APPROXIMATIONS
We implement our proposed low-rank kernel approximation algorithm in Jax (Bradbury al. 2018 with distributed multi-GPU parallel computation support. For the baseline methods for comparison. we first compute the full kernel Gram matrix using the neural sarrraltaaggete. (Novak et al. 2020) library, and then use sk ea decompos tion. Truncatedsvi to obtain the truncated SVD results. All model and algorithm hyper-parameters parameters are included in the Appendix.
Computational Costs. We start by comparing running time costs of computing top NFK eigen- vectors via truncated SVD. We use two models for the comparison, a DCGAN-like GAN model in (Zhai al. 2019) and Wide ResNet (WRN) with 40 layers and 2 times wider than original network (denoted as WRN-40-2). Please see appendix for the detailed description of hhper-parameteer We observed that our proposed algorithm could achieve nearly linear time scaling, while the baseline method would not be able to handle more than 214 data examples as the memory usage and time complexity are high to afford. We also see in Fig. 3 that by utilizing multi-GPU parallelism, we achieved further speed-up which scales almost linearly with the number of GPUs. We emphasize that given the number of desired eigenvectors, the time complexity of our method scales linearly with the number of data examples and the demanded memory usage remains constant with adequate data batch size, since explicit computation and storage of the full kernel matrix never needed.
Approximation accuracy. We investigate the approximation error of our proposed low-rank approxi- mation method. Since we did not introduce any additional approximations, our method shares the same approximation error bound with the existing randomized SVD algorithm (Martinsson and Tropp 2020 Halko et al., 2001I) and would only expect differences compared to the baseline randomized SVD algorithm up to numerical errors. To evaluate the quality of the low-rank kernel approximation, we use L and compute its full NFK Gram matrix on MNIST dataset. Please see appendix for detailed hyper-pant paramear setups. We show in Fig, 3 the approximation emrors of top-128 eigenvalues along with corresponding explained variances. We obtain less than le 8 absolute error and less than le 7 relative error in top eigen-modes which explains most of the data.
7

---

Published as a conference paper at ICLR 2022
Table I: CIFAR- 10 accuracies of linear evaluation on top of representations learned with unsupervised and self-superviser methods. NFK-122 denotes the 128 dimensional embeddings from the low-rank approximation of the NFK (ie AFV), Remarkably, we can use 128 dimensions to exactly recover the performance of the 5.9M dimensional Fisher Vectors.
Model
Acc
Category
#Features
Examplar CNN Dosovitskiy et al.
2015
84.3
Unsupervised
BiGAN (Mu al. 2020
70.5
Unsupervised
RotNet Lincar Gidaris ct al.
2018
81.8
Self- Soff-Supee
25K
AET Linear (Zhang el al. 2019
83.3
Self-Supervised
25K
VAE(Mu et al. 2020)
61.5
Unsupervised
VAE-NFK- (ours)
63.2
Unsupervised
128
MNE-NFK-2566 (ours)
68.7
Unsupervised
256
GAN-Supervised
92.7
Supervised
GAN-Activations
65.3
Unsupervised
GAN-AFV (Zhai et al
2019)
89.1
Unsupervised
5.9M
GAN-AFV (re-implementation) Zhai et al. 2019
89.8
Unsupervised
5.9M
GAN-NFK- 128d (ours)
89.8
Unsupervised
128
GAN-NFK-2566 (ours)
89.8
Unsupervised
256
4.2 LOW-RANK NFK EMBEDDING AS DATA REPRESENTATIONS
In this section we evaluate NFK to answer the following: Q1. In line with the question raised in Sec. [ how does our proposed low-rank NFK embedding differ from the intermediate layer activations for data representation? Q2. How does the low-rank NFK embedding compare to simply using gradients (Jacobians) as data representation? Q3. To what extent can the low-rank NFK embedding preserve the information in full Fisher vector? Q4. Does the NFK embedding representation lead to better generalization performance
terms of better sample efficiency and faster adaptation? We conduct comparative studies on different tasks understand the NFK embedding and present empirical observations to answer these questions in following sections.
NFK Representations from Unsupervised Generative Models. In order to examine the effective- ness of the low-rank NFK embeddings as data representations in unsupervised learning setting, we consider GANs and VAEs as representative generative models and compute the low-rank NFK embed- dings. Then we adopt the linear probing protocol by training a linear classifier on top of the obtained embeddings and report the classification performance quantify the quality of NFK data representa- tion. For GANs, we use the same pretrained GAN model from ZZhai et al.) 2019) and reimplemented the AFV baseline. For VAEs, we follow the same model architecture proposed in (Child 2020). We then apply the proposed truncated SVD algorithm with 10 power iterations to obtain the 256 dimen- sional embedding via projection. We present our results on CIFAR 10 (Krizhevsky et al. 2009a) in Table. 1 We use GAN-NFX-122 128d (GAN-NFK- K- 2-25 6d) to denote the NFK embedding obtained from using 128 (top-256) eigenvectors in our GAN model. Our VAE models (VAE-NF 1 8d and VAE-NFK NFK-256d) follow the same notations. For VAE baselines, the method proposed in (Mu et al. 2020) combines both gradients features and activations- first consider using intermediate layer activations only
features into one linear model, denoted
as VAE in the table. For GAN baselines, as data representation, referred to as the GAN-Act vvat ions model. We then consider using full Fisher vector as representation, namely using the normalized gradients w.r.t all model parameters as features, denoted as the GAN-AFV model as proposed in (Zhai et al. (201)). Moreover, we also compare our results against training whole neural network using data labels in a supervised learning way, denoted as GAN- -NN--Suppiiiee model.
As shown in Table. D. by contrasting against the baseline GAN-AFV from GAN-Act ivat ions, as well as validation in recent works (Zhai et al., 2019 Mu et al. 2020). gradients provide additional useful information beyond layer activations based features. However, it would be impractical to use all gradients or full Fisher vector as representation when scaling up to large-scale neural network models. For example, VAE (Child) 2020) has ~ 40M parameters, would not be possible to apply the baseline methods directly. Our proposed low-rank NFK embedding approach addressed this challenge by building low-dim vector representation from efficient kernel approximation algorithm, making it possible to utilize all model parameters' gradients information by embedding it into low- dim vector, e.g. 256-dimensional embedding in VAE-NEK- 256 from ~ 40M parameters. As our low-rank NFK embedding is obtained by linear projections of full Fisher vectors, it naturally provides answers for Q2 that the NFK embedding can be viewed as a compact yet informative representation containing information from all gradients. We see from Table. that by using top-128 eigenvectors,
8

---

Published as a conference paper at ICLR 2022
Table 2: Error rates of semi-superrvisd classification CIFARIO and SVHN, varying labels from 500 to 4000. NFK-1280 yields extremely competitive performance, compared to other more sophisticated baselines, Mixup (Zhang al. 2018), , VAT (Miyato et al. 2019), Meahhra Tarvainen and Valpola 2017). Mix.Match (Berthelot et al. 2219). Improved GANimalimans et al. (0166) all are jointly learns with labels. Also note that the architecture used by MixMatch yields 4. 4.1%% supervised learning error rate, which is much stronger than our supervised baseline (7.3%).
Model
Category
CIFAR-10 10
SVHN
500
1000
2000
4000
500
1000
2000
4000
Mixup
Joint
36.17
25.72
18.14
13.15
29.62
16.79
10.47
7.96
VAT
Joint
26.11
18.68
14.40
11.05
7.44
5.98
4.85
4.20
MeanTeacher
Joint
47.32
42.01
17.32
12.17
6.45
3.82
3.75
3.51
MixMatch
Joint
9.65
7.75
7.03
6.24
3.64
3.27
3.04
2.89
Improved GAN
Joint
19.22
17.25
15.59
18.44
8.11
6.16
NFK-128d (ours)
Pretrained
20.68
14.77
13.82
12.95
8.74
4.47
3.82
3.19
Table 3: Supervised knowledge distillation results (classification accuracy on test dataset) on CIFARI0
against baseline methods KI) (Hinton et al., 2015), FitNet (Romero et al. (0155), AT (Zagoruyko and
Komodakis 2017). NST (Huang and Wang 2017).
VID-I (Ahn et al. 2019), numbers are from (Ahn
al. 2019
Teacher
Student
KD
FitNet
AT
NST
VID-I
NFKD (ours)
ACC
94.26
90.72
91.27
90.64
91.60
91.16
91.85
92.42
the low-rank NFK embedding is able to recover the performance of full ~ M--dimensionn Fisher vector, which provides positive evidence for Q3 that we can preserve most of the useful information in Fisher vector by taking advantage of the low-rank structure of NFK spectrum.
NFK Representations for Semi-Supervised Learning. We then test the low-rank NFK embeddings in the semi-supervised learning setting. Following the standard semi-supervised learning benchmark settings (Berthelot et al. 2019; Miyato et al., 2019; Laine and Aila 2017; Sajjadi et al. 22066] Tarvainen and Valpola| 2017). we evaluate our method on CIFAR- TO (Krizhevsky et al. (2009a) and SVHN datasets (Krizhessky et al. 2009b). We treat most of the dataset as unlabeled data and use few examples as labeled data. We use same GAN model as the unsupervised learning setting above, and compute top-128 eigenvectors using training dataset (labeled and unlabeled) to derive the 128-dimensional NFK embedding. Then we only use the labeled data to train a linear classifier on top of the NFK embedding features, denoted the WFK-1288 model. We vary the number of labeled training examples and report the results in Table. 2 in comparison with other baseline methods. We see that NFK-1288 8d achieves very competitive performance. On CIFAR-10 10, NFK-128d is only outperformed by the state-of-the-ant semi-supervised learning algorithm MixMatch (Berthelot al. 2019), which also uses a stronger architecture than ours. The results on SVHN are mixed though NFK- 28d is competitive with the top performing approaches. The results demonstrated the effectiveness of NFK embeddings from unsupervised generative models in semi supersupe learning, showing promising sample efficiency for Q4.
NFK Representations for Knowledge Distillation. We next test the effectiveness of using low-rank NFK embedding for knowledge distillation in the supervised learning setting. We include more details of the distillation method the Appendix. Our experiments are conducted on CIFAR10, with a teacher set as the WRN-40-2 model and student being WRN-16-1, After training the teacher, we compute the low-rank approximation of NFK of the teacher model, using top-20 eigenvectors. We include more details about the distillation method setup in the Appendix. Our results are reported in Table, 3 We see that our method achieves superior results compared to other competitive baseline knowledge distillation methods, which mainly use the logits and activations from teacher network as distillation target.
CONCLUSIONS
In this work, we propose novel principled approach to representation extraction from pre-trained neural network models. We introduce NFK by extending the Fisher kernel to neural networks in both unsupervised learning and superevised learning settings, and propose novel low-rank kernel approximation algorithm, which allows us to obtain a compact feature representation in a highly efficient and scalable way.

---

Published as a conference paper at ICLR 2022
REFERENCES
Tommi S. Jaakkola and David Haussler. Exploiting generative models in discriminative classifiers. In Michael J. Kearns, Sara A. Solla, and David A. Cohn, editors, Advances in Neural Information Processing Systems 11, (NIPS Conference, Denver, Colorado, USA, November 30 December
5. 1998], pages 487-493. The MIT Press, 1998. URL httL:ttp::2 /paptess/ppppprss.ee/ n wtwpscccpppppcp.......eeeepeepee
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016. doi: 10.1 10.1109/CVP..2066.00 htLhttps: /ww. www.oogg...o..... 10.
Geoffrey F. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, a03/0503.0 2015. UtLbtt:: wwt/ w10.aa./1551.... 0203.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv: 1406. (e...14066660066 1 2014. URL /ttpsss/aarxx..orggaaa////eeeeee 2661
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:
6114
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ily Sutskever. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event. volume 119 of Proceedings of Machine Learning Research, pages 1691-1703. PMLR, 2020a. URL http:: URR::tt::ppp/peed.gg
mlr. ptl.s/res//vee httpsss/v1l///hhaa chen2 html
Fangzhou Mu, Yingyu Liang, and Yin Li. Gradients as features for deep representation learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL BKecaeHKDS
net/ forum?
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicold Cesa-Bianchi, and Roman Garnett, editors, Advances in Neu- ral Information Processing Systems 31: Anmual Conference on Neural Information Pro- cessing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 8580 8589, 2018. URL Ba44e.113eeebbbbcccce2222222---2
newrips. cc/ wttpcc/ppp/pp...... 8/ahh// .html
Thomas Hofmann, Bernhard Schölkopf, and Alexander Smola. Kernel methods machine learning. The annals of statistics, pages 1171-1220, 2008
Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pages 29-53. Springer, 1996.
Christopher K. I. Williams. Computing with infinite networks. In Michael Mozer, Michael I. Jordan, and Thomas Petsche, editors, Advances in Neural Information Processing Systems NIPS. Denver, CO, USA. December 2-5, 1996, pages 295-301. MIT Press, 1996. URL http: Sipapers.nis..o wttpppppccccaaapaeaaee///ecceeea httppppssnipssccccppp///133/ ingg://ttmtttttt-iii-ee with wnwinite-netttee..
Nicolas Le Roux and Yoshua Bengio. Continuous neural networks. In Marina Meila and Xiaotong Shen, editors, Proceedings of the Eleventh International Conference Artificial Intelligence and Statistics, AISTATS 2007, San Juan, Puerto Rico, March 21-24, 2007, volume 2 of JMLR Proceedings, pages 404-411. 1...411..M.... JMLR.org, 2007. URL http:::::::ppeeeeeeeeoott
122/eroox072 html
Tamir Hazan and Tommi Jaakkola. Steps toward deep kernel methods from infinite neural networks. CoRR,
2015. URL
05133
10

---

Published as a conference paper at ICLR 2022
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz. Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver. BC. Canada, April May 3. 2018. Conference Track Proceedings. OpenReview.net, 2018. URL htt.hhtts:///gtt httb::://ppenrvviie
ew.. ew net. forum? id=
B1 1IAA-M 1EAM--O2 07
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver. BC. Canada, April 30 May 3, 2018, Conference net/ forum? ?
Track Proceedings. OpenReview.net, 2018. URL wttpttt:ss/ppeeeeeeeeeeee
B1l-nGgWC-
Lin Chen and Sheng Xu. Deep neural tangent kernel and laplace kernel have the same RKHS. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL. htttps://ppererri
net/ / forum? ? d=
vk rKQgr
Amnon Geifmann Abhay Kumar Yadav, Yoni Kasten, Meirav Galun, David W. Jacobs, and Ronen Basri. On the similarity between the laplace and neural tangent kernels. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
hntp://.r/poreeeennn.n..e....... newrips....p.... wwt.rriis.cccpeeee oat.:aeeee.e.e111ooaAaaarrarae.a html
Abst
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need understand kernel learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th Interna- tional Conference Machine Learning. ICML 2018, Stockholmsmüssann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 540-548. PMLR, 2018. UR//attp://prcceeiiing
html
Behrooz Ghorbani. Song Mei, Theodor Misiakiewicz. and Andrea Montanari. When do neural networks outperform kernel methods? In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information P Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL
nwurips..pp.... wwtpuurris..ccpppppeeeeee cc/paper/ AaeetB--bbttaacc .ttm
2020/hash 9df2255ad62aeeccc
Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, edi- tors, Advances in Neural Information Processing Systems 29: Anmual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2253-2261, 2016. URL http:tt://pooceeedmmmmee neurips httpp/ptc...ppcpppp////////p///e wwtpppppppp.cc0cc 6/hash/ 2.bea7bba2.0 42e1122416666 6b 2122a22122221117/8822424444444 Abst html
Florent Perronnin and Christopher R. Dance. Fisher kernels on visual vocabularies for image categorization. In 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007), 18-23 June 2007, Minneapolis, Minnesota, USA. IEEE Computer Society, 2007. doi: 10. 00.11099/VPR.2207.3888600 2007. 383266
org/:..ooggl000 wtt.://o....../1/....... 1109/CVPR
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Workshop Truck Proceedings, 2015. URL httpt::p://ii....t///// org/ http///bs/11000 11t...1111111
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Marii-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference Machine Learning. ICML 2016, New York City, NY, USA. June 19-24. 2016. volume 48 of JMLR Workshop and Conference Proceedings, pages 1747-1756 JMLR.org, 2016. URL http:: http://e//pocceeeiiee ngs /..p.eess//////rrr htmll

---

Published as a conference paper at ICLR 2022
Ruslan Salakhutdinov. Deep learning. In Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang, and Rayid Ghani, editors. The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '4, New York, NY, USA August 24 27, 2014, page 1973. ACM, 2014. doi: 10.1 00.1145//622333226600000 URL https: //doi .tw.dooooorrooooooooo 1r.11...115//
2623330 2630809
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pictro Perona. Task2vec: Task embedding for meta-lcarning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 04429-6438 IEEE, 2019. doi: 10.1109//CCV22219..0055
URL https: org 10. 1100/ CCV 2019. 20199000511
Zihang Dai, Amjad Almahairi, Philip Bachman, Eduard H. Hovy, and Aaron C. Courville. Calibrating energy-based generative adversarial networks. In 5th International Conference on Learning Representations, ICIR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. net /wwt/eetttoaaee ftr.mmormmm/mm d- d SyxeqhP 911
OpenReview.net, 2017. URL hattps://oppezeeeee
Shuangfei Zhai, Walter Talbott, Carlos Guestrin, and Joshua M. Susskind. Adversarial fisher vec- tors for unsupervised representation learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buee Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 11156-11166, 2019. URL https: httpt:p:/pppcccceeiiiee 7 fal.cctfa72222222355522444444342
www.diiiinnnn................... cttccc//ppape wtc/paper/200 100011/hahh
.html
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN is secretly energy-based model and you should use discriminator driven latent sampling. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tier Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL htppstts////popcciiinaa neurips...........mm.m cc htcpc//ppcc..cc 2022r/20207 h2211//0222217733333333356661113
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representa- fions, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Truck Proceedings, 2016. URL
orw. abs 1509 00519
Will Grathwohl, Kuan-Chich Wang. Henn-k Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpp:://operreeeeee net/ Eorum? Brrum?1d==HxzxONEEI
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290(55000:2333-3388 2000.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In Lise Getoor and Tobias Scheffer, editors, Proceedings of the 28th International Conference
Machine Learning. ICML 2011, Bellevue, Washington, USA, June 28 July 2, 201 I, pages 833-840. Omnipress, 2011a. URL https://icct wttpcc//////peeeeeeee
455 wtw__mcmlppppppp paper.
Salah Rifai, Yann N. Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The man- ifold tangent classifier. In John Shawe-Taylor, Richard Zemel, Peter L. Bartlett, Fer- nando C. N. Pereira, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Sys- tems 2011. Proceedings of meeting held 12-14 December 2011, Granada, Spain, pages 2294 2302, 2011b. URL
12

---

Published as a conference paper at ICLR 2022
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-25 2324, 1998.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: /arxiv..vv...////////a////aaaa org/ abs
org/abs////aoooooooo/////////bar 1511
06434
J Mercer. Functions ofpositive and negativetypeand theircommection with the theory ofintegral equations. Philos. Trinsdictions Rogyal Soc, 209:4 415, 1909.
Christopher K. Williams and Matthias W. Seeger. Using the nyström method to speed up kernel machines. In Todd K. Leen, Thomas G. Dietterich, and Volker Tresp, editors, Advances in Neural Information Processing Systems 13, Papers from Neu- ral Information Processing Systems (NIPS) 2000, Denver, CO, USA, pages 682 688. MIT Press, 2000. URL Abst html 19de Oadbaa b2eel
Abst
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-Firs! Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007, pages 1177-
1184. Curran Associates, Inc., 2007. URL https::/prooeednngg. nwwrips........... cc/paper/ 2007 / htt.hhashhh/// 01 3a00 03 92ee fe www.ffefff1fff55eee -aa11118rra555555ttaatttttt 0
Abst htm
Nathan Halko, Per-Gunnar Martinsson, and Joel A. Tropp. Finding structure with randomness:
Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev,, 53(2): 217-288, 2011 1. doi: 10.1137/090771866 / wtt.///.....r.11.11/00//00011110
Gene H Golub and Henk A Van der Vorst. Eigenvalue computation in the 20th century. Journal of Computational and Applied Mathematics, 123(1-2):35-66 65, 2000.
Klaus-Jürgen Bathe. Solution methods for large generalized eigenvalue problems in structural engineering. National Technical Information Service, US Department of Commerce, 1971.
James Bradbury. Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:/ /egg/gttuuuue com/ google/
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dicksted
and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020, URL Sk1D9yrFPS
net/ forum?
Per-Gunnar Martinsson and Joel A. Tropp. Randomized numerical linear algebra: Foundations and algorithms. Acta Numer. 29:403-572, 2020. doi: 10.1077/50022429220000022 URL
https::/doi.orr httppp///oo...rr/000000////22222
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence, 3809) 2015.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by pre- dicting image rotations. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC. Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httptttss//oeeeeve.eeee
net
id- -orump2d--S14422110
Liheng Zhang, Guo-Jun Qi. Liqiang Wang, and Jiebo Luo. AET AED: unsupervised representation learning by auto-encoding transformations rather than data. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20,
13

---

Published as a conference paper at ICLR 2022
2019. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVVR.221..0226.. URL
http: ctt...asss..eeee.cccoooo
CVPR_ 2019/htm1/2hang A
AED Unsupervised Representation Learnineaannn by Auto-En.codia Transformal hatprioos___at__cP_aae_e pa_____019__ppp______ .ttml
Rewon Child. Very deep vaes generalize autoregressive and can outperform them on images.
CoRR, 1../2001..105000 2020.
/2011/11.b 10650
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009a.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC. Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
net httts//frmmmmddrrrrr.. Ddp1-Rb
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 1199):199-11993, 2019. doi: 10.1 10.1109/PAMM..218.22555 7/doi.org 10. 1109//199M.... 2018.285882.
URL https
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- con- sistency targets improve semi-supervised deep learning results. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio. Hanna M. Wallach, Rob Fergus. S. V. N. Vishwanathan, and Roman Gameet, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 1195-1204, 2017. URL httppp::
neuri i ps. wttpcc//pppp.ee wtcppaper/20:101 // html
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference
Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5050-5060, 2019. URL https: / http:///ccccceeeeeeeeeee neurips cc/ httpp/ppppppp2/// 2aaper/2019/hahh html
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spuin, pages hatp://
2016. URL /RL/httpps///ppceeedingg new.ieeeripppp cc/ .tt.acttttmml.lmmm..
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
net/
Fooum?iddBJJooffg
Mehdi Sajiadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg. Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1163-1171, 2016. URL httpps://pooceeddnnneee/ B06E3000644004333322222222eeee22 -atpr/eeeeeet70o---tttttttteeeee Abstract html
meeurrps... wwwpuriisscccpaapee wwcpppppr//////ccc 201 6/hash/
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009b.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun, editors, 3rd
14

---

Published as a conference paper at ICLR 2022
International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. (RL/bttp://axxiit org/ abs 1112/ 6550
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Pro- ceedings. OpenReview.net, 2017. URL lattps://openreviccr net net/Forum?id-kks ajex
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. CoRR, abs/1 70700129, 2017. URL htttttttp///xii.///eee
abs///b////// at/////s/////// 01219
Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, and Zhenwen Dai.
Variational information distillation for knowledge transfer. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 9163-9171. Computer Vision Foundation / IEEE, 2019. doi: 10.1 Ptt//////V.......00.999 URL httpp://openaaceess..t thecv com/ http://ootetttceeeee Sennt_CVPR__00 NTPP_2219/httm/Aet Variational Information Distillation for Knowledge Transfer.
CVPR 2019 paper.html
Aristide Baratin, Thomas George, César Laurent, R. Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pages 2269-2277 PMLR, 2021, URL http::/pprcceeddgg/
mlr
html
Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21 2020
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment
explain generalization in kernel regression and infinitely wide neural networks, 2020.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pitore-Antoine Manzagol, and
Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research. 11(12), 2010.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03744 2018. URL httptttpp////aeeeee /awpiv..ivv/.////////aa//aaaa org/ergg////ooo. httpp//////xxx...o//bb////////// 03748
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 2020b. URL httt:://procee:e w.r.ppress/vll c9e.c/2c/chnnncpc html
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9726-9735. IEEE, 2020. doi: 10. CVPR42600. 2020. 00975
URL https: doi orgoor////ooo www.oorggg...... 1109
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL tttLtttt/////eeeeiiii net/Eorum?iddBBae 30ccX
Ben Poole, Sherjil Ozair, Aäron van den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9- June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research pages 5171-51880 PMLR, 2019. URL htl.http:/// http::://pocceeee///tt/ ngs ml wwwpppasssssp.. peress/v9770 htwp://///p//////////// 9a html
15

---

Published as a conference paper at ICLR 2022
Ruixiang Zhang, Masanori Koyama, and Katsuhiko Ishiguro. Learning structured latent factors from dependent datac generative model framework from information theoretic perspective. In Proceed- ings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1 11111-1111 11152. PMLR, 2020.
zhang 0m. html
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Zoubin Ghahra- mani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference Neural In- formation Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2654 2662, 2014. URL httpps::/pprocceett//tttt/
neurips httpccccpppppppeee / 4/hash/ 8881:77755551177666777766-6---63
Frederick Tung and Greg Mori. munilartty-ressrvingg knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 1365-1374. IEEE, 2019. doi: 10.1109/ICCV220190001 URL
aittps://dos .or.ioorgg.oo.. 100 1et.11199//CC 2019 00145
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. URL
net./ forum?rrrmm
SkepBJrtvS
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-276. 1998.
Ryo Karakida and Kazuki Osawa. Understanding approximate fisher information for fast convergence of natural gradient descent wide neural networks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, edi- tors, Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020. NeurIPS 2020. December 6-12, 2020, virtual, 2020. URL httpps://proceeednggs htw.roccseddngss..er.i... neuriss. ccs.cc/ppe wwtp.cccpcp.///////////.. be220055000000a22224445000e22220
.ttll
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016. URL http:/ //Wp://www bmva. www.orgggooo... htwp////////////////////////eee/ 2014 6/ wwwpaapapss//pp/pppp www.ppsspppperr..... wndexxndexxx.//// html
16

---

Published as a conference paper at ICLR 2022
A EXTENDED PRELIMINARIES
We extend Sec. 2 to introduce additional technical background and related work.
Kernel methods in Deep Learning. Popularized by the NTK work Dacot et al. (2018), there has been great interests in the deep learning community around the kernel view of neural networks. In particular, several works have studied the low-rank structure of the NTK, including (Baratin et al.) 2021, Papyan 2020 Canatar et al., 2020), which demonstrate that empirical NTK demonstrates low-rankness and that encourages better generalization theoretically. Our low-rank analysis of NFK shares a similar flavor, but generalizes across supervised and unsupervised learning settings. Besides, make explicit effort in proposing efficient implementation of the low-rank approximation, and demonstrate strong empirical performances.
Unsupervised/self supervised representation learning. Unsupervised representation learning an old idea in deep learning. A large body of work is dedicated to designing better learning objectives (self supervised learning), including denoising (Vincent al. 2010), contrastive learning (Oord et al. 2018:Chen Chen et al. 2020b; He et al., (020), mutual information based methods (Hjelm et al. 2019 Poole et al., 2019 Zhang et al. (020) and other "pretext tasks" Ting and Tian (2020). Our attempt falls into the same category of unsupervised representation learning, but differs in that we instead focus on effectively extracting information from standard probabilistic model. This makes our effort
orthogonal to many of the related works, and can be easily plugged into different family of models.
Knowledge Distillation. Knowledge distillation (KD) is generally concerned about the problem of supervising a student model with teacher model (Hinton et al., 2015 Ba and Caruanal 2014). The general form of KD is to directly match the statistics of one or a few layers (default is the logits). Various works have studied the layer selection (Romero et al. 2015) loss function design aspects (Ahn et al. 2019). J. More closely related to our work is efforts that consider the second order statistics between examples, including (Tung and Mori, 2019 Tian et al., 2020). NFKD differs in that represent the teacher's ' knowledge in the kernel space, which is tlyectly tied to the kernel interpretation of neural networks which introduces different inductive biases than layerwise representations.
Neural Tangent Kernel. Recent advancements in the understanding of neural networks have shed light on the connection between neural network training and kernel methods. In (Jacot et al., 2018). it is shown that one can use the Neural Tangent Kernel (NTK) to characterize the full training of a neural network using a kernel. Let f(0: x) denote a neural network function with parameters O The NTK is defined as follows:
(6) where Pe is the probability distribution of the initialization of 0. (Jacot et al., 2018) further demonstrates that in the large width regime, a neural network undergoing training under gradi- ent descent essentially evolves as linear model. Let 0g denote the parameter values at ini- tialization. To determine how the function ((,;; evolves, we may naively taylor expand the output around oy: fi+1 ( R (((x)) f(((x))) [0 0z). As the weight updates are given by 0.11 A: fe(0;;)) - n EN (x,
Knti(x, 2) = Ea-p, (Vo](0;)), (0)(0)
x Em 1 Vo, La (xx), hence we have fe+1(0 w
The significance of the NTK stems from two observations. 1) When suitably initialized, the NTK converges to a limit kernel when the width tends to infinity limwidh--oc Kmutex, z; 0n) = Knet 2). 2) In that limit, the NTK remains frozen in its limit state throughout training.
B ON THE CONNECTIONS BETWEEN NFK AND NTK
In Sec 3.1 we showed that our definition of NFK in the supervised learning setting bares great similarity to the NTK. We provide more discussion here on the connections between NFK and NTK.
For the L2 regression loss function, the empirical fisher information reduces to I = N EN, Vele (effxx))eef(()
Note that the fisher information matrix I is give by covariance matrix of J. while the NTK matrix is defined as the Gram matrix of J. where J is the Jacobian matrix, implying they share the same spectrum, and that the NTK and the NFK share the same cigenvectors. The addition of I in the definition of K,, can be seen as a form of conditioning, facilitating fast convergence in all directions spanned by J.
17

---

Published as a conference paper at ICLR 2022
Algorithm 3 Our proposed method: compute low-rank NFK feature embedding
1: V
@diag(E))P" P
R via sruncated_sváck Sa. topk=K,
kernel_type="NFKK
2:
3: compute K (xx X) P, Rs Vidiag E, P. via JVP evaluation
4: obtain enfk (x*) E Re via Eq. and Eq.
Equation 3 also has immediate connections to NTK. In NTK, the kernel (x, x) E RNXN is matrix which measures the dot product of Jacobian for every pair of logits. The NFK, on the other hand, reduces the Jacobian fe for each example x to a single vector of dimension (i.c., size of 0), weighted by the predicted probability of each class po(y(x). The other no- table difference between NFK and NTK is the subtractive and normalize factors, represented by (x)~pe(x)) Σy ( f (x') and I, respectively. This distinction is related to the difference between Natural Gradient Descent (Amari 1998; Karakida and Osawa 2020) and gradient descent. In nutshell, our definition of NFK in the supervised learning setting can be considered as a reduced version of NTK, with proper normalization. These properties make NFK much more scalable w.r.t. the number of classes, and also less sensitive to the scale model's 's parameters.
To better this, we define an "unnormalized" version of NFK (x, [Ey (o V [x)]T Ey (o V) (x). It is easy to see that K,,, has the same rank as the original NFK K,, as I-1 is full rank by definition. We can then further rewrite it
(x, = =E poly x))o(y(3)Voofl Vola
ΣΣ poly x) R)K (x, x)
In words, the unnormalized version of NFK can be considered as a reduction of NTK, where the weights of each element is weighte by the predicted probability for the respective class. If we further assume that the model of interest is well trained, as is often the case in knowledge distillation, we approximate the K., as KCy .Y" (x,x), x), where y arg maxy pe (y x) and likewise for y*. This suggests that the unnormalized NFK can roughly viewd as a downsampled version of NTK. As result, we expect the unnormalized NFK (and hence the NFK) to exhibit similar low rank properties as demonstrated in the NTK literature.
On the low-rank structure of NTK. Consider the NTK Gram matrix Katk NNN of some network Given the dataset [x; J (for simplicity we assume a scalar output) and its eigen decompo- sition Kntx Σx= u Let f R. denote the concatenated outputs. Under GD in the linear regime, the outputs ft evolves according to:
Vi, uz -241 - VCC
VYC.
The updates f++1 - fe projected onto the bases of the kernel therefore converge at different speeds, determined by the eigenvalues { Intuitively, good kernel-data alignment means that the C is spanned by a few eigenvectors with large corresponding eigenvalues, speeding up convergence and promoting generalization.
C NEURAL FISHER KERNEL WITH LOW-RANK APPROXIMATION
C.I NEURAL FISHER KERNEL FORMULATION
We provide detailed derivations of the various NFK formulations presented in Section. 3
NFK for Energy-based Models. Consider an Energy-based Model (EBM) Po(x) exp( E(x
2:0)
where F(x) is the energy function parametrized by O and Z(0) f eexp(- -x(:= 0)) dx is the
18

---

Published as a conference paper at ICLR 2022
Algorithm 4 truncated _sddd Truncated SVD Algorithm for Low-rank Kernel Approximation. Comments are based on NTK for simplicity.
Input Dataset X = {x
Input Neural network model fe Input Kernel type kernel. NFK or NTK
Input Low-rank embedding size K
Input Number of power iterations I. = 10
Input Number of over samples U = 10
Output Truncated SVD of Jacobian (X) R Q
1: U == = + +U
b Size of augmented set of vectors in power iterations
2: Draw random matrix Q NR
3: Ω =matrixx =mmarrixjaacoaaaaaa htap_ii_jjaccaanaaaa__ producct (fe. X, S2 Kernell)
b f = Je(X)s e M
4: for step
do
5:
Ω
X, Q kernel) X, ft kernel)
D Ω J0 (X) E NXU
6: Ω bttiriix_jaccbia_pprooooe
b Ω Jo(X)s E R
n ====_decompons ===__decompsiiionn(
8: end for
9: B
kerne)))))
B JJ (X)Ω NNU
P P.E, Q T))
11: P -PP
12: Keep top rank-K vectors to obtain the truncated results Pk: Σk, Q
Return Pk, Σx, Q2
Algorithm jacoobann_matrix_ppooo
Input Neural network model fe
Input Input data X E Bx where batch size
Input Input matrix M
Input Kernel type kernel, NFK or NTK
Output J (X)M for NTK, Fisher--ector--matiiieeeeee VoJ (X)M for NFK
1: jmp_fn fn jaxx vwwp()ax.
2: P ==jmp_fn(fe M)
3: if kernel "NFK" then
4: P diag(Z) : (P (P- Zg M)
5: end if
6: Return P
partition function, we could apply the Fisher kernel formulation to derive the Fisher score Ux as
Ux Vo log
V log (exp(-F(xx:)) 0))] Vo log Z(0)
(xE(x;0 0) - Va log 2(0)
VVF 0) Ex~ (x) V log (op0-FE(xx:00
Ex~ro(x) EoF(x; (xF(x;
Then we can obtain the FIM I and the Fisher vector Vx from above results, shown as below
I Ex-mo(x) [
(10)
Vx I b
NFK for GANs. As introduced in Section 3 we consider the EBM formulation of GANs. Given pre-trained GAN model, D(x: 0) to denote the output of the discriminator D, and use G(h) to denote the output generator G given latent code h ~ p(h). Then we have the energy-function defined as Exx; 0) D(x; 0). Based the NFK formulation for EBMs, simply substitute
19

---

Published as a conference paper at ICLR 2022
Algorithm 6 mmttrix_jacobiannPe
Input Neural network model f
Input Input data X E Bx where B is batch size
Input Input matrix M
Input Kernel type kernel, NFK or NTK
Output Jg ( for NTK, wwwprr-eeoooooraiipooooeee V (X)M for NFK
1: mip_fn fn jax. vmap(jax.
2: P P= ==jjp__ae fnnfff(ooo M)
3: if kernel= = "NFK" then
4:
P diag(I) & PP-ZeN
5: end if
6: Return P
Exx; 0) Dx(x; 0) into Eq. 9 and Eq. 10 and derive the NFK formulation for GANs as below
Ux VxD(x; 0) En VoD(G(
I = En-p(h) (U(())
(I1)
V (I) U)Ux
(x, (Vx, Vi)
Note that we use diagonal approximation of FIM throughout this work for the consideration of scalability, Also, since the generator of GANs is trained to match the distribution induced by the discriminator's EBM from the perspective variational training for GANs, we could use the samples generated by the generator to approximate
(x), which is reflected in above formulation.
NFK for VAEs, Flow-based Models, Augr-Regres Models. For models including VAEs, Flow- based Models, Auto-Rette Regresseee Models, where explicit approximate density estimation is available, we can simply apply the classical Fisher kernel formulation as introduced in the main text.
NFK for Supervised Learning Models. In the supervised learning setting, we consider conditional probabilistic models ( x) = p(y x; 0). In particular, we focus on classification problems where the conditional probability is parameterized by a softmax function over the logits output f(x; 0): po x) = exp( ( (x; 0)) Ey exp( (x; 0)), where y is a discrete label and f> 0) denotes y-th logit output. We then borrow the idea from JEM (Grathwohll et al. 2020) and write out joint energy function term (x, y) as Fx(x, y;
f (x; 0). It is easy yields exactly the same conditional probability, at the same time leading to a free energy function:
that joint energy
E(x: 0) llg Σ exp(fV(xx:0)
(12)
VoE(x; Σ poly x) [x
Based on the NFK formulation for EBMs, simply substitute above results into Eq. 9 and Eq. and derive the NFK formulation for GANs as below
Ux Σ poly (x; (xo Σ poly (x
(13)
I = Ex-pp(x) [U2U]]
Vxx = (diag(I)) b)Ux Kan (x, (Vx, Vz)
(14)
C.2 EFFICIENT LOW-RANK NFK/NTK APPROXIMATION VIA TRUNCATED SVD
We provide mode details on experimental observations on the low-rank structure of NFK and the low-rank kernel approximation algorithm here.
20

---

Published as a conference paper at ICLR 2022
3
9
2
8
3
3
9
2
8
33
7
5
5
6
6
1
9
8
7
5
3
6
/
9
8
5
7
7
2
9
2
3
57
5
7
2
9
2
4
4
3
7
5
4 3
4
4 6
4
0
9
1
4
O /
4114970
8
2 1
7 >
7
4
0 8
2 7
2
8
2
9
4
5
4
9 9
2
4
5
7
4 9
8
2
9
4
2
3
8
3
8
2
9
4
2
3
8
(a) NFK
3
8
6
y
2
3
8
6
/
4
2 4
9
5
7
4
9
3 4
9
5
7
4
9
9
3
O
2
/
2
9
0
3
0
2
/
3
9
O
1
9
5
4
2
2 6
/ 1
1
9
5
4
7
2
/
1
1
4
5
b
5
9
1
4
5
6
5
6
9
9
3
0
9
1
9
3
0
2
9
d
5
2
6
0
9
4
4
0
5
6
O
9
4
0
5
3
4
8
9
0
5
3
4
8
9
O 3
(b) PCA
Figure 4: Inverting a DCGAN with 100d NFK embeddings (a), compared with image reconstruction with 100d PCA embeddings (b). In either case, the left plot corresponds to real test images and the right corresponds to the reconstructions. Note that NFK embeddings care capable of inverting GAN by producing high quality semantic reconstructions. With PCA, embeddings with the same dimensionality produces more blurry reconstructions (thus less semantic).
Low-Rank Structure of NFK.
For supervised learning models, we trained a LeNet-5 (LeCun et al., 1998) CNN and 3-layer MLP network by minimizing binary
entropy loss, and then compute the eigen-decomposition of the NFK Gram matrix. For unsupervised learning models, we trained small unconditional DC- GAN (Rafford al. 2016) model on MNIST dataset. We deliberately selected small discriminator, which consists of 17K parameters. Because of the relatively wwwdiimenioonality of Ô in the discrimi- nator, we were able to directly compute the Fisher Vectors for a random subset of the training dataset. We then performed standard SVD on the gathered Fisher Vector matrix, and examined the spectrum Eim X
statistics. In particular, we plot the explained variance ration quantity, defined as Simn X where A; is the i-th singular value. In addition, have also visualized the top principle components, by showing example images which have the largest projections on each component in Fig.
Furthermore, conducted GAN inversion experiment. We start by sampling set latent variables from the generator's ' prior h E phh), and get a set of generated example {x;}, x; G(h;), 1. 1,... n.. We then apply Algorithm 2 on the generated example (x,) obtain their NFK embeddings {e(x:)) and we set the dimension of both h and to 100. We now have a compositional mapping that reads h x e. We then learn a linear mapping W 100x from (e( (h,)) to (h,) auto encoder from a regular GAN, with compositional mapping of x - - -> &, where the reconstruction of an input The reconstructions shown in Figure (a). Interestingly, the 100d SVD embedding gives rise to a qualitatively faithful reconstruction on real images. n contrast, a PCA embedding with the same dimension gives much more blurry reconstructions (eg., noise in the background), as shown in Figure (b). This a good indication that the 100d embedding captures most of the information about an input example.
by minimizing Σ h; (h;G( In doing so, we have constructed
Power iteration of NEK as JVP/VJP evaluations. Our proposed algorithm is based on the Power method Golub and Van der Vorst 2000): Bathe (1971) for finding the leading top eigenvectors of
21

---

Published as a conference paper at ICLR 2022
1.00
0.95
esplained varlance ratio
0.90
0.85
0.80
0.75
0.70
0.65
random model
0.60
10a
10a
10'
10a
10*
number of components
Figure The low-rankness of the NFK on DCGAN trained on MNIST. For a trained model, the first 100 principle components of the Fisher Vector matrix explains 99.5% of all variances. An untrained model with the same architecture on the other hand, demonstrates a much lower degree of low-rankness.
9
4
2 1 5
Figure 6: Images with the largest projections on the first five principle components. Each row corresponds to principle component.
the real symmetric matrix. Starting from a random vector vo drawn from a rotationally invariant distribution and normalize it to unit norm |oll = 1, the power method iteratively constructs the sequence V [[ up to g power iterations. Given the special structure of K that it's Gram matrix of the Jacobian matrix Jo E RDNN to evaluate Kv, in each power iteration step we need evaluate Jo x)) Jo (TXX,,,, which can be decomposed as: (i) evaluating z, (X(X)vr and then (ii) Kv, = Jo x) z;. Note that when K is in the form of NTK of neural networks, step (i) of evaluating z, Vector-Jacobian saa---Jaaobia-ooduuee -roddct (VJP) and step (ii) is Jacobian www--eeeetoooouuuee (JVP). With the help of which only requires the same order of computational costs of one backward- pass and forward pasd-dpssssss of neural networks respectively. In this way, we can reduce the Kernel matrix vector product operation each power iteration step to one VJP evaluation and one JVP evaluation, without the need to computing and storing the Jacobian matrix and kernel matrix explicitly.
Kv,
diffettiidiiferetttation techniques, we can evaluate both JVP and VJP efficiently.
As introduced in Section. 3.2 we include detailed algorithm description here, from Algorithm, 3 to Algorithm. 6 In Algorithm. we show the algorithm compute the low-rank NFK em- bedding, which can be used as data representations. In Algorithm. 4 we present our proposed autommtii-diiffreettttion based truncated SVD algorithm for kernel approximation. Note that in Algorithm. [ and 6, we only need to follow Equation. 3 to pre-compute the model distribution statisties, Zo (xx(')((')) Ey po [of (x'), and FIM I Ex/opo(x) (Ux/ UZ). We adopt
22

---

Published as a conference paper at ICLR 2022
0.9
0.7
0.5
0.3
25
5.0
7.5
10.0
12.5
15.0
17.5
ombooding 2ize
Figure 7: Linear probing accuracy on CIFARIO with different number of principle components in embedding. We use our proposed low-rank approximation method to compute the embedding from the teacher model on CIFARIO for knowledge distillation.
the EBM formulation of classifier fx(x) then replace the Jacobian matrix Jx(X) with the Fisher vector matrix (X( = -dia(Z) (X) Zo). Note that our proposed algorithm also readily applicable to empirical NTK via replacing the FIM by the identity matrix.
D EXPERIMENTS SETUP
D.1 QUALITY ANDEFFICIENCY OFLOW-RANKNER APPROXIMATIONS
Experiments Computational Cost. We randomly sample N {2 : 7 k S 16} data examples from CIFAR-I0 dataset, and compute top-32 eigenvectors of the NFK Gram matrix NxR) by truncated SVD. We use same number of power iterations (10) in baseline method and our algorithm. We show in Fig. the rumning time of SVD for both methods terms of number of data examples N.
Experiments on Approximation Accuracy. We randomly sample 10000 examples and compute top-128 eigenvalues using both baseline methods and our proposed algorithm. Specifically, we compute the full Gram matrix and perform degeopsccomooiiioonn to obtain baseline results. For our implementation, we run 10 power iterations in randomized SVD.
D.2 NEURAL FISHER KERNEL DISTILLATION
With the efficient low-rank approximation NFK, one can immediately obtain a compact representa- tion of the kernel. Namely, each example can be represented as a l dimension vector. Essentially, have achieved a form of kernel distillation, which is a useful technique on its own.
Q generalized form for teacher student styled knowledge distillation (KD), as in (Hinton et al. 2015). In standard KD, one obtain teacher network (c.g., deep model) and use it to train student network (e.g., shallow model) with distillation loss in the following format:
Furthermore,
a ( (((])))))))(
+
(15)
where Lets is standard classification loss (e.g., cross entropy) and L, is teacher loss which forces the student network's output fa to match that of the teacher fc. We propose a straightforward extension of KD with NFK, where modify the loss function to be:
Lana(x, = a (x) (1-
(x(,,(x), Qx(x)):
(16)
where (x) denotes the dimensional embedding from the SVD of teacher NFK, for example x. h, prediction head from the student, and L, is overloaded to denote suitable loss (e.g., (2 distance or cosine distance). Equation 16 essenntily uses the low dimension embedding of the teacher's NFK supervision, inplace of the teacher's logits. There arguable benefits of using Iinfkd over Lgd-
23

---

Published as a conference paper at ICLR 2022
For example, when the number of classes is small, the logit layer contains very little extra information (measured in number of bits) than the label alone, whereas Qi can still provide dense supervision to the student.
For the Neural Fisher Kernel Distillation (NFKD) experiments, we adopt the WideResNet- 40x2 (Zagoruyko and Komodakis, 2016 neural network as the teacher model. We train another WideResnet with 16 layers as the student model, and keep the width unchanged. We run 10 power iterations compute the SVD approximation of the NFK of the teacher model, obtain the top-20 eigenvectors and eigenvalues. Then we train the student model with the additional NFKD distillation loss using mini-bate' stochastic gradient descent, with 0.9 momentum, for 250 epochs. The initial learning rate begins at 0.1 and we decay the learning rate by 0.1 at 150-th epoch and decay again by 0.1 200-th epoch. We also show the linear probing accuracies on CIFAR 10 by using different number of embedding dimensions in Figure. 7
24


Reviews:
[{"review_id": "L8vtMt4N7NK", "reviewer": "Reviewer_945A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is interesting and technically sound, its novelty comes from two perspectives. First, it considers to represent/modify neural networks from the view of kernels. The proposed NFK serves as a unified kernel for both supervised and unsupervised learning models. Besides, the authors take advantage of the low-rank structure of NFK to avoid directly handling unmanageably high dimensional vectors.\n\n1. The effectiveness of the proposed NFK is mainly evaluated based on CIFAR-10 dataset, how is the performance of the proposed method on other datasets?\n\n2. The authors mention some representative unsupervised learning tasks such as denoising. Have the authors tried to combine the proposed NFK with the simple CNN for image denoising/super-resolution/enhancement?\n\n3. Some figures could be used to help to better explain the framework of the proposed method and also its effectiveness.\n\n4. Some terms such as VAEs and GANs should be defined,  for readers without too much background knowledge may be confused.", "summary_of_the_paper": "The paper investigates the representation of modern neural networks from the perspective of kernels by extracting features from pre-trained network models. The authors show the effectiveness of the proposed neural fisher kernel (NFK) for both unsupervised and supervised learning tasks. A low-rank approximation strategy of NFK is adopted to reduce computational burdens.", "summary_of_the_review": "The paper is technically interesting, but the authors should carefully revise their experimental section to address the concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G5qDBZUrsLH", "reviewer": "Reviewer_z7Tb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a formulation for the NFK for unsupervised (GANs and VAEs) and supervised neural networks. The kernels are defined as inner products of the Fisher vectors (derived from the Fisher score) computed from pre-trained neural networks. \n\nStrength:\n\n•\tNice idea to see extensions of kernels, here the Fisher kernel, to neural networks.\n\n•\tSimilar formulations for supervised and unsupervised settings.\n\nWeaknesses:\n\n•\tMore motivation and derivations of the different Fisher scores $U_x$ (for GANs, VAEs, supervised) would be beneficial for understanding better the paper. \n\n•\tMore discussions on the non-diagonal version of the Fisher information matrix models (see comments below) would be beneficial.\n\n•\tDiscussion on the dependence of the quality of the NFK embedding on the quality of the pre-trained neural network.\n\n•\tDiscussion on the choice of the low-dimensional embedding dimensionality $k$\n\nGeneral comments:\n\n•\tUsing the diagonal of the Fisher information matrix (FIM) seems desirable from a computational reason, however a natural question is what happens if one tries to use the full matrix. Given the size of the parameters $\\theta$ in a neural network, estimating the whole matrix would indeed be extremely computationally expensive, but by discarding them, one loses significant information. Could the authors comment on that? Is the diag of FIM related to other known concepts in statistics? Does using only the diagonal imply that the off-diagonal elements are zero meaning the parameters are orthogonal? How does this affect the results and interpretation?\n\n•\tCould the authors give the derivation of $U_x$ in eq (1) for GANs (I see part of the derivation is in the paper by Zhai et al, 2019)? For VAEs and supervised case, FIM $\\mathcal{I}$ is the inner product using $U_x$, but for GANs it acts in the output space of the generator. Why is this the case (some explanations are given in the original paper but would be helpful to discuss this a bit more here)? The derivation of eq (4) would also be useful.\n\n•\tLow-rank structure of NFK and Alg 1: How does one choose the feature dimensionality $k$? Many methods that rely on kernels and manifold learning make the assumption of low-dimensionality/low-rankness and show that a small number of eigenfunctions is sufficient to reconstruct the input data. How is this different for NFK? The way I understand “low rank” is that the data has its own rank, which is low, and could potentially be learned. However here the authors input the dimensionality/rank $k$ which might be or not close to the true rank in real applications.\n\n•\tHow does this work relate to the work of Belkin et al (2018) – “To understand deep learning we need to understand kernel learning”, where the authors look at other kernels (Laplacian and Gaussian)?\n\n•\tCould the approach be used for neural networks that are not pre-trained, as the neural tangent kernel NTK?\n\n•\tThe experimental results are nice, however the focus on computation is not so relevant given that it only uses the diagonal of the Fisher information matrix. Comparisons using the whole matrix would also be needed. What error is used in Table 3 (MSE, MAE, RMSE)? The goal of the paper is to present a method for supervised and unsupervised settings, however in the results an example on semi-supervised is also presented. I wonder if the examples on semi-supervised and knowledge distillation could leave room to improve the supervised and unsupervised settings discussions, and potentially be moved to the Appendix?\n\nOther comments:\n\n•\tPlease update reference (Jaakkola et al): year, conference, also there should be no “et al” there are only two authors\n\n•\tDoesn’t, don’t, won’t, it’s , etc -> does not, do not, would not, it is\n\n•\tBoth the concepts of “data representation” and “feature representation” are used. Do they always refer to the same thing? If yes, would be good to specify that.\n\n•\tExpression of $K_{fisher}$ => second $U$ should be subscript $z$ not $x$? \n\n•\t“FIM defined as the variance of the score …” -> the FIM matrix is defined between all pairs of parameters $\\theta_i$ and $\\theta_j$, so it should be a covariance?\n\n•\tAppendix Fig 3: Not sure I fully understand this example. Could one try the reconstruction of the digits using a simple method, such as PCA using the first 100 principal components as a baseline?\n\n•\tNot familiar with the “Fisher vector” terminology, except in image classification and the “Adversarial Fisher vector” from Zhai et al, 2019. Are there other references?\n", "summary_of_the_paper": "The paper introduces a new kernel, the neural Fisher kernel (NFK), to learn data representations for generative models in both unsupervised and supervised settings. NFK extends the Fisher kernel to neural networks, and a low-rank approximation of NFK is used to scale to large datasets. ", "summary_of_the_review": "The paper presents interesting ideas of extending kernels, here the Fisher kernel, to neural networks. I find that the paper would need to make a stronger connection with existing literature (eg, Belkin et al, 2018) and give more motivation/discussion/intuition on the derivation of the different $U_x$s. My grading suggestion is a 6, as I believe the paper presents nice ideas but would require more work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "42GLACAc_O", "reviewer": "Reviewer_9i1k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\nThe general idea of using the linear space associated with the kernel feature map for extracting data representation is interesting. Simplifying the representation through low-rank approximation of the kernel matrix is an interesting technique and may have a good impact on representation learning. Using the power method for computing the low-rank approximation through automatic differentiation could have applications that go beyond the proposed approach.\n\nweaknesses:\nThe authors should specify better what is new and what has been already done. For example, it looks like using Fisher vectors in representation learning and approximating kernels through truncated SVD are not new ideas. Also, the authors could have spent some more words to outline the main differences and advantages of using the Fisher kernel instead of the standard Tangent kernel. It is not clear if the main contribution of the paper is the extension to unsupervised learning or the proposed low-rank approximation.   \n\nquestions:\n- Is the approximation of the fisher tangent kernel the key novelty of the paper? If so, what is the technical challenge compared with approximating other kernels?\n- I like the idea of using automatically computed Jacobians in the power method. Is this a new idea? Can this be applied to the standard tangent kernel?\n- The experiments do not compare the proposed approach with non-fisher similar methods. Would it be possible to run a quantitative comparison with the tangent kernel (at least for the supervised case)? \n- Are there computational gains associated with using the Fisher kernel for its low-rank approximations to train a model or is the proposed method only suitable for obtaining more intuitive representations?\n- Is Montecarlo Sampling needed in the proposed method? And what are the computational cost and feasibility limits of that power iteration approach?\n- In the third line of the last paragraph of p2, Ux I Ux should be Ux I Uz, right?\n", "summary_of_the_paper": "The paper proposes a new strategy for extracting compact and intuitive representations of the data from a neural network. The approach exploits the feature map associated with the kernel representation of a neural network. In particular, the authors consider the Neural Fisher\nKernel and propose a series of linear approximations to make their approach scalable.\n", "summary_of_the_review": "An interesting idea but the authors should specify better what is new and what comes from existing work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper which elicited much discussion.
The paper proposes to extract features from pre-trained networks through Kernel functions. It develops the idea of Fisher kernels for Neural networks calling it NFK. The methodology applies to both supervised and un-supervised setting. 
The paper shows that proposed kernel has low rank structure and serves as the basis for developing an algorithm for computing the kernel on large datasets. The idea of extending Fisher kernels, their efficient computation, and investigating their usage in both Supervised and Unsupervised are some of the key strengths of the paper.
The reviewers though appreciative suggested (1) several new experiments, (2) inclusion of more background work related to Power method 
and (3)  have more technical discussion clarifying the contributions related to background. 
The author(s) during rebuttal tried to incorporate most of the suggestions in the revised draft. 

Since there was consensus on the novelty, the detailed discussions, and the results of the additional experiments, one could potentially accept this paper if there is space. The results will be interesting will be those who investigate the interplay of kernel methods and Deep Networks.
Decision: Accept (Poster)