Some Practical Concerns and Solutions for Using Pretrained Representation in Industrial Systems
Authors: Da Xu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Representation Learning
Stability
Generalization
Convergence
Predictability
Industry Application

Abstract: Deep learning has dramatically changed the way data scientists and engineers craft features -- the once tedious process of measuring and constructing can now be achieved by training learnable representations. Recent work shows pretraining can endow representations with relevant signals, and in practice they are often used as feature vectors in downstream models. In real-world production, however, we have encountered key problems that cannot be justified by existing knowledge. They raise concerns that the naive use of pretrained representation as feature vector could lead to unwarranted and suboptimal solution.
Our investigation reveals critical insights into the gap of uniform convergence for analyzing pretrained representations, their stochastic nature under gradient descent optimization, what does model convergence means to them, and how they might interact with downstream tasks. Inspired by our analysis, we explore a simple yet powerful approach that can refine pretrained representation in multiple ways, which we call "Featurizing Pretrained Representations". Our work balances practicality and rigor, and contributes to both applied and theoretical research of representation learning. 

Under review as a conference paper at ICLR 2023
SOME PRACTICAL CONCERNS AND SOLUTIONS FOR USING PRETRAINED REPRESENTATION IN INDUS- TRIAL SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning has dramatically changed the way data scientists and engineers craft features the once tedious process of measuring and constructing now be achieved by training learnable representations. Recent work shows pretraining can endow representations with relevant signals, and in practice they are often used as feature vectors in downstream models. In real-world production, however, we have encountered key problems that cannot be justified by existing knowl- edge. They raise concerns that the naive use of pretrained representation as feature vector could lead to unwarranted and suboptimal solution. Our investigation re- veals critical insights into the gap of uniform convergence for analyzing pretrained representations, their stochastic nature under gradient descent optimization, what does model convergence means to them, and how they might interact with down- stream tasks. Inspired by our analysis, we explore a simple yet powerful approach that can refine pretrained representation in multiple ways, which we call Featur- izing Pretrained Representations. Our work balances practicality and rigor, and contributes both applied and theoretical research of representation learning.
INTRODUCTION
The ability of neural networks to learn predictive feature representation from data has always fas- cinated practitioners and researchers (Bengio al. 2013). The learnt representations, proved reliable, can potentially renovate the entire life cycle and workflow of industrial machine learn- ing. Behind reliability are the three core principles for extracting information from data, namely stability, predictability, and computability (Yu) 2020). These three principles can not only justify the practical value of learnt representation, but also lead to the efficiency, interpretability. and repro- ducibility that are cherished in real-world production. Since pretrained representations are optimized to align with the given task, intuitively, they should satisfy all three principles in a reasonable set- ting. However, when productionizing an automated pipeline for pretrained representations in an
(a) Th ree rondemly sampled oreara ned (b) embedd rgg mntrall pedi, with ther mean and stander devation
The snesraining loss and zecurzee as the training progress each refependent run. Their trspectoree almoet overtap.
Disoribution of the telemonc sei emp rical vortance for pretrained representation as triiing progressen.
Distribuiion of emcinica veriance (d)
for the output of pretriiin ing motel (oner product betweer emcedding! under diferrer. baden-dimenitin
Hi
0.99
epores
Pretin LoGs
Protr ACc
Density
Dersity
Rdp
0.15
30
25
35
0.00
0.55
0.50
6.75
1,00
1.5
1.5
0.00
0.03
0,02
0,33
2.04
5.05
Dimension
Epoch
Emptrical Varfance
Empirical Varience
Figure 1: Illustrating the stability issue of pretrained representation with MovieLens-Im. The details of the experiments are deferred to Appendix(F) The empirical variances computed from ten independent runs.
industrial system, we encountered key problems that cannot be justified by existing knowledge. In particular, while the daily refresh follows the same modelling and training configurations and uses essentially the same data downstream model owners reported unexpectedly high fluctuations in
Since the pretraining uses years of history data, the proportion of daily data is quite small.

---

Under review as a conference paper at ICLR 2023
performance when retraining their models. For illustration purpose, here we reproduce the issue using benchmark data, and take one further step where the pretraining is repeated on exactly the same data, under the same model configuration, training setup, and stopping criteria. We imple- ment ten independent runs to essentially generate the i.i.d versions of the pretrained representation.
We first visualize the dimension-we -iise empirical variances of the pre- trained representations, provided in Figure ... It is surprising to find out that while the pretraining losses almost converge to the same value in each run (Figure [p), there is such a high degree of uncertainty about the exact values of each dimension. Further, in Figure D. we observe that the uncertainty (empirical variance) of pretrained representation will increase as the pretraining progresses. In the downstream task where pre- trained representations are used as feature vectors (see the right figure), observe that the performance does fluctuate wildly from run to run. Since we use logistic regression as the downstream model, the fluctua- tion can only be caused by the instability of pretrained representations because we can effectively optimize the downstream model to global optimum. To demonstrate that the above phenomenon is not caused by using a specific model or data, we also experiment with a completely different pre- training model and benchmark data from from another domain. We perform the same analysis, and unfortunately the same issues persist (Figure A. in the Appendix).
The tae performmnze of semaged (across the runs protrained oap resercat
0.22
Meuratyd 0553
Existing deep learning theory, both the convergence and generalization results (we will discuss them more in Section D. can fail to explain why shall we expect pretrained representation to work well in a downstream task when their exact values are unstable. This is especially concerning for industrial systems as the issue can lead to unwarranted and suboptimal downstream solutions. We experienced this issue firsthand in production, so we motivated to crack the mysteries behind pre- trained representations, and understand if and how their stability can be improved without sacrificing predictability and computability. We summarize contributions below.
We provide a novel uniform convergence result for pretrained representations, which point out gaps that relate to the stability and predictability issues.
We break down and clarify the stability issue by revealing the stochastic nature of pretrained representation, the convergence of model output, and the stable and unstable components involved.
We investigate the interaction between pretrained representation and downstream tasks in both parametric and non-parametric settings, each revealing how predictability can benefit or suffer from stability (or instability) for particular usages of pretrained representations.
We discuss the idea of featurizing pretrained representation, and propose a highly practical so- lution that has nice guarantees and balances stability, predictability. and computability. We also examine its effectiveness in real-world experiments and online testings.
2 RELATED WORK
It is not until recent years that deep learning theory sees major progress. Zhang et al. (2016) observed that parameters of neural networks will stay close to initialization during training. At initialization, wide neural networks with random weights and biases are Gaussian processes, a phenomena first discussed by Neal (1995) and recently refined by Lee al. (2017); Yang (2019). However, they do not consider effect of optimization. The Neural Tangent Kernel provides a powerful tool to study the limiting convergence and generalization behavior of gradient descent optimization (Jacot et al. 2018 Allen-Zhu et al. 2019), but it sometimes fails to capture meaningful characteristics of practical neural networks (Woodworth et al. 2020 Fort et al. 2020). However, those works require parameters being close to initialization, in which useful representation learning would not take place.
Indeed, it has also caught to people's s attention that representation learning can go beyond the neural langent kernel regime (Yehudai & Shamir 2019 Wei et al.) 2019 Allen-Zhu & Li, 2019 Malach et al. 2021), among which a line of work connects the continuous time training dynamics with mean field approximation (Mei et al.) 2018; Sirignano & Spiliopoulos 2020), and another direction is to study the lazy training regime (Chizat et al, Gho9/Gorbam
Ghorbani et al., 2019)) where only the last layer of
neural network is trained. Unfortunately, their assumed training schemas all deviate from practical
2

---

Under review as a conference paper at ICLR 2023
representation learning. Still, part of our analysis in Section can be viewed practical discrete- time extension of the mnean-field method. Perhaps the most practical setting for studying pretrained representation is Arora et al. (2019), which analyzes the contrastive representation learning under a particular data generating mechanism. However, their results do not generalize to broader setting, and they cannot justify the stability issue of pretrained representation.
3 PRELIMINARIES
Notations. We use x E X' S Rd and E R to denote the raw feature and outcome, upper- case letters to denote random variables and measures, and bold-font letters denote matrices. Let : X' + IRd be the representation hypothesis, and f IRd - R be the prediction hypothesis. The hypothesis classes are given by H. and FF respectively. Denote by o the operator for function compo- sition, and é RxR- + [0, 1] the loss function. We assume é is I-Lipschitz without loss of generality. Then the risk for a pair of (h E H. E F) is given by: R(h, f) == (x [f[f o h(X). Y)], where P is a measure on (R, We also use Pa to denote the corresponding product measure for (X1, Y)).... (X,,
The one-layer pereptoyeerrerreeeprro (MLP) is perhaps the most fundamental representation learning model, given by: f h(z) = Go (Wz). Here, is the activation function, and W Rdoxd @ E RR We mention that adding the bias terms will not affect our analysis, so we drop them here for brevity. In practice. @ and W are often initialized
perceptron
scaled i.i.d Gaussian random variables that follow ((0, /d). We will use such as [W]; to denote the 2th row of a matrix. The popular contrastive representation learning can also be considered as special case of this configurationf
Define the shorthand g(x) == @o(Wx) x). A typical pretraining process involves optimizing the risk function defined for pretraining and extracting the hidden representation. The optimization is done via stochastic gradient descent (SGD), e.g. W(++1) ((') V VV y()), where is the learning rate. For convenience, we consider each mini-batch having one random sample, denoted by (x))), (t) that corresponds to the eth step.
Given a representation hypothesis we define: fn, := arg min 1/n ((f(())))))))))))) In the sequel, how well fr.n o h can generalize to new i.i.d sample of the downstream task is:
R(h)= := B(x,y)~PEP [e(sh.n n h(X), Y)),
where the second expectation Ep- is taken with respect to the downstream data (X,, Yi under- lying fr.n. Its empirical version is given by R, (h) == 1/n E, [f M().),
4 MAIN ANALYSIS
4.1 THE GAP OF UNIFORM CONVERGENCE FOR PRETRAINED REPRESENTATION
Suppose l and f are optimized jointly (end-to-end) via empirical risk mininization (ERM), which amounts to solving: arg minl 1/n Σ, {(ʃ o h(x;), v:). In this setting, the gemeralization behavior of the solution well-studied. In particular, using the notion of Gaussian (or Rademacher) c 3] the generalization eиror be bounded by (F H)/n + v(log 1/8)/n) with probability at least 1 - å (Barllett & Mendelson] 2002). This result, known as uniform convergence, especially appealing because it both includes proclem- aspects and applies all functions in the composite hypothesis class F H == {f o h: f f F. h E H).
Is it possible achieve a comparable result for pretrained representation? Perhaps the most ideal setting for uniform convergence to hold under pretrained representation is:
C1: the pretraining and downstream training will use the same data {(X,, Y. i... h arg min n En_ {(f h(X,), Y.). Jh., = arg minjer E. (()(..))
We can simply set 2i E TR" as one-hot encodings, and W, e E Re where they are allowed to coincide. Then we let h(as)) [W] or [+]: depending on the context. The activation hecomes the identity function,
and (m:), my) = log(1 (1 - a (T,,)))(( (or log ((h(,,)T)) (h,,))))),,)),,...)))))))))))))) with af-) being the Sigmoid function. We will use Gaussian complexity (()) here for some of its technical convenience. Then we let G. be the empirical Gaussian complexity. See Appendix[A for detail.
3

---

Under review as a conference paper at ICLR 2023
C2: they rely the same prediction function class F.
These two conditions essentially eliminate the confounding effects of model and data mismatch. Thus, if uniform convergence cannot hold in this setting, it is unlikely to serve more general use cases. We first summarize the common intuition behind why pretrained representation might work:
the pretraining objective, when well-designed, reasonably predicts the empirical downstream risk for Jh., (intuition I);
fhan' 's empirical downstream risk can be generalized to the true downstream risk (intuition 2).
These two intuitions have also been exemplified for contrastive representation learning in Arora et al. (2099) and its following work. Our main contribution here is to make the above intuitions rigorous, and reveal whether they are indeed sufficient for uniform convergence general settings.
Recall that, given the complete information min R(h. f'). We denote the representation hypothesis that achieves this minimum by h². Let h be given in C1. Then the generalization error is simply given by: R(h))mingE Following the standard derivation which decomposes the generalization error and takes the supre- mum to upper bound each term,
downstream task, the best
do is:
R(h, f)-
into terms that exactly characterize the above two intuitions. As we show in Appendix B holds that:
R(h) min BEH.JEF
R(h,
S suu P. Ru (1)) + sup Epm (B(x ( [(Innn ((X), - -Rn(hi]] remainder, h
{
h
where the first term sup,, {Ep= Rr, (h) - Rn (h)) exactly seeks to match intuition 1. and the second term can be further upper bounded using: PX [e(nn. o h()),)) - (h)] <
h(X).Y) - Rn (h)] }. which underlies intuition 2. The remainder terms can be bounded using standard concentration results.
sup, (Ex
However, we also spot a critical issue with the first term, and we first expand it for clarity:
Epm L Σ e(fnn
M(X.). ((n,n,,((X,,
Σ (fn. (()(n.oM(X))xx))) MXx))
sup
Notice that this is not the typical empirical process encountered in a standard generalization setting, and we show that its upper bound is actually given by (g,, (H) + Vlog /0) following the same procedure as Bartlett & Mendelson (2002). Compared with the standard generalization bound, here the slack term / /log 1/5 does not vanish as we increase n. Therefore, there exist gaps between common intuitions and achieving uniform convergence. Before we discuss the cause of the gaps and its implications, we first present the complete result as below.
Proposition 1. Let (-(()) be slightly modified Gaussian complexity term. Under the conditions and definitions C1 and C2, it holds with probability at least 1 that:
(n(H) Vn
(z)F)) / BIII(X) +
R(h)-
min REH,JEF
1og 1/5.
Vn
The proof is deferred to Appendix B Proposition I can be viewed as "no free lunch" result for using pretrained representation: even the most ideal setting we study here, uniform convergence cannot be expected for all representation hypothesis. The gap is that not for every h E H6 can the pretraining objective predictive of Sn,n 's empirical downstream risk. Imagine a entirely random h. Then both its pretraining objective and the empirical downstream risk of fi., may have high variances that do not scale with n. Thus the prediction will not concentrate whatsoever.
Takeaway. The implications of this gap two folds. Firstly, it does not suffice only study H. and the data distribution the statistical and algorithmic convergence properties of h(X) could be more relevant as they suggest its stability. Secondly, we cannot take the performance of f S/.n for granted,
at least not without understanding how h(X) interacts with the downstream learning and generates fi,n which ultimately relates to its predictability. Unfortunately, we find lack of discussion these two issues in existing literature, we will investigate them in the next two sections.
4

---

Under review as a conference paper at ICLR 2023
4.2 STOCHASTIC NATURE OF PRETRAINED REPRESENTATION, AND THE CONVERGENCE OF PRETRAINING MODEL
In this section, we reveal two important statistical and algorithmic properties of pretrained repre- sentation. We show that while they persist as random vectors during SGD optimization (as shown in Figure D. the output the pretraining model can be deterministic and converge to some optimal solution. Two contributing factors are scaled i.i.d initialization and the inductive bias of gradient descent. Our findings provide critical insight to the stability of pretrained representations.
We motivate our statistical analysis by deriving the optimization path of the one-layer MLP intro- duced in Section 3 For notation convenience, we introduce O and W as the rescaled version of O and W such that ( ( 1,2.d (0, 1). We let (g((()), y) be the derivative of the loss function and similarly for other functions. In contrast to the existing theoretical work that studies optimiza- tion path under gradient flow or infinitesimal learning rate, we fix the learning rate as a 1 to reflect real-world practice. The output dimension is also set to k = 1 without loss of generality.
In the first forward pass, since o (W(0)) has i.i.d coordinates, as d + co it holds that:
1 = d
+
B. EO o (w(()) (denote by gi (x(())))
where we use 6 (t)) to denote an i.i.d element (or row) of ( and ( As result, ll (g(o)()(()) (() also converges to the deterministic value (()) = l' (( (0) (x)())))()))
(o)). Then
in the first backward pass, the updated parameters will follow:
6 ( - (W z )
w(1)
w C
An important observation is that the updated parameters remain to be element-wise i.i.d. Conse- quently, the model output of the second forward pass will also converge to deterministic value:
(2) 0.5; E(0 c a (w(()) (w(0) c 2f 0 oo0oo (w
As we show in the following Proposition, the (statistical) convergence result will hold for any t, and there exists general iterative update rule for (x)). For some intuition, suppose a(-) is the identity function, then (+), (()) will simply be linear combinations of (0),, (
Proposition 2. For the one-layer MLP we consider, with the learning rate a = for any step I > 1, as d + oc, the model output g (z) will converge almost surely to gi (x) defined as follows:
g. (x) (C + +c(00(10)
with (cft+2) clt+1), ((+1),c((t)) c((tt)) ,,((t))c((tt)) (c cht) oh c(a) ) (00 cf0 cl c.o))
As a corollary, while the hidden representations will remain random vectors throughout the SGD process (which can be seem from the update rule):
(x) : o = -o
-o (w(+=)) (tt-
(wr(t-))) 1))
(x),, (x')) will nevertheless also converge to some deterministic value as d > xo The proof and detail are deferred to Appendix [ In Figure see that the statistical convergence of model output is indeed evident even with moderately small d, and its variance is by magnitudes smaller than the variance of the hidden representation o (W(())) xr) (see the x-axis of Figure and 1t).
On the other hand, the algorithmic convergence of model prediction has received considerable atten- tion. It has been shown that over weer-paammterized models will converge to minimum norm interpolants due to the inductive bias of gradient descent (Bartlett al. 2021 (oudry et al. 2018). For the sake of space, here we focus on their implications and leave the details to Appendix C Roughly speaking, among the many locally optimum solutions that interpolate the training data, gradient descent will converge to the one with the smallest norm, which usually has nice properties such as smoothness.
We let 9n be that particular solution such that lim,-sso g (x) = (z().. Since (h(*), ((() converge statistically to a deterministic value at every optimization step, immediately conclude that:
5

---

Under review as a conference paper at ICLR 2023
if gd) takes the form of (h( h such as in contrastive representation learning, the inner product between hidden representations also converge algorithmically to 9o 's prediction;
if g() = (h i.e. the last hidden layer is used as the representation, note that a necessary but
not sufficient condition for (x) - (x²') to be small is that (1) - (x'(())) is small
as well. Suppose ((9) are normalized, then upon the algorithmic convergence, (1 (x), (t),""((x))
are likely to be larger if z x' are close each other under go 's prediction.
Takeaway. The stochastic nature of h lim h(t and the (approximate) convergence of (h(x), h(x') under gradient descent reveal two important properties of pretrained representations:
1. Instability of h(x)): the exact position of Mar) in Rd is stochastic, depending on the initialization and the order of the pretraining data that is fed SGD;
Stability of (f(x), h(z')): the pairwise inner product of (f()), h(a")) converges (approximately) to value that is consistent with the minimum-nowe interpolant of the pretraining task.
These results will also play a crucial role in understanding how h can interact with the downstream learning, which we will study in the next section.
4.3 INTERACTION WITH DOWNSTREAM TASK
To be comprehensive, we consider both the parametric and non- per--arramtric set up for downstream task. Interestingly, they will reveal different aspects on the predictability of h.
Parametric setup. To eliminate the interference of label noise, consider the noiseless setting where the output of downstream task is generated by: y/ = f (E([(z)))) hxx) might be high-dimensional, we assume there is some sparsity f". The conditions below provide perhaps the easiest parametric setup for pretrained representations to perform well.
I,.... n. Because
C3: Let (h) == ()),
10 S q, and let the inputs h, := Eh(x;) be sampled from: NN0, oI where d% is the strength of the signal. We show previously that h is stochastic, we simply set h4 = h4 + Ei where e (O, If captures the variance of the pretrained representation.
Intuitively, since are i.i.d, it holds that E. [(((a,), M(r,)) = (h(zs), h(x;)) so recovering 6* should be less challenging. However, we show that the variance will again prohibit efficient learning. and the best fi., can do is controlled by a. 0z/ax- resentation. The result below takes the form of a minimax lower bound: an iiformation-thoorttiin quantity that characterize the inherent difficulty of a problem. Our proof (in Appendix D) is based on Le Cam's 's method that was previously used to prove a lower bound result under label noise (Raskutti) ct al. (011), which is very different from our setting.
notion of signal-to-noise ratio for pretrained rep-
Proposition 3. Under C3, holds with probability least 1/2 that:
inf sup 110 0- Z (o2/oR) qn qn log(d/))) 11
where inf, is taken with respect to any learning procedure that is based on (f(z;)) V.
Takeaway. The result in Proposition] is alarming because during pretraining, the variance of h(x:) might increase as more and more stochastic terms are being added (suggested by both the derivations in Section 4.ction/422aan and the empirical result in Figure The above lower bound shows the predictability of f((x)) can be compromised by its variance inherited from pretraining. This also explains the instability in downstream machine learning that we experienced during real-world production.
Non- non-parametric setup. Among the parameerriiic regression estimators, the Nadaraya- Watson (NW) estimator has received considerable attention due to its simplicity and effectiveness (Nadaraya 1964.. It can be thought of as a smoothing neighorr estimator under a weighting schema:
fn.n o h(xx) = Σ (y 2x), wa (x, xx) = K <(h(()) - hxx:))/

---

Under review as a conference paper at ICLR 2023
where K : R -> R is kernel, and is normalizing constant. Here, we omit the bandwidth pa- rameter for convenience. The Gaussian kernel K(2) & exp((||u|[) is a common choice, so when pretrained representations are normalized, it only depends on h via (h(z), (((=), h(x')) quantity according to the previous section. We effectively denote this kernel by K K( ((h(z), h(z)))) is well-understood that the generalization of a kernel support vector machine is controlled by the kernel-target alignment (Cristianini et al. 2001), i.e. (V. Kÿ), where y 1Bt....,yun and K., K[ ((((z,), h(z)))). We prove that this is also the case for NW estimator, with a simple result that does not resort to the concentration arguments. The proof is in Appendix [
more stable
Lemma 1. Under 0-1 loss, with probability at least 8, the risk of NW estimator satisfess:
R(f o h) $ 11-ve Vō. E || (1)
where the expectation is taken with respect to (X,Y) ~ P. (X) P.
Takeaway. Lemma I shows the predictability of h(z),, when expressed and measured through the more stable (h(x;), h(')) ), is strictly guaranteed. Therefore, using h(x) in downstream task in the form of h(xx) [e ((()), ...., (((())),(En)]]] can be beneficial, and it can be interpreted
representation of weights in the NW estimator. Further, h(x) contains all the pairwise relationship that can be more closely related to the pretraining objective. Note that h(x) can also be viewed as the compression of h(() because: [h(x;)], = (h)())).)) h(xx,(()))))))>)),)))))) ))). Nevertheless, h(z) and h(x) cannot be compared directly because they have different intrinsic dimensionality. In terms of computability,
h(x) E R" is also no compare to h(z) I computing h(x) itself can be non-trivial for large- scale applications. We aim to resolve these issues in the next section.
5 FEATURIZING PRETRAINED REPRESENTATION
Our next goal is to build on top of hxx)) features or representations that are comparable to T((x) in terms of stability and predictability, and have similar computability to h(x).
Suppose (a<x:))== are normalized. Then h(x;)) is simply the exponential of pairwise cosine distances between h(x;) and all the pretrained representations. Notice that the angle between any pair of (h(x;), h(z))) can be decomposed into their respective angle with a baseline direc- tion u E RRd /|ull2 = 1. When the set of baseline directions is rich enough, we can recover all the pairwise cosine distances in h(z;) using their angles with the baseline directions. Given U : [ui..... um] Rdxm the set of angles between h((;)) and U forms a measurement for the relative location of h(x) E Rd. We refer to such a measurement process as featurizing pretrained representation, as it is similar to how features are constructed by measuring experimental subjects.
While featurizing (xx) according to its geometrically property is an appealing solution, it is un- known how many baseline directions are needed to preserve the stability and predictability of Ti as well as the optimal way to choose those directions. Fortunately, the Bochner's Theorem (Loomis) 2013) from harmonic analysis lays a solid foundation for selecting the directions and providing approximation and learning guarantees. Also, the resulting measurements will coincide with the random Fourier feature (Rahimi & Recht, 2007 Liu et al.) (02T) that plays critical role in many machine learning communities. For the Gaussian kernel we studied, Bochner's Theorem states that there exists measure Q on Rd such that:
K(h(z),h(x')) = | chulM(e) (())du part Euwo cos (u(h(z1)- -(a))))
Rd
Since cas(a- a') cos(a) cos(a') + sin(a) sin(a'), we can approximate the kernel value using the Monte Carlo method as below:
m
K(h(z),h(z'))
cos (u,((z)) cos (u,h(z')) + sin (ug((z)) sin (urh(z')), ut Q.
iiid
m
Let 6m (h(z). := 1/ V m cos (u,h(x)) sin (uzh(z)).... cos (umh((z) sin sin(innn (umh(x))) be the featurization of h(x) according to Bochner's Theorem. Note that amounts to measuring h(x)'s
7

---

Under review as a conference paper at ICLR 2023
distances with respect to random directions drawn from Q(u), and then transforming them through trigonometric functions. Furthermore, { @ (h(-), ), Q), 4m (h(-), Q)) can approximate any entries in h. To be more precise, Rahimi Recht (007) shows that only requires m = (dd/e log(ar/))) to achieve K (h(x),, h(x²)) - (0m ( Q), pm (h(x'), Q)) S where oă is the second moment Q. Therefore, when is comparable to d, the featurized @m (h(z), achieves the stability and predictabiliiy of h as well as the computability of h.
Converting h(x) oin (h(x)) Q) is computationally efficient, since u1,-.., Um only need to be drawn from Q once and apply to all h(z;), i I....,n. However, there is still the obstacle of finding the optimal Q", Strictly speaking, Q is obtained from the inverse Fourier transform, but in practice the standard Gaussian distribution is often used. Indeed, compute the inverse Fourier transform and sample from it poses another challenging task. To our knowledge, there no existing study on whether we can safely sample from proxy Q. In the following proposition, we show that using Q instead of Q* will not cost stability as long as their discrepancy is bounded.
In particular, we state our result in the context of Lemma [ that is, the downstream risk is controlled by the alignment A == B[1[y = (]K((M( ((M(X), h [x)]]]] We use D2(Q, Q*) = s(dQ/ /e///d"""""d""" to denote the f-divergence induced by s(-).
Proposition 4. Let Q(Q": 5) == (Q D2(Q,Q") S 5} be a D,-ball with radius cen- tered Q' Ler (h(z)), 1 be the downstream data, and AQ( = m(n-1) Eizs 1lys (.l(mm(h(xi)) Q), om (h(x,), Q)). It holds that:
-
Pr sup QE2(Q*;)) (Q) - A0) 2
+ eexp (- exp (- 1 22
mc2
ne?
64 8)
where dQ = maxqe Øq.
The significance of Proposition 4 is that even if the optimal Q" is not used, in the worst case sce- nario, the instability caused by it (reflected via 8) vanishes quickly as the sample size gets larger. Similarly, increasing the dimension of featurized representation dm also speeds up the convergence exponentially. They provide the guarantee for predictability even if Q is not used. The proof is provided in Appendix E
Takeaway. Featurzing pretrained representation as om (m, Q) offers simple and practical solution to balance stability, predictability, and computability. We just showed that Q can simply be stan- dard Gaussian distribution, and the dimension of 6m (h) can be obtained by satisfying specific approximation threshold c. It can also be treated as a tuning parameter in downstream tasks.
6 BENCHMARK AND REAL-WORLD EXPERIMENTS
We conduct experiments on the benchmark dataset MovieLens-Im (MI-Im) for illustration and reproducibility purposes. The real-world production experiments took place at major US e- commerce platform anonymized as Ecom. The detailed descriptions for ML-1m and the intro- duction of Ecom's production environment are provided in Appendix
On ML-1m. The dataset supports two types of eeetraining-downsteee task combination: (a). leverage the sequences of user viewing data to pretrain movie embeddings, then use the embed- dings to predict the genre of the movie (ML-Im task I);
(b). pretrain movie embeddings using the title and other descriptions, then use the embeddings for downtream sequential recommendation (ML-Im task 2).
The detailed data processing, model and pretraining configurations, downstream training/testing setup, evaluation metrics, and sensitivity analysis are deferred to Appendix [F] On ML- task 1, we use contrastive representation learning to pretrain the movie embeddings, and employ lo- gistic regression to predict the genre using movie embeddings as features. On ML-Im task 2, we use bidirectional- -NN-tt structure on movies' NLP data, and extract the final hidden layer as pretrained representation. The downstream sequential recommendation task employs two-tower structure, and a RNN is used to aggregate the history viewing sequence. In Table we first see that dn (h) improves the stability of h by at least x10 in both tasks. Even under the same dimension,
8

---

Under review as a conference paper at ICLR 2023
(A) (h) outperforms h, and is highly comparable avg(h) the manually stabilized version of by averaging it over ten independent runs. Note that avg(h) is almost never a good practical solution because it requires repeating the same pretraining process multiple times. Here, use it as an an- alytical baseline, and show that @m (h) is just as good. When the dimension increases for 6m (h), it delivers much more superior results. Although changing dimension can also change the downstream model complexity, but as we discuss below, it offers more flexibility for real-world problems.
ML-Im Task1
Mm Task2
Ecom A/B testing
Accuracy
FI-Macro
Recaller
NDCG@5
Task
Task2
Task3
Task4
(a)
44.79(2.11) 64.82(2.83)
66.55(3.03)
33.96(1833)
used as control
(z))
68.72(-)
59.04(-)
68,98(-)
35.02(-)
(not applicable)
03 (h()))
68.64(0.27)
58.91(0.25)
68.90(.33)
34.97(17)
+0.24%
+0.13%
+0.14%
+0.10%
(b(x)(
85.77(0.19)
90.46(0.21)
71.30(.27)
37.48(115)
P < 0.01
P.O.1
p 0.05
p < 0.05
Table 1: For ML-1m experiments, the results are multiplied by 100, and in the parenthesis are the standard deviations computed from ten independent runs. The subscripts of h and e refer the dimension of repre- sentation, avg(h) refers to using the representation averaged from the ten independent pretrainings. Note that avg(h) cannot be used for large-scale production since it takes ten times the resources needed. For Ecom A/B testing, we present the relative lift over the control, and the p-value associated with the lift.
On Ecom. The item representation learning pipeline is being used by several downstream pro- ductions: item-page recommendation (Task!), search ranking (Task2), email recommendation (Task3), and home-page marketing (Task4). They all have task-specific features and non-trivial model architectures different. The refreshing of pretrained item embedding is done on a daily ba- sis, and downstream model owners may have separate schedules to update and refresh the relevant parts of their models. Appendix [F:4] we describe our engincering solutions of deploying the featurization process the frontend and backend.
During A/B testing, we observe performance lifts (in terms of click-through rate) that are statistically significant for all four downstream applications. The average prenue-per-vising lift is also positive during the testing period. The detailed online results and analysis are provided in Appendix F
Lessons learnt. In addition to improved stability and performance, important feedback we ceived from downstream model owners is that the flexibility in choosing 6m (h)'s dimension is very helpful for their tasks. Prior to our featurization technique, is almost impossible personalize the dimension of pretrained representation for different applications, let alone tuning it in down- stream tasks. Now knowing that the predictability will not vary much, experimenting with different dimensions often allows them to find better bias-variance tradeoff for downstream tasks.
DISCUSSION
The analytical results and the proposed featurization method in our work can apply to a broad range of applications and research problems. Nevertheless, our results may still be rudimentary and far from providing the complete picture or optimal practice for using pretrained representation. We hope the progress made will lead more advanced future research and applications.
Scope and limitation. Most of our analysis are performed in basic settings: while they ensure the results will hold in generality, advanced methods for pretraining representation are not considered. Also, we do not include additional downstream features and their correlation with pretrained repre- sentations, or connections between the pretraining objective and downstream task. Those additional knowledge can be useful for deriving task-specific results (Arora et al. 2019). For application, our featurization technique may be less helpful if the downstream task simply uses embedding distance like KNN search. Optimizing the space and time complexity by such as embedding quantization might be more useful for such tasks (Chen et al., 2020). which not discussed in our paper.
A future direction. While our work studies h(x) as whole, it can be inferred from Figure D that the element-wise variance of h(x)) is bimodal, which suggests heterogeneity within h(x). Possible explanations are that a (random) subset of h(x) is responsible for overfitting the pretraining task Bartlett et al.] 2020), that some dimensions are foreed to become more independent of others so the representation matrix has nice spectral properties (Hastic et al. 2022). is thus an important future direction to identify the internal structure of h(x)) to better featurize pretrained representations.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. What can resnet learn efficiently, going beyond kernels? Ad- vances in Neural Information Processing Systems, 32, 2019.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pp. 242-252. PMLR, 2019.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
Oren Barkan and Noam Koenigstein. Item2vec: neural item embedding for collaborative filtering. In 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2016.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research. 3(Nov):463-482 2002.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):33063333077 2020.
Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint.
Acta numerica, 30:87-201, 201, 2021.
Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Sci- ence, 59()):3413337777 2013.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 3798)11798-1888, 2013.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: nonasymp- lotic theory of independence. Oxford university press, 2013.
Supriyo Chakraborty, Richard Tomsett, Ramya Raghavendra, Daniel Harborne, Moustafa Alzantot, Federico Cerutti, Mani Srivastava, Alun Preece, Simon Julier, Raghuveer M Rao, et al. Inter- pretability of deep learning models: A survey of results. In 2017 IEEE smartworld, ubiqui- tous intelligence computing, advanced irusted computed, scalable computing commu- nications. cloud & big data computing, Internet of people and smart city innovation (smart- BAAAL//CCALLOOMIICAACCCCCOO////
pp. 1-6. IEEE, 2017.
Ting Chen, Lala Li, and Yizhou Sun. Differentiable product quantization for end-to-end embedding compression. In International Conference Machine Learning, pp. 1617-1626. PMLR, 2020.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recom- mender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10, 2016.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pp. 191-198, 2016.
Nello Cristianini, John Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in neural information processing systems, 14, 2001.
Andrew M Dai, Christopher Olah, and Quoe V Le. Document embedding with paragraph vectors.
arXiv preprint arXiv: 107.0998, 2015.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv: 1810:/810004400 2018.
10

---

Under review as a conference paper at ICLR 2023
Yixing Fan, Xiaohui Xie, Yinqiong Cai, Jia Chen, Xinyu Ma, Xiangsheng Li, Ruqing Zhang, Jiafeng Guo, et al. Pre-training methods in information retrieval. Foundations and Trends& in Information Retrieval, (6(3):78-3111 2022.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-58611 2020.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Limitations of lazy training of two-layers neural network. Advances in Neural Information Processing Systems, 32, 2019.
Vinay Chakravarthi Gogineni, Vitor RM Elias, Wallace A Martins, and Stefan Werner. Graph dif- fusion kernel Ims using random fourier features. In 2020 54th Asilomar Conference Signals, Systems, and Computers, pp. 1528-1532. IEEE, 2020.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pp. 1832- 1841. PMLR, 2018.
Maxwell Harper and Joseph Konstan. The movielens datasets: History and context. Acm Iransactions on interactive intelligent systems (lis), 5(4):1-19, 2015.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949 2022.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918-4927,201
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnll: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Dietmar Jannach and Malte Ludewig. When recurrent neural networks meet the neighborhood for session-based recommendation. In Proceedings of the eleventh ACM conference on recommender systems, pp. 306-310, 2017.
Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent nonseparable data. In Conference on Learning Theory, pp. 1772-1798. PMLR, 2019.
Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and processes, volume 23, Springer Science & Business Media, 1991.
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv: 1.00166 2017.
Fanghui Liu, Xiaolin Huang, Yudong Chen, and Johan AK Suykens. Random features for kernel ap- proximation: A survey on algorithms, theory, and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Lynn H Loomis. Introduction to abstract harmonic analysis. Courier Corporation, 2013.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2019.
Eran Malach, Pritish Kamath, Emmanuel Abbe, and Nathan Srebro. Quantifying the benefit of using differentiable learning over tangent kernels. In International Conference on Machine Learning, pp. 7379-7389. PMLR, 2021.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665-7777) 2018.
11

---

Under review as a conference paper at ICLR 2023
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1): 141-142, 1964.
Radford M Neal. BAYESIAN LEARNING FOR NEURAL NETWORKS. PhD thesis, University of Toronto, 1995.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Garvesh Raskutti, Martin J Wainwright, and Bin Yu. Minimax rates of estimation for high- dimensional linear regression over lq-balls. IEEE transactions on information theory, 57(10): 6976-6994, 2011.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: central limit theorem. Stochastic Processes and their Applications, 130(3):1820-1855 2020.
David Slepian. The one-sided barrier problem for gaussian noise. Bell System Technical Journal, 41(2):463500 -01 1962.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias gradient descent on separable data. The Journal of Machine Learning Research, 19 (1):2822-2878, 2018.
Martin Wainwright. High-dimensional statistics: non-asymptotic viewpoint, volume 48. Cam- bridge University Press, 2019.
Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun Lee. Billion-scale commodity embedding for e-commerce recommendation in alibaba. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 839-848, 2018.
Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. Advances in Neural Information Processing Systems, 32, 2019.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp. 3635-3673. PMLR, 2020.
Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Product knowledge graph embedding for e-commerce. In Proceedings of the 13th international conference on web search and data mining, pp. 672-680, 2020a.
Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. A temporal kernel approach for deep learning with continuous-time information. In International Conference on Learning Representations, 2020b.
Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Theoretical under- standings of product embedding for e-commerce machine learning. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp. 256-264, 2021.
Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian pro- cesses. Advances in Neural Information Processing Systems, 32, 2019.
Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for understanding neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Bin Yu. Veridical data science. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 4-5, 2020.
12

---

Under review as a conference paper at ICLR 2023
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2016.
Daokun Zhang, Jie Yin, Xingquan Zhu, and Chengqi Zhang. Network representation learning: A survey. IEEE transactions on Big Data, 6(1):3-28, 2018.
A TECHNICAL BACKGROUND
In this part of the paper provide the technical background for both the discussions in the paper and the following proofs.
A central role for proving uniform convergence results is the Gaussian / Rademarcher complexity. For set C IR", defined as:
G(A) = Ee
sup aEA 1=1
where e are i.i.d Gaussian / Rademarcher random variables. It essentially measures how good function class can interpolate random sign pattern assigned to a set of points. Given a function class F and samples (f1,...,Ta) Tn), the empirical Gaussian Rademarcher complexity is given by:
(F) = E sup fET
f(.))
Remark A.1. We mention that in some versions of the definition, there 1/n factor in the com- plexity term. Here, we explicit pull that factor out and place it in the resulting bound.
As we mentioned earlier, an important reason for us using Gaussian complexity is because some of its technical properties, which is the Slepian's Lemma (Slepian 1962) and its corollary, which state as below:
Lemma A.1 (From Slepian's Lemma). Suppose & : A + RR9 has Lipschitz constant L. Then holds that:
((o(A)) - LG(A).
This result can be viewed as the contraction lemma for Gaussian complexity (Ledoux & Talagrand 1991..
A.1 INDUCTIVE BIAS GRADIENT DESCENT
Our introduction primarily follows Soudry et al. 2018)) & Telgarsky (019); ?) Gunasekar et al.
(2018) and their follow-up works. The key factor that contributes the implicit bias gradient decent is the divergence of model parameters after separating the data under loss functions that has exponential tail behavior.
When the predictor f E F parameterized by 0 is peer-parameterized, other than certain degener- ated cases, the data can separated at certain point the predictor class satisfies some regularity assumptions (Lyu & Li 2019), c.g.
* f E F is homogeneous such that: f(x; c- 0) c² f(x; 0), Vc > 0:
. f F is smooth and has bounded Lipschitz constant.
These conditions can be met for many neural network structures and activation functions. The exponential-tail of loss function can be satisfied by the common exponential loss and logistic loss (which we use through our discussions and experiments).
To why the the norm of model parameters will diverge, simply note that under such expo- nential loss, both the risk and the gradient will take the form of: Σ. c; exxp(x:ff(le (x,; 8)), where e; lower order terms. Since gradient descent will converge stationary point due the nice
13

---

Under review as a conference paper at ICLR 2023
properties of F, expect E, G exp(-9p (x;; 0)) 0 to hold upon convergence. A necessary condition for that is: fxp 0) = 0, = 1,... n, and this condition is actually sufficient with high probability (Soudry et al. 2018
Therefore, for all exp(-): 0)) ) to reach O, ||0]| must diverge so (f(::)) + 3c. With that said, since the loss function decays exponentially fast around 0, the data points with the largest margin will dominate both the gradient and the loss function.
As direct consequence, the decision boundary will share characteristics with the hard margin SVM, given by:
min ||0||2 s.l. f 0) 2 1, Vi = l,...,,
Indeed, recent work shows that the optimization path of wver-parameterized models will indeed converge to some minimum-nore predictor:
Corollary A.I (Chizat al. 2211); Woodworth al. 2020), and others). Under the conditions specified in the reference work, which are mostly exponential loss, scaled initialization, appropriate learning rate, and regularity conditions for the predictor class, it holds that:
lim lim F(0()) (10(01)))01) cetiimaryppintt of { arg min fIll s.t. yf fx,)) 2 1, e [m]] (-000 (-0 d
where F the decision boundary of f. d the dimension of the hidden layer(s) of f. and || an appropriate RKHS norm.
ll is
Note that in Section 4.2 we use go to denote the converged result, and the above corollary guarantees its existence and uniqueness. However, one open question which particular RKHS norm best de- scribes the solution, because it will particularly affect the convergence of the parameters. Therefore, in our work, we leave the convergence of parameters out our discussion.
Remark A.2. It is also worth mentioning that the convergence of EE (9) (z)] plays no part in our arguments and results. Indeed, it will not change the stochasticity of h(tt) (x), and (in some cases) can be implied from the convergence of (x) Lyu & Li 2019) Therefore, we do not discuss it in our work.
B PROOF OF THE RESULTS IN SECTION
We prove Proposition 1 in this part of the appendix. An important result we will be using is the Gaussian complexity bound for empirical risk minimization, and we will use the version of Bartlett & Mendelson (2002).
Lemma A.2. Let F. be real-valued function class from X' to [0, 1]. Let (X1,... Xn) be iii.d random variables, then for all f F. it holds with probability at least 1 that:
E [[((x] S 5(x)) (2)
9 log 2/5
2n
We now provide the proof, part of which will be using Corollary Lemma A.I and Lemma A.2 We also assume FF has a Lipschitz constant of most L.
Proof. Recall that h", ;= arg minnen, R(h, f). We decompose the generalization error via:
R(h)
min NEH R(h, f) = (R(h)
-min 1 Σ e(f ((())((x))))))
JEF n
min Σ E((foh(x)))
min Σ (x))))((x))
FEF ni
JEF n
(A.1)
(min 1 JEF
Σ ((Sol((X)))
Epn
Σ [((ron(()))))) min JEF
+ Epn
min L
Σ
e(foh
min JeF
Ein (f
14

---

Under review as a conference paper at ICLR 2023
We first discuss the first term, which incurs major discussion in Section 4.1 the first term can be bounded via:
By standard practice,
R(h)
1 Σ ef [((foo(())))) min - JEF
(((h)
min Σ ((1 [((oo(())))))
sup heH
lEF n
E ((fnn ((X),)) - Ra(hi]]
sup pp hEHl
(a)
sup hER
Epm
Σ ol n
(b)
Using Lemma
A.2
term (b) can be bounded as:
Σ
n E (x) (49g)(4
sup hch
Pm n
where the set A(H) is given by:
{(2 e o h(X1), (frn o h(Xn), h E nn
It is easy to examine that A(H)) invokes Slepian's 's lemma, so we can use the contraction result from Lemma to further bound
(A(H)) S Gn
Vn
Combined together, the term (b) is upper bounded as:
VEHLOm(H) vm
+ 99
Now we bound term (a) as below.
Define the shorthand ((F(h) =
{((f(N(X,),)) ((((,..,,(((((,,,)) Y.))) : E F}. It holds that:
BxxYr (Jhn ( ( h(X), v) - Rn, (1)]
sup pm hE7
I sup pm sup heH
JEF (f ((),) - ((Foo(()),)))
I 22 sup Epm Gn(e(F(h))) kCH
(using Lemma A.2) and A.I
(A.2)
Epm (((F((()) [((X)| [|x(X]I
22n
sup DEM
(where ((X) = ((X)),...,,(144.
< 22t
sup JEF Σ +f(AA])) & iid N(0,
sup bEA
sup AERuxd
|||]
We let (, SUP
[ Esup E, ([Al)) be the modified Gaussian complexity, so
the term (b) is finally bounded by: 2 SFssheH v /E)1( Next, notice in the last term that:
(()oN())))))) s Epn 1 Eeff JEF
E
min
Therefore, the last term is always non-positive. Similar, by definition, the second term is non-positive well.
15

---

Under review as a conference paper at ICLR 2023
Finally, as for the second term, since there is already non-concentrating terms appearing in the bound of the first term, it does not hurt to simply bound using Hoeffding's bound. i.c. the first term will not exceed O /log 1/6) with probability at least 1 - 6. Putting things together, we conclude the final result.
C TECHNICAL DETAILS FOR SECTION 4.2
We first restate the proposition:
Proposition A.l. For the one-layer MLP we consider, with the learning rate L, for any step
t t> 1. as + oc, the model output gl (x) will converge almost surely to g:D) (z) defined as follows:
g- (x) = (cl) c + +Cs )l
with (c((t)) clett), , c(tt)) c(+)))) (c cl C cI 2 (0 (c.) 10 c c
The above iterative update result can be shown by making explicit of the terms following the forward and backward pass in 1 gradient step.
In partieular, it holds that:
(x) S EO (t) (W)())) g (x)), (x)1), y0)) $ ( y/(0)))) c())
(= ((U),
del
((++1) = 6( - (W(((()))))) ((++)) w - ot
(W)))()))
The only extra requirement for the above convergence to hold is that the activation function is well- behaved (see Yang (2019) for a detailed description).
To see how the above system of equations lead to the result in Proposition A.I imagine the activation the identity function. this case, ( and ( are always deterministic linear combinations of (0) and (.)) Observe that the update becomes:
( 6 (O)) w( O + +C (O))
We mention that as corollary, ((++)) (z) is also dementt-wss i.i.d, so the inner product of the hidden representations
(w(++)((), w(++) 3 [[ w(
where (()) an i.d.d row of ((++)) which is the rescaled version of ((+1).
D PROOFS OF THE RESULTS SECTION
Proof for Proposition 3
Proof. The proofs for the lower bound often starts by converting the problem to a hypothesis testing task. Denote our parameter space by Bik)= = {0 E Rd ||0l|o < k}.
The intuition is that suppose the data is generated by:
1. drawing 0 according to an uniform distribution on the parameter space;
2. conditioned the particular 0, the observed data is drawn.
16

---

Under review as a conference paper at ICLR 2023
Then the problem is converted to determining according to the data if we recover the underlying 8 as a canonical hypothesis testing problem.
For any 8-packing {0I..... 0m) } of B(k), suppose B is sampled uniformly from the d-packing. Then following a standard argument of the Fano method Wainwright (2019), it holds that:
P( min sup
||0 12 2 6/2) 2 min P(0 + B):
(A.3)
110" llo
where 0 a testing function that decides according to the data if the some estimated equals to an element sampled from the 5-packing.
The next step is to bound ming P(O * B), whereas by the information-theorrtice. lower bound (Fano's Lemma), we have:
[ B) + log 2
min P(O + B) 21
(A.4)
log M
where ((:-)) denotes the mutual information.
Then we only need to bound the mutual information term. Let Po be the distribution of (which the vector consisting of the n samples) given B = 8. Since y is distributed according to the mixture of: E, Po,. it holds:
I(y, B)
- M
DKL Po.ll M
Po. S
DKL (PP: 1
M
where DxL is the Kullback-Leibler divergence. The next step is to determine M: the size of the 8 packing, and the upper bound on D (Po, Il,)s where Pa,, Po, are elements of the d papakingg.
For the first part, it has been shown that there exists 1//2- 1/2-packing of ((k) in 2o-norm with log M 2 d-k (Raskutti et al. 2011).
log 1/2
As for the bound the KI-ddiergeece term, that given 0, Po is product distribution of the condition Gaussian: g/e ~ (0T OTo(o? -²//?)) where o² == of + o.
Henceforth, for any 01, 02 E B(k), it easy to compute that: (Po
||y - 03 03 12
||y - 0J e 11
n EEp, log (0? - -02-0//03 + 20 of/o?)
2 0,02(0² o4/0?)
2070: - of/o?)
a² ((( 02)11
2n
where y and are the vector and matrix consists of the n samples, i.e. R and c E Rax
Since each row in the matrix is drawn from NOO, Taxa), standard concentration result shows that with high probability, [|=(0, 02) can be bounded by Cl|0, 02 l/ for some constant C. It gives us the final upper bound on the KI. divergence term:
no
DRI (1pb))
2n
Substituting this result into (A.4) and (A.3). by choosing == Cha² oān
log d-k and rearranging terms, 1/2
we obtain the desired result that with probability at least 1/2:
o²
inf sup ||0 8* 1|12 2
kn -1 - log(d/k))
8 0=:li0- Il
o
Proof of Lemma
17

---

Under review as a conference paper at ICLR 2023
Proof. We first express the NW predictor in its expectation form:
fo(X) Ex y (X, xX)]
where Z is the normalization constant. Recall that y E {-1, R(-) is risk associated with the 0 1 classification loss. We first define for X::
Ex K(X,
To (x))
Z
where the expectation is taken w.r.t. the underlying distribution. Using the Markov inequality, we immediately have: [x() s 1 with probability at least It then holds that: v
((R()- P(VS(( 2 0)
Y f(X)
1[ 2 o|
2
(X)
Y ((X)
B[I[Y = =X"] vō with probability 1 1 5,
2 E
(X)
Z
which concludes the proof.
E PROOF OF THE RESULT IN SECTION 5
The proof of Proposition 4 relies on two important results, which we state below.
Lemma A.3 (Ben- Tal et al. (013 Let e be any closed convex function with domain [0, + and
this conjugate is given by (8) sup (1s c(()]. Then for any distribution Q* and any function
g IRd R. holds:
X20,1 / ((u))d(u)= inf {a f (0((())) (g(u) QEQ(Q):))
(s(u)-))dQ((0 (u) + 62 +
sup
(A.5)
The next lemma is adapted feature in Rahimi & Recht 2007 ). cos (um (mm(())) sin (umh(x))] comes from the Monte
from the concentration of random Recall that om (h(x), Q)
Fourier
1/ /mmmm cos (uth(z)), sin uz))))).......... Carlo approximation of K K(h(x), h(x"))
Lemma A.4. Let A 12d has diameter dA such that h(x) E for all æ E holds that:
Pr sup Mxx).((z))
[((h))) (6m (h(x), Q) om ()(1))
(A.6)
28 (2(( (QQAA) exp
mE²
4(d 2
where Q is given by the inverse Fourier transform of K, and σQ is the second moment of Q.
Recall that AQ( :..(Q)==iiTy))...m Ein, 1[y - vil( (1(z)) Q), om (h(z;), Q)). For notation conve- nience, in what follows, let h; h(zz), and further define o(h U) = (U) sin Sin(UTh) h)] as the actual random Fourier feature underlying om (h, Q), where U Q. Also, we let K (Y, Y") == Y Y' to be the labelling kernel of the downstream task.
Proof. Following Lemma A.3
we work with scaled version of the f-divergence under c(t) (th 1) (because its dual function has a cleaner form). is easy to check that c* (s) [s]]
with + 1
First note that the sampling error of the alignment E [[K(Y, expectation by the sample average, be given by:
H2)], i.c. replacing the
18

---

Under review as a conference paper at ICLR 2023
1
An(U) = n(n
K (u yj U))) U)"ā(h, -/KK,,,,))__KK
1
Σ
Ky, yo ) U\2 S(h, U) (L) Ul
n(n
We show that (U) is Sub-Gaussian. Let {M/ v. 1i be an i.i.d copy off the observation expect for one element such that (hj, Mj) + (K1,, v/). Without loss of generality, assume the last element is different: (M,, yv.) * (h/=3%))
V2). Let (U) be computed by replace (hs. with (M4, M= and their difference can be bounded via:
An (U) - (T))
1
K(n, (la..u))0l v,),,h, U)" (h,,,), (k,,,))( UY" Σ
nnn
1
K(v Vn ) U)² ((hn, U) K(v,, Un o( U)
n(n2
1)
ien
(un, U3 (b u(, U) K( W, U)
Jkn
4
where the last inequality comes from the fact that the random Fourier features o and the labelling kernel K (y. y²) are both bounded by Therefore, the above bounded difference result tells that (U) is -subGaussian random variable.
To bound An (U), we use:
f (nnUU) QEQ(Q-:8)
Ann(U)ddQ
sup
sup QEQ(Q=:))
X(5 X(0 + (using Lemma A.3 Eq [IAn()))*
inf A20
k'
(ō + 1/K
|I (U)) 1/2 (solving for X" from above)
vō LE (U)) 1/2 (let =k = 1/22.
(A.7)
means that in order to bound ( (U))QQ we only need to bound (A))..
Using classical results for sub-Gaussian random variables (Boucheron et al. 2013), it holds for S n/8 that:
E exp (AA,, S exp (- (- log(1 - (og(1-8A/n)).
We can take its integral and further upper bound the result with:
pl / An (U))2QQ
2
z 8
8+11
s E eeEe exp (a/ An(()P(Q) exp
dca (Chernoff bound)
8 8
I <exp
(- 1 2
log (1
)))
81
dc²
(apply Jensen's inequality).
8 111
Hence, it holds that:
ne2 16(1 + )
Pr sup (6)
A, 2) < exp
Combine this result with the approximation error of random Feature feature in Lemma obtain the desired result.
A.4
we
19

---

Under review as a conference paper at ICLR 2023
[b]
0.7
1.75
1.50
Protm Loss
0.-
1.25
VOlUs
Denaity
1.00
Protmm
0.75
0.30
0.25
0.00
0.0
2.5
5.0
7.5
10.0
12.5
0.2
0.0 Empirical Variance
0.4
Epoch
Figure A.I: Replicating the results shown Figure 1 Db. and D on IMDB dataset. (a). The wime variance of the pretrained embedding values of three randomly sampled reviews; (b). The convergence of the pretraining loss and accuracy. (c). The distribution of the empirical variances of the pretrained embedding values, pretraining progresses.
F SUPPLEMENTARY MATERIAL FOR THE EXPERIMENTS
We provide the descriptions, details, and additional results of our experiments in this part of the appendix.
F.1 REPLICATING THE INSTABILITY ISSUE WITH IMDB DATASET
The IMDB dataset is binary sentiment analysis dataset consisting of 50,000 reviews from the Internet Movie Database (IMDb) labeled as positive or negative"] We particularly consider using this dataset for addition proof of concept because it appears on the official rutorial Tensorflow
We directly adopt the implementation from the tutorial, including the text prepossessing pipeline and model architecture. In particular, the raw input texts pass to text vectorization layer, an embedding layer, a bidirectional RNN layer, and finally two dense layers to produce the final score for binary classification. We extract the hidden representation from the last hidden layer the hidden representation. In our experiments, we set the number of hidden dimension as 32.
The results are provided in Figure A.I where observe patterns highly similar to the MI-Im data. In particular, the pretrained embeddings both have high variances in their exact values even if their pretraining objectives converge to similar loss and accuracy, and the variances gets larger the pretraining progress. Two minor differences from the 1L-1m result are that the pretraining process less stable for IMDB (Figure A.Ip) p). and the variance distribution here is unimodal instead of the bimodal distribution we observed in Figure T
F.2 DETAILS OF THE RENCHMARK EXPERIMENTS
The main benchmark experiments in our paper are conducted the Movielens-Inte dataset, which a well-established public dataset for movie & user contextual analysis and examining recommen- dation.
The ML-Im dataset consists of million movie ratings from 6,000 users on 4,000 movies, with a one-to-five rating scale. According Harper & Konstan 2015). the data is collected from the initial and follow-up stages, where the imitial stage mainly involves popularity- very small proportion involves random exposure), while in the follow-up stage, rating feedback is collected under some deterministic recommender systems. By convention, we convert the dataset to implicit feedback, which amounts to treating all rating events as clicks. For contextual information, each movie is provided with its title and genre, in the form of English words or sentences. There are 18 genres in total.
exposure (a
Pretraining movie embeddings from user behavior data
http:://ww.innb.ccmm//eeeeeeeeee
20

---

Under review as a conference paper at ICLR 2023
We use Item2vee (Barkan & Koenigstein 2016) to train movie embedding from users' consecutive viewing data. Item2vec uses the same objective function as Word2vec (Mikolov et al., 2. where the words become movies and the corpus become each user's viewing sequence. Movies belong to a consecutive viewing window of #ws are treated as positive pairs, and for each positive pair, we randomly sample #hs negative movies. Together with the embedding dimension d and fz- regularization parameter (weight decay) X, the training schema is described by the quadruplet of (#ws, #ns, X). Since our goal is not to find the best pretraining schema, we fix #ws=3 and #ns=3, and focus on studying how the our results may change under different d.
Pretraining movie embeddings from movie contextual data
Since the movie titles and other contextual information can be relatively short, large NLP models may not be ap- propriate. Therefore, we use the Doc2vec model[Dai et al.) (015) to pretrain the movie embeddings. Since Doc2vec is built on top Word2vec, the training schema can also be described by the quadruplet of (#ws, #ns, d, X). There- fore, we also #ws=3 and #ns=3.
Score
Dense
Using pretrained movie embedding for downstream genre prediction
Figure A.2: The two-tower architecture.
Given pretrained movie embeddings h(x). we employ lo- gistic regression to predict the score for the movie to be-
long to a particular genre, i.e, p(Y; = k) exp(0xh(z))
Due to its simplicity, we use the logistic regression subroutine from the scikit-learn package.
Using pretrained movie embedding for downstream sequential recommendation
We employ a two-tower model structure (Figure A.2 for the downstream sequential recommen- dation, which very common in the recommendation community. In particular, we use RNN to aggregate the past interaction sequence, so the whole model is very similar to GRU4Rec (Jannach & Ludewig 2017). We use the sigmoid function as the activation function for the dense layers. The model training can be done in a seq-to-seq fashion, where for each positive target, we randomly sample 3 negative targets. We fix the hidden dimension both the RNN and the dense layers as 16.
Model Training
Besides Doc2vec and the logistic regression, all of our models are optimized using the Adam opti- mizer with early stopping, which stops the training if the improvement in the loss is less than le for three consecutive epochs. For all the experiments, we the initial learning rate to 0.001, and set the weight decay to le 4. Our main implementation is with Tensorflow, and all the computations are conducted on 16-core Linux cluster with 128 Gb memory, and two Nvidia Tesla V100 GPU each with 16 Gb memory. We use the Doc2vec subroutine from the Gensim package to pretrain the movie embeddings for ML-Im task2.
Train/test split and metrics
Since the goal our experiments is not to find the best modelling and training configuration, we do not use validation set tune the hyperparameters. Instead, provide sensitivity analysis certain parameters of interest in Appendix
For ML-1m task1, we randomly split the movies by 80%-20% to construct the training and testing set for genre classification. For evaluation, we use the accuracy and Macro Fl score as metrics.
For ML-1m task2, follow the convention of using the last user- interaction for testing, and use all the previous interactions for training. For evaluation, we use Recall@ i.e. if the movie that the user truly viewed is among the top-5 recommendation, and NDCG@5 5 that further discounts the position of the viewed movie in the top-5 recommendation.
Inttps://radimrehureee
21

---

Under review as a conference paper at ICLR 2023
F.3 SUPPLEMENTARY RESULTS
We provide the sensitivity analysis for the featurization method. We focus on two variables, the dimension and the variance of Q (denoted by oz). Recall that we consider Q as Gaussian distri- butions. We vary d in {16, 32, 64), and vary oă in (0.2, 0.4, 0.6, 0.8).
ML-1m task1
ML-1m task2
100
75
70
Accuracy
Recalig5
65
4u(h)
Pu(h)
60
#z)hh)
60
$2xth)
35
40
50
d-16
d-32
d-64
d-16
d-32
d-64
Dimension
Dimension
Figure A.3: Sensitivity analysis on the representation's 's dimension.
In particular, we first compare side-to-side h,, @2(h), and (2a(h), while fixing Q as the standard Gaussian distribution. We see from Figure A.3 that @a(h) consistently outperforms ha on both ML- Im task? and ML-Im task2. (2) also significantly improves upon the performance of (a(h), which suggests the benefits of allowing extra model complexity in the particular tasks we consider. Further, the performance of both (a)h) and ( have considerable smaller variances than h(z).
Sensitivity analysis de on ML-1m task1
Accuracy
04
36
Sensitivity analysis for dg on ML-1m task2
Recalloss
Variance of (0g)
Variance of (ag)
Variance Q (dp)
Figure A.4: Sensitivity analysis on the variance of Q (TQ) from which U is drawn.
We then examine the sensitivity of the downstream performance w.r.t. Q the sampling distribution for constructing (h).. As stated before, we let Q be zero-mean Gaussian distribution, and vary observe that for all the dimensions task under (a(h) is very stable under different aq- This echos Corollary 4 that our approach enjoys robustness to the selection of Q. In real-world productions, have been using standard Gaussian distribution and observed no issues.
its variance. From Figure [A.4]
consider, the downstream
F.4 ONLINE DEPLOYMENT
To avoid potential conflict of interest, we provide an overview of our production experiments. We aim to provide enough detail for interested practitioners draw inspiration for both developing their own solutions and replicating ours.
Some background introduction.
22

---

Under review as a conference paper at ICLR 2023
In e-commerce application, the representation of items serves as central component for almost all machine learning algorithms Wang et al. (0188); ();(Xu Xu et al. (2021). In the past few years, we have built dedicated item representation learning pipeline that uses multiple sources of data optimize item emheddings. Since there are over billions of items on our platform Ecom, it took us considerable effort optimize the data pipeline and training routines such that the model refresh can be done on a daily basis.
We point out that the daily refresh is necessary for item representation learning because the catalog of items, which is a major source of pretraining data, also gets minor updates on a daily basis. For example, new items can be added, and the features of items (e.g. title, price, description) can be modified by the vendors. The other major source of pretraining data is the historical customer behavior data. They are critical for revealing the relationship (e.g. similarity, complementariness, compatibility, substitutabilit)) among items. These relationships are relatively stable in the customer population, so the more data we use, the more likely for us to discovery useful signals.
Our model for pretraining item embeddings has both feed-forward component, recurrent-uni com- ponent, as well as contrastive learning component. The reason for using these different components is to effective handle data that has different structures. It is expected that the pretrained item em- beddings are stable. As we mentioned above, the relationship among items are quite stable, and the catalog data has very minor differences in limited span of time. Therefore, downstream machine learning models may follow a weekly or bi-weekly refresh schedule and are expecting very stable performances.
The four major applications that depend on our pretrained item embeddings, which we first intro- duced in Section 6 are item-page recommendation, search ranking, email recommendation, and home-page marketing. Each of the four tasks use both item embeddings and task-specific features to optimize their objectives. Most of them use model structures similar to the Wide and Deep net- work (Covington et al. 2016) to effectively combine information from different sources. Item-page recommendation aims to provide items that are related to the anchor item on that particular page that the customer is viewing. Item embeddings are in both the recall and reranking stage. Search ranking a huge system that combines multiple components. In particular, the item embeddings are used in particular recall stage. Email recommendation is a simpler task that aims to recommendation items related to what the customers recently interacted with, or are supposed to interact again. Item embeddings is used along with other features build model that optimizes CTR. Marketing is also a huge system in Ecom, and the particular component that uses item embedding is to build the predicted click-through rate model support bidding and placement ranking.
Brief summary of the production environment and implementation.
Our production environment is relative standard in the e-commerce industry, with Hive/Spark sup- porting the offline data streaming and Tensorflow Server supporting the online inference of deep learning models.
Featuring h())) via ((h(-), Q) can be easily implemented in production. Some practical advantages are:
the algorithm is very simple and requires no training;
it fits seamlessly into the current big-data infrastructure and frontend service;
it require no change to the downstream model;
the overhand for both the training and inference time are small;
the communication can be easily done by simply recording the dimension and random seed under which particular U is generated.
On the backend, fcaturizing pretrained representation is engineered into subroutine (on top of the original automated representation learning pipeline) callable by downstream applications. For in- stance, it can be simple PySpark function if the end point of the automated representation learning pipeline is a feature store in HDFS.
The dimension and the random seed for generated the random directions U j....... Uy] are the two main inputs. Configuring and logging the random seed used by each experiment is important because U might be reused for deploying the model frontend. If the dimension and random seed
23

---

Under review as a conference paper at ICLR 2023
logged in the configuration, then there is need pass the particular U around across different infrastructures. We mentioned in Section that the dimension can be chosen by either specifying a threshold the approximation error (Proposition 4. for which provide implementation that takes as input. We find that in most cases, downstream model owners are more willing to treated tuning parameter.
For frontend service, notice that featurizing h into 6m (h, Q) amounts to simply adding an initial fully-connected layer whose parameters are given by the U E Rdxm used offline training. The activation function of that initial layer is given by the sin(-) and cos(-) functions. Therefore, it fits seamlessly with the serialization or signature building processes of the downstream model at no extra complication. In term of the inference time, it costs little overhead for large models. For smaller models and applications, on the other hand, we find out that it might be more efficient to directly cache the featurized @m (h, Q) at the expense of space complexity.
A/B testing and result analysis
Deploying unified A/B testing across the different downstream applications is extremely challeng- ing. Therefore, the A/B testing is conducted with each downstream application, but we coordinated with the different teams such that they approximately start at the same time.
02%
-06
App1
App2
App3
App4
VARATION
Figure A.5: Online A/B testing results, and the monitoring the per-vistor gross- coss-merchandise value (GMV) during the testing period. Appl, App2. App3, App4 correspond item-page recommen- dation, search ranking, email recommendation, and home-page marketing.
The control baseline is simply the existing implementation of each downstream applications. They use the raw item embeddings to refresh their models as usual. The treatment is switching to d(h) in their implementations and obtain another version of their models. The outputs of the control model and treatment model are exposed to users pre-allocated into different buckets. The exact testing logic quite involved and beyond the scope of our paper. The key takeaway is that the testing begins almost at the same time for all four applications, and the only treatment variable is which item embedding to use. The detailed testing results are provided in Figure (A.5] For brevity, only show the performance in terms of CTR since it is indicative for all four applications.
As we shown Table all four A/B testings achieves statistically significant result under the level of 1. In particular, the lift of item-page recommendation CTR has p-value less than 0.01. Further- more, we observe a very positive change to the per- visitor gross-merchandiig value (GMV) during the testing period. Although this metric can be confounded by many other factors, nevertheless believe featurizing pretrained representations benefits both the individual machine learning task and the overall business metric.
Summary
We examine the performance of featurizing pretrain item embedding via the method proposed in this paper. We discuss how the approach fits scamlessly with our production scenarios, the overview of our implementation, well as the final testing results. Based experience, featurizing
24

---

Under review as a conference paper at ICLR 2023
representation could be important step toward more stable and productive system for industrial machine learning.
G ADDITIONAL DISCUSSIONS
In this part of the appendix, we will provide additional discussions that provide readers with better understandings of our contributions.
Connections between our theoretical and empirical results
Proposition I and its subsequent discussion outline a potential problem with using pretrained rep- resentations in practice: the variance of h(z) may cause the empirical downstream solution not concentrating to the optimal downstream solution. In Proposition 2, we provide a comprehensive result that characterize the variance of l(x)), which corroborates 1. In particular, it shows that although the pretraining algorithm may converge, the entires of h(() can remain stochastic, which negatively affects the downstream performance. Together, Proposition and 2 suggests that directly using (xx)) in downstream tasks may result in poorer performances, which is corroborated by the empirical results in Table 1.
empirical findings in Figure
Proposition 3 and Lemma 1 analyze the potential interactions scenarios between pretrained rep- resentation and downstream tasks. They highlight the role of the embedding kernel and motivate the featurization approach we later developed. Proposition 4 provides stability guarantee for the proposed featurization approach, which indeed delivers superior performances as shown in Table 1.
Additional relevant literature
Besides the recent develop in machine learning theory summarized in Section 2 our work is related to number of active domains in machine learning.
In terms of applications, pretrained representations are being used by a broad range of industrial machine learning tasks including information retrieval, recommeder systems, advertisement, knowl- edge completion, social network, user understanding, and many others (Cheng al. 2016 |aan et al,
2022 Xu et al. 2020a Zhang et al. 2018). Unfortunately, no existing work investigates their effec- tiveness from the industry's 's standpoint, that is, whether they strike a good balance between stability, predictability, and computability. Parallel to the above industrial tasks the large natural language and computer vision models (Devlin et al.) 2018: He et al.. 2019), whose complexity exceeds the typical settings studied in this paper.
The proposed featurization approach relies on random Fourier feature a well-established and im- portant tool of modern machine learning (Rahimi & Recht 2007). Its properties are known and are being actively applied to obtain kernel representations for many applications, such as signal processing, time series, and graphs (Liu al. 2021 Gogineni al. 2020 Xu et al.. 2020b).
Finally, compared with the recent line of research on interpretability of deep learning models (Chakraborty et al. 2017), we focus particularly on the properties of learnt representations and how to improve them.
Experiments with NLP dataset
We further conduct experiments with the IMDB dataset described in Appendix F, where we use Word2vec for pretraining word embeddings, and apply them to a downstream bi-directional RNN model for sentiment analysis. We mention that large language models are not suitable for our pur- pose because the existing model checkpoints provided by the developers are from one-off imple- mentation. They cannot be used to recover the variance information of the pretrained embeddings. Also, our analysis requires retraining those large models from scratch multiple times, which are very time-consuming and resource-intensive.
The experiments are straightforward, they closely reveal the instability issues of pretraining em- bedding and the benefits of the proposed featurization approach. In particular, we consider the window size (#ws) and number of negative samples (#ns) as the configuration parameters of the Word2vec model. We vary their values and present the array of downstream performances. For each pretraining configuration. we conduct 10 independent runs to compute the empirical vari- ance measure of h(x)'s instability in the downstream task. Our Word2vec pretraining follows
25

---

Under review as a conference paper at ICLR 2023
the original implementation of (Mikolov al. 2013). 3. which uses the standard stochastic gradi-
ent descent with a decaying learning-rate schedule for 30 epochs. All the remaining settings are kept default, and observe they are sufficient for achieving very small training loss (< 1c-6).
The architecture of the downstream sentiment classifi- cation model is shown in Figure A.6 which is bi-
directional RNN standard in the NLP literature. To align
with the original Tensorflow implementationf we adopt the same model configuration and treat the weight decay as the only tuning parameter of the downstream task. For each set of pretrained word embeddings, we select À from (1e-6, le-5, le-4,le-3, le-2} according to the vali- dation accuracy.
great
Input
120
35
TextVectorization
Embedding
The train- saain-vaidation-tess split follows the standard 80%- 10%-10%. We use the binary cross entropy loss and use accuracy as the metric. We also use the Adam optimizer with initial learning rate le-4 and early stopping (when the validation accuracy stops improving for five epochs). The remaining settings are kept as default.
Bidirectional
Dense
For reference, when the embedding layer is trained jointly, the downstream accuracy following the above im- plementation is: 86.25 (0.12). The downstream accuracy of using the different sets pretrained embeddings are pro- vided in Table The featurized representations (a(h) obtained in the same way as described in Appendix F. In A. we observe almost identical patterns to our particular, shows that di- rectly using (ax) in the downstream task results in a high degree of instability under all pretraining configurations. The proposed featurization approach not only achieves better testing accuracy, but it improves the downstream stability by almost eight times average.
Classification
Figure A.6: The model architecture of downstream sentiment analysi In the original experiments, the embedding layer is trained jointly with the rest of the model. Here we replace it by pre- trained word embeddings.
previous results in Table
Results on the NLP task provides additional evidence for both the instability issue of pretrained representations, and the benefits of the proposed featurization approach.
d=16
#ws=3, #ns=3
#ws=3, #ns=4
#ws=4, #ns=3
#ws=4, #ns=4
h(x))
74.87 (1.98)
75.52 (1.96)
76.02 (1.96)
75.79 (1.97)
(h(())))
76.65 (.24)
77.34 (.22)
77.95 (.22)
77.50 (.23)
d=32
#ws=3, #ns=3
#ws=3, #ns=4
#ws=4, #ns=3
#ws=4, #ns=4
(xx)))
76.64 (2.01)
78.23 (1.98)
79.33 (1.75)
78.87 (1.82)
(a((()))
78.26 (.23)
80.07 (.21)
81.05 (.20)
80.79 (.21)
d=64
#ws=3, #ns=3
#ws=3, #ns=4
#ws=4, #ns=3
#ws=4, #ns=4
h((x)
76.35 (2.14)
78.31 (2.16)
78.48 (2.14)
78.22 (2.09)
(a(((x)))
78.49 (.24)
80.56 (.24)
80.86 (.21)
80.32 (.22)
Table A.1: Testing accuracy of using pretrained word embeddings for downstream sentiment classi- fication. All results are multipled by 100, and in the paranthesis the standard deviation computed from 10 independent pretraining runs.
httppp//www..eeaaaaooooo////a/aa f fication_rmi ion_rnn
26


Reviews:
[{"review_id": "uP818A5ErD", "reviewer": "Reviewer_vPVs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The Author's investigation reveals critical insights into the gap of uniform convergence for analyzing pre-trained representations, their stochastic nature under gradient descent optimization, what model convergence means to them, and how they might interact with downstream tasks. Authors propose a simple approach which contributes to both applied and theoretical research of representation learning. ", "summary_of_the_review": "\n1) In section 3, why is \\Theta d x k when y is just a scalar? How to make sense of loss in this case?\n\n2) Section 4.2 \"instability of ˆh(x): the exact position of ˆh(x) in Rd is stochastic, depending on the initialization and the order of the pretraining data that is fed to SGD\": What if the initializations are same every time? \n\n3) \"In addition to improved stability and performance\": How is stability measured in Table 1?\n\n4) How was hyperparameter tuning done for the main results? \n\n\n", "strength_and_weaknesses": "Strengths:\n1) Authors explain concerns with existing methods clearly.\n2) Authors divide each of their theoretical results and give a takeaway summary for better understanding for readers \n3) Authors back their theoretical claims with empirical results \n\nWeaknesses:\n1) Setting considered in the paper is too simplistic to be broadly used in the industry. \n2) More connection between theoretical results and empirical results is not discussed well enough. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pzTmW5zvIck", "reviewer": "Reviewer_BWrq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to solve the suboptimal problems caused by the pretraining models in industrial systems. And the authors analyze the problem with four aspects( the gap of uniform convergence for analyzing pretrained representations, their stochastic nature under gradient descent optimization, what model convergence means to them, and how they might interact with downstream tasks). Then they propose a method to refine the features of pretraining models in various ways to improve the stability of the downstream task model. Experiments on six tasks prove the effectiveness of the proposed method.\n\n", "summary_of_the_review": "Overall, the paper proposed an interesting idea and showed strong empirical results, but it should have a more comprehensive analysis of both pretraining and finetuning. Hence, I would suggest the author complete more related works to give more convincing conclusions. \n", "strength_and_weaknesses": "Pros:\n\n-The problem addressed has high practical value: it tries to make pre-trained model more accessible to a range of industrial systems. The \"Featurizing Pretrained Representations\" idea will promote the performance of downstream tasks compared to other approaches.\n\n-This paper verifies its thought with a detailed formula. By analyzing the features of the pre-training models, the author proves that the features from the pre-training models are related to the stability of the performance of the downstream tasks. In detail, the variance of the pre-training features h (x) plays an essential role in the learning process of downstream tasks, and the variance will increase with the increase of random variables in the pre-training model. \n\n-The proposed method (Featurizing Pretrained Representations) is intuitive and effective, which uses a more stable h (x), e.g., the representation of weights in the NW estimator in Section 4.3, in downstream tasks. Moreover, experiments on six downstream tasks indicate that the method has superior performance and stability.\n\n\nCons:\n\n-It may have a more suitable h(x). The authors think the optimal h(x) can be obtained through Fourier Transform. But it is not easy to get the representation of the sample by the inverse Fourier transform. So they replaced this method by increasing the number of samples and the dimension of features. Nevertheless, this method may not be the most suitable one in industrial systems due to the limited sample and computing resources. Maybe the authors can seek a better method from the perspective of math or Fourier Transform.\n-The analysis is not comprehensive enough. Although the paper proves that the features of pre-training model do affect the stability of downstream tasks, it ignores the features of downstream tasks, which are more essential for downstream tasks. Have the authors tried to analyze the downstream tasks, and what does the result look like?\n\n-The essence of h(x) is still not clear. The paper discussed the impact of the variance of h(x) on downstream tasks, and then the authors improved the performance with the optimization of h(x). However, they did not give any description of the reason why the pretraining models generate such features h(x). Maybe it is more important.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cF3PRERFHjR", "reviewer": "Reviewer_VPk5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper observed an instability issue of downstream prediction based on features from pretrained models (i.e., the accuracy of a downstream task is varing, which depends on the pretrained representation trained on the same data, model configuration, training setup, and stopping criteria). This empirical observation is theoretically analyzed via a novel uniform convergence bound for pretrained representations, and eventually proposes a solution, called featurizing pretrained representation, which is empirically justified to outperform a baseline (i.e., a downstream task with usual pretrained features) using the MovieLens1m dataset and an in-house real-world production dataset. \n", "summary_of_the_review": "I like the way of this paper to attack the observed representation instability; but the paper does not follow the actively changing trends — large language models are widely used, but didn’t account for this line of work (including in the related work section). Moreover, as mentioned in the weaknesses, I have concerns on the proof that supports the paper’s main contribution. Thus, I vote for rejection for now, but I’m willing to adjust my understanding. \n\n\n=== After the rebuttal\n\nThanks for providing details.\n\nFor the Proposition 1 proof, it needs to provide that the slack term does not disappear for some f and h. In particular, the generalization error in Proposition 1 is decomposed in three terms in the proof. And each term is *independently* bounded by a uniform convergence bound. However, this does not mean that there exist f and h such that the slack term is not canceled by other terms. Moreover, the argument on the existence of an irreducible slack term may not be achievable via upper-bounding the generalization error since I can take any upper bound and then artificially add some slack term to claim that there is an irreducible slack term. \n\nI think having a correct proof Proposition 1 is necessary to generalize the paper’s intriguing observation — “In real-world production, however, we have encountered key problems that cannot be justified by existing knowledge. They raise concerns that the naive use of pretrained representation as feature vector could lead to unwarranted and suboptimal solution.” I agree that Proposition 1 is an intermediate step, but this is the basis of the paper’s argument. Otherwise, authors’ need to empirically justify their observations with a large number of experiments with various models and datasets.\n\nFor these reasons, I still lean toward rejection, but I would see the value of the proposed algorithms (as heuristic) and empirical results.  \n\n\n", "strength_and_weaknesses": "**Strengths**:\n* Provide theoretical interpretations on empirical observations. \n* A solution to address the claimed instability issue is simple.\n\n**Weaknesses**:\n* The main empirical observations (Figure 1) are made in weak setups\n* Proposition 1 is not necessarily true. \n* The experiment result is weak\n\n(Weakness 1)\n\nThe paper first made empirical observations on MovieLens-1m along with the traditional Doc2vec model as a pretrained representation, along with logistic regression for a downstream task. Considering a huge advance on a large language model, this experiment setup does not represent the current state of representation learning. If this observation is made with IMDB (or any larger dataset) along with BERT (or RoBERTa), the story would be much stronger. \n\nMoreover, the following statement needs to be more carefully explained: \n“Since we use logistic regression as the downstream model, the fluctuation can only be caused by the instability of pretrained representations.”\nWhy logistic regression cannot be the source of the instability? In particular, do you mean the optimization for the logistic regression is done until it converges to the global optimum?\n\n\n(Weakness 2)\n\nProposition 1 implies that the uniform convergence bound for pretrained representation has an irreducible term, contributing to the instability of the downstream task performance. However, I think Proposition 1 is not necessarily true. In particular, (A.1) decomposes the generalization bound, and the proof claims some terms are non-negative, which is not always true. \nConsider the second term in (A.1), where the proof said “by definition, the second term is non-positive “. First, I think \"non-positive\" is a typo as if it is non-positive, it can cancel out the irreducible term, thus the claim does not make sense; so I assume that the term is \"non-negative\". Here, h* is the optimal representation w.r.t. the expected loss R(h, f) instead of the empirical loss, thus the second term can be negative. In other words, if $h^* = argmin_h \\sum_i \\ell(f \\circ h(X_i), Y_i)$, the second term cannot be negative; but the if statement does not hold. Additionally, the non-negative argument on the last term is the same issue. \n\nThis proof and the affected arguments in the following sections need to be fixed and adjusted. \n\n\n(Weakness 3)\n\nI liked the experiments on real-world production experiments, which demonstrates how knowledge in academia is actually used in practice. But, to justify the efficacy of the proposed approach, it is always good to evaluate multiple public datasets, but currently only one is used. \n\nFor the ML-1m, I’m not sure if this is a widely acceptable dataset in representation learning; the main reason is that its input dimension looks not large enough (e.g., based on the paper, saying “each movie is provided with its title and genre, in the form of English words or sentences. There are 18 genres in total.”). Based on this limited information, classifying five ratings sounds very challenging (contrast to this, a human rates each movie based on richer information). I personally think this unusual property of the dataset may lead to the unstable results in Figure 1. Appendix F1 includes some results on IMDB, but I cannot see the downstream accuracy results as in the figure in page 2. \n\nIn short, evaluating the proposed approach at least 2 widely acceptable datasets would make the experiment results stronger. I guess IMDB's downstream accuracy results need to be presented to strongly show the observed instability issue. Moreover, I’d recommend using features from BERT or RoBERTa for pretrained model features. Based on Table 1, the featurting pretrained representation looks promising, and hoping that the same trend holds for other datasets and representations. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject