Functional Risk Minimization
Authors: Ferran Alet
Clement Gehring
Tomás Lozano-Pérez
Joshua B. Tenenbaum
Leslie Pack Kaelbling
Venue: Submitted to ICLR 2023 (2023)
Keywords: learning framework
theory
meta-learning
supervised learning

Abstract: In this work, we break the classic assumption of data coming from a single function $f_{\theta^*}(x)$ followed by some noise in output space $p(y|f_{\theta^*}(x))$. Instead, we model each data point $(x_i,y_i)$ as coming from its own function $f_{\theta_i}$. We show that this model subsumes Empirical Risk Minimization for many common loss functions, and provides an avenue for more realistic noise processes. We derive Functional Risk Minimization~(FRM), a general framework for scalable training objectives which results in better performance in small experiments in regression and reinforcement learning. We also show that FRM can be seen as finding the simplest model that memorizes the training data, providing an avenue towards understanding generalization in the over-parameterized regime.

Under review as a conference paper at ICLR 2023
FUNCTIONAL RISK MINIMIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
In this work, we break the classic assumption of data coming from a single func- tion fo- (x) followed by some noise in output space fo (z:)). Instead, we model each data point (Is. yt) as coming from its own function fo.- We show that this model subsumes Empirical Risk Minimization for many common loss func- tions and captures more realistic noise processes, We derive Functional Risk Min- imization (FRM), general framework for scalable training objectives that results in better performance in supervised, unsupervised, and reinforcement learning ex- periments. We also show that FRM can be seen as finding the simplest model that memorizes the training data, providing an avenue towards understanding general- ization in the over-parameterized regime.
INTRODUCTION
1.1 MOTIVATION
In most machine learning settings, we only have limited control over how data is collected and even less so over the process generating it. For this reason, data is often correlated in complex ways, like data coming from similar times or locations. When these correlations are known, one can handle them appropriately as is done in frameworks such as multi-task or meta learning. However, in the absence of obvious reasons to specialize models to subsets of the data, practitioners often take an opposing perspective where differences in labels belonging to similar inputs are regarded as noise, often modeled in the output space. This idea serves as the basis for the training objectives we prefer, e.g., mean-squared error objective for gaussian noise cross-entropy objective for multinomial distributions. By not accounting for highly structured noise, we expect that a singular model will appropriately average out noise differences during training.
For instance, consider training a language model on Wikipedia, then fine-tuning it to work on a dataset of books. In doing so, In contrast, when we train a model on general internet data, using Wikipedia and the dataset of books, typically use a single function feunternes and we explain each training example with multinomial noise in output space, i.e., yi ~ P ([fiittrree(x:))) However, whether we arrange the data into different datasets or a single one, the datapoints remain the same. Therefore, it is contradictory to handle the same variability using two different models: functional diversity vs. output noise.
two different functions fonooks and foom with formoro fown--
To remedy this contradiction, this paper proposes to model noise in function space instead of output space. We propose Functional Generative Models (FGMs), where each point (æ,, y:) comes from its own (unseen) function, fe., which fits it: y: fo, (r.). FGMs don't assume the existence of a privileged function fo-, but consider a distribution over functions P()), see fig.
Most supervised machine learning is based on variants of empirical risk minimization (ERM), which searches for a single function that best fits the training data. There, the training objective acts in output space, comparing the true answer with the prediction. In contrast, assuming that data comes from FGM, derive the Functional Risk Minimization (FRM) framework, where training objectives act in function space. Although the full version requires a high-dimensional integral, we derive reasonable approximation that scales to training neural networks.
Recently, neural networks have been observed to generalize despite memorizing the data, contra- dicting the classic understanding of ERM (Zhang et al., (017). Interestingly, find connection between FRM and recent theory explaining this benign overfitting of parrpaaraeterrized neural networks under ERM.

---

Under review as a conference paper at ICLR 2023
6r
pla)
P(O)
ERM formulation
FRM formulation
yr fo- (z:²)
0 ~ p(0)
24
Yi ~ p(-1)
yi = fo. (xi)
Figure 1: For many common losses, ERM and FRM can be related to maximum likelihood under simple generative models. Red lines ending in a circle are stochastic, blue arrows are deterministic.
The main contributions of this work are the following:
1. We introduce Functional Generative Models, a simple class of generative models that as- signs a function to each datapoint.
2. We derive the Functional Risk Minimization framework, compute a tractable and scalable approximation and link it to the generalization of weerpaaamettrized neural networks.
3. We provide empirical results showcasing the advantages of FRM in supervised learning, unsupervised learning, and reinforcement learning.
2 BACKGROUND AND RELATED WORK
2.1 INFERENCE AND RISK MINIMIZATION
In parametric machine learning, the user specifies dataset ((x,, 3t)) و parameterized function class fo, and a loss function L (y. fa()).. Our goal is to design learning framework that provides the i that minimizes the expected risk over unseen data: min, E [ (v. fx())) However, since we do not have access to unseen data, we canmot compute this expectation.
Empirical risk minimization (ERM) machine learning, we often rely on variants of ERM where a loss function L evaluated on the given dataset is optimized, i.e., ming Ei=1 fo However, what we want to have low expected risk (test loss), not empirical risk (training loss). In general, the best choice for training objective depends on the loss function L. but also on the (known) functional class fe and the (unknown) data distribution P(x, y). Often, ERM can be seen as doing maximum likelihood by assuming very particular noise model for the data that makes P(y(z) function of Pyylfo (x)) for some unknown, but fixed, d* However, in general, the user- defined loss function and thus the optimal f*, need not have any relation to the data distribution.
Bayesian learning The Bayesian setting explicitly disentangles inference of P(y(z) from risk minimization of L. However, it usually assumes the existence of a true 6* and further assumes comes from some known prior q: 0" ~ q(-). Then, similar to maximum likelihood, the Bayesian setting often assumes noise model Plyl (xx)) on the output. Thus inference about the posterior, P)) x (0)) P(D(0), becomes independent of the loss. Only in the final prediction step, the loss function is used, together with the posterior, to find the output with the lowest expected risk.
Relations to FRM Similar Bayesian learning, Functional Risk Minimization benefits from clean distinction between inference and risk minimization. However, FRM assumes fundamental aleatory noise in function space P(0), not to be confused with epistemic uncertainty in the Bayesian setting. Similar to ERM, FRM aims at only using a single parameter 6* at test-time, which avoids the challenging integration required in the Bayesian setting and its corresponding inefficiencies.
2.2 RELATED WORK
FGMs essentially treat each individual point as its own task or distribution. In this way, FGMs are related to multi-task learning (Thuu & Pratt 1998) and meta-learning (Hospedales et al. 2020). Within them, connections between learning
learn and Hierarchical Bayes the most rele- FRM closer to works looking distances in parameter space (Nichol al.] 2018) using implicit
vant Tenenbaum 1999; Grifiths et al.] 2008 Grant et al., 2018). Implementation-viiee
2

---

Under review as a conference paper at ICLR 2023
High functional loss
Empiical
loss
tunctionll
(a) Changes in light and translation naturally cause large, but struc- tured, variations.
(b) Images with high and low functional loss for series of fixed empirical losses, when predicting edges of the image on the left. The model a simple two- layer fully- colyycoonvlltiinnll network. One can see that images with low functional loss retain most of the structure despite having high errors in output (pixel) space.
Figure 2: Functional losses provide a way to capture structured noise, typical in natural settings.
gradients (Lorraine et al. 2020 Rajeswaran al. 2019). However, these are still fundamentally ERM-based as noise is modeled in output space within each task.
Other works have noted the importance of funetion space for applications such minimizing catastrophic forgetting in continual learning (Kirkpatrick et al.) (2177) optimization (Martens & Grosse 2015). exploration in reinforcement learning (Fortunato et al. 2017). Information geom- etry (Amari 2016), formalizes the geometrical structure of distributions using tools from differential geometry. contrast, we leverage stochasticity in function space for modeling and learning.
Multiple alternatives to ERM have been proposed, particularly in the multi-task setting, such as adaptive (Zhang al. 2020) and invariant risk minimization (Arjovsky al. 2019). It also rele- vant the line aiming at flat minima (Hochreiter & Schmidhuber 1997 /997immii sharpness (Foret al. 2020) in order to improve generalization on standard supervised learning. In contrast these works, our perturbations are per-point, and they come from the data distribution giving rise to the noise, instead of a regularization made on top of ERM with classic loss functions. Other works proposed per-point adaptations to tailor a model to each specific input either to encode an inductive bias (Alet et al.) 22200: 202I) or adapt to a new distribution (Sun et al., 2019 Wang et al.) 2020). However, these adaptations fine-tune ideal model. In contrast, in this work, per-point models are not a mechanism, but a fundamental part of reality, which then defines losses in function space rather than output space.
imperfect model trained with ERM to get it closer to an
FUNCTIONAL GENERATIVE MODELS: SAMPLING PER-POINT FUNCTIONS
3.1 DESCRIPTION
In machine learning, we want to reach conclusions about a distribution P(x, y) from a finite dataset ((r;, v))) However, there is no generalization without assumptions. From convolutions to graph neural networks and transformers, most research has focused on finding the right inductive biases for the mappings + y. However, much less research has challenged the assumptions about the uncertainty of those mappings: (v) For instance, whenever we minimize mean-squared error on an image-prediction problem we doing maximum likelihood assuming gaussian noise in pixel space. However, the actual noise is usually much more structured, as we show in figure 2
In this work, start from single principle, which call Functional generative models (FGMs): we model each data-point (zi, 3:) as coming from its own function fe, such that = fo, (æ:) and 0. ~ P(O). Notably, P(0) is unknown in the same way that we do know P(r, y). FGMs can be seen as a special type of hierarchical Bayes (Heskes 1998 Griffiths et al. 2008), where each group has single point, the lower-level deterministic and each 0, an unobserved latent variable.
Example: predicting house prices with linear regression Let's consider predicting the price of a house given its surface area using linear regressor: y = Az + 3 and the mean mean-squared error loss function. ERM would simply find the X, 3 leading to the lowest squared error on the training data. This is equivalent to doing maximum likelihood on a gaussian noise model y/ ~ N ((r; + 3, o²) with constant a. However, this may be suboptimal. For instance, we intuitively know that prices of bigger houses tend to be higher, but also have larger variances: we expect the price of a large house to vary by 500k, but we would not expect the same 500k variation for small house.
3

---

Under review as a conference paper at ICLR 2023
priccts)
surface/m
surface
skge
(a) Functional subspaces (lines in this example) that fit each point in the dataset (top plot). Each line in is colored according to its datapoint.
(b) The hest parameter distribution (in green) being quite certain in the offset 99 and uncertain in the slope A.
Figure 3: Functional generative models for a linear function class in house price prediction. Since only have two parameters, we can plot the function space 2D on the bottom-left of each fig-figure, with the actual data is plotted on the top-right.
When using the FRM framework, assume that, for each house (I:, 34) there different X, Bi, satisfying y4 = X;2; + Br. For instance, we may believe that agent commissions vary and are well- modeled by 3., and that the price-per-metter-ssuare (captured by A:) changes depending the neighborhood. This the modeling made by FGMs, which is more flexible than the output-level noise model corresponding mean-squane seuan- error. We show this effect figure
3.2 PROPERTIES OF FGMS
FGMs model the arbitrariness of dataset definitions A dataset implicitly defines which points belong the data distribution Pax, y) and which points do not. For instance, dataset of houses sold in Boston in the last 5 years, doesn't I contain houses sold in other cities, or Boston houses sold in 2005. Each of these categories would follow a slightly different distribution and, using Hierarchical Bayes, we could model them as similar parameter assignments to a single function class.
More subtly, even the dataset a single city encompasses multiple distributions, such as houses from different neighborhoods, years, colors. These hidden intra-distributions are source of noise when not described in the input. the absence of any information, the least restrictive assumption is that each point comes from its own distribution, giving rise to what refer noise. It therefore natural to use Hierarchical Bayes to model the differences in P(y (z;) from a single 0; P(0).
FGMs entrust what the user already trusts A user needs provide learning framework with three ingredients: dataset ((x,, ())) 1. a function class fe. and loss function L. Compared to the Bayesian setting, FGMs don't assume an independent noise model, which may have little connection with the user specifications. Instead, they leverage the user's trust in the function class fo to be a good model of the mapping + y. They simply go one step further and also entrust the uncertainty in that mapping to the same function class, which now also models individual mappings x; + yi-
FGMs encode structure through their function class FGMs draw their representational power from the function class fe. Therefore, if the function class has a particular constraint, the FGM will have a corresponding constraint in probability space. For example, for the function class of linear functions, the expectation of Prylar) is also linear. Similarly, as shown in figure 2 using convo- lutional neural networks we can create meaningful, structured noise priors image space. From graph neural networks and neural differential equations to probabilistic programs, FGMs leverage structured function maps to construct structured probability distributions.
FGMs can be arbitrarily expressive FGMs assume that P(ylæ) = PewP(e)) [fo(x) v). As just described, this need not be arbitrarily expressive. However, for some arbitrarily expressive function classes, such multi-layer perceptrons, their corresponding FGM can be shown to be arbitrarily expressive, in probability space. We formalize this the following definition.
Definition 1. Given function class F with parameterisation O define Functional generative model (P(x), P())) E FGM [Fa X as a probability density function P(z, y) E L x y with P(x)) [2) and y 8(fo (z)),
(x)) ),0 0 P(0) E 1.2
4

---

Under review as a conference paper at ICLR 2023
grobabiltty denety
probabiitty denstr
probeislity dereaty
urbbabilty dersty
a)
b)
c)
Figure 4: ERM with common losses equivalent to maximum likelihood under an FGM that is only stochastic in the output parameters. The particular distribution depends on the loss: a) MSE with a Gaussian b) LI with a Laplace c) cross-entropy with a Gumbel d) accuracy with a delta plus flat distribution. In practice, the axis for "other parameters" will often refer to thousands of parameters.
Note that P(0 E ②) and (( E 2) are independent and is deterministic given x; 0; see figure 1
Theorem 1 (Universal Distribution Theorem). Let q(x,y) E L x VI, = =0.1] [0" c IR, [0, 1]mh C Rm be given probability density distribution function. Let Fé be the class of 3-layer neural networks with sigmoidal activation function and neurons in the hidden layer. For any c O, 3K and a functional generative model (P(x), P(0)) FGM [F5 X s.t. Drv ((O)) g) < with Drv being the total variation distance. (Proof in appendix I
FGMs is a superset of some instances of ERM In appendix B we prove that ERM for four com- mon objectives (MSE, LI loss, accuracy and cross-entropy) can be seen as subcase of maximum likelihood on an FGM where all the stochasticity is restricted to the 'output' parameters. Figure A provides a visual intuition on how empirical losses correspond to functional losses in output space.
FUNCTIONAL RISK MINIMIZATION LEARNING IN FUNCTION SPACE
Now, we look at the supervised learning problem under the FGM assumption.
4.1 MATCHING PROBABILITY DISTRIBUTIONS IN FUNCTION SPACE
We start with our goal minimize the expected risk, impose the FRM generative model assumption and do basic math manipulations. In the derivation, whenever we use P(0) we refer to an unknown probability distribution entirely characterized by the data distribution P(x,, y) and function class f.
arg min Ev.y [C(y. [x)( = (1) 2o (0)
Qoy (0)
Qug (0)
0" / C ((o())) P(O)P(2))OOL = (2)
(())
arg min 0"
/ P(0) log (e- (e (o)) dd (3)
arg min 6-
/ P(0) log (1 Z(0*)
arg min
2(0') ) dB (4) Figure 5: Finding the pro-
jection of the unknown dis-
P(()) E (. log ((0)))- = (5) tribution P(0) to the fam-
arg min g+
Z(0*)
ily Q (f) of probability dis-
arg (P(O), Co. (0)) log (Z(0")).
(6) tributions in function space.
Here 03 (green) is best.
with H(P, Q) being the cross-entropy operator and Qo- (0) =
- F...((o( (+)) (+)
Z = Sge ((fo()))) (=r) d0 being class of probability distributions and their normalizers. [
To gain
intuition, first observe that the second term - log Z(0") log 1/Z(0*) log 1/ (L small I e (o d0; i.e. sharp distribution. Now, we can see that the first term encouraging the matching of two probability distributions in function space:
(2) is a label-independent regularizer that penalizes 0- leading to
1. P(0): the unknown data-dependent distribution, which does depend the loss func- tion L. This target distribution is defined entirely by the model class f and the unknown data distribution P(x, y), which we will have estimate from the training data.
5

---

Under review as a conference paper at ICLR 2023
2. Qo- (0): class of probability distributions which depends the loss function L and the 8" used to make predictions, but not on the labels. This approximating distribution makes parameter 0 more likely the closer the function is to fo- according to the problem- specified loss L. Intuitively, is a gaussian-like distribution centered at 0", with a metric that captures the differences in task space. This will be formalized in section 4.2
This equation also shows that we need not know the exact shape and distribution of P(0), which could be very complex without further assumptions. We only need to know its 'projection' particular class of probability distributions defined by the task at hand. This also happens in ERM- based learning: we need know P Pyy2z) in order estimate a + y map.
We would like to optimize equation 6 but we do not have access to samples for P(0), we only have (ar, y) pairs. However, we can compute the cross-entropy on P(y(a) following the FRM generative model. Thus, for given dataset D Dirain = ((Ig, 3:)) the FRM objective is:
arg max 6"
Σ (mi))e)
log
(c d0,-
(z:
Note that often we will not have access to the true input distribution P(x) to compute Ey [Co (o), (x)].. that case, we can also estimate it from samples.
4.2 APPROXIMATING THE FRM OBJECTIVE BY LEVERAGING OVER- PARAMETERIZATION
Equation 7 is an integral in high dimensions under non-linear constraint. In general, this is well- known to be computationally challenging. Fortunately, for this particular class of systems, we can rely on over- paramterization to propose reasonable approximation. First, observe that all constraints fo, (I;; = Yi are independent and that they all have a viable solution, only trying fit each single data-point (I y:) with the entire parameter set 6,, For instance, even a constant model fxx = e fits the data with G = y Vi. In other words, the system (0", ..,.....,,, 0n) ) is always peeepparamtteiize.
sanity check,
Moreover, is often extremely wee-pparameterized. For reasonably parameterized models this is in- deed the case: even small models of 10 parameters (compared to modern models of more than 10 parameters) may be underparameterized w.r.t. the entire dataset, but extremely weer-paaaetterized w.r.t. fitting a single data point. Therefore, similar to the Neural Tangent Kernel literature (Jacot et al. 2018) for extremely wide neural networks, we can assume that a very small perturbation will be enough to fit each datapoint
Now, we assume that we only need to analyze small perturbations A, around a parameter 6* for A << 1. We can therefore take the Laplace approximation of the probabílity distribu- tion we want to fit and assume it is a Gaussian with mean 0": NN00, H,, (1, );,k 8 (c)
Similarly, we can take the first-order Taylor approximation of the function f94a (x.) Re fo(a.)+Jffo(an .AA, assuming it is linear. Omitting the normalizer term, this leads to:
dA
arg max
Σ log (Alssta))HoddaaTTT...... (#L.01)
dA,.
Z(0*)
Under these conditions, computing the likelihood of fe+a, fitting x; involves integrating a gaussian distribution over either subspace (for regression) half-space (for binary classification).
Regression We first note that the integral of the gaussian under constraint can be seen as the pdf of yı (z.) A + fo(x;), A NO (.,) Because it is a fixed linear transformation of a gaussian distribution it can also be expressed as a gaussian. In particular using the notation
Note that this justifies that there is large prohability mass for 10, 0 << 1, but it does not justify that this is accurate approximation of the entire integral. However, this is common and useful approximation.

---

Under review as a conference paper at ICLR 2023
J4 == (z)) we have p(y)) N (fo(z)), (f H. Computing its log-likelihood we obtain the following training objective where both J: and F.C. depend on 0:
arg min Σ (ui (x))))))) (I. (o- +
Σ log
(9)
Classification For binary classification the solution is similar, except that we integrate over a half- space instead of a shper-ppane. Thus, we take the gaussian cedf (complementary cumulative distribu- tion function) instead of the gaussian pdf. Therefore, to maximize the logprobability of a function fit- ting a point, we minimize the gaussian logedf of the signed distance function to the decision bound- ary: ming Σ logedf (A,) where A, == sign (o:ff(z;)) 2) (f (z))),)) the signed distance to the decision boundary.
10, 0 ,
Note that in classification the best perturbation is not zero, but a very negative (i.e. opposite to the gradient) value, since this implies that the parameter 0 is well within the correct classification region. This is also similar to regular ERM in binary cross-entropy, where we maximize the sigmoid, which has a very similar shape as the gaussian edf.
For multi-class class classification the integral is over an intersection of C - half-spaces (comparing each class with the correct class y:). The efficient integration in that case is still an active area of research (Gessner et al., 2020). Two potential alternatives may be practical: turning the training of an n- n-way classification into n binary classifications, and linearizing the softmax of incorrect classes jointly instead of linearizing each one independently.
4.3 FRM MAY DO EXPLICTLY WHAT OVER-VVV PARAMETERIZED ERM DOES IMPLICITLY
It has been observed that neural networks often generalize despite memorizing the training dataset (Zhang et al. 2017 Poggio et al. 2017 Belkin et al. 2019] Nakkiran al. 2021]. seemingly contradicting classic understanding of generalization ERM, which relies on controlled capacity.
FRM implicitly assigns to every datapoint (I;, Wi) its own latent model So, which fits it: Sa, (x:) yt. In this way, we can turn a model fo into an over- wer-parameterized hyper-model. Although unobserved in FGMs, the previous Taylor ver- sion of FRM becomes equivalent to this optimization:
-035 -C33 -033 C3S ods
Figure 6: Minimal functional adap- tations using generalized linear model with Fourier features.
Σ 100--01imp B min Σ min fo, 2.
min 01 Ap......o.
10 105-0l2y,cr 0l
(10)
0 (m.)=
where explicit 6g are sought that are as close possible according to the metric M. Whereas ERM finds the function that best fits the data among class of simple functions, FRM finds the simplest moper-model to explain the data, related to the principle of Occam's Razor.
This can be seen finding the simplest hyper-model (01.... 0,) that fits the data. Simplicity is measured as the distance of parameters being close to central parameter given a metric that captures the relationship between the function class fa and the loss L. This encourages each independent function to be close to the central one, and thus all functions being close to each other, as shown in figure 6 This is related to the line of research exposed by Bartlett al. (2021), which conjectures that ERM under gradient descent may implicitly find a function with two components fstable + Ssiiky, such that the spiky component has negligible norm but allows overfitting. In this regard, FRM can be seen as explicilty searching for the smallest necessary perturbation for each point.
EXPERIMENTS
To scale to neural networks, we leveraged the Taylor approximation in section 4.2 However, that re- quires inverting Hessian, which is usually too big to even instantiate in memory. We bypassed this

---

Under review as a conference paper at ICLR 2023
ERM loss /RRM loss
1-D
10-D
15-
12-
test
30-
test
11
FRM better
ERM better
training
1.-
training
log(oftset noise
scale noise)
logloftset noise scale noise)
Figure 7: Ratio of train and test error between ERM and FRM function of the ratio between noise in the scale vs. offset components in I-D and 10-D linear regression. As expected, we can see that ERM always has lower training loss well as slightly lower test loss (12% lower) when its assumption (gaussian noise only on the offset) is perfectly satisfied. When noise is heteroscedastic, ERM has up to 40% higher test error. In 10 dimensions, the advantage of FRM even starker: ERM can have 4 times more test error than FRM, despite having lower training error.
problem by I) relying on iterative solvers to avoid the cubic cost and 2) materializing only Hessian- vector products. To do so, we use JAX (Bradbury al. 2018) and the jaxopt package (Blondel et al. 2021), which implements implicit gradients.
5.1 LINEAR LEAST SQUARES
To better understand the trade-offs between FRM and ERM we analyze the simple case of linear regression under mean-squared error risk. We consider d-dimensional input and one-dimensional output. The classic ERM solution minimizes the risk on the training data: minx,8 Ei.,,)) (A I; + B - y²)² This is equal to doing maximum likelihood on a fixed gaus- sian noise on B3 Thus, we expect ERM to do well in this situation, but not necessarily otherwise.
For linear regression with squared loss, the Taylor approximations in section 4.2 are exact. Furthermore, both the Hessian and the gradients are independent of the parameters, which further simplifies the objective function to just specific re-weighting of the per-point risks:
minx,3 Exx.))) ( [K. with H E. 1J, [x, 11] Figure 7 shows that indeed ERM does slightly better with gaussian noise in the bias, but FRM does much better when the noise is en- tirely in the slope. We also observe that the FRM is more than 4 times better in higher dimensions.
5.2 VALUE FUNCTION ESTIMATION
We demonstrate here that the proposed approach can be broadly applied on an illustrative offline value estimation task using the mountain domain (Sutton Barto 2018). We consider the problem of learning a linear value function using a 15 x 15 grid of radial basis functions (RBFs) using the 1-step temporal difference (TD) error (Sutton] 1988) the training loss function and using sampled transition gathered by a near-optimal policy. Both approaches were optimized with stochastic gradient descent with constant learning rate best suited for it, selected by grid search over
and batch size of 256. Performance is then evaluated using the root mean squared error (RMSE) between predictions and the true values on unseen samples.
We consider two different arrangements of RBFs, a uniform layout and one that is denser towards the center of the environment. Note that although the true value function has a discontinuity spiraling out from the center, which might benefit from finer resolution, the more poorly conditioned nature of this non-uniform arrangement of features makes the problem harder, as can be seen in figure g We see that FRM is competitive in the easier of the two cases while outperforming ERM by over 20% in the harder one. We hypothesize that TD loss is commonly subject to complex noise that severely hinder ERM when its features bootstrapping (C(s, r, s') := (fo(s) r fo((s))) the temporal difference error is inherently functional through the term fo(s') affecting the label.
poorly aligned. Furthermore, due the use of
8

---

Under review as a conference paper at ICLR 2023
10
Objective FRM
Objective
FRM
RMSE
RMSE
2
0 2500 5000
7500 Iteration
10000 12500 15000
2500
5000
7500
10000 12500 15000
Iteration
Figure 8: Comparison of the RMSE for ERM and FRM for the learned value function in mountain car under a fixed policy using a temporal difference loss with different features: (left) using a uni- form grid of radial basis functions, (right) using distorted grid of radial basis functions denser in the middle. Solid lines are the average over 20 seeds; shaded areas show the 95th percentile interval.
0.9
ERM
0.90
0.8
FRM
0.7
0.5
0.75
ERM
0.5
ERM FRM
ERM
0.4
FRM
0.5
FRM
Figure 9: Accuracies of an MLP trained from latents of two CNN-based VAEs, trained with ERM and FRM. FRM provides small gains in vanilla MNIST, and large gains in all three variants.
5.3 FGM-BASED VAE FINDS BETTER REPRESENTATIONS WITHIN STRUCTURED VARIATIONS
To better understand when FRM works better than ERM, we build Variational AutoEncoder (VAE) on top of MNIST (LeCun et al.) 1998) and combinations of two popular variations: colored MNIST (Arjovsky al.] 2019)) and translated MNIST (Jaderberg et al. 2015). We build a vanilla VAE with MLP encoder and CNN decoder. Then, we evaluate the quality of the representation to do classification for the vanilla VAE and an FGM-based decoder where noise is modeled in function space. For FGM, train small MLP top of the latent representation, with stop-gradient, and measure accuracy depending on the size of the latent. We see that MNIST, where natural variations in orientation, translation, and color have been unnaturally removed, some gains exist but are small. In the datasets containing variations in color or translation, the FRM gains are substan- tial. This is because noise in CNN weights can easily explain these structured variations, as shown in figure g Similarly, papers such as Deep Image Prior (Ulyanov et al. 2018) have argued that neural networks are good models for real-world variability, making FRM particularly appealing for modeling real-world data. Results are shown in figure
CONCLUSION
The main limitation of FRM in its current form is its compute cost. Thanks the approximations proposed in sections 4.2 and 5 we can run FRM on a ResNet-50 using a single GPU, but with a prohibitive iteration cost. However, long term, FRM could be orders of magnitude more efficient than ERM-based approaches. As explained in section 4.3 under- under-parametent parameterized FRM may behave similarly to over-paramen parameterized ERM by making models have n times more parameters .i..... 9n- There, each 0: is instantiated on the fly for loss computation and thus doesn't ' need to be in memory, this could provide orders of magnitude of benefit for modern datasets where often 106.
In the last years, there has been a clear tendency towards building large models capable of per- forming many tasks which were previously modeled individually. FGMs propose the natural step to model the diversity in these datasets in function space rather than output space, allowing for richer and more meaningful noise models. Despite noise being pervasive across real-world data, modern deep learning approaches still largely use simple unstructured noise models. As we keep moving to- wards larger, more varied datasets, properly modeling the internal data diversity will become crucial. We believe FRM provides first step towards effective solution.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ferran Alet, Maria Bauza, Kenji Kawaguchi, Nurullah Giray Kuru, Tomas Lozano-Perez, and Leslie Pack Kaelbling. Tailoring: encoding inductive biases by optimizing unsupervised ob- jectives at prediction time. arXiv preprint arXiv:2009. 10623, 2020.
Ferran Alet, Dylan Doblar, Allan Zhou, Josh Tenenbaum, Kenji Kawaguchi, and Chelsea Finn.
Noether networks: meta-learning useful conserved quantities. Advances in Neural Information Processing Systems, 34:16384-16333 2021.
Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:190..02893, 2019.
Peter I. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: statistical viewpoint. Acta numerica, 30:87-201, 2021.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical barinnare trade-off. Proceedings of the National Academy 2019.
of Sciences, 116(32):15449-1555
Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares- López, Fabian Pedregosa, and Jean-Philippe Vert. Efficient and modular implicit differentiation. arXiv preprint arXiv:2105.15188 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, github, 2018. URL
http: /ttt:///ithub. com/
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv: /706 770.. 10295, 2017.
Alexandra Gessner, Oindrila Kanjilal, and Philipp Hennig. Integrals over gaussians under linear do- main constraints. In International Conference Artificial Intelligence and Statistics, pp. 2764- 2774. PMLR, 2020.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learnng as hierarchical bayes. arXiv preprint 1801.088000 2018.
Thomas L Griffiths, Charles Kemp, and Joshua B Tenenbaum. Bayesian models of cognition. 2008.
TM Heskes. Solving huge number of simular tasks: A combination of multi-task learning and hierarchical bayesian approach. 1998.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey, arXiv preprint 7XXx:20044054499 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Ad- vances in neural information processing systems, pp. 2017-2025, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzaiion with gumbel-softmax. arXiv preprint arXiv: 1. 1.01 144, 2016.
10

---

Under review as a conference paper at ICLR 2023
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3511-3526, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-222) 1998.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pp- 1540-1552. PMLR, 2020.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:/611.00072,, 2016.
James Martens and Roger Grosse. Optimizing neural networks with curvature. In International conference on machine learning, pp. 2408-2417. PMLR, 2015.
approximate
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12): 2021.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. CoRR, 02999
2018. URL
Tomaso Poggio, Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary, and Hrushikesh Mhaskar. Theory deep learning ii:: explaining the con-overfitting puzzle. arXiv preprint 1tt1::80000007.. 2017.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with im- plicit gradients. Advances in neural information processing systems, 32, 2019.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training for out-of-distribution generalization. arXiv preprint arXiv:1909.1333 2019.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9-44, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Joshua Brett Tenenbaum. A Bayesian framework for concept learning. PhD thesis, Massachusetts Institute of Technology, 1999.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 1998.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446-9454 2018.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell, Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017.
Marvin Zhang, Henrik Marklund, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adap- tive risk minimization: A meta-learning approach for tackling group shift. arXiv preprint arXiv:2007.02931, 8:9, 2020.
11

---

Under review as a conference paper at ICLR 2023
Table 1: FRM outperforms ERM in a small CNN environment despite ERM having 0 training loss.
Furthermore, the Hessian can be enough to express the dependence on the loss function.
Objective
Train
Test
positives
negatives
positives
negatives
ERM
positives
.000 = .000
.283 = 0016
.130 = .006
.278 = .013
negatives .366 .220
.000 = .000
.323 + .018
.119 = .005
FRM
positives
.052
.002
.009 = .007
.085 1 .004
.124 = .006
negatives
.131 + .010
.022 - .002
.136 + .009
.0841.004
A FUNCTIONAL NOISE IN A CNN
To show the value of the Taylor approximation, we create a dataset by sampling different parameter assignments 4-layer CNN architecture. The CNN takes in CIFAR-IO 10 image and outputs a real number. We provide only 8 labels to each method, allowing empirical risk minimization to easily memorize the dataset. Despite FRM obtaining substantially higher training losses (.000 .052), we observe FRM obtains significantly less test error (.125 to .085).
We also test the ability of FRM to modify its training depending the loss function. Although this is obviously the case for ERM, in approximate FRM the loss function enters only in an indirect way, affecting the hessian in equation 8 We modify the objective by creating two different losses, which assign zero loss to labels that are either positive or negative, respectively. Table I shows that indeed FRM performs better when trained and tested the same loss (0.085 0.128).
B PROOFS OF EMPIRICAL LOSSES BEING SUB-CASES OF FUNCTIONAL LOSSES
B. MEAN-SQUARED ERROR AND L1 LOSS AS A FUNCTIONAL LOSSES
Let our dataset D Drrain = {(..:)))); y.)) Vi E IR!,, and let LMSE = 1 E. (f - y:) i.e. the mean-squared error loss.
Lemma 1. For any arbitrary function class fe.s(z) expressible as fa,a(x) = folx) + B, there exists a functional loss restricted to functional adaptations B, that only change B - Br which is equivalent to the mean- seuun--uarrdd error loss.
Proof Since we can only change 3 there is single solution to the per per-point constraint: fo(x,) fo(2;) + 3 y/ + 3, y/ fo(x;). We can now model the probability distribution over functions F (0, 0, B, LASEE) as gaussian centered at (0,3). 3). Since 0 doesn't change, this will just be (o,,N33,350 Maximizing the mean of the log bog-probabblitiis will result in Σ, log (B,- =
N (B, - = = E. - fa(x,) 3) = E, - fa,s(z;)) = LMSE.
Of note, the Gaussian model of the functional distribution satisfies
F (0, 3, B, (MSE) = N ((0, 3x) - (0, x e -18-8,8 = BB (mee((e.s.ooo...
This is because for all x, LMSE (z) for .a (z)) = (f,,(()) - for (x)) 3112.
Finally, we note that the entire derivation can be equivalently followed for the L1 loss by swapping |- for |- and the Gaussian distribution for the Laplace distribution.
B.2 CLASSIFICATION ERROR AS A FUNCTIONAL. LOSS
Let now look at multi-class classification and let our dataset Dirain {II; y:)) i/ {1. (1...., C}. Our function class will output in unconstrained logit space I and we define Lete I -E [[y = arg max, (fo (fo, (x;))) i.c. the classification error. As in previous sections, abusing notation will refer to [yy = arg max, (J:))( as 1 [y: fo,s(x:)].
12

---

Under review as a conference paper at ICLR 2023
Lemma 2. For any arbitrary function class fo,s(z) expressible as fo,s(z) fo(a) 3, B R constrained on fo(z) being finite, there exists a functional loss restricted to functional adaptations 8. = 0 that only change 3 + Bg which is equivalent the classification error.
Proof We will show that a solution is given by F(0, B, Ldts) = p 3) (1 p) No (0, or) (3), with
e-1 C+e-I (0, 1). In other words, a specific positive (note the open brackets) combination of an infinitely-sharp distribution (Dirac's delta) with an infinitely-flat distribution. Given a fixed p. 0.3. the probability of y; arg max, .8. (2,) will be equal to p- 1 = arg maxe (fo.s).l yl + J. This comes directly from the definition of the functional prob- ability distribution: with probability p. we have (0,, B,) (0, 3) and thus the result depends solely
on (0, 9); with probability (1 - p) the logits are perturbed by an infinitely strong noise and thus the arg max will just be uniform distribution over the classes, i.e. c.
Now, the average log- ligllikelihood of the functional loss will be:
1-P ) p [y = [1,(
log
1 1 P
1
1[/ = [0,s( + +11-D)/C)-
log
+
log
C
-1-p)/C
i=1
1 p
p (1 (1-p)/C 1
log
+ +log
1l = fo,p(x]]]
C
(1 - (1-p)/C
n2 iml
log 1-p
pC
+ log
1+ 1 1-p) Ldds =
C
llo (C + e 1) Leds-
where in the second step we observe that the log term within the sum is zero when yi + (e;) and, in the last step, we have set = C+c-I. which by construction is in (0, 1). We can now easily see that this is equivalent to Lds up to constant additive term, which will not affect any optimization.
e-1
B.3 CROSS-ENTROPY LOSS AS A FUNCTIONAL LOSS
Continuing in multi-class classification and let our dataset Deruin = (1) Yi E f1.....C). Our function class will output in an unconstrained logit space BCC and we define LCE D iisi log a (fe.z)y. i.c. the cross-entropy loss. Here, a-)) corresponds to taking the e-th com- ponent of the softmax of a given logit to obtain the probability of a given class e given the logit predictions.
Lemma 3. For any arbitrary function class fo.)(1) expressible as fa,s(x) = fo(x) + B. 3 R.C, that only change B - 3,
there exists a functional loss restricted to functional adaptations which is equivalent to the cross-entropy loss.
Proof As shown in (Jang et al., 2016 Maddison et al. 2016 if we have logits % = So (x:)) + go we can sample from the probability distribution of distribution equal to a (7) by e = arg max, (7% + gd) where each 9, follows
independent Gumbel distribution, i.e. 9i - log(- log u,), Uq (0, 1). This gives us a trivial expression for a functional distribution over which to make maximum likelihood: 3, 3 + G, where G consists of e independent Gumbel noise variables. This is because, since 33 lives in logit space, adding noise to 3 is equivalent to adding noise to the logits. Finally, since the cross-entropy loss is the maximum likelihood assuming probability distribution given by the logits and we have shown a functional distribution with the same distribution, performing maximum likelihood on that distribution equivalent minimizing the cross-entropy loss.
C UNIVERSAL DISTRIBUTION THEOREM
Definition 2. Given function class F with parameterisation θ we define a Functional generative
model (P(x), P())) E FGM [Fe, X1 as probability density function p(x,z) y) E L x x }) with P(x) [2) and y fo(x), 0 P(O) E 12)(0).
13

---

Under review as a conference paper at ICLR 2023
Note that, in particular, P(0 ②) and P(r E X) are independent and y is deterministic given 2; 8; as shown in figure]] I
Theorem 2 (Universal Distribution Theorem). Let q(x,)) y) E L2IX x VI, = [0. 10" C R [0, C Rm be given probability density distribution function. Let Flg be the class of 3-layer neural networks with sigmoidal activation function and l neurons in the hidden layer. For any E O, 3K and a functional generative model (P(x), P(O)) FGM [ XX] s.t. Drv (((()),(())) q) c, with Drv being the total variation distance.
For the first layer we use deterministic weights with arbitrarily- slope to implement the functions e, 1]. For the second layer,
a|l 2 cy]l for all coordinates 1 S 1 S v and c2 = again use deterministic weights implement functions 1 [[x E [01,a] a1 + e) x ..... [an, an + c)] to determine whether given input is within a hyper-cube of side 6. Exactly one of those two-layer nodes will be active for any given input. From the node corresponding
[a1, Q1 + +6) x x [on, an +e) the each of the output nodes there are m weights 0, we assign them a distribution equal to f1:m ~ P(y] x (d1,...;a,)) Because P(ylæ) is continuous, P(y|æ (a1...., an)) will be arbitrarily close to (()(x)) for any in the hyper-cube [a1, 01 + c) x ... x (an, + c) for sufficiently-smie
c.
We note that this universality also holds for a 2-layer neural network as well (also universal func- tion class). However, the prove for that case is more cumbersome and less insightful for our pur- poses.
D FURTHER UNDERSTANDING THE DIFFERENCE BETWEEN ERM AND FRM
The ERM assumption: by assuming that the training objective is equal to the test loss L, ERM can be suboptimal for certain P(0). like the house example on section 3] As shown in appendix B for many loss functions L, including most of the common ones, ERM equivalent assuming the functional generative model and then doing maximum likelihood on P(0) by assuming it has a form parameterized by 0 whose uncertainty is only the output offset parameters. In other words, the assumption equivalent to performing ERM is often strictly more assuming than FGMs.
For instance, consider predicting the price of different houses as a function of their size and having MSE as the loss. Doing empirical risk minimization with the MSE would be equivalent to doing maximum likelihood on the following price model: y ~ (fff(z)), However, noise to be heteroskedastic with higher variations for higher prices.Thun even if we are evaluated on MSE the test data, it may not be advisable to use it as our training criteria.
would expect
Similarly, consider a child learning a concept from examples on a textbook rather than from stan- dardized images of a dataset. Images may receive different illuminations from the sunlight, or be in different positions than we expect. These factors will produce massive changes in pixel space, but in very structured ways (fig 2 However, humans still easily grasp the idea because the "conceptual" noise is small.
How can we have more meaningful noise models? By construction, we will often believe that the function class fo is a good characterization of the relationship between æ and It is thus a natural assumption to define a noise model by leveraging the function class itself. More concretely, we can think of generative model of the data as first sampling the input v;, then sampling a function Ji (LC, 0) from some parameterized distribution over functions, which will depend on both the problem-specified loss function L as well as the function class Sq. Once the function and the input have been sampled, the output is automatically determined y; = fz,)) see the right of figure [
For example, in our house- prisc prediction, if we using linear model f(x) A Z + 8, then it makes sense to think about our data as coming from first sampling 24 ~ p(x) and (Ax, 3.) ~ F(C, (X, B)), then computing y4 X Iz + Bi, as shown the right of figure [ For instance, 3: can model different commissions or taxes, and X; can model the per-meter-squee variable across neighborhoods. Even if we about making accurate predictions in dollar-space, assuming our uncertainty is only in the offset term Br may be too restrictive.
price being
14

---

Under review as a conference paper at ICLR 2023
D.1 ERM Vs FRM FOR THE LINEAR CASE
Let us now take a deeper look at our linear regression example. We have a dataset Dirain ((r,, y:)]. depicted in the top-right of figure 3a with an arbitrary color per point. For every point, there is subspace of models (A,, 3x) s.t. X + 3 = Ys- Since we only have two parameters, we can also look at function-space space in 2-D, and plot the corresponding subspace for each point, in the bottom-left of figure Sa We observe that every point gives us a line in function space, which we plot with the corresponding color.
Our goal is to produce probability distribution P(A, () such that the sum of the log- log-densities of each line (Λ,, +,, is maximal. Intuitively, this means that each line should pass through high-density area of the probability distribution, but it does not mean that the line should be covered by the high-density area (which is not possible, since they're 're unbounded). This can be seen in figure 3b where all lines pass near the center of the distribution generating the data (marked in green).
We can further see that ERM with the MSE loss is equivalent to finding point (XERM BERM)) that minimizes the vertical distance to each line:
(AERM BERM)
Σ (3/s - Λx; - -8)
min X
min
Σ (v: - Axt;; - 3)
A.d.
>11'==
E(e: - 3)2.
min A,
Auai + -.
In contrast, if the probability distribution in parameter space is Gaussian, FRM involves taking the distance of the entire vector (A, 3), using the inverse covariance matrix as the metric. For cases where most of the uncertainty is in the slope, in figure 33b] ERM measures the distance in the vertical direction and FRM measures it almost horizontally, leading to different results.
D.2 VISUALIZATION FOR A SIMPLE FULLY CONVOLUTIONAL NETWORK
Figure shows the difference between MSE and its functional correspondent for a small fully-convolutional network mapping images to images fe. Images y with the same empirical loss /o=) could require very different functional adaptations to explain: millerffeeee
10' 10'-0' 0 0 For instance, one does edge detection and mistakenly translates its prediction a bit to the right, this small change in functional space could lead to large error in pixel space. Similarly, if we have a pattern detector and we slightly change its threshold, could make the entire prediction darker or lighter.
Conversely, we add unstructured noise onto our image, is be expected that it will have a high functional loss as no small perturbation of the function could simultaneously explain pure noise. That's indeed what we observe figure 2b when we look for images with high and low functional loss for fixed empirical loss. Images with high functional loss contain salt-and-pepper-ll that breaks the smooth pattern the original image. In contrast, images with low functional loss preserve the overall structure while uniformly shifting large blocks of pixels to much lighter color. the noise in our data is better represented by our functional class than noise in the output, we can take this into account to improve learning.
noise
15


Reviews:
[{"review_id": "JCWrLInMCJ", "reviewer": "Reviewer_6yNo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new basic data generation framework where each data point is associated with a function and the `noises' are assumed to be from the variation of its function rather than its output used in classical ERM.  Two corresponding models are proposed for unsupervised learning (generative model) and supervised learning.  ", "summary_of_the_review": "This paper proposes an interesting and valuable idea to extend the classical ERM to FRM. The benefits are well introduced. However, there are sill some confusions of the concepts, like the relation with hierarchal Bayesian. Please see Strength and Weakness for more details. ", "strength_and_weaknesses": "Strength\n\n1. the idea of associating each data point with a function is interesting.\n2. the paper is well-written and examples are given to show the motivation and clear some concepts.\n3. the functional risk minimization may extend the existing ERM to a new era.\n\nWeakness\n\n1.  Even the authors claims there are some difference between the proposed FRM to the Bayesian learning on page 2, it is still confusing to me that the proposed model, at least the functional generative model, is just a kind of hierarchal Bayesian model as illustrated in Fig 1. The authors claim that 'FRM only use a single parameter at test-time'. It looks weird because FRM believes every data point is associated with its own function f_{\\theta_i} then why and how you use single \\theta at test time? like generate new data? \n\n2. The similar thing happens to FSM for supervised learning. When training using (1) and predicting using single \\theta^*, the objective function for training and test will be different which implies that the assumptions behind training and test are different. Is that OK? Note that this is not happened in ERM where the assumptions behind training and test are consistent. Is there any effect from such property? \n\n3. There is no experimental evaluation on the proposed functional generative model in section 5. \n\n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Du3ZGCDjH83", "reviewer": "Reviewer_EeLk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a general framework Functional Risk Minimization (FRM), a general framework for scalable training objectives which results in better performance in small experiments in regression and reinforcement learning. FRM model each data point (x_i, y_i) as coming from its own function f_\\theta_i. The authors also show that FRM can be regraded as finding the simplest model that memorizes the training data, providing an avenue towards understanding generalization in the over-parameterized regime.\n\n", "summary_of_the_review": "The paper is marginally above the acceptance line. I would suggest to accept this paper. ", "strength_and_weaknesses": "Pro: The paper rethink the common assumption in machine learning community and proposes a new comprehensive model that can subsumes classic Emipirical Risk Minimization for many common loss functions. \nCons: 1) The computation cost of FRM in its current form is high. 2) Maybe the authors can discuss more why the FRM is really crucial and why we could not just modify the classic ERM for instance and get similar performance. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-oFAPwV8CQA", "reviewer": "Reviewer_vBxE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel framework for supervised learning problems. First, it introduces functional generative models which represent $ P(x,y)$ in terms of a latent variable $ \\theta$ that is sampled independently of $ x $ and then determines $ y $ as a function $ f_\\theta(x)$. This is contrasted to a standard additive noise model, where $ y = f_\\theta(x) + \\epsilon$. Using this noise model, the paper proposes functional risk minimization (FRM) as an objective to find the maximum likelihood parameter $ \\theta^*$ and proposes an approximate algorithm that relies on approximating an integral over all parameters by a local Laplace approximation.  ", "summary_of_the_review": "I think this paper clearly has some interesting and promising ideas. However, as currently presented I cannot support acceptance because of the significant issues connecting the actual algorithm to the motivation, lack of basic theory, and general lack of clarity.", "strength_and_weaknesses": "### Strengths\n\n1. This paper presents a very interesting and (to my knowledge) novel take on representing noise in a learning framework. This could be highly impactful as it may allow for more easily dealing with the variation in real datasets than standard MLE in additive noise models. \n\n2. The empirical results on linear regression with non-uniform noise seem to be strong, demonstrating that as the noise distribution changes, FRM begins to outperform ERM. \n\n### Weaknesses\n\n1. It is not clear how closely the implementable algorithm relates to the theoretical presentation and derivations. Equations 7 and 8 seem to be the core algorithmic contribution of the paper, but they are never formally derived and it is highly unclear under what conditions they will actually relate to the population objective. \n\n2. The implementable algorithm as proposed does not seem to be scalable. In particular, it requires both a significant approximation to integrate over parameters and then uses Hessian information that could be difficult to get for large models. This is born out by the fact that all the experiments, while interesting, are quite small in scale. \n\n3. There is no formal argument made as to why or when FRM will outperform ERM. In fact, there are no arguments at all showing that (a) the FRM minimizer will achieve low loss, (b) the finite-sample version of FRM approximates the population variant, or (c) the finite-sample FRM minimizer will achieve low loss.    \n\n4. The presentation is confusing.\n\n    a. The use of \"ERM\" is non-standard and confusing. ERM, as used in foundational work like [1], simply refers to minimizing an empirical loss as a proxy for an expected loss. There is no mention of any particular assumption about additive noise in the ERM principle. Moreover, the entire reason to use the ERM term is to contrast it with learning objectives like structural risk minimization. This paper instead uses ERM to refer to the noise model induced by ERM with particular loss functions by viewing them as MLE. It would perhaps be more clear to frame the novelty of the approach as replacing MLE under an additive noise model with a novel MLE under the functional latent variable noise model.  \n\n    b. Similarly, FRM is often used to refer to the noise model, which is in fact the FGM under the nomenclature introduced in the paper. The paper would be much improved by being more clear about the difference between the FGM modeling assumption and the FRM learning objective.\n\n    c. The FRM learning objective is never clearly and explicitly layed out. When the authors say FRM, what implementable algorithm are they explicitly referring to? This needs to be more clear.\n\n\n[1] Vapnik, V. (1991). Principles of risk minimization for learning theory. Advances in neural information processing systems, 4.\n\n\nMinor: There are a few typos throughout the paper. For example, in the abstract \"regraded\" should be \"regarded\". In the \"FGMs encode...\" paragraph on page 4, many spaces are missing after periods. In the Table 1 caption \"Furthermoe\" should be \"furthermore\". ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject