Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time
Authors: Anshul Nasery
Soumyadeep Thakur
Vihari Piratla
Abir De
Sunita Sarawagi
Venue: NeurIPS 2021 Poster (2021)
Keywords: continuous domain generalization
time-evolving classifiers
gradient interpolation

Abstract: In several real world applications, machine learning models are deployed to make predictions on data whose distribution changes gradually along time, leading to a drift between the train and test distributions. Such models are often re-trained on new data periodically, and they hence need to generalize to data not too far into the future. In this context, there is much prior work on enhancing temporal generalization, e.g. continuous transportation of past data, kernel smoothed time-sensitive parameters and more recently, adversarial learning of time-invariant features. However, these methods share several limitations, e.g, poor scalability, training instability, and dependence on unlabeled data from the future. Responding to the above limitations, we propose a simple method that starts with a model with time-sensitive parameters but regularizes its temporal complexity using a Gradient Interpolation  (GI) loss. GI allows the decision boundary to change along time and can still prevent overfitting to the limited training time snapshots  by allowing task-specific control over changes along time. We compare our method to existing baselines on multiple real-world datasets, which show that GI outperforms more complicated generative and adversarial approaches on the one hand, and simpler gradient regularization methods on the other.

Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time
Anshul Nasery *f
Soumyadeep Thakur'
Vihari Piratla Department of Computer Science Indian Institute of Technology. Bombay
Abir De Sunita Sarawagi
Abstract
In several real world applications, machine learning models are deployed to make predictions on data whose distribution changes gradually along time, leading to drift between the train and test distributions. Such models are often re-trained new data periodically, and they hence need to generalize to data not too far into the future. In this context, there is much prior work on enhancing temporal generalization, e.g. continuous transportation of past data, kernel smoothed time- sensitive parameters and more recently, adversarial learning of time-invariant features. However, these methods share several limitations, e.g, poor scalability, training instability, and dependence on unlabeled data from the future. Responding to the above limitations, propose simple method that starts with model with time-sensitive parameters but regularizes its temporal complexity using a Gradient Interpolation (GI) loss. GI allows the decision boundary to change along time and can still prevent overfitting to the limited training time snapshots by allowing task-specific control over changes along time. We compare our method existing baselines on multiple real-world datasets, which show that GI outperforms more complicated generative and adversarial approaches on the one hand, and simpler gradient regularization methods on the other.
1 Introduction
Many organizations operate their machine learning pipeline as follows: collect labeled data based on day-to-day operations e-g. user-clicks for a recommendation model; periodically (say every = 48 hours) train the model on the data collected from the past; deploy the model to serve requests of the next A period; repeat the cycle. Real-world examples of such usage include recommendation models for e-retails, loan approval models in banks. regression models to forecast future popularity of media content, classification malicious websites, and activity recognition models from sensor data.
A key motivation behind such periodic retraining is the implicit knowledge that the data distribution not stationary, requiring a revision of the decision boundary with time. While incorporating fresh data in training does make the trained model more current, we investigate if we can improve the training even further. In particular, when training the model at a time f, we know that the model will not be deployed on data samples in the training interval, but on data from the immediate future [1, + A] which could differ from even the most recent training data. Our goal this paper is to use this knowledge more centrally to train a model tailored to the intended deployment period.
Formal Problem Statement. We consider prediction tasks over an input space X' and output space ) where the joint distribution P(X.))) ))) evolves with time. The prediction model should therefore be conditioned on time as P(y]x.() for x E X, y E y, and time stamp 4. During training we are provided with labeled data from T time snapshots 1 S l2... < lr. Call these D D2 D2... D where ...) assumed to sampled from P(x, y)(,). The trained model
each D= ((x,, y:)
Eqqual Contribution
lanshulnasery@@maill com
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

will only be deployed on data from a time interval in the future which we denote as tr+ that is on examples sampled from P(x Note that we only evaluate performance once on time tr+1- differing from online learning where regret is computed incrementally. Following most practical setups, we assume that the temporal drift across time is not too high. Since the deployment time is in the future, we cannot assume the presence of even unlabeled target data at the time of training the model. Further, at different time snapshots, completely unrelated sets of x; may comprise a D This makes the problem much harder than standard time-series forecasting tasks, since we do not have values of the same instance at different points in time. Our goal is not to generalize to all future data, but rather to data relatively close in the future, since motivating application is one where models are retrained periodically. The former generalization would require very different models and approaches than what we propose. We assume that P(y|x x t) is implemented as general purpose neural network F(x, 1)) that takes as input x, L and outputs either (1) a vector of logits for classification tasks or (2) mean and other scores defining a continuous distribution over predicted y for regression tasks.
A solution often used in practice is to only train the model on recent data and/or apply a graceful ageing penalty on past data. However, this may not be adequate with data hungry modern networks, especially when non-obvious seasonality makes distant data relevant. So we seek general purpose solutions that can harness all available data for making optimal decisions into the future. Any such solution first needs to make the model time sensitive. This can be done by feeding time as an input feature, or by making all or some of the model parameters to be a function of time. However, that by itself cannot be sufficient to generalize to future data if we train the model to minimize loss on the training snapshots. In fact, sufficiently complex model can easily overfit on training time-stamps when the number of instances per snapshot large and the number of time stamps T small.
The field of continuous learning [1] offers one approach to tackle this problem. However, in our case the training data can be revisited and fresh models retrained making issues like catastrophic forgetting irrelevant. More relevant to our problem formulation are recent continuous domain adaptation methods [2. 4, 5, 6] which either try to predict how the training domains evolve as time passes, or learn feature representations which are invariant across time [4, 5]. However, these domain adaptation approaches assume the presence of unlabeled data in the future, which is not available in our case. Another set of approaches aim to make network parameters a function of time [7, 8, 9] but they require external smoothness kernels as parammarrsestrs for each parameter-type, which cannot be easily trained end-to-end on deep neural networks.
The key objective of our desired training algorithm is to generalize to the near future, even at the cost of performance on past data. In order to achieve this goal, the training algorithm needs to encourage the model to adjust for temporal drifts both in the data distribution and the decision boundary, and be able to generalize and extrapolate well taking into account such shifts. To this end, we aim to develop a supervised learning model which would be robust to small perturbations in time which is fed as feature. We attempt learning functions that are smooth in time, ensuring a limited temporal drift from past to future. However, given a limited number of training staa-s a moderately complex model is prone to fit the training data perfectly, without generalizing it well to the future. To tackle this problem, we propose a method which can learn to generalize well in time by interpolating the predictor between two training time steps using first order Taylor series expansion. This interpolated predictor is supervised using the available labels. While doing so, it makes optimal use of the temporal information, which allows it to extrapolate well on future time steps.
Our gradient interpolation method gives us the privilege to use a complex temporal model even if the number of timestamps is small without the fear of overfitting. To that end, we develop novel neural model that captures the evolving decision boundary over time. More specifically, we design novel time dependent activation function called leaky Temporal ReL.U (TReL.U), which changes the underlying activation threshold smoothly along time. Such a model along with the Gradient Interpolation (GI) based loss function allows our method to make the optimal use of the temporal information and thus extrapolate well to future time.
Our Contributions. In this work, we propose a simple training algorithm to encourage model to learn functions which can extrapolate well to the near future. We do this by supervising the first order Taylor expansion of the learnt function. We also propose a method to make neural networks time aware by modifying the leaky ReLU activation function to change with different simesstamss. We demonstrate strong empirical performance of our method by comparing against state-of-the-ar methods and practical baselines on several real world datasets, and also give insights into its workings through a simple synthetic setup.
2

---

2
Related Work
As we discussed in Section 1, our work is closely related to continuous domain adaptation where time can be treated as continuous valued domain index. Approaches tackling such problems can be broadly classified into three categories: (1) biasing the training loss towards future data via transportation of past data, (2) using time-sensitive network parameters and explicitly control their evolution along time, (3) learning representations that are time-invariant using adversari methods. The first category augments the training data, the second category reparameterizes the model, and the third category redesigns the training objective. We discuss each of these in detail.
Transportation Methods Transformation based approaches [10, 11] map data samples from the past distribution into the target distribution, and train a classifier on the transported samples. These have been adapted situations in which the domains change continuously in [3] and [2]. [3] proposes CDOT, which applies optimal transport with regularizer to take into account the similarity between two successive distributions. [2] modifies the approach to work in an environment where the target data is given as a stream, which is continually changing. To compute the transformations, these methods require similarity measure on x, along with unlabelled data from the target domain, which makes them unsuitable for our settings. Further, the transformations induced may be too simple for high-dimensional inputs x, and may not scale on large training sets.
Parameter smoothness Another approach is to make the model parameters a function of time, and to control their smoothness along time. Mancini et. al. in [9] propose Adagraph, a deep-neural network for image data where the scale and shift batch norm parameters are domain-specific. They propose using user-specified kernel to capture the relationship between the domains, and computing the test domain-speciinn parameters using a kernel regression on the train domain parameters. This kernel a
and needs to be carefully designed. The method cannot hence discover relationships between domains in an end-to-end fashion. Other methods like [7] and [8] consider only a logistic regression classifier and model the parameters as stochastic processes like vector autoregression (VAR) or a Gaussian process (GP). These methods require computing the posterior of GP-smoothed parameters, and this inference does not scale for multi-layered neural networks.
Time invariant representations A recent state of the art approach for continuous domain adaptation CIDA [5] that takes time as input but requires the neural network to encode [x, Ej as a time-invariant feature vector to be passed to a common classifier. The encoder is trained using standard two-party adversarial game where the discriminator tries recover time using a regression loss, and the encoder attempts to suppress time. It is well-known that training such two-party games is challenging and does not converge easily. Even when such convergence achieved, we argue that a better strategy is to allow the features to evolve with time, while controlling the smoothness of this change.
Domain Generalization The goal of such methods is generalize an open unordered set of domains given multi-domain data during training but no unlabeled data during deployment. Several methods have been proposed for this task including domain invariant networks [12], decomposition methods [13], meta-learning [14] or data augmentation with domain perturbations [15]. Our problem can be considered special case of domain generalization where the domains are ordered, and we
only care about generalizing to an adjacent closed interval.
Another emerging area deals with settings where the labeled data is from fixed domain but the target domain evolves with time and is unlabeled [16, 17]. This is different from our setup where we assume evolving labeled data in the source, and one target domain without any data.
Online Learning [18] Also called continual learning [19] and life-long learning, attempts to learn models using training data that arrives in a stream. These have been extended to deep networks [20] using techniques like meta-lcarning [21] and randomized stochastic gradient descent [22]. Methods such as FTPL and FTRL [23] could also be extended to provide alternate approach to the problem. However, online methods do not allow revisiting the training data, unlike in our case.
In addition to the above techniques we explored a number of other strategies for augmenting training data in the future in addition to transportation methods. We devised generative model and local perturbation method inspired by [24, 15]. We also explored the use of neural ODEs for transporting data along time since these are specifically meant to capture continuous dynamics. These approaches are discussed in detail in secction A.2 of the supplementary. None of these approaches provided as much consistent gains on a variety of data types on modern neural networks, as this surprisingly simple gradient interpolation method that we describe in the next section.
3

---

3 The Gradient Interpolation Approach
Our approach comprises of two parts: (1) modelling Fxx, t) as time sensitive neural network Fo(x,t) t) with 0 being the trainable parameters and (2) imposing a special loss to encourage the network to generalize to the near future. We make the network time-sensitive by both taking time as an additional input concatenated with x, and by computing subset of the network parameters as a function of time. When a parameter W, of a network is made simestensivve, then W, becomes the output of small neural network wy (t(O,) with time as input and parameters fg. Now instead of W3 we train 0, during the and training. The set of Wjs that are made time-sensitive is part of network architecture design. We present our proposed design of time-sensitive network architecture in Section 3.3. The rest of our algorithm treats the fJs as any other network parameter and does not impose any custom smoothing on them unlike some prior work [8, 9].
3.1 Training loss induced by gradient interpolation
Overfitting issues for time-sensitive network. Despite building a time- network, if we apply standard ERM training on data from T training snapshots Di...., DT,, we have no guarantee that the network would generalize to samples from the P(x[T + 1) distribution, since the network parameters could easily overfit individually on the T training distributions. Responding to this challenge, we propose to regularize the output of the entire network, described below
Introducing gradient interpolation induced training loss. We encourage learning functions that can be approximated linearly by regularizing the curvature of Fo(x, 1) along time. We impose a loss on the first order Taylor approximation of the model output. Formally, if E(::)) denotes a loss metric to be minimized, our gradient interpolated loss is defined as
Fox 6) + 0F (x, (x, 6)
max JGI(y: Fo (x. t)) = Ely; (x(x 6E(-A.A)
(1)
Ot
Here, the first term is the usual loss for fitting Fx(x, () to y. while the second term is the loss on a regularized approximation of Fo Fx [x, f) us- ing the Taylor Expansion at - 8. We choose the 6 adversarially by gradient ascent within a user-provided window A, which we set as a task-dependent hyper-parameter. When ò is neg- ative we are computing the logits of an instance at t by extrapolating from the future t + 5. This indirectly serves to supervise the function and its gradients to near-by points. We thus expect this loss to provide better generalization to near future points in the neighbourhood of the last training time step.
Algorithm 1 GI based training algorithm to esti-
mate F
Require: Dataset (D* = -xx, yi), E (T]}, time stamps {t, E [T] a neural model for F, i.e., Fe. initial model parameters Co. learning
rate 7, and number of finetune domains k.
1: /* Pre-train Fo
on each D */
2: + 0o
3: for E [T] do
4: 0 <-PRTRRAINOO, D*)
5: end for
6: /* Train Fo o) using loss (1) */
7: for 8 E [T k. T] do
We summarize our training procedure in Algo-
8:
(B) -SPLLTINBATCHESS(D
rithm 1. It works in three steps. Given pa- rameterized neural model Fg it first pretrains 0 using ERM loss on the entire dataset (for loop in lines 3-4). Then, for each minibatch in each time snapshot, we first adversarially select a 5 by gradient ascent to maximize the loss ((::)) and update the model parameters 9 defined in Eq. (1) using 8 (for loop in lines 9-15). More- over, while doing so, we always clamp the value of 8 in (-A, A), as suggested in Eq. (I).
9:
for each minibatch B do
10:
ån +- Unif.[-A, A]
11:
8 - TRAINDELTA (o, B)
12:
8 CLAMP(-A, A)
13:
JGI + COMPUTELOSS((,bb
14:
0 t 1--0-VA VeJes
15: end for
16:
end for
17: return O.
3.2 Understanding GI with synthetic setup
We present here an empirical analysis with a simple setup to understand how GI could be more effective in generalizing to the future compared to existing methods. Consider dataset where the input is d Boolean features i.e., X' E [o, i]d, and output label y E [o, 1] is binary. Assume their joint distribution factorizes Pr(x, y/t) Pr(y) II Pr(æzlv, t) where prior class probability is

---

independent of time and Pr(y) Pr(y]t) = 1/2, and features are conditionally independent given the class label. Assume Pr(z,) y. t) Pr(zz = y/t) = py(t) where (t))) denotes the time-vane -naryig correlation between feature J and label y at time L. Further, let pj (1) be defined as (for 2 3):
pi (1) - 0.6 + 0.14, p2()) 0.6, (1) - 0.5 + 0.49[[ - 3]]
Here the first feature's correlation with class label evolves linearly with time, the second feature has a constant but slightly positive correlation, whereas each of the remaining features have strong correlation at exactly one time stamp, and are uninformative elsewhere. Given the conditional independence of features, the following parametric form can perfectly fit the data while allowing for easy interpretation:
d Σ wy ((0))) + (t)())))) j=1
(x t)
where the w. (t]0,) can be any neural network with input and parameters 0,. We used a two layer network with ReLU activation of width 50 and 20 respectively for each (t)())
Assume we have training data at O, 1, 2 and we need to deploy at t = 3. According to the above parameterization, the baseline ERM could fit training data well by giving a high weight to 23, I4, 2s times / 0, 1, 2 respectively, and low weight other times. There is no control on how the (y)()) values evolves with time, and there is no reason for the classifier to generalize to ( = 3. In fact here the test accuracy is 49.5% (slightly worse than random) while the train accuracy is 99%. Figure I shows the weights as function of time (solid lines) for w1, W2, W1, Wy and we see that wy is largely ignored while W4, Wy take large values and have large swings.
When GI is applied on this dataset, we impose an additional loss on logits computed from E, - 5) + 5 ( This loss provides supervision on not just the parameter values at the training times. but also requires that their gradients be correctly oriented to allow reconstruction at near-by points. This encourages information sharing between training time-stamps for the model, and has the effect of smoothing the (t) functions for the transient features, and recovering the correct slope of the linear features. Further, when is negative then for instances close to we also indirectly supervise the logits and their gradient at future times T + 8.
Figure shows the weights learned by GI in contrast with those of ERM. The accu- racy obtained by is 70.9, which is higher than that of any other model. Without clair- voyance, and without access even unla- beled data from = 3, it may be difficult to get better results in expectation over future datasets.
10
20
20
ERM
(r(())
10
3
200
20
ong -
-5
Is GI loss just a gradient regularizer? Since one of the main outcomes of GI is smoothing wy)s),, a natural question is whether the benefits of GI could also be obtained by a simple regularizer on the gra- dient of the network with respect to time. In that context we explored an alternative gradient penalized training objective as follows:
10
Pm
110
-15
-10
2
-10
0
2
3
-20
0
2
Figure/ 1: Weights learnt as a function of time
@ (x t)
2
JGR Fo(,,) E(y; Fox, +
(2)
Ot
This objective requires us to carefully select the weight of the regularizer. X is large, the objective will converge to constant wy (L), even for weights that have a predictable linear change along time. Such a classifier gives a test accuracy of 47.4% in this setting. For small x, the weights would not be sufficiently smooth. Contrast the above with the GI loss in Equation where the true label y supervises how much the gradient needs to be smoothed along time.
Why time-invariant representation would not work. GI allows the last layer features to evolve with time, unlike recent methods such as CIDA [5] that attempt to learn time-invariant representations at the last layer. We implement this method by making the vector w(t) @ (where @ denotes element-
5

---

to be our train wise product) as the time invariant representation of a data point. A successful implementation of the adversarial loss would make the expected value of E, w (t)x)) independent of time. In our generative model, it can be see that the marginals [2y] = 1/2, and thus are independent of time. This implies that an ideal classifier as per CIDA would make w; (t) a constant independent of time. When such a classifier is fit on the data, we get test accuracy of 50.5% which is again random. The merit of GI is that it can let the feature representation evolve with time to generalize to the future interval.
3.3 Our Proposed Time-sensitive Model Architecture
In this section, we present our proposed time-sensitive model architecture F/ which approximates F using neural network model parameterized with 0. Given the feature vector x E D at a time snapshot i, the predictor (x(x) uses two time-sensitive components:
Time representation vector. First, we featurize the time t associated with the data point x. A naive way to do that concatenate f with x to obtain an augmented feature vector [x, t]. However, such an approach cannot capture complex trends in data, c.g., periodicity. To tackle this challenge, obtain a representation vector T, E IR" of time i by building upon an existing time embedding model TIME2VEC proposed in [25], which is computed as,
[Wat t + bas 1 S mp [sin(wa t + ba), mp < a S m
T: [a|
Hence the first mp entries of T use linear transformation of t, whereas the last m - my entries use periodic transformation. Here, {wa) and {b,] are trainable parameters.
TReLU activation function. Another way to make F time-evolving is to model the feature weights in time dependent manner. Previous work model such weights using Gaussian Processes [8], batch normalization using kernel regression [9], etc. However, extensions of these models in deep neural network may be computationally infeasible for training.
We note that many successful universal approximators often consists of a cascade of linear and ReLU layers. Therefore, in order to encapsulate the time-evolving nature of F, we ideally require universal approximator of time-dependent function. To this aim, we develop a novel time dependent ReLU function, called as TReLU, as follows.
(To(T)) @ (x go((T)) + v(Te), when x < (T( x, when x 2 (T,)
TReLUU
(4)
Note that, h& E Rdd Uy and I& E Rd are neural networks with parameters Φ, which control the slope and threshold. If h. and na = g 0, TReLU becomes equivalent to traditional ReLU activation function. We then use these TReLU units as drop in replacement for all the ReLU units in a neural network. We compute the output y for the input x by successively passing it through linear and TReLU layers according to the network architecture. Note that each TReLU unit in the network is parameterized by separate set of parameters Φ
Note that the outputs of h and g are d dimensional. In our experiments, we use neural networks 8 units to parameterize these functions, and also consider the
with single hidden layer of size dimension of time vectorization, i.e. m to be 8. This introduces 2(mk + kid) 128 + 16d extra parameters per TReIU layer into the network, which is much less than ((d) parameters of the linear layer in the network. This is also less than (Td) extra parameters introduced by Adagraph [9] to model temporal dependencies.
Finally 0 = {wo, b.. d. }. are estimated using the training algorithm shown in Algorithm 1.
4 Experiments
We compare GI with several existing approaches on seven datasets and present a detailed ablation study. All results are reported across five random restarts. Our experimental setup is coherent with our problem setup as described in Section 2. The datasets under consideration have a temporal component associated with them. We consider data with timestamps 1, .., T data, and data from T to be the test data. We train all models on the training data, and report the

---

Classiication
Regression
Method
2-Moons
Rot-MNIST
ONP
Shuttlc
Reuters
Elec2
House
M5-Hob
M5-House
Baseline
2.,444.6
18.6=4.0
33.8
0.77
89 1.02
23.0
11.0
0,24 0.27
0.08
LasiDomain
1.99
17.
36.0
0.91
0018
1.1
25.8+0.6
10.3 0.16
2.72
0.75
+.54
IncFinetune
3..4
+0.1
34.0+.3....
0.07
27.3 8.7+
9.01
0.12
0.05
0.17
CDOT
9.3
1.22=100
14.100000..40
0.17
1.15
17.8
CIDA
1..
9.3
34.7+0.6
DNC
DNC
14.1+0.2
9.7 9.7-0.06
0.06
0.40
0.07
0.58 0.11
Adagraph
8.0
9.9
0.47
0.04
0.51
20.1
9.7
1.64
0.28
0.87
GI
1,
7.7
3..9
0.05
7.4 0.28
16.9 0.7
9.6 0.02 0.09
0.03
0.05 0.02
Table 1: Comparison of our proposed method against existing methods all the nine datasets in terms of misclasssication error (in % %)for first six datasets and mean absolute error (MAE) for last three datasets. The standard deviation over five follows the + mark. DNC indicates that the method did not converge. We observe that GI outperforms almost all the baselines.
performance the test data. For tuning hyperparameters, consider data from T - 1 to be the validation set. Code for reproducing our experiments has been open-sourced".
4.1 Datasets
We experiment with six classification datasets: Rotated 2 Moons (2-Moons), Rotated MNIST (Rot-MNIST), Online News Popularity (ONP) [26], and Electrical Demand (Elec2), Shuttle and Reuters; and three regression datasets: House prices dataset (House), M5-Hobbies (M5-Hob) and M5-Household (M5-House). The first two datasets are synthetic where the angle of rotation is used as a proxy for time. The remaining are real world temporally evolving datasets. Details of the datasets available in section A.I of the Appendix.
4.2 Methods compared
We compare our method with three practical baselines and three state of the art continuous domain adaptation methods.
Baseline. This time-oblivious model is trained using ERM on all the source domains.
LastDomain. This time-oblivious model is trained using ERM on the last domain.
IncFinetune. In this model, we bias the training towards more recent data by applying the Baseline method described above on the first time-stamp and then, fine-tuning with a reduced learning rate on the subsequent time-stamps sequential manner. However, the underlying network is otherwise not time-sensitive. Such approach is similar in spirit to the gradual domain adaptation method of [16], modified to account for labelled data at each time snapshot. This is also similar to online learning methods adapted for our setting.
CDOT [3]. transports most recent labeled examples DT to the future using learned coupling from past data, and trains classifier on them.
CIDA [5]. This method is representative of typical domain erasure methods applied to continuous domain adaptation problems.
Adagraph [9]. This method makes the batch norm parameters time-sensitive and smooths them using a given kernel.
More details about these methods can be found in section A.2 of the appendix. Comparisons with modified online learning methods from [23] are also present in section C of the appendix.
4.3 Comparative analysis
We first compare the performance of our proposed GI based method against existing approaches, in terms of misclasssication error for first six datasets, i.e., 2-Moons, Rot-MNIST, ONP, Elec2 and mean absolute eтror (MAE) for last three datasets, i.e., House, M5-Hob and M5-House. Table I summarizes the results. In most cases, we see that GI provides significantly lower errors. The gains we obtain on M5-House and M5-Hob are particularly impactful because these are large real-life datasets and the gains over the second-best method are statistically significant by a big margin. Additionally, we make a number of other interesting observations from these detailed experiments: (i) All datasets except
7

---

ONP exhibit strong temporal drift as seen by the poor performance of the time-oblivious Baseline, (ii) More data helps and heuristically choosing suffix of available data may be sub-optimal seen by the mixed results on the simple LastDomain baseline. (iii) Just incremental fine-tuning where most recent data seen by the model often strong baseline. time-invariant representations shows improvements but in two cases it is worse than IncFinetune.
(iv) The CIDA method that creates
T= 10, =1
T y
T= 7=
T= 7,y=1
feature 22
feature 212
feature
Ceature Ø1
feature 27
feature xt (c) CDOT
(a) Baseline: ERM
(b) Incremental Fineutining
leature
feature
feature
-2
-2
fenture
fenture it
feature at1
(d) CIDA
GI
(f) Gradient Regularization
Figure 2: Qualitative Analysis on 2-moons across different methods. Pink (+1) and light purple (-1) are samples from 7 7, and magenta(+1) and dark blue (-1) are samples from T 10 differing by 54" rotation. Decision boundaries at T 10 are in dark blue lines and T 7 are in light blue lines.
4.4 Qualitative Analysis on 2-moons
Next we compare different methods in a qualitative manner. Figure displays the results, which shows the decision boundary learned by the different classifiers at T 10 and T time-stamps the 2-Moons dataset. We observe that the time-agnostic baseline overfits the training data and shows poor performance for T 10. Furthermore, IncFinetune significantly improves the baseline but seems to be over-fitted for the last timestamp. We also note that performance of CDOT depends on how closely the transported images resemble the true target image. Since the transported samples quite noisy. it does not perform quite as well as other methods that model the dependence on time directly. CIDA better but misclassifies the outer ends. Gradient regularization makes the decision boundary almost time-invarian significantly hurting performance due to lack of generalizability to the future. On the other hand, GI is the only method that manages learn decision boundary of the correct complexity, and rotates correctly along time.
4.5 Ablation study
GI differs from existing methods in two ways: (1) time-sensitive network with time as input via Time2Vec encoding and TReLU, and (2) the gradient interpolated loss. Here, we perform an ablation study to investigate which of these components contribute the most to GI's gains and also discuss if simpler regularizers could achieve what our GI loss does. More specifically, we consider four alternatives to compare to our proposed method. (1) Base-Time Time: Here we make the network architecture time-sensitive
the same way as GI. However, we train using simple ERM. (2) InCFinetune-Time Here, we bias the time-sensitive network towards recent data using incremental fine-tuning. (3) Grad-Reg: Here we regularize the 1.2-norm the gradient of the model along time in Equation 2, (4) TimePerturb: Here adapt the popular adversarial training method to make

---

Ablation
2-Moons
Rot-MNIST
Elec2
Shuttle
M5-Hob
M5-House
Base-Time
4.1
0.9
0.35 0.09
0.29 0.14
IncFinetune- Time
3.3
9.2
1.
0::0000000000000:000000000000000 0.10 0.07 0.07
Grad-Reg
1..4
1.5
2.57 1.01 0.15 0.90 17.3
26.3
TimePerturb [24]
9.7
10000.77=0000 0.06
0.09 0.01
0.11
GI
3.5 + 1.37
7.7
16.9
0.29 0.05 0.09 + 0.03 0.05 0.01
Table 2: Ablation study. Comparison of performance between our method and four alternatives across four datasets for classification tasks and two datasets for regression tasks.
model robust to small changes in time [24]. In particular, the loss minimized is:
Jrr(y; Fo(x, t)) E(y, Fo t)) X(((y, Fo t + +))),
(5)
were 8 is chosen adversarially through gradient ascent on ((:-))
Table 2 summarizes the results. We make the following observations. (i) GI outperforms other forms of regularizations most dataset. (ii) The poor performance well stability issues of Grad-Reg are due to the fact that a supervised gradient regularization makes the model more time agnostic. (iii) TimePerturb can also provide significant gains since the method is designed to provide robustness against small perturbations of inputs. GI also perturbs time, but the key difference is that TimePerturb assumes that y stays the same after perturbation. In constrast, GI makes milder linearity assumption on the computed logits allowing GI to explore larger perturbation windows without introducing undue smoothness bias.
4.6 Evaluation of time-sensitive architecture of our model
There are many ways to make the network parameters a function of time. We chose to modify only the ReLU units of the network and make their activation a function of time using our TReLU module. Here, we investigate the impact of these units through an ablation study. We start with a network with only ReLU activation functions. We then change the last few layers increasingly to be time-dependent and finally train network using ERM to understand how well our temporal parameterization does on its own Figure 3 summarizes the results. We observe that the performance becomes better with the number of time dependent units. Even a single TReLU unit gives significant gain against a time-insensitive architecture, indicating its effectiveness in capturing temporal trends.
4.7 Accuracy Along Time
We compare the efficiency of GI with CIDA and Adagraph by training the model using only limited number of time snapshots on the Rot-MNIST dataset. We do this by fixing the test time snapshot T. and training independent models on data from times {T 1}, (T Figure 4 summarizes the results, which shows that GI works better especially in the low data regime, though the performance gap closes as the number of train time-stamps increases. It is also notable that performance across all methods is improved by including older training data, indicating the importance of harnessing longer term temporal trends the data.
T - 2} and so on.
4.8 Impact of adversarial selection of
In this section, we present a comparative analysis of various 8 selection schemes. Adversarially selecting 8 incurs additional overhead with regards running time of our algorithm. We tried the following methods:
Random ó: We randomly select value of d as 5 + Unif. A, A] for each mini-batch. Adversarial Following Algorithm 1, we adversarially pick a for each mini-batch Adversarial 8 WS: Same above, but instead of sampling å for each mini-batch, we warm start, ie. continue with the learnt in the previous training step.
From Table 3, it is clear that selecting 8 adversarially according to Algorithm 1 provides some performance gains. However, warm-starting the value of from the previous training minibatch gives similar or better performance minibatch. Randomly selecting performs slightly worse, but can be used for situations in which the training process has time constraints.
most datasets. This reduces the overhead of sampling å for each

---

Dataset
2-Moons
Rot-MNIST
M5-Hob
Random
3,8 + 1.2 3.6
8.0 1.0
0.12
Adversarial 8
8.3
0.08 = 0.05
Adversarial 8 WS
1.4
7.7
0.09 1
Table 3: Effect of adversarial selection on GI.
Hominghhiee
14
RAMNNIT
9.60
25
Adagophh
131
Eeror-
20
8.5
Number TimeHeLU units->
Number TimeReLU units->
Number of time stegs-+
(a) House dataset
(b) Rot-MNIST dataset
Figure 4: Effect of number of training snapshots.
Figure 3: Effect of changing the number of TReL.U units
4.9 Importance of number of finetune domains
When fine-tuning with GI, we finetune only on a subset of the training data, picking only k most recent train time stamps. There is trade-off inherent to this process, wherein one could pick larger have more training data, the cost of higher training time. As observed in table k performs comparably with higher values of and at lower computational expense, and hence we use this value for all our experiments.
Dataset
Rot-MNIST
House
Flec2
8.6
9.9
17.2
k 2
7.7
9.6
16.9
h
8.0
9.7
16.9
k = 4
7.9
9.6
17.3
Table 4: Effect of number of finetune domains on performance of GI.
Conclusions, Limitations and Future Work
In this work we identify the problem of generalizing prediction models for the future, and propose a simple method to address it. To that aim, we propose a Gradient Interpolation based approach which is agnostic of the deep network architecture used and be used in various classification and regression tasks that exhibit data or label distribution shifts along time. Furthermore, we use TReLU and time representation vectors which can make any model robust non-linear distribution shifts in time, like rotation. However, this is not without its shortcomings.
Limitations. One of the key elements in proposed algorithm is the running time overheads of the current timestamp. While we propose
searching for an adversarial some engineering tricks reduce this overhead in section A.4 of the appendix, it remains significant component of the total training time of our algorithm. However, we note that the total wallclock training time of GI is still less than adversarial approaches like CIDA [5] due faster convergence of our algorithm, as we detail in section B of the appendix.
a A neighbourhood
Future work includes theoretically analyzing the GI loss, and extending GI loss to structured prediction tasks.
Acknowledgements
Vihari Piratla is supported by Google PhD fellowship. This research was partly sponsored by the IBM AI Horizon Networks IIT Bombay initiative.
10

---

References
[1] German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71 71, 2019.
[2] Judy Hoffman, Trevor Darrell, and Kate Saenko. Continuous manifold based adaptation for evolving visual domains. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, Columbus, OH, USA, pages 867-874, 2014.
[3] Guillermo Ortiz Jimenez, Mireille El Gheche, Effrosyni Simou, Hermina Petric Maretic, and Pascal Frossard. Cdot: Continuous domain adaptation using optimal transport. arXiv preprint arXiv: 11448, 2019.
[4] Qicheng Lao, Xiang Jiang, Mohammad Havaci, and Yoshua Bengio. Continuous domain adaptation with variational domain-agnosto feature replay. arXiv preprint arXiv:2003.044: 2020.
[5] Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. arXiv preprint arXiv:2007.01807, 2020.
[6] Yongxin Yang and Timothy M Hospedales. Multivariate regression on the grassmannian for predicting novel domains. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, Las Vegas, NV, pages 5071-5080, 2016.
[7] Atsutoshi Kumagai and Tomoharu Iwata. Learning future classifiers without additional data. In Thirtieth AAAI Conference on Artificial Intelligence, Phoemix, AZ, USA, 2016.
[8] Atsutoshi Kumagai and Tomoharu Iwata. Learning non-linear dynamics of decision boundaries for maintaining classification performance. In Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 2017.
[9] Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Adagraph: Uni- fying predictive and continuous domain adaptation through graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, pages 6568 6577, 2019.
[10] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 39(9):1833- 1865, 2016.
[11] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In Proceedings of the 2012 IEEE Conference Computer Vision and Pattern Recognition, Providence, RI, USA, pages 2066-2073 IEEE, 2012.
[12] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 100i:1997.0000 999999989,, 2019.
[13] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common-specifie low-rank decomposition. ICMI., 2020.
[14] Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain general- ization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems, pages 6447-6458, 2019.
[15] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint 1804 10745, 2018.
[16] Ananya Kumar, Tengyu Ma. and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference Machine Learning, pages 5468 5479. PMLR, 2020.
[17] Hong Liu, Mingsheng Long, Jianmin Wang, and Yu Wang. Learning to adapt to evolving domains. In Advances in Neural Information Processing Systems, volume 33, 2020.
11

---

[18] Brendan McMahan. hwwp:ttteerrgulrizee--eeeeee and mirror descent: Equivalence theorems and 11 regularization. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 525-533, 2011.
[19] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting in classification tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, pages 1-1, 2021.
[20] Steven C. H. Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. Online learning: A comprehensive survey, CoRR, abs/I 1a2/00002,2,7,, 2018.
[21] Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based RL. CoRR, abs/1812.07671. 2018.
[22] Shibhansh Dohare, Rupam Mahmood, and Richard S. Sutton. Continual backprop: Stochastic gradient descent with persistent randomness, 2021.
[23] Arun Sai Suggala and Praneeth Netrapalli. Follow the perturbed leader: Optimism and fast parallel algorithms for smooth minimax games. In NeurIPS, 2020.
[24] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint 141iv::1422..510 2014.
[25] Seyed Mchran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning vector representation of time. arXiv preprint 107.05397 2019.
[26] Kelwin Fernandes, Pedro Vinagre, and Paulo Cortez. A proactive intelligent decision support system for predicting the popularity of online news. In Portuguese Conference on Artificial Intelligence, pages 535-546. Springer, 2015.
12


Reviews:
[{"review_id": "l0zBkqsDzGi", "reviewer": "Reviewer_F2t2", "summary": "This paper presents the Gradient Interpolation (GI) loss to regularize a time-dependent network, aiming to generalizing the predictions to the near future. The GI loss is computed as the maximum of the loss within a short window, which encourages the network to be locally linear in time. The paper further designs the Time Leaky ReLU (TReLU) which uses the time feature to scale the standard ReLU functions. Through a list of classification and regression datasets, this paper demonstrates its superiority against baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Overall Comment\nTo make the agent to extrapolate towards future, it is important to uncover the temporal dependencies. This paper attempts to learn a network that take in both the input and the timestamp. This paper presents the GI loss to regularizing the predictions of the network within a short time window. However, I am not fully convinced by the arguments and the analyses.\n\n### The Synthetic Problem\nSec 3.2 is very difficult to understand for me .\n1) I cannot understand the conditional distribution: $P(x_j | y, t) = P(x_j = y | t) = p_j (t) $. Why $x_j$ appears only in the first equation ? \n2) Since a random guess is $50$%, the numbers $49$%, $55.4$%,$49.5$%  do not seems to statistically different to me. In consequence, I do not think this experiment makes a positive evidence. In fact, I don't think any simple neural network could uncover the one-to-one relationship between $ j $ and $ t  $ with data only in past domains.\n4) I think a more proper gradient regularizer than Eq(2) is to regularizing the gradients over losses, e.g., $ \\| \\frac{ \\partial l(y, x, t) }{\\partial t } \\|_2^2 $. This regularizer is also similar to the TimePerturb approach in Sec 4.5, which works comparably to GI.\n\n### TRAIN DELTA\nGiven that the GI loss involves the gradient already, performing gradient descent to find $\\delta$ requires evaluating the second-order derivatives. And multiple gradient descent updates are needed to find the optimum. Therefore, the resulting approach seems computationally very heavy. Could you also provide a running time comparison with the baselines in your experiments ? \n\n### Pretraining\n1) How the pretraining is done in an online setting, where tasks come one-by-one ? \n1) Line 3-5 in Algorithm 1. Is the pretraining done one-by-one sequentially ? Given you assumed knowing all tasks, this does not seem necessary. \n\n### Experiments\n1) Could you present a clear definition of the training and the testing procedure ? \n2) How many more trainable parameters are used in TReLU, compared to the standard network and the baseline approaches ? \n\n\n### minor comments\n1) Line 75: \"we we \"\n2) Line 278: \"the angle of rotations\"\n3) Line 301: \"lower errors\"\n4) Line 305: \"drifts\"\n5) Table 2: \"performances\", \"the classification task\", \"the regression task\"\n6) Line 369: \"Furthermore, we use ..... can make any model....\"\n7) Line 192: $ [0, 1]^d $, should use curly brackets for the binary case.\n8) Line 196: $p_j(t), t\\geq 3$\n9) Line 262: I cannot understand the logic of this sentence: \"we ideally require a universally approximator of time-dependent function\", given that the proposed TReLU is not provenly universal.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "To further improve the paper, I think these are potential directions,\n1) A clearer discussion of Sec 3.2\n2) Discussing the computational overheads by training deltas, and the parameter overheads by TReLU\n3) A clear description on the workflow\n\n (a) how the pretraining is conducted \n\n (b) what is the training and the testing procedure\n\nThis paper does not lead to negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "hp15eIJPY1W", "reviewer": "Reviewer_rfub", "summary": "ML models deployed in production pipelines are trained continuously to account for gradual shift in the data distribution. This paper deals with model generalization w.r.t. data in the near future in this continuous distribution shift. The paper proposes training a model with time-sensitive parameters where the temporal complexity is regularized by a Gradient Interpolation (GI) loss. Time-sensitive network architecture has two components: time-insensitive (basically standard neural network parameters) and time-sensitive parameters (which are the output of another small neural network which takes into account the time parameter). It also proposes a novel time dependent activation function, Leaky Temporal ReLU (TReLU), that allows for smooth changes along time dimension. Empirical evaluations demonstrate that the proposed scheme outperforms competitive baselines such as generative and adversarial approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n---------------\n\n- Separation of the neural architecture into time-sensitive and time-insensitive parameters: (a) incorporating time as a feature using TIME2Vec embedding model, (b) time dependent feature processing using the proposed TReLU activation function.\n\n- Gradient Interpolation loss to incorporate adversarial time dependence on the loss w.r.t. time-dependent component of the network.\n\nWeakness:\n---------------\n\n- The underlying problem seems like an ideal setting where online learning should have been explored first. \n\n- I have some reservations on the empirical evaluation of the scheme ( see below ). Barring the two synthetic classification datasets, the proposed scheme performs poorly than the baselines on the remaining classification datasets. Are the gains on the regression datasets mainly due to the fact that the time-representations of the input is better than the raw features?\n\n\nQuestions for Authors:\n---------------\n- Why can the underlying problem not be taken as an instance of an online learning? In online learning, at any time t, you play parameters W_t, and you receive the next data (which could even be adversarial in nature, specifically designed to fool W_t), and you update your parameters to adjust to this data? There are many learning schemes which have been deployed in this context such as Follow-The-Regularized-Leader, Follow-The-Perturbed-Leader, and other advanced variants of these schemes etc.\n\n- What is the best one can hope for this continual setting? Since the assumption is that near future data may have no relation to the data recently seen. Could you expect any classifier to perform better than one jumbo network trained with all the data in the past (paper mentions that data in the past could be repeated in future)? Correct me if I'm missing some critical peace of this puzzle.\n\n- The Toy example in the Sec.~3.2 with the synthetic setup is not convincing that the proposed scheme works as intended. Coin toss accuracy is 50%, ERM achieves 49%, GI (proposed scheme) achieves 55.4%, and scheme with gradient regularizer achieves 47.4%. Its not clear if this example can help you make any conclusion about any of these methods as they are very close to the random chance on this problem. Maybe this is not a good toy example to showcase the benefits of the proposed scheme.\n\n- I'm a little lost in the empirical evaluation section. Given a dataset that has time components and a train/test split. How is the training done? How is the network evaluated? Is it just trained on the training set and simply evaluated on the test set? I expected the evaluation to be incremental where in you train with train data upto a certain time and then evaluate on the data past this time instance, and so on.\n\n- Figure 4 is missing in the main text and I see no figure 4 in the appendix.\n\nWriting Clarity:\n---------------\n- Suggestion to include an explicit assumptions paragraph, which states whether the near-by future has any correlation to what was seen recently or could it be related to anything in the distant past?\n\n- Missing related works and connections with online learning\n\n- Line 75: we we \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "eUT-od5N6Yx", "reviewer": "Reviewer_M61X", "summary": "The authors propose a novel loss (the gradient interpolation loss), paired with various architectural innovations and a training algorithm, in order to train models that generalize gracefully under distribution shifts that happen over time. The main contributions are as follows:\n1. The gradient interpolation (GI) loss, a regularization term that penalizes the curvature of the learned network along time. \n2. The TReLU activation, a novel activation whose slope and threshold are learned to vary wrt. time. \n3. A training algorithm that uses the GI loss to train networks that are robust against time-dependent distributions shifts. \n\nThe contributions of the experiments are as follows:\n1. The proposed method seems to perform better than a number of selected baselines and state-of-the-art domain adaptation methods. \n2. Further insight is derived from synthetic experiments that aim to understand the mechanisms by which the proposed method achieves robustness against domain shifts. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe that this is a good paper that makes a solid and interesting contribution on an important problem. I'm ready and inclined to increase my score based on the author response to the main weakness I list below. \n\nThe main strengths include:\n1. The proposed approach is interesting and, as far as I know, novel. \n2. The GI loss can, in principle, play well with arbitrarily complex models. \n3. The fact that time-dependence of network parameters can be achieved only through modifying the activation functions is noteworthy. (I'm not sure if this is a novelty)\n4. The paper does relatively extensive ablation studies and runs other synthetic experiments to gain insight into how the proposed method works. \n\n___ The main weakness ___:\n1. It's not clear how the hyper-parameters were tuned, both for the competing benchmarks and in the ablation studies. For example, was the learning rate separately tuned, albeit coarsely, for all different tasks on all different models? It's also important to learn more about how the competing approaches for picking the delta term (i.e. adversarial, random, adversarial with warm start etc.) were tuned, as well as the lambda coefficient of the gradient-penalty loss (Equation 2). In my opinion, if it turns out that selecting delta randomly from a reasonable distribution works as well as the adversarial version, I think that'd be a positive for the proposed method, as it'd remove the costly optimization step to optimize delta each step. \n2. It is specified in the appendix that the fine-tuning steps, which is where the GI loss is enforced, is applied only to the last k time stamps of the dataset. This is, I think, an important design choice and should be moved to the main paper. Additional clarification on the importance of choosing k properly is also needed (it seems like most experiments used k=2. How would be performance be if you used k=1, or k=3? ). (Please correct me if you've actually discussed this in the main paper and I've missed it somehow)\n\nOther weaknesses include:\n1. The proposed method is expensive (due to the online selection of the delta term in Algorithm 1), as acknowledged by the authors. The tasks that the proposed method is applied on are relatively small scale, making it difficult to see how much of a scalability issue the cost of enforcing the GI loss really incurs. A more detailed discussion (perhaps in the appendix) on how much of the compute is spent on different parts of the algorithm would improve the paper, though I don't think this is crucial.\n2. (This might not be a weakness - I might have misunderstood something about TReLUs) You mentioned that each TReLU has neural networks \"h\" and \"g\" that have different parameters. If a single TReLU is a replacement for a single ReLU, then that'd mean we have to train thousands, maybe millions, of small neural networks that are inherently unparallelizable on GPUs. Doesn't this lead to the algorithm being very inefficient? (Again, I might have misunderstood this part and this might not be a weakness)\n3. The delta term is optimized across minibatches and, in some configurations of the algorithm, warm started from the result found in the previous mini-batch. These break the iid. assumption. This should be acknowledged, as it could lead to unexpected effects. \n4. Overselling phrases like \"[the proposed method] makes optimal use of the temporal information\", as in line 77, should be avoided, unless the notion of optimality is defined properly and justified theoretically or empirically. \n5. Further details on the \"TrainDelta\" step in line 11 of Algorithm 1 would be useful to have in the paper. \n\n\n\n\nQuestions to the authors: \n1. Also, do you backpropagate through the steps to find delta (I assume not, but this would be useful to clarify). \n2. How important is it to pick the capital delta terms (i.e. the window in which the perturbations are picked)?\n3. Did you inspect how the learned slope and threshold values look like? \n4. Is the x axis of Figure 3 \"number of TimeReLU units\", or \"number of timeReLU layers\"? \n5. Could you clarify on which task Figure 4 was obtained on? \n\nWriting, typos etc.\n1. The paper is well written, well organized and easy to follow. \n2. It has a small number of typos: \n- line 75: \"we\" is duplicated. \n- line 155: repeated \"..\"\n- line 180: nerual -> neural\n3. The math expression under line 196 is a big confusing. \n\n====Edit after author response====\nI thank the authors for their response. Acknowledging some of the limitations that were pointed out during the review process, I believe that the methods and results are strong/relevant enough to be of interest to the NeurIPS community working on domain generalization with extrapolation to near future. This is why favour acceptance. \n\nI encourage the authors to improve upon their expository toy example to demonstrate the utility of their approach better, and in general try to gain a more mechanistic understanding for why their approach seems to have good empirical performance and how this compares with the ablation methods, such as regularizing the gradient. I also urge the authors to improve on writing to make the problem definition as well as key design decisions clearer to the audience. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations, which are closely intertwined with the weaknesses, are outlined above. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers feel that this paper introduces an interesting and potentially useful approach to learning in changing environments. They consider the approach sensible. Various reviewers originally had concerns both about the computational cost and about the experimental validation (especially details of hyperparameter optimization and choices of tasks and baselines). After some discussion, the reviewers feel like the authors have addressed their main concerns, and all of them recommend acceptance. I concur.


Decision: Accept (Poster)