Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time

1 Introduction . Many organizations operate their machine learning pipeline as follows : collect labeled data based on day-to-day operations e.g . user-clicks for a recommendation model ; periodically ( say every ∆ = 48 hours ) train the model on the data collected from the past ; deploy the model to serve requests of the next ∆ period ; repeat the cycle . Real-world examples of such usage include recommendation models for e-retails , loan approval models in banks , regression models to forecast future popularity of media content , classification of malicious websites , and activity recognition models from sensor data . A key motivation behind such periodic retraining is the implicit knowledge that the data distribution is not stationary , requiring a revision of the decision boundary with time . While incorporating fresh data in training does make the trained model more current , we investigate if we can improve the training even further . In particular , when training the model at a time t , we know that the model will not be deployed on data samples in the training interval , but on data from the immediate future [ t , t+ ∆ ] which could differ from even the most recent training data . Our goal in this paper is to use this knowledge more centrally to train a model tailored to the intended deployment period . Formal Problem Statement . We consider prediction tasks over an input space X and output space Y where the joint distribution P ( X , Y ) evolves with time . The prediction model should therefore be conditioned on time as P ( y|x , t ) for x ∈ X , y ∈ Y , and time stamp t. During training we are provided with labeled data from T time snapshots t1 ≤ t2 ... ≤ tT . Call these D1 , D2 , . . .DT where each Ds = { ( xi , yi ) : i = 1 . . . ns } is assumed to be sampled from P ( x , y|ts ) . The trained model ∗Equal Contribution †anshulnasery @ gmail.com 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . will only be deployed on data from a time interval in the future which we denote as tT+1 , that is on examples sampled from P ( x|tT+1 ) . Note that we only evaluate performance once on time tT+1 , differing from online learning where regret is computed incrementally . Following most practical setups , we assume that the temporal drift across time is not too high . Since the deployment time is in the future , we can not assume the presence of even unlabeled target data at the time of training the model . Further , at different time snapshots , completely unrelated sets of xi may comprise a Ds . This makes the problem much harder than standard time-series forecasting tasks , since we do not have values of the same instance at different points in time . Our goal is not to generalize to all future data , but rather to data relatively close in the future , since our motivating application is one where models are retrained periodically . The former generalization would require very different models and approaches than what we propose . We assume that P ( y|x , t ) is implemented as a general-purpose neural network F ( x , t ) that takes as input x , t and outputs either ( 1 ) a vector of logits for classification tasks or ( 2 ) mean and other scores defining a continuous distribution over predicted y for regression tasks . A solution often used in practice is to only train the model on recent data and/or apply a graceful ageing penalty on past data . However , this may not be adequate with data hungry modern networks , especially when non-obvious seasonality makes distant data relevant . So we seek general purpose solutions that can harness all available data for making optimal decisions into the future . Any such solution first needs to make the model time sensitive . This can be done by feeding time as an input feature , or by making all or some of the model parameters to be a function of time . However , that by itself can not be sufficient to generalize to future data if we train the model to minimize loss on the training snapshots . In fact , a sufficiently complex model can easily overfit on training time-stamps when the number of instances per snapshot is large and the number of time stamps T is small . The field of continuous learning [ 1 ] offers one approach to tackle this problem . However , in our case the training data can be revisited and fresh models retrained making issues like catastrophic forgetting irrelevant . More relevant to our problem formulation are recent continuous domain adaptation methods [ 2 , 3 , 4 , 5 , 6 ] which either try to predict how the training domains evolve as time passes , or learn feature representations which are invariant across time [ 4 , 5 ] . However , these domain adaptation approaches assume the presence of unlabeled data in the future , which is not available in our case . Another set of approaches aim to make network parameters a function of time [ 7 , 8 , 9 ] but they require external smoothness kernels as hyper-parameters for each parameter-type , which can not be easily trained end-to-end on deep neural networks . The key objective of our desired training algorithm is to generalize to the near future , even at the cost of performance on past data . In order to achieve this goal , the training algorithm needs to encourage the model to adjust for temporal drifts both in the data distribution and the decision boundary , and be able to generalize and extrapolate well taking into account such shifts . To this end , we aim to develop a supervised learning model which would be robust to small perturbations in time— which is fed as feature . We attempt learning functions that are smooth in time , ensuring a limited temporal drift from past to future . However , given a limited number of training time-stamps , a moderately complex model is prone to fit the training data perfectly , without generalizing it well to the future . To tackle this problem , we propose a method which can learn to generalize well in time by interpolating the predictor between two training time steps using a first order Taylor series expansion . This interpolated predictor is supervised using the available labels . While doing so , it makes optimal use of the temporal information , which allows it to extrapolate well on future time steps . Our gradient interpolation method gives us the privilege to use a complex temporal model even if the number of timestamps is small without the fear of overfitting . To that end , we develop a novel neural model that captures the evolving decision boundary over time . More specifically , we design a novel time dependent activation function called leaky Temporal ReLU ( TReLU ) , which changes the underlying activation threshold smoothly along time . Such a model along with the Gradient Interpolation ( GI ) based loss function allows our method to make the optimal use of the temporal information and thus extrapolate well to future time . Our Contributions . In this work , we propose a simple training algorithm to encourage a model to learn functions which can extrapolate well to the near future . We do this by supervising the first order Taylor expansion of the learnt function . We also propose a method to make neural networks time aware by modifying the leaky ReLU activation function to change with different time-stamps . We demonstrate strong empirical performance of our method by comparing against state-of-the-art methods and practical baselines on several real world datasets , and also give insights into its workings through a simple synthetic setup . 2 Related Work . As we discussed in Section 1 , our work is closely related to continuous domain adaptation where time can be treated as a continuous valued domain index . Approaches tackling such problems can be broadly classified into three categories : ( 1 ) biasing the training loss towards future data via transportation of past data , ( 2 ) using time-sensitive network parameters and explicitly control their evolution along time , ( 3 ) learning representations that are time-invariant using adversarial methods . The first category augments the training data , the second category reparameterizes the model , and the third category redesigns the training objective . We discuss each of these in detail . Transportation Methods Transformation based approaches [ 10 , 11 ] map data samples from the past distribution into the target distribution , and train a classifier on the transported samples . These have been adapted to situations in which the domains change continuously in [ 3 ] and [ 2 ] . [ 3 ] proposes CDOT , which applies optimal transport with a regularizer to take into account the similarity between two successive distributions . [ 2 ] modifies the approach to work in an environment where the target data is given as a stream , which is continually changing . To compute the transformations , these methods require a similarity measure on x , along with unlabelled data from the target domain , which makes them unsuitable for our settings . Further , the transformations induced may be too simple for high-dimensional inputs x , and may not scale on large training sets . Parameter smoothness Another approach is to make the model parameters a function of time , and to control their smoothness along time . Mancini et . al . in [ 9 ] propose Adagraph , a deep-neural network for image data where the scale and shift batch norm parameters are domain-specific . They propose using a user-specified kernel to capture the relationship between the domains , and computing the test domain-specific parameters using a kernel regression on the train domain parameters . This kernel is a hyper-parameter and needs to be carefully designed . The method can not hence discover relationships between domains in an end-to-end fashion . Other methods like [ 7 ] and [ 8 ] consider only a logistic regression classifier and model the parameters as stochastic processes like vector autoregression ( VAR ) or a Gaussian process ( GP ) . These methods require computing the posterior of GP-smoothed parameters , and this inference does not scale for multi-layered neural networks . Time invariant representations A recent state of the art approach for continuous domain adaptation is CIDA [ 5 ] that takes time as input but requires the neural network to encode [ x , t ] as a time-invariant feature vector to be passed to a common classifier . The encoder is trained using standard two-party adversarial game where the discriminator tries to recover time using a regression loss , and the encoder attempts to suppress time . It is well-known that training such two-party games is challenging and does not converge easily . Even when such convergence is achieved , we argue that a better strategy is to allow the features to evolve with time , while controlling the smoothness of this change . Domain Generalization The goal of such methods is to generalize to an open unordered set of domains given multi-domain data during training but no unlabeled data during deployment . Several methods have been proposed for this task including domain invariant networks [ 12 ] , decomposition methods [ 13 ] , meta-learning [ 14 ] or data augmentation with domain perturbations [ 15 ] . Our problem can be considered a special case of domain generalization where the domains are ordered , and we only care about generalizing to an adjacent closed interval . Another emerging area deals with settings where the labeled data is from a fixed domain but the target domain evolves with time and is unlabeled [ 16 , 17 ] . This is different from our setup where we assume evolving labeled data in the source , and one target domain without any data . Online Learning [ 18 ] Also called continual learning [ 19 ] and life-long learning , attempts to learn models using training data that arrives in a stream . These have been extended to deep networks [ 20 ] using techniques like meta-learning [ 21 ] and randomized stochastic gradient descent [ 22 ] . Methods such as FTPL and FTRL [ 23 ] could also be extended to provide an alternate approach to the problem . However , online methods do not allow revisiting the training data , unlike in our case . In addition to the above techniques we explored a number of other strategies for augmenting training data in the future in addition to transportation methods . We devised a generative model and a local perturbation method inspired by [ 24 , 15 ] . We also explored the use of neural ODEs for transporting data along time since these are specifically meant to capture continuous dynamics . These approaches are discussed in detail in secction A.2 of the supplementary . None of these approaches provided as much consistent gains on a variety of data types on modern neural networks , as this surprisingly simple gradient interpolation method that we describe in the next section .

Summary: The authors propose a novel loss (the gradient interpolation loss), paired with various architectural innovations and a training algorithm, in order to train models that generalize gracefully under distribution shifts that happen over time. The main contributions are as follows: 1. The gradient interpolation (GI) loss, a regularization term that penalizes the curvature of the learned network along time.  2. The TReLU activation, a novel activation whose slope and threshold are learned to vary wrt. time.  3. A training algorithm that uses the GI loss to train networks that are robust against time-dependent distributions shifts.   The contributions of the experiments are as follows: 1. The proposed method seems to perform better than a number of selected baselines and state-of-the-art domain adaptation methods.  2. Further insight is derived from synthetic experiments that aim to understand the mechanisms by which the proposed method achieves robustness against domain shifts.