Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time

1 Introduction . Many organizations operate their machine learning pipeline as follows : collect labeled data based on day-to-day operations e.g . user-clicks for a recommendation model ; periodically ( say every ∆ = 48 hours ) train the model on the data collected from the past ; deploy the model to serve requests of the next ∆ period ; repeat the cycle . Real-world examples of such usage include recommendation models for e-retails , loan approval models in banks , regression models to forecast future popularity of media content , classification of malicious websites , and activity recognition models from sensor data . A key motivation behind such periodic retraining is the implicit knowledge that the data distribution is not stationary , requiring a revision of the decision boundary with time . While incorporating fresh data in training does make the trained model more current , we investigate if we can improve the training even further . In particular , when training the model at a time t , we know that the model will not be deployed on data samples in the training interval , but on data from the immediate future [ t , t+ ∆ ] which could differ from even the most recent training data . Our goal in this paper is to use this knowledge more centrally to train a model tailored to the intended deployment period . Formal Problem Statement . We consider prediction tasks over an input space X and output space Y where the joint distribution P ( X , Y ) evolves with time . The prediction model should therefore be conditioned on time as P ( y|x , t ) for x ∈ X , y ∈ Y , and time stamp t. During training we are provided with labeled data from T time snapshots t1 ≤ t2 ... ≤ tT . Call these D1 , D2 , . . .DT where each Ds = { ( xi , yi ) : i = 1 . . . ns } is assumed to be sampled from P ( x , y|ts ) . The trained model ∗Equal Contribution †anshulnasery @ gmail.com 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . will only be deployed on data from a time interval in the future which we denote as tT+1 , that is on examples sampled from P ( x|tT+1 ) . Note that we only evaluate performance once on time tT+1 , differing from online learning where regret is computed incrementally . Following most practical setups , we assume that the temporal drift across time is not too high . Since the deployment time is in the future , we can not assume the presence of even unlabeled target data at the time of training the model . Further , at different time snapshots , completely unrelated sets of xi may comprise a Ds . This makes the problem much harder than standard time-series forecasting tasks , since we do not have values of the same instance at different points in time . Our goal is not to generalize to all future data , but rather to data relatively close in the future , since our motivating application is one where models are retrained periodically . The former generalization would require very different models and approaches than what we propose . We assume that P ( y|x , t ) is implemented as a general-purpose neural network F ( x , t ) that takes as input x , t and outputs either ( 1 ) a vector of logits for classification tasks or ( 2 ) mean and other scores defining a continuous distribution over predicted y for regression tasks . A solution often used in practice is to only train the model on recent data and/or apply a graceful ageing penalty on past data . However , this may not be adequate with data hungry modern networks , especially when non-obvious seasonality makes distant data relevant . So we seek general purpose solutions that can harness all available data for making optimal decisions into the future . Any such solution first needs to make the model time sensitive . This can be done by feeding time as an input feature , or by making all or some of the model parameters to be a function of time . However , that by itself can not be sufficient to generalize to future data if we train the model to minimize loss on the training snapshots . In fact , a sufficiently complex model can easily overfit on training time-stamps when the number of instances per snapshot is large and the number of time stamps T is small . The field of continuous learning [ 1 ] offers one approach to tackle this problem . However , in our case the training data can be revisited and fresh models retrained making issues like catastrophic forgetting irrelevant . More relevant to our problem formulation are recent continuous domain adaptation methods [ 2 , 3 , 4 , 5 , 6 ] which either try to predict how the training domains evolve as time passes , or learn feature representations which are invariant across time [ 4 , 5 ] . However , these domain adaptation approaches assume the presence of unlabeled data in the future , which is not available in our case . Another set of approaches aim to make network parameters a function of time [ 7 , 8 , 9 ] but they require external smoothness kernels as hyper-parameters for each parameter-type , which can not be easily trained end-to-end on deep neural networks . The key objective of our desired training algorithm is to generalize to the near future , even at the cost of performance on past data . In order to achieve this goal , the training algorithm needs to encourage the model to adjust for temporal drifts both in the data distribution and the decision boundary , and be able to generalize and extrapolate well taking into account such shifts . To this end , we aim to develop a supervised learning model which would be robust to small perturbations in time— which is fed as feature . We attempt learning functions that are smooth in time , ensuring a limited temporal drift from past to future . However , given a limited number of training time-stamps , a moderately complex model is prone to fit the training data perfectly , without generalizing it well to the future . To tackle this problem , we propose a method which can learn to generalize well in time by interpolating the predictor between two training time steps using a first order Taylor series expansion . This interpolated predictor is supervised using the available labels . While doing so , it makes optimal use of the temporal information , which allows it to extrapolate well on future time steps . Our gradient interpolation method gives us the privilege to use a complex temporal model even if the number of timestamps is small without the fear of overfitting . To that end , we develop a novel neural model that captures the evolving decision boundary over time . More specifically , we design a novel time dependent activation function called leaky Temporal ReLU ( TReLU ) , which changes the underlying activation threshold smoothly along time . Such a model along with the Gradient Interpolation ( GI ) based loss function allows our method to make the optimal use of the temporal information and thus extrapolate well to future time . Our Contributions . In this work , we propose a simple training algorithm to encourage a model to learn functions which can extrapolate well to the near future . We do this by supervising the first order Taylor expansion of the learnt function . We also propose a method to make neural networks time aware by modifying the leaky ReLU activation function to change with different time-stamps . We demonstrate strong empirical performance of our method by comparing against state-of-the-art methods and practical baselines on several real world datasets , and also give insights into its workings through a simple synthetic setup . 2 Related Work . As we discussed in Section 1 , our work is closely related to continuous domain adaptation where time can be treated as a continuous valued domain index . Approaches tackling such problems can be broadly classified into three categories : ( 1 ) biasing the training loss towards future data via transportation of past data , ( 2 ) using time-sensitive network parameters and explicitly control their evolution along time , ( 3 ) learning representations that are time-invariant using adversarial methods . The first category augments the training data , the second category reparameterizes the model , and the third category redesigns the training objective . We discuss each of these in detail . Transportation Methods Transformation based approaches [ 10 , 11 ] map data samples from the past distribution into the target distribution , and train a classifier on the transported samples . These have been adapted to situations in which the domains change continuously in [ 3 ] and [ 2 ] . [ 3 ] proposes CDOT , which applies optimal transport with a regularizer to take into account the similarity between two successive distributions . [ 2 ] modifies the approach to work in an environment where the target data is given as a stream , which is continually changing . To compute the transformations , these methods require a similarity measure on x , along with unlabelled data from the target domain , which makes them unsuitable for our settings . Further , the transformations induced may be too simple for high-dimensional inputs x , and may not scale on large training sets . Parameter smoothness Another approach is to make the model parameters a function of time , and to control their smoothness along time . Mancini et . al . in [ 9 ] propose Adagraph , a deep-neural network for image data where the scale and shift batch norm parameters are domain-specific . They propose using a user-specified kernel to capture the relationship between the domains , and computing the test domain-specific parameters using a kernel regression on the train domain parameters . This kernel is a hyper-parameter and needs to be carefully designed . The method can not hence discover relationships between domains in an end-to-end fashion . Other methods like [ 7 ] and [ 8 ] consider only a logistic regression classifier and model the parameters as stochastic processes like vector autoregression ( VAR ) or a Gaussian process ( GP ) . These methods require computing the posterior of GP-smoothed parameters , and this inference does not scale for multi-layered neural networks . Time invariant representations A recent state of the art approach for continuous domain adaptation is CIDA [ 5 ] that takes time as input but requires the neural network to encode [ x , t ] as a time-invariant feature vector to be passed to a common classifier . The encoder is trained using standard two-party adversarial game where the discriminator tries to recover time using a regression loss , and the encoder attempts to suppress time . It is well-known that training such two-party games is challenging and does not converge easily . Even when such convergence is achieved , we argue that a better strategy is to allow the features to evolve with time , while controlling the smoothness of this change . Domain Generalization The goal of such methods is to generalize to an open unordered set of domains given multi-domain data during training but no unlabeled data during deployment . Several methods have been proposed for this task including domain invariant networks [ 12 ] , decomposition methods [ 13 ] , meta-learning [ 14 ] or data augmentation with domain perturbations [ 15 ] . Our problem can be considered a special case of domain generalization where the domains are ordered , and we only care about generalizing to an adjacent closed interval . Another emerging area deals with settings where the labeled data is from a fixed domain but the target domain evolves with time and is unlabeled [ 16 , 17 ] . This is different from our setup where we assume evolving labeled data in the source , and one target domain without any data . Online Learning [ 18 ] Also called continual learning [ 19 ] and life-long learning , attempts to learn models using training data that arrives in a stream . These have been extended to deep networks [ 20 ] using techniques like meta-learning [ 21 ] and randomized stochastic gradient descent [ 22 ] . Methods such as FTPL and FTRL [ 23 ] could also be extended to provide an alternate approach to the problem . However , online methods do not allow revisiting the training data , unlike in our case . In addition to the above techniques we explored a number of other strategies for augmenting training data in the future in addition to transportation methods . We devised a generative model and a local perturbation method inspired by [ 24 , 15 ] . We also explored the use of neural ODEs for transporting data along time since these are specifically meant to capture continuous dynamics . These approaches are discussed in detail in secction A.2 of the supplementary . None of these approaches provided as much consistent gains on a variety of data types on modern neural networks , as this surprisingly simple gradient interpolation method that we describe in the next section .

Summary: ML models deployed in production pipelines are trained continuously to account for gradual shift in the data distribution. This paper deals with model generalization w.r.t. data in the near future in this continuous distribution shift. The paper proposes training a model with time-sensitive parameters where the temporal complexity is regularized by a Gradient Interpolation (GI) loss. Time-sensitive network architecture has two components: time-insensitive (basically standard neural network parameters) and time-sensitive parameters (which are the output of another small neural network which takes into account the time parameter). It also proposes a novel time dependent activation function, Leaky Temporal ReLU (TReLU), that allows for smooth changes along time dimension. Empirical evaluations demonstrate that the proposed scheme outperforms competitive baselines such as generative and adversarial approaches.