Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a novel loss (the gradient interpolation loss), paired with various architectural innovations and a training algorithm, in order to train models that generalize gracefully under distribution shifts that happen over time. The main contributions are as follows:
1. The gradient interpolation (GI) loss, a regularization term that penalizes the curvature of the learned network along time. 
2. The TReLU activation, a novel activation whose slope and threshold are learned to vary wrt. time. 
3. A training algorithm that uses the GI loss to train networks that are robust against time-dependent distributions shifts. 

The contributions of the experiments are as follows:
1. The proposed method seems to perform better than a number of selected baselines and state-of-the-art domain adaptation methods. 
2. Further insight is derived from synthetic experiments that aim to understand the mechanisms by which the proposed method achieves robustness against domain shifts. 
main_review: I believe that this is a good paper that makes a solid and interesting contribution on an important problem. I'm ready and inclined to increase my score based on the author response to the main weakness I list below. 

The main strengths include:
1. The proposed approach is interesting and, as far as I know, novel. 
2. The GI loss can, in principle, play well with arbitrarily complex models. 
3. The fact that time-dependence of network parameters can be achieved only through modifying the activation functions is noteworthy. (I'm not sure if this is a novelty)
4. The paper does relatively extensive ablation studies and runs other synthetic experiments to gain insight into how the proposed method works. 

___ The main weakness ___:
1. It's not clear how the hyper-parameters were tuned, both for the competing benchmarks and in the ablation studies. For example, was the learning rate separately tuned, albeit coarsely, for all different tasks on all different models? It's also important to learn more about how the competing approaches for picking the delta term (i.e. adversarial, random, adversarial with warm start etc.) were tuned, as well as the lambda coefficient of the gradient-penalty loss (Equation 2). In my opinion, if it turns out that selecting delta randomly from a reasonable distribution works as well as the adversarial version, I think that'd be a positive for the proposed method, as it'd remove the costly optimization step to optimize delta each step. 
2. It is specified in the appendix that the fine-tuning steps, which is where the GI loss is enforced, is applied only to the last k time stamps of the dataset. This is, I think, an important design choice and should be moved to the main paper. Additional clarification on the importance of choosing k properly is also needed (it seems like most experiments used k=2. How would be performance be if you used k=1, or k=3? ). (Please correct me if you've actually discussed this in the main paper and I've missed it somehow)

Other weaknesses include:
1. The proposed method is expensive (due to the online selection of the delta term in Algorithm 1), as acknowledged by the authors. The tasks that the proposed method is applied on are relatively small scale, making it difficult to see how much of a scalability issue the cost of enforcing the GI loss really incurs. A more detailed discussion (perhaps in the appendix) on how much of the compute is spent on different parts of the algorithm would improve the paper, though I don't think this is crucial.
2. (This might not be a weakness - I might have misunderstood something about TReLUs) You mentioned that each TReLU has neural networks "h" and "g" that have different parameters. If a single TReLU is a replacement for a single ReLU, then that'd mean we have to train thousands, maybe millions, of small neural networks that are inherently unparallelizable on GPUs. Doesn't this lead to the algorithm being very inefficient? (Again, I might have misunderstood this part and this might not be a weakness)
3. The delta term is optimized across minibatches and, in some configurations of the algorithm, warm started from the result found in the previous mini-batch. These break the iid. assumption. This should be acknowledged, as it could lead to unexpected effects. 
4. Overselling phrases like "[the proposed method] makes optimal use of the temporal information", as in line 77, should be avoided, unless the notion of optimality is defined properly and justified theoretically or empirically. 
5. Further details on the "TrainDelta" step in line 11 of Algorithm 1 would be useful to have in the paper. 




Questions to the authors: 
1. Also, do you backpropagate through the steps to find delta (I assume not, but this would be useful to clarify). 
2. How important is it to pick the capital delta terms (i.e. the window in which the perturbations are picked)?
3. Did you inspect how the learned slope and threshold values look like? 
4. Is the x axis of Figure 3 "number of TimeReLU units", or "number of timeReLU layers"? 
5. Could you clarify on which task Figure 4 was obtained on? 

Writing, typos etc.
1. The paper is well written, well organized and easy to follow. 
2. It has a small number of typos: 
- line 75: "we" is duplicated. 
- line 155: repeated ".."
- line 180: nerual -> neural
3. The math expression under line 196 is a big confusing. 

====Edit after author response====
I thank the authors for their response. Acknowledging some of the limitations that were pointed out during the review process, I believe that the methods and results are strong/relevant enough to be of interest to the NeurIPS community working on domain generalization with extrapolation to near future. This is why favour acceptance. 

I encourage the authors to improve upon their expository toy example to demonstrate the utility of their approach better, and in general try to gain a more mechanistic understanding for why their approach seems to have good empirical performance and how this compares with the ablation methods, such as regularizing the gradient. I also urge the authors to improve on writing to make the problem definition as well as key design decisions clearer to the audience. 
limitations_and_societal_impact: The limitations, which are closely intertwined with the weaknesses, are outlined above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: ML models deployed in production pipelines are trained continuously to account for gradual shift in the data distribution. This paper deals with model generalization w.r.t. data in the near future in this continuous distribution shift. The paper proposes training a model with time-sensitive parameters where the temporal complexity is regularized by a Gradient Interpolation (GI) loss. Time-sensitive network architecture has two components: time-insensitive (basically standard neural network parameters) and time-sensitive parameters (which are the output of another small neural network which takes into account the time parameter). It also proposes a novel time dependent activation function, Leaky Temporal ReLU (TReLU), that allows for smooth changes along time dimension. Empirical evaluations demonstrate that the proposed scheme outperforms competitive baselines such as generative and adversarial approaches.
main_review: 
Strengths:
---------------

- Separation of the neural architecture into time-sensitive and time-insensitive parameters: (a) incorporating time as a feature using TIME2Vec embedding model, (b) time dependent feature processing using the proposed TReLU activation function.

- Gradient Interpolation loss to incorporate adversarial time dependence on the loss w.r.t. time-dependent component of the network.

Weakness:
---------------

- The underlying problem seems like an ideal setting where online learning should have been explored first. 

- I have some reservations on the empirical evaluation of the scheme ( see below ). Barring the two synthetic classification datasets, the proposed scheme performs poorly than the baselines on the remaining classification datasets. Are the gains on the regression datasets mainly due to the fact that the time-representations of the input is better than the raw features?


Questions for Authors:
---------------
- Why can the underlying problem not be taken as an instance of an online learning? In online learning, at any time t, you play parameters W_t, and you receive the next data (which could even be adversarial in nature, specifically designed to fool W_t), and you update your parameters to adjust to this data? There are many learning schemes which have been deployed in this context such as Follow-The-Regularized-Leader, Follow-The-Perturbed-Leader, and other advanced variants of these schemes etc.

- What is the best one can hope for this continual setting? Since the assumption is that near future data may have no relation to the data recently seen. Could you expect any classifier to perform better than one jumbo network trained with all the data in the past (paper mentions that data in the past could be repeated in future)? Correct me if I'm missing some critical peace of this puzzle.

- The Toy example in the Sec.~3.2 with the synthetic setup is not convincing that the proposed scheme works as intended. Coin toss accuracy is 50%, ERM achieves 49%, GI (proposed scheme) achieves 55.4%, and scheme with gradient regularizer achieves 47.4%. Its not clear if this example can help you make any conclusion about any of these methods as they are very close to the random chance on this problem. Maybe this is not a good toy example to showcase the benefits of the proposed scheme.

- I'm a little lost in the empirical evaluation section. Given a dataset that has time components and a train/test split. How is the training done? How is the network evaluated? Is it just trained on the training set and simply evaluated on the test set? I expected the evaluation to be incremental where in you train with train data upto a certain time and then evaluate on the data past this time instance, and so on.

- Figure 4 is missing in the main text and I see no figure 4 in the appendix.

Writing Clarity:
---------------
- Suggestion to include an explicit assumptions paragraph, which states whether the near-by future has any correlation to what was seen recently or could it be related to anything in the distant past?

- Missing related works and connections with online learning

- Line 75: we we 


limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents the Gradient Interpolation (GI) loss to regularize a time-dependent network, aiming to generalizing the predictions to the near future. The GI loss is computed as the maximum of the loss within a short window, which encourages the network to be locally linear in time. The paper further designs the Time Leaky ReLU (TReLU) which uses the time feature to scale the standard ReLU functions. Through a list of classification and regression datasets, this paper demonstrates its superiority against baselines.
main_review: ### Overall Comment
To make the agent to extrapolate towards future, it is important to uncover the temporal dependencies. This paper attempts to learn a network that take in both the input and the timestamp. This paper presents the GI loss to regularizing the predictions of the network within a short time window. However, I am not fully convinced by the arguments and the analyses.

### The Synthetic Problem
Sec 3.2 is very difficult to understand for me .
1) I cannot understand the conditional distribution: $P(x_j | y, t) = P(x_j = y | t) = p_j (t) $. Why $x_j$ appears only in the first equation ? 
2) Since a random guess is $50$%, the numbers $49$%, $55.4$%,$49.5$%  do not seems to statistically different to me. In consequence, I do not think this experiment makes a positive evidence. In fact, I don't think any simple neural network could uncover the one-to-one relationship between $ j $ and $ t  $ with data only in past domains.
4) I think a more proper gradient regularizer than Eq(2) is to regularizing the gradients over losses, e.g., $ \| \frac{ \partial l(y, x, t) }{\partial t } \|_2^2 $. This regularizer is also similar to the TimePerturb approach in Sec 4.5, which works comparably to GI.

### TRAIN DELTA
Given that the GI loss involves the gradient already, performing gradient descent to find $\delta$ requires evaluating the second-order derivatives. And multiple gradient descent updates are needed to find the optimum. Therefore, the resulting approach seems computationally very heavy. Could you also provide a running time comparison with the baselines in your experiments ? 

### Pretraining
1) How the pretraining is done in an online setting, where tasks come one-by-one ? 
1) Line 3-5 in Algorithm 1. Is the pretraining done one-by-one sequentially ? Given you assumed knowing all tasks, this does not seem necessary. 

### Experiments
1) Could you present a clear definition of the training and the testing procedure ? 
2) How many more trainable parameters are used in TReLU, compared to the standard network and the baseline approaches ? 


### minor comments
1) Line 75: "we we "
2) Line 278: "the angle of rotations"
3) Line 301: "lower errors"
4) Line 305: "drifts"
5) Table 2: "performances", "the classification task", "the regression task"
6) Line 369: "Furthermore, we use ..... can make any model...."
7) Line 192: $ [0, 1]^d $, should use curly brackets for the binary case.
8) Line 196: $p_j(t), t\geq 3$
9) Line 262: I cannot understand the logic of this sentence: "we ideally require a universally approximator of time-dependent function", given that the proposed TReLU is not provenly universal.

limitations_and_societal_impact: To further improve the paper, I think these are potential directions,
1) A clearer discussion of Sec 3.2
2) Discussing the computational overheads by training deltas, and the parameter overheads by TReLU
3) A clear description on the workflow

 (a) how the pretraining is conducted 

 (b) what is the training and the testing procedure

This paper does not lead to negative societal impacts.
review_title: null

Meta-review: The reviewers feel that this paper introduces an interesting and potentially useful approach to learning in changing environments. They consider the approach sensible. Various reviewers originally had concerns both about the computational cost and about the experimental validation (especially details of hyperparameter optimization and choices of tasks and baselines). After some discussion, the reviewers feel like the authors have addressed their main concerns, and all of them recommend acceptance. I concur.


Decision: Accept (Poster)