Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The authors proposed a new embedding for time - Time2Vec. Unlike previous research that is either proposing a new architecture or proposing expensive handcrafted features, this work proposes a model-agnostic learnable time embedding.

I would like to recommend an accept based on the following reasons:
* Modeling time is crucial for quite a few machine learning tasks. With the two most desired properties, learnable and model-agnostic, this time embedding will be very useful in various applications.
* The authors are good at story-telling and this makes the paper very readable and approachable. This increases the chance of the contribution made in this paper to be applied in real-world applications.
* This work did clear and detailed analysis on both the empirical results and the probing experiments.


 
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper introduces a particular learnable vector representation of time which is applicable across problems without the use of a hand-crafted time representation. Their representation makes use of a feed-forward layer with sine activations which operates on time data. As it is a vector representation, it combines well with other deep neural network methods. They motivate their problem well, explaining why time data is important to a variety of problems and situate their solution as an orthogonal approach to many current solutions in the literature. They make reference to fourier analysis as motivation for their representation. Finally, they provide experimental results to support their claims using fabricated and real-world time series datasets, as well as ablation studies to support their design decisions.

While I think this work has the potential to be a significant contribution, I rate this a weak reject because the theoretical motivation and analysis of the experimental results are lacking the depth of evidence I would expect for an ICLR paper. If you provide a deeper discussion of the provable claims about the power of your model via Fourier analysis and provide a table of test accuracy/recall@K with/without your representation for more than one other state of the art algorithm for these datasets, I would be convinced to strong accept.

Specific comments:

* p.3 third paragraph: you repeat yourself in math notation a few times here. Repeated equations usually indicate that there is something new happening, but all of these are just restatements of your theta sin(omega tau + phi) term. I would introduce the notation for t2v(tau) upfront and use that to define a(tau, k)[j] and f_j
* p.3 A clearer explanation of the theory here would help, as I think Fourier's theorem nicely supports your claims.
* p.4 first paragraph you claim that this method responds well to data which exhibits seasonality, but none of your datasets deal with data that would exhibit seasonality. There are plenty of simple real-world datasets available which show multi-scale periodic phenomena (activity or location data, weather data, travel data, etc.). In fact, segmentation and recognition of wearable device activity would be a great application for this method.
* p.4 third paragraph: Your claim of invariance to time rescaling is technically correct, but I am not convinced that a model can learn the correct omega values for an arbitrary rescaling (e.g. if the period is smaller than the time unit). You show that this works for a rescaling from 2pi/7 to 2pi/14, but it would be nice if there was experimental confirmation of this property with frequency > 1.
* p.6 Showing accuracy/recall across training epochs is not sufficient evidence to show that this is a useful representation. There should be some kind of comparison with test set results from other state-of-the-art work on these datasets. If adding your representation to the SOTA model improved test set performance (or at least sped up training without hurting test set performance), then that would be better evidence. If LSTM+T is the SOTA, say so and restate the author's test performance compared to yours. If this is what these graphs show, consider using a different visualization to make it clearer that you're improving the final performance, not just the training process.
* p.8 I think sine functions make optimization harder because they make the gradient function periodic with respect to the weights, creating infinitely many local extrema. Historically this may have been an issue, but deep neural networks have so many local minima it might not matter. Still, it would be good to show that trained performance doesn't depend on the initialization values more than a standard LSTM+T model.
* You have an interesting corner case where your neural network parameters are interpretable: you can interpret the omega values from your model as frequencies and investigate their values to see which kinds of periodicity your model uses. You do something like this on p.7, but it would be neat to see a histogram like the one you have for EventMNIST for one of the real-world datasets to see if it learns the domain-relevant time knowledge you claim that it should learn.
review_title: Official Blind Review #5

final_score: rating: 1: Reject
review_content: ## Summary

The authors propose a method for encoding time features using a sine function with learned phase and frequency. They apply this method to several synthetic and real-world datasets.

Temporal and positional encoding is important to many applications, including NLP, sound understanding and time series modeling, so the topic is certainly of interest. However, the method they propose offers very little that is new when compared to e.g. Vaswani (https://arxiv.org/pdf/1706.03762.pdf, section 3.5) (the authors acknowledge this work several times). In addition, the authors compare to a baseline that seems to consist of passing time as a float. This seems like a very weak baseline---there are any number of other reasonable ways to encode this.

Due to the incremental nature of the improvement and the weak baseline, I don't think this paper should be accepted to ICLR.


## Specific Comments

1. In Section 2, I find the sentence "We follow a similar intuition but instead of decomposing a 1D signal of time into its components, we transform the time itself and feed its transformation into the model that is to consume the time information" really unclear. Could you rephrase it?

2. Often, positional encodings are used to encode ordering for a model architecture that is not inherently sequential. This is the case for the positional encodings in the transformer model. Did you try these encodings with non-recurrent architectures?

3. In Section 5.2, did you mean 'fixing t2v(\tau)[n] = sin(2\pi n \tau / 16)'? i.e. I think it's missing a 'tau'

4. In Section 5.2 "Fixed frequencies and phase shifts" you compare Time2Vec to a fixed set of frequencies. Since positional encoding with Fourier transforms is well known, this seems like the relevant benchmark but it receives only a brief treatment. The authors compare these methods only on Event-MNIST and only for 16 frequencies. I would like to see this comparison expanded.

5. Could you clarify exactly how time is encoded for LSTM + T? Are you, in fact, just passing a float value? How is this encoded for each data set? For example, the "times" for Event-MNIST is always [0, 783] while the SOF data has timestamps. What is the encoding scheme for each?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: # Summary
This paper proposes a simple representation of time (Time2Vec) for modelling sequential data. The idea is to apply multiple sine functions to the time with trainable period and offset and concatenate them together, which is similar to positional encoding [Vaswani et al.] except that the periods and offsets are learned. The results on several sequential modelling datasets show that Time2Vec performs better than naive representations and alternative baselines. 

# Originality
- Although the proposed representation looks simple and similar to positional encoding [Vaswani et al.], the idea of parameterizing sine functions is novel and interesting.

# Quality
- The proposed idea is very simple but seems very effective in practice as shown by the empirical results. 
- The paper also provides in-depth analysis and ablation studies showing that each of the proposed component is helpful. 
- Although the results presented in this paper look very promising, it would be much stronger if the paper presented results on other sequential modelling tasks such as machine translation and language modelling that the research community cares about much more. For example, it would be great if the paper showed that replacing the fixed positional encoding with Time2Vec improves the performance on a machine translation dataset.
- It would be good to show the effect of the number of sine units (k) in Time2Vec. 
- (Minor) Clockwork RNN [Koutn´ık et al.] introduces a nice toy periodic sequential prediction problem, where the model has to recover a mixture of sine/cosine function. It looks like a natural task to show in this paper as well (which can potentially replace the synthesized data experiment). 

# Clarity
- The paper is overall well-written.
- Figure 4 is not mentioned in the main text.
- Figure 5 is mentioned earlier than Figure 3. It would be good to swap them. 

# Significance
- This paper proposes a simple but effective idea that can be potentially widely used by the research community. The paper would be stronger if it included more results on high-impact sequential modelling tasks and datasets such as machine translation. 
review_title: Official Blind Review #1

Meta-review: This paper investigates and evaluates learning high-dimensional embeddings of time, which is useful for a variety of applications. This paper received 4 reviews (due to a missing review, we requested several emergency reviews). R1 recommends Weak Accept, calling the method simple but saying it could be of wide interest and utility in practice. R3 recommends Reject, identifying concerns about the significance of the contribution, caused by the simplicity of the approach, the connection to existing work, and missing comparisons to baselines. In a short review, R4 recommends Accept with several positive comments. In a long, thoughtful review, R5 recommends Weak Reject, due to concerns and questions about the theoretical motivation and depth of experiments. The authors have submitted detailed responses that have addressed many of the questions of the reviewers; however, R3 feels the response does not address their concerns, and R5 is closer to accepting but still feels additional improvements in presentation and experimentation are needed.

Given the split decision, the AC also read the paper. The AC agrees with R1 and R4 that this is an interesting problem and the approach here may be useful in practice, but shares concerns with R3 and R5 about the depth of contribution with respect to existing work, and need for additional experimental validation against stronger baselines. 
Decision: Reject