Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes topological attention, a type of self-attention mechanism that captures the topological information of the signals. In contrast to earlier works, this work focuses on local instead of global topological summaries. The main difference is attributed to the extraction of such summaries for each sliding window and not over the global historic data. The paper discusses how such sets of local summaries can be vectorized and incorporated into neural network architectures. The experimental results on an existing forecasting approach show the benefit of topological attention vs. the regular (global) attention mechanism.
main_review: The paper is well-written and the introductory figures help in the understanding of the proposed approach. 

The novelty of the paper is somewhat limited, considering that, conceptually, the main difference is the use of an existing approach over sliding windows. However, the vectorization and integration of such extracted topological summaries are not trivial and, therefore, there is merit in having such an approach documented.

The results are not impressive. For the single time series experiments the method used to incorporate the topological attention seems to already perform surprisingly well due to the simplicity of the task. However, it is interesting to see that standalone topological features without attention mechanism reduce performance and, therefore, the benefit of using both approaches combined becomes evident.

The results on the M4 forecasting competition show similarly not impressive trends: the performance depends on the horizon, some times a simple attention mechanism performs better. However, even a 1% improvement might be statistically significant considering the size of the corpus here. The ablation study creates a much clearer picture of the benefits.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents an approach to combine topological features to time-series forecasting.
Essentially, the method computes 1D topological barcodes at each time-steps, pass them to a transformer and then provides those features as additional input to NBEATS. Experiments are conducted on two datasest when performing local forecast (e.g. forecasting the time-series independently) and on M4 when using global forecast (learning the model jointly across time-series). An improvement of 0.4%/1.2% is reported on M4 for OWA/sMAPE.

main_review: The paper is clearly written and well positioned compared to existing work both in forecasting and application of persistent homology for ML. The method is original and while reusing existing techniques from persistent homology, the contribution on time-series is novel and interesting on its own.

My main concern for this paper lies on the experiments. Tiny improvements are reported compared to existing approach and it was not clear how much improvement can be attributed to the technique proposed compared to other factors (noise, HPO).

First, the paper reports a specific setup for their method where learning rates are adjusted per components: for instance for the univariate setting, the learning rates are 9e-2 (for topological feature mapping), 5e-2 for the transformer taking topological features and 9e-2 for the output linear projection. For the M4 setting, different learning rates values are used. Introducing different learning rates is problematic, it would require an additional HPO budget (especially given that the two settings have different values) and it is not clear if the improvement reported by the authors would hold if a similar setup would have been used for NBEATS. I believe, a single learning rate should be used in all models to have a more apple-to-apple comparison.

Second, the results reported by the authors are very small (0.4%/1.2% improvement on M4 for OWA/sMAPE). This also raises a few questions. The noise on M4 is very large: different network initializations may give results differences that are close to the range reported by the author. As such, multiple seeds should be run to clearly show that the improvement reported is not simply due to noise. Also, while a useful ablation study has been carried out, it is a bit insufficient in my view. Part of the improvement could be done solely to the increase of parameters. As such, an ablation should also consider a model trained with roughly the same number of parameters to validate that the improvement comes from the topological features and could not have been achieved by "just" increase the hidden size of NBEATS. 

I hope those points can be improved by the author during the rebuttal. One thing I was hopping when reading the paper was also to provide more insights or qualitative visualizations on the topological features as they are quite new for time-series/ML readers (why do they work, how do they help to detect complex seasonality pattern).

Additional remarks:

- l148 the description barcode coordinate function should be included in order to make the paper self-contained, some space can be gain in this section by just saying in words that you concatenate the e topological features at each time-steps
- l196: I agree that the computational complexity would not suffered with the computation of barcodes. However, a more important discussion should be on the number of additional parameters to the initial method.
- l249: how did you find the learning-rate per component? It would be challenging for the method to tune 3 learning-rate on each dataset and this diminish the value of the contribution.
- Table 2 what are (2) and (10)?
- l288: "However, in the latter strategy, the input dimensionality for the transformer
 encoder scales with the sliding window size n." I do not find this argument convincing, one could slice the number of observations to the same dimensionality as your topological representation.
- l324: "1.2% improvement over the M4 winner" this number has to be compared with the noise obtained with multiple seeds.
- l69: *Single* time series experiments


-- edit after rebuttal

I would like to thank the authors for the clear answer to the points raised in my reviews regarding hybrid learning-rates, difference in number of parameters and improvement reported compared to noise sensitivity (the latter point that seems has been noted by the other reviewers as well).

The answers and detailed analysis clarified my doubts, there is an improvement that can be clearly attributed to the method proposed (but require careful examination, I hope the analysis can be included in the manuscript). While the improvement is small, the benchmark considered (M4) is also highly competitive and the method is original and promising, as noted by other reviewers, as such I will raise my score.



limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new approach for one-dimensional time-series forecasting that is based on augmenting the data with topological information. Specifically, the authors design an attention module that extract the local connections in their windowed persistent homology barcodes, and concatenates this information to the original time-series. The authors evaluate their method on two scenarios: a single series and multiple time-series elements. Their techniques shows modest improvement over existing approaches.
main_review: This submission proposes an interesting idea: in addition to operating on the raw data, it suggests to augment the data with local topological information that is drawn from persistent homology computations of the data. In practice, the proposed method includes a module that generates the persistent barcodes, and an attention module which extract local connections in the windowed barcodes. To the best of my knowledge, this idea is novel and was not suggested before. In general, I like this approach, as it allows to connect a fundamental problem (time-series forecasting) with an established theory and practice (persistent homology). As such, this work has the potential to inspire further research that combines persistent homology in the context of time-series processing or other machine-learning problems. Overall, the exposition is clear, and the authors made an effort to include many details about persistent barcodes and their computation. I believe their description in 3.1 & 3.2 could be further, however, I do not have a concrete suggestion on how to do it---perhaps use less general descriptions. For instance, you can directly write what is the particular $mathcal{V}_\theta$ you used right below eq. 2.

The main issue I found with this submission is the somewhat on-par results in the multi-series setting. In particular, the improvement over N-BEATS in Table 3 seems marginal. Moreover, the comment at the end of pg. 8 makes me think that the problem is more fundamental. Namely, since the authors did not use the ``best'' version of N-BEATS, and improve on a subpar version of it, it might be that the proposed mechanism yields no improvement when added to the full N-BEATS. At this point---we have no way of knowing. With that being said, I am still on the positive side, given the potential of this work.

limitations_and_societal_impact: See above.
review_title: null

Meta-review: The paper proposes a form of local topological attention to improve the performance of a time series forecasting model. The reviewers agree that the proposed method is interesting and relevant, and the writing is clear. 
On the other hand, all reviewers caution that the performance improvements demonstrated empirically are small (~ 1% on M4), and two reviewers see the novelty as somewhat limited. However, the authors were able to alleviate further concerns regarding the empirical evaluation during the discussion period, leading all reviewers to vote for (weak) acceptance. The reviewers belief that the merits outweigh the flaws of the paper, and I concur with their assessment, recommending acceptance.
Decision: Accept (Poster)