Venue: NeurIPS 2022 Workshop NeurReps

Reviews:

final_score: rating: 4: Technically solid paper with substantial limitations.
review_content: summary: The authors propose a method to address the problem of catastrophic forgetting in the context of continuous learning tasks in NN, by regularizing the loss term by a term that enforces the topological structure of the network through its 1-skeleton representation, and is computed using closed forms of the Wasserstein distances and Barycenter.

They test and benchmark their method on various datasets and against state of the art methods, with improved performance reported by adding the regularization to reservoir sampling and ring buffer strategies.
strengths_and_weaknesses: The authors provide an interesting and original approach by enforcing the topological features of the neural networks in the context of continuous learning, and use a nice mathematical graph representation to do so. While the approach is technically sound, I find the authors perhaps needed more time before the submission to generate more results to support their claims and interpret the current findings. As it probably led the submission to be put in the extended abstract track, it also made the paper tedious to read, with a lot of back and forth between the main sections and appendices, with a structure and arrangement of figures that is suboptimal for clarity. With that said, I believe that with more maturation, this work can lead to a significant contribution to the field.
questions: As already mentioned, the structure of the paper should be much improved for clarity. Appendices should not be written as an augmented version of the text, but section that complement the main text with technical details. They should also be put in order of appearance. Figures should also be put in the main manuscript since they are fundamental to understand and visualize the approach (especially Fig 1 and 2). The paper also crucially lacks of a discussion about the limitations and future works that would help to gain a better perspective on its potential contribution to the field.

I am confused by the results obtained by Orthog-Sub, since  much better performance was obtained on a similar set of tasks as the ones in the paper. (86.6% ACC on permuted MNIST, Chaudhry et al. 2020, as cited in the paper). Can the authors explain the discrepancy?

Why don't the authors also consider the FWT as a metric of performance?

Found few typos or grammar mistakes (but I suspect there could be more): sequantial -> sequential ; we penalize changes (...) while allow -> while allowing p. 13
limitations: The authors did not address the limitations of their work. More experiments should in particular be conducted to assess the impact of the method, e.g. looking at the performance as the number of tasks increases, convergence of the loss function etc.
recommended_decision: 2: Borderline
review_title: Review of Topological Continual Learning with Wasserstein Distance and Barycenter

final_score: rating: 6: Technically solid, moderate impact paper, with no major concerns with respect to evaluation and reproducibility.
review_content: summary: The authors propose a new topology-based regularization that encourages a neural network to learn modular structure during training by penalizing cycle structure. They have covered the cases for both shallow and deep neural networks of limited width. Finally, they demonstrate this regularization method on Permuted MNIST and Rotated MNIST datasets, using average accuracy (ACC) and backward transfer (BWT) as the perfomance measures. 
strengths_and_weaknesses: Originality:
- (strength) The paper clearly demonstrates originality. It is a very interesting application of topological data anlaysis in continual learning. The connections seem surprising at first but make sense in retrospect. There is a lot of potential in this idea to be developed into a full paper. 

Quality:
- (strength) There are both theoretical and numerical results. Some of the details are in the appendix, but overall the results are substantially justified. 
- (strength) The numerical results are based on well-established datasets (P-MNIST and R-MNIST). 
- (weakness) The paper claims to show that the method applies to both shallow and deep neural networks. However, numerical results are only for neural networks with two hidden layers (deep neural networks). 

Clarity:
- (strength) The paper is mostly well-written, with clear statements and explanations for relevant mathematical definitions/theorems. 
- (weakness) Certain parts of the paper lack elaborations - particularly the results section. Both the presentation and analysis of results are not sufficiently clear.

Significance. 
- (strength) The ideas presented in this paper are very relevant to the theme of the workshop. Including this paper will benefit the discussion at the workshop. 
- (weakness) The sections on conclusion and limitations are missing from the main text. (Given the preliminary nature of the workshop, however, this might not be a crucial drawback.) 
- (weakness) The paper has never explained how significant the improvements (in terms of ACC and BWT) are the in the context of the tasks or in the context of continual learning in general. 
questions: - Since the aim only involves detecting graph cycles, could there be potentially simpler graph-theoretic methods that achieve the same aim without invoking topological methods? Why is topological data analysis particularly suitable for this application? 
- Why it is desirable for the network to learn modular structure? Perhaps the authors have some justifications in mind, but the paper has not explicitly addressed this in the main text. 
- What are the implications of the improvements in terms of ACC and BWT? How to put a few percentage of improvements into context? Are such improvements considered a major progress in the context of the tasks or in the context of continual learning in general?
limitations: The paper does not discuss limitations, which might be due to page limit. 

Here are some suggestions: 
- The authors could further develop a more thorough analysis and evaluation of results. 
- The results for shallow networks seem to be missing from the paper. 
- Some ideas could be better motivated. For example, the authors could elaborate more on: 
-- Why is topology particularly suitable for this task? Why not use other graph-theoretic approaches to 
-- Why should the network learn modular structure (i.e. why is it good to avoid cycle structures)? 
-- How significant are the improvements? What are the implications for continual learning?


recommended_decision: 3: Accept
review_title: Interesting idea with potential, but requires some further work on the analysis and evaluation of results

final_score: rating: 7: Technically solid paper, with moderate-to-high impact, and good-to-excellent evaluation and reproducibility.
review_content: summary: This article proposes a topological continuous learning method that uses a regularization based on Wasserstein distances from optimal transport, and the addition of episodic memory to prevent forgetting in neural network learning.
strengths_and_weaknesses: The article's idea is interesting, and the presentation is good. I thus recomment acceptance.

The quality of the writing is good in general, but a bit casual in some places. The extended abstract would be improved by redoing the language in this instance:
- Section 2: "Represent a neural network as an undirected ... [...].", "Create [...]."

The mathematics presented at the end of Section 2 and in Section 3 is a bit heavy and sudden. The article could be improved by adding a few more definitions.
questions: -
limitations: The submission does not really discuss limitations in enough detail. It could be improved by adding an expanded discussion of this sort.
recommended_decision: 3: Accept
review_title: Review of extended abstract "Topological Continual Learning with Wasserstein Distance and Barycenter"

Decision: Accept (Poster)