Title: No title found

Abstract: No abstract found

:



Figures and Tables:

Peer review (iclr2020):
The paper addresses the question of how the topology of intermediate representation spaces learned by a neural network changes, depending on the architecture and, in particular, depending on the used activation functions. One of the main conclusions is that the effectiveness of ReLU networks stems from the fact that ReLUs are not homeomorhisms, thereby allowing the topology to change more rapidly and in ways not possible with, e.g., a tanh activation. The authors study the Betti numbers of the intermediate representation spaces and provide empirical evidence for their claims mostly on toy data and on one (small-scale) real-world experiment. Overall, the paper is written in an understandable manner with a clear motivation. I did appreciate the well-done introduction of homology & Betti numbers (App. D) and implementation aspects (in App. E) which clarified some issues which were not clear from the main text. While the authors experiments confirm their claims about the effectiveness of ReLU-networks, I think this conclusion is also somewhat obvious, given that ReLUs would allow for more *freedom* in changing the topology. In that sense, the novelty of the paper is somewhat limited, especially as many other people have investigated the behavior of neural networks from a topological perspective recently (e.g., the Neural persistence work by Rieck et al., or the work mentioned in the comment by W. Guss). Some parts of the paper are also fuzzy (or imprecise) in terms of the statements made: - what is meant by *doubly complicated topologies* or *complicated topology* in Sec. 3? - the notion of the term *dimension* is inherently ambiguous in many parts: e.g., *Note that dimension is a topological invariant ...* - What is meant here? E.g., the Lesbeque covering dimension? Such statements need to be more precise. - In App. C. Thm. C.3. I would repeat the precise statement of what is meant by *inside* - Without reading Lemma C.1. this is unclear. - Also, in App. C. Thm C.3. you argue that *for any set of parameters* the network is a near-diffeomorphism. So this would allow all weights to be 0, right? I think the argument, as the authors state, is that a constant map with a *tiny* gradient (whatever tiny means) turns it into a diffeomorphism and then you are in the setting of *near-diffeomorphisms* (C.2). Is that correct? - Thm. C.3. also requires d>=n_1 >= ..., so most conv.-nets would not fall into this setting, as typically the output in the first few layers is larger (due to the #filters) than the input. Maybe I missed that point, but this should be highlighted. On a more general note, the experiments, in my point of view, do not meet the ICLR standards, given that the theoretical part is rather shallow (except for some parts of the Appendix). I do appreciate the experiments on toy data, for which you actually have a good handle on the topology of the input and can then study how this changes throughout a forward pass. However, on real data (e.g., CIFAR10/100, etc.), there is very little that we would know (as admitted by the authors) . The experiments on MNIST behave somehow similarly to the toy experiments, but the effects are less obvious in my point of view. This also brings me to the point of why limit the study to Betti numbers and not take advantage of the actual barcodes (which the authors compute anyway as far as I understood), thereby eliminating the pain of identifying a scale. The authors argue that statistics on barcodes is still an active research field (which is true), but many people have worked on this in the past, and many good approaches exist (e.g., P. Bubenik*s work on statistics with persistence landscapes (JMLR), Kwitt et al.*s work on statistical analysis with kernels on persistence diagrams (NIPS), Fred Chazal et al.*s work on Stochastic convergence of persistence landscapes and silhouettes, Monod et al.*s work on Tropical Sufficient Statistics for Persistent Homology, etc.). These ideas could be easily leveraged in my point of view (or at least acknowledged). In summary, the paper touches upon a quite interesting subject (which deserves further investigation), however, the paper appears (1) immature in many parts and (2) could benefit from a more extensive experimental study on real-world data. Toy experiments are fine to illustrate the concept, but at some point, we need to move on to actual data.