Title: No title found

Abstract: No abstract found

:



Figures and Tables:

Peer review (iclr2020):
The paper studies the superiority of ReLU activations and the benefit of many layers in a DNN from a topological data analysis perspective. An experimental study is conducted based on Betti numbers, which are adopted as a measure of topological complexity. It is shown that deeper layers attain progressively smaller Betti numbers until minimal values are reached. This is interpreted as the network changing the topological structure of the dataset, transforming it from complicated to simple. The advantage of ReLU*s is attributed to their nonhomeomorphism that enables topological changes, as opposed to smooth activations that preserve topology (and only change geometry). I think the paper is interesting and have enjoyed reading it. The proposed perspective is elegant and the experiments and illustrations are clearly presented. My main concern is twofold. First, the proposed approach doesn*t look applicable to large real-world dataset, for which DNN classifiers oftentimes behave differently as compared to small nets for synthetic data. This naturally limits the empirical reach of the paper and is the source of my 2nd and larger concern. I do not think the paper provides sufficient evidence to believe its observations generalize to large systems. Adding a larger scale experiment, even 10-class MNIST would somewhat help (though MNIST is typically too well-behaved). With that said, I don*t see the authors being able to deal with anything larger like, e.g., CIFAR-10? These are my main concerns. To sum up, while the paper is an enjoyable read, it is hard to quantify its contribution in progressing our understanding or practical systems. The results are not strong enough to convince they are not artifacts of the synthetic setups. Furthermore, it is unclear what is the paper*s take home message. The last sentence says something about designing NNs with an eye toward topological changes. But what does it actually mean? How should a systems designer take this into account? The paper doesn*t provide tools to do so for actual systems, which goes back to the points above. Can the authors say what they perceive as the broader impact here? Seeing similar results for larger networks would make a stronger case. I am open to increasing my score depending on the improvements the authors are able to introduce. * Note: I wanted to rate the paper as *`5: Borderline*, but for some reason can only choose 1, 3, 6 or 8. The current rank is 3, thought I think this paper deserves more. I am open to increasing the ranking if the above concerns are addressed. A few additional comments: 1) Betti numbers are only defined in Section 4. Yet, they are constantly mentioned in the abstract, introduction and all other preceding sections. After carefully reading the first 3.5 pages, there wasn*t even a heuristic description of Betti numbers, just that they measure topological complexity. A heuristic interpretation is provided at the top of page 5, but something of this flavor must appear much earlier. I suggest not mentioning Betti numbers in the abstract and presenting them properly in the introduction. Otherwise, it*s hard to appreciate anything being said about or in terms of these objects until one get to Section 4. This also makes the introduction read very hand-wavy, when the authors in fact do have precise ideas in mind. Should be easy to fix. 2) The authors keep saying they train the network to perfection. This sounds much like over-fitting. How would such a network generalize? Does the proposed perspective projects on generalization error somehow? This was a big missing link in the discussion for me. The current framework doesn*t seem to account for unseen examples (train and test sets are combined), but this is a crucial aspect of machine learning that shouldn*t be overlooked. 3) The authors mention they assume real data has shape. What does *real* refers to here? Do learning parity of binary matrices counts as real data? Would they expect even (or odd) parity matrices to be nicely clumped together like the classes in the synthetic experiments? 4) I suggest not rotating the axes in Fig. 2. The authors did so to provide a clearer view of the shapes, but it*s mostly confusing. The changes will be easier to see if the axes stay fixed. 5) Some of the observations seem strongly related to disentanglement or clustering of samples observed, e.g., in: - https://arxiv.org/pdf/1706.01350.pdf - https://arxiv.org/pdf/1810.05728.pdf The authors should do a further literature review and discuss possible connections to clustering. The latter is a more sensible thing to measure and could provide a way to reason about topological changes in larger networks.