Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a neural network to model data (or machine learning tasks) with fiber bundle structure. Machine learning tasks like regression or classification are often many to one. They assign a "label" (discrete or continuous) $Y \ni y = \pi(x)$ to each $x \in X$. Many different $x$ may receive the same label. It is interesting to explore / parameterize the set of all $x \in X$ that yield a given label $y$, that is to say, the set $\pi^{-1}(y)$. This is nicely modeled by the topological construction of a fiber bundle (a space that is locally a product space). Operationally, it corresponds to a covering of the label space $Y = \bigcup_{i} U_i$ by "patches" $U_i$ so that $\pi^{-1}(U_i)$  are homeomorphic to product spaces $U_i \times Z$, with $Z$ being the fiber space. 

The proposed network is in essence a conditional generative model with the added twist that the conditioning is also performed on the "patch" that a label (or a regression target) belongs to (for training; inference is only performed in the "reverse" direction of fiber sampling.)

The authors show interesting results in modeling familiar low-dimensional bundles as well as real-world datasets. Comparisons with conditional generative models suggest that the new architecture may be a better fiber bundle model. Finally, the authors numerically explore the role of "latent space" topology, a welcome addition.

main_review: ### Strengths of the paper

- I like the idea to model many-to-one tasks in machine learning using fiber bundles. I find this natural and elegant. The authors do a good job of motivating introducing and motivating the model.
- The proposed architecture is natural and more or less directly maps the fiber bundle technology to neural networks to learn fiber bundles from data.
- The author propose an effective training strategy to optimize the parameters of the proposed model which includes multiple well-motivated loss functions.
- Empirically the proposed architecture seems to outperform conditional generative models.
- I very much appreciate the discussion of the topological aspects and the exploration of the influence of topology of the latent space on performance.
- Overall the paper is well written and easy to follow. The illustrations help understand the technical points in the text.


### Weaknesses of the paper

- At a high level the fiber bundle idea (in a machine learning context) is very similar to that of conditional generative models. The authors argue that a specialized architecture is warranted and that networks like conditional flows cannot work well. These discussions are however at a rather heuristic level, with argument such as ... . To me it remains unclear why conditional networks underperform by such a significant margin.
- One claim the authors make is that the improved performance of their model (relative to conditional generators) comes from space partitioning, but partitioning can be interpreted as a particular choice of conditional architecture. In conditional generative models the only conditioning object is $y$; same seems to hold in BundleNet except that $y$ enters the architecture in two different ways, once through the argmin calculation of the nearest mean, and once on the "reverse" input on the right hand side. The question that comes to mind is then: why not use a slightly more advanced conditional generator architecture? My gut feeling is that it should perform well.
- I am uncertain about the significance of the "CGAN-nbhd" model in your comparisons. You state that this model is your best effort at modifying a standard architecture to sample from fibers, but its performance is _worse_ than that of its unmodified version. I am not sure about the message: that including fiber-bundle-specific details in a model makes its performance worse? Arguing from empirics, it means that the unmodified model is closer to the bundle ideal, no? I would suggest to remove this model from all comparisons and perhaps add a proper "in-between" model that actually does better than vanilla conditional generators. It is impossible to interpret why the performance of CGAN-nbhd is the way it is (it may have to do with things like hyperparameters, architecture details, training strategy, ...)
- Overall, I would like to see a much more thorough and convincing comparison between the various models and a better effort at training the existing conditional architectures. (Again, there are no convincing hypotheses, tested numerically, for why conditional generative models perform worse.)
- The current architecture seems to be suitable for very low-dimensional datasets; things like $k$-means are hard to motivate in higher dimension. Could you discuss examples and implications of applying your ideas to high-dimensional problems?
- I believe that considerable care is required in interpreting the empirical statements about the influence of topology. I am not convinced that these empirical results are broadly representative (they are likely to strongly depend on the chosen models and hyperparameters). What is more, many real-world data distributions are believed or known to be low-dimensional yet invertible flows model them without difficulty. The example shown by the authors might be an artifact of low dimension for two reasons: (topological) low-dimensional topology and embeddings are more quirky than high-dimensional and (expressivity) the flow networks used have limited expressivity when approximating functions $\mathbb{R}^2 \to \mathbb{R}^2$. They might work better in high dimension.
- A minor point: I would appreciate having the dataset descriptions in the main text. The main results of the paper are empirical, on Airfoil Noise and Wine Quality datasets—it is hard to build intuition without knowing the relevant details of the datasets. One would need to go to the appendix, but even there you refer to some external resources.

### Notes and questions
- Out of curiosity: your explore the case where the fiber bundle structure is essentially known. Are there applications where such a structure would have to be discovered? Could you adapt your network to these scenarios?
- Is there a connection between your model and the recently-proposed multichart flows? (I am thinking about the label space partitioning)
- How are the conditioning vectors $r_i$ structured? (are they simply scalars or they have the structure of labels $y$ or inputs $x$?)
- Since you're using an invertible network, it seems to me that $\text{dim}(y) + \text{dim}(z) = \text{dim}(x)$, but $x$ corresponds to points in the ambient space. Many datasets are believed to be intrinsically low-dimensional. Would it then make sense to use injective flows like https://arxiv.org/abs/2102.10461 or https://arxiv.org/abs/2003.13913 instead of invertible flows?

summary_of_the_review: This interesting paper suggests that fiber bundles are good models of many-to-one tasks in machine learning. The authors propose a deep neural network to model fiber bundles and argue that it outperforms conditional generative models. I find the premise of the paper natural and convincing, but the numerical results, evaluations of conditional generative models, and the empirical claims about the influence of topology lacking, hence my recommendation. Overall I think the paper has a lot of potential but the narrative and the comparisons should be much more solid, especially when it comes to existing conditional generative models and diagnosing the (topological) modes of failure. My initial recommendation is thus a rejection but I am more than willing to adjust it after the author response. (I would've chosen a 4 but it is not available anymore.)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The manuscript introduces Bundle Networks which are neural networks designed with an explicit fiber bundle in the architecture. With this in hand one has an explicit description of the fibers of the fibers associated with a machine learning task. The model is demonstrated on both artificial data, showing that it is capable of learning the fiber bundles where these can be explicitly computed, as well as real world data sets where the fiber bundles are of interest.

main_review: The manuscript is presented clearly and succinctly and is generally well written, and the technical contributions are novel as far as I can tell. I believe this work has potential in influencing future papers and could become important in the future. 

When the range is continuous it seems to me that by choosing the clusters to be large or small, one has a trade-off between the "true" fibers for very small clusters, and a regularised version for large ones. It would be interesting to see this discussed in a bit more depth. I would also like to see a little more elaboration on how the number of neighbourhoods for k-means affects the performance, as this seems to affect the overall architecture quite a bit. 

It would also add a lot if the task of learning the fiber structure was motivated more by e.g. an application where the fiber structure is helpful in analysing the problem.

My main point of contention is that the datasets considered are very low dimensional, and the manuscript mentions that the model trains very slowly in higher dimensions, but this isn't shown and I suspect that the huge improvement in performance over the baselines would drop off pretty fast as the dimensionality increases. It would be interesting if this could be expanded on (maybe in the appendix).

Overall I think this is a well written and stimulating paper with theoretically motivated technical innovation. The experimental results look impressive besides the lack of any higher dimensional results, but as this is a more theoretical paper, this omission does not hurt the manuscript too much.

summary_of_the_review: I think this is an important addition to the literature and has the possibility of being useful for future work about explainability and disentanglement.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a new architecture for generative modeling, called Bundle Networks. This is based on the mathematical concept of fiber bundles and allows to explore the networks fibers above the labels (their preimages). Training the network then corresponds to learning the local trivialization of the fiber bundle, i.e., of the labeling function. 
main_review: *Pro*

+ The paper is clearly written and easy to follow. The figures are of high quality.

+ The method outperforms the baselines at the studied generative modeling tasks.

+ The method is based on a standard construction in differential topology. As such the underlying concepts are mathematically well-founded. Bringing these ideas to machine learning is quite interesting on its own.

*Contra*

- The labelling function on real world data is typically not a fiber bundle $Z\to \operatorname{supp}(\mathcal{D_X}) \stackrel{\pi}{\to} Y$. In other words, the local trivialization the networks tries to learn do not exist.  
Admittedly, this limitation is discussed towards the end of the paper in section 6. However, from my point of view, this should be stated more prominently already in section 3.1, when introducing the general idea. Also, it's theoretical implication should have been explored to some extent.

  The empirical examination of this issue in section 6 is insufficient. The topological space which is studied therein, is locally homeomorphic to $\mathbb R \times \mathbb S^1$ except on a measure zero set (with respect to the uniform distribution). So it is unsurprising, that the algorithm is able to learn this space. Overall, the setting is too simple to transfer to real world data, whose topology usually is much more difficult.

  I also want to remark, that Figure 4 reveals problems with predicting the geometry (opposed to the topology) close to the singularity. Do the baselines (WGAN, CGAN) suffer from similar artifacts? Could you provide similar plots as Figure 4 for them?

- The method introduces new hyperparameters, in particular the fiber space $Z$, a probability distribution $\mathcal {D_Z}$ on the fiber and the number $q$ of local homeomorphisms (centroids for k-means). Analysis of the robustness with respect to these choices on real world data is missing.  

   Regarding $q$, according to the theory, there should not be an issue, if $q$ is sufficiently large. Is this indeed true? Does choosing $q$ too large lead to a noticable increase of computational cost.  

  Regarding $Z$ and $\mathcal{D_Z}$. Choosing these seems not obvious at all on non-synthetic data. Are there any heuristics? Is the prediction robust with respect to these choices? Are there any tradeoffs regading the fiber dimensionality?

- (minor) Figure 6 in Appendix A.4 is not sufficiently described. It is unclear, which points come from training data and which ones come from the reconstruction. Is only the bottom right reconstructed?
summary_of_the_review: The idea of the submission is quite interesting and I really like its presentation in the paper. However, the theoretical setting seems not to be applicable to real word data. Furthermore, a study on the robustness with respect to the additional hyperparameters is missing. I therefore give a score of 5.

----

**Update**

I thank the authors for addressing my concerns in the update. It seems that the good performance is rather independent of particular hyperparameter choices and prior distributions, but mostly from the guiding idea of allowing the model to be a *local* (as opposed to *global*) trivialization, as indicated by the experiments with q=1. Yet, a convincing argument is missing. In general, I still think, that there should have been a theoretical study under which conditions and  assumptions on the data Bundle Networks can be successful. 

I raise my score to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the Bundle Networks as a framework to learn many-to-one maps. The empirical performance is promising on two synthetic datasets and two real datasets. 
The main contribustions are three-fold: 
1) The problem of learning the fibers of a ML task is formalized.
2) An approach is proposed to learn the fibers.
3) A family of deep generative models called a Bundle Network is designed.
 
main_review: [Strengths] 
1) (A new problem) The formalization of a new machine learning problem, i.e., the problem of learning fibers of a ML task.  
2) (A new solution) An effective scheme is proposed to solve the fiber learning problem with encouraging results.


[Weakness]
1) (The motivation). The proposed fiber learning problem is interesting and novel to me. However, the rationality of modeling the many-to-one relationship via the strict "product structure" seems not sufficiently explained. It is suggested to give more vivid examples in CV/ML to show the model's suitability.
2) (Conditioning may weaken the expressive power). To avoid training a different model for each $U_i$ in $\mathcal{U}$, conditioning is used in the proposed BundleNet. However, using conditioning instead of optimizing may limit the model's expressive power. 
3) (Clarity of introduction to BundleNet). The description of BundleNet seems not sufficiently clear. I find it a bit difficult to understand the whole process. For example, "elements of $\mathcal{R}$ are in bijection with neighborhoods contained in $\mathcal{U}$ with the objective of having $\Phi_i$ play the role of $\psi_i^{-1}$". It is suggested to rewrite this section for better readability. 
4) (Typos). There are typos in this paper. For example, "Finally, each measurable subset $U\in Y$"  --> "$U \subset Y$".
 


 
summary_of_the_review: This paper proposes a new problem, i.e., the fiber learning problem of a ML task, as well as an effective solution to it. The proposal is novel and interesting to me. However, the motivation, the clearity, and the empirical support can be improved.
review_title: null

Meta-review: The paper studies the problem of learning fiber distributions associated with a machine learning task, in which the goal is to predict Y, given X. One chooses a fiber space / distribution Z / D_Z, and learns a trivialization \varphi : (Y,Z) -> X. The proposed architecture first clusters the label space Y. Within each cluster i it fixes a fiber space and distribution, and then learns a mapping \Phi_i, parameterized by an invertible neural network, by minimizing the discrepancy between the generated distribution and the distribution of the training data. The paper performs experiments on the wine dataset and a dataset coming from an aerospace application, as well as synthetic data with fiber bundle structure. Since the task here is generative modeling, the paper compares to standard GAN architectures (WGAN and conditional GAN) and argues that they are not fiber-learners, in the sense of this paper. 

Initial reviews were split, with reviewers appreciating the novelty of the fiber learning task, while also raising questions about the paper’s relationship to conditional GANs, some points of clarity, and limitations of the experiments. After interaction in the response period, the reviewers converged to a decision to accept. The paper’s primary strength is its clear formulation — the paper provides useful language for describing conditional generative models (in particular, for discussing when a factorization of the distribution over Y and Z is appropriate), a valuable contribution to the discussion in this area.
Decision: Accept (Poster)