Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes CoFrNet, a neural architecture that stacks neural layers in a way inspired by Continuous Fractions. The paper shows some interpretability advantage of the proposed model.
main_review: Pros:
1. There is some novelty as this is (to my best knowledge) the first work that combines Continuous Fractions with Neural Networks.
2. Theoretical development on the interpretability via Power Series and universal approximation seems sound.
3. The paper is in general clearly written.

Cons:
1. While the paper does show the interpretability advantages on synthetic functions and some of the datasets tested, discussion on other datasets ( such as Cifar-10) is missing. 
2. I also feel that the interpretability discussion on non-synthetic datasets is very limited. The authors only listed a few top features. I think a more thorough analysis, possibly including plots illustrating the feature attributions, is needed.
3. The performance results (in term of accuracy) are poor compared to SOTA models. While the proposed CoFRNet does have interpretability advantages, the poor performance will not allow it to be used in any real-world scenarios. I also doubt if CoFRNet can be applied to more difficult datasets (such as ImageNet) as the inverse activation will have serious gradient variance issues when you stack many more layers to build a deeper model.



limitations_and_societal_impact: Limitations are discussed adequately. I don't find any discussion on societal impact but I guess there is no need.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a new neural architecture inspired by continued fractions, which has some great theoretical properties in terms of function approximation. As a neural architecture, CoFrNets can learn function with much less number of parameters compared to fully connected neural networks. The authors also claims that CoFrNets is interpretable since it can represented as a polynomial.
main_review: The idea of considering continued fractions for neural networks is novel, however, the claim that the model is interpretable is not supported well. In (1), the authors discussed first-order attributions and high-order attributions. Arguably, if one wants to get a sparse linear/ polynomial function, learning linear models on polynomial expansions and kernel methods would be what comes to mind. I would argue that these two have at least the same explainability to CoFrNets, but they are not compared to.

Some theoretical discussion may also be added compared to polynomial functions. It is mentioned that the rational approximations formed by any of its finite truncations (termed convergents) are closer to the true value than any other rational number with the same or smaller denominator. However, this is not useful as a function approximator. If the Ground Truth is a high order polynomial of order k with d terms, a polynomial expansion would need to represent the function with (# features)^k input features with d non-zero term. Thus, the number of parameter here is  (# features)^k, and the actual parameter needed is d. In this case, can you prove the number of parameters needed to accurately estimate the polynomial function? It seems like you definitely need less than (# features)^k parameters (although I don't know if polynomials are a subset of continuous fractions). Some analysis will also be helpful to readers unfamiliar with continued fractions.

To raise the quality of the work, I would consider 1. convolution CoFrNets, where each ladder only receives the input dimensions that are structurally similar. 2. using CoFrNets in existing NN layers, so that it is comparable with SOTA models. If CoFrNets is only replacing the most memory-heavy component of some existing model, it might make the model more lightweight.

The model architecture kind of reminds me of transformer models, and by the self-attention mechanism, it would be much easier to construct polynomial functions for transformers. However, I have doubts that whether continuous fractions is better at learning functions that are not polynomial (see sin cos example in appendix). I feel like the authors did not do enough empirical studies to understand why CoFrNets is better or worse compared to MLP. Universal Approximation already exists for MLPs, and NNs are known to be notoriously over-parameterized (which is one of the reasons that it succeeds), and thus outperforming NNs with the same number of parameters is not really impressive. I would like to understand what is the function space where CoFrNets performs better, and what is the function space where CoFrNets performs better. For example, in image classification CNN will perform better than CoFrNet even if CoFrNet has the same number of parameters compared to CNN (at least for now). If for any function CoFrNets outperforms MLP with "any'' number of parameters (not just the parameters cherry picked for CoFrNets), I will be very surprised. I would like to see the performance of CoFrNet and MLP both varying the number of parameters (instead of only one set of parameters which is probably cherry picked for CoFrNets). Also, I would like to see more function classes being compared to. For example, you can set the ground truth to be some random forest regression output, and compare how well MLP and CoFrNet compare since most function spaces (estimated in practice) are not polynomials (at least not with sparse parameters). Discussing and understanding when CoFrNet performs worse than MLP (and also when it is better) is going to make the paper more valuable .

--------------------------------------------------------------------------------

After the authors showed results for imagenet I have faith for this method to become very impactful in the future. This is a very clear accept in my opinion. The interpretability claim is not needed to make this a strong work as the potential of the new model architecture is very high. More theories and applications can be developed along this type of research.


limitations_and_societal_impact: yes, but not enough, see main review
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose three variants of a novel neural architecture based on continued fractions, where the coefficients are learned linear combinations of features.
The authors claim global intepretability of this architecture through the additive nature of one variant and introduce two approaches exploiting the architecture's analytical form to compute first and higher order feature attributions.
They provide an original theoretical proof that a linear combination of continued fractions, i.e. their architecture, is a universal approximator.
Finally, the authors demonstrate the performance and interpretability of the architecture on a 2-dimensional toy experiment, as well as on multiple real-world data sets compared to other models.
main_review: ### Originality:
Continued Fractions are well-known, and to the best of my knowledge, this is the first time they were suggested for designing a neural architecture.
As far as I can tell, the interpretability the authors provide by its functional form has not been done like this before.
While I must admit that I had difficulties following the proof for universal approximation, it is original as far as I can tell.

### Quality:
The quality of this paper is quite high, and the work feels well polished.
This work motivates the introduced neural architecture well, and provides not only a wide range of experiments to justify its existence, but also proves its property as a universal approximator.
The authors discuss weaknesses of their architecture.

Since the authors put a strong emphasis on the interpretability aspect of their architecture, I find the demonstration thereof somewhat short-coming in the manuscript.
Given the strong emphasis, I would have liked to see a more quantitative analysis of the interpretability aspect, especially compared to the other globally interpretable models.


### Clarity:
I find the manuscript is well written and well organized.
Beyond technicality, the manuscript is relatively easy to follow.
Methods are introduced adequately.

### Significance:
The architecture is a new approach for globally interpretable models, based on the well-known continued fractions.
Although the manuscript is limited in its practical demonstration of the architecture's iterpretability,
I believe it provides a foundation for equally interesting research and demonstrates a new instance to prove universal approximation.


limitations_and_societal_impact: Some limitations are discussed with respect to with the architecture's practical applicability.
Societal impact is left unaddressed in the paper.
A very short remark in Section 7 to its (and other globally interpretable neural network architectures) impact may be beneficial.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper introduces CoFrNet, a new neural net architecture inspired by continued fractions. Several different training strategies are suggested, such as joint training, incrementally fit each layer, etc. Synthetic data and real data experiments are provided, and accuracy comparisons compared to several interpretable generalized additive models, trees, neural nets, etc. are provided
main_review: The ideas presented in this paper are interesting. I would like to clarify some questions about the evaluation: 
-	Are the numbers in Table 1 accuracies or AUCs? The caption says accuracies but just wanted to confirm. If some of these datasets are imbalanced, accuracies would not be a good measure. 
-	Why does MLP have such a low accuracy on Waveform (0.34), a lot lower than the other methods where the lowest accuracy was 0.69?
-	That the GAM and EBM results are exactly the same for the 3 tabular datasets (Waveform, Magic, Credit Card) seems a little unusual to me because the pygam package uses different types of spline base learners, if I’m not mistaken, while EBM uses a tree base learner, and previous work in the literature (e.g. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.433.8241&rep=rep1&type=pdf which first introduced EBMs) showed that the choice of base learner can matter a lot. 
-	Please provide standard deviations for Table 1 so that the reader can tell if the results are statistically significant.

Since interpretability is one of the goals here, it would be nice to provide some visualizations of the output that humans can interpret. It appears that CoFrNet-D can be visualized as an additive model (one feature at a time), while CoFrNet-F and CoFrNet-DL cannot. However CoFrNet-D seems to have worse performance than other additive models like EBM, NAM, etc.

Writing: 
- “Reference [2] is suited for tabular and image data, 112 whereas [52] is suitable for text data. Reference [19] is a …”  seems a little non-standard to say Reference [x], maybe use author names, e.g. name et al. …..?
limitations_and_societal_impact: I don't have any concerns here.
review_title: null

Meta-review: The reviewers have agreed on many positive aspects of the paper and the authors have done a great job in addressing the concerns. The paper is on a highly important topic and contains novel ideas. 

There were many important points in the reviewer responses that should be integrated into the final version of the paper. 
Decision: Accept (Poster)