Kati8/ICLR_2023_2025 huggingface dataset:
authors: Xinshun Liu
authors: YiZhi Fang
authors: Michael Jiang
reviews: 
Comments: This manuscript proposes to organize the neurons in a neural network to a graph-based structure along with an regularization method. While the experimental results demonstrate advancement, the motivation of reorganizing the neural networks from tree-like structure to flat graph structure requires further clarification. The contribution of this work is not convincing since existing works such as reservoir computing have also adopted such structure.
Strengths: 1. The rethinking and redesigning of the neural network structure is in urgent need and the idea of this work originates from real biological neurons.
2. The methods and contends in this paper are easy to follow.
3. Experimental results demonstrate advancement of the proposed framework.
Weaknesses: 1. The motivation of reorganizing the neural networks from tree-like structure to flat graph structure requires further clarification.
2. The literature review to existing works is not extensive, while the differences with similar works such as reservoir computing have not been explained.
3. The figures are not expressive.
4. While the parameter \gamma in definition 1 lies within the core of this work, its usage and selection are not detailedly described, which should be explicitly explained in equations or pseudocode.
5. The proposed NM regularization method requires theoretical verification. While the authors claimed that "a larger value of z would cause the edges connected to these subgraphs to tend toward zero", how to initialize z is not described.
6. The experimental results require detailed description to the experimental environment as in other referenced works.
7. The results in section 4.2-4.4 are quite foreseeable, which bring minor information to the readers.
8. The contents in appendix are redundant and also bring little information.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper proposes Neural Modules, a graph-based framework for organizing neural units within neural networks that aims to go beyond the traditional constrained structure of neural networks (NNs). Unlike the tree-like structure of traditional NNs which does not allow neurons to establish connections across the same layer, Neural Modules defines a graph-based structure that allows all neural units to communicate with all other units, allowing for more collaborative learning. Performance across three datasets from the UCI data repository are reported, showing that Neural Modules achieves lower error rates compared to other architectures and faster runtime compared to the DEQ model.
Strengths: 1. The discussion about traditional tree-like neural network tructure and the motivation behind redesigning the connectivity pattern behind neurons is well-written and clearly stated. A better organization of neurons could potentially allow for more efficient learning closer to that of real biological neural networks.
2. The methodology provides a formulation for the forward and backward pass of the Neural Modules architecture.
3. The authors show performance comparisons when varying the NM regularization parameter, and show a runtime comparison against one baseline model.
Weaknesses: 1. Performance of Neural Modules with several hundred nodes is reported on several datasets, however there are many other datasets with thousands of input features, requiring more nodes. The results would be stronger with benchmarking results on other datasets with more nodes in the graph, to demonstrate scalability to large graphs of neurons and more diverse datasets.
2. The running time reported for Neural Modules in Figure 3 lacks units, which makes it difficult to gauge what the relative improvement in runtime is against DEQ. Additionally, runtime is not compared to regular NNs, making it unclear if there are runtime disadvantages for Neural Modules versus traditional NNs.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper contends that current tree-like/hierarchical structures of neural networks are insufficiently expressive compared to examples of biological neural networks, proposes that networks of arbitrary directed graph structure (where "later" nodes can influence "earlier" ones) might get us access to levels of expressivity not possible with existing structures. However, they recognize that, because this model structure necessitates solving a system of equations in the forward pass, it scales poorly with the size of each system of equations being solved (which in this case is the size of the fully connected subcomponent being solved). Therefore, they formulate a regularizer (which they call NM regularization) which penalizes nodes for being connected to large subcomponents within the graph. This means that nodes will tend to have their edges with larger subgraphs decay to zero, which reduces the size of the subgraphs. They also implement threshold-based sparsification, only including equations in the system of equations to be solved if they have a weight > lambda to the subcomponent
Strengths: - The paper provides a seemingly practical way of scaling computation graphs without scaling the systems of equations to be solved to an impractical degree, while still staying within the paradigm of soft, gradient-based learning, and not needing to know the network structure ahead of time 
- The paper is well-presented and clear to read
Weaknesses: - The paper didn't provide any estimates for how computational cost would scale for more plausibly-sized models (which I think could have been done with some estimates of different regularization parameters and size k without necessarily having to run those experiments)
- The paper did not necessarily make an argument for whether certain _kinds_ of dataset benefit from this problem structure or require it to be solved well, which would make the argument for this computationally costly procedure more compelling. Instead it just tested the approach on some generic ML datasets.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper introduces the idea of replacing feed-forward neural networks with densely connected networks of neural-units that allow all-to-all communication. It also defines the idea of a "Neural Module", a locally connected subgraph is such all-to-all networks.
Strengths: This paper does a good job of introducing and motivating the idea of graph-based neural networks and Neural Modules. Presentation of results on multiple datasets helps gain confidence in the ability for this idea to positively impact current research. The work opens up new avenues of research in the fundamental design of neural networks.
Weaknesses: The paper is missing certain details about the method - such as what algorithm is used to solve the set of simultaneous equations in the forward process, and how neural-modules are identified and factorized.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 3