Unsupervised Dynamic Routing Via Competition Over Network Loss
Authors: Lei Zhao
Lin Wang
Jiuqiao Yu
Venue: NeurIPS 2021 Submitted (2021)
Keywords: unsupervised learning
deep learning
self organizing map
back propagation

Abstract:   This paper proposes a novel neural network architecture that can simultaneously do normal network optimizing while attaining the ability of unsupervised learning. Almost all existing unsupervised learning algorithms are based on doing calculations on the input space or feature space, this paper proposes a new possibility to discover a structure in the functional space without supervision. Using the self-organizing map over the competition of the loss of individual neural column, we route the input to the most appropriate modules dynamically, by doing this we separate the input functional space into different sub spaces which are represented by each individual neural column. At the end of the paper, we propose several possible architectures based on the philosophy of this paper that could build a neural network system block by block. 

Unsupervised Dynamic Routing Via Competition Over Network Loss
Anonymous Author(s)
Affiliation
Address
email
Abstract
This paper proposes novel neural network architecture that can simultaneously do normal network optimizing while attaining the ability of unsupervised learning. Almost all existing unsupervised learning algorithms are based on doing calcu- lations on the input space or feature space, this paper proposes new possibility to discover a structure in the functional space without supervision. Using the self self-organizing map over the competition of the loss of individual neural column, we route the input the most appropriate modules dynamically, by doing this we separate the input functional space into different sub spaces which represented by each individual neural column. At the end of the paper, propose several possible architectures based on the philosophy of this paper that could build a neural network system block by block.
5
6
10
11
Introduction
13
The unsupervised learning ability of the natural structured biological neural network is always fascinating scientists. The structure of some primitive neural networks is generated from evolution D. while other superior networks have base neural substrate that could adapt to the input neural signal and generate suitable structure automatically 2 Even computer simulation of this process gives the same result. By simulating the evolution process by genetic algorithm, scientist could evolve the same network structure for the insect's path integration(5). On the same subject, one could also use general purpose recurrent neural network (RNN) to optimize the path integration problem and still get the same network structure by interpreting the network (6 This method still works for more complicated problems, some scientists could replicate the neuron activation pattern for the grid cell in the hippocampus of the rat, which is used for rat's ''s navigation 10 Although different in optimizing algorithm and network structure.
14
15
16
19
20
21
22
23 and path integration RNN and hippocampus could get the same activation pattern eventually the subject of navigation which means that develop artificial network structure to mimic the biological plausible structure
26 is not impossible.
27 Information has its own structure and the structure is important. Convolutional neural network (CNN)
28 works best in the domain of image information [1] while natural language sequence information
29 works best under the structure of recurrent neural network (RNN) or deep self attention network (Transformer) [12] Image information has a structural match with the structure of CNN, even 31 without training the structure of CNN could give us relatively high accuracy in classification
32 task which is better than random guessing [33] So the hierarchical structure of image information
33 and the hierarchical network structure of CNN is having a resonance. On the other hand, natural
34 language sequence doesn't 't have such a strict information structure, the "pixels" " of the sequence has
Submitted to 35th Conference Neural Information Processing Systems (NeurIPS 2021). Do distribute.

---

correlation of each other some random distance away, so the self attention algorithm works best under 35
36 this scenario 12]. Can any neural network learn such information structure automatically that it can adapt to any kind of information universally? The biological neural network of vertebrate animals seems achieved this goal already by evolved a columnar structure 33, so we continue adopt this philosophy on the design of neural network architecture.
37
38
39
Contribution We proposed novel network architecture called functional self organizing map (ISOM) for making connection between network layers and also method to discover the connection target automatically by dynamic routing among the neural columns between layers by mutual competition between network columns.
41
42
43
44
2
Related Work
Capsule networks For the area of columnar networks that can do dynamic routing, method has been [14) Hinton first introduced the concept of capsules which is multi-layer columnar network which are trained to generated images with certain transformations. However in their method, if the feature size is large then the mutual connection matrix will get large quickly which will make the optimization slow. Actually if considering employing regularization to enhance the generalization of the network, most of the number the feature vector is zero with only a small portion of the vector non zero after activation function [3]. As seen in Figl in appendix we can partition the feature vector into columns and only transfer this non-zero column to the next layer. And also, their method is based on a local routing rule of mutual agreement which although to some extend have a biological plausibility, doesn't 't take into account the mutual competition of the columns involved so the network lacks of a global interpretability because the columns it generated don't 't have 56 a topological relationship.
47
48
49
50
51
54
55
Functional Self Organizing Map
57
se
3.1
Introduction to self organizing map
59
Here we give an introduction to the unsupervised learning algorithm of the self organizing map (SOM) which selected as the mutual competition algorithm of our method[[8] Each data from
60
61
our dataset is a vector X E R m of dimension m. The SOM composed of a group of vectors (also called the weight of the SOM) which have the same dimension m that is distributed on two dimensional map. The index of W is the position of the weight vectors seen in Fig [ The input vector is then compared with each of the vectors the map, the one with the minimum distance d from the input vector will be selected for update, which is called the best matching unit (BMU):
82
85
66
d,
Σ (X W
(I)
m
67
BMU Th arg (d.)
68
where d, is called distance function, BMU is the best matching unit vector. To update the weight, 69 we need to make the vector of the BMU closer to the input vector according to hebbian learning
70 rule We also need to update the vectors that are close to BMU, otherwise, the same BMU will always be selected no matter what is input. To do this, we need to define neighborhood function of
71
72 the BMU and assign different rates when updating the weight.
14 exep |r; -BMUUl
202
Where h2 is called the neighborhood function, F; E R and TBMU R are the position coordinates on the SOM, is the radius of the neighborhood function. The weights of the SOM will be updated according to the following rule.
W W
W + +h, W

---

73
This means we can scatter the input representation space map X" + W.r We'll show how
74 to regress some output space to some input space G(X = Y by scattering this functional space
75 topologically on a map (Xmr Y") = G Gm as we did on SOM.
X
A
Input Vector
B
Winning Node
W
W.th
Winning Node
Comparison
Weight Update
Neighborhood Function
Figure I: A) Input vector XTh is compared with every node vector on the SOM W the node with minimum distance d is selected for the best matching unit (BMU). B) Each node the SOM is updated following the input vector. node BMU and other node that close to the BMU have larger learning rate while other nodes have lower learning rate according to the neighborhood function.
78
3.2
Functional Self Organizing Map
The main idea of the SOM is competition. By competing over every other node, we get a winning node and update the node that position. Adopting the same idea, instead of using vectors, use neural network columns to compose the SOME Each column two layer encoder-decoder unit. For an input vector X each of the columns will give output vector as follows:
Σ E!
77 Where X is the input vector, E.TL the encoder matrix of column D is the decoder matrix and
78 Y is the output vector of dimension n. Note no activation function and weight regularization are
79 needed explained in Figl of appendix. Now we can define the loss function of each column be:
L =
T Y
71
BMU index arg min(L;)
where T is the target vector. Then the node with minimum loss is selected as the BMU and has the priority to fully back propagate the error. Other error back propagation is penalized by the value of the neighborhood function by limiting the loss value associated with that node, so the overall gradient needs to propagate back is:
V
L,
|r
T TBMUll 1
h; = exp(
20
where the parameters of all the columns. Note that not all columns can propagate gradient back, only those within some certain radius of the BMU have the opportunity as seen in Fig At this stage, there's mutual connection between columns, different columns will be optimized different directions although the loss function could be the same. To make columns more coherent with their surrounding, need to add a coherence update to the parameters of the columns. Now define the

---

optimized parameters of the network to be E and Dir, we can make coherence update like the following:
E Eq + ah, (E E
D l/ D + ah, D
X
Input Vector
E
Enooder Columns
Auxilliary Classitier
D
Decoder Colurns
Vectors
Loss
Losing
Losing
Losing
Losing
Wining
Losing
Gradient Propagation
Weight Update
Figure 2: Functional Self Organizing Map. please see section 3.2 for detail
80 where a is the coherence update rate. This means, we going to have topologically functional
81 space composed of each column, and also we dynamically routed the input to output by doing
82 winner-take-all competition. Note that during optimization, not all the columns are optimized by
85 back propagation, only those surrounding BMU are. So make inference on the network we need
84 record the chosen BMU location and train an auxiliary classifier as seen in Fig 2] then by using select the correct column to output. By doing this, the scope of the calculation only limited to the classifier and the selected column during inference.
this classifier
87
3.3 Multi-layered Functional Self Organizing Map
For multi-layered networks, need dynamic routing algorithm between layers. In other works [4 columns between different layers are fully connected which poses problem such that we need

---

every loss of the columns to find the BMU and do the optimization. A multi-layered network has combinatorial number of losses to calculate and the number of losses will grow exponentially large. Here we adopt simplification strategy by expanding the fully connected layers to a hierarchical cascade network. Fig shows a two layered functional SOM network. The output is calculated by columns in different layer as following:
Y?
Σ x
Ex
Emi F E
milk
Input Vector
Losing
E
Wining
Losing
Columns
F
Bottom Encoder Columns
D
Decoder Columns
Y!
Output Vectors
Loes
Losing
Losing
Losing
Losing
Wining
Losing
Lasing
Losing
Losing
Gradient Back Propagation
Weight Coherence Update
Figure Multi-layered Functional Self Organizing Map, please see section 3.3 for detail
Where the selected index the first layer. The calculations to find the distance function and BMU the same as Equation 12 However for there are two layers, we need two neighborhood functions:
1r: r
BMMU
by = exp( ||r r BMU 2
h2 = exp(
20
20
5

---

Coherence update rate Where 0102 are the neighborhood radius of each layer, ri :73 B. are the position coordinates of the BMU. To make gradient back propagation work as in the single layer case, we need to apply the neighborhood function to the loss. Different layers have different map sizes, so the neighborhood function each layer has different granular scale. In Fig 3 each top layer node connects to three bottom layer nodes so the neighborhood function needs triple the size match the bottom layer with the scale-up function upsize² (), the resulting neighborhood function the sum of the two:
BMU
88
89
90
91
92
|r: r BAU I1,
(exsize"(exp(( |r,
3 BMU 1
h exp(
(5)
2
20
where B controls the relative ratio of the top layer neighborhood function. And the gradient back propagation is the same:
V
L;
94 As in the single layer case, we still need coherence update all the column parameters:
E E + ahy(E (É ml Eyd
Fike = F.th + ah, (F,k (F
F(!)
D D oh,
D
After the optimization, we going have coarse functional map for the top layer and finer map belongs each top node for the bottom layer. We'll show the generated map in the next section.
4
Experiments
97
All the experiments are conducted GPU machine of 2080Ti. The code will be published in Github online repository!
100
4.1
One Layer Functional Self Organizing Map
101
We first test our method on the one-layer case explained in Section 3.2 The dataset we used the handwritten digit dataset mnist (20.. The training target T in Equation simply same as input
102
103
vector X which makes the optimization into an auto-encoder regression. All hyperparameters are summarized in the following table.
Table I: Single layer network hyper parameters
Phase
Map size
Radius
Adam learning rate
Start
16
2.0
le-4
1.0
End
16
0.5
1
0.1
106
We terminated the optimization after the decoder weight is stabilized and generated the figure of decoder weights for visualization. Please see Figure 4 for detail.
4.2
Two Layer Functional Self Organizing Map
109
Secondly. we test our multi-layer method on the same dataset of mnist. The training target is again an auto-encoder regression.
110
111
112 We generated the figure of the top encoder and decoder weights, the bottom encoder is visually interpretable, just show sample of it. We also generated the overall neighborhood function defined in Equation 5 We then test the BMU indices of the top layer against the true label to check
114
https://gi cwm/ http://h....ceeeteeeeeeoe//0

---

7
7
7
7
7
7
7
5
5
7
7
7
5
5
7
7
7
7
7
7
7
7
7
9
4
1
5
5
9
7
7
7
7
9
5
9
9
9
4
9
7
7
7
7
4 6
4 9
9
9
7
7
|
/
4 4 9
9
7
/
4 4 4 4
9
9
3
3
3
/////
6 4 4
9
9
8
3
3
////
/
6 9 4
4
9
5
3
3
3
8
6 9
7
5
5
3
3
8
2
9
4
6
6
6
9
66655550
3
8
8
2 2
5
2
6 6
9
6
3
3
3
8
5
5
8
2
2
6
6
0
3
6
2
1
2
7
2
2
2
8
8
C
S
5
5
3
3 3
8
8
8
2
8
8
3
3
3
8
(a) Decoder map with coherence update
(b) Decoder map without coherence update
Figure 4: Single layer experiments result. Selected encoder and decoder channel is tiled grid.
16 x 16
Table 2: Single layer network hyper parameters
Top map size
Bottom map size
Top radius
Bottom radius
Adam learning rate
Coherence update rate
Relative ratio
Phase
Start
16
1.0
2.0
le le-4
1.0
1.0
End
4
16
0.1
0.5
le-
0.1
0.1
115 the unsupervised classification ability. We can see that the decoder map generated is partitioned into
116 16 blocks which correspond to the size of the top layer map. Each block again having 16 columns
117 corresponding to each top layer column
Limitation As can in Fig 8 it's possible that some input handwritten numbers routed the wrong columns. This problem could be solved by adding more layers, increasing the size of the map or casting the network into CNN structure in future research.
4
4
7
7
7
4
9
4
9
7
4
4
4
4
7
7
7
4
4
7
9 4 4 4
2
2
8
8
8
9
9
4 9 4
2
2
8
9
9 4 4
2
2
2
8
9 4 1 444922221
8
6
6
6 0666622200060
6 9 0066000000000
6 9
6 6
5 5 3
3 5
(a) Decoder map with coherence update
(b) Overall neighborhood function
Figure 5: Two layer experiments result. We can see the decoder map is partitioned into 4 x 4 blocks.

---

77
1/ 7
7
8
9 8
8
4
2 3
6252
5253.
O 6
5
6 055
Figure 6: Selected top encoder map
Figure 7: Selected bottom encoder column
14
13
12
11
Cokin Indox Top Layes
1-
Figure 8: Unsupervised classification. Horizontal axis is true number label, vertical axis is the winning top layer column index. Circle sizes reflect the possibility which the input number is classified to.
Discussion
121
122
Bus structure and bandwidth saving The location of the BMU of each layer makes the classifi- cation while the vector of the BMU column passed to the next layer attains the details within this class which means that the information not belonging the current class has been got rid of. Unlike traditional deep neural networks whose feature vector dimension always gets bigger when the network get deeper, in our architecture, the feature vector will always gets smaller with the accompanying BMU classification.
123
126
127
128
The auxiliary classifier of each layer determines which column has the priority to output to the next layer which resembles bus structure in the computer system. This point of view gave us an inspiration that the methodology of computer system design could also apply to neural network system design least some extend.
129

---

132
Winner take all competition is too strict, we need a finer competition algorithm One limitation of our work is that in the method section all the competition strategies we used are winner take all (WTA) which means only one node selected as the BMU while all others lose. WTA is a special case of the mutually inhibitory neural algorithm. Normally after mutual inhibition, there will be multiple BMUs(21] which means we can build more complicated structure the future.
133
134
135
136
Neural columns can latch into each other just like the flip-flop circuits by mutual inhibition and lateral excitation 23 24 With neural columns as the functional unit, mutually latched columns by some certain trace can be recalled as functional group that optimized for some function domain while with another trace we can recall another functional group for another function domain. This means it's possible for us to develop a universal architecture for the human-like like neural network machine.
139
140
141
142
Conclusion
143
144
novel neural network architecture of functional self organizing map is presented in this paper. By introducing competition and coherence update to columnar network we can route different columns each other dynamically. We conducted experiments that our method inherited the self organizing ability of the self organizing map, especially on two layered case, we can generate a hierarchically topologically related map which have a unsupervised classificatton ability. This result suggest that we successfully mapped the functional space of the auto-encoder to topological space. As for future work, it is interesting to adopt this method to convolutional neural network or other fine grained model for better regression ability. Finally, we
145
handwritten digits dataset and verified
148
149
150
151
152
hope this novel architecture provide new substrate for neural network research which could bring advanced artificial intelligence.

---

References
154
[1] Stone et al. An Anatomically Constrained Model for Path Integration in the Bee Brain, Current
156
Biology (2017), http::t/t.dpxo/ doi. org/ 10. 1.w./006jj.coo cub. 2017. .0....8..0. .522
157
[2] Horton JC, Adams DL. The cortical column: a structure without a function. Philos Trans R Soe Lond B Biol Sci. 2005 Apr 29:360(145)):336622, doi: 10. 1098 PMID:
158
159
15937015; PMCID: PMC1569491.
[3] Maya-Vetencourt F Caleo M
Experience-Dependeal Plasticity in the Central Nervous System[M]. Springer Berlin Heidelberg, 2013.
162
[4] Elliott, K., Houston, & Fritzsch, B. Sensory afferent segregation in three-eared frogs resemble
163
the dominance columns observed three-eyed frogs. Sci Rep 5, 8338 (2015). https://do
184
10. 10w.01008/seeppe0
[5] Haferlach T, Wessnitzer J. Mangan M et al. Evolving Neural Model of Insect Path Integration[]]. Adaptive Behavior, 2007, 15(3).
186
167
[6] Cueva CJ, Wang Y, Chin M, et al. Emergence of functional and structural properties of the head direction system by optimization of recurrent neural networks[J]. 2019.
168
Kanitscheider 1, Fiete I. Training recurrent networks to generate hypotheses about how the brain solves hard navigation problems[J]. 2016.
171
[8] Hafting T, Fyhn M, Molden S et al. Microstructure of spatial map in the entorhinal cortex[J]. Nature, 2005, 436(7052)8801-806
172
173
[9] Cueva J Wei X Emergence of grid-like representations by training recurrent neural
networks perform spatial localization(J). 2018.
175
[10] Banino A, Barry C, Uria B et al. bector-based navigation using grid-like representations in artificial agents[J]. Nature, 2018.
178
177
[11] Simonyan K Zisserman A Very Deep Convolutional Networks for Large-Scale Image Recognition[]]. Computer Science, 2014.
178
[12] Vaswani A, Shazeer N, Parmar N al. Attention Is All You Need[J]. arXiv, 2017.
180
[13] D Ulyanov, Vedaldi A Lempitsky V Deep Image Prior[J]. IEEE, 2017.
181
[14] Sara Sabour, Nicholas Frosst, Geoffrey Hinton. Dynamic Routing Between Capsules. arXiv: 1710.09829, 2017
182
[15] Krogh A A Simple Weight Decay Can Improve Generalization(J], Advances in Neural Infor- mation Processing Systems, 1992, 4.
185
[16] Hinton G. How to represent whrt-whole hierarchies neural network[J]. 2021.
186
[17] Chen C CC, Yang L,, Cheng H Y Efficient and Robust Parallel DNN Training through Model
187
Parallelism Multi-GPU Platform. 2018.
[18] Kohonen T. The self-organizing map[J]. IEEE Proc Ienn, 1990, 1(1-3):1-6.
189
[19] Hebb, Donald O. The Organization of Behavior. 1949.
190
[20] Lecun Y, Cortes C. The mnist database of handwritten digits. 2010.
[21] Fldik P. Forming sparse representations by local anti- nati-Hebbian learning[J]. Biological Cybernet-
ics, 1990, 64(2).
10

---

193
[22] Olshausen B A Field D J. Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by VIJJ. Vision Research, 1997, 37(23):607-609.
194
[33]
Stern M, Bolding K, Abbott L, al. A Transformation From Latency To Ensemble Coding In A Model Of Piriform Cortex[J]. Elife, 2017, 7.
197
[24] Haberly Parallel-distributed Processing in Olfactory Cortex: New Insights from Morpho-
198
logical and Physiological Analysis of Neuronal Circuitry[J]. Chemical Senses, 2051(5):551-57.

---

Checklist
199
200
1. For all authors...
201
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
202
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see 4.2 5
204
(c) Did you discuss any potential negative societal impacts of your work? [No] This research is general research but not a applied algorithm research that could have any socictal impacts
205
208
207
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
208
209
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
211
(b) Did you include complete proofs of all theoretical results? [N/A]
212
3. If you ran experiments..
213
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] Please see
214
215
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see 4
217
218
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] Please see
219
220
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see M
221
222
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Please see 4.1
224
(b) Did you mention the license of the assets? [No] Cant' find licenses anywhere, even on the homepage of the dataset http:: //yp:////nnmm.mm lecun comcun.commeee www///ed///mm..//////////e//e/ee open to use
225
should be
227
(c) Did you include any new assets either in the supplemental material or as a URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [No] Dataset is open to use
230
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] No personally identifiable information in dataset
231
232
5. If you used crowdsourcing or conducted research with human subjects...
239
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
234
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
236
237
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
238
12


Reviews:
[{"review_id": "yWlez_t7Tez", "reviewer": "Reviewer_DBew", "summary": "This paper develops a 2D mapping method by extending the self-organizing map (SOM) to a multi-layer version. While the simple version of SOM can be regarded as a single-layer network, the proposed functional SOM (fSOM) adopts the multilayered encoder-decoder architecture. The training technique, called weight coherence update, is introduced to obtain consistent neighborhood structures. The proposed fSOM is applied to the 2D embedding problem using the MNIST dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes an extension of the self-organizing map (SOM). The research direction that develops a deep SOM architecture seems to be interesting.\n\nIn the equation in section 3.2, the operation in the encoder and decoder seems to be linear projection. There is no activation function, meaning that no non-linear mapping exists in the proposed architecture. Why don't the authors introduce non-linearity into the proposed architecture?\n\nMoreover, the experimental comparison to existing methods is not conducted. The reviewer cannot recognize the advantage of the proposed method against naive SOM and other unsupervised learning methods.\n\nThis paper has a lot of unclear points. For example, what is the contribution of an auxiliary classifier? How to determine the hyperparameter in the proposed method? How sensitive is the experimental result against the hyperparameter setting?\n\nThere exist several deep SOM methods (see the following literature). The authors should elaborate on the difference between the proposed fSOM and existing methods and the advantages of fSOM.\n\nN. Liu, J. Wang and Y. Gong, \"Deep Self-Organizing Map for visual classification,\" 2015 International Joint Conference on Neural Networks (IJCNN), 2015, pp. 1-6, doi: 10.1109/IJCNN.2015.7280357.\n\nC. S. Wickramasinghe, K. Amarasinghe, D. Marino and M. Manic, \"Deep Self-Organizing Maps for Visual Data Mining,\" 2018 11th International Conference on Human System Interaction (HSI), 2018, pp. 304-310, doi: 10.1109/HSI.2018.8430845.\n\nVincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, Gunnar Rätsch, \"SOM-VAE: Interpretable Discrete Representation Learning on Time Series,\" ICLR 2019\n\nFlorent Forest, Mustapha Lebbah, Hanene Azzag and Jérôme Lacaille (2019). Deep Embedded SOM: Joint Representation Learning and Self-Organization. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2019).\n\n---\n[Comment after rebuttal] The authors did not provide the responses. So, I keep my score.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not provide a comparison between the proposed method and the existing methods. It is not clear the advantages and disadvantages of the proposed method compared to other baselines. Therefore, the authors should conduct the experimental comparison to reveal the limitations of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "x_inkqcahW_", "reviewer": "Reviewer_6pPz", "summary": "The paper introduces an approach to function-level clustering. Function-level clustering means that a conditional computation model is optimized (in this case, parallel copies of a neural network), and that the optimization step is grouped by the respective computation path. This is achieved by using a  \"self organizing map\" that can compose  to select minimal loss computation paths. In more detail, it:\n1. computes the function through all of its existing computation paths\n2. selects the lowest-loss path\n2. optimizes that path and its neighbors only", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- the paper tackles an interesting idea and a research direction arguably fundamental for interpretable ML and biologically plausible networks\n- the idea of updating a neighborhood of computational units is novel and interesting, because it could lead to a functional continuum\n\nCons:\n- the paper does not address the similarities of this work and the rich literature on conditional computation. In particular the \"Routing\" paradigm (https://arxiv.org/pdf/1904.12774.pdf) is an extremely similar and relevant approach to combining computation paths. \"Dispatched Routing Networks\" (https://nlp.stanford.edu/projects/sci/dispatcher.pdf) even explicitly models routing and conditional computation as function-level clustering, and discusses several additional aspects not mentioned in the paper\n- the paper is poor on experiments; neither does it fully evaluate how the computation paths develop or how they relate, nor does it evaluate on multiple datasets\n\nOriginality:\nUnfortunately, the authors do not seem to be aware of the existing literature. Given the existing highly relevant (and richer) literature, however, originality of this work is low.\n\nQuality:\nThe quality of the submission is ok. There are some minor issues around language, but overall the paper is well written, well structured, and has well-thought out illustrations.\n\nClarity:\nThe clarity of the submission is ok. There are some minor issues when the authors move from single layer routing to multi-layer routing, and it's not immediately clear how one relates to the other.\n\nSignificance:\nThe area of study - modular neural architectures - is highly relevant for several subfields of ML and the brain sciences. Unfortunately, the limited scope of their experiments (and the low originality) strongly lowers the overall significance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "not applicable", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "3-KOc_JthFz", "reviewer": "Reviewer_H5F1", "summary": "The paper proposed a novel network architecture called functional self-organizing map (fSOM) for making connections between network layers and also a method to discover the connection target automatically by dynamic routing among the neural columns between layers by mutual competition between network columns. The authors presented the results of experiments on a handwritten digits dataset and verified that the method inherited the self-organizing ability of the self-organizing map, especially on the two-layered case, we can generate a hierarchically topologically related map that has an unsupervised classification ability.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposed a novel network architecture called functional self-organizing map (fSOM) for making connections between network layers and also a method to discover the connection target automatically by dynamic routing among the neural columns between layers by mutual competition between network columns. The authors presented the results of experiments on a handwritten digits dataset and verified that the method inherited the self-organizing ability of the self-organizing map, especially on the two-layered case, we can generate a hierarchically topologically related map that has an unsupervised classification ability.\n\nThe presented idea is quite original and the description is rather clear. However, the quality of the paper is not sufficient for NeurIPS. Since the experiments were conducted only on a relatively simple (to contemporary machine learning methods) dataset (mnist) and the method was not compared with any other approaches / algorithms, it is hard to assess its significance. Running experiments only on MNIST suggest that it is still relatively early-stage research and further investigation is required. It is possible that for more complex tasks and datasets this method will not be as successful.\n\nAlso, there is a link to a Github repository that was created but there is no code the verify or reproduce the results. The last commit was approximately 2 months ago, so it looks that the authors had sufficient time to submit the code used for experiments.\n\nAlso, there are many typos / language mistakes, e.g.,:\n- \"is always fascinating scientists\" -> \"always fascinates scientists\"\n- \"the \"pixels\" of the sequence has a correlation of each other\" -> \"the \"pixels\" of the sequence have a correlation of each other\"\n- \"the concept of capsules which is a multi-layer columnar network which are trained\"-> \"the concept of capsules which is a multi-layer columnar network which is trained\" (or \"the concept of capsules which are multi-layer columnar networks which are trained\")\n- \"for making connection between\" -> \"for making a connection between\"\n- \"if the feature size is large then the mutual connection matrix\" -> \"if the feature size is large, then the mutual connection matrix\"\n- \"is the same as in Equation 1 2\" -> \"is the same as in Equations 1 and 2.\"\n- \"map which have a unsupervised classification ability\" -> \"map which has an unsupervised classification ability\"\n\nAll in all, the idea seems to be interesting and novel but it requires further investigation and at the current stage, the article is not good enough to be published at NeurIPS. However, it might be appropriate for other conferences.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations are discussed. This research doesn't seem to have a societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers suggested reject and the authors did not provide any answers to all the questions raised. 
Decision: Reject