Graph Tree Neural Networks
Authors: Seokjun Kim
Jaeeun Jang
Heeseok Jung
Hyeoncheol Kim
Venue: ICLR 2022 Submitted (2022)
Keywords: Graph neural networks
Tree based convolutional neural network
Domain general learning

Abstract: In the field of deep learning, various architectures have been developed. However, most studies are limited to specific tasks or datasets due to their fixed layer structure. In this paper, we do not express the structure delivering information as a network model but as a data structure called a graph tree. And we propose two association models of graph tree neural networks(GTNNs) designed to solve the problems of existing networks by analyzing the structure of human neural networks. Defining the starting and ending points in a single graph is difficult, and a tree cannot express the relationship among sibling nodes. On the contrary, a graph tree(GT) can express leaf and root nodes as its starting and ending points and the relationship among sibling nodes. Instead of using fixed sequence layers, we create a GT for each data and train GTNN according to the tree's structure. GTNNs are data-driven learning in which the number of convolutions varies according to the depth of the tree. Moreover, these models can simultaneously learn various types of datasets through the recursive learning method. Depth-first convolution (DFC) encodes the interaction result from leaf nodes to the root node in a bottom-up approach, and depth-first deconvolution (DFD) decodes the interaction result from the root node to the leaf nodes in a top-down approach. To demonstrate the performance of these networks, we conducted two experiments. The first experiment is whether various datasets can be processed by combining GTNN and feature extraction networks(processing various datasets). The second experiment is about whether the output of GTNN can embed information on all data contained in the GT(association). We compared the performance of existing networks that separately learned image, sound, and natural language datasets with the performance simultaneously learned by connecting these networks. As a result, these models learned without significant performance degradation, and the output vector contained all the information in the GT. 

Under review as a conference paper at ICLR 2022
GRAPH TREE NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In the field of deep learning, various architectures have been developed. However, most studies are limited to specific tasks or datasets due to their fixed layer structure. In this paper, we do not express the structure delivering information as a network model but as a data structure called a graph tree. And we propose two association models of graph tree neural networks (TNNNs) designed to solve the problems of existing networks by analyzing the structure of human neural networks. Defining the starting and ending points in single graph is difficult, and tree cannot express the relationship among sibling nodes. On the contrary, graph tree(GT) can express leaf and root nodes as its starting and ending points and the relationship among sibling nodes. Instead of using fixed sequence layers. create GT for each data and train GTNN according to the tree's 's structure. GTNNs are data-driven learning in which the number of convolutions varies according the depth of the tree. Moreover, these models can simultaneously learn various types of datasets through the recursive learning method. Depth-first convolution (DFC) encodes the interaction result from leaf nodes to the root node in a bottom-up approach, and depth-first deconvolution (DFD) decodes the interaction result from the root node to the leaf nodes in a top-down approach. To demonstrate the performance of these networks, we conducted three experiments. In the first experiment, we verified whether it can be processed by combining graph tree neural networks and feature extraction networks(combining association models and feature extraction models). The second experiment is about whether the output of GTNN can embed information on all data contained in the (aasooccitioo In the third experiment, we simultancously learn the structure of images, sounds, graphs, and trees structure datasets (processing various datasets). We compared the performance of existing networks that separately learned image, sound, and natural language, relation datasets with the performance simultaneously learned by connecting these networks. As a result, these models learned without significant performance degradation, and the output vector contained all the information in the GT. And the importance of leasfin-learnigg and fin-tuning in these models explained.
INTRODUCTION
In deep learning, various architectures have been designed such as CNN, RNN, and LeCun al.) 1989 Hopfield 1982. Scarselli et al., 2008 Wu al. 2020). These networks show good performance in various fields such as image, text, and sound etc. However, most of the existing studies are focused on a specific dataset or task. And there are many tasks that humans can do but are difficult to perform with neural networks.
First, some of the difficulties with imitating human neural networks are introduced below. (C-i) Sensory organs that process information for each data type exist at several starting points. (C-ii) A tree cannot express the relationship between sibling nodes, and difficult to define the graph's 's starting point and ending point. (C-iii) The structure that receives information is hierarchical and freely integrated with various structures. But existing networks are designed to learn only specific tasks or datasets because layers are fixed. (C-iv) The number of activated neurons and the processing depths differ depending on the data type and complexity. And sometimes the information is entered and sometimes not. (C-v) Human neurons are very numerous and process various types of data.
C-I-] denotes the characteristics. (C-i) Humans have various sensory organs from which they receive information such as sight, hearing, and smell, and transfer it to the cerebral cortex using different

---

Under review as a conference paper at ICLR 2022
information unformation-processingg organs.
The optic nerve also begins with the visual cortex of the occipital
Prontal lobe
Parictal kobe
Terpporal lobs
Oecipiaal
(a) Human brain
(b) Graph Tree
Figure 1: Comparison of the human brain and graph tree neural network
lobe, and the auditory nerve begins with the auditory cortex of the temporal lobe; this means that the two paths have different starting points. (C-ii) A graph can describe the relationship depending on whether all nodes are connected, but it is not easy to define the start and end points. A tree is a data structure that can define direction with the leaf and root nodes. On the other hand, a tree structure cannot express the relationship between sibling nodes. (C-ii)) Information from different sensory organs is gathered in the association area. Typically, the posterior parietal cortex Malach al. 1995) is located at the top of the head as part of the parietal lobe, and sensory information such as vision and hearing is fused and interpreted; Then, information is coordinated in the frontal lobe and humans act. (C-iv) There is also an over-fitting problem DNN ANN(Widrow et al.) 1960), and various attempts have been made to solve this in (Srivastava et al., 2014). In DNN, simply deeper layers lead to an over- over-fitting problem{Dai et al. 2017). In particular, NasNet(Zoph et al., 2018) has attempted to design a layer automatically through reinforcement learning and RNN. This means that there is an appropriate network depth depending on the complexity of the dataset; we want to design network that could adjust the depth of the layers according the complexity of each data, not the data set. (C-v) Also, the number of human neurons is about 85 billion(Hereuland) Houzel 2009), which is difficult to express in network. Therefore, GTNNs designed to solve these problems.
A-[+] denotes our approach of C-I-). (A-i) This network can perform information processing, such as that done by sensory organs before their signals are entered into the cerebral cortex. In this paper, we express this sensory organ as a fcature-extraction process(sec2..)) Type information is stored with the data and converted into an input vector by different feature extraction processes for each type. (A-ii) We propose a new data structure called Graph Tree Node(GTNN and Graph Tree(GT). graph tree can be expressed using leaf and root nodes as the starting and ending points, as well as the relationship among sibling nodes. In addition, we can modify the existing tree dataset at no high cost. (A-iii) subtrees of different types are merged and make a final decision at the root node; this structure is difficult to express with the existing forward-learnin
method. Therefore, we used a recursive-lcariing (ooler & Kuchler, 1996). This methodology has been effective in analyzing the semantics of programming source code natural language(Socher et al. 2013 Mou et al. 2014). We call the secursive-convllutee convolution (DFC methodology(see) 2.3 DFC is a convolution method in which subtrees originating from different Leaf nodes are integrated into a bottom-up approach and represent hierarchical and relational information. GTNNs can simultaneously learn various types of datasets through the recursive learning method because the graph tree data strueture can express various model structures. (A-iv) GTNNs are data-driven learning in which the number of convolutions varies according the depth of the tree. Instead of using fixed sequence layers, we create a graph tree for each data and learn according to the tree's structure. If we modify this network further, we can adjust the depth according to the type and complexity of each data. In addition, We introduce two models of GTNNs called graph tree convolutional networks (GTC) and graph tree recursive (eTR) and these models are mathematically related to MLP(Hinton al.) 2006) and RNN. The Fully Connected layer(FC layer) in MLP as Level Layer of GTC, and Time in RNN as depth of GTR on a special case. (A-v) The number of human neurons is enormous, and it is difficult to express all of their structures in a network; on the other hand, this network can represent multiple neurons because it can perform recursive end-to-end learning according to the amount of information.
methodology used in GTNNs the depth-first
2

---

Under review as a conference paper at ICLR 2022
This network theory holds that "units of information have relationship in the form of graph, then become a bigger unit of information, and have a relationship with other units of information. At this point, the unit of information is set of neurons, and we express it as a vector with GTNN."
To demonstrate the performance of this network, we conducted three experiments. And we used several benchmark datasets (mage(MNISTT(LCCC 2018), text(IMDB (Mass et al. 2011),SS1 Socher et al. 2013)), graph( et al. 2021))
al. 1998)), sound(Speech Commands(Wardes
In the first experiment, It verifies whether feature extraction networks and association networks can be learned together. we compared the performance of existing networks that separately learned (enge((eeNet-5(LeCC performance simultaneously learned by connecting these networks for feature extraction into GTNN.
al. 1989), ,, sound(M5
al.) 2017)), text(CNN(Kim)) 2014)) with the
The second experiment contained one or three types of data into GT, and GTNNs learned these GTs. Then, we verified whether the output contained all the information inside the GT. Then, we conducted a verification experiment on whether the network's output contained all information on the tree.
In the third experiment, we verifies whether data of various cnnecturesimagee sound, tree, graph) can be learned and checked the plot the learning process in appendix E As a result, the network was learned without significant degradation in performance compared to when learning existing networks separately, and all information on the tree could be embedded as a vector.
GRAPH TREE NEURAL NETWORKS
In this section, we proposes graph architecture. The architecture consists of three parts
(i) Defining the data structure of graph tree node(GTN) and designing the graph tree structure (GT). (ii) Defining the feature-extraction((E)) process. (iii) Defining the GTNN model.
(xx r)
CNN
X,T
RNN
Bix
GCN
(a) Graph Tree Node
(b) Feature Extraction Networks (c) Graph Tree Neural Network
Figure 2: Graph Tree Architecture
2.1 GRAPH TREE DATA STRUCTURE
The graph tree (GT) is a data structure that can express relational and hierarchical information.
(Input) we can store the data such as images, sound, text, or tabular to each node. T: (Type) 1 This refers to the type of information matching the input æ. the feature-extraction function(y)) is selected by - like Eqn
A.: (Children Adjacency Matrix) - This refers to the relationship information that exists in GTN. the number of children is N and we can express as Ae E IR NN
C: (Children) This refers the child nodes of GTN matching to the node of A,, If there is no matching child, replaced with GTN,, which carries the initial hidden state 0' instead. We can express this C {GTN1, GTN,, ....GTNNN] ).
We can express GTN A,, C). GTN, E GT. GTN root denotes the root node of GT. The reason for defining the relationship among child nodes is the convenience of implementation. we define GTN the above way, we can convert the tree dataset into the GT dataset without significantly modifying the structure that has previously been useful.
3

---

Under review as a conference paper at ICLR 2022
2.2 FEATURE EXTRACTION NETWORKS & TYPE BIAS
= ( (2)) onehol(T))
(I)
The ariiput) and r(type) exist together GTN, and the feature extraction process of is selected by Eqn [ We expressed the function of feature extraction as V, and this function converts to X. And the one-hot vector has effect of having different bias value for each type(r). The weight parameter corresponding to the type onc-hot vector means the bias value for the corresponding type, and the activation threshold value is adjusted for each sype-bias. can represent data type information and that it is transmitted from which nerve. In addition, we do not need to perform adding operations(+) for bias. Therefore, the empty space of GT can be used as a zero-vector when batch learning is performed. And the dictionary structure is very useful for batch learning; we attached the methodology to appendix A This methodology divides mini-batch samples by type, and the lype-mini-batch size varies each time. Therefore, we recommend using a weight standardization Qiao et al. 2019) or group normalization Wu & He 2018) to avoid batch normalization(Loft & Szegedy affected by the batch size. We described the details of the network used in appendix
2013
2.3
DEPTI-FFRRST CONVOLUTION & DECONVOLUTION
Depth Firt Convolsion Foel (ratr
Depth First Decomvdlation
Pro Pre-orddr Pre-ordee(RiggttFni
Figure 3: Depth First Convolution & Deconvolution
This section introduces two propagation methods of depth-firs firth-firstto convolution (DFC) (a convolution method for traversing all nodes of GT) and depth-first deconvolution (DFD). Depth-first search (DFS) search algorithm for the tree data structure. Unlike general recursive convolution, DFC and DFD) are methodologies for learning children's 's relationships, feature extraction networks together.
Algorithm Depth First Convolution
1: function DFC p. (v)
10:
if len( Ien(C is then
b leaf node
2:
Tp: App Cp GTN,.iieme ti & V/ (Tpp((pp)
11:
GTN pmrre (+i))
3:
12:
return end if
T p. 0]
4:
Children MiidrenTist..
13:
5:
for q L...N, do
left-first
14:
[x h + Stack(Childreniis)
6:
pqo & DFC 1) Challdreniislappenddi pq: N pqD)
15:
h hp 22 t GTNN x,, h/, p',)))
7:
16:
GTN p.more (si1, +d2)
8: end for
17: return ( hpl
9:
18: end function
Post-order Depth-first convolution (Left-First) If DFS is post-order, iteration starts from the leaf node. Likewise, DFC a recursive convolution that propagates from the deepest nodes. GTNN Utilizes the hidden vector expressing the current depth and state.
First, we need to bring four items(1p, Tp: Acps Cp) of the current GTN,, the p means the convolution order, and up is embedded by the feature extraction network(V) according to the tурe(тp) and becomes the feature vector( d" p.).
Graph tree network models perform the convolution by receiving information 7t pay h pa) from the current node's children and the relationship( (Ae)) among children's information. We can use this process like learning the relationship between siblings nodes. the h means the convolution output of the child nodes. finally. 3 and 7 are delivered to the parent node. If the child node does not exist( C,), the current node means a leaf node. Therefore, the feature vector( 7 p) and initial hidden state( 0 transmitted to the parent node.
4

---

Under review as a conference paper at ICLR 2022
The important thing is that the root node only receives information from their children and does not the convolution with the root feature vector. Therefore, like algo B convolution is performed once more the output of the DFC. If we repeat this process, we finally get h root- Additionally, we can store information(#1g) *12) generated during the DFC process in GTNN more and use during the DFD process (ex. If the aggregate function is maxpool, we the indices information during deconvolution).
Algorithm 2 Depth First Deconvolution
function DFD hip GTN ps +lv)
9:
GTN
P.N...A. A. t Rep
2: Tp: ops Cp GTN p.doow dccooviteems(
10:
for Np... do
right-first
3: #11, *ig &- -TN,,more
11:
F 2pp My x kp
4: GTN p...Pp Ppp t v (Tplppp #h1)
12:
DFD(1 pq: pg:
5: if len( Cp) is 0 then
> leaf node
13:
end for return
6: return
14:
7: end if
15:
end function
8:
App X 1 - GTNN GGTNN Ass hp. bu *i2)
Pre-order Depth-first Deconvolution (Right-First) DFD a decoding methodology GTNNs. The DFC propagates from the leaf node to the root node, whereas the DFD propagates from the root node to the leaf node in the order of pre-order depth first to decode this.
First, we need to bring three items(Tp. Aep: Gp) of the current GTN p- Tp is decoded by the feature decoder network(V-)) according the type(T,) and the decoded output is stored in the GTNp..p.
the child does not exist, this means a leaf node and performs return. Otherwise, The hidden state(hp) is restored to X,, hi through GTNN- and hí transferred to child nodes. Also, DFC proceeds from left to right as left first (I....), DFD proceeds from right left as right first (N...1).
Please see Fig
Algorithm 3 Propagate (for AutoEncoder Models)
function PROPAGATE(GT. root)
4: Frogt N Broot +- GTNN root.0,+12)
2: FFroots N6' raot]= +- DFC(GTN raot, 1)
5: DFD(I' root, h root, GTNNood 1)
3:
h root: #h2 t GTNN(L, root, r0ot.0)
6: return h root, GTN root
7: end function
Like the propagate function (algo 3), we can freely encode and decode through DFC and DFD. We want to show the relationship between DFC and DFD and the possibility expanding to the widely used autoencoder(Hinton & Salakhutdinov 2006) models. In addition, Implementing this recursive function as a loop(while will speed up.
2.4 ASSOCIATION NEURAL NETWORK MODELS GTR & GTC SERIES
This section proposes GTR and GTC series that perform inductive learning in GTNNs. We describe the convolution performed in one graph tree node by DFC. These networks can be used as an association cell or layer, and we explained the difference between the two models appendix B
These association models do not use any fixed architecture and integrate information according the GT structure. And by fusing graph and tree structures, we can learn more diverse data structures with one cell and various architectures can be expressed by data(please these models are freer to perform various tasks and learn a lot of information using fewer parameters.
the appendix D). Accordingly,
These association models are as follows: The feature extraction models create the information from data; the "smallest unit" of information in the GT. The parent GTN receives information from their children and perform the convolution operation with the relation term; the convolution output is "bigger unit" of information. Finally, root GTN has GT information and perform the convolution operation; the output is "the biggest unit" of information.
5

---

Under review as a conference paper at ICLR 2022
2.4.1 ASSOCIATION CELL: GRAPH TREE RECURSIVE NEURAL NETWORK
We will describe a GTR cell that trains the GT described previously in See 2.1 GTR model can be expressed as GTR = [W, V, ". W E R (FF*x(F+++F") F is the node-feature size, B is the type-hias size, and F' is the output and hidden size, which is the information received from the child node, y denotes the feature-extractionnEe function, and a denotes the activation function.
D process described above, denotes the aggregate
オ, [,,, (xp)) onehot (Tp)] V (Zp. Tp), Z E ERR F+B
- g(o A Drp (W (I pq. T pm ))))
(3)
The propagation for GTNN is proceeding by the DFC traversing from leaf nodes to a root node (sec 2.3) is the order of DFC, is the child node number, Np is the number of children of GTN,, Ip is the node-input GTN,
The v extracts the feature vector( d'p) from Zp of GTN, through the feature extraction pro- cess(Eqn 2 considering the type information(r,)) The first location to start is leaf nodes by DFC. In the Leaf node, the initial hidden state is a zero vector( 0 ), and this is concatenated with Z to become I Ti E F++++F through the concatenation process as [.] and then delivered to the parent node. a child node exists, the current node receives information from a child node as [2'pg pg ol
pq> N pg] matching the graph nodes(A.p) existing in the p-th node are stacked as much Np. which is 11 pq. h pq
pq: n pg) means that (7]
T E (RNN>x(++++F))
We applied the GCN methodology (Kipf & Welling 2016), which useful in the GNN field. Therefore we expressed as Acp Acp I, where is the Identity matrix. If we express the connection value as just the more connected nodes becames the scale value larger than others. Therefore, D Acp D
applied using the order matrix Dep of Acp as a method of normalizing the relationship matrix in Eqn. 3 used F as 128, B (image, sound, language), and F' as 128, and a ReLU & Hinton 2010), and g was the readout of Max. If there no sibling node of the current node, GTR mathematically identical the RNN IW J. RNN can be special case of GTR, and the depth of the tree means times the RNN.
2.4.2 ASSOCIATION LAYERS: GRAPI TREE CONVOLUTIONAL NEURAL NETWORK
GTC is composed of GTC ((.W....W. V g. r). There is W E x BB in charge of each level, and the network in charge of each level is called the level layer. The input size of the level is Fi.. its output size is FV.- and the output size of the child is Fl Therefore, it is possible to adjust the input, hidden size; and can learn the depth-limited tree, m means Maximum-Deptte We set Fie and Fi, to 128 for all Ivs in the same way as GTR in the experiment. If the input size of Fi is and there is no sibling node of the current node, this network is mathematically identical to the FC layer as IW,, 2 and the MLP (IWo o.. (IW fy-1 (IW,. to Z²))),. I is the Identity matrix. The mathematical expression of GTC is similar to Eqn 3 The depth of the tree indicates the number of layers of the MLP. Therefore, the MLP and FC layer be special case of GTC.
2.5 ATTENTION MODELS
There is an attention process in the human brain. Therefore, GATs al. 2017), model that has been useful recently, was combined.
2.5.1 GRAPH TREE RECURSIVE ATTENTION NETWORKS
We introduce GTRAs that learn the importance through attention by slightly modifying the expression of the GTR. is composed GTRAs = {W, V, g, a, Go, d' that added (o.. d in GTR. A parameter for attention mechanism is added 2R2 x 2 + R) and the attention's 's activation function used LeakyReL.U(XX al. 2015) in the same way GATs. Nog is set of nodes connected

---

Under review as a conference paper at ICLR 2022
to the q-th child's node in the Acp of GTN and we express this
exp(LeakyRe
pay Th pq
I W pry h pri (DD)
Opper
(4)
EkEND, exp(LeakyReLU (a" pq h W phs h p&l
With the attention methodology introduced GATs Veličkov et al. 2017). it learns how the r-th node is of importance to the q-th node. This information is replaced with the part to which the adjacency matrix is connected.
If Acp does not exist in the GTN, we utilized to select critical features as
Sigmoid a [W]]
H ml
prs W
H ml I'D
if q + if =
Scor
(5)
exp(scorepgr)
Oppqr
Nyg
6
k=l exp(scorépgh)
The self-node set and the relationship values of the node are replaced with a real value of 0 ~ 1 using Sigmoid. Therefore, train the attention learning without Acp. It means to learn relationships based on node- Other useful models is self- adlfaadaptive model et al. 2019).
Therefore, the GTRAs are follows:
h = =lo(((AA G aa)) )))
In Eqn indicates point-wise and x', p is stack of 3' || 2 pa and it x', E (R2x)(FBB)) h', p is stack of W pq as o pa To further stabilize the self-attention process, we introduce multi-head attention mechanism:
and h', E RNN, Therefore [x',, p. h RNxx(P++BP))
h g(1 I(lk o a (W* (W (x',, (,,"'))))
where K is the number of multi-heads. The results from multiple heads are concatenated and delivered to parent node. set to and set output dim for each head 16(=P),. K). therefore we set Fy to 128. This process becomes a cell and delivers the result from the leaf node to the root node.
2.5.2 GRAPH TREE ATTENTION NETWORKS
The GTAs model is modified from GTC, and GATs' attention mechanism was applied. this model can be expressed as GTASs ((Wo... .....,,,,,,, Wn).
(d.... V, g, a, J.), and (0a, tv are added in GTC( TR ). Unlike GTRAs, GTAs can design different sizes of F, F' to match the Iv. this model is designed to select critical features, and it is similar to Sec Sec
Supervised learning Supervised learning uses datasets with labels. In this network, the h root was mapped to the dimension of the class using fully connected layer( F" to class count) and we calculated the log softmax and negative log likelihood loss for supervised learning as:
fog_soottaar(TTro Task root))
(9)
loss negative looeiiieoo
(10)
EXPERIMENTAL RESULTS
This section introduces the contents the experiment and the GT dataset consisting of the image, sound, and natural language. How to design GT is introduced in appendix [
Datasets Our goal is to express the information delivery process of existing models as a graph tree of data. The first experiment is whether various datasets can be simultaneously learned by combining GTNN and feature extraction networks, the GT we used in the experiment is illustrated by Fa). Layer node means that it has input value, only performs convolution. If there is sibling node, GTC performs the same operation as one FC layer. Sound and natural language GT have two nodes, and image GT has three nodes because It was configured similarly to figure (likes this fi The
7

---

Under review as a conference paper at ICLR 2022
Layer
fnyer
Layer
layer
Layer
Layer
sound
Layer
lang
sownd
ftayer
lang rand
pounnd rand
layer
lang rand
foound gand
Ltayer
lang
image rand
simage gand
Image
image
(a) Expt 1,2: One type
(b) Expt 2: All sound
(c) Expt 2: to image (d) Expt All language
Figure 4: Graph tree datasets of the experiments
important point is not limited to any specific architecture; later, this GT could be modified according the task or complexity.
The second experiment about whether the output( It root) of GTNN is embedded information on all data contained in the Ga(association). Therefore, we performed image, sound, and text classification by putting image, sound, text data in one GT, and we also learned the GT used in the first experiment simultaneously. One problem that when the three types of data are combined, The number of combinations of the dataset becomes too huge. For example, there are 50,000 images, 105,000 sounds, and 17,500 natural language samples, and when these data are combined, we need to generate 50,000 x x 17, 17,500 GT samples. For this problem, we used sampling method. When loading each data, two data of different types are sampled randomly. Thus, when we load one data, two GTs are generated of the one type GT and the three type GT. The test accuracy is the result of averaging 5 5-fold. the GT dataset of the second experiment is illustrated by Fig/4a,bbccd) the third experiment, we added the graph and tree structure dataset, This is described in Appendix
3.1 THE EXPERIMENT
INDIVIDUAL & SIMULTANEOUS (PROCESSING VARIOUS DATASETS)
Table 1: the result of Experiment 1
Task layers
17< class
Transfer leaming
Fine tuning
Model
MNIST
SC
IMDB
MNIST
IMDB
MNIST
SC
IMDB
MNIST
SC'
IMDB
LeNet-5
98.16
98.46
98.49
98.49
(Croup Nomn)
97.63
100.0
CNN
GTC
98.84
97.14
86, 10
98.55
96.87
85.88
98.
98.81
82.27
98.87
99.07
86 99
GTAs
98.96
96.91
85.38
98.77
96.85
85.76
98.65
98.91
82.20
98.64
97.39
86.62
CTRAs
98.79
96.66
36,28
98.78
96.68
85.79 77
98.47
98.64
82.18
98.92
98.89
87.27
Fpochs
30
30
95
We trained without using pre-trained model to check whether various feature extraction models can be learned simultancously. Only word embedding network (Pennington al. 2014) was used as pretrained model. Existing networks learn only about specific datasets. Therefore, LeNet-5(LeCum et 1998), M5 Dai et al. al.] 2017), and CNN 2014) were selected for feature extraction networks of the image (MNiee((LeLLCCC et al. 1998)), SSpeeh Command
2018)), and natural IIMD ( Maas et al., 201 I)). We described more details in appendix C The reason for
choosing these networks is based Convolutional Neural Networks specialized extract features.
Since the number of classes is different for each data, the test was divided into two cases. (Task Layers): The last layer placed differently for each dataset and mapped to the class dimension of the dataset. (47 class): The one last layer was used for all datasets, and the number of class dimensions equal the sum of class dimensions of all datasets. Because if the last layer is placed differently for each dataset, information on the input type may be informed. All models were learned during 30 epochs, and the results
Table Task layers, 47 class), and the learning process is (Fig
The learning processes of GTNNs are similar to the process of individually learning feature extraction networks; if the model learns like this, several problems found. LeNet-5 is well trained, but M5
8

---

Under review as a conference paper at ICLR 2022
103
66.
(4,
20 sncam
(a) Simultaneous learnnnn (47class)
(b) Transfer-learing
(c) Fine- Funinn
Figure 5: Test Accuracy (The Experiment 1)
needs to be more trained, and CNN has an overfitting problem. The reason is that each network has different epochs to have optimal performance, and setting learning parameters is more complex than learning individually.
transfer learning & fine-tuning Therefore, after learning individually, combined the association model to perform mensfer-learingg and fine-tuning (Pan & Yang 2009). The result is Table Transfer- learning, Fine-tuning). In transfer learning, the parameters of feature extraction networks are not modified, and in fine-tuning, the parameters are modified. We used the validation set to learn each feature extraction network and adopt the lowest value of the validation loss to use the model. As a result, the LeNet-5, M5, CNN were pre- pre-trained with 26, 95, 3 epochs. These models were combined with GTNN to perform fine-tuning and transfer learning. Then, the network was re-learned, and learning was stopped at epochs with the lowest value of validation loss using each of the same validation datasets. As in the results of [able[] Traasrrrr-eaaanann case of transfer learning, performance was poor in the IMDB dataset. We thought was overfitting because the IMDB dataset was relatively small. We reduced the learning rate from 0.001 to 0.0001, but the results were similar. While fine-tuning was similar to the performance learned individually in all datasets. And the performance improved compared to when fine-tuning was not performed. Consequently, If transfer learning and fine tuning are used by setting parameters well according to the network, the performance of the network can be improved. In Experiment 3, we learn datasets of various structures(Image. sound, tree, graph) through GTNN. It is described in Appendix E
Fine-tuning) and Fige) in the
3.2
THE EXPERIMENT 2 : CAN ALL DATA INFORMATION BE CONTAINED (ASSOCIATION)
Table 2: the result of association test
Task layer (%)
Lask + fine tuning (%)
Model
MNIST
MINISRAAI
SC&AAl
IMDB
IMDBAATI
MNIST
MNIST&AI
SC&ATI
IMDB
IMDB&AII
98.88
96.83
95.71
N7.20
96.63
98.83
86.76
86.61
GTAs
98.87
98.85
97.30
96.37
86.73
98.93
98.87
98.52
87.43
37.30
GIK
99.05
99.04
96.46
97.05
87.16
87.21
98.97
98.84
98.83
98.58
87.00
87.14
GTRAs
98.95
98.96
97.36
95.76
86.55
87.08
98.82
98.73
98.60
98.22
87.30
87.26
FBI Boseline
98.46
98.46
97.63
97.63
87.13
87.13
98.499
98.49
100.0
100.0
87.48
87.48
We constructed the GT dataset described above contain all the information in the GT, and the results are as follows Table The meaning of FE Baseline is the performance of networks of LeNet-5, M5, and CNN. In these experiments, it has been verified that is possible to learn various types of datasets using one network cell and that information can be embedded together. Consequently,
to validate if the output vector can
share association cell or layers to learn without being limited to the dataset type and embed all information inside the GT into a vector.
4 CONCLUSION
We introduced a data-driven network that can jointly learn relationships and hierarchical information. This study is to be free from architecture and has been developed to connect various types of information being developed. And The author of this paper believes that one day a neural network will emerge that can perform all human tasks. In addition, this is an association model that behaves like human sensory organs and can be described as similar to a human neural network. We think this network can be used as a deep neural network part of DQN DQN(Mnih et al. 2013). We will try to leverage this network to approach problems that have not been solved before. This paper is part of series; in the next paper, we introduce Graph Tree Deductive (GTD))

---

Under review as a conference paper at ICLR 2022
REFERENCES
Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das. Very deep convolutional neural networks for raw waveforms. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 421-425. IEEE, 2017.
Yingtong Dou, Kai Shu, Congying Xia, Philip S Yu, and Lichao Sun. User preference-aware fake news detection. arXiv preprint MKKi:2104.122555 2021.
Christoph Goller and Andreas Kuchler. Learning task-dependent distributed representations by backpropagation through structure. In Proceedings of International Conference on Neural Networks (ICNN'96) volume 1. pp. 347-352. IEEE, 1996.
Suzana Herculano-Houzed The human brain in numbers: linearly scaled-up primate brain. Frontiers in human neuroscience, 3:31, 2009.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-007 2006.
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1577-154, 2006.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554 2558, 1982.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456.
PMLR, 2015.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746-1755
Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D144 1181.
URL / anthol ogt.olog1ooo1o111111111
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Thomas Kipf and Max Welling. Semi- sumi-supervised classification with graph convolutional networks. arXiv preprint arXiv: (xXiv:660900200) 2016.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2324 1998.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 160iv:1608.033
2016.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language lechnologies, pp. 142-150, 2011.
Rafael Malach, JB Reppas, RR Benson, KK Kwong. H Jiang, WA Kennedy, PJ Ledden, TJ Brady, BR Rosen, and RB Tootell. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Sciences, 92(18): -135-8139, 1995.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 132iv:13/2.5602, 2013.
Lili Mou, Ge Li, Zhi Jin, Lu Zhang, and Tao Wang. Tbenn: A bree-based convolutional neural network for programming language processing. arXiv preprint arXiv: 2014.
10

---

Under review as a conference paper at ICLR 2022
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Icml, 2010.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. JEEE Transactions on knowledge and data engineering, 22(10))1345 1359, 2009.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings the 2014 conference empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batel training with batch-channel normalization and weight standardization, arXiv preprint arXiv:I903.10 10.20000.000,, 2019.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions neural networks, 20(1):61-8 2008.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods Natural Language Processing, pp- 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL ottpps//aelaatiioooo..rr//////
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, lya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting The journal of machine learning research, 15(1):129-1958, 2014.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 2017.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03320, 2018
Bernard Widrow et al. Adaptive" adaline" Neuron Using Chemical" memistors." ". 1960.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chenggi Zhang. Graph wavenet for deep spatial-temporal graph modeling. arXiv preprint arX1v:1906000221, 2019.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and $ Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4-24 24, 2020.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint 1rXiv:1505.00855 2015.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8710, 2018.
11

---

Under review as a conference paper at ICLR 2022
A
TYPE EMBEDDING NETWORKS FOR BATCH TRAINING
Algorithm 4 Batch Type Embedding
1: function TYPFEMMIREDDINN((Gmeeel
2: dict, =
key(T) value( key(batch (ectbattch___rrr)________ value(T, index
3:
dictid==11
4:
for idCsattch T, + enumerate((ZZuttee do
5: dict;drliddsaaed (T, len(dict_[7))
6:
b How to combine data the same type
7: end for
8:
for
T; Xr t- didddiiiiimmss do
D T (key), X (values) Type Batch Convolution
9:
x, [ onchoo(r).rpeeca((en(X..))
10:
dictal=]=X,
end for
12:
outpuibbtch =
for idTbotch f- ....N do
Restoration
14:
7,idæ, & dictiad -ddchiddiddddaatt
15:
X, = d[cctu[T]
16:
arrtputoaach.appendeee [idx,])
17: end for
18: return Stack puttpulatte))
19:
end function
First, make dictionary data structures. One is dictionary(dict.) that stores data for each data type. The other is a dictionary(dictaa.) that preserves the batch-index information of the data. We tour the mini-batch data in order and the key is the type(r) of data, and the value is the input data(æ). And the other dictionary stores the type and index.
Therefore, the input data collected for each type, which is called X,- And the input data become bype-mini-batch data for each type and becomes batch-inputs for the feature extraction network of that type.
Let x, denote the output of the feature extraction network. Finally, the x, moves to the original batch index through the previously-stored batch index(dict
B COMPARE ARCHITECTURE: GTC & GTR
Table 3: Compare Architecture
Name
GTC
GTR
Level layer The number W
O
X
The number of levels
Depth-limited
Fixed
Not fixed
The number of parameters Input size level The special cases
Use more
Use less than
Adjustable
Fixed
FCNN, MLP
RNN(recurrent, recursive)
In this section, we compare the network characteristics of GTC-series and GTR-series recursive models. In the of GTC, there is different Wiv for each level, expressing each level layer and the convolution of all information. On the other hand, in the case of GTR, the main difference is the recursive convolution with only W cell. This is the most significant difference when comparing the networks.
Therefore, GTC adjust the number of features by level; for example, the size of input features can differ between Levels and 0. It is possible for GTC to train datasets by adjusting parameters at any level. On the other hand, GTR must be trained with the same input and hidden size.
12

---

Under review as a conference paper at ICLR 2022
Thus, divided GTNN into two groups. Since GTC has network in charge of each level, more parameters are needed, and it is appropriate for depth-limited GT. if the Flo of input size is 0 and the number of nodes is in GTN, FCNN and MLP can be special cases of GTC as IW in z and (TW te-1 (IW )))) On the other hand, GTR traverses all GTNs with a GTR cell. Therefore is possible train even if the maximum depth is not fixed and fewer parameters used.
in addition, there is no sibling node in GTN p. RNN can be special cases of GTR as IW E ETT. Consequently, the compare result in Table 3
C USED MODELS
GTR Cell
GTC Lv
GTRAs Cell
GTAS Lv
M5 (Found))
LoNet-5 (Timage)
CNN (Tiang)
GTC Lv1
GTAS Lv1
GTC
GTAs
X
(a) Feature Extraction Networks
(b) GTR & GTC
Figure 6: Used Models
Table 4: Feature Extraction Networks for Image dataset
LaNee-5
LoNet-5 for
Model
kernel
(5.5)
tanh
AvgPool2D
(2:2)
(2,2)
Conv2D
16
(5.5)
tanb
16
(5.5)
tanh
AvgPoo12D
(2.2)
(2.2)
(5:5)
(5,5)
tanh
layer
120
tanh
PC layer 2
B4
10
solimax
Zero padding
128
Final
128
1 al. 1989) was used the image feature extraction network. We create a dimension of 128 by applying zero padding to the extracted features without using the affine- lFyere- layers) and then forward it to GTNN.
Table 5: Feature Extraction Networks for Sound dataset
MSNCrroup Nomm)
MSCGr Norm) for GINN
sorm
activation
kennel
stride
acrm
activation
128
group
relu
group 16
retu
MaxPoolIL
ComvlD
128
128
16
relu
128
128
group 16
rein
MaxPoelID
group
retu
group
relu
MaxPoolID
ConvlD
256
512
16
relu
256
512
group 16
relu
MisPoelID
FC layer
512
log softmax
512
128
lealy ralla
Final
128
13

---

Under review as a conference paper at ICLR 2022
M5 al. 2017) was used as the sound feature extraction network. As described above(Sec 2.2
we used group normalization(Wu & He 2018) without using Batch norm[Loffe & Szegedy 2015); We the affine laC layer) and deliver it to the GTNN in 128 dimensions. It could be implemented in torchaudid
Table 6: Feature Extraction Networks for Natural language dataset
CNN
CNN GINN
Model
kernel
stride
activation
In
Out
komel
stide
activation
Cony2D
100
(3,5)
relu
100
(3.3)
rel
ArgRoel2D
Com22D
(4.4)
relu
(4.4)
AogPool2DD
(2.2)
(2.2)
Cony2D
100
(5.5)
relu
100
(5.5)
relo
(300 (00,40005000
(300,400,500)
1200
Dropout
RC layer
1200
1200
128
Final
128
Kim 2014) was used as the feature extraction network for natural language processing. We slightly modified the contents in this paper and combined them with GTNN. We did not use drop out because we wanted to emphasize the difference between transfer learning and general learning in the association model. The glove(Pennington network with 100 dimensions. We used an adam aptimizer(Kingna & Ba. (0044), a learning rate of 0.001, a cosine annealing(T max=2, eta min=le-05) for batch-size is 32 and the MNIST Speech Command and IMDB classes are 10, 35, 2.
al. 2014) was used for the pre-trained word embedding
Hutter 2016).
D
How DO WE DESIGN THE STRUCTURE OF THE GRAPH TREE?
FC Layer
0.5g
FC Layer 2
6.ro
RNN Cell
x: hg
FC Layer
FC Layer 1
RNN Cell
X
x, lbg
FC Layer
0.
X
RNN Cell
X
x, hb
FC Layer
U.Ta
RNN Cell
X2
CNN
x, ht
CNN (r,)
FC Layer
J.T
RNN Cell
X
x
X
x
ho
MLP
GT
CNN
GT
RNN(recurrent)
GT
(a) MLP & GT
(b) CNN & GT
(c) RNN((recurrent) GT
Figure 7: Model structure & Data structure
These architectures are the models underlying existing networks. First, Fig (() means a multi-layer perceptron model. we express this architecture GT and learn with GTC, which has input only in leaf node, it becomes the same operation process.
recoen t oo
14

---

Under review as a conference paper at ICLR 2022
Second, Fig.2(b) means CNN model. Using the CNN network for the feature extraction network part with GTC will be the same operation.
Third, Fig ( means recurrent neural network model(RNN). Using tree with only child to express sequence and learn with GTR will be the same operation.
FC Layer
readout
X
0,
XF
X
X
X,T
x,
x,
X.T
GNN
GT
RNN(recursire)
GT
(a) GNN
(b) RNN(recursive)
Figure 8: Model structure & Data structure
These models are architectures that learn relational and hicrarchical information. First, Fig a) means GNN model. The same delivery process is obtained if the relationship and inputs are expressed as relationship and inputs among sibling nodes of the graph tree with GTC.
Second, Fig (() means a recursive neural network model(RNN). If we learn using identity matrix with GTR, it will be the same delivery process.
O,
0,
FC Layer
Iocund
Xasape
Xtusa
X,
X, T2
X
CNN
CNN
X
WoveZiec (foend)
CNN (1,)
CNN (r2)
(Kung)
x
2-stream CNN & GT
(b) GT contained various data
Figure 9: Model structure & Data structure
These networks are structures in which each feature extraction process is combined. If we design feature extraction networks for each type and construct graph tree with two or three children Fig (a,,), it will be the same delivery process.
15

---

Under review as a conference paper at ICLR 2022
Gom!l
Fremch
Impoitth
Suseget
Comart
Pracue)
Xonage
Xiangange
Figure 10: Ideal GT
Finally, this structure is an ideal graph tree structure (Fig 10 The feature extraction network cor- responds the sensory organ. And the feature extraction process(V) is selected according to the type(r) of data x(x), and x becomes vector F.
And information integration occurs. We call it association network. This network serves to embed various extracted information into one vector( h root) by applying relationship and hierarchical information.
At this time, expressing a tree structure that learns well for each type of data or task is the same as designing the architecture of the network.
Therefore, instead of using fixed layers to learn according to the flow of information, build graph tree to learn according the flow of information. It will enable the network to learn networks that can handle multiple tasks or multiple datasets simultaneously.. not just specific tasks or specific datasets. Because various architectures can be expressed by data.
Visual information is input when we open our eyes, and information is not input when we close our eyes. It can be expressed when there is does not have image information in the graph tree.
In addition, it could express a structure such as V1, V2, V3, V4, V5/MT in visual cortex, and graph tree has hicrarchical information and relational information. The grammar of natural languages can also be expressed as a tree parser.
Voice information can also be extracted from MFCC algorithms etc. And all information is integrated and embedded in association networks of Graph Tree Neural Networks.
E THE EXPERIMENT 3 INDIVIDUAL & SIMULTANEOUS
O.5o
d,
Xised
Xivna
Tira
FirE
Sound
Image
Graph
Tree
(a) Sound
(b) Image
(c) Graph
(d) Tree
Figure Graph tree datasets of the experiment 3
16

---

Under review as a conference paper at ICLR 2022
40-Class
ga, mage
gio, image
STC sound
09
07. sound
gF, sound
X
0.-
gTT, sound
M5
gTC longuage
0.-
www.legggggg
LeNet5 Howage
FC Falayer (Faieatool
Glove Hanmy
(Fooned)
git grogh
gragh
10
Epoch
15
20
gtra, graph
CCN
gT, graph
X
CAT
(a) Feature Extraction Networks
(b) Learning plot (49class- Test acc)
Figure 12: Feature Extraction Networks & Learning plot
As third experiment, we learned datasets of various structures with GTNN. Data in the form of images, sound data, language data(Socher et al. 2013) in the tree structure, and data in the graph structure(Dou al., 202T) were learned the same time, and the results are follows, All of the performances at this time were similar to those of existing networks.
In addition, in the case of GTR and GTRAs, it can be seen that the learning speed for language data is lower than Recursive Neural Networks.
The reason is that the size of language dataset is relatively less than others. Therefore, since the language model has relatively less influence on loss, it seems to have been pushed back in the optimization process.
In the case of GTC and GTAs, the result of overfitting in the language model is shown. This is natural result because the layers of tree data are learned differently for each level.
As result, we proved that learning to express various information delivery structures with one network cell does not significantly affect performance. When we stop learning this network at 30
Table 7: the result of Experiment 3
49(=10+35+2+2) class
Model
MNIST
SST
UPFD-GOS
LeNet-5
98.52
M5 (Group Norm)
98.37
RNN(Recursive)
77.69
GCN
93.67
GATs
93.93
GTC
98.88
97.18
73.30
94.74
GTAS
98.80
96.67
73.39
94.85
GTR
98.68
97.46
76.74
93.15
GTRAS
98.35
97.44
76.79
94.67
class count
10
35
2
2
type
image
sound
tree
graph
epochs, the performance in test dataset is in the tabld7
17


Reviews:
[{"review_id": "d6Rmwlq9dFz", "reviewer": "Reviewer_oPzs", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally, I don't (and shouldn’t) reject a paper because of writing.\nHowever, the writing of this paper is too hard to follow.\nAmong hundreds of papers I have reviewed, it is the most distracting one.\nEven if there are significant technical contributions, I still cannot accept this paper because of the writing.\nHowever, the idea of applying a tree-like structure to generalize neural networks is still interesting.\nI am sorry that I cannot complete the reading because of the writing.", "summary_of_the_paper": "It **seems** that this paper tries to propose a general neural network architecture that is composed of multiple layer types based on a tree structure.", "summary_of_the_review": "Writing is too hard to follow.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "RDzJWSEbfBs", "reviewer": "Reviewer_Svha", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "From what I understand, I really like the research idea and direction of the paper. Unfortunately, I had extreme difficulties understanding it because the paper lacks a good structure and a fluent narrative (details below). Therefore, I cannot properly comment on the paper's strengths and weaknesses. Instead, I list the questions I had while reading the paper.\n\n* Apart from having a new architecture, I do not understand the exact problem setting of the paper: is it supposed to be multi-modality? Multi-task learning? An analysis of the relationship of brain-like learning and artificial learning? This is important, because a proper problem setting would determine the choice of baseline architectures and experiments. As things are, the paper feels \"ungrounded\".\n* In the paper's claims, several related works are not mentioned. In particular regarding dynamic computation graphs and dynamic resource allocation, the work around conditional computation (mixture of experts; more relevant, the \"Routing\" literature (Routing Networks and the Challenges of Modular and Compositional Computation) come to mind)\n* Terminology throughout the paper is strange; instead of \"inputs\" and \"outputs\" or \"predictions\" we have \"start\" and \"endpoints\". I understood this eventually, but it created some confusion for me in the beginning.\n* Somewhat related to the first point, I'm missing a \"big picture\" overview of what the proposed architecture tries to achieve, and why the proposed architecture is good at achieving this. This is particularly relevant for section 2: I wish for an overview at the beginning of each subsection (2.1-2.5), describing what the respective component is mean to achieve, how it does so, and why it was designed in this particular way.\n* Unfortunately, my confusion continues in the experimental section. I suspect that the experiments are non-standard, but I cannot actually tell, because the experimental setup is not described anywhere in detail. \n* I am not perfectly sure on the experiments. However, I still want to point out that the baselines are out of date. That may be fine, but then the choice of these baselines needs to be motivated.", "summary_of_the_paper": "The paper introduces a new neural network architecture called a \"Graph Tree Neural Network\". This architecture is inspired by several properties of the human brain: 1. it's multi-modal; 2. it allows for cross-input reasoning; 3. it can adapt its internal structure to the current input; 4. it can adapt the amount of computation per input; and 5. the human brain is larger than most neural networks.\n\nThe architecture itself relies on mode-specific representation components: later, in the experiments, they introduce targeted architectures for images, speech and text. These representations are then consumes by a convolutional logic that allows full communication within the different components of the network. Finally, these components can be recursively combined to yield a full graph tree neural network.\n\nThis architecture is then evaluated on several domains covering speech, vision and nlp datasets, showing consistent improvement over the selected baselines.", "summary_of_the_review": "Overall, I like the goal and the ideas behind the paper. Unfortunately, the paper is extremely unclear in its presentation: I could not put individual sections into context, I do not know what the goal of the paper is, and I don't understand the results. Therefore, I cannot recommend this paper for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KUvMl_MKV4D", "reviewer": "Reviewer_sfjx", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## What I like about this paper\n\n- The idea of unifying different architectures (RNN, Convolution, MLP) into one big neural network organized as a graph tree.\n\n- Dealing with multiple types of input modality is an important research direction.\n\n\n## Concerns\n\n- The paper is hard to follow. Several sentences are not well-formatted. I found it hard to extract the relevant contributions and claims from this paper.\n\n- I'm not convinced by the empirical results presented in section 3. All the numbers look very close to each other and might be within the error margin. I would have liked to have the source code, to test it myself.\n\n- On page 7, the authors mention the image GT has three nodes since \"the occipital lobe is the farthest from the frontal lobe in the human brain\". I find that justification dubious as longer axonal connections don't mean more processing. Citations would be needed to back up this claim.\n\n- A related work section is missing? I doubt there is no other work that can be related to this one. What are the limitations of let's say Graph Neural Network compared to Graph Tree Neural Network?\n\n- P.1: what does it mean to \"freely integrated\"?\n- P.1: there's a claim about \"existing networks are designed ... layers are fixed\" which is not true. Several papers describe approaches to adapt how information is processed in dynamic ways, e.g. ResNet [1], Adaptive Computation Time [2], etc.\n\n#### References\n- [1] He, Kaiming et al. “Deep Residual Learning for Image Recognition.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)\n- [2] Graves, Alex. “Adaptive Computation Time for Recurrent Neural Networks.” ArXiv abs/1603.08983 (2016)\n\n### Minor\n\n- This paper has heavy usage of acronyms. For the sake of clarity in the paper, I'd use the full names rather than the acronyms when talking about GT, GTN, GTNN, GTR, and GTC. Otherwise, I'd try to reduce their usage as much as possible.\n\n- I'm concerned by the name used to refer to the proposed approach. To me, Graph Tree Neural Networks means a neural network capable of taking graphs shaped as trees as input.\n\n-----\n### Typos\n- A lot of sentences don't start with an uppercase letter.\n- Missing whitespaces before several opening parentheses.\n", "summary_of_the_paper": "In this paper, the authors propose Graph Tree Neural Network (GTNN) a new learning model that is structured as a graph tree (i.e., a tree with links between siblings) where each node has to process the data coming from its children. GTNN can take as input data of various types where each node will be processing the data differently according to the data type. The authors describe how (de)convolution and recurrent neural networks can be achieved within GTNN. Experiments were conducted on some classic deep learning datasets of three different data types: images (MNIST), language (IMDB), and sound (Speech Command). The goals of those experiments were two see whether GTNN can process various datasets of different types and if GTNN can produce an output vector that retains all the information coming from the different data types.", "summary_of_the_review": "I believe designing novel architectures inspired by the human brain is an interesting and important research direction. However, in its current state, this paper was not able to convince me that the GTNN architecture was one worth pursuing. To be honest, it wasn't clear to me what were the actual contributions of the paper, and how the experiment results are backing up the claims made in this paper. For those reasons, I recommend rejecting this paper for ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: First, I would like to thank all the reviewers for their efforts in reading and understanding this paper. I tried to read the paper as well and I also find it's really difficult (if possible) for me to understand the ideas presented here. The most important task in writing a paper (as Reviewer Svha also suggested in his/her review) in the field of machine learning is to explain to your peers what is the problem you are trying to solve and how you solve (or partially solve) that problem. I think there is a consensus among the reviewers that the paper did not do a great job of that. I am not questioning the quality of the idea or the research here, but I think the paper here will need to do a significantly better job here in explaining the idea before it can be a good ICLR publication.
Decision: Reject