Year: 2021

Abstract:
In the field of deep learning, various architectures have been developed. However, most studies are limited to specific tasks or datasets due to their fixed layer structure. In this paper, we do not express the structure delivering information as a network model but as a data structure called a graph tree. And we propose two association models of graph tree neural networks(GTNNs) designed to solve the problems of existing networks by analyzing the structure of human neural networks. Defining the starting and ending points in a single graph is difficult, and a tree cannot express the relationship among sibling nodes. On the contrary, a graph tree(GT) can express leaf and root nodes as its starting and ending points and the relationship among sibling nodes. Instead of using fixed sequence layers, we create a GT for each data and train GTNN according to the tree’s structure. GTNNs are data-driven learning in which the number of convolutions varies according to the depth of the tree. Moreover, these models can simultaneously learn various types of datasets through the recursive learning method. Depth-first convolution (DFC) encodes the interaction result from leaf nodes to the root node in a bottom-up approach, and depth-first deconvolution (DFD) decodes the interaction result from the root node to the leaf nodes in a top-down approach. To demonstrate the performance of these networks, we conducted three experiments. In the first experiment, we verified whether it can be processed by combining graph tree neural networks and feature extraction networks(combining association models and feature extraction models). The second experiment is about whether the output of GTNN can embed information on all data contained in the GT(association). In the third experiment, we simultaneously learn the structure of images, sounds, graphs, and trees structure datasets (processing various datasets). We compared the performance of existing networks that separately learned image, sound, and natural language, relation datasets with the performance simultaneously learned by connecting these networks. As a result, these models learned without significant performance degradation, and the output vector contained all the information in the GT. And the importance of transfer-learning and fine-tuning in these models is explained.

heading: 1 INTRODUCTION
text: In deep learning, various architectures have been designed such as CNN, RNN, and GNN(LeCun et al., 1989; Hopfield, 1982; Scarselli et al., 2008; Wu et al., 2020). These networks show good performance in various fields such as image, text, and sound etc. However, most of the existing studies are focused on a specific dataset or task. And there are many tasks that humans can do but are difficult to perform with neural networks.
First, some of the difficulties with imitating human neural networks are introduced below. (C-i) Sensory organs that process information for each data type exist at several starting points. (C-ii) A tree cannot express the relationship between sibling nodes, and it is difficult to define the graph’s starting point and ending point. (C-iii) The structure that receives information is hierarchical and freely integrated with various structures. But existing networks are designed to learn only specific tasks or datasets because layers are fixed. (C-iv) The number of activated neurons and the processing depths differ depending on the data type and complexity. And sometimes the information is entered and sometimes not. (C-v) Human neurons are very numerous and process various types of data.
C-[·] denotes the characteristics. (C-i) Humans have various sensory organs from which they receive information such as sight, hearing, and smell, and transfer it to the cerebral cortex using different
information-processing organs. The optic nerve also begins with the visual cortex of the occipital
lobe, and the auditory nerve begins with the auditory cortex of the temporal lobe; this means that the two paths have different starting points. (C-ii) A graph can describe the relationship depending on whether all nodes are connected, but it is not easy to define the start and end points. A tree is a data structure that can define direction with the leaf and root nodes. On the other hand, a tree structure cannot express the relationship between sibling nodes. (C-iii) Information from different sensory organs is gathered in the association area. Typically, the posterior parietal cortex(Malach et al., 1995) is located at the top of the head as part of the parietal lobe, and sensory information such as vision and hearing is fused and interpreted; Then, information is coordinated in the frontal lobe and humans act. (C-iv) There is also an over-fitting problem in DNN(Widrow et al., 1960), and various attempts have been made to solve this in (Srivastava et al., 2014). In DNN, simply deeper layers lead to an over-fitting problem(Dai et al., 2017). In particular, NasNet(Zoph et al., 2018) has attempted to design a layer automatically through reinforcement learning and RNN. This means that there is an appropriate network depth depending on the complexity of the dataset; we want to design a network that could adjust the depth of the layers according to the complexity of each data, not the data set. (C-v) Also, the number of human neurons is about 85 billion(Herculano-Houzel, 2009), which is difficult to express in a network. Therefore, GTNNs are designed to solve these problems.
A-[·] denotes our approach of C-[·]. (A-i) This network can perform information processing, such as that done by sensory organs before their signals are entered into the cerebral cortex. In this paper, we express this sensory organ as a feature-extraction process(sec.2.2). Type information is stored with the data and converted into an input vector by different feature extraction processes for each type. (A-ii) We propose a new data structure called Graph Tree Node(GTN) and Graph Tree(GT). A graph tree can be expressed using leaf and root nodes as the starting and ending points, as well as the relationship among sibling nodes. In addition, we can modify the existing tree dataset at no high cost. (A-iii) subtrees of different types are merged and make a final decision at the root node; this structure is difficult to express with the existing forward-learning method. Therefore, we used a recursive-learning method(Goller & Kuchler, 1996). This methodology has been effective in analyzing the semantics of programming source code or natural language(Socher et al., 2013; Mou et al., 2014). We call the recursive-convolution methodology used in GTNNs the depth-first convolution(DFC) methodology(sec.2.3). DFC is a convolution method in which subtrees originating from different Leaf nodes are integrated into a bottom-up approach and represent hierarchical and relational information. GTNNs can simultaneously learn various types of datasets through the recursive learning method because the graph tree data structure can express various model structures. (A-iv) GTNNs are data-driven learning in which the number of convolutions varies according to the depth of the tree. Instead of using fixed sequence layers, we create a graph tree for each data and learn according to the tree’s structure. If we modify this network further, we can adjust the depth according to the type and complexity of each data. In addition, We introduce two models of GTNNs called graph tree convolutional networks(GTC) and graph tree recursive networks(GTR), and these models are mathematically related to MLP(Hinton et al., 2006) and RNN. The Fully Connected layer(FC layer) in MLP as Level Layer of GTC, and Time in RNN as depth of GTR on a special case. (A-v) The number of human neurons is enormous, and it is difficult to express all of their structures in a network; on the other hand, this network can represent multiple neurons because it can perform recursive end-to-end learning according to the amount of information.
This network theory holds that "units of information have a relationship in the form of a graph, then become a bigger unit of information, and have a relationship with other units of information. At this point, the unit of information is a set of neurons, and we can express it as a vector with GTNN."
To demonstrate the performance of this network, we conducted three experiments. And we used several benchmark datasets (image(MNIST(LeCun et al., 1998)), sound(Speech Commands(Warden, 2018)), text(IMDB(Maas et al., 2011),SST(Socher et al., 2013)), graph(Dou et al., 2021)).
In the first experiment, It verifies whether feature extraction networks and association networks can be learned together. we compared the performance of existing networks that separately learned image(LeNet-5(LeCun et al., 1989), sound(M5(Dai et al., 2017)), text(CNN(Kim, 2014)) with the performance simultaneously learned by connecting these networks for feature extraction into GTNN.
The second experiment contained one or three types of data into a GT, and GTNNs learned these GTs. Then, we verified whether the output contained all the information inside the GT. Then, we conducted a verification experiment on whether the network’s output contained all information on the tree.
In the third experiment, we verifies whether data of various structures(image, sound, tree, graph) can be learned and checked the plot of the learning process in appendix E. As a result, the network was learned without significant degradation in performance compared to when learning existing networks separately, and all information on the tree could be embedded as a vector.

heading: 2 GRAPH TREE NEURAL NETWORKS
text: In this section, we proposes graph tree architecture. The architecture consists of three parts
(i) Defining the data structure of graph tree node(GTN) and designing the graph tree structure (GT). (ii) Defining the feature-extraction(FE) process. (iii) Defining the GTNN model.

heading: 2.1 GRAPH TREE DATA STRUCTURE
text: The graph tree (GT) is a data structure that can express relational and hierarchical information.
x : (Input) – we can store the data such as images, sound, text, or tabular to each node. τ : (Type) – This refers to the type of information matching the input x. the feature-extraction
function(ψ) is selected by τ like Eqn.1. Ac : (Children Adjacency Matrix) – This refers to the relationship information that exists in GTN.
the number of children is N and we can express it as Ac ∈ RN×N . C : (Children) – This refers to the child nodes of GTN matching to the node of Ac. If there is no
matching child, it is replaced with GTN∅, which carries the initial hidden state −→ 0 instead. We can express this as C = {GTN1,GTN2, ..GTNN}.
We can express GTN = {x, τ,Ac,C}, GTNi ∈ GT. GTNroot denotes the root node of GT. The reason for defining the relationship among child nodes is the convenience of implementation. If we define GTN in the above way, we can convert the tree dataset into the GT dataset without significantly modifying the tree structure that has previously been useful.

heading: 2.2 FEATURE EXTRACTION NETWORKS & TYPE BIAS
text: −→x = Ψ(x, τ) = [ψτ (x), onehot(τ)] (1) The x(input) and τ (type) exist together in GTN, and the feature extraction process of x is selected by τ as Eqn 1. We expressed the function of feature extraction as Ψ, and this function converts x to −→x . And the one-hot vector has the effect of having a different bias value for each type(τ ). The weight parameter corresponding to the type one-hot vector means the bias value for the corresponding type, and the activation threshold value is adjusted for each type-bias. It can represent data type information and that it is transmitted from which nerve. In addition, we do not need to perform adding operations(+) for bias. Therefore, the empty space of GT can be used as a zero-vector when batch learning is performed. And the dictionary structure is very useful for batch learning; we attached the methodology to appendix.A. This methodology divides mini-batch samples by type, and the type-mini-batch size varies each time. Therefore, we recommend using a weight standardization(Qiao et al., 2019) or group normalization(Wu & He, 2018) to avoid batch normalization(Ioffe & Szegedy, 2015) affected by the batch size. We described the details of the network we used in appendix.C.

heading: 2.3 DEPTH-FIRST CONVOLUTION & DECONVOLUTION
text: This section introduces two propagation methods of depth-first convolution (DFC) (a convolution method for traversing all nodes of GT) and depth-first deconvolution (DFD). Depth-first search (DFS) is a search algorithm for the tree data structure. Unlike general recursive convolution, DFC and DFD are methodologies for learning children’s relationships, feature extraction networks together.
Algorithm 1 Depth First Convolution
1: function DFC(GTNp, lv) 2: xp, τp,Acp,Cp ←GTNp.items() 3: −→x p, ∗i1← Ψ[τp](xp) 4: ChildrenList← [ ] 5: for q ← 1...Np do ▷ left-first 6: −→ x′ pq, −→ h′pq ← DFC(Cp[q], lv + 1) 7: ChildrenList.append([ −→ x′ pq, −→ h′pq]) 8: end for 9: 10: if len(Cp) is 0 then ▷ leaf node 11: GTNp.more← (∗i1) 12: return [−→x p, −→ 0 ] 13: end if 14: [x′p,h′p]← Stack(ChildrenList) 15: −→ h p, i2←GTNN(Ac,x′p,h′p, lv) 16: GTNp.more← (∗i1, ∗i2) 17: return [−→x p, −→ h p]
18: end function
Post-order Depth-first convolution (Left-First) If DFS is post-order, iteration starts from the leaf node. Likewise, DFC is a recursive convolution that propagates from the deepest nodes. GTNN Utilizes the hidden vector expressing the current depth and state.
First, we need to bring four items(xp, τp, Acp, Cp) of the current GTNp. the p means the convolution order, and xp is embedded by the feature extraction network(Ψ) according to the type(τp) and becomes the feature vector(−→x p). Graph tree network models perform the convolution by receiving information( −→ x′ pq, −→ h′pq) from the current node’s children and the relationship(Acp) among children’s information. We can use this process like learning the relationship between siblings nodes. the −→ h p means the convolution output of the child nodes. finally, −→x p and −→ h p are delivered to the parent node. If the child node does not exist(Cp), the current node means a leaf node. Therefore, the feature vector(−→x p) and initial hidden state( −→ 0 ) are transmitted to the parent node.
The important thing is that the root node only receives information from their children and does not the convolution with the root feature vector. Therefore, like algo.3, convolution is performed once more on the output of the DFC. If we repeat this process, we finally get −→ h root. Additionally, we can store information(∗i1, ∗i2) generated during the DFC process in GTN.more and use it during the DFD process (ex. If the aggregate function is a maxpool, we can use the indices information during deconvolution).
Algorithm 2 Depth First Deconvolution
1: function DFD(x̃p, h̃p,GTNp, ∗lv) 2: τp,Acp,Cp ←GTNp.dconv_items() 3: ∗i1, ∗i2←GTNp.more 4: GTNp.x̂p ← Ψ−1[τp](x̃p, ∗i1) 5: if len(Cp) is 0 then ▷ leaf node 6: return 7: end if 8: Âcp, x̃′p, h̃ ′ p←GTNN −1(∗Acp, h̃p, lv, ∗i2) 9: GTNp.Âc ← Âcp 10: for q ← Np...1 do ▷ right-first 11: x̃′pq ,h̃ ′ pq ← x̃′p[q], h̃′p[q] 12: DFD(x̃′pq, h̃ ′ pq,GTNp.C[q], lv + 1) 13: end for 14: return 15: end function
Pre-order Depth-first Deconvolution (Right-First) DFD is a decoding methodology for GTNNs. The DFC propagates from the leaf node to the root node, whereas the DFD propagates from the root node to the leaf node in the order of pre-order depth first to decode this.
First, we need to bring three items(τp, Acp, Cp) of the current GTNp. x̃p is decoded by the feature decoder network(Ψ−1) according to the type(τp) and the decoded output is stored in the GTNp.x̂p.
If the child does not exist, this means a leaf node and performs a return. Otherwise, The hidden state(h̃p) is restored to x̃′p, h̃ ′ p through GTNN
−1; and h̃′p is transferred to child nodes. Also, if DFC proceeds from left to right as left first (1...N ), DFD proceeds from right to left as right first (N...1). Please see Fig.3.
Algorithm 3 Propagate (for AutoEncoder Models)
1: function PROPAGATE(GTroot) 2: [−→x ′root, −→ h′ root]← DFC(GTNroot, 1) 3: −→ h root, ∗i2←GTNN(I, −→ x′ root, −→ h′ root,0) 4: ∗_, x̃′root, h̃′root←GTNN −1(I,
−→ h root,0,∗i2)
5: DFD(x̃′root, h̃′root,GTNroot, 1) 6: return −→ h root,GTNroot 7: end function
Like the propagate function (algo.3), we can freely encode and decode through DFC and DFD. We want to show the relationship between DFC and DFD and the possibility of expanding to the widely used autoencoder(Hinton & Salakhutdinov, 2006) models. In addition, Implementing this recursive function as a loop(while) will speed up.

heading: 2.4 ASSOCIATION NEURAL NETWORK MODELS : GTR & GTC SERIES
text: This section proposes GTR and GTC series that perform inductive learning in GTNNs. We describe the convolution performed in one graph tree node by DFC. These networks can be used as an association cell or layer, and we explained the difference between the two models in appendix B.
These association models do not use any fixed architecture and integrate information according to the GT structure. And by fusing graph and tree structures, we can learn more diverse data structures with one cell and various architectures can be expressed by data(please see the appendix.D). Accordingly, these models are freer to perform various tasks and learn a lot of information using fewer parameters.
These association models are as follows: The feature extraction models create the information from data; It is the "smallest unit" of information in the GT. The parent GTN receives information from their children and perform the convolution operation with the relation term; the convolution output is a "bigger unit" of information. Finally, a root GTN has all GT information and perform the convolution operation; the output is "the biggest unit" of information.

heading: 2.4.1 ASSOCIATION CELL : GRAPH TREE RECURSIVE NEURAL NETWORK
text: We will describe a GTR cell that trains the GT described previously in Sec 2.1. GTR model can be expressed as GTR = {W,Ψ, g, σ}W ∈ RF ′×(F+B+F ′). F is the node-feature size, B is the type-bias size, and F ′ is the output and hidden size, which is the information received from the child node. Ψ denotes the feature-extraction(Eqn 1) process described above, g denotes the aggregate function, and σ denotes the activation function.
−→x p = [ψτp(xp), onehot(τp)] = Ψ(xp, τp),−→x p ∈ RF+B (2)
−→ h p = g(σ(D̃ − 12 cp ÃcpD̃ − 12 cp (W(∥Npq=0[ −→ x′ pq, −→ h′pq] T )T ))) (3)
The propagation for GTNN is proceeding by the DFC traversing from leaf nodes to a root node (sec.2.3). p is the order of DFC, q is the child node number, Np is the number of children of GTNp, xp is the node-input of GTNp. The Ψ extracts the feature vector(−→x p) from xp of GTNp through the feature extraction process(Eqn.2) considering the type information(τp). The first location to start is leaf nodes by DFC. In the Leaf node, the initial hidden state is a zero vector( −→ 0 ), and this is concatenated with−→x p to become [−→x p, −→ h p] ∈ RF+B+F ′ through the concatenation process as [, ] and then delivered to the parent node. If a child node exists, the current node receives information from a child node as [ −→ x′pq, −→ h′pq]. ∥Npq=0[ −→ x′ pq, −→ h′pq] T means that [ −→ x′ pq, −→ h′pq] matching the graph nodes(Acp) existing in the p-th node are stacked as much as Np, which is ∥ Np q=0[ −→ x′ pq, −→ h′pq]
T ∈ RNp×(F+B+F ′). We applied the GCN methodology(Kipf & Welling, 2016), which is useful in the GNN field. Therefore we expressed as Ãcp = Acp + I, where I is the Identity matrix. If we express the connection value as 1, just the more connected nodes becames the scale value larger than others. Therefore, D̃− 1 2 cp ÃcpD̃ − 12 cp
is applied using the order matrix D̃cp of Ãcp as a method of normalizing the relationship matrix in Eqn. 3. we used F as 128, B as 3(image, sound, language), and F ′ as 128, and σ as ReLU(Nair & Hinton, 2010), and g was the readout of Max. If there is no sibling node of the current node, GTR is mathematically identical to the RNN as IW[−→x , −→ h ]. RNN can be a special case of GTR, and the depth of the tree means times of the RNN.

heading: 2.4.2 ASSOCIATION LAYERS : GRAPH TREE CONVOLUTIONAL NEURAL NETWORK
text: GTC is composed of GTC = {{W0, ...Wm},Ψ, g, σ}. There is Wlv ∈ RF ′ lv×(Flv+B+F ′ lv+1) in charge of each level, and the network in charge of each level is called the level layer. The input size of the level is Flv , its output size is F ′lv , and the output size of the child is F ′ lv+1. Therefore, it is possible to adjust the input, hidden size; and it can learn the depth-limited tree, m means Maximum-Depth. We set Flv and F ′lv to 128 for all lvs in the same way as GTR in the experiment. If the input size of Flv is 0 and there is no sibling node of the current node, this network is mathematically identical to the FC layer as IWlv−→x T and the MLP as (IW0..(IWlv−1(IWlv−→x T ))), I is the Identity matrix. The mathematical expression of GTC is similar to Eqn 3. The depth of the tree indicates the number of layers of the MLP. Therefore, the MLP and FC layer can be a special case of GTC.

heading: 2.5 ATTENTION MODELS
text: There is an attention process in the human brain. Therefore, GATs(Veličković et al., 2017), a model that has been useful recently, was combined.

heading: 2.5.1 GRAPH TREE RECURSIVE ATTENTION NETWORKS
text: We introduce GTRAs that learn the importance through attention by slightly modifying the expression of the GTR. It is composed of GTRAs = {W,Ψ, g, σ, σa,−→a } that added {σa,−→a } in GTR. A parameter for attention mechanism is added (R2F ′ × R2F ′ → R) and the attention’s activation function used LeakyReLU(Xu et al., 2015) in the same way as GATs.Npq is a set of nodes connected
to the q-th child’s node in the Acp of GTNp, and we can express this as:
αpqr = exp(LeakyReLU(−→a T [W[ −→ x′ pq, −→ h′pq] T ,W[ −→ x′ pr, −→ h′pr]
T ]))∑ k∈Npq exp(LeakyReLU( −→a T [W[ −→ x′ pq, −→ h′pq]T ,W[ −→ x′ pk, −→ h′pk]T ]))
(4)
With the attention methodology introduced in GATs(Veličković et al., 2017), it learns how the r-th node is of importance to the q-th node. This information is replaced with the part to which the adjacency matrix is connected.
If Acp does not exist in the GTN, we utilized to select critical features as :
scorepqr =
{ Sigmoid(−→a T [W[ −→ x′ pq, −→ h′pq] T ,W[ −→ x′ pr, −→ h′pr]
T ]) if q ̸= r 1 if q = r
(5)
αpqr = exp(scorepqr)∑Npq k=1 exp(scorepqk)
(6)
The self-node is set to 1 and the relationship values of the node are replaced with a real value of 0 ∼ 1 using Sigmoid. Therefore, we can train the attention learning without Acp. It means to learn relationships based on self-node. Other useful models is self-adaptive model(Wu et al., 2019).
Therefore, the GTRAs are as follows: −→ h p = g(σ((Acp ⊙ αp)(W[x′p,h′p]T ))) (7)
In Eqn.7, ⊙ indicates point-wise and x′p is a stack of −→x ′pq as ∥ Np q=0 −→ x′Tpq and it is x ′ p ∈ RNp×(F+B). h′p is a stack of −→ h′pq as ∥ Np q=0 −→ h′Tpq and h ′ p ∈ RNp×F ′ . Therefore it is [x′p,h′p] ∈ RNp×(F+B+F ′).
To further stabilize the self-attention process, we introduce a multi-head attention mechanism: −→ h p = g(||Kk=1σ(Acp ⊙ αkp)(Wk[x′p,h′p]T )) (8)
whereK is the number of multi-heads. The results from multiple heads are concatenated and delivered to a parent node. We set K to 8 and set output dim for each head to 16(=F ′lv/K). therefore we set F ′ lv to 128. This process becomes a cell and delivers the result from the leaf node to the root node.

heading: 2.5.2 GRAPH TREE ATTENTION NETWORKS
text: The GTAs model is modified from GTC, and GATs’ attention mechanism was applied. this model can be expressed as GTAs = {{W0, ...Wm}, {−→a 0, ...−→a m},Ψ, g, σ, σa}, and {σa,−→a lv} are added in GTC(−→alv ∈ R2F ′ lv ). Unlike GTRAs, GTAs can design different sizes of F, F ′ to match the lv. this model is designed to select critical features, and it is similar to Sec.2.5.1
Supervised learning Supervised learning uses datasets with labels. In this network, the −→ h root was mapped to the dimension of the class using a fully connected layer(F ′ to class count) and we calculated the log softmax and negative log likelihood loss for supervised learning as:
ŷ = log_softmax(TaskLayer( −→ h root)) (9)
loss = negative_log_likelihood_loss(y, ŷ) (10)

heading: 3 EXPERIMENTAL RESULTS
text: This section introduces the contents of the experiment and the GT dataset consisting of the image, sound, and natural language. How to design GT is introduced in appendix.D. Datasets Our goal is to express the information delivery process of existing models as a graph tree of data. The first experiment is whether various datasets can be simultaneously learned by combining GTNN and feature extraction networks. the GT we used in the experiment is illustrated by Fig.4(a). Layer node means that it has no input value, only performs convolution. If there is no sibling node, GTC performs the same operation as one FC layer. Sound and natural language GT have two nodes, and image GT has three nodes because It was configured similarly to figure (likes this fig.1). The
important point is not limited to any specific architecture; later, this GT could be modified according to the task or complexity.
The second experiment is about whether the output( −→ h root) of GTNN is embedded information on all data contained in the GT(association). Therefore, we performed image, sound, and text classification by putting image, sound, text data in one GT, and we also learned the GT used in the first experiment simultaneously. One problem is that when the three types of data are combined, The number of combinations of the dataset becomes too huge. For example, there are 50,000 images, 105,000 sounds, and 17,500 natural language samples, and when these data are combined, we need to generate 50,000×105,000×17,500 GT samples. For this problem, we used a sampling method. When loading each data, two data of different types are sampled randomly. Thus, when we load one data, two GTs are generated of the one type GT and the three type GT. The test accuracy is the result of averaging 5-fold. the GT dataset of the second experiment is illustrated by Fig.4(a,b,c,d). In the third experiment, we added the graph and tree structure dataset, This is described in Appendix E.

heading: 3.1 THE EXPERIMENT 1 : INDIVIDUAL & SIMULTANEOUS (PROCESSING VARIOUS DATASETS)
text: We trained without using a pre-trained model to check whether various feature extraction models can be learned simultaneously. Only word embedding network(Pennington et al., 2014) was used as a pretrained model. Existing networks learn only about specific datasets. Therefore, LeNet-5(LeCun et al., 1998), M5(Dai et al., 2017), and CNN(Kim, 2014) were selected for feature extraction networks of the image(MNIST(LeCun et al., 1998)), sound(Speech Command(Warden, 2018)), and natural language(IMDB(Maas et al., 2011)). We described more details in appendix C. The reason for choosing these networks is based on Convolutional Neural Networks specialized extract features.
Since the number of classes is different for each data, the test was divided into two cases. (Task Layers): The last layer was placed differently for each dataset and mapped to the class dimension of the dataset. (47 class): The one last layer was used for all datasets, and the number of class dimensions is equal to the sum of class dimensions of all datasets. Because if the last layer is placed differently for each dataset, information on the input type may be informed. All models were learned during 30 epochs, and the results were as Table.1(Task layers, 47 class), and the learning process is (Fig.5(a)).
The learning processes of GTNNs are similar to the process of individually learning feature extraction networks; if the model learns like this, several problems are found. LeNet-5 is well trained, but M5
needs to be more trained, and CNN has an overfitting problem. The reason is that each network has different epochs to have optimal performance, and setting learning parameters is more complex than learning individually. transfer learning & fine-tuning Therefore, after learning individually, we combined the association model to perform transfer-learning and fine-tuning(Pan & Yang, 2009). The result is Table.1(Transferlearning, Fine-tuning). In transfer learning, the parameters of feature extraction networks are not modified, and in fine-tuning, the parameters are modified. We used the validation set to learn each feature extraction network and adopt the lowest value of the validation loss to use the model. As a result, the LeNet-5, M5, CNN were pre-trained with 26, 95, 3 epochs. These models were combined with GTNN to perform fine-tuning and transfer learning. Then, the network was re-learned, and learning was stopped at epochs with the lowest value of validation loss using each of the same validation datasets. As in the results of Table.1(Transfer-learning, Fine-tuning) and Fig.5(b,c), in the case of transfer learning, performance was poor in the IMDB dataset. We thought it was overfitting because the IMDB dataset was relatively small. We reduced the learning rate from 0.001 to 0.0001, but the results were similar. While fine-tuning was similar to the performance learned individually in all datasets. And the performance improved compared to when fine-tuning was not performed. Consequently, If transfer learning and fine tuning are used by setting parameters well according to the network, the performance of the network can be improved. In Experiment 3, we learn datasets of various structures(Image, sound, tree, graph) through GTNN. It is described in Appendix.E

heading: 3.2 THE EXPERIMENT 2 : CAN ALL DATA INFORMATION BE CONTAINED? (ASSOCIATION)
text: We constructed the GT dataset described above(Fig.4(a,b,c,d)) to validate if the output vector can contain all the information in the GT, and the results are as follows Table.2. The meaning of FE Baseline is the performance of networks of LeNet-5, M5, and CNN. In these experiments, it has been verified that it is possible to learn various types of datasets using one network cell and that information can be embedded together. Consequently, we can share an association cell or layers to learn without being limited to the dataset type and embed all information inside the GT into a vector.

heading: 4 CONCLUSION
text: We introduced a data-driven network that can jointly learn relationships and hierarchical information. This study is to be free from architecture and has been developed to connect various types of information being developed. And The author of this paper believes that one day a neural network will emerge that can perform all human tasks. In addition, this is an association model that behaves like human sensory organs and can be described as similar to a human neural network. We think this network can be used as a deep neural network part of DQN(Mnih et al., 2013). We will try to leverage this network to approach problems that have not been solved before. This paper is part of a series; in the next paper, we introduce Graph Tree Deductive Networks(GTDs).

heading: A TYPE EMBEDDING NETWORKS FOR BATCH TRAINING
text: Algorithm 4 Batch Type Embedding 1: function TYPEEMBEDDING(GTbatch) 2: dictx = {} ▷ key(τ) : value(listx) 3: dictidx = {} ▷ key(batch_idx) : value(τ , indexdictx) 4: for idxbatch, τ, x← enumerate(GTbatch) do 5: dictidx[idxbatch] = (τ , len(dictx[τ ])) 6: dictx[τ ].append(x) ▷ How to combine data of the same type 7: end for 8: for τ,Xτ ← dictx.items() do ▷ τ(key), X(values) 9: xτ = [Ψ[τ ](Xτ ), onehot(τ ).repeat(len(Xτ ))] ▷ Type Batch Convolution 10: dictx[τ ] = xτ 11: end for 12: outputbatch = [] 13: for idxbatch← 1...N do ▷ Restoration 14: τ, idxτ ← dictidx[idxbatch] 15: xτ = dictx[τ ] 16: outputbatch.append(xτ [idxτ ]) 17: end for 18: return Stack(outputbatch) 19: end function
First, make two dictionary data structures. One is a dictionary(dictx) that stores data for each data type. The other is a dictionary(dictidx) that preserves the batch-index information of the data. We tour the mini-batch data in order and store the key is the type(τ ) of data, and the value is the input data(x). And the other dictionary stores the type and index.
Therefore, the input data are collected for each type, which is called Xτ . And the input data become type-mini-batch data for each type and becomes batch-inputs for the feature extraction network of that type.
Let xτ denote the output of the feature extraction network. Finally, the xτ moves to the original batch index through the previously-stored batch index(dictidx).

heading: B COMPARE ARCHITECTURE: GTC & GTR
text: In this section, we compare the network characteristics of GTC-series and GTR-series recursive models. In the case of GTC, there is a different Wlv for each level, expressing each level layer and the convolution of all information. On the other hand, in the case of GTR, the main difference is the recursive convolution with only one W as a cell. This is the most significant difference when comparing the networks.
Therefore, GTC can adjust the number of features by level; for example, the size of input features can differ between Levels 1 and 0. It is possible for GTC to train datasets by adjusting parameters at any level. On the other hand, GTR must be trained with the same input and hidden size.
Thus, we divided GTNN into two groups. Since GTC has a network in charge of each level, more parameters are needed, and it is appropriate for depth-limited GT. if the Flv of input size is 0 and the number of nodes is 1 in GTN, FCNN and MLP can be special cases of GTC as IWlv−→x T and (IW0..(IWlv−1(IWlv−→x T ))). On the other hand, GTR traverses all GTNs with a GTR cell. Therefore it is possible to train even if the maximum depth is not fixed and fewer parameters are used.
in addition, If there is no sibling node in GTNp, RNN can be a special cases of GTR as IW[−→x , −→ h ]T . Consequently, the compare result in Table 3.

heading: C USED MODELS
text: LeNet-5(LeCun et al., 1989) was used as the image feature extraction network. We create a dimension of 128 by applying zero padding to the extracted features without using the affine-layer(FC layers) and then forward it to GTNN.
M5(Dai et al., 2017) was used as the sound feature extraction network. As described above(Sec.2.2), we used group normalization(Wu & He, 2018) without using Batch norm(Ioffe & Szegedy, 2015). We use the affine layer(FC layer) and deliver it to the GTNN in 128 dimensions. It could be implemented in torchaudio1.
CNN(Kim, 2014) was used as the feature extraction network for natural language processing. We slightly modified the contents in this paper and combined them with GTNN. We did not use drop out because we wanted to emphasize the difference between transfer learning and general learning in the association model. The glove(Pennington et al., 2014) was used for the pre-trained word embedding network with 100 dimensions. We used an adam optimizer(Kingma & Ba, 2014), a learning rate of 0.001, a cosine annealing(T max=2, eta min=1e-05) for schedulers(Loshchilov & Hutter, 2016). the batch-size is 32 and the MNIST, Speech Command and IMDB classes are 10, 35, 2.

heading: D HOW DO WE DESIGN THE STRUCTURE OF THE GRAPH TREE?
text: These architectures are the models underlying existing networks. First, Fig.7(a) means a multi-layer perceptron model. If we express this architecture as GT and learn with GTC, which has input only in leaf node, it becomes the same operation process.
1https://pytorch.org/tutorials/intermediate/speech_command_recognition_ with_torchaudio_tutorial.html
Second, Fig.2(b) means a CNN model. Using the CNN network for the feature extraction network part with GTC will be the same operation.
Third, Fig.7(c) means a recurrent neural network model(RNN). Using a tree with only one child to express sequence and learn with GTR will be the same operation.
These models are architectures that learn relational and hierarchical information. First, Fig.8(a) means a GNN model. The same delivery process is obtained if the relationship and inputs are expressed as relationship and inputs among sibling nodes of the graph tree with GTC.
Second, Fig.8(b) means a recursive neural network model(RNN). If we learn using an identity matrix with GTR, it will be the same delivery process.
These networks are structures in which each feature extraction process is combined. If we design feature extraction networks for each type and construct a graph tree with two or three children Fig.9(a,b), it will be the same delivery process.
Finally, this structure is an ideal graph tree structure (Fig.10). The feature extraction network corresponds to the sensory organ. And the feature extraction process(Ψ) is selected according to the type(τ ) of data x(x), and x becomes vector −→x . And information integration occurs. We call it an association network. This network serves to embed various extracted information into one vector( −→ h root) by applying relationship and hierarchical information.
At this time, expressing a tree structure that learns well for each type of data or task is the same as designing the architecture of the network.
Therefore, instead of using fixed layers to learn according to the flow of information, build a graph tree to learn according to the flow of information. It will enable the network to learn networks that can handle multiple tasks or multiple datasets simultaneously, not just specific tasks or specific datasets. Because various architectures can be expressed by data.
Visual information is input when we open our eyes, and information is not input when we close our eyes. It can be expressed when there is or does not have image information in the graph tree.
In addition, it could express a structure such as V1, V2, V3, V4, V5/MT in visual cortex, and a graph tree has hierarchical information and relational information. The grammar of natural languages can also be expressed as a tree parser.
Voice information can also be extracted from MFCC algorithms etc. And all information is integrated and embedded in association networks of Graph Tree Neural Networks.

heading: E THE EXPERIMENT 3 : INDIVIDUAL & SIMULTANEOUS
text: As a third experiment, we learned datasets of various structures with GTNN. Data in the form of images, sound data, language data(Socher et al., 2013) in the tree structure, and data in the graph structure(Dou et al., 2021) were learned at the same time, and the results are as follows. All of the performances at this time were similar to those of existing networks.
In addition, in the case of GTR and GTRAs, it can be seen that the learning speed for language data is lower than Recursive Neural Networks.
The reason is that the size of language dataset is relatively less than others. Therefore, since the language model has a relatively less influence on loss, it seems to have been pushed back in the optimization process.
In the case of GTC and GTAs, the result of overfitting in the language model is shown. This is a natural result because the layers of tree data are learned differently for each level.
As a result, we proved that learning to express various information delivery structures with one network cell does not significantly affect performance. When we stop learning this network at 30
epochs, the performance in test dataset is in the table7.

Review summaries:
The paper introduces a new neural network architecture called a "Graph Tree Neural Network". This architecture is inspired by several properties of the human brain: 1. it's multi-modal; 2. it allows for cross-input reasoning; 3. it can adapt its internal structure to the current input; 4. it can adapt the amount of computation per input; and 5. the human brain is larger than most neural networks.  The architecture itself relies on mode-specific representation components: later, in the experiments, they introduce targeted architectures for images, speech and text. These representations are then consumes by a convolutional logic that allows full communication within the different components of the network. Finally, these components can be recursively combined to yield a full graph tree neural network.  This architecture is then evaluated on several domains covering speech, vision and nlp datasets, showing consistent improvement over the selected baselines.

In this paper, the authors propose Graph Tree Neural Network (GTNN) a new learning model that is structured as a graph tree (i.e., a tree with links between siblings) where each node has to process the data coming from its children. GTNN can take as input data of various types where each node will be processing the data differently according to the data type. The authors describe how (de)convolution and recurrent neural networks can be achieved within GTNN. Experiments were conducted on some classic deep learning datasets of three different data types: images (MNIST), language (IMDB), and sound (Speech Command). The goals of those experiments were two see whether GTNN can process various datasets of different types and if GTNN can produce an output vector that retains all the information coming from the different data types.