Graph Tree Neural Networks

1 INTRODUCTION . In deep learning , various architectures have been designed such as CNN , RNN , and GNN ( LeCun et al. , 1989 ; Hopfield , 1982 ; Scarselli et al. , 2008 ; Wu et al. , 2020 ) . These networks show good performance in various fields such as image , text , and sound etc . However , most of the existing studies are focused on a specific dataset or task . And there are many tasks that humans can do but are difficult to perform with neural networks . First , some of the difficulties with imitating human neural networks are introduced below . ( C-i ) Sensory organs that process information for each data type exist at several starting points . ( C-ii ) A tree can not express the relationship between sibling nodes , and it is difficult to define the graph ’ s starting point and ending point . ( C-iii ) The structure that receives information is hierarchical and freely integrated with various structures . But existing networks are designed to learn only specific tasks or datasets because layers are fixed . ( C-iv ) The number of activated neurons and the processing depths differ depending on the data type and complexity . And sometimes the information is entered and sometimes not . ( C-v ) Human neurons are very numerous and process various types of data . C- [ · ] denotes the characteristics . ( C-i ) Humans have various sensory organs from which they receive information such as sight , hearing , and smell , and transfer it to the cerebral cortex using different information-processing organs . The optic nerve also begins with the visual cortex of the occipital lobe , and the auditory nerve begins with the auditory cortex of the temporal lobe ; this means that the two paths have different starting points . ( C-ii ) A graph can describe the relationship depending on whether all nodes are connected , but it is not easy to define the start and end points . A tree is a data structure that can define direction with the leaf and root nodes . On the other hand , a tree structure can not express the relationship between sibling nodes . ( C-iii ) Information from different sensory organs is gathered in the association area . Typically , the posterior parietal cortex ( Malach et al. , 1995 ) is located at the top of the head as part of the parietal lobe , and sensory information such as vision and hearing is fused and interpreted ; Then , information is coordinated in the frontal lobe and humans act . ( C-iv ) There is also an over-fitting problem in DNN ( Widrow et al. , 1960 ) , and various attempts have been made to solve this in ( Srivastava et al. , 2014 ) . In DNN , simply deeper layers lead to an over-fitting problem ( Dai et al. , 2017 ) . In particular , NasNet ( Zoph et al. , 2018 ) has attempted to design a layer automatically through reinforcement learning and RNN . This means that there is an appropriate network depth depending on the complexity of the dataset ; we want to design a network that could adjust the depth of the layers according to the complexity of each data , not the data set . ( C-v ) Also , the number of human neurons is about 85 billion ( Herculano-Houzel , 2009 ) , which is difficult to express in a network . Therefore , GTNNs are designed to solve these problems . A- [ · ] denotes our approach of C- [ · ] . ( A-i ) This network can perform information processing , such as that done by sensory organs before their signals are entered into the cerebral cortex . In this paper , we express this sensory organ as a feature-extraction process ( sec.2.2 ) . Type information is stored with the data and converted into an input vector by different feature extraction processes for each type . ( A-ii ) We propose a new data structure called Graph Tree Node ( GTN ) and Graph Tree ( GT ) . A graph tree can be expressed using leaf and root nodes as the starting and ending points , as well as the relationship among sibling nodes . In addition , we can modify the existing tree dataset at no high cost . ( A-iii ) subtrees of different types are merged and make a final decision at the root node ; this structure is difficult to express with the existing forward-learning method . Therefore , we used a recursive-learning method ( Goller & Kuchler , 1996 ) . This methodology has been effective in analyzing the semantics of programming source code or natural language ( Socher et al. , 2013 ; Mou et al. , 2014 ) . We call the recursive-convolution methodology used in GTNNs the depth-first convolution ( DFC ) methodology ( sec.2.3 ) . DFC is a convolution method in which subtrees originating from different Leaf nodes are integrated into a bottom-up approach and represent hierarchical and relational information . GTNNs can simultaneously learn various types of datasets through the recursive learning method because the graph tree data structure can express various model structures . ( A-iv ) GTNNs are data-driven learning in which the number of convolutions varies according to the depth of the tree . Instead of using fixed sequence layers , we create a graph tree for each data and learn according to the tree ’ s structure . If we modify this network further , we can adjust the depth according to the type and complexity of each data . In addition , We introduce two models of GTNNs called graph tree convolutional networks ( GTC ) and graph tree recursive networks ( GTR ) , and these models are mathematically related to MLP ( Hinton et al. , 2006 ) and RNN . The Fully Connected layer ( FC layer ) in MLP as Level Layer of GTC , and Time in RNN as depth of GTR on a special case . ( A-v ) The number of human neurons is enormous , and it is difficult to express all of their structures in a network ; on the other hand , this network can represent multiple neurons because it can perform recursive end-to-end learning according to the amount of information . This network theory holds that `` units of information have a relationship in the form of a graph , then become a bigger unit of information , and have a relationship with other units of information . At this point , the unit of information is a set of neurons , and we can express it as a vector with GTNN . '' To demonstrate the performance of this network , we conducted three experiments . And we used several benchmark datasets ( image ( MNIST ( LeCun et al. , 1998 ) ) , sound ( Speech Commands ( Warden , 2018 ) ) , text ( IMDB ( Maas et al. , 2011 ) , SST ( Socher et al. , 2013 ) ) , graph ( Dou et al. , 2021 ) ) . In the first experiment , It verifies whether feature extraction networks and association networks can be learned together . we compared the performance of existing networks that separately learned image ( LeNet-5 ( LeCun et al. , 1989 ) , sound ( M5 ( Dai et al. , 2017 ) ) , text ( CNN ( Kim , 2014 ) ) with the performance simultaneously learned by connecting these networks for feature extraction into GTNN . The second experiment contained one or three types of data into a GT , and GTNNs learned these GTs . Then , we verified whether the output contained all the information inside the GT . Then , we conducted a verification experiment on whether the network ’ s output contained all information on the tree . In the third experiment , we verifies whether data of various structures ( image , sound , tree , graph ) can be learned and checked the plot of the learning process in appendix E. As a result , the network was learned without significant degradation in performance compared to when learning existing networks separately , and all information on the tree could be embedded as a vector . 2 GRAPH TREE NEURAL NETWORKS . In this section , we proposes graph tree architecture . The architecture consists of three parts ( i ) Defining the data structure of graph tree node ( GTN ) and designing the graph tree structure ( GT ) . ( ii ) Defining the feature-extraction ( FE ) process . ( iii ) Defining the GTNN model . 2.1 GRAPH TREE DATA STRUCTURE . The graph tree ( GT ) is a data structure that can express relational and hierarchical information . x : ( Input ) – we can store the data such as images , sound , text , or tabular to each node . τ : ( Type ) – This refers to the type of information matching the input x. the feature-extraction function ( ψ ) is selected by τ like Eqn.1 . Ac : ( Children Adjacency Matrix ) – This refers to the relationship information that exists in GTN . the number of children is N and we can express it as Ac ∈ RN×N . C : ( Children ) – This refers to the child nodes of GTN matching to the node of Ac . If there is no matching child , it is replaced with GTN∅ , which carries the initial hidden state −→ 0 instead . We can express this as C = { GTN1 , GTN2 , .. GTNN } . We can express GTN = { x , τ , Ac , C } , GTNi ∈ GT . GTNroot denotes the root node of GT . The reason for defining the relationship among child nodes is the convenience of implementation . If we define GTN in the above way , we can convert the tree dataset into the GT dataset without significantly modifying the tree structure that has previously been useful . 2.2 FEATURE EXTRACTION NETWORKS & TYPE BIAS . −→x = Ψ ( x , τ ) = [ ψτ ( x ) , onehot ( τ ) ] ( 1 ) The x ( input ) and τ ( type ) exist together in GTN , and the feature extraction process of x is selected by τ as Eqn 1 . We expressed the function of feature extraction as Ψ , and this function converts x to −→x . And the one-hot vector has the effect of having a different bias value for each type ( τ ) . The weight parameter corresponding to the type one-hot vector means the bias value for the corresponding type , and the activation threshold value is adjusted for each type-bias . It can represent data type information and that it is transmitted from which nerve . In addition , we do not need to perform adding operations ( + ) for bias . Therefore , the empty space of GT can be used as a zero-vector when batch learning is performed . And the dictionary structure is very useful for batch learning ; we attached the methodology to appendix.A . This methodology divides mini-batch samples by type , and the type-mini-batch size varies each time . Therefore , we recommend using a weight standardization ( Qiao et al. , 2019 ) or group normalization ( Wu & He , 2018 ) to avoid batch normalization ( Ioffe & Szegedy , 2015 ) affected by the batch size . We described the details of the network we used in appendix.C .

Summary: In this paper, the authors propose Graph Tree Neural Network (GTNN) a new learning model that is structured as a graph tree (i.e., a tree with links between siblings) where each node has to process the data coming from its children. GTNN can take as input data of various types where each node will be processing the data differently according to the data type. The authors describe how (de)convolution and recurrent neural networks can be achieved within GTNN. Experiments were conducted on some classic deep learning datasets of three different data types: images (MNIST), language (IMDB), and sound (Speech Command). The goals of those experiments were two see whether GTNN can process various datasets of different types and if GTNN can produce an output vector that retains all the information coming from the different data types.