Graph Tree Neural Networks

1 INTRODUCTION . In deep learning , various architectures have been designed such as CNN , RNN , and GNN ( LeCun et al. , 1989 ; Hopfield , 1982 ; Scarselli et al. , 2008 ; Wu et al. , 2020 ) . These networks show good performance in various fields such as image , text , and sound etc . However , most of the existing studies are focused on a specific dataset or task . And there are many tasks that humans can do but are difficult to perform with neural networks . First , some of the difficulties with imitating human neural networks are introduced below . ( C-i ) Sensory organs that process information for each data type exist at several starting points . ( C-ii ) A tree can not express the relationship between sibling nodes , and it is difficult to define the graph ’ s starting point and ending point . ( C-iii ) The structure that receives information is hierarchical and freely integrated with various structures . But existing networks are designed to learn only specific tasks or datasets because layers are fixed . ( C-iv ) The number of activated neurons and the processing depths differ depending on the data type and complexity . And sometimes the information is entered and sometimes not . ( C-v ) Human neurons are very numerous and process various types of data . C- [ · ] denotes the characteristics . ( C-i ) Humans have various sensory organs from which they receive information such as sight , hearing , and smell , and transfer it to the cerebral cortex using different information-processing organs . The optic nerve also begins with the visual cortex of the occipital lobe , and the auditory nerve begins with the auditory cortex of the temporal lobe ; this means that the two paths have different starting points . ( C-ii ) A graph can describe the relationship depending on whether all nodes are connected , but it is not easy to define the start and end points . A tree is a data structure that can define direction with the leaf and root nodes . On the other hand , a tree structure can not express the relationship between sibling nodes . ( C-iii ) Information from different sensory organs is gathered in the association area . Typically , the posterior parietal cortex ( Malach et al. , 1995 ) is located at the top of the head as part of the parietal lobe , and sensory information such as vision and hearing is fused and interpreted ; Then , information is coordinated in the frontal lobe and humans act . ( C-iv ) There is also an over-fitting problem in DNN ( Widrow et al. , 1960 ) , and various attempts have been made to solve this in ( Srivastava et al. , 2014 ) . In DNN , simply deeper layers lead to an over-fitting problem ( Dai et al. , 2017 ) . In particular , NasNet ( Zoph et al. , 2018 ) has attempted to design a layer automatically through reinforcement learning and RNN . This means that there is an appropriate network depth depending on the complexity of the dataset ; we want to design a network that could adjust the depth of the layers according to the complexity of each data , not the data set . ( C-v ) Also , the number of human neurons is about 85 billion ( Herculano-Houzel , 2009 ) , which is difficult to express in a network . Therefore , GTNNs are designed to solve these problems . A- [ · ] denotes our approach of C- [ · ] . ( A-i ) This network can perform information processing , such as that done by sensory organs before their signals are entered into the cerebral cortex . In this paper , we express this sensory organ as a feature-extraction process ( sec.2.2 ) . Type information is stored with the data and converted into an input vector by different feature extraction processes for each type . ( A-ii ) We propose a new data structure called Graph Tree Node ( GTN ) and Graph Tree ( GT ) . A graph tree can be expressed using leaf and root nodes as the starting and ending points , as well as the relationship among sibling nodes . In addition , we can modify the existing tree dataset at no high cost . ( A-iii ) subtrees of different types are merged and make a final decision at the root node ; this structure is difficult to express with the existing forward-learning method . Therefore , we used a recursive-learning method ( Goller & Kuchler , 1996 ) . This methodology has been effective in analyzing the semantics of programming source code or natural language ( Socher et al. , 2013 ; Mou et al. , 2014 ) . We call the recursive-convolution methodology used in GTNNs the depth-first convolution ( DFC ) methodology ( sec.2.3 ) . DFC is a convolution method in which subtrees originating from different Leaf nodes are integrated into a bottom-up approach and represent hierarchical and relational information . GTNNs can simultaneously learn various types of datasets through the recursive learning method because the graph tree data structure can express various model structures . ( A-iv ) GTNNs are data-driven learning in which the number of convolutions varies according to the depth of the tree . Instead of using fixed sequence layers , we create a graph tree for each data and learn according to the tree ’ s structure . If we modify this network further , we can adjust the depth according to the type and complexity of each data . In addition , We introduce two models of GTNNs called graph tree convolutional networks ( GTC ) and graph tree recursive networks ( GTR ) , and these models are mathematically related to MLP ( Hinton et al. , 2006 ) and RNN . The Fully Connected layer ( FC layer ) in MLP as Level Layer of GTC , and Time in RNN as depth of GTR on a special case . ( A-v ) The number of human neurons is enormous , and it is difficult to express all of their structures in a network ; on the other hand , this network can represent multiple neurons because it can perform recursive end-to-end learning according to the amount of information . This network theory holds that `` units of information have a relationship in the form of a graph , then become a bigger unit of information , and have a relationship with other units of information . At this point , the unit of information is a set of neurons , and we can express it as a vector with GTNN . '' To demonstrate the performance of this network , we conducted three experiments . And we used several benchmark datasets ( image ( MNIST ( LeCun et al. , 1998 ) ) , sound ( Speech Commands ( Warden , 2018 ) ) , text ( IMDB ( Maas et al. , 2011 ) , SST ( Socher et al. , 2013 ) ) , graph ( Dou et al. , 2021 ) ) . In the first experiment , It verifies whether feature extraction networks and association networks can be learned together . we compared the performance of existing networks that separately learned image ( LeNet-5 ( LeCun et al. , 1989 ) , sound ( M5 ( Dai et al. , 2017 ) ) , text ( CNN ( Kim , 2014 ) ) with the performance simultaneously learned by connecting these networks for feature extraction into GTNN . The second experiment contained one or three types of data into a GT , and GTNNs learned these GTs . Then , we verified whether the output contained all the information inside the GT . Then , we conducted a verification experiment on whether the network ’ s output contained all information on the tree . In the third experiment , we verifies whether data of various structures ( image , sound , tree , graph ) can be learned and checked the plot of the learning process in appendix E. As a result , the network was learned without significant degradation in performance compared to when learning existing networks separately , and all information on the tree could be embedded as a vector . 2 GRAPH TREE NEURAL NETWORKS . In this section , we proposes graph tree architecture . The architecture consists of three parts ( i ) Defining the data structure of graph tree node ( GTN ) and designing the graph tree structure ( GT ) . ( ii ) Defining the feature-extraction ( FE ) process . ( iii ) Defining the GTNN model . 2.1 GRAPH TREE DATA STRUCTURE . The graph tree ( GT ) is a data structure that can express relational and hierarchical information . x : ( Input ) – we can store the data such as images , sound , text , or tabular to each node . τ : ( Type ) – This refers to the type of information matching the input x. the feature-extraction function ( ψ ) is selected by τ like Eqn.1 . Ac : ( Children Adjacency Matrix ) – This refers to the relationship information that exists in GTN . the number of children is N and we can express it as Ac ∈ RN×N . C : ( Children ) – This refers to the child nodes of GTN matching to the node of Ac . If there is no matching child , it is replaced with GTN∅ , which carries the initial hidden state −→ 0 instead . We can express this as C = { GTN1 , GTN2 , .. GTNN } . We can express GTN = { x , τ , Ac , C } , GTNi ∈ GT . GTNroot denotes the root node of GT . The reason for defining the relationship among child nodes is the convenience of implementation . If we define GTN in the above way , we can convert the tree dataset into the GT dataset without significantly modifying the tree structure that has previously been useful . 2.2 FEATURE EXTRACTION NETWORKS & TYPE BIAS . −→x = Ψ ( x , τ ) = [ ψτ ( x ) , onehot ( τ ) ] ( 1 ) The x ( input ) and τ ( type ) exist together in GTN , and the feature extraction process of x is selected by τ as Eqn 1 . We expressed the function of feature extraction as Ψ , and this function converts x to −→x . And the one-hot vector has the effect of having a different bias value for each type ( τ ) . The weight parameter corresponding to the type one-hot vector means the bias value for the corresponding type , and the activation threshold value is adjusted for each type-bias . It can represent data type information and that it is transmitted from which nerve . In addition , we do not need to perform adding operations ( + ) for bias . Therefore , the empty space of GT can be used as a zero-vector when batch learning is performed . And the dictionary structure is very useful for batch learning ; we attached the methodology to appendix.A . This methodology divides mini-batch samples by type , and the type-mini-batch size varies each time . Therefore , we recommend using a weight standardization ( Qiao et al. , 2019 ) or group normalization ( Wu & He , 2018 ) to avoid batch normalization ( Ioffe & Szegedy , 2015 ) affected by the batch size . We described the details of the network we used in appendix.C .

Summary: The paper introduces a new neural network architecture called a "Graph Tree Neural Network". This architecture is inspired by several properties of the human brain: 1. it's multi-modal; 2. it allows for cross-input reasoning; 3. it can adapt its internal structure to the current input; 4. it can adapt the amount of computation per input; and 5. the human brain is larger than most neural networks.  The architecture itself relies on mode-specific representation components: later, in the experiments, they introduce targeted architectures for images, speech and text. These representations are then consumes by a convolutional logic that allows full communication within the different components of the network. Finally, these components can be recursively combined to yield a full graph tree neural network.  This architecture is then evaluated on several domains covering speech, vision and nlp datasets, showing consistent improvement over the selected baselines.