—In this paper, a two -way pre -training method is presented to combine the learning of deep neural networks with the learning of others. The learning of these networks often does not converge due to facing a high number of local minima. It is now possible to avoid many local minima with the appropriate initial value of network weights. The two -way layer -by-layer pre - training method is a fast and efficient method that adjusts the initial values of its weights in a forward and backward direction using the desir ed inputs and outputs of the network. For this purpose, a hidden layer based on the weights of the pre -trained layer of the deep network and the auxiliary weights is used to teach the auxiliary networks. Then, the weight values obtained from their training are put under pre -training in the main structure of the network, and for the accurate adjustment of the weights, integral training is done. This method was used for pre -training the weights of three deep neural networks that recognize the person, emotiona l states and handwritten digits, and it was shown that by using this pre -training method, the speed of learning convergence increases dramatically. Also, the number of recognitions in the face database improves significantly, which indicates the increase i n the network's generalization power using this method. Keywords - pre-learning, bidirectional, cross -association, deep structure, multi -layer neural networks, convergence of learning I. INTRODUCTION In the last few years, according to the evidence based on the presence of deep structures in the human brain, researchers have started to learn deep multi -layer neural networks. These structures are capable of learning multiple levels of input. Therefore, it is possible to extract the ranks of the components in them. As the high -level components are formed as a combination of the low -level components in several levels [1]. At present, the usual learning of networks with more than two hidden layers often leads to weak results. So when all the layers are trained through a criterion function such as the degree of similarity to the inputs or the similarity to the desired layers, the results may be worse than shallow models. They depend a lot on the type of functions and initial value of the network parameter s [2-4]. In other words, in learning the deep structures of the network, with the increase in the number of layers and neurons, the problem of local minima becomes more noticeable in the middle of the way, it is needed. Therefore, in addition to the direct and usual approach of learning a multilayer neural network that uses the error backpropagation method, there are new approaches that become useful with increasing data complexity and the number of neurons, which are under the heading of advanced methods [6]. Pre-learning methods are used to free the learning process from the topical minims that exist in the middle of the way as a basic obstacle in the learning process. These methods seek to find a suitable starting point for the weights of the network. In addition to facilitating and accelerating the process of learning convergence, pre -learning methods also improve the generalization power of the network. It has been shown in [ 7-8], that with equal learning error, the network that has an appropriate initia l value before learning has a lower test error. In other words, pre -training forces the parameters of the network to be in a region of the parameters space where the desired answer is. One of the existing methods for pre -training deep networks is the goal -oriented method. This method was used for the first time to learn deep neural networks in [ 9] to extract the basic non - linear components, and then it was used in [ 10-11]. In this method, at first, the neural network used is a neural network (BNN), and a hidden layer is assumed, which is added to its depth during training. At each stage of the growth of the number of layers and the depth of the network, the learned values of the weights of the previous shallower network are used as the initial pre-trained va lues for the deeper network of the next stage [12]. In this way, the initial values of the weights of the deeper neural network are closer to the desired final values than the final values, so they are closer to the final goal. It has been shown that this method can eliminate local minimizations in the path of deep neural networks [13]. In 2006, Hinton presented the 2 method of decomposition into restricted Boltzmann machines for pre -training multilayer neural networks to reduce the nonlinear dimension [14-16]. Other applications have been presented. Considering that the existing pre -training methods have been presented for self -association learning, this method is considered a new solution for the other -association learning of deep neural networks. The two -way layer -by-layer pre -training method, which is a key mode of the layer -by-layer pre -training method [ 17-18], as a basic pre -training method with the appropriate selection of the initial conditions, helps the error backpropagation algorithm to reach the des ired minimum. The presented method for pre -training deep neural networks with the purpose of image recognition has been evaluated. In continuation, the two -way pre -education method will be introduced. Then, in the third section, the implementations and their results are given. Finally, the summary and conclusion will be presented in the fourth section. II. TWO-WAY LAYER -BY-LAYER PRE -LEARNING METHOD Neural networks with deep structures using multiple layers of neurons have the capacity to learn multi -stage and complex nonlinear transformations. But as mentioned, with the increase in the number of neural network layers, it becomes difficult and sometimes impossible to converge their lear ning through the error backpropagation algorithm due to the existence of local minima. Now, if the number of possible local minimums can be limited in the way of learning the network, the convergence will be improved. It should be noted that deep neural ne tworks are able to produce outputs with the desired accuracy if they are able to analyze and analyze the input information in different layers without losing information. Therefore, it is necessary to express the different inputs in different layers. If tw o distinct samples of the input data are reproduced in the network layers with a single and undifferentiated expression from each other, then the nonlinear transformation of the input up to this layer could not work for these samples one by one and discrim inatively. Therefore, some of the input information has been lost [2]. In the case of deep neural networks whose neurons use the step function, considering that each of these neurons creates a hypersurface in its input space, they will be able to different iate the input samples in its two inner layers. Do not separate. In such networks in pattern learning, in order to be able to express all distinct patterns in the output in a different way, it is necessary that all non -matching patterns are expressed with different codes in all layers. For this purpose, it is necessary that in the input space, between the two non -matching samples, at least one hyper -page should pass and the same situation should continue in all the subsequent layers of the network. This mea ns that in all layers, a separate area should be formed by the hypersurfaces of that layer for each layer, so that two separate samples with a single expression are not presented in any layer, and the discriminating information is not reduced. As a result, it means that in the output, the differentiation of the patterns is reduced to a great extent. In the two -way layer -by-layer pre -training method, the above problem can be avoided by placing the neuron surfaces step by step in the appropriate place to pres erve the differentiation in the layers. In this method, the desired input and output information is directly used in the initial setting of the weights of the deep network. The initial values of the weights of different layers based on input or output are determined according to which layer is closer. In these stages, due to the fact that the networks of one hidden layer are taught, fast learning and the role of local minimums is less. This pre -training method transfers the initial weights of the deep neura l network to a situation that is free from many possible local minima. In this way, the weights of the neurons of different layers of the network benefit from a relatively efficient pre -training method. It should be kept in mind that this layer -by-layer pr e- training method can be used for any convolutional network with any number of hidden layers. It can be shown that the presented method for pre -learning weights is a very efficient method. In the two -way layer -by-layer pre -training method, if you have a de ep network with 2n weight layers according to figure (1), its structure is analyzed in parallel from the beginning and the end, respectively, forward and backward. In this way, in the sequential path, BNNs of a hidden layer is defined whose input weights ( Wi) represent the weights of the layer in question from the deep network. Their output weights (Vi) are also auxiliary weights that are used to determine Wis. In this path, the first BNN is trained with the input vector and the subsequent BNNs are trained with the image of the input vector of the previous BNN in their bottleneck layer. In this path, the first BNN is trained with the output vector and the subsequent BNNs are trained with the image of the input vector of the previous BNN in its bottleneck lay er. Unlike the BNNs of the forward path, in the backward path, the output weights of the BNNs will be used in the deep network structure. Figure 1. How to analyze a deep neural network in the two -way layer -by- layer pre -training method For pre -training with the two -way layer -by-layer method to determine the values of weights in the forward path, the local cost functions are defined based on the correct recons truction of the input of the deep neural network and in the backward path based on the desired output reconstruction [4]. 𝐸𝑖𝐿𝐹=1 2‖𝑌𝑖−1𝐹−𝑍𝑖𝐹‖2 𝑛−1≥𝑖>1 (1) 𝐸𝑗𝐿𝐵=1 2‖𝑌𝑗−1𝐵−𝑍𝑗𝐵‖2 2𝑛−1≥𝑗≥𝑛+1 (2) 3 EiLF and E jBF in relations (1) and (2), respectively, the self - adherence errors of BNNs i -th and j -th are in the forward and backward paths, which are defined as the input reconstruction error in the output layer. In these relations , Y i-1F and Yj-1B the input images of the (i -1)-th and j -th BNNs are in the two paths corresponding to the relations (3) and (4) in their throat layer [6]. 𝑌𝑖−1𝐹=𝑓(𝑌𝑖−1𝐹𝑤𝑖−1−𝑏) 𝑛≥𝑖≥2 (3) 𝑌𝑗−1𝐵=𝑓(𝑌𝑗𝐵𝑉𝑗−𝑏) 2𝑛≥𝑗≥𝑛+2 (4) Also Y 0F The input vector of the first BNN in the forward path is equal to X, the input vector of the deep neural network, and Y 2nB, the input vector of the first BNN in the backward path, is equal to the desired output vector L of the deep neural network. ZiF and ZjB also, which are the i -th and j -th BNN outputs in the forward and backward directions, are defined in relations (5) and (6). 𝑍𝑖𝐹=𝑓(𝑌𝑖𝐹𝑉𝑖−𝑏) 𝑛−1≥𝑖≥1 (5) 𝑍𝑗𝐵=𝑓(𝑌𝑗−1𝐵𝑤𝑗−𝑏) 2𝑛+1≥𝑗≥𝑛+1 (6) In the above relationships, b determines the value of the threshold level for the activity function of neurons. This process continues until the n -1st weights from the forward path and the n+2nd weight from the backward path. Then, the images of the inputs in the throat layer of the last two BNNs are used for the calculation of two paths and for the purpose of training a hidden layer consisting of the weights of W n and W n+1. In this way, the initial values of all the weights of the deep network are obtained, and t heir learning is based on the error backpropagation algorithm. After completing the pre -training steps, the obtained weights are considered as the initial weights in the structure of the integrated deep neural network, and with the help of error backpropag ation operations, more accurate values of the network's weight matrices are obtained. III. IMPLEMENTATIONS AND RESULTS In this section, the two -way layer -by-layer pre -training method is evaluated in order to properly adjust the weights of other deep neural netw orks. Therefore, its effectiveness in learning deep classification neural networks with the purpose of recognition in several different databases is investigated. Therefore, in the following, three databases will be used. A. Database s A.I. Bosphorus database The Bosphorus database [16] includes 3D and 2D images of the faces of 150 subjects, which were collected at Begaze University, Turkey. This database includes a rich set of facial expressions, head rotation, and different types of obstructions, the collection related to different facial expressions has been used for evaluation in this article. Figure (2) These situations were randomly selected for one of the pictures of 95 people from this group. These images were used in black and white with 256 gray levels. Also, in order to reduce the load of calculations, the clarity of these images was reduced to 114*92. Figure 2. Images related to different states of the face for one of the people from Besphorus database [16] The pictures of 95 people from this group were randomly selected for examination. These images were used in black and white with 256 gray levels. Also, in order to reduce the load of calculations, the clarity of these images was reduced to 114*92. In this research, this database has been used for face recognition. Since the number of images related to different states of the face is not the same for all individuals, the images related to 12 common states among all individuals were set aside for training and other images for testing. In the end, the training and test sets included 1126 and 1443 images, respectively. People's label was also defined as 7 binary codes that represent 95 different codes that identify people's identity. A.II. CK+ database The CK+ database is an extension of the Cohen -Kund database. [17] In the Cohen -Kund database, there are sequences of images for 6 emotional states of people, which change from the neutral state to the intended state. These emotional states include happy, surprised, angry, scared, hating and upse t states. Also, for each face image, there is a vector of signs that can be used to align the faces. In this article, the faces are aligned so that the eyes are in a horizontal direction. Additional information around the images has also been removed in su ch a way that all the important details are present in the image. Finally, the dimensions of all images were changed to 50x50. In figure (3), some samples of the images of this database are given for the used emotional states. Figure 3. Samples of Cohen -Kand datab ase images for different emotional states after aligning the faces and removing additional information around the images Also, the images of the first half of each of these categories were deleted, so that the images that really represent the related emoti onal state are kept. In this article, this is the base The data has been used to recognize emotional states. Therefore, the images are divided into two groups. The training set is the images related to 96 people ( 2816 images) that are used for training the deep neural network of the recognizer. The pictures related to the remaining 10 people (351 pictures) are also used for the test. Emotional states are also defined as 4 binary codes that represent the state of people's faces. A.III. MNIST database 4 This da tabase includes English handwritten numbers from 0 to 9, the image of each number is stored as a 28x28 pixel gray image. Its training set includes 60,000 images and the test set contains 10,000 images [18]. The labels defined in this article are also 3 -way binary codes expressing numbers. B. Preparation Each image is considered a two -dimensional array of M×N light values. But since in the input of the network, this array must be presented as one dimension, each image is converted into an MN×1 vector. Also, the pixel values of each face image are normalized to the range of zero to one. C. Model The structure of the deep classification neural network for recognition in three databases is given in table (1). This network is a network with three hidden layers, whose input corresponds to the light values of the image pixels and whose output corresponds to the defined labels. This network is trained separately for each data set and its results are evaluated. To show the effectiveness of the two -way layer -by-layer pre-training method, first random initial values are set for the weights, and training is done. In the next step, using the proposed pre -training method, the initial values of the weights are determined and then the network is trained. TABLE I. PARAMETERS OF DEEP NEURAL NETWORK The number of hidden layers 3 The number of hidden layer neurons 50-200-400 The function of neurons in the hidden layer Unipolar nonlinearity The function of output layer neurons Unipolar nonlinearity Learning coefficient 0.001 Torque coefficient 0.7 D. Two-way layer -by-layer pre -training method for deep recognition network pre -training According to the method presented in section (2), this deep network is broken into two single -layer BNN and another network with one hidden layer. One of the BNNs is trained with a vector of pixels and another with a vector of binary codes defined as corresponding labels. The number of training repetitions in each pre -training stage is assumed to be 500 repetitions for face recognition including Bosp horus and CK+ database images, and 10 repetitions for handwritten recognition due to the large volume of recognition. Because learning more than the size in the pre -learning stages leads to setting more than the limit of the weights on the training judges, as a result of which the generalization power of the deep network will decrease. Then, the images of the inputs are calculated in the hidden layer of each BNN and used to train another network of a hidden layer. In this way, the initial values of all the weights of the deep network are obtained. E. Results The graph of Figure 4 shows the changes of learning error for two deep person recognition networks, one with random initial value and the other with initial value by the proposed pre - training method, on Bes phorus database. As it can be seen, with the appropriate initial setting of network weights instead of setting random values for them, the speed of learning convergence increases dramatically. Figure 4. The diagram of the changes of the learning error of the deep network of the person recognizer with two methods of randomly estimating the initial weights and using two -way layer -by-layer pre -training on the Besphorus database As it reaches its minimum value in the same initial repetitions. For the network with a random value (according to the process of reducing the learning error), the weights of the last iteration of learning and for the pre -trained network, the weights of the 40th iteration (after that, the learning error remains constant). For the recognition of people, the test data was used. TABLE II. COMPARISON OF THE ACCURACY PERCENTAGE OF PEOPLE RECOGNITION IN BOSPHORUS DATABASE BY THREE HIDDEN LAYERS NEURAL NETWORK AFTER ACCURATE ADJUSTMENT OF WEIGHTS How to estimate the initial weights of deep neural network The number of repetitions Training data Test data Randomly 5000 99.72% 86.52% Two-way layer by layer pre-education 40 100% 97.16% The results of table (2) show that by using this pre -training method, the results of the person's recognition for the judges of the exam have improved by 11%. It is Especially, in the pre - training stages, one -layer hidden networks are taught, and the duration of their training is much less in each repetition compared to three -layer hidden networks. In the diagram of figure (5) and table (3), the results on the CK+ database with the aim of recognizing the emotional state in face images are given. As can be seen in the diagram of figure (5), the network error diagram with random initial values does not de crease significantly after the 2500th repetition of learning. Therefore, the weights of this stage are used for the state recognition network. For the pre -trained network, the weights of the 90th repetition of training have been used. 5 Figure 5. The diagram of the changes of the learning error of the deep network of the person recognizer with two methods of randomly estimating the initial weights and using two -way layer -by-layer pre -training on the CK+ database The recognition results in table (3) show that the use of the bidirectional layer -by-layer pre -training method for the initial estimation of the weights of the recognition network improves the recognition accuracy percentage by 14%. TABLE III. COMPARISON OF THE ACCURACY PERCENTAGE OF PEOPLE RECOGNITION IN CK+ DATABAS E BY THREE HIDDEN LAYERS NEURAL NETWORK AFTER ACCURATE ADJUSTMENT OF WEIGHTS How to estimate the initial weights of deep neural network The number of repetitions Training data Test data Randomly 2500 99.41% 73.70 % Two-way layer -by-layer pre-education 90 99.96 % 87.29 % In the case of the MNIST handwritten digit database, due to a large number of samples, the number of repetitions of learning in the pre -learning stages was considered to be 10 repetitions. Experiments also showed that after this number, the error in the pre-training stages for this data does not improve. In the graph of figure (6), the changes of the learning error of the deep network of the digit recognizer in the stage of accurate adjustment of the weights, for two me thods of initial estimation of the weights randomly and using the pre -training of the two - dimensional layer are given. Figure 6. The diagram of the changes of the learning error of the deep network of the person recognizer with two methods of randomly estimating the initial weights and using two -way layer -by-layer pre -training on the MNIST database Table (4) shows the number recognition error for the two methods of initial estimation of network weights. For handwritten judges, no improvement in digit recognition ha s been achieved by the pre -trained network. It can be mentioned that the problem of local minima in the learning process of deep networks appears more recently in complex problems such as face and face mode, so the use of appropriate pre -training in their learning process is more effective. But for handwritten numbers, due to the simplicity of the learning task, the estimated network with initial random weights has a favorable result, as the pre - training could not create a significant improvement in the results. Of course, it should be mentioned that the result obtained from the network with pre -training is three times faster. Especially that the stages of its pre -education were short. TABLE IV. COMPARISON OF THE ACCURACY PERCENTAGE OF PEOPLE RECOGNITION IN MNIST DATA BASE BY THREE HIDDEN LAYERS NEURAL NETWORK AFTER ACCURATE ADJUSTMENT OF WEIGHTS How to estimate the initial weights of deep neural network The number of repetitions Training data Test data Randomly 600 99.88% 98.24 % Two-way layer -by-layer pre-education 200 99.92 % 98.56 % IV. CONCLUSION In some cases, networks with one or two hidden layers in the input data mapping to the output space do not work successfully. Because for more complex data with high dimension, such as face images, the sudden tr ansformation of the input data features into the output, which can be labeled by the input layers, increases the network error significantly. Therefore, there is a need for deeper structures of neural networks in order to extract data through multiple laye rs of deeper components, and then with their help, proper mapping is done to the output. This is when the number of layers of neural networks increases, the convergence of their learning faces a problem due to the increase of local minima. In order to solv e this problem, some methods for pre -learning the deep structures of neural networks in self - learning applications have been presented so far. However, in this article, the two -way layer -by-layer pre -training method for the appropriate estimation of the in itial weights of the multi - layer networks with the other function has been presented, which pre-trains the network weights in parallel in a two -way path forward and backward. This method is presented for other applications of deep neural networks, unlike p revious pre - training methods that were defined only for self -associative neural networks. This method was used for pre -training the weights of three deep neural networks for person recognition, classification of emotional states and recognition of handwrit ten digits, and it was shown that by using this pre -training method, the convergence speed of the learner can be increased. Also, the rate of recognition in the face database improves significantly, which indicates the increase in the generalization power of the network using this pre -training method.