# Neural Architecture Search for a Highly Efficient Network with Random Skip Connections
## 3 A Highly Efficient Network with Random Skip Connections
### Initialization of the Auto-Generated Network  
The networks are trained using a gradient backpropagation algorithm to effectively capture the dependencies that [PERSON] et al. [11] proposed, which use a chrono-initializer in LSTM and GRU cell structures. Here, we adopted the initializer in our designed cell and demonstrate two aspects of theoretical advantages: capturing dependencies and unhindered gradient propagation. We took the auto-generated network as an example, where the coefficients were initialized as \(\omega_{1}=1\) such that the cell structure became:  
\[c_{t}=f_{t}\odot c_{t-1}+i_{t}\odot tanh(W_{c}x_{t}+b_{c}), \tag{4}\]which was then considered to be a leaky RNN [12]:  
\[c_{t}=(1-\alpha)\odot c_{t-1}+\alpha\odot tanh(W_{c}x_{t}+b_{c}), \tag{5}\]  
where \(1-\alpha\) and \(\alpha\) represent combination coefficents and should be learned with \(\mathrm{f_{t}}\) and \(\mathrm{i_{t}}\), respectively. By taking the first-order Taylor expansion [4], we have:  
\[\frac{dc(t)}{dt}=-\alpha\odot c(t)+\alpha\odot tanh(W_{c}x_{t}+b_{c}), \tag{6}\]  
with the inputs \(\mathrm{x_{t}}\) and hidden vectors centered around zero, and \(\mathrm{b_{c}}\) initialized to be 0, we have:  
\[\begin{array}{c}\frac{dc(t)}{dt}=-\alpha\odot c(t)\\ \int_{t_{0}}^{t}\frac{c(t)}{c(t)}dc(t)=-\alpha\int_{t_{0}}^{t}dt\\ c(t)=c(t_{0})exp(-\alpha(t-t_{0})),\end{array} \tag{7}\]  
\(\mathrm{c}(t)\) will decrease to \(\mathrm{e^{-1}c(t)}\) over a time of \(\frac{1}{\alpha}\), which is called the forgetting time and is symbolized as T. Then, the forget and input gates in the auto-generated network should approximate \(1-\frac{1}{T}\) and \(\frac{1}{T}\), respectively. As T cannot be known previously, we set:  
\[b_{f}\sim\log(\mu([1,T_{max}-1],\ b_{i}=-b_{f} \tag{8}\]  
where \(\mu\) represents a uniform distribution. Under the conditions of zero-centered inputs and hidden layers, we have \(\sigma(W_{f}x_{t}+b_{f})=\sigma(b_{f})\approx 1-\frac{1}{T}\) and \(\sigma(U_{i}h_{t-1}+b_{i})=\sigma(b_{i})\approx\frac{1}{T}\). This means that the forgetting time of our network can lie in approximately the same range as the dependencies of the sequential data.  
Moreover, the gradient used for updating one parameter can be formulated as:  
\[\begin{array}{c}\frac{\partial I}{\partial U_{i}}=\frac{\partial I}{ \partial c_{T}}\frac{\partial c_{T}}{\partial U_{i}}+\frac{\partial I}{ \partial c_{T}}\frac{\partial c_{T}}{\partial c_{T-1}}\frac{\partial c_{T-1}} {\partial U_{i}}+\cdots+\frac{\partial I}{\partial c_{T}}\prod_{k=2}^{T}\left[ \frac{\partial c_{k}}{\partial c_{k-1}}\right]\frac{\partial c_{1}}{\partial U _{i}}\\ \frac{\partial c_{T}}{\partial c_{t-1}}=\sigma(W_{f}x_{t}+b_{f})+U_{i}\sigma^{ \prime}(U_{i}h_{t-1}+b_{i})\odot tanh(W_{c}x_{t}+b_{c}),\end{array} \tag{9}\]  
where \(J\) symbolizes the objective function, \(\sigma(W_{f}x_{t}+b_{f})\) approaches 1 when the dependency length \(T\) is large, and other items will approximate 0. Thus, we have \(\frac{\partial c_{k}}{\partial c_{k-1}}\approx 1\). In this case, the vanishing gradient can be prevented remarkably well as the multiplication effect is dramatically eliminated; meanwhile, the long-term gradient propagation can be retained.  
Figure 1: The framework of our proposed model. RNN: Recurrent neural network.