# NDARTS: A Differentiable Architecture Search Based on the Neumann Series
## 2 Methods
### DARTS Optimization Algorithm Based on Neumann Series Approximation (NDARTS)  
#### 2.2.1 Ndarts  
We perform a one-step expansion of \(\
abla_{a}L_{2}\) based on the Implicit Function Theorem (we denote \(L_{1}\) and \(L_{2}\) the training loss \(L_{train}\), and the validation loss \(L_{val}\), respectively). At present, there has been some research work in related fields, and [PERSON] et al. (2016) used this method to optimize neural network architecture search algorithms.  
When \(\omega^{*}\) is the optimal weight of the model achieved by the training set, \(\frac{\partial L_{1}(\omega^{*},\alpha)}{\partial\omega}=0\), thus,  
\[\frac{\partial}{\partial\alpha}(\frac{\partial L_{1}(\omega^{*}(\alpha), \alpha)}{\partial\omega})=0,\]  
\[\frac{\partial^{2}L_{1}}{\partial\alpha\partial\omega}+\frac{\partial^{2}L_{ 1}}{\partial\omega\partial\omega}\frac{\partial\omega^{*}(\alpha)}{\partial \alpha}=0, \tag{5}\]  
\[\frac{\partial\omega^{*}(\alpha)}{\partial\alpha}=-\left[\frac{\partial^{2}L_ {1}}{\partial\omega\partial\omega}\right]^{-1}\frac{\partial^{2}L_{1}}{ \partial\alpha\partial\omega}.\]  
Based on this consequence, sub-gradient \(\
abla_{a}L_{2}\) can be represented as:  
\[\
abla_{a}L_{2}=\frac{\partial L_{2}}{\partial\alpha}+\frac{\partial L_{2}}{ \partial\omega}\frac{\partial\omega}{\partial\alpha}=\frac{\partial L_{2}}{ \partial\alpha}-\frac{\partial L_{2}}{\partial\omega}\left[\frac{\partial^{2} L_{1}}{\partial\omega\partial\omega}\right]^{-1}\frac{\partial^{2}L_{1}}{ \partial\alpha\partial\omega}. \tag{6}\]  
According to Neumann series, for matrix A, when \(||I-A||<1\), we have \(A^{-1}=\sum\limits_{k=0}^{\infty}(I-A)^{k}\).  
When \(\gamma<\frac{1}{L_{1}^{\
abla\omega}}\), according to \(\frac{\partial^{2}L_{1}}{\partial\omega\partial\omega}<L_{1}^{\
abla\omega}\), we have \(||I-\gamma\frac{\partial^{2}L_{1}}{\partial\omega\partial\omega}||<1\).  
So, we can formulate the sub-gradient:\[\
abla_{a}L_{2} =\frac{\partial L_{2}}{\partial\alpha}-\frac{\partial L_{2}}{ \partial\omega}\gamma\bigg{[}I-(I-\gamma\frac{\partial^{2}L_{1}}{\partial \omega\partial\omega})\bigg{]}^{-1}\frac{\partial^{2}L_{1}}{\partial\alpha \partial\omega} \tag{7}\] \[=\frac{\partial L_{2}}{\partial\alpha}-\gamma\frac{\partial L_{2} }{\partial\omega}\sum\limits_{j=0}^{\infty}(I-\gamma\frac{\partial^{2}L_{1}}{ \partial\omega\partial\omega})^{j}\frac{\partial^{2}L_{1}}{\partial\alpha \partial\omega}.\]  
For this result, take the first K terms as an approximation, we can derive:  
\[\
abla_{a}L_{2}^{\prime}=\frac{\partial L_{2}}{\partial\alpha}-\gamma\frac{ \partial L_{2}}{\partial\omega}\sum\limits_{j=0}^{K}(I-\gamma\frac{\partial ^{2}L_{1}}{\partial\omega\partial\omega})^{j}\frac{\partial^{2}L_{1}}{ \partial\alpha\partial\omega}. \tag{8}\]  
In summary, this article proposes the NDARTS algorithm, which involves the following steps:  
1. Create a mixed operation \(o(i,j)\) parameterized by \(\alpha(i,j)\) for each edge \((i,j)\), initialize the hyper-net weight \(\omega\), and operate weight \(\alpha\);
2. Train on the training dataset \(D_{train}\), using \(\omega=\omega-\gamma\
abla_{\omega}L_{1}(\omega,\alpha)\) to update \(\omega\);
3. Train on the validation dataset \(D_{val}\), using \(\alpha=\alpha-\gamma_{a}\
abla_{a}L_{2}^{\prime}(\omega(\alpha),\alpha)\) to update \(\alpha\);
4. Stop when the termination condition is met, otherwise return to step 2.