# NDARTS: A Differentiable Architecture Search Based on the Neumann Series
## 2 Methods
### DARTS Optimization Algorithm Based on Neumann Series Approximation (NDARTS)
#### 2.2.2 Proof
 \tag{24}\]  
where, according to Lemma 2, we know \(L_{\
abla_{\alpha}L_{2}}\) exists and is bounded. Let \(e_{m}=\
abla_{\alpha}L_{2}(\omega^{*}(\alpha_{m}),\alpha_{m})-\
abla_{\alpha} L_{2}^{\prime}(\omega^{*}(\alpha_{m}),\alpha_{m})\), hence  
\[E[L_{2}(\omega^{*}(\alpha_{m}),\alpha_{m})]=E\big{[}L_{2}^{\prime}(\omega^{*}( \alpha_{m}),\alpha_{m})+e_{m}\big{]}\leqslant E\big{[}L_{2}^{\prime}(\omega^ {*}(\alpha_{m}),\alpha_{m})\big{]}+E[e_{m}], \tag{25}\]  
in that way,  
\[ \begin{split} E[L_{2}(\omega^{*}(\alpha_{m+1}),\alpha_{m+1})| \alpha_{m}]&\leqslant E\big{[}L_{2}^{\prime}(\omega^{*}(\alpha_{m }),\alpha_{m})\big{]}-\gamma_{\alpha_{m}}E\Big{[}||\
abla_{\alpha}L_{2}^{ \prime}(\omega^{*}(\alpha_{m}),\alpha_{m})||^{2}\Big{]}\\ &-\gamma_{\alpha_{m}}E\big{\langle}e_{m},\
abla_{\alpha}L_{2}^{ \prime}(\omega^{*}(\alpha_{m}),\alpha_{m})\big{\rangle}+\\ &\frac{L_{\
abla_{\alpha}L_{2}}}{2}\gamma_{\alpha_{m}}^{2}E\Big{[} ||\
abla_{\alpha}L_{2}^{\prime}(\omega^{*}(\alpha_{m}),\alpha_{m})||^{2}\Big{]}.\end{split}  \tag{26}\]  
According to Lemma 1,  
\[||e_{m}||\leqslant C_{L_{1}^{\omega}}C_{L_{2}^{\omega}}\frac{1}{\mu}(1-\gamma \mu)^{K+1}, \tag{27}\]  
therefore, for any \(\
abla_{\alpha}L_{2}^{\prime}(\omega^{*}(\alpha_{m}),\alpha_{m})\),  
\[ \begin{split}\big{\langle}e_{m},\
abla_{\alpha}L_{2}^{\prime}( \omega^{*}(\alpha_{m}),\alpha_{m})\big{\rangle}&\geq-C_{L_{1}^{ \omega}}C_{L_{2}^{\omega}}\frac{1}{\mu}(1-\gamma\mu)^{K}||\
abla_{\alpha}L_{2}^{ \prime}||\\ &=-\frac{C_{L_{1}^{\omega}}C_{L_{2}^{\omega}}\frac{1}{\mu}(1-\gamma \mu)^{K}}{\mu||\
abla_{\alpha}L_{2}^{\prime}||}||\
abla_{\alpha}L_{2}^{\prime}|| ^{2}\\ &=-P||\
abla_{\alpha}L_{2}^{\prime}||^{2}.\end{split}  \tag{28}\]  
Let \(P=\frac{C_{L_{1}^{\omega}}C_{L_{2}^{\omega}}\frac{1}{\mu}(1-\gamma\mu)^{K}}{ \mu||\
abla_{\alpha}L_{2}^{\prime}||}\), Equation (22) can be rewritten as:  
\[ \begin{split}& E[L_{2}(\omega^{*}(\alpha_{m+1}),\alpha_{m+1})] \leqslant E\big{[}L_{2}^{\prime}(\omega^{*}(\alpha_{m}),\alpha_{m})\big{]}-\\ &\gamma_{\alpha_{m}}(1-P)E\Big{[}||\
abla_{\alpha}L_{2}^{\prime} ||^{2}\Big{]}+\frac{L_{\
abla_{\alpha}L_{2}}}{2}\gamma_{\alpha_{m}}^{2}E\Big{[} ||\
abla_{\alpha}L_{2}^{\prime}||^{2}\Big{]}\\ &\leqslant E\big{[}L_{2}^{\prime}(\omega^{*}(\alpha_{m}),\alpha_{m })\big{]}-\gamma_{\alpha_{m}}\Big{[}(1-P)-\frac{L_{\
abla_{\alpha}L_{2}}}{2} \gamma_{\alpha_{m}}\Big{]}E\Big{[}||\
abla_{\alpha}L_{2}^{\prime}||^{2}\Big{]}.\end{split}  \tag{29}\]In the equation, select a small learning rate \(\gamma_{\alpha_{m}}<\frac{1-P}{\frac{\gamma_{\alpha_{1}\gamma_{2}}}{2}}\), can result in  
\[(1-P)-\frac{L_{\
abla_{\alpha}L_{2}}}{2}\gamma_{\alpha_{m}}>0. \tag{30}\]  
Since the learning rate is positive, \(1-P\) should also be a positive number. This can be achieved by adjusting \(\gamma\) and \(K\) to make \(\frac{C_{L_{2}\gamma^{\alpha}}\frac{1}{2}(1-\gamma_{\alpha})^{K+1}}{\mu|| \
abla_{\alpha}L_{2}||}<1\). At this point, it can be seen from the recursive formula that as \(\alpha_{m}\) iterates, \(L_{2}\) will decrease and the loss function \(L_{2}\) is bounded, so \(L_{2}\) is convergent. The difference in recursive equation  
\[ \begin{split}& E[L_{2}(\omega^{*}(\alpha_{m}),\alpha_{m})]-E[L_{2}( \omega^{*}(\alpha_{m+1}),\alpha_{m+1})]\\ &\geq\gamma_{\alpha_{m}}\Big{[}(1-P)-\frac{L_{\
abla_{\alpha}L_ {2}}}{2}\gamma_{\alpha_{m}}\Big{]}E\Big{[}||\
abla_{\alpha}L_{2}^{\prime}( \omega^{*}(\alpha_{m}),\alpha_{m})||^{2}\Big{]},\end{split}  \tag{31}\]  
denote \(c_{m}=\gamma_{\alpha_{m}}\Big{[}(1-P)-\frac{L_{\
abla_{\alpha}L_{2}}}{2}\gamma _{\alpha_{m}}\Big{]}\), we can derive  
\[ \begin{split}& E[L_{2}(\omega^{*}(\alpha_{m}),\alpha_{0})]-E[L_{2}( \omega^{*}(\alpha_{m}),\alpha_{m})]\\ &=\sum_{i=1}^{m}\{E[L_{2}(\omega^{*}(\alpha_{i-1}),\alpha_{i-1}) ]-E[L_{2}(\omega^{*}(\alpha_{i}),\alpha_{i})]\}\\ &\geq\sum_{i=0}^{m-1}c_{i}E[L_{2}(\omega^{*}(\alpha_{i}),\alpha _{i})].\end{split}