# NDARTS: A Differentiable Architecture Search Based on the Neumann Series
## 2 Methods
### DARTS Optimization Algorithm Based on Neumann Series Approximation (NDARTS)
#### 2.2.2 Proof
\begin{split}||\
abla_{a}L_{2}(\omega(\alpha),\alpha)-\
abla_{a }L_{2}(\omega(\alpha^{\prime}),\alpha^{\prime})||\\ =&||\
abla_{a}L_{2}(\cdot,\alpha)-\
abla_{a}L_{2}( \cdot,\alpha^{\prime})+\
abla_{a}L_{2}(\omega(\alpha),\cdot)-\
abla_{a}L_{2} (\omega(\alpha^{\prime}),\cdot)||\\ =&||\
abla_{a}L_{2}(\cdot,\alpha)-\
abla_{a}L_{2}( \cdot,\alpha^{\prime})+\
abla_{\omega}L_{2}(\omega(\alpha),\cdot)\
abla_{a} \omega(\alpha)-\
abla_{\omega}L_{2}(\omega(\alpha^{\prime}),\cdot)\
abla_{a} \omega(\alpha^{\prime})||\\ \leqslant&||\
abla_{a}L_{2}(\cdot,\alpha)-\
abla_{a }L_{2}(\cdot,\alpha^{\prime})||+||\
abla_{\omega}L_{2}(\omega(\alpha),\cdot) \
abla_{a}\omega(\alpha)-\
abla_{\omega}L_{2}(\omega(\alpha^{\prime}),\cdot) \
abla_{a}\omega(\alpha^{\prime})||.\end{split}  \tag{17}\]  
Among them, the left half on the right side of the equation,  
\[||\
abla_{a}L_{2}(\cdot,\alpha)-\
abla_{a}L_{2}(\cdot,\alpha^{\prime})|| \leqslant L_{2}^{\
abla_{a}}||\alpha-\alpha^{\prime}||, \tag{18}\]  
the right half,  
\[ \begin{split}||\
abla_{\omega}L_{2}(\omega(\alpha),\cdot)\
abla_ {a}\omega(\alpha)-\
abla_{\omega}L_{2}(\omega(\alpha^{\prime}),\cdot)\
abla_ {a}\omega(\alpha^{\prime})||\\ =&||\
abla_{\omega}L_{2}(\omega(\alpha),\cdot)\
abla_ {a}\omega(\alpha)-\
abla_{\omega}L_{2}(\omega(\alpha^{\prime}),\cdot)\
abla_ {a}\omega(\alpha)\\ &-\
abla_{\omega}L_{2}(\omega(\alpha^{\prime}),\cdot)\
abla_{a} \omega(\alpha^{\prime})+\
abla_{\omega}L_{2}(\omega(\alpha^{\prime}),\cdot) \
abla_{a}\omega(\alpha)||\\ \leqslant&||\
abla_{\omega}L_{2}(\omega(\alpha^{ \prime}),\cdot)-\
abla_{\omega}L_{2}(\omega(\alpha^{\prime})||||\
abla_{a} \omega(\alpha)||\\ &+||\
abla_{\omega}L_{2}(\omega(\alpha^{\prime}),\cdot)||||\
abla_ {a}\omega(\alpha)-\
abla_{a}\omega(\alpha^{\prime})||.\end{split}  \tag{19}\]  
According to assumptions (1), (3), and (4), we obtain  
\[||\
abla_{\omega}L_{2}(\omega(\alpha)-\
abla_{\omega}L_{2}(\omega(\alpha^{ \prime})||\leqslant L_{2}^{\omega}||\omega(\alpha)-\omega(\alpha^{\prime})||, \tag{20}\]  
and  
\[||\omega(\alpha)-\omega(\alpha^{\prime})||\leqslant L_{\omega}||\alpha-\alpha^{ \prime}||,||\
abla_{a}\omega(\alpha)-\
abla_{a}\omega(\alpha^{\prime})|| \leqslant L_{\
abla_{a}\omega}||\alpha-\alpha^{\prime}||. \tag{21}\]  
According to assumptions (1) and (3), we know \(\
abla_{\omega}L_{2}(\omega(\alpha^{\prime}),\cdot)\), \(\
abla_{a}\omega(\alpha)\) are bounded and \(||\
abla_{\omega}L_{2}(\omega(\alpha^{\prime}),\cdot)||\leqslant L_{2}^{\omega},|| \
abla_{a}\omega(\alpha)||\leqslant L_{\omega}\). Consequently,\[||\
abla_{\omega}L_{2}(\omega(\alpha)\
abla_{\alpha}\omega(\alpha),\cdot)- \
abla_{\omega}L_{2}(\omega(\alpha^{\prime})\
abla_{\alpha}\omega(\alpha^{\prime }),\cdot)||\leqslant L_{2}^{\
abla_{\omega}}L_{\omega}^{2}||\alpha-\alpha^{ \prime}||+L_{2}^{\omega}L_{\
abla_{\alpha}\omega}||\alpha-\alpha^{\prime}||. \tag{22}\]  
In summary,  
\[||\
abla_{\alpha}L_{2}(\omega(\alpha),\alpha)-\
abla_{\alpha}L_{2}(\omega( \alpha^{\prime}),\alpha^{\prime})||=(L_{2}^{\
abla_{\alpha}}+L_{2}^{\
abla_{ \omega}}L_{\omega}^{2}+L_{2}^{\omega}L_{\
abla_{\alpha}\omega})||\alpha-\alpha^ {\prime}||. \tag{23}\]  
\(\square\)  
**Proof of Theorem 1. **  
\[ \begin{split}& E[L_{2}(\omega^{*}(\alpha_{m+1}),\alpha_{m+1})| \alpha_{m}]\\ =& E[L_{2}(\omega^{*}(\alpha_{m}+(\alpha_{m+1}- \alpha_{m})),\alpha_{m}+(\alpha_{m+1}-\alpha_{m}))|\alpha_{m}]\\ \leqslant& E[L_{2}(\omega^{*}(\alpha_{m}),\alpha_{m} )|\alpha_{m}]+E[\langle\
abla_{\alpha}L_{2}(\omega^{*}(\alpha_{m}),\alpha_{m} ),\alpha_{m+1}-\alpha_{m}\rangle|\alpha_{m}]+\frac{L_{\
abla_{\alpha}L_{2}}}{2 }E\Big{[}||\alpha_{m+1}-\alpha_{m}||^{2}\Big{]}\\ =& L_{2}(\omega^{*}(\alpha_{m}),\alpha_{m})+\big{\langle} E[\
abla_{\alpha}L_{2}(\omega^{*}(\alpha_{m}),\alpha_{m})],-\gamma_{\alpha_{m}}E \big{[}\
abla_{\alpha}L_{2}^{\prime}(\omega^{*}(\alpha_{m}),\alpha_{m})| \alpha_{m}]\big{\rangle}\\ &+\frac{L_{\
abla_{\alpha}L_{2}}}{2}\gamma_{\alpha_{m}}^{2}E\Big{[} ||\
abla_{\alpha}L_{2}^{\prime}(\omega^{*}(\alpha_{m}),\alpha_{m})||^{2}\Big{]}.\end{split}