# NDARTS: A Differentiable Architecture Search Based on the Neumann Series
## 2 Methods  
### Overview: DARTS  
We first briefly review DARTS, the basis of our proposed architecture search method. Following the NASNet searches space [12], DARTS search for a computation cell as a building block of the final architecture, and the overall architecture of the networks is obtained by stacking two types of cells: a Normal cell that returns a feature map of the same dimension and a Reduction cell that reduces the size of input feature maps by half. A cell is a directed acyclic graph (DAG) consisting of an ordered sequence of N nodes. Each node \(x_{i}\) is a latent representation (e.g., a feature map in convolutional networks). The first two nodes \(x_{0}\) and \(x_{1}\) in each unit are defined as input nodes, and they receive the outputs of the precursor cells as input. The node \(x_{N}\) is the output of the current cell, which concatenates the outputs of all intermediate nodes \(x_{N-1}=concat(x_{0},\ldots,x_{N-2})\). A directed edge \(e^{(i,j)},0<j<i<N-1\), indicates the existence of an operation to perform a transformation (convolution, pooling, etc.) on the feature representation. We use \(O\) to represent a set of candidate operations (e.g., convolution, max pooling, zero), where each operation represents some function \(o\) to be applied to \(x_{i}\).  
Each intermediate node is computed based on all of its predecessors:  
\[x^{(ij)}=\sum_{i<j}o^{(ij)}(x^{(j)}). \tag{1}\]To make the search space continuous, DARTS relaxes the categorical choice of a particular operation to a softmax over all possible operations:  
\[\delta^{(i,j)}(X)=\sum\limits_{o\in O}\frac{exp(\alpha_{o}^{(i,j)})}{\sum\
olimits _{o^{\prime}\in O}exp(\alpha_{o^{\prime}}^{(i,j)})}, \tag{2}\]  
where the operation mixing weights for a pair of nodes \((i,j)\) are parameterized by a vector \(\alpha^{(i,j)}\) of dimension \(|O|\). It represents that there is a mixed operation parameterized by \(\alpha^{(i,j)}\) instead of selecting a certain operation on each edge.  
After relaxation, the task of architecture search then reduces to learning a set of continuous variables \(\alpha=\alpha^{(i,j)}\). DARTS used a bi-level optimization to jointly learn the architecture parameters \(\alpha\) and the weights \(\omega\) within all the mixed operations (e.g., weights of the convolution filters).  
\[min_{a}L_{val}(\omega*(\alpha),\alpha), \tag{3}\]  
\[s.t.\omega^{*}(\alpha)=argmin_{\omega}L_{train}(\omega,\alpha). \tag{4}\]  
DARTS alternately optimizes the architecture \(\alpha\) and the weights \(\omega\) on the validation set and the training set, respectively.  
The process of DARTS is as follows:  
1. Create a mixed operation \(o^{(i,j)}\) parameterized by \(\alpha^{(i,j)}\) for each edge \((i,j)\), initialize the hyper-net weight \(\omega\), and operate weight \(\alpha\);
2. Train on the training dataset \(D_{train}\), update \(\omega\) by \(\omega=\omega-\gamma\
abla_{\omega}L_{1}(\omega,\alpha)\);
3. Train on the validation dataset \(D_{val}\), update \(\alpha\) by \(\alpha=\alpha-\gamma_{\alpha}\
abla_{\alpha}L_{2}^{\prime}(\omega(\alpha), \alpha)\);
4. Stop when the termination condition is met, otherwise return to step 2.