## 3 Materials and Methods
### Continuous Relaxation  
To construct neural architecture automatically, NAS algorithms must perform operator selection in the search space. Each operator from the operator pool is denoted as \(o\). Searching the best operator for every edge \(i\) to \(j\) is the task given. Where the search performed is a discrete space and can contain multiple operators. Thus, input/output and operators in NAS are defined as Equation (1) [18].  
\[x_{j}=\sum_{i<j}o_{i,j}(x_{i}) \tag{1}\]  
DARTS employs the notion of \"mixed operators\" and represents the mixture as a set of edges which is denoted as \(\delta_{i,j}\) in Equation (2) [10].  
\[\delta_{i,j}(x_{i})=\sum_{o\in\mathcal{C}}\frac{\exp\left(\alpha_{i,j}^{o} \right)}{\sum_{o^{\prime}\in\mathcal{O}}\exp\left(\alpha_{i,j}^{o^{\prime}} \right)}o(x_{i}) \tag{2}\]  
Given the input \(x_{i}\) to each operator between \(i\) and \(j\), \(o_{i,j}\), the result is defined as their weighted sum of each output from the operator pool. DARTS adopts the Softmax function to encode these mixed operators in the discrete search space into a mixture of candidate edges in the continuous space. As shown in Equation (2) [10], the outputs of each operator in the pool are multiplied with a weight parameter \(\alpha\) after putting the \(\alpha\) to the Softmax function, and they are being summed. A mixed operator is denoted as \(\delta\).  
For every edge between nodes \(x_{i}\) to \(x_{j}\), mixed operator \(\delta\) will be calculated. Mixed operator \(\delta_{i,j}\) binds operators with architectural weight \(\alpha_{i,j}\). This mechanism parameterizes the importance of each operator by giving each operator an architectural weight \(\alpha\), enabling backpropagation. Consequently, a set of \(\alpha\) values with operators represents continuously encoded architecture.  
\[\text{Softmax}\Big{(}\alpha_{i,j}^{o}\Big{)}\approx\beta_{i,j} \tag{3}\]  
where, \(\beta^{(i,j)}=\left(\beta_{i,j}^{o_{1}},\,\beta_{i,j}^{o_{2}},\,\ldots,\, \beta_{i,j}^{o_{M}}\right)\) is a normalized \(\alpha\) vector by Softmax, and \(\alpha_{i,j}^{o}=\left(\alpha_{i,j}^{o_{1}},\alpha_{i,j}^{o_{2}},\ldots,\alpha _{i,j}^{o_{M}}\right)\) is an architectural weight vector [18].  
Figure 1: Operator pool for Neural Architecture Search (NAS).  
Here, DARTS denotes \(\alpha\) to approximate the class value by Softmax, which is in a one-hot encoded format. In the resulting formulation, \(\alpha\) represents a collection of importance values assigned to multiple operators. However, only the singular operator with the highest contribution is retained during the test process, while the remaining operators are disregarded.  
DARTS seeks to find the \(\alpha\) that minimizes \(\mathcal{L}_{val}(w^{*}(\alpha),\,\alpha)\) as follows:  
\[\mathcal{L}_{val}(w^{*}(\alpha),\,\alpha)=\mathcal{L}_{val}\left(\operatorname* {argmax}_{w}\mathcal{L}_{train}(w,\,\alpha),\,\alpha\right) \tag{4}\]  
This can be solved by bi-level optimization, where \(w\) is the lower-level variable and \(\alpha\) is the upper-level variable. \(L_{train}\) is the loss obtained when training \(w\) with fixed \(\alpha\) and \(L_{val}\) is the loss obtained when training \(\alpha\) with fixed \(w\). Training of \(w\) is done with the training dataset, and training of \(\alpha\) is done with the validation dataset. In this study, we regularize the optimization process in architecture search by annealing the Softmax function with temperature. Thereby we perform relaxation by scaling the weight parameter distribution of the operator(\(\alpha\)) so that they are close to the one-hot encoding distribution of the operators.