## 4 Experimental Results  
### TetraOpt for Hyperparameter Optimization on NATS-Bench  
We compared TetraOpt for task hyperparameter optimization (HPO) against several widely used optimization algorithms algorithms using the NATS-Bench dataset--a comprehensive benchmark for evaluation of HPO algorithms for the task of image classification--a accessed via the NATS-Bench Python package [[https://github.com/D-X-Y/NATS-Bench](https://github.com/D-X-Y/NATS-Bench)]([https://github.com/D-X-Y/NATS-Bench](https://github.com/D-X-Y/NATS-Bench)), (accessed on 4 February 2025). Our comparative baseline included the following: BayesianOptimization [4], Particle Swarm Optimization (PSO) [5], the Genetic Algorithm [7; 40], Differential Evolution (DE) [6], and Random Search. NATS-Bench consists of 15,625 different convolutional neural network (CNN) model architectures, varying in terms of hyperparameters with different pre-evaluated test accuracies and predefined model training times on different datasets (CIFAR-10, CIFAR-100 [14], and ImageNet [41]) and training durations (12, 90, and 200 epochs). Figure 4 provides a comparison of TetraOpt, PSO, Genetic Algorithm, DE, Bayesian Optimization, and Random Search across the six NATS-Bench subsets. Each subfigure corresponds to a specific dataset and the training duration of models in the subset. Specifically, we used the subsets corresponding to models trained and evaluated on CIFAR-10, CIFAR-100, and ImageNet for 12 and 90 epochs each. The search space of 15,625 CNN architectures was parameterized according to the optimization algorithm in use. During each iteration, an architecture was sampled from the search space, and its pre-computed accuracy and training time were recorded. The horizontal axis in Figure 4 represents the iteration number of the HPO process, which corresponds to the number of evaluated models with hyperparameters sampled by the respective optimization algorithms. The vertical axis shows the maximum test accuracy achieved up to the current iteration on the given dataset for a model trained for a specific number of epochs, as recorded in the NATS-Bench package. To further evaluate the computational efficiency of these algorithms, Figure 5 presents a comparison of their runtime. Each subfigure corresponds to the same dataset and training duration as in Figure 4. The horizontal axis represents the runtime of the algorithm, where the total runtime accumulates based on the training time of each sampled configuration. Figure 4: Maximum accuracy found at each iteration for different optimization algorithms across six experiments using NATS-Bench subsets. (**a**–**f**) corresponds to a specific dataset and training duration. The vertical axis displays the maximum accuracy achieved up to the corresponding runtime. Both the accuracy and training time for each configuration were obtained from the NATS-Bench package. For the implementation of the Differential Evolution (DE) algorithm, we utilized the pre-existing implementation from the SciPy library, available at [[https://github.com/scipy/scipy](https://github.com/scipy/scipy)]([https://github.com/scipy/scipy](https://github.com/scipy/scipy)), (accessed on 4 February 2025). For the Genetic Algorithm (GA), we used the 'eaSimple' algorithm from the DEAP library [7], available at [[https://github.com/deap/deap](https://github.com/deap/deap)]([https://github.com/deap/deap](https://github.com/deap/deap)) (accessed on 4 February 2025). This implementation is based on the simplest evolutionary algorithm described in Chapter 7 of [40]. The GA was configured with a population size of 50 individuals and ran for 40 generations. The crossover probability was set to 0.7, allowing genetic material to be exchanged between individuals during the crossover step, while the mutation probability was set to 0.3, introducing variability into the population to help escape local optima. For the Particle Swarm Optimization (PSO) algorithm, we set the number of particles to 20.