SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Method	CIFAR-10	CIFAR-100	ImageNet-16-120	CIFARTile	AddNIST
cell-based	hierarchical	cell-based	hierarchical	cell-based	hierarchical	cell-based	hierarchical	cell-based	hierarchical
Best ResNet [100]	6.49 ± 0.24 (32)	27.1 ± 0.67 (110)	53.67 ± 0.18 (56)	57.8 ± 0.57 (18)	7.78 ± 0.05 (34)
Best EfficientNet [1]	11.73 ± 0.1 (B0)	35.17 ± 0.42 (B6)	77.73 ± 0.29 (B0)	61.01 ± 0.62 (B0)	13.24 ± 0.58 (B1)
DARTS [6]	45.7 ± 0.00†	-	61.03 ± 0.00†	-	81.59 ± 0.00†	-	76.1 ± 0.07	-	73.21 ± 0.13	-
DrNAS [45]	5.64 ± 0.00†	-	26.49 ± 0.00†	-	53.66 ± 0.00†	-	54.77 ± 0.68	-	10.18 ± 0.74	-
DrNAS (progressive) [45]	-	-	-	-	-	-	55.15 ± 2.3	-	9.92 ± 0.63	-
NAS-Bench-201 oracle*	5.63	-	26.49	-	52.69	-	-	-	-	-
RS	6.39 ± 0.18	6.77 ± 0.1	28.75 ± 0.57	29.49 ± 0.57	54.83 ± 0.78	54.7 ± 0.82	52.72 ± 0.45	40.93 ± 0.81	7.82 ± 0.36	8.05 ± 0.29
RE [9, 12]	5.76 ± 0.17	6.88 ± 0.24	27.68 ± 0.55	30.0 ± 0.32	53.92 ± 0.6	55.39 ± 0.54	52.79 ± 0.59	40.99 ± 2.89	7.69 ± 0.35	7.56 ± 0.69
AREA [65]	-	6.37 ± 0.03	-	29.35 ± 0.79	-	54.24 ± 0.66	-	38.11 ± 2.39	-	7.4 ± 0.23
NASWOT (N=10) [65]	6.55 ± 0.1	8.18 ± 0.46	29.35 ± 0.53	31.73 ± 0.96	56.8 ± 1.35	58.66 ± 0.29	41.83 ± 2.29	49.46 ± 2.95	10.11 ± 0.69	11.81 ± 1.55
NASWOT (N=100) [65]	6.59 ± 0.17	8.56 ± 0.87	28.91 ± 0.25	31.65 ± 1.95	55.99 ± 1.3	58.47 ± 2.74	41.63 ± 1.02	43.31 ± 2.0	10.75 ± 0.23	14.47 ± 1.44
NASWOT (N=1000) [65]	6.68 ± 0.12	8.26 ± 0.38	29.37 ± 0.17	31.66 ± 0.72	58.93 ± 2.92	58.33 ± 0.91	39.61 ± 1.12	45.66 ± 1.29	10.68 ± 0.27	13.57 ± 1.89
NASWOT (N=10000) [65]	6.98 ± 0.43	8.4 ± 0.52	29.95 ± 0.42	32.09 ± 1.61	54.2 ± 0.49	57.58 ± 1.53	39.9 ± 1.2	42.45 ± 0.67	10.72 ± 0.53	14.82 ± 0.66
NASBOT [18]	-	6.68 ± 0.15	-	29.25 ± 0.55	-	54.61 ± 0.28	-	36.47 ± 2.25	-	7.45 ± 0.23
NASBOWL [48]	5.68 ± 0.11	6.98 ± 0.5	27.66 ± 0.18	28.7 ± 0.64	53.67 ± 0.39	53.47 ± 0.86	52.81 ± 0.27	35.75 ± 1.58	7.86 ± 0.41	8.2 ± 0.37
BOHNAS	5.68 ± 0.11	6.0 ± 0.16	27.66 ± 0.18	27.57 ± 0.46	53.67 ± 0.39	53.43 ± 0.61	52.81 ± 0.27	32.28 ± 2.39	7.86 ± 0.41	6.09 ± 0.34
BOHNAS (best)	5.64 ± 0.14	5.65 ± 0.09	27.03 ± 0.23	27.63 ± 0.2	53.54 ± 0.43	52.78 ± 0.23	53.18 ± 0.91	30.33 ± 0.77	8.04 ± 0.45	6.33 ± 0.59
BOHNAS°	5.37 ± 0.12	25.28 ± 0.19	49.47 ± 0.73	37.44 ± 1.62	4.27 ± 0.23
BOHNAS° (best)	5.02 ± 0.1	25.41 ± 0.28	48.16 ± 0.17	39.2 ± 2.24	4.57 ± 0.38
(a) Test error vs. number of parameters (#params (MB)).
(b) Test error vs. number of FLOPs (#FLOPs (M)).
Figure 9: Test error vs. number of parameters (a) and FLOPs (b) for each architecture candidate in the cell-based search space (blue dots), the best cell (orange cross), and our best found architecture in the hierarchical NAS-Bench-201 search space (green star).
zero, id)), Residual2(Cell(Sequential1(Sequential3(hardswish, conv1x1, instance)), Sequential1(Cell(Sequential3(hardswish, dconv3x3, batch)), Sequential1(Sequential3(mish, dconv3x3, instance)), Sequential1(Sequential3(relu, conv1x1, batch)), id, id), down, down), Residual2(Cell(Sequential1(Sequential3(hardswish, conv1x1, layer)), Sequential1(Sequential3(hardswish, dconv3x3, batch)), Sequential1(Sequential3(relu, conv1x1, batch)), Sequential1(Sequential3(hardswish, conv3x3, layer)), id, avg_pool), down, down)), Residual3(Cell(id, zero, Sequential1(Sequential3(hardswish, conv1x1, layer)), Sequential1(Sequential3(hardswish, conv3x3, instance))), Cell(Sequential1(Sequential3(relu, dconv3x3, layer)), id, avg_pool, Sequential1(Sequential3(relu, dconv3x3, layer)), id, zero), Cell(zero, Sequential1(Sequential3(relu, conv1x1, layer)), id, Sequential1(Sequential3(hardswish, conv1x1, instance)), Sequential1(Sequential3(hardswish, conv1x1, layer)), Sequential1(Sequential3(hardswish, dconv3x3, layer))), Residual2(Cell(id, zero, Sequential1(Sequential3(mish, conv1x1, layer)), Sequential1(Sequential3(mish, conv3x3, layer)), Sequential1(Sequential3(hardswish, dconv3x3, batch)), zero), Cell(id, zero, id, Sequential1(Sequential3(relu, conv3x3, batch)), id, id), Cell(Sequential1(Sequential3(hardswish, conv3x3, layer)), Sequential1(Sequential3(hardswish, conv1x1, layer)), Sequential1(Sequential3(hardswish, conv1x1, batch)), Sequential1(Sequential3(hardswish, dconv3x3, batch)), Sequential1(Sequential3(hardswish, conv1x1, batch)), id, Sequential1(Sequential3(hardswish, conv1x1, batch))), Residual2(Cell(Sequential1(Sequential3(hardswish, conv1x1, batch)), Sequential1(Sequential3(hardswish, dconv3x3, batch))), Sequential1(Sequential3(relu, conv1x1, batch)), id, id), down, down), Residual2(Cell(Sequential1(Sequential3(hardswish, conv1x1, layer)), Sequential1(Sequential3(hardswish, dconv3x3, batch))), Sequential1(Sequential3(hardswish, dconv3x3, layer)), Sequential1(Sequential3(hardswish, conv1x1, batch)), id), Cell(Sequential1(Sequential3(relu, conv1x1, instance)), Sequential1(Sequential3(relu, conv1x1, layer)), Sequential1(Sequential3(relu, conv1x1, layer)), Sequential1(Sequential3(hardswish, conv1x1, batch))), Sequential1(Sequential3(hardswish, conv1x1, batch))), Sequential1(Sequential3(hardswish, conv1x1, batch))), Sequential1(Sequential3(hardswish, conv1x1, batch))), Sequential1(Sequential3(hardswish, conv1x1, batch))), Sequential1(Sequential3(hardswish, conv1x1, batch))), Sequential1(Sequential3(hardswish, conv1x1), id), Sequential2(Cell(Sequential1(Sequential3(hardswish, conv3x3, batch))), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid), Sequential2(cellsid)
layer)), Sequential1(Sequential3relu, conv1x1, layer)), Sequential1(Sequential3relu, conv3x3, layer)), id), Cell(Sequential1(Sequential3relu, conv1x1, batch)), id, id, Sequential1(Sequential3relu, conv1x1, instance)), Sequential1(Sequential3relu, conv1x1, instance)), Sequential1(Sequential3relu, dconv3x3, layer)), zero, Sequential1(Sequential3(hardswish, dconv3x3, layer))), Cell(Sequential1(Sequential3relu, dconv3x3, instance)), zero, zero, id, zero, id)), Sequential4(Residual2(Cell(Sequential1(Sequential3(hardswish, conv3x3, layer)), Sequential1(Sequential3relu, conv3x3, layer)), Sequential1(Sequential3relu, conv1x1, layer)), Sequential1(Sequential3relu, conv1x1, layer)), Sequential1(Sequential3relu, conv1x1, batch)), id, id, Sequential1(Sequential3relu, conv1x1, layer)), Sequential1(Sequential3relu, conv1x1, instance)), Sequential1(Sequential3relu, conv1x1, layer)), zero, Sequential1(Sequential3(hardswish, conv3x3, batch))), Residual2(Cell(Sequential1(Sequential3(hardswish, conv3x3, layer)), Sequential1(Sequential3relu, conv3x3, layer)), Sequential1(Sequential3relu, conv1x1, layer)), Sequential1(Sequential3relu, conv1x1, instance)), Sequential1(Sequential3relu, conv1x1, layer)), Sequential1(Sequential3relu, conv1x1, batch)), id, Cell(Sequential1(Sequential3relu, conv1x1, batch)), id, id, Sequential1(Sequential3(hardswish, conv3x3, layer))), Sequential2(Cell(Sequential1(Sequential3(hardswish, conv3x3, layer)), Sequential1(Sequential3relu, conv3x3, layer)), Sequential1(Sequential3relu, conv1x1, layer)), Sequential1(Sequential3relu, conv1x1, instance)), Sequential1(Sequential3relu, conv1x1, layer)), Sequential1(Sequential3relu, conv1x1, batch)), id, Cell(Sequential1(Sequential3(hardswish, conv3x3, layer)), Sequential1(Sequential3relu, conv3x3, layer)), Sequential1(Sequential3relu, conv1x1, instance)), Sequential1(Sequential3relu, conv3x3, layer)), id), Cell(Sequential1(Sequential3(hardswish, conv3x3, layer)), Sequential1(Sequential3relu, conv1x1, instance)), Sequential1(Sequential3relu, conv1x1, batch)), id, Cell(id, Sequential1(Sequential3(hardswish, conv1x1, layer)), Sequential1(Sequential3(hardswish
CIFAR-100 (mean test error  $27.63\%$ , #params 0.962 MB, FLOPs 115.243 M):
Sequential3(Residual3(Sequential3(CellSequential3(mish, conv3x3, layer), avg_pool, Sequential3(hardswish, conv1x1, instance), zero, Sequential3(mish, conv3x3, batch), zero), Cell(Sequential3(hardswish, dconv3x3, batch), zero, Sequential3(hardswish, dconv3x3, batch), sequential3(relu, dconv3x3, batch), id, id), Cell(Sequential3(mish, conv3x3, batch), zero, id, zero, Sequential3(hardswish, dconv3x3, batch), zero, zero, Sequential3(mish, conv3x3, batch)), sequential3(mish, conv3x3, batch), sequential3(hardswish, conv3x3, batch), sequential3(hardswish, conv3x3, instance), id, id, avg_pool), Cell(id, id, zero, zero, id, id)), Residual3(Sequential3(CellSequential3(mish, conv3x3, layer), id, Sequential3(hardswish, dconv3x3, batch), Sequential3(mish, conv3x3, instance), Sequential3(mish, conv3x3, batch)), Cell(Sequential3(hardswish, conv1x1, layer), id, Sequential3(hardswish, dconv3x3, batch), Sequential3(relu, conv3x3, layer), id, id), Cell(Sequential3(relu, conv3x3, instance), zero, id, zero, Sequential3(mish, conv3x3, batch), avg_pool)), Sequential3(Cell(zero, id, Sequential3(hardswish, conv1x1, layer), Sequential3(mish, conv3x3, instance), Sequential3(mish, conv3x3, instance), Sequential3(hardswish, dconv3x3, batch), Sequential3(relu, conv3x3, batch), id, id), Cell(Sequential3(relu, conv3x3, instance), Sequential3(hardswish, dconv3x3, batch)), sequential3(mish, conv3x3, batch), sequential3(hardswish, dconv3x3, batch), sequential3(hardswish, dconv3x3, batch)), sequential3(hardswish, dconv3x3, batch), sequential3(hardswish, dconv3x3, batch)), sequential3(hardswish, dconv3x3, batch), sequential3(hardswish, dconv3x3, batch), sequential3(hardswish, dconv3x3, batch), sequential3(hardswish, dconv3x3, batch), sequential3(hardswish, dconv3x3, batch), sequential3(hardswish, dconv3x3, batch), sequential3(hardswish, dconv3x3, batch), sequential3(hardswish, dconv4x1, layer), id (Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zero id. zero Sequential2Cell(zeros 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0)
down, down), Residual2(Cell(zero, id, zero, Sequential3(mish, conv3x3, batch), avg_pool, Sequential3(mish, conv3x3, layer)), down, down)))
ImageNet-16-120 (mean test error  $52.78\%$ , #params 0.626 MB, FLOPs 23.771 M):
Sequential3(Sequential4(Residual2(Cell(id, avg_pool, id, id, Sequential3(relu, dconv3x3, layer), zero), Cell(Sequential3(hardswish, conv1x1, batch), zero, zero, Sequential3(mish, dconv3x3, layer), zero, zero), Cell(Sequential3(relu, dconv3x3, layer), Sequential3(mish, dconv3x3, layer), zero, Sequential3(hardswish, conv3x3, layer), Sequential3(relu, dconv3x3, instance)), Sequential3(hardswish, conv3x3, instance))), Sequential2(Cell(zero, Sequential3(relu, conv3x3, layer), Sequential3(mish, conv1x1, batch), Sequential3(mish, conv1x1, batch), avg_pool, Sequential3(relu, conv3x3, layer)), Cell(id, id, Sequential3(mish, conv3x3, layer), Sequential3(relu, conv3x3, instance), id, id)), Residual2(Cell(zero, avg_pool, Sequential3(mish, conv1x1, batch), Sequential3(mish, conv1x1, layer), zero, zero), Cell(id, Sequential3(relu, dconv3x3, layer), id, zero, zero, id)), Cell(zero, Sequential3(hardswish, conv3x3, layer), avg_pool, zero, Sequential3(hardswish, conv1x1, layer), id)), Residual3(Sequential2(Cell(Sequential3(relu, conv1x1, instance), Sequential3(mish, conv1x1, layer), Sequential3(mish, conv1x1, instance), zero, Sequential3(hardswish, dconv3x3, layer), id), Cell(id, avg_pool, avg_pool, Sequential3(relu, conv1x1, instance), id), Residual2(Cell(Sequential3(mish, conv1x1, batch), Sequential3(mish, conv1x1, instance), id), Residual2(Cell(zero, Sequential3(relu, conv3x3, layer), Sequential3(mish, conv1x1, batch), Sequential3(mish, conv1x1, batch), avg_pool, Sequential3(relu, conv3x3, instance))), Cell(id, zero, Sequential3(mish, conv1x1, batch), Sequential3(mish, conv1x1, instance), id), Residual2(Cell(Sequential3(mish, conv1x1, batch), Sequential3(mish, conv1x1, batch), id), Residual2(Cell(zero, Sequential3(relu, conv1x1, batch), Sequential3(mish, conv1x1, batch), id), Residual2(Cell(zero, Sequential3(mish, conv1x1, instance), Sequential3(mish, conv1x1, batch), id), Residual2(Cell(Sequential3(mish, conv1x1, instance), id), Residual2(Cell(zero, Sequential3(relu, conv1x1, batch), Sequential3(mish, conv1x1, instance), id), Residual2(Cell(Sequential3(mish, conv1x1, instance), Sequential3(mish, dconv3x3, layer), id), Cell(id, avg_pool, avg_pool, Sequential3(relu, conv1x1, instance), id), Residual2(Cell(Sequential3(mish, conv1x1, instance), Sequential3(mish, dconv3x3, instance)), Sequential2(Cell(zero, Sequential3(relu, conv3x3, layer), Sequential3(mish, conv1x1, batch), Sequential3(mish, conv1x1, batch), avg_pool, Sequential3(relu, conv3x3, layer)), Cell(id, zero, Sequential3(mish, conv3x3, layer), Sequential3(relu, conv3x3, instance), id), Residual2(Cell(Sequential3(mish, conv1x1, batch), Sequential3(mish, dconv3x3, instance)), Sequential2(cells(zero, Sequential3(relu, conv3x3
CIFARTile (mean test error  $30.33\%$ , #params 2.356 MB, FLOPs 372.114 M):
Sequential4(Residual3(Residual2(Cell(Sequential3(hardswish, conv3x3, instance), id, zero, Sequential3(relu, dconv3x3, instance), Sequential3(mish, conv1x1, instance), avg_pool), Cell(avg_pool, avg_pool, id, zero, Sequential3(hardswish, conv3x3, batch), avg_pool), Cell(Sequential3(relu, dconv3x3, instance), zero, id, Sequential3(relu, dconv3x3, layer), id, id)), Residual2(Cell(zero, zero, Sequential3(mish, conv1x1, instance), Sequential3(mish, conv3x3, batch), zero, Sequential3(relu, dconv3x3, batch), id, Sequential3(mish, conv3x3, batch), id), Cell(Sequential3(hardswish, dconv3x3, batch), Sequential3(relu, conv3x3, batch), Sequential3(relu, conv1x1, batch), zero, Sequential3(relu, conv3x3, batch), id)), Sequential2(Cell(Sequential3(relu, dconv3x3, layer), Sequential3(mish, conv1x1, layer), id, zero, Sequential3(mish, conv3x3, batch), Sequential3(relu, dconv3x3, layer)), down), Sequential2(Cell(id, Sequential3(hardswish, conv1x1, layer), id, Sequential3(relu, conv1x1, instance), avg_pool, Sequential3(relu, conv1x1, layer)), down)), Residual3(Residual2(Cell(id, avg_pool, avg_pool, Sequential3(hardswish, dconv3x3, instance), Sequential3(mish, conv1x1, layer), Sequential3(hardswish, dconv3x3, instance))), Cell(id, id, Sequential3(relu, dconv3x3, layer), id, id, zero), Cell(Sequential3(relu, conv3x3, layer), id, avg_pool, Sequential3(mish, dconv3x3, instance), Sequential3(relu, conv1x1, layer), zero)), Residual2(Cell(Sequential3(mish, conv3x3, batch), Sequential3(mish, conv3x3, instance), zero, avg_pool, avg_pool, Sequential3(mish, conv1x1, batch)), Cell(Sequential3(mish, conv1x1, batch), Sequential3(relu, dconv3x3, layer), zero, id, avg_pool, avg_pool), Cell(avg_pool, Sequential3(hardswish, conv1x1, instance), id, avg_pool, avg_pool, Sequential3(hardswish, conv1x1, instance))), Residual2(Cell(Sequential3(relu, dconv3x3, batch), avg_pool, id
avg_pool, id, zero), down, down), Residual2(Cell(zero, zero, Sequential3(relu, dconv3x3, batch), avg_pool, Sequential3(hardswish, conv1x1, instance), avg_pool), down, down)), Sequential4(Sequential3(Cell(Sequential3(hardswish, conv3x3, batch), Sequential3(hardswish, conv3x3, batch), Sequential3(relu, conv1x1, instance), id, Sequential3(relu, conv1x1, layer), Sequential3(relu, conv3x3, layer)), Cell(id, Sequential3(relu, conv3x3, instance), Sequential3(hardswish, conv1x1, instance), Sequential3(relu, conv3x3, layer), avg_pool, Sequential3(mish, conv1x1, layer)), Cell(zero, zero, id, Sequential3(relu, conv3x3, batch), id, Sequential3(relu, conv1x1, layer))), Sequential3(Cell(Sequential3(hardswish, conv3x3, batch), Sequential3(relu, conv1x1, instance), Sequential3(relu, dconv3x3, layer), Sequential3(mish, conv1x1, layer), Sequential3(relu, conv3x3, batch)), Cell(id, Sequential3(relu, conv3x3, instance), Sequential3(hardswish, conv1x1, instance), Sequential3(relu, dconv3x3, instance), Sequential3(mish, conv1x1, layer)), Cell(zero, zero, id, Sequential3(relu, conv3x3, batch), id, avg_pool)), Sequential3(Cell(id, id, avg_pool, Sequential3(mish, conv1x1, layer), Sequential3(mish, conv3x3, batch), zero), Cell(id, Sequential3(relu, conv1x1, batch), avg_pool, Sequential3(relu, conv1x1, layer), avg_pool), Sequential3(mish, conv1x1, batch), sequential3(mish, dconv3x3, batch), sequential3(mish, conv1x1, batch), id), Cell(id, Sequential3(hardswish, conv1x1, layer), zero, id, zero, id)), Sequential3(Sequential2(Cell(id, zero, Sequential3(mish, dconv3x3, instance), Sequential3(mish, conv3x3, batch), Sequential3(mish, dconv3x3, instance), Sequential3(relu, conv1x1, instance)), Cell(Sequential3(relu, dconv3x3, instance), avg_pool, Sequential3(mish, conv1x1, instance), Sequential3(hardswish, dconv3x3, instance), id, Sequential3(hardswish, conv1x1, layer))), Sequential2(Cell(zero, zero, Sequential3(mish, dconv3x3, instance), Sequential3(relu, conv1x1, instance), Sequential3(hardswish, conv1x1, instance), avg_pool, zero, Sequential3(hardswish. conv3x3,batch)), cell(avg_poolSequential3(mish,dconv3x3,layer)zeroavg_pool)
AddNIST (mean test error  $6.33\%$  #params 2.853 MB, FLOPs 593.856 M):
dconv3x3, batch), Sequential3(mish, dconv3x3, batch), id, Sequential3(mish, conv3x3, layer)), Cell(zero, Sequential3(relu, dconv3x3, layer), Sequential3(hardswish, conv3x3, instance), avg_pool, avg_pool, zero)), Sequential2(Cell(id, Sequential3(relu, conv3x3, instance), Sequential3(relu, conv3x3, layer), Sequential3(hardswish, dconv3x3, batch), id, Sequential3(relu, conv3x3, layer)), Cell(Sequential3(mish, conv1x1, batch), id, avg_pool, id, avg_pool, id)), Cell(id, Sequential3(relu, conv3x3, layer), Sequential3(mish, conv1x1, instance), Sequential3(hardswish, conv3x3, batch), Sequential3(mish, dconv3x3, instance), Sequential3(hardswish, conv1x1, instance))), Sequential4(Sequential2(Cell(Sequential3(relu, conv3x3, instance), id, Sequential3(relu, conv3x3, batch), avg_pool, zero, id), Cell(zero, Sequential3(relu, conv3x3, batch), avg_pool, Sequential3(mish, conv3x3, batch), Sequential3(relu, dconv3x3, instance), id)), Sequential3(Cell(Sequential3(relu, conv3x3, batch), id), Sequential3(relu, dconv3x3, instance), cell(mish, conv1x1, layer), id, Sequential3(relu, dconv3x3, instance)), Cell(Sequential3(mish, conv3x3, batch), Sequential3(mish, conv1x1, instance), Sequential3(hardswish, dconv3x3, instance), zero, Sequential3(mish, dconv3x3, layer), Sequential3(relu, conv3x3, batch)), Cell(avg_pool, Sequential3(mish, conv1x1, instance), Sequential3(relu, conv3x3, batch), avg_pool, id, Sequential3(mish, dconv3x3, batch))), Sequential2(Cell(id, Sequential3(relu, conv3x3, layer), Sequential3(hardswish, conv3x3, layer), Sequential3(relu, conv3x3, layer), Sequential3(hardswish, dconv3x3, batch), id, Sequential3(relu, conv3x3, layer)), Cell(Sequential3(mish, conv3x3, batch), id, avg_pool, id), Cell(id, Sequential3(relu, conv3x3, layer), Sequential3(mish, conv1x1, instance), Sequential3(hardswish, conv3x3, batch), Sequential3(mish, dconv3x3, instance)), Sequential3(mish, conv3x3, instance))))
Below we also report the best found architecture per dataset on the hierarchical search space with more non-linear macro topologies (instead of linear macro topologies) and a single, shared cell (hierarchical+shared+non-linear). For sake of readability we report macro topology and cell separately. Different to the best architectures found in the hierarchical NAS-Bench-201 search space (hierarchical), the found architectures on this variant tend to have higher computational complexity (parameters and FLOPs).
CIFAR-10 (mean test error  $5.02\%$ , #params 4.82 MB, FLOPs 439.796 M):
Sequential3(Residual3(Residual2(Shared, Shared, Shared), Diamond2(Shared, Shared, Shared, Shared), Residual2(Shared, down, down), Residual2(Shared, down, down)), Residual3(Residual2(Shared, Shared, Shared), Diamond2(Shared, Shared, Shared, Shared), Diamond2(Shared, Down, down), Diamond2(Shared, Shared, Down, down)), Diamond3(Residual2(Shared, Shared, Shared), Residual2(Shared, Shared, Shared), Residual2(Shared, Shared, Shared))
Cell(Sequential3(hardswish, conv3x3, layer), Sequential3(hardswish, conv3x3, layer), Sequential3(mish, conv3x3, layer), Sequential3(relu, conv3x3, batch), Sequential3(mish, conv3x3, instance), Sequential3(relu, conv1x1, batch))
CIFAR-100 (mean test error  $25.41\%$ , #params 2.68 MB, FLOPs 307.87 M):
Sequential4(Residual3(Residual2(Shared, Shared, Shared), Residual2(Shared, Shared, Shared), Residual2(Shared, down, down), Residual2(Shared, down)), Residual3(Residual2(Shared, Shared, Shared), Residual2(Shared, Shared, Shared), Residual2(Shared, down, down), Residual2(Shared, down))), Residual3(Residual2(Shared, Shared, Shared), Residual2(Shared, Shared), Residual3(Diamond2(Shared, Shared, Shared, Shared), Residual2(Shared, Shared), Shared))
Cell(Sequential3(mish, conv3x3, batch), Sequential3(relu, conv3x3, batch), Sequential3(relu, conv3x3, batch), Sequential3(mish, conv3x3, batch), Sequential3(mish, dconv3x3, instance), id)
ImageNet-16-120 (mean test error  $48.16\%$ , #params 0.852 MB, FLOPs 89.908 M):
Sequential3(Sequential3(Diamond2(Shared, Shared, Shared, Shared), Residual2(Shared, Shared, Shared), Shared), Diamond3(Residual2(Shared, Shared, Shared), Diamond2(Shared, Shared, Shared, Shared), Residual2(Shared, Shared, Shared), Diamond2(Shared, Shared, Shared, Shared), Diamond2(Shared, Shared, Shared, Shared), Residual2(Shared, Shared, Shared, Shared), Residual2(Shared, Shared, Share, Share, Share))
(a) CIFAR-10.
(b) CIFAR-100.
(c) ImageNet-16-120.
(d) CIFARTile.
(e) AddNIST.
Figure 10: Visualization of the best found architectures in our hierarchical NAS-Bench-201 search space. Abbreviations are defined as follows: ap=avg_pool, ba=batch, c1=conv1x1, c3=conv3x3,CLS=classifier, dc=dconv3x3, ha=hardswish, in=instance, la=layer, mi=mish, re=relu, and stm=stem. Best viewed with zoom.
Figure 11: Density estimates of the validation errors of all architecture candidates proposed by the search strategies and across all datasets from our experiments in Section 6.
Cell(Sequential3(mish, conv3x3, batch), Sequential3(mish, conv1x1, batch), Sequential3(relu, conv3x3, instance), Sequential3(mish, conv3x3, instance), Sequential3(relu, conv3x3, batch), id)
CIFARTile (mean test error  $39.2\%$ , #params 10.712 MB, FLOPs 1519.944 M):
Sequential4(Sequential3(Residual2(Shared, Shared, Shared), Residual2(Shared, Shared, Shared), Diamond2(Shared, Shared, Down, down)), Residual3(Sequential2(Shared, Shared), Residual2(Shared, Shared, Shared), Residual2(Shared, Down), Diamond2(Shared, Shared, Down, down)), Diamond3(Sequential2(Shared, Shared), Sequential2(Shared, Shared), Diamond2(Shared, Shared, Shared), Residual2(Shared, Shared, Shared), Residual2(Shared, Shared), Residual2(Shared, Shared))
Cell(Sequential3(mish, conv3x3, batch), Sequential3(relu, conv3x3, instance), avg_pool, Sequential3(mish, conv3x3, layer), Sequential3(relu, dconv3x3, layer), Sequential3(mish, conv3x3, layer))
AddNIST (mean test error  $4.57\%$ , #params 3.345 MB, FLOPs 347.959 M):
Sequential4(Diamond3(Diamond2(Shared, Shared, Shared, Shared), Diamond2(Shared, Shared, Shared, Shared), Diamond2(Shared, Shared, Shared, Shared), Residual2(Shared, down, down), Residual2(Shared, down, down)), Residual3(Diamond2(Shared, Shared, Shared, Shared), Diamond2(Shared, Shared, Shared, Shared), Residual2(Shared, down, down), Residual2(Shared, down, down)), Residual3(Diamond2(Shared, Shared, Shared, Shared), Diamond2(Shared, Shared, Shared, Shared), Residual3(Diamond2(Shared, Shared, Shared), Diamond2(Shared, Shared, Shared))
Cell(Sequential3(relu, conv3x3, batch), zero, avg_pool, Sequential3(hardswish, conv3x3, instance), Sequential3(hardswish, conv3x3, instance), Sequential3(hardswish, conv3x3, instance))