SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Performance comparison under for two different sizes of initial samples, where the best is shown in bold while the second best is underlined for query times.
Initial Size	Algorithm	Problem 1	Problem 2	Problem 3
State Dimension:11	State Dimension:20	State Dimension:30
Failure times	Query times	Failure times	Query times	Failure times	Query times
N=32	BOGP	0	9.60±3.89	100	1000±0	15	239.12±367.12
GA	0	32.00±0	0	241.60±71.75	21	270.80±382.51
PSO	0	32.00±0	18	311.20±333.45	14	283.28±324.54
CMAES	0	77.56±4.38	1	321.01±188.6	22	280.54±363.18
ISRES	0	64.00±0	3	416.24±209.23	21	276.24±386.75
NSGA2	0	32.00±0	1	239.44±150.26	22	262.88±394.99
UNSGA3	0	32.00±0	2	218.72±136.53	22	260.64±396.51
SVPEN	100	1000±0	100	1000±0	100	1000±0
GESESE (Ours)	0	33.63±19.35	0	233.96±180.01	10	167.77±284.31
N=16	BOGP	0	10.62±5.53	100	1000±0	17	249.88±372.99
GA	0	16.00±0	43	657.04±352.42	23	364.40±373.30
PSO	0	32.00±0	10	293.76±271.02	21	247.76±392.87
CMAES	0	77.56±4.38	1	333.49±156.24	17	320.07±350.84
ISRES	0	17.00±0	2	260.50±189.71	20	243.20±392.70
NSGA2	0	32.00±0	33	590.96±355.93	25	377.20±385.78
UNSGA3	0	32.00±0	28	487.04±360.22	28	408.80±397.77
SVPEN	100	1000±0	100	1000±0	100	1000±0
GESESE(Ours)	0	36.72±22.52	0	248.26±176.64	9	163.26±279.34
Below we further discuss separately the effects of different hyperparameters and analyze the reasons behind: (1) We experiment with three base network numbers  $L = 2, 4, 8$ . It can be seen from Table 6 that there is a performance improvement as  $L$  increases in terms of the required query times, but this is on the expense of consuming higher training cost. Thus, we choose the more balanced setting  $L = 4$  as the default in our main experiments. (2) We test different candidate state numbers  $N_{\mathrm{IT}} = 1, 32, 64, 128$  used for exploitation. Results show a performance increase followed by a decrease as  $N_{\mathrm{IT}}$  increases. Using a candidate set containing one single state is insufficient, while allowing a set with too many candidate states can also harm the efficiency. This can be caused by the approximation gap between the surrogate error model and the true physical evaluation. In our main experiments, we go with the setting of 64 for problem 1 as we mentioned in Appendix B, because it provides a proper balance between the exploitation performance and the overfitting risk. (3) We also examine different settings of the learning rate for training the exploitation generator, i.e.,  $\eta_{\mathrm{IT}} = 1e^{-1}, 1e^{-2}, 1e^{-3}$ . Similarly, there is a performance increase first but followed by a decrease, as in changing  $N_{\mathrm{IT}}$ . A larger learning rate can accelerate the learning of the exploitation generator and subsequently enable a potentially faster search of the feasible state. But an overly high learning rate can also cause fluctuation around the local optimum, and this then consumes more query times. Although a smaller learning rate can enable a more guaranteed convergence to the local optimum, it requires more iterations, thus more query times. (4) We experiment with three values of early stopping threshold, i.e.,  $\epsilon_e = 1e^{-3}, 1e^{-4}, 1e^{-5}$ . It can be seen from Table 6 that a decreasing  $\epsilon_e$  can first improve the efficiency but then reduce it, however without changing much the standard deviation. An inappropriate setting of the early stopping threshold can lead to base neural networks overfitting (or underfitting) to the actual data distribution, thus harm the performance.
Table 6: Results of sensitivity Analysis, where a better performance is highlighted in bold.
(1): Effect of Base Network Number
Base Network Number	Query times
L = 2	20.20 ±16.37
L = 4	15.44 ±13.86
L = 8	15.09 ±13.01
(2): Effect of Latent Vector Number
Latent vector number	Query times
NT=1	72.56 ±36.13
NT=32	28.21 ±17.05
NT=64	20.20 ±16.37
NT=128	26.95 ±14.12
(3): Effect of Learning rate for Exploration Generator
Learning Rate	Query times
ηIT=1e-1	27.56 ±9.28
ηIT=1e-2	20.20±16.37
ηIT=1e-3	64.36 ±44.50
(4): Effect of Early Stopping Threshold
Early stopping threshold	Query times
εe=1e-3	37.55±17.28
εe=1e-4	20.20 ±16.37
εe=1e-5	26.20 ±15.45