# Self-Tuning Lam Annealing: Learning Hyperparameters While Problem Solving
## 2 Methods
### Self-Tuning Lam
#### 2.3.5 Putting It All Together
We continue by examining the results of the experiments on minimizing \(f_{2,\phi}(x)\). Comparisons of the average solutions found by the two algorithms at different run lengths \(N\), for the four scaled versions of the problem that we consider, \(f_{2,1}(x)\), \(f_{2,10}(x)\), \(f_{2,100}(x)\), and \(f_{2,1000}(x)\), are shown in Tables 11-14, respectively. Note that the value of the optimal solution scales according to our scale factor \(\phi\). First, for the original version of the problem (Table 11), no statistically significant difference was seen, except for the shortest runs (\(N=1000\)), where the Modified Lam exhibited a slight performance advantage. The Self-Tuning Lam achieved a near-optimal solution on average with fewer iterations than the Modified Lam (e.g., beginning at 10,000 iterations for the Self-Tuning Lam vs 100,000 iterations for the Modified Lam). However, the difference at 10,000 iterations was not statistically significant (\(p=0.32\)). Figure 8: Minimize \(f_{1,1}(x)\): Self-Tuning Lam, Modified Lam, and target acceptance rates for (**a**) 1000 iterations, (**b**) 10,000 iterations, (**c**) 100,000 iterations, and (**d**) 1 million iterations. Figure 10: Minimize \(f_{1,100}(x)\): Self-Tuning Lam, Modified Lam, and target acceptance rates for (**a**) 1000 iterations, (**b**) 10,000 iterations, (**c**) 100,000 iterations, and (**d**) 1 million iterations. Figure 9: Minimize \(f_{1,10}(x)\): Self-Tuning Lam, Modified Lam, and target acceptance rates for (**a**) 1000 iterations, (**b**) 10,000 iterations, (**c**) 100,000 iterations, and (**d**) 1 million iterations. \begin{table}
\begin{tabular}{c c c c} \hline \hline \(N\) & **Modified Lam** & **Self-Tuning Lam** & **T-Test _p-Value** \\ \hline
1000 & 22.978824 & 13.303961 & \(<\)\(10^{-4}\) \\
10,000 & 18.422196 & 6.650951 & \(<\)\(10^{-8}\) \\
100,000 & 6.650951 & 6.650951 & 0.69 \\
1,000,000 & 6.650951 & 6.650951 & 0.95 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Average solution for minimizing \(f_{2,100}(x)\). The minimum is \(f_{2,100}(x)\approx 66.5095\). Figure 11: Minimize \(f_{1,1000}(x)\): Self-Tuning Lam, Modified Lam, and target acceptance rates for (**a**) 1000 iterations, (**b**) 10,000 iterations, (**c**) 100,000 iterations, and (**d**) 1 million iterations. \begin{table}
\begin{tabular}{c c c c} \hline \hline \(N\) & **Modified Lam** & **Self-Tuning Lam** & **T-Test _p-Value** \\ \hline
1000 & 1.044128 & 1.690334 & 0.002 \\
10,000 & 0.703067 & 0.665095 & 0.32 \\
100,000 & 0.665095 & 0.665095 & 0.32 \\
1,000,000 & 0.665095 & 0.665095 & 0.07 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Average solution for minimizing \(f_{2,1}(x)\). The minimum is \(f_{2,1}(x)\approx 0.665095\). At all other scales (Tables 12-14) the Self-Tuning Lam achieved far superior solutions on average than the Modified Lam at very statistically significant levels for \(N\leq\) 10,000. For all three of those scales, the Self-Tuning Lam's solutions for \(N=\) 10,000 were on average near-optimal, just as achieved for the original version of the problem; whereas the Modified Lam required an order of magnitude longer to achieve solutions of that quality. No statistically significant difference was seen for the longest runs of \(N\geq\) 100,000, as average solutions for both algorithms are very near the optimal. When minimizing \(f_{2,\phi}\), the Self-Tuning Lam optimally solves the problem an order of magnitude faster than the Modified Lam. This is likely due to the Self-Tuning Lam's ability to more consistently follow the target acceptance rate, independent of cost scale (see Figures 12-15) and run length (e.g., parts (a) to (d) of those figures). Figure 14: Minimize \(f_{2,100}(x)\): Self-Tuning Lam, Modified Lam, and target acceptance rates for (**a**) 1000 iterations, (**b**) 10,000 iterations, (**c**) 100,000 iterations, and (**d**) 1 million iterations.