# Self-Tuning Lam Annealing: Learning Hyperparameters While Problem Solving
## 2 Methods
### Self-Tuning Lam
#### 2.3.3 Tuning the Initial Temperature \(T_{0}\)  
One of the two hyperparameters that directly controls the evolution of the temperature \(T\) throughout the run is the initial temperature \(T_{0}\). The Modified Lam sets it to a constant, independent of run length and cost function scale. The initial temperature has a direct effect on the Modified Lam's problem solving efficacy, with respect to cost function scale. If the average difference of cost between neighboring solutions is significantly above \(T_{0}\), then the early portion of the search will fail to explore, choosing instead to reject all neighbors with inferior cost, increasing the impact of local optima in the search space. Since the Modified Lam is capable of increasing temperature, it will eventually adjust for this if the run is sufficiently long, but will waste time early in the process. In the Self-Tuning Lam, we utilize the \(M\) iterations of our new tuning phase to gather data related to the average change in cost of random neighbors. First, recall from Section 2.3.1 that the tuning phase of the Self-Tuning Lam accepts all neighbors, and thus begins with a random walk from a random initial solution. Using the \(M\) tuning iterations, we compute the average difference in cost between neighbors, but excluding cases where the neighboring solutions have the same cost (those will be accounted for separately as we will see). Let \(S_{0}\) be the random initial solution, and let \(S_{i}\) be the random neighbor of \(S_{i-1}\) generated during iteration \(i\). Additionally, let \(C(S_{i})\) be the cost of solution \(S_{i}\), where the problem is to minimize this cost. Define \(k\) as the count of the number of pairs of neighboring solutions with different costs found during the tuning phase, as shown here:  
\[k=\sum_{i=1}^{M} \begin{cases}1&\text{if }C(S_{i})\
eq C(S_{i-1})\\ 0&\text{if }C(S_{i})=C(S_{i-1}).\end{cases}  \tag{11}\]  
Now, define the average change in cost, \(\Delta C\), as follows:  
\[\Delta C= \begin{cases}\frac{1}{k}\sum_{i=1}^{M}|C(S_{i})-C(S_{i-1})|&\text{if } k>0\\ 1&\text{if }k=0.\end{cases}  \tag{12}\]  
The case above when \(k=0\) occurs if all of the tuning samples reside on a plateau in the search space. In this case, we set \(\Delta C=1\) as we require it to be non-zero, which for an integer cost problem is the lowest possible non-zero cost difference. Define \(d\) as the number of cases where the cost of neighbor \(S_{i}\) is either the same as or superior to that of the prior solution \(S_{i-1}\) during the tuning phase, as shown here:  
\[d=\sum_{i=1}^{M} \begin{cases}1&\text{if }C(S_{i})\leq C(S_{i-1})\\ 0&\text{if }C(S_{i})>C(S_{i-1}).\end{cases}  \tag{13}\]  
These are cases where SA would deterministically choose to accept the neighbor. Next, use \(d\) to define the proportion \(\gamma\) of tuning phase samples where SA would have deterministically chosen to accept the neighbor, independent of the temperature:  
\[\gamma= \begin{cases}\frac{d}{M}&\text{if }d\
eq M\\ \frac{d}{1+M}&\text{if }d=M.\end{cases}  \tag{14}\]  
The case above when \(d=M\) occurs when all of the tuning samples reside on a plateau. Our tuning use of \(\gamma\) requires \(\gamma<1\), which is the reason for the adjustment in this case. We now use \(\Delta C\) and \(\gamma\) to estimate the probability \(P(T_{0})\) that SA accepts a random neighbor using the usual Boltzmann decision at temperature \(T_{0}\):  
\[P(T_{0})=\gamma+(1-\gamma)\cdot e^{-\Delta C/T_{0}}, \tag{15}\]  
which is the probability \(\gamma\) of the deterministic acceptance case (i.e., SA always accepts neighbors that are not worse than the current solution), plus the probability, \(1-\gamma\), thatthe random neighbor has worse cost times the probability of accepting it anyway (the SA's Boltzmann distribution), where we use \(\Delta C\) as an estimate of the cost difference. In Equation (9) we defined \(\text{AcceptRate}(0)\) at the end of the tuning phase to be equal to the target acceptance rate \(M\) iterations into the run.