## 3 Methodology
### TetraOpt Algorithm for Hyperparameter Optimization  
TetraOpt is an optimization algorithm that utilizes tensor train programming approaches to guide the optimization process. In our work, we apply TetraOpt for the task of hyperparameter optimization of convolutional neural networks. The TetraOpt algorithm represents a hyperparameter search space as a tensor of configurations with different combinations of hyperparameters and performs corresponding measurements of target optimization metrics for this configuration. Instead of evaluating all possible configurations, TetraOpt estimates the values of this tensor using the cross-approximation technique developed by [PERSON] and [PERSON] [39]. In addition to that, TetraOpt utilizes a MaxVol operation, proposed by [PERSON] and [PERSON] [2], to identify the most promising areas of the hyperparameter search space. This allows for reducing the algorithm runtime by allocating resources for exploring only the regions that are likely to yield high target metric values and discarding less promising areas of the search space. The TetraOpt algorithm is described through the following 9 steps:  
1. Define each of the \(d\) hyperparameters on a discrete interval with \(n\) values \(h_{i}\in\left[h_{i}^{\min},h_{i}^{\max}\right]\), where \(i\in[1,d]\): \[\left\{h_{i}(1),h_{i}(2),\ldots,h_{i}(n)\right\}_{i=1}^{i=d}\] (1)
2. Select the rank \(r\): a smaller \(r\) ensures faster computations, while a larger \(r\) retains more accuracy. 3. Choose \(r\) combinations of hyperparameters: \[\left\{h_{2}^{1}(j),h_{3}^{1}(j),\ldots,h_{d}^{1}(j)\right\}_{j=1}^{j=r}\] (2)
4. In the following three steps, we perform an iterative process known as the \"sweep right\". The first step is evaluating the first TT core:  
Figure 1: General schema of neural network hyperparameter optimization problem. Evaluate the accuracy of \(n\cdot r\) configurations, combining all \(n\) values of the first hyperparameter \(\{h_{1}(i_{1})\}_{i_{1}=1}^{i_{1}=n}\) with the \(r\) selected combinations of \(\{h_{2}^{1}(j),h_{3}^{1}(j),\ldots,h_{d}^{1}(j)\}_{j=1}^{j=r}\): \[\{A\big{(}h_{1}(i_{1}),h_{2}^{1}(j),h_{3}^{1}(j),\ldots,\] \[h_{d}^{1}(j)\big{)}\}_{j=1,h_{1}=1}^{j=r,i_{1}=n}.\] (3)
* Identify the \(r\) most promising configurations in this \(n\times r\) matrix by selecting the submatrix with the largest determinant: \[\Big{\{}A\big{(}h_{1}^{1}(i_{1}),h_{2}^{1}(j),h_{3}^{1}(j),h_{d}^{1}(j)\big{)} \Big{\}}_{j=1,h_{1}=1}^{j=r,i_{1}=r}.\] (4) And fix the corresponding \(r\) values of the first hyperparameter: \(\{h_{1}^{1}(i_{1})\}_{i_{1}=1}^{i_{1}=r}\):
5. Evaluate the second TT core: * Fix the \(r\) values of \(\{h_{1}^{1}(i_{1})\}_{i_{1}=1}^{i_{1}=r}\) from the previous step and \(r\) combinations of \(\big{\{}h_{3}^{1}(j),h_{4}^{1}(j),\ldots,h_{d}^{1}(j)\big{\}}_{j=1}^{j=r}\) from the third step. Evaluate the accuracy for \(nr^{2}\) configurations by varying all \(n\) values of the second hyperparameter \(\{h_{2}(i_{2})\}_{i_{2}=1}^{i_{2}=n}\): \[\big{\{}A\big{(}h_{1}^{1}(i_{1}),h_{2}(i_{2}),h_{3}^{1}(j),\ldots,\] \[h_{d}^{1}(j)\big{)}\big{\}}_{j=1,h_{1}=1,h_{2}=1}^{j=r,i_{1}=r,i_{2}=n}.\] (5)
* From this \(nr\times r\) matrix, identify the \(r\) most promising configurations by identifying a submatrix with the maximal determinant module: \[\big{\{}A\big{(}\big{(}h_{1}^{2}(k),h_{2}^{2}(k)\big{)},h_{3}^{1}(j),\ldots,\] \[h_{d}^{1}(j)\big{)}\big{\}}_{j=1,k=1}^{j=r,k=r}\] (6) The \(r\) combinations \(\big{\{}\big{(}h_{1}^{2}(k),h_{2}^{2}(k)\big{)}\big{\}}_{k=1}^{k=r}\) for the first and second hyperparameters are now fixed. 6. Continue the process for the \((d-1)\)th TT core. Once this step is completed for all \((d-1)\) TT cores, the \"sweep right\" process is considered finished. 7. Similar to step 3, we now have \(r\) combinations of hyperparameters. However, these combinations are no longer random. Perform a \"sweep left\" starting from the last TT core, reversing the order of indices. This process is similar to the \"sweep right\" but works backward, refining the configurations iteratively.