# Optimized Score Level Fusion for Multi-Instance Finger Vein Recognition
## 4 Proposed Method
### Optimization  
Optimization is an iterative process to find the desired solution (minimum or maximum) of a problem while satisfying all of its constraints. An optimization problem could be non-linear or linear. In non-linear optimization, search methods are used iteratively before a solution is achieved. In a typical machine learning model, the model is trained with data \(\{(x_{i},\,y_{i})\}_{i=1}^{N}\) to learn or set the parameters \(p\) of the model. While parameters are set by the model, the model also has hyperparameters \(\lambda\) that are typically set by the user manually. In other words, given an objective function\(f\) to be minimized, a typical machine learning problem solves the following problem:  
\[\underset{p}{minimizef}\,\left(p\mid\{(\,x_{i},y_{i}\,)\}_{i=1}^{N}\,, \lambda\right) \tag{11}\]  
While this gives decent performance and is easily solvable for some machine learning models, it, unfortunately, leaves out potential higher performance of the model by not optimizing \(\lambda\). It is also much more difficult to solve when optimizing both as:  
\[\underset{p,\lambda}{minimizef}\,\left(p,\lambda\mid\{(\,x_{i},y_{i}\,)\}_{ i=1}^{N}\right) \tag{12}\]  
The basic method to solve this is with grid-based search or random search of the hyperparameters. While usable, the problem with hyperparameter optimization is that each evaluation of the hyperparameters is very expensive, since it requires retraining the model. Hence, it is important for hyperparameter optimization to use as little evaluation of objective function as possible. Bayesian hyperparameter optimization is better than grid or random search since it uses previous optimization results for the next optimization, allowing it to be more efficient [13]. Bayesian optimization is also used due to its nature as a non-gradient-based optimizer and has high compatibility. since not all objective functions of machine learning models provide gradient information for optimization [11]. Specifically, the proposed Bayesian optimization can be formalized as sequential model-based optimization (SMBO) [14]. SMBO is started by first initializing a probabilistic regression model \(M\) using a small set of samples from domain \(\chi\), which is the range of hyperparameters to be optimized. A new sample is selected by optimizing an acquisition function \(A\) which uses the model as a cheap surrogate for the expensive objective function \(f\) (for BSSF or BSBF). Each sample \(x_{i}\) produces an observation of the objective function \(y_{i}=f\left(x_{i}\right)\). The new sample is appended to the set of all samples or data \(Q=\{(x_{1},y_{1}),\,\ldots,\,\{x_{i},y_{i}\}\}\), which is later used to update the model. A limit \(T\) is used to restrict the number of optimization runs. This process is presented in Algorithm 1 [Adapted with permission from [14]]. ```
Data:\(f,\chi,A,M\) // initialize dataset with random samples from domain \(Q\leftarrow\text{InitSamples}(f,\chi)\); // run for T steps for\(i\leftarrow|Q|\) to \(T\)do // train a model based on data \(p(y|x,Q)\leftarrow\text{FitModel}(M,Q)\); // select best hyperparameters based on model \(x_{i}\leftarrow\text{argmax}_{x\in\chi}A(x,p(y\big{|}x,Q))\); // evaluate the hyperparameters \(y_{i}\gets f(x_{i})\); // append new data \(Q\gets Q\cup(x_{i},y_{i})\); end
```  
**Algorithm 1** Sequential Model-Based Optimization (SMBO)  
SMBOs can be roughly differentiated by their regression models and acquisition functions. In this work, tree-structured parzen estimator (TPE) [15] is used as the regression model and expected improvement (EI) is used as the acquisition function. The SMBO with TPE algorithm is used to optimize the HOG hyperparameters, the SVM hyperparameters, and the weights of the score level fusion. The use of optimization exhibits better results than using a fixed combination of hyperparameters while being faster than using the brute force approach to search for the best combination of hyperparameters. In our implementation, the objective function of BSSF method can be seen in Algorithm 2.