## 5 Methods
### Support Vector Regression
\[\frac{\partial L}{\partial\xi_{i}^{*}}=C-\alpha_{i}^{*}-\eta_{i}^{*}=0 \tag{42}\]  
Constants \(\alpha_{i}\) and \(\alpha_{i}^{*}\) define the resulting support vectors. \(\eta_{i}\) and \(\eta_{i}^{*}\) can be eliminated through Equations (41) and (42) as \(\eta_{i}^{(*)}=C-\alpha_{i}^{(*)}\). Furthermore, \(\omega\) can also be expressed as  
\[\omega=\sum_{i=1}^{t}(\alpha_{i}-\alpha_{i}^{*})x_{i} \tag{43}\]  
which is known as Support Vector expansion, as \(\omega\) can completely be described by a linear combination of the training data [39].The substitution of Equations (39) to (42) into (38) results in the optimization problem  
\[ \begin{array}{ll}\text{maximize}&-\frac{1}{2}\sum_{i,j=1}^{t}\bigl{(}\alpha _{i}-\alpha_{i}^{*}\bigr{)}\bigl{(}\alpha_{j}-\alpha_{j}^{*}\bigr{)}\bigl{\langle} x_{i},x_{j}\bigr{\rangle}+\\ &-e\sum_{i=1}^{t}\bigl{(}\alpha_{i}+\alpha_{i}^{*}\bigr{)}+\sum_{i=1}^{t}y_{i} \bigl{(}\alpha_{i}-\alpha_{i}^{*}\bigr{)}\\ \text{subject to}&\sum_{i=1}^{t}\bigl{(}\alpha_{i}-\alpha_{i}^{*}\bigr{)}=0 \text{ and }\alpha_{i},\alpha_{i}^{*}\in[0,C]\end{array} . \tag{44}\]  
The result of the optimization problem determines the linear combination of the training data that is used to describe \(\omega\). Finally, the complete algorithm can be described using the data, even the evaluating \(f(x)\):  
\[f(x)=\sum_{i=1}^{t}(\alpha_{i}+\alpha_{i}^{*})\bigl{\langle}x_{i},x\bigr{\rangle} +b. \tag{45}\]  
According to the Karush-Kuhn-Tucker conditions, the product of the primal variables and the constraints vanishes at the point of solution:  
\[ \begin{array}{l}\alpha_{i}(\epsilon+\xi_{i}-y_{i}+\langle\omega,x_{i} \rangle+b)=0\\ \alpha_{i}^{*}\bigl{(}\epsilon+\xi_{i}^{*}+y_{i}-\langle\omega,x_{i}\rangle-b \bigr{)}=0\end{array}  \tag{46}\]  
and  
\[ \begin{array}{l}(C-\alpha_{i})\xi_{i}=0\\ \bigl{(}C-\alpha_{i}^{*}\bigr{)}\xi_{i}^{*}=0\end{array} . \tag{47}\]  
Firstly, Equation (47) means that only samples (\(x_{i}\), \(y_{i}\)) with \(\alpha_{i}^{(*)}=C\) lie outside the \(\epsilon\)-tube [39], as either \(\xi_{i}^{(*)}\) is zero, which means that the sample lies inside the \(\epsilon\)-tube or \(\xi_{i}^{(*)}>0\), but then \(\alpha_{i}^{(*)}\) has to equal \(C\) in order to fulfill the equation. Secondly, according to Equation (46) \(\alpha_{i}^{(*)}=0\) for samples that lie inside the \(\epsilon\)-tube. The inner part of the brackets is non-zero when the sample lies in the tube, as \(\xi_{i}^{(*)}\geq 0\). Thus, \(\alpha_{i}^{(*)}\) have to be zero in order for the equation to vanish. In conclusion, only samples outside the \(\epsilon\)-tube have non vanishing coefficients and are used to create the regression model. They are called Support Vectors. Until now, this subsection only described the linear function case. A simple method for archiving a nonlinear algorithm is a preprocessing map \(\Phi\) of the input space into a higher dimensional feature space [40]. However, this approach can quickly become computationally infeasible for higher dimensional input space or feature space [39]. As already seen in Equation (44), it is sufficient to know the result of the dot product of the mapped input samples instead of explicitly calculate the mapped samples. The result of the dot product of the mapped input samples is called kernel  
\[k\bigl{(}x_{i},x_{j}\bigr{)}=\langle\Phi(x_{i}),\Phi(x_{j})\rangle. \tag{48}\]and substitutes the former dot product in the optimization problem:  
\[ \begin{array}{ll}\text{maximize}&-\frac{1}{2}\sum_{i,j=1}^{i}\big{(}\alpha_{i}- \alpha_{i}^{*}\big{)}\left(\alpha_{j}-\alpha_{j}^{*}\right)k\big{(}x_{i},x_{j} \big{)}+\\ &-e\sum_{i=1}^{i}\big{(}\alpha_{i}+\alpha_{i}^{*}\big{)}+\sum_{i=1}^{i}y_{i} \big{(}\alpha_{i}-\alpha_{i}^{*}\big{)}\\ \text{subject to}&\sum_{i=1}^{i}\big{(}\alpha_{i}-\alpha_{i}^{*}\big{)}=0\text{ and }\alpha_{i},\alpha_{i}^{*}\in[0,C]\end{array} . \tag{49}\]  
There are several types of kernels that fulfill the necessary conditions, but note that the optimization now finds the flattest function in feature space, not in input space [39].