## 3 Evaluation  
We implemented the feature extraction and attribute reduction in Java. We used the _weka_[5] and _libsvm_[6] libraries for SVM training and evaluation. For training and test, different traces with valid and invalid SIP messages have been used. The used traces were derived together with security experts of a tier one national carrier. Their experts recorded a typical test session which is used to qualify SIP network elements. We used two different training data sets: trace 1 contained 610 messages (12 invalid),  
Figure 3: Example of the attribute reduction step to increase throughput. and trace 2 contained 928 messages (28 invalid). The true positive rate (TPR) and false positive rate (FPR) in our tests have been obtained by using a test trace containing 12,923 messages (10,000 invalid). A major problem was setting the SVM parameters in a way to ensure a correct classification for different kind of traces, _i.e._, when considering different SIP protocol extensions. _Libsvm_ trains a model based on the given \(l\) input data (\(\textbf{x}_{i}\ ;i=1,...,l\)), each with corresponding class (\(y_{i}\ ;\ y_{i}\in\{1,-1\}\)). For the determination _libsvm_ solves the following primal problem:  
\[\min_{\textbf{w},b,\xi}\frac{1}{2}\textbf{w}^{T}\textbf{w}+C\sum_{i=1}^{l}\xi _{i}\hskip 72.27 pt\text{subject to}\hskip 28.452756 pty_{i}(\textbf{w}^{T}\phi( \textbf{x}_{i})+b)\geq 1-\xi_{i}\]  
with **w** as normal vector of the separating hyperplane, \(b\) as displacement of **w**, \(\xi_{i}\) as slack variable according to [3] and \(\phi(x)\) as mapping function into the high dimensional space. This leads to the non-parametric decision function:  
\[\text{sgn}\left(\sum_{ \begin{subarray}{c}support\\ vectors\end{subarray} }y_{i}\alpha_{i}K(\textbf{x}_{i},\textbf{x})\ +\ b\right)\hskip 72.27 pt\text{ with }K(a,b)\text{ as the used kernel}\]  
Based on this formulation, we have to choose \(C\), the kernel and based on the kernel selection its parameter. We use the default _libsvm_-kernel (RBF-kernel defined as \(K(a,b)=\exp(-\gamma\|a-b\|^{2})\)). The C-parameter describes the ratio between the usage of the hyperplane and additional support vectors [3]. This parameter has shown the largest influence on detection accuracy. Therefore, we left the kernel and \(\gamma\) to the default values (\(\gamma=(\#features)^{-1}\)) and changed only C during evaluations. _Libsvm_ uses C-Parameter settings between 0 and \(\infty\), which corresponds to the interval between 0 and 1 that is mostly used in literature. For our evaluations, we use an SVM classifier based solely on 4-grams as our reference. We refer to this as the _basic_ approach, _i.e._, without keyword extraction and attribute reduction. Figure 4 shows TPR (top chart) and FPR (bottom chart) for two different traces. The basic approach is indicated with gray lines. The result achieved with our keyword extension is presented with black lines and markers. We can observe an increase of the TPR close to 100% with increasing C-Parameter. There is a steep increase of the TPR for the basic approach at a C-parameter of 10 for trace 2 and at a C-Parameter of 22 for trace 1. The FPR is very low and increases for trace 1 at a C-Parameter of 22 from 0.51% to 0.71%. For trace 2 the FPR stays constant at about 0.2%. We realized that this constant FPR is due to message properties that are covered only partly in the traces. A better coverage of these properties in the training set could improve detection accuracy here. Such a problem arises if e.g. additional optional parameters of a SIP-Header are not contained in the training set. An example is the optional _user_ parameter in the following two _From_ definitions:  
From: [PERSON] $<$sip:[PERSON]AIL_ADDRESS]:5060;user=phone>;tag=4711 From: [PERSON] $<$sip:[PERSON]AIL_ADDRESS]:5060>;tag=4711 As we can see from Figure 4, setting the C-Parameter for trace 2 is unproblematic, but for trace 1 there is a very small gap between the rising slopes of the TPR and FPR. This means, that the classifier is not stable for this trace.