# Efficacy Improvement of Anomaly Detection by Using Intelligence Sharing Scheme
## 4 Proposed Architecture
### Proposed Algorithm
#### 4.2.5 Pairwise Optimization of Normalization Error  
This method is similar to pairwise deletion (available-case analysis) when used to estimate a correlation matrix. In that case, pairwise deletion was used in pairs of networks \(n_{i}\),\(n_{j}\) to compute correlation coefficients \(c_{i,j}\) and build a correlation matrix [\(c_{i,j}\) ]. For each pair of networks \(n_{i}\),\(n_{j}\), a set of common hostnames was used to compute the correlation; hostnames missing from some of them were discarded. A much greater proportion of samples was utilized in this way. Then, in the other case with complete-case analysis, only identical sets of hostnames are ever compared together in contrast to available-case analysis. However, the resulting correlation matrix [\(c_{i,j}\) ] is not guaranteed to be positive-semi-definite. Since our goal is normalizing outlier scores rather than estimating the correlation matrix, a normalization error function was built instead of a correlation matrix, but it performed in the same pairwise manner. This error function encompasses the statistical differences of all pairs of networks. By minimizing it, the parameters were estimated that minimize the said differences. Derivation  
Let \(N\) be a set of all networks. Each network \(n\ \in N\) contains a set of hostnames,  
\(Hn.\) Let \(fn:\ Hn\ \rightarrow\ \mathbb{R}\) be an anomaly score of a hostname \(h\ \in\ Hn\) in network \(n.\)  
Let \(g\theta n\): \(\mathbb{R}\rightarrow\mathbb{R}\) be a normalization function parameterized by \(\theta n.\)  
We wish to derive the parameters \(\theta n\) that normalize best the anomaly scores \(fn(\cdot)\). Using these parameters, it is possible to compute the normalized anomaly scores \(q_{n,h}\):  
\[q_{n,h}\ =\ gn\ (fn\ (h)),\ h\ \in\ Hn \tag{6}\]  
Now, by using the least squares optimization method to find the best parameters \(\theta n\) for every pair of networks \(n\),\(m\), the normalization error function \(En\),\(m\) is:  
\[E_{n,m}=\sum_{h\in Hn\cap Hm}(g\theta n\ fn(h))-g\theta m(fm(h)))^{2}=\sum_{h \in Hn\cap Hm}(q_{n,h}-q_{m,h})^{2} \tag{7}\]  
Here, for simplicity, we substituted anomaly scores in the normalized anomaly scores \(q_{n,h}\). By minimizing \(E_{n,m}\) over \(\theta n\), \(\theta m\), we obtained the normalization parameters \(\theta n\), \(\theta m\) for networks \(n\), \(m\). To compute the global error function \(E\), we can sum the errors \(E_{n,m}\) over all pairs (\(n\), \(m\)):  
\[E=\sum_{n,m\in N,}E_{n,m}=\sum_{n,m\in N,}\ \sum_{h\in Hn\cap Hm}(q_{n,h}-q_{m,h}) ^{2} \tag{8}\]  
This can be re-arranged to sum over hostnames in the final results. By doing so, the descriptions of the summation sets would be complicated, so they will no longer print, and we simply assume that a summand is present only if its value is defined. After re-arranging we get:  
\[E=\sum_{h}\ \sum_{n}\ \sum_{m}(q_{n,h}-q_{m,h})^{2} \tag{9}\]  
where according to the stated convention, \(n\) and \(m\) are summed only over the networks that observe the hostname \(h\). It is now visible that the influence of a single hostname \(h\) is quadratic to the number of networks it is present in (there is a quadratic number of summands referencing this hostname). This is caused by using a pairwise error function instead of an error function comparing to a single baseline. This is certainly not desired; due to the nature of the dataset, it can be expected that the differences between the number of occurrences of different hostnames is large. They would be quadratically exacerbated, and the influence of less frequent host-names would vanish.