# Using Common Spatial Patterns to Select Relevant Pixels for Video Activity Recognition
## 4 Experimental Setup
### Experiments  
In this section the performed experiments are presented. They were developed using the technique introduced in Section 3, where the main idea is to consider the temporal sequence of pixel \(i,j\) as a signal to be fed to the CSP. Taking that algorithm as a basis, different methods have been computed to make a comparison between them and see which one performs better.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Videos** & **Frames/Video** & **Frames/Class** \\ \hline Brush hair & 107 & 648 & 69,336 \\ \hline Fencing & 116 & 301 & 34,916 \\ Walk & 282 & 534 & 150,588 \\ Punch & 126 & 502 & 63,252 \\ Smoke & 109 & 445 & 48,505 \\ Cartwheel & 103 & 132 & 13,596 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Data-set class details.  
Modalities  
CSP algorithm was used in several modalities. The main change between the performed tests consists of deciding which pixels of the image are used to make the signals (channels) to calculate the CSP.  
* Separation in quadrants. Taking into account that different actions have to be recognized, where in the video they occur should be taken into account, that is, in which area of the window. In many of the sequences used, the camera is static and the individuals performing the interaction appear almost centered. In these cases, it can be supposed that if the action they are performing is, for instance, _smoke_, it will be happening at the top of the images, while if the action is _kick_, it will be happening at the bottom part. In order to consider this approach, it was decided to divide each frame of the video in 16 quadrants as can be seen in Figure 3, and perform the whole classification pipeline for every one of them. Thus, each classifier focuses on an exact area of the videos. The final prediction is the result of the majority voting of these 16 classifiers.
* Pixels with maximum variance. CSP works with the variance of each channel. In some of the video sequences some objects from the scene, such as the background, are static, and do not change over time. The pixels corresponding to these static areas will produce a zero (or near zero, due to random fluctuations in pixel intensity) variance. This, apart from not providing useful information, can cause some problems at the time of executing the algorithm due to the calculation of the logarithm of the variances in the computation of CSP, yielding negative infinite values. In this case, it was decided to first select a group of pixels to extract the features, these are the pixels that change the most over time and therefore have the maximum variances. The hypothesis here is that they are the best candidates to represent the performed action. To select these pixels, the frames were transformed to grayscale, in order to have just one channel when calculating the variance of each pixel. In Figure 4 an example can be seen, where the 25 most relevant pixels have been selected for each quadrant.  
Figure 3: The division of a 25 \(\times\) 25 frame in 16 quadrants.  
Once the relevant pixels are selected and the quadrant separation has been decided, the classification is performed using different features extracted after the CSP filter. The main focus of the experimentation is the use of the variances of the signals after applying the Common Spatial Pattern filter. However, apart from the variances, many other information can be extracted from these transformed signals. Hence, some experiments are performed with just the information of the variances and other experiments also with information about the maximum and minimum values of the signal and the interquartile range (\(IQR=Q3-Q1\)). This information may be useful when performing the classification, and a comparison has been made between the results obtained by these two ways of performance:  
* var
* var, min, max, IQR taken together