# Analyzing Brain Waves for Activity Recognition of Learners

Hiromichi Abe, Kazuya Kinoshita, Kensuke Baba, Shigeru Takano, Kazuaki Murakami

► To cite this version:

Hiromichi Abe, Kazuya Kinoshita, Kensuke Baba, Shigeru Takano, Kazuaki Murakami. Analyzing Brain Waves for Activity Recognition of Learners. 3rd International Conference on Information and Communication Technology-EurAsia (ICT-EURASIA) and 9th International Conference on Research and Practical Issues of Enterprise Information Systems (CONFENIS), Oct 2015, Daejon, South Korea. pp.64-73, 10.1007/978-3-319-24315-3_7. hal-01466252

HAL Id: hal-01466252

https://inria.hal.science/hal-01466252

Submitted on 13 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Analyzing Brain Waves for Activity Recognition of Learners

Hiromichi Abe, Kazuya Kinoshita, Kensuke Baba, Shigeru Takano, and Kazuaki Murakami

Kyushu University, Fukuoka, Japan
hiromichi.abe@soc.ait.kyushu-u.ac.jp

**Abstract.** Understanding the states of learners at a lecture is expected to be useful for improving the quality of the lecture. This paper is trying to recognize the activities of learners by their brain wave data for estimating the states. In analyses on brain wave data, generally, some particular bands such as $\alpha$ and $\beta$ are considered as the features. The authors considered other bands of higher and lower frequencies to compensate for the coarseness of simple electroencephalographs. They conducted an experiment of recognizing two activities of five subjects with the brain wave data captured by a simple electroencephalograph. They applied support vector machine to 8-dimensional vectors which correspond to eight bands on the brain wave data. The results show that considering multiple bands yielded high accuracy compared with the usual features.

## 1 Introduction

Understanding the states or emotions of learners is useful for improving the quality of the lecture. Learners' states are usually observed visually by the lecturer at the time or estimated by exams conducted after the lecture as a degree of understanding. However, especially in massive or on-line lectures, it is difficult to know the state of each learner at the lecture. Detecting learners' states automatically and in real time is expected to innovate in the current style of learning.

The aim of our research is to detect the states of learners from several kinds of sensor data. On the assumption that some particular activities of a person correspond to the states, we try to recognize activities to understand the states of learners. Activity recognition has been widely explored using video data. Recently, a spread of depth sensors enabled us to analyze the subject in terms of 3-dimensional coordinates [4]. A camera with a depth sensor can recognize some motions, facial expressions, and eye movements with high accuracy. However, the target of our study is learners who are sitting with small moves, and hence we need another kind of sensor data in addition to image data for activity recognition with high accuracy.

We focused on brain waves, electroencephalography (EEG), captured by a simple electroencephalograph for activity recognition of learners as additional data to video data. Generally, it is said that EEG changes by the influence of
the emotion of the person. However, usual electroencephalographs have a lot of electrodes and are not suitable for use by learners in a lecture. Moreover, some expert systems are expensive and unsuitable for a large amount of use. Especially for massive lectures, electroencephalographs are required to be easy to use and cheap.

In this paper, we evaluated the validity of using multiple bands of brain wave data captured by a simple electroencephalograph for activity recognition of learners. In usual researches of activity or emotion recognition by EEG data, some particular bands, called α and β waves, are used as features for analysis. We considered other bands of higher and lower frequency in addition to the usual features, α and β waves and their combinations, to improve low accuracy of the activity recognition by simple electroencephalographs. In our experiment, the subjects performed two activities, calculation and meditation, with a simple electroencephalograph which can output a spectrum of brain waves with the 8 kinds of bands. Then, we applied support vector machine (SVM) [5] to the 8-dimensional vectors, and investigated the accuracy of the activity recognition by comparison with scalar data based on the values of one or two bands.

As a result, the accuracy of the activity recognition by the data of the 8 kinds of bands was high compared with considering one or two features. Additionally, we confirmed that the simple electroencephalograph has a possibility of use for the activity recognition of learners.

The rest of this paper is organized as follows. Section 2 surveys related work of activity recognition of learners with EEG data. Section 3 describes the activities, the collected brain wave data, and the data analysis in our experiments. Section 4 reports the results of the experiments. Section 5 shows considerations about the results and future directions of our study.

## 2 Related Work

Our study is considering EEG data of learners captured by a simple electroencephalograph for state recognition.

A viewpoint of this survey is EEG data of learners. Some studies indicated a possibility of state or emotion recognition of learners by analyzing their EEG data. Wan et al. [11] investigated the characteristics of EEG data of learners in a Web-based lecture. Chaouachi and Frasson [3] showed a relation between EEG data of learners and the response time to some questions. These studies were using electroencephalographs with many electrodes rather than simple one.

The other viewpoint is EEG data by a simple electroencephalograph. Electroencephalographs with a few (dry) electrodes are less accurate than expert one, but those are easy to use for learners. Ishino and Hagiwara [6] proposed a system which estimates the feeling of subjects by EEG data captured by a simple electroencephalograph with three electrodes. Vourvopoulos and Liarokapis [10] evaluated a usability of a commercial and simple electroencephalograph for brain-computer interfaces. These studies were not necessarily considering learners as the target.
The study by Yoshida et al. [13] is similar to ours in the sense of the two viewpoints. They considered the changes of EEG data to evaluate the states of learners who repeated simple calculations. Then, they reported that the ratio of $\alpha$ wave to $\beta$ wave was related to the activity in their experiment. Some recent studies investigated more complex relations between emotional states and the range of brain waves rather than the discriminated one such as the relation between relaxation and $\alpha$ wave. For example, Wang et al. [12] classified EEG data into some categories based on emotional states by machine learning. In our experiment, we considered the difference between two kinds of activities and compared the other kinds of waves in EEG data systematically in addition to $\alpha$ and $\beta$ waves and the combinations of the waves.

# 3 Methods

The purpose of this paper is to clarify whether considering multiple bands of brain wave is useful for activity recognition compared with usual features based on a single band or a scalar from a few bands. We collected brain wave data of persons with two activities, and tried to recognize the activities from the data with considering several kinds of features.

## 3.1 Collected Data

We collected brain wave data of five subjects with the two activities:

- Calculation of simple additions and

- Meditation with the eyes closed.

We used 10 × 10 matrixes with randomly chosen 20 single-digit numbers (Fig. 1) for the activity of calculation. We prepared a sufficient number of matrixes printed on papers, and the subjects tried to fill the matrixes as much as possible in each trial. The time for a single trial of an activity was 2 minutes, and the subjects repeated the two activities alternately 5 times for each with no interval. Therefore, the total time of the experiment for a subject was 20 minutes.

We used a simple electroencephalograph MindWave Mobile (Fig. 2) by NeuroSky<fnref n="1" /> for collecting brain wave data. The simple electroencephalograph has only two electrodes and does not need any gel on the head of a subject for capturing potential differences. One of the electrodes is placed at front polar (the middle point of Fp1 and Fp2 in the international 10-20 system) and the other is placed at left earlobe (A1 in the international 10-20 system). The electroencephalograph (and an attached module) can output a spectrum of brain waves with the following 8 bands:

- $\delta$: 0.5-2.75Hz,

<footnote n="1">¹ NeuroSky, MindWave Mobile, http://mindwavemobile.neurosky.com/ (accessed in May 2015).</footnote>
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** An example of the matrixes used for the activity of calculation. The subject fills each blank with the sum of the corresponding numbers.</figcaption></figure>

<table><caption>**Table 1.** The numbers of the vectors for the five subjects and the two activities in the sample set.</caption><thead><tr><th></th><th>Calculation</th><th>Meditation</th><th>Total</th></tr></thead><tbody><tr><td>Subject 1</td><td>602</td><td>605</td><td>1,207</td></tr><tr><td>Subject 2</td><td>602</td><td>625</td><td>1,227</td></tr><tr><td>Subject 3</td><td>602</td><td>609</td><td>1,211</td></tr><tr><td>Subject 4</td><td>604</td><td>622</td><td>1,226</td></tr><tr><td>Subject 5</td><td>601</td><td>628</td><td>1,229</td></tr><tr><td>Total</td><td>3,011</td><td>3,089</td><td>6,100</td></tr></tbody></table>

- θ: 3.5–6.75Hz,

- Low-α: 7.5–9.75Hz,

- High-α: 10–11.75Hz,

- Low-β: 13–16.75Hz,

- High-β: 18–29.75Hz,

- Low-γ: 31–39.75Hz, and

- Mid-γ: 41–49.75Hz.

The sampling frequency was 512Hz. The output rate of the spectrum was about 1 for a second. Therefore, we obtained about 6,000 (=$60 \times 20 \times 5$) 8-dimensional vectors with the label for one of the two activities. Table. 1 shows the numbers of the generated vectors for the five subjects and the two activities. We simultaneously took video data of the subject for adjusting the time of each trial to the labels.

## 3.2 Data Analysis

We tried to recognize the two activities from the collected data. First, we considered the following 19 kinds of scalars:
<figure><img src="image_3.png" /><figcaption>Fig. 2. The simple electroencephalograph MindWave Mobile used for collecting brain wave data. The picture was downloaded from http://mindwavemobile.neurosky.com/ in May 2015.</figcaption></figure>

- Each element of the 8 bands,

- The sum of low-$\alpha$ and high-$\alpha$,

- The sum of low-$\beta$ and high-$\beta$, and

- The ratio of $\beta$ to $\alpha$ for the 9 combinations of (low-$\alpha$, high-$\alpha$, and the sum) and (low-$\beta$, high-$\beta$, and the sum).

Next, we considered the 8-dimensional vectors by the 8 elements.

Additionally, we considered the simple moving averages of the collected data with ranges 10 and 20. The simple moving averages of a time series data $v_t$ with a range $n$ is the time series data

$$u_t = \frac{v_t + v_{t-1} + \cdots + v_{t-n+1}}{n}.$$

Then, we obtained two data sets $S_{10}$ and $S_{20}$ in addition to the collected data set $S_1$.

For the 20 cases, we applied SVM with a linear kernel to the three data sets. Concretely, we used the function `ksvm` in the package `kernlab` [8] for $R^2$ with the default values for the parameters. Then, we conducted 5-fold cross-validation [2], where the accuracy of an activity recognition is the ratio of the numbers of the vectors whose label was predicted correctly to the total number of the vectors. We set each test set in the cross-validation to correspond to a single trial of a subject, because vectors in the same trial tend to have similar values. The activity recognition was conducted for each subject separately, and the accuracy was calculated as the mean of the results for the 5 subjects.

<footnote n="2">² The R Project for Statistical Computing, http://www.r-project.org/ (accessed in May 2015).</footnote>
<table><caption>**Table 2.** The accuracy of the activity recognition by SVM for the 20 kinds of features with the three data sets $S_1$, $S_{10}$, and $S_{20}$.</caption><thead><tr><th></th><th>S<sub>1</sub></th><th>S<sub>10</sub></th><th>S<sub>20</sub></th><th></th></tr></thead><tbody><tr><td>8 bands</td><td><strong>79.1</strong></td><td><strong>86.3</strong></td><td><strong>86.3</strong></td><td>(%)</td></tr><tr><td>δ</td><td>69.1</td><td>78.7</td><td>76.6</td><td></td></tr><tr><td>θ</td><td>64.7</td><td>78.3</td><td>79.0</td><td></td></tr><tr><td>Low-α (α<sub>1</sub>)</td><td>55.3</td><td>64.7</td><td>66.3</td><td></td></tr><tr><td>High-α (α<sub>2</sub>)</td><td>57.2</td><td>68.3</td><td>72.9</td><td></td></tr><tr><td>Low-β (β<sub>1</sub>)</td><td>57.4</td><td>68.8</td><td>69.9</td><td></td></tr><tr><td>High-β (β<sub>2</sub>)</td><td>61.4</td><td>71.4</td><td>73.2</td><td></td></tr><tr><td>Low-γ</td><td>69.6</td><td>81.2</td><td>80.0</td><td></td></tr><tr><td>Mid-γ</td><td>68.8</td><td>78.1</td><td>76.4</td><td></td></tr><tr><td>α<sub>3</sub> = α<sub>1</sub> + α<sub>2</sub></td><td>54.2</td><td>62.0</td><td>64.1</td><td></td></tr><tr><td>β<sub>3</sub> = β<sub>1</sub> + β<sub>2</sub></td><td>60.6</td><td>71.6</td><td>72.0</td><td></td></tr><tr><td>β<sub>1</sub>/α<sub>1</sub></td><td>50.4</td><td>51.0</td><td>50.4</td><td></td></tr><tr><td>β<sub>2</sub>/α<sub>1</sub></td><td>51.9</td><td>57.4</td><td>55.2</td><td></td></tr><tr><td>β<sub>3</sub>/α<sub>1</sub></td><td>51.5</td><td>53.7</td><td>53.8</td><td></td></tr><tr><td>β<sub>1</sub>/α<sub>2</sub></td><td>57.5</td><td>67.8</td><td>70.1</td><td></td></tr><tr><td>β<sub>2</sub>/α<sub>2</sub></td><td>59.6</td><td>70.4</td><td>72.1</td><td></td></tr><tr><td>β<sub>3</sub>/α<sub>2</sub></td><td>59.7</td><td>71.0</td><td>72.6</td><td></td></tr><tr><td>β<sub>1</sub>/α<sub>3</sub></td><td>56.8</td><td>65.5</td><td>67.0</td><td></td></tr><tr><td>β<sub>2</sub>/α<sub>3</sub></td><td>59.8</td><td>71.1</td><td>70.5</td><td></td></tr><tr><td>β<sub>3</sub>/α<sub>3</sub></td><td>59.6</td><td>70.7</td><td>70.6</td><td></td></tr></tbody></table>

# 4 Results

Table 2 shows the accuracy of the activity recognition for the 20 cases with the three data sets. The accuracy by considering the 8 bands was higher than other 19 cases for any data set, the accuracy was the highest (86.3%) with $S_{10}$ and $S_{20}$. The accuracy by low-$\gamma$ was the highest in the 8 bands for the three data set. The accuracy by considering the ratio of $\beta$ to $\alpha$ was not high in any combination even compared with each case of the 8 bands. The graphs for some cases showed 5 peaks which correspond to the change of the activities. Fig. 3 is an example of the graph for the change of brain wave data over time. We can see from the graph that the data of low- and mid-$\gamma$ divide the two activities.

# 5 Discussion

## 5.1 Major Conclusion

We found that the two activities could be recognized with high accuracy by considering multiple bands in brain wave data compared with usual features. At the same time, we confirmed that brain wave data obtained by the simple electroencephalograph includes useful information for this kind of activity recognition.
<figure><img src="image_4.png" /><figcaption>**Fig. 3.** The changes of the brain data over time, where the values are for the 8 bands with the simple moving average of the range 20 of a subject.</figcaption></figure>

## 5.2 Key Findings

The optimal length of the range of the simple moving average was 16. By Table. 2, the accuracy was improved when we considered the average of the data for 10 or 20 seconds for any criterion. We additionally investigated the relation between the accuracy and the range of the simple moving average for the case of the 8 dimensional vectors. Figure 4 shows the accuracy against the range. The accuracy was the highest (86.9%) when the range was 16. Considering the average of data for about 20 seconds seems to be suitable for the activity recognition, which corresponds to the same analysis on another kind of activity recognition with EEG by a simple electroencephalograph [1].

We experimentally conducted personal identification by the collected data. In the experiments, we observed differences of brain waves between the subjects. Table 3 shows the results of the personal identification. We applied SVM with a linear kernel and 5-folds cross-validation as the activity recognition. We could not earn high accuracy for the personal identification with the data. The accuracy by the data for the activity calculation (43.9%) was better than meditation (28.8%). The results indicate that we need other techniques to utilize the EEG data for personal identification.

## 5.3 Future Directions

The original goal of our research is to recognize the states of learners such as being concentrated or absentminded. In this paper we found that the collected
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** The accuracy of the activity recognition against the range of the simple moving average with the 8 dimensional vectors.</figcaption></figure>

<table><caption>**Table 3.** The confusion matrix of the personal identification with the five subjects by $S_{20}$.</caption><thead><tr><th>Predicted\Actual</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>Precision (%)</th><th>F-measure (%)</th></tr></thead><tbody><tr><td>Subject 1</td><td>83</td><td>106</td><td>42</td><td>27</td><td>84</td><td>24.3</td><td>29.1</td></tr><tr><td>Subject 2</td><td>66</td><td>79</td><td>2</td><td>50</td><td>26</td><td>35.4</td><td>35.1</td></tr><tr><td>Subject 3</td><td>14</td><td>18</td><td>70</td><td>30</td><td>14</td><td>47.9</td><td>37.5</td></tr><tr><td>Subject 4</td><td>22</td><td>11</td><td>113</td><td>90</td><td>7</td><td>37.0</td><td>38.3</td></tr><tr><td>Subject 5</td><td>42</td><td>13</td><td>0</td><td>30</td><td>96</td><td>53.0</td><td>47.1</td></tr><tr><td>Recall (%)</td><td>38.6</td><td>34.8</td><td>30.8</td><td>39.6</td><td>42.3</td><td>36.8</td><td></td></tr></tbody></table>

brain wave data could divide the two activities, calculation and meditation, with a degree of accuracy, however there are some deviations between the activities and the states of the subjects. For example, we can consider the following noises: the brain wave data might have effects of (1) the move of the hands and (2) whether the eyes were closed. Therefore, our future work includes extra experiments with other activities such as

(1) Calculation without writing, or moving the hand and

(2) Meditation with the eyes opened, or opening and closing the eyes.

We are also planning to use EEG data for activity recognition with data collected by other sensors. Kamizono et al. [7] conducted a similar experiment of activity recognition of learners using video data with depth information. They considered four activities with great moves of the face (meditation, reading texts, looking away, and talking to another person). Moreover, it is possible to capture the heart rate of the subject from video data [9]. Therefore, we can expect improvement of the accuracy of activity recognition by combining these kinds of data with EEG data.
# 6 Conclusion

We evaluated the validity of using brain wave data with multiple bands for activity recognition of learners in a lecture. We conducted the experiments of activity recognition for two activities, calculation of simple additions and meditation with the eyes closed, with a simple electroencephalograph. We found that two activities could be recognized with high accuracy by considering multiple bands in the collected brain wave data compared with usual features. Additionally, we confirmed that brain wave data obtained by the simple electroencephalograph includes useful information for this kind of activity recognition.

# Acknowledgment

This work was partially supported by a joint research with Panasonic Corporation from 2014 to 2015 and the Center of Innovation Program from Japan Science and Technology Agency, JST.

# References

1. H. Abe, K. Baba, S. Takano, and K. Murakami. Towards activity recognition of learners by simple electroencephalographs. In *Proceedings of the International Conference on Information Systems and Design of Communication*, ISDOC '14, pages 161–164, New York, NY, USA, 2014. ACM.

2. C. M. Bishop. *Pattern Recognition and Machine Learning (Information Science and Statistics)*. Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006.

3. M. Chaouachi and C. Frasson. Exploring the relationship between learner EEG mental engagement and affect. In *Proceedings of the 10th International Conference on Intelligent Tutoring Systems - Volume Part II*, ITS’10, pages 291–293, Berlin, Heidelberg, 2010. Springer-Verlag.

4. L. Chen, H. Wei, and J. Ferryman. A survey of human motion analysis using depth imagery. *Pattern Recogn. Lett.*, 34(15):1995–2006, Nov. 2013.

5. T. Hastie, R. Tibshirani, and J. Friedman. *The elements of statistical learning: data mining, inference and prediction*. Springer, 2 edition, 2008.

6. K. Ishino and M. Hagiwara. A feeling estimation system using a simple electroencephalograph. In *Systems, Man and Cybernetics, 2003. IEEE International Conference on*, volume 5, pages 4204–4209 vol.5, Oct 2003.

7. T. Kamizono, H. Abe, K. Baba, S. Takano, and K. Murakami. Towards activity recognition of learners by kinect. In *Advanced Applied Informatics (IIAIAAI), 2014 IIAI 3rd International Conference on*, pages 177–180, Aug 2014.

8. A. Karatzoglou, A. Smola, K. Hornik, and A. Zeileis. *kernlab – an S4 package for kernel methods in R*. *Journal of Statistical Software*, 11(9):1–20, 2004.

9. M.-Z. Poh, D. J. McDuff, and R. W. Picard. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. *Opt. Express*, 18(10):10762–10774, May 2010.

10. A. Vourvopoulos and F. Liarokapis. Evaluation of commercial brain-computer interfaces in real and virtual world environment: A pilot study. *Computers & Electrical Engineering*, 40(2):714–729, 2014.
11. J. Wan, B. Hu, and X. Li. EEG: A way to explore learner's affect in pervasive learning systems. In P. Bellavista, R.-S. Chang, H.-C. Chao, S.-F. Lin, and P. Sloot, editors, *Advances in Grid and Pervasive Computing*, volume 6104 of *Lecture Notes in Computer Science*, pages 109–119. Springer Berlin Heidelberg, 2010.

12. X.-W. Wang, D. Nie, and B.-L. Lu. Emotional state classification from EEG data using machine learning approach. *Neurocomput.*, 129:94–106, Apr. 2014.

13. K. Yoshida, H. Hirai, Y. Sakamoto, and I. Miyaji. Evaluation of the change of work using simple electroencephalography. *Procedia Computer Science*, 22:855–862, 2013. 17th International Conference in Knowledge Based and Intelligent Information and Engineering Systems - KES2013.