# Multimodal Deep Learning for Group Activity Recognition in Smart Office Environments
## 4 Empirical Results
### Unimodal Results Analysis
#### 4.2.1 Visual Modality  
As a starting point we set out to investigate the effects of different camera angles relatively on the activities occurring. In Figures 4 and 5 we plotted the model's accuracy and loss respectively over the course of training. It is interesting to note that regardless of the camera's positioning and field of view, both the overhead and the corner camera achieved a high accuracy; however, the learning curve slope for the overhead camera is slightly steeper than in the case of the corner camera. That is, while the overhead camera achieved 99% accuracy after about 200 epochs, the corner one got to 98% in roughly 250 epochs. In addition, when activity recognition was performed based on video streams from both camera angles, the resulting accuracy was also 99%. Thus, it appears that the overhead camera provided the best field of view for addressing activity recognition in an office setup, while the corner angle did not provide any performance increase. Importantly, in contrast to the previous work [17] on the AMI dataset, our residual learning-based approach outperformed by a margin of 5%.  
Figure 4: Model accuracy based on the visual data from (**a**) the corner camera, (**b**) the overhead camera, and (**c**) the corner and overhead camera angles combined.  
Given that the two classes _presentation_ and _meeting_ clearly display a level of similarity much higher compared to the _empty_ class, which may impact the overall results, we also conducted a separate experiment wherein the classification task was strictly limited to distinguishing between _presentation_ and _meeting_. The accuracy remained the same; Figure 6 presents the confusion matrix, which indicates that the model is more likely to misclassify a meeting as presentation than vice-versa.  
We set out to investigate the performance for the activity classification task based solely on the audio feed. Namely, in Figure 7 we compare several deep learning architectures that showed promising results in [17] on the AMI dataset, against the residual learning approach. Noticeably, as well as in the case of visual data, residual learning outperformed both the VGG-16 [26] and the Inception V3 [27] models, by extracting features that made it easier for the model to classify the audio data correctly. Moreover, the differences in the F1-score are considerably larger, as can be seen in Table 1. At the same time, it is important to point out that audio-based activity recognition did not perform anywhere close to the accuracy attained when using visual data, which is consistent with similar findings reported for activity recognition tasks [16; 28]. Additionally, we mentioned that the results reported here have been obtained based on a transfer learning approach, which is further detailed in Section 4.2.3.  
Figure 5: Model loss based on the visual data from (**a**) the corner camera, (**b**) the overhead camera, and (**c**) the corner and overhead camera angles combined.  
Figure 6: Confusion matrix for the corner and overhead camera angles combined.