# Environmental Feature Correlation and Meta-analysis for Occupancy Detection - A Real-Life Assessment

Asimina Dimara, Alkiviadis Kyrtsoglou, Iakovos Michailidis, Stelios Krinidis,

Elias B. Kosmatopoulos, Dimosthenis Ioannidis, Christos-Nikolaos

Anagnostopoulos, Dimitrios Tzovaras

► To cite this version:

Asimina Dimara, Alkiviadis Kyrtsoglou, Iakovos Michailidis, Stelios Krinidis, Elias B. Kosmatopoulos, et al.. Environmental Feature Correlation and Meta-analysis for Occupancy Detection - A Real-Life Assessment. 18th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2022, Hersonissos, Greece. pp.253-264, 10.1007/978-3-031-08341-9_21. hal-05335647

HAL Id: hal-05335647

https://inria.hal.science/hal-05335647v1

Submitted on 28 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Environmental Feature Correlation and Meta-analysis for Occupancy Detection - A Real-Life Assessment

Asimina Dimara<fnref n="1" />,<fnref n="2" />✉, Alkiviadis Kyrtsoglou<fnref n="1" />✉, Iakovos Michailidis<fnref n="1" />,<fnref n="3" />✉, Stelios Krinidis<fnref n="1" />,<fnref n="4" />✉, Elias B. Kosmatopoulos<fnref n="1" />,<fnref n="3" />✉, Dimosthenis Ioannidis<fnref n="1" />✉, Christos-Nikolaos Anagnostopoulos<fnref n="2" />✉, and Dimitrios Tzovaras<fnref n="1" />✉

**Abstract.** Even though occupancy inference is of utmost importance for numerous real-time and real-life applications a widely-accepted approach to predict occupancy does not exist. In this paper, an assessment of widely-recommended approaches and data processing for occupancy is overviewed. Furthermore, the correlation and meta-analysis between various sensor features like motion sensing, temperature, humidity, and energy consumption were tested. Random Forest classifier a widely-applied artificial model for occupancy inference prediction is evaluated in 4 different real-life data sets including various features. The results of both a univariate and multivariate model are examined. Random Forest classifier results during an experimental phase are presented to reveal the best model. The outcomes of the current research indicate that even in similar spaces data analysis and correlation have different results while the multivariate model is more accurate than the bivariate model.

**Keywords:** Occupancy inference · Environmental sensors · Meta-analysis · Data correlation

## 1 Introduction

Occupancy inference is the process that estimates if a space is occupied or not. Occupancy either reports absence or presence in a room or space, or the number of occupants [1]. Occupancy factor is of key importance for many energy-related applications as it dictates the energy loads inside the building [2]. Specifically, most of the controllable energy saving actions are based on whether the room is

<footnote n="1">¹ Information Technologies Institute, Centre for Research and Technology Hellas, 57001 Thessaloniki, Greece

adimara@iti.gr</footnote>

<footnote n="2">² Department of Cultural Technology and Communication, Intelligent Systems Lab, University of the Aegean, Mytilene, Greece</footnote>

<footnote n="3">³ Electrical and Computer Engineering Department, Democritus University of Thrace, 67000 Xanthi, Greece</footnote>

<footnote n="4">⁴ Management Science and Technology Department, International Hellenic University (IHU), Kavala, Greece</footnote>
occupied [3]. Furthermore, occupancy is important for occupants safety therefore many application are based on motion tracking and alerts [4]. Additionally, occupancy is highly related to Indoor Environmental Quality, occupants well-being and comfort [5].

Consequently, there exist numerous approaches, methods and technologies to infer occupancy for real-life and real-time applications. Furthermore, each of them depends on various and diverse type of data and information. Non Artificial Intelligence (AI) methods use specific devices or hardware means to sense occupancy (e.g., smart-phones, thermal cameras, motion sensors, camera sensor network, etc.) [5]. AI methods use data and information deriving from various environmental sensors (e.g., temperature, illuminance, CO₂ sensors) and energy meters to predict occupancy based on machine learning techniques and models (e.g., regression algorithms, neural networks, classification algorithms, supervised and unsupervised learning) [6].

Nonetheless, the plethora of methods does not reassure that occupancy prediction is a straightforward task and choosing one of the aforementioned methods will guarantee high accuracy. Primarily, sensor raw data is filled with prevalent errors, noise, duplicates, constant stack values and many other faults [7] making it almost impossible to be used as it is. As a result, data pre-processing is necessary to heal sensor data before using and saving it using cleaning tools and applications (e.g., imputation, erase duplicates) [8]. Furthermore, in a smart home there are many sensors installed and there is a huge amount of data streams. Therefore, a data correlation must be performed to facilitate feature selection [9]. There exist many data-driven models for occupancy prediction each using different features [10]. Finally, some methods use a univariate model to predict occupancy [11], while others use a multivariate model instead into12.

Based on the previous analysis, the current paper intends to overview the basic assumptions of occupancy inference from a real-life use case scenario. Initially, data check will be performed to assess the quality and type of errors. Basic techniques like heat-maps and data correlation will be examined and applied on a raw data set [13]. Moreover, the same classification model will be tested using a univariate and a multivariate approach. Eventually, all findings will be analyzed and suggestions will be made for the best approach to use for an accurate occupancy prediction.

The remainder of the paper is structured as follows: Sect. 2 presents all the investigated approaches and widely used data-analysis methods. In Sect. 3 the aforementioned approaches are tested and results are presented. Subsequently, results are drawn in Sect. 4.

## 2 Investigated Approaches and Data Analysis

In this section commonly-used techniques for data pre-processing and data correlation are addressed. Moreover, feature selection methods are examined and the relationship between various environmental features and occupancy inference are evaluated. Eventually, two methods for classification models (i.e., univariate,
<figure><img src="image_3.png" /><figcaption>Fig. 1. Overview work flow</figcaption></figure>

multivariate) is assessed. The overall work flow of the investigated approaches and data analysis is depicted in Fig. 1.

## 2.1 Data Pre-processing

Data pre-processing is a very important process implied in machine learning. Especially, if the data to handled is raw data and not simulated. Some of the most used techniques are:

**Impute Missing Values.** Imputation is the method used to handle all missing data (e.g., NaN, Null, missing timestamps, etc.). Every feature that appears in a data set demands a unique imputation technique based on its behaviour. Specifically, for continuous features (e.g., humidity, temperature, CO₂), gaps may be filled exploiting an interpolation method (e.g., linear interpolation, spline interpolation) [14]. On the other hand, boolean data may be filled with the backward filling technique [15].

**Feature Encoding.** Label encoding [16] is applied to the occupancy state feature to convert the occupants schedule to a numeric format as described below:

$$ occupancy\ state = \begin{cases} 0, & \text{Absent} \\ 1, & \text{Presence} \end{cases} \qquad (1) $$

## 2.2 Data Correlation

Every data set that is retrieved by a sensor monitoring network includes various features (e.g., temperature, humidity, illuminance, CO₂, window state, motion
detection, active power). Eventually, all the retrieved features are used to create a data set to test and fit an appropriate model to predict the selected output. Before creating, testing and fitting the proper machine learning model the features must be cleaned and analyzed. Feature cleaning refers to all the variables that are irrelevant to the problem, therefore they must be omitted from the training data set. Moreover, an analysis between the retrieved variables must be performed to showcase data's patterns. Certain behaviours and relations among the features are revealed by correlation and auto-correlation [20].

Correlation is a metric to estimate the strength of the relationship among two data-set features. Auto-correlation estimates the linear link among a feature and its past values. Specifically, the closer a value is to zero the weakest the correlation is. On the other hand, the closer a value is to 1 or -1 indicates a strongest correlation. Consequently, the correlation and auto-correlation process facilitates features association while giving a clear perspective for how accurate a model will perform based on them.

## 2.3 Bivariate Random Forest Versus Multivariate Random Forest Classification for Occupancy

Random Forest (RF) is a widely-used and accurate tree-based machine learning algorithm. It is really popular for classification predictive modeling data sets especially for occupancy inference as it has proven efficient enough [17]. RF could be exploited for time-series prediction and inference, although a transformation of the timestamp (e.g., timestamp to unix) should be performed first. It could be used following two different approaches [19]:

**Bivariate Time Series RF.** Bivariate is a model that uses two features to predict the output. This type of model associates the relationship and strength of those two variables.

**Multivariate Time Series RF.** A multivariate model exploits multiple features to predict the output. This type of model associates the relationship and strength of multiple features.

# 3 Experimental Results

All the above were tested on a real-case scenario. In 4 almost similar offices (i.e., size, occupancy schedule), sensors and energy analyzers were installed as depicted in Fig. 2. Data streams from the sensors and equipment were pushed periodically every 5 min to a database. No further data process was handled before saving data to the database. As a result, all aforementioned approaches will be tested, using the sensors' and energy analyzers' raw data.
<figure><img src="image_4.png" /><figcaption>Fig. 2. Floor plan of the sensors and energy meters used</figcaption></figure>

In Table 1 all sensors and equipment used for the experiments is presented. For each of the sensors an identifier was used to save data streams. As a result, a variable called coTemperature indicates that this values derives from the CO2 sensor. Furthermore, if there are more than one of the indicated type sensor, a number is added after the identifier (e.g., dw1).

<table><caption>Table 1. Sensors and equipment used for the experiments</caption><thead><tr><th>Type of equipment</th><th>Name</th><th>Values</th></tr></thead><tbody><tr><td>Occupancy schedule</td><td>Occupancy</td><td>0,1 (presence, absence)</td></tr><tr><td>CO2 sensor</td><td>co-</td><td>Humidity<br>Carbon_Dioxide<br>Temperature</td></tr><tr><td>Motion sensor</td><td>ms-</td><td>Illuminance<br>Temperature<br>state: 0,1 (presence, absence)</td></tr><tr><td>Door window</td><td>dwXX-</td><td>Temperature<br>state: 0,1 (close,open)</td></tr><tr><td>Smart meter</td><td>sm-</td><td>Consumption_total</td></tr></tbody></table>
## 3.1 Pre-processing

**Impute Missing Values.** An example of filling missing temperature values in the data-set is depicted in Fig. 3. It may be observed that the complete temperature time-series is accurate enough and may be used as feature for a training data set.

<figure><img src="image_5.png" /><figcaption>Fig. 3. Example of imputation of missing temperature values using interpolation</figcaption></figure>

**Feature Encoding.** An example of feature encoding for occupancy schedule in the data-set is depicted in Fig. 4. It may be observed that the occupancy schedule from the offices was encoded to zero and one for absence and presence respectively.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Example of feature encoding for occupancy schedule</figcaption></figure>
## 3.2 Data Correlation

<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Heat map of all temperature variables from all sensors for all the offices</figcaption></figure>

Initially, a heat map is created for all the different temperatures received from the various sensors (i.e., door window, CO₂, motion sensor) depicted in Fig. 5. Highly correlated features of the same variable may be some how associated as they do not contribute more to the prediction. Specifically, they have the same information quality so merging those features will reduce running speed and storage needs. It may be observed, from Fig. 5, that temperatures in offices 1, 3 and 4 are highly correlated. As a result, the average temperature was used for these offices. On the contrary, in office 2 only temperatures from door window 1 and 2 were highly correlated, so all of the temperatures were used.

Two out of the four offices are depicted in Figs. 6 and 7 indicatively, and present the total analysis of the correlation among the retrieved features from the sensors. Specifically, the diagonal line presents each variable's distribution. While, under the diagonal line the bivariate scatter plots are presented. Moreover, over the diagonal line the probability of no relationship (significance level)
and the correlation values are depicted. Finally, the stars indicate the statistical
significance of the values.

<figure><img src="image_8.png" /><figcaption>Fig. 6. Distribution and correlation characteristics between all features of office 1.</figcaption></figure>

It may be observed that almost all variables of the data-set are statistical
significant for the Occupancy feature for all the offices. Moreover, it may be seen
that features showcase different correlation between the offices indicating that
the features of each office must be selected separately for each one of them.

3.3 Bivariate Classification Results

Bivariate classification was implemented for all the offices and office 3 is indica-
tively presented here due to limited space. In more detail, the features of focus for
this experiment were the timestamp and the occupancy information described
in Sect. 3. Moreover, for the occupancy feature, its own two past values (lags)
were also taken into account. Since we are dealing with time sensitive data, the
information yielded by past values is necessary to preserve the feature’s time
dependence. In addition, due to the nature of the problem (supervised learn-
ing), a pre-processing step was mandatory. RF requires the conversion of some
feature values in order for it to “understand” them.
<figure><img src="image_9.png" /><figcaption>Fig. 7. Distribution and correlation characteristics between all features of office 4.</figcaption></figure>

Specifically, the timestamp of the dataset was converted from python's datetime format to the unix format. For the occupancy, besides the addition of the two lagged versions of itself, all of their values were converted to integers. Before executing the classification, the data were split into train and test sets with a ratio of 80-20 respectively. The purpose of the classifier is to predict whether the office has occupants at a specific moment. For this purpose the RF classifier was used along with some metrics for the evaluation of the output results. The most commonly used features in a classification are the precision, recall and f1 score [21]. Table 2 depicts the results for the case of office 3. The high precision score and the very low score on recall metric, for the 'occupancy' class, indicates a small percentage of correctness for this class which means that further investigation is needed using these features.

<table><caption>Table 2. Random forest classifier results for the bivariate case</caption><thead><tr><th></th><th>Precision</th><th>Recall</th><th>f1-score</th></tr></thead><tbody><tr><td>Not_occupancy</td><td>0.78</td><td>1</td><td>0.87</td></tr><tr><td>Occupancy</td><td>1</td><td>0.01</td><td>0.02</td></tr><tr><td>Accuracy</td><td></td><td></td><td>0.78</td></tr><tr><td>Macro avg</td><td>0.89</td><td>0.51</td><td>0.45</td></tr><tr><td>Weighted avg</td><td>0.83</td><td>0.78</td><td>0.68</td></tr></tbody></table>
## 3.4 Multivariate Classification Results

Likewise to the previous section, office 3 is also presented here. Similar to the bivariate case, the multivariate experiments included feature selection, pre-processing and a training and an evaluation step. As the name multivariate implies, multiple features were selected to train the classifier. These include the same features as the ones on the bivariate experiment and data from other sensors located in the office room. Sensors, such as, the motion sensor which detects movement in the room, the CO dioxide sensor which detects the level CO dioxide in the room, along with the door-window sensors that indicate if a window or door is open at a certain time. Finally, total energy consumption is included, measuring the total electric energy consumption of the room, and thus revealing presence in the room. The selection of these features was based on their impact on occupancy based on the data analysis in Sect. 3.

The aforementioned features along with their lagged values, from up to two previous timesteps, comprise the whole dataset for the multivariate case. Likewise, the test and train sets were defined with a ratio of 80-20 respectively and the target of prediction remained the occupancy feature. The results from the random forest classifier with a multivariate input are shown on Table 3 revealing an improvement on the recall metric for the class "no_occupancy", which leads to an improvement to the overall accuracy of the algorithm in comparison with the bivariate case.

<table><caption>Table 3. Random Forest Classifier results for the multivariate case</caption><thead><tr><th></th><th>Precision</th><th>Recall</th><th>f1-score</th></tr></thead><tbody><tr><td>Not_occupancy</td><td>0.99</td><td>1</td><td>0.99</td></tr><tr><td>Occupancy</td><td>0.98</td><td>0.97</td><td>0.98</td></tr><tr><td>Accuracy</td><td></td><td></td><td>0.99</td></tr><tr><td>Macro avg</td><td>0.99</td><td>0.98</td><td>0.99</td></tr><tr><td>Weighted avg</td><td>0.99</td><td>0.99</td><td>0.99</td></tr></tbody></table>

## 4 Conclusions

The experiments on the occupancy detection as well as the data analysis of all the available sensor data yielded interesting results regarding the effect of similar datasets on different locations. One observation regarding the selection of features concerns the correlation between them. Typically, highly correlated features can be merged to one (e.g. average value) or can be discarded, as there is not additional information that can be shared. On the other hand, such correlation may or may not be observed on, intuitively, same data. Results indicate this statement, since office 2 temperature heatmap depicts temperature features with no correlation in contrast to the remaining offices, which have high correlated data.
A similar observation is based on the correlation results. Each office presents a different set of features which are affected by the occupancy attribute. In other words, although every office produces data from similar sensors, each case differs and thus the feature selection procedure cannot be the same for all of them.

In conclusion, the indicative classification results for the occupancy detection problem, are distinct between the two approaches, bivariate and multivariate. Using the date information and the past values of the occupancy feature produced not so satisfying results with an accuracy of 0.78 but a very low recall ratio. On the contrary the augmentation of the feature set with environmental and energy related features helped the random forest model raise its metrics indicating the importance of the additional information provided by the added features. Still, further research is required to compare different aspects of the whole procedure (e.g. different data imputation methods), to conclude to a more accurate outcome. Lastly, an undeniable observation is that when dealing with similar datasets, depending on the context, the behavior of the data may vary substantially, rendering separate investigation mandatory.

In the future, we intend to work on further improving and optimizing the data analysis procedures. One scope is the feature analysis automization, where highly correlated features will be merged automatically, and another scope is the feature selection automization where only relevant features will automatically be selected after analysis, meaning features with correlation above 0.5 or under -0.5.

**Acknowledgements.** The research leading to these results was partially funded by the European Commission "LC-EEB-07-2020 - Smart Operation of Proactive Residential Buildings" - PRECEPT H2020 project (Grant agreement ID: 958284) https://www.precept-project.eu/, accessed on 8 March 2022; and "LC-SC3-B4E-3-2020 Upgrading smartness of existing buildings through innovations for legacy equipment" - Smart2B H2020 project (Grant agreement ID: 101023666) https://www.smart2b-project.eu/, accessed on 2 March 2022.

## References

1. Dimara, A., Krinidis, S., Tzovaras, D.: occupI: a novel non-intrusive occupancy inference tool. In: 2020 International Conferences on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics). IEEE (2020)

2. Salimi, S., Hammad, A.: Critical review and research roadmap of office building energy management based on occupancy monitoring. Energy Build. **182**, 214–241 (2019)

3. Jafarinejad, T., et al.: Bi-level energy-efficient occupancy profile optimization integrated with demand-driven control strategy: university building energy saving. Sustain. Cities Soc. **48**, 101539 (2019)

4. Luo, K., et al.: Safety-oriented pedestrian occupancy forecasting. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE (2021)
5. Jung, W., Jazizadeh, F.: Human-in-the-loop HVAC operations: a quantitative review on occupancy, comfort, and energy-efficiency dimensions. Appl. Energy **239**, 1471–1508 (2019)

6. Rueda, L., et al.: A comprehensive review of approaches to building occupancy detection. Build. Environ. **180**, 106966 (2020)

7. Wang, X., Wang, C.: Time series data cleaning: a survey. IEEE Access **8**, 1866–1881 (2019)

8. Kenda, K., Mladenić, D.: Autonomous sensor data cleaning in stream mining setting. Bus. Syst. Res. Int. J. Soc. Adv. Innov. Res. Econ. **9**(2), 69–79 (2018)

9. Saha, H., et al.: Occupancy sensing in buildings: a review of data analytics approaches. Energy Build. **188**, 278–285 (2019)

10. Golestan, S., Kazemian, S., Ardakanian, O.: Data-driven models for building occupancy estimation. In: Proceedings of the Ninth International Conference on Future Energy Systems (2018)

11. Kutafina, E., et al.: Recursive neural networks in hospital bed occupancy forecasting. BMC Med. Inform. Decis. Mak. **19**(1), 1–10 (2019)

12. Hepler, S.A., Erhardt, R.J.: A spatiotemporal model for multivariate occupancy data. Environmetrics **32**(2), e2657 (2021)

13. Barter, R.L., Bin, Yu.: Superheat: an R package for creating beautiful and extendable heatmaps for visualizing complex data. J. Comput. Graph. Stat. **27**(4), 910–922 (2018)

14. Verma, V., et al.: Interpolation consistency training for semi-supervised learning. arXiv preprint arXiv:1903.03825 (2019)

15. Saad, M., et al.: Machine learning based approaches for imputation in time series data and their impact on forecasting. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE (2020)

16. Jia, B.B., Zhang, M.L.: Multi-dimensional classification via decomposed label encoding. IEEE Trans. Knowl. Data Eng. (2021)

17. Amayri, M., et al.: Estimating occupancy in heterogeneous sensor environment. Energy Build. **129**, 46–58. Zeigermann, Lars (2016)

18. TIMESTAMP: Stata module to obtain a UNIX timestamp and the current time of a user-specified timezone (2016)

19. Denis, D.J.: SPSS Data Analysis for Univariate, Bivariate, and Multivariate Statistics. John Wiley & Sons, Hoboken (2018)

20. Liebhold, A.M., Sharov, A.A.: Testing for Correlation in the Presence of Spatial Autocorrelation in Insect Count Data. Population and Community Ecology for Insect Management and Conservation, pp. 111–118. CRC Press, Boca Raton (2020)

21. Tharwat, A.: Classification assessment methods. Appl. Comput. Inform. **17**(1), 168–192 (2021). https://doi.org/10.1016/j.aci.2018.08.003