<figure><img src="image_1.png" /></figure>

FRANÇOIS CHARPILLET, STÉPHANE PLOIX, PATRICK REIGNIER

Introduction (EN)

Volume 4, n° 1 (2023), p. 11-19.

https://doi.org/10.5802/roia.48en

© Les auteurs, 2023.

<figure><img src="image_2.png" /></figure>

Cet article est diffusé sous la licence

CREATIVE COMMONS ATTRIBUTION 4.0 INTERNATIONAL LICENSE.

http://creativecommons.org/licenses/by/4.0/

<figure><img src="image_3.png" /></figure>

La Revue Ouverte d'Intelligence Artificielle est membre du
Centre Mersenne pour l'édition scientifique ouverte

www.centre-mersenne.org

e-ISSN : 2967-9672
# Introduction (EN)

François Charpillet<sup>a</sup>, Stéphane Ploix<sup>b</sup>, Patrick Reignier<sup>c</sup>

<sup>a</sup> Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France
E-mail: francois.charpillet@inria.fr

<sup>b</sup> Univ. Grenoble Alpes, CNRS, Grenoble INP, G-SCOP, 38000 Grenoble, France
E-mail: stephane.ploix@grenoble-inp.fr

<sup>c</sup> Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France
E-mail: patrick.reignier@univ-grenoble-alpes.fr.

## 1. A BIT OF HISTORICAL CONTEXT

The smart home was one of the first applications of personal computing. The Echo IV computer, designed in 1966 by a Westinghouse engineer and never commercialized, was intended to control the heating of the house or to manage the shopping list. The arrival of microprocessors and the miniaturization of components allowed the development of commercial products and the arrival of home automation in the 1980's. Thanks to a law passed by Congress in 1984, it became possible for American companies to cooperate in the field of research and development (but not marketing) in private consortiums without violating antitrust laws. That same year, the National Association of Home Builders (NAHB) created the “Smart House” consortium to promote and develop the smart home. In 1987, the NAHB research laboratory at Bowie University in Maryland created the first house for their own research. Large companies like Honeywell also got into the game with their own lab in Golden Valley, Minnesota. The enthusiasm is strong for these new technologies. An executive with the Smart House consortium predicts that one in two new homes built will be smart by 1997 [25].

The vision at that time was a better management of resources (energy savings) and scenarios based on improving comfort by delegating everyday actions to the system. The Chicago Tribune, for example, in an article published in 1991, illustrates the intelligent house by [29]: "Imagine that it lights your way to the bathroom in the morning while it opens the curtains, heats the coffee and fills your bathtub to the right temperature. In the evening, you call from your car phone and ask the house to select good music, turn up the thermostat and heat up the roast." Lutolf [20] defined the smart home in 1992 as the integration of different services within a home using a communication system. These services ensure economical, safe and comfortable operation of the house and include a high degree of "smart" functionality and flexibility.
Despite the presence of major industrial groups and significant press coverage, the commercial success is not there. The approaches are too techno-centric. The devices deployed are often proprietary, difficult to interoperate, complex to configure and sometimes requiring external intervention to do so. As Trulove [33] explains, under the cover of simplifying the lives of occupants and giving them better control of their homes, the opposite effect was often obtained: multiplication of passwords and remote controls, impossibility of controlling a device when losing of one of these remote control, etc. "Initially enthusiastic, those high -end home owners were reporting that they would try to use the smart house system once or twice and then give up on it. They would never bother programming the software. Or they would not repair the system if it broke down."

## 1.1. A SECOND WIND

At the beginning of the 2000's, Ambient Intelligence, supported in particular by Europe [10], put Human back at the center of technological developments. The objective is the development of an “intelligent” and “ubiquitous” digital environment that helps users in their daily, personal and professional tasks. In parallel, the technical devices deployed in homes for the perception of the environment (temperature sensors, CO², cameras etc. ) or to control this environment (thermostats, roller shutters, multimedia), until now limited at best to the perimeter of the house, are now opening up far beyond and are beginning to become interoperable through their connection to the Internet, either directly (IP cameras, for example), or through hardware and/or software gateways such as openHAB (1) or, more recently, Jeedom (2), for example. The massive arrival of connected objects has given a second wind to the smart home. Since 2013, it has been a major feature at CES. In 2014, the market was structured by the acquisition of startups by major IT players. The field has been steadily and rapidly growing ever since with a significant industrial presence (see [18] for example). In a study on smart homes in Europe, Sovacool et al. [30] mention the presence of 113 companies (including very large groups like Philips, Siemens, Apple, Amazon or Google). The Smart Building Alliance for Smart Cities, a group of building industry players, mentions in a 2020 blog the massification of the connected home market in France [28].

## 2. APPLICATION AREAS AND CHALLENGES

What services can we expect from a smart home ? Marikyan et al. conducted a bibliographic study of articles selected between 2002 and 2017 around the keywords *Smart Home*, *Smart Technology* and *Smart Buildings* [21]. 5 main services emerged (with overlaps between them):

* Comfort (41 citations): automation of daily routines, remote control of the house, taking into account energy and environmental constraints in comfort management and interconnected multimedia devices

<footnote>(1) https://www.openhab.org/.</footnote>

<footnote>(2) https://www.jeedom.com/.</footnote>
* Monitoring – supervision (31 citations): parameters related to health (physiological parameters, fall detection, etc.), to the quality of the environment (temperature, humidity, CO² for example) or to security (intrusion detection)

* Therapies (19 citations): Telemedicine

* Assistance (17 citations): help for people with visual or hearing impairments (voice recognition, adapted interfaces for home control), mobility difficulties (robotics), etc.

The main application fields that emerge are in the health, environment and leisure sectors. At the national level, we find research activities in the health domain around the assistance for the maintenance of the elderly and vulnerable people at home. In this context, there are works related to machine perception, for the detection of immediate events such as fall for instance. For example, there is research carried out by the Larsen team at INRIA Nancy [7, 9]. There are also works focused on the longer term such as the detection of activities or behavior patterns and their slow evolution over time, associated with the evolution of pathologies. We can cite in this context the work of the S4M team at LAAS ([4] or [5] for example) or the STARS team at Inria Sophia Antipolis [34], which is more specialized in the use of video sensors.

Homecare, particularly in medical deserts, also requires the development of telemedicine. For example, the STARS team at INRIA Sophia Antipolis is interested in the feasibility of early detection of cognitive disorders through remote medical consultations [19] during which the practitioner would also have access to elements such as emotion, commitment or stress measured from the automatic analysis of captured audio and video streams. In 1999, the Maia team at Inria (now the Larsen team) developed one of the very first telemedicine system for monitoring people undergoing dialysis at home, the Diatelic system [14, 16], in collaboration with the Trio and Dialogue teams at Nancy and in collaboration with doctors from L'Altir and the Nancy University Hospital. The device included a blood pressure meter and a scale as well as a networked computer to transmit the measurements.

In the environmental domain, we find the notion of energy manager whose objective is to control the air renewal, heating or cooling, hot water production, shutters etc. in order to reduce energy consumption while offering a desired level of comfort. These managers can range from simple rule-based systems to complex systems based on physical models of the buildings and available devices. Historically based on a complete control of the devices and taking little or no account of the activities of the inhabitants or users (in the case of tertiary buildings), work is currently underway to put the users back at the center and to better take their constraints into account. We are thus moving from an automatic control system to a cooperative system co-constructed by the inhabitants themselves. Putting the inhabitant back at the heart of energy management is central, for example, in the work carried out by the GCSP team of the G-SCOP laboratory [1, 2, 27].

The shift from a purely technophile view to a user-centric approach is part of a general movement within smart environments. As explained by Menniken and al [23], one of the important elements for the development and adoption of smart homes is the
support for the goals and values of the inhabitants. Technology needs to be understood less as something intelligent but more as something whose intelligence emerges through interaction with it (Taylor et al. [31]). This user involvement can be present from the design phase in order to develop technologies that really meet their expectations. Living Labs in particular are an essential part of this process by allowing a more ecological feedback of the needs, constraints and expectations of future users toward the industrial and academics. This allows the design, experimentation and evaluation of devices in a controlled environment in which a scientific approach can be developed. One example is the work of the ELIPSE team at IRIT, based on the Maison Intelligente de Blagnac ([26] and an article in this issue). Putting the user back at the center is also at the heart of the concerns for the configuration of the behavior of its intelligent habitat (which actions to trigger in which situation). Historically, the first home automation systems were based on automatic approaches and imposed rigid usage scenarios that were difficult to configure. As explained in the historical context part, this led to a distance of the public for these systems, leading to a loss of control instead of the promised freedom of use. In the field of IoT and interconnection of services, rule-based approaches have developed with in particular the notion of “market” allowing users to share applications (set of rules) reusable by less experienced users. We can quote for example the commercial system IFTTT(3). The IIHM team of the LIG laboratory proposes an approach based on a CCBL programming language specifically dedicated to non-programmer users ([32] and an article in this issue).

Designing systems that are truly usable and accepted by users is the 2^th^ point among 17 cited in a survey of 31 experts on potential barriers to the development of smart homes [30]. In first place is the technical reliability of these devices and the ability of a home to behave properly. The problem of technical reliability is linked to the multiplication of devices (often equipped with batteries which low level is not detected), to the multiplication of heterogeneous protocols and operating modes, as well as to the dynamic aspect of the environment (arrival or removal of new sensors and services). This complexity, this heterogeneity, this dynamicity of the environments, as well as taking into account the users’ needs require the implementation of specific architectures and software infrastructures (middlewares for example) allowing to take in charge these various points and to reinforce the quality of the produced software. Becker and al. [3] propose for instance an overview of existing approaches as well as the challenges to be solved. In the middleware domain, we can mention the work of the Adèle team (which has since joined the M-PSI team) of the LIG laboratory. They propose a service-oriented software component model iPOJO on top of OSGi [13] as well as an execution platform ApAM [8] allowing to take into account the dynamic and evolutive aspect of environments. Beyond the work on software architecture and runtime support, we can also mention an initial work led by the Vasco team of the LIG aiming at verifying at runtime that the behavior of the intelligent building conforms to a set of properties enacted during the design [15].

<footnote>(3)https://ifttt.com.</footnote>
Finally, security and privacy are also cited in first place, ex-aequo with reliability. Most home automation solutions are based on data uploading and processing in the cloud of service providers. This is accompanied by the fear of exploitation by these groups of increasingly private data. A first possible response can be provided by edge computing, which aims to exploit the computing resources that are increasingly present in the local environment (cell phones, intelligent sensors, raspeberry-type computers, etc.) in order to deport certain calculations traditionally carried out in the cloud to the level of these devices directly, as close as possible to the data. The coupling of Artificial Intelligence and Edge Computing approaches is called Edge AI. Several approaches are possible in this context. For example, all processing can be deported to the devices present in the home. There is no more cloud access (other than for software updates) and the data no longer leaves the users' sphere. The houses are usually all considered different from each other and there is no direct sharing of experiences through the construction of a more global model. This sharing of general knowledge between houses can nevertheless be envisaged by transfer learning type approaches where an initial model can be specialized for each house in order not to start from scratch. We can cite for example the work carried out in the RAMBO team of the Lab-STICC ([6] and an article in this issue). Edge Computing can also be considered as building a global model aggregating data from all connected homes by learning the model at the cloud level and uploading the learned model to local devices in order to perform inferences from sensor data. This approach reduces the volume of data transmitted and reduces latency by performing processing as close as possible to the data production, while exploiting the computing resources available within the homes. However, this does not completely solve the risks concerning privacy, as the data is sent to a cloud during the learning phase of the model. The problem of building a model that aggregates the information of different users while respecting their privacy is a problem addressed by Federated Learning approaches initially proposed by Google [22]. These approaches aim to locally train a model in each habitat with locally generated data. The parameters of these models are then centralized in a cloud and aggregated. The new model is then sent back to the local systems for a new learning phase. In this context, we can mention for example the work of the Getalp and M-PSI [12] teams as well as Datamove [24] of the LIG. We can also mention a study of the LIST3N [17] laboratory of the Troye University of Technology in the context of a professional environment (enterprise 4.0). This theme is also the subject of the Edge Intelligence Chair of the MIAI Artificial Intelligence Institute [11].

Research on Home and Intelligent Environments is multi-disciplinary, involving engineering sciences and human and social sciences. It offers important societal benefits and many scientific and technical challenges remain to be met.

### 3. IN THIS ISSUE

This special issue is composed of five articles. They are part of the movement to put the user back at the center of his intelligent habitat, particularly in the context of aging and home assistance for people in fragile situations. The first two articles present
the role that the user can play upstream during the design or validation of innovative devices, or “at runtime”, in the construction of the provided services. The last three articles present approaches and challenges related to the design of services capable of perceiving the context, the activities, and the needs of the user within his habitat.

The first paper, by Brulin, Campo, Val, Van Den Bossche, Vella and Vigouroux focuses on the issue of “ageing well at home”. In particular, it highlights Living Lab approaches that put end users and their needs at the heart of the design and evaluation of innovative devices. The Maison Intelligente de Blagnac is presented, in particular the technical choices that were made to meet the objectives of a living lab, the experimental methodology deployed, as well as a more complete description of an experiment that was carried out to assess the needs in home automation for the elderly.

The second paper, by Demeurre and Caffiau, proposes to put the user, not at the center of the design of new innovative devices, but at the heart of the design of the behavior resulting from the cooperation of the various home automation systems deployed in his home. This problem of programming by the users themselves of the behavior of their smart home is usually addressed in experimental or commercial approaches by “user friendly” languages based on rules Event – Condition – Action. These approaches have a certain number of limitations that the authors resolve by defining a new dedicated language: CCBL (Cascading Contexts Based Language).

The third paper, by Vacher and Portet, looks back on fifteen years of research conducted by the Getalp team of the LIG laboratory in the field of voice control for an intelligent habitat, one of the privileged application domains being the assistance for the maintenance of elderly and fragile persons at home. The specific domain of voice control in the home brings a large set of specific constraints such as the need to continuously adapt to the speaker, the consideration of noisy sound environments, the presence of several speakers, etc. The Getalp team has been strongly involved in the setting up of living labs allowing controlled experiments to be carried out in ecological situations and also allowing the constitution of corpora made available to the community.

The fourth paper, by Campo, Brulin, Estève and Chan, presents the evolution of technologies in the complex field of maintaining elderly or frail people in autonomy as long as possible in their homes while offering good security conditions without being too intrusive. It questions in particular the role that artificial intelligence and domestic robotics can have in this context, in particular around the modeling of activities and interaction with the environment and users.

The last paper by Bouchabou, Nguyen, Lohr, Leduc and Kanellos presents an approach for activity recognition based on the temporal succession of sensor activations deployed in the habitat. Interpreting a sensor activation as a word and a succession of activations as a sentence, they are inspired by the work in automatic natural language processing to build a syntactic and semantic representation of these activations allowing to improve the recognition of activities and facilitating the transfer of knowledge from one house to another.
Introduction (EN)

We would like to thank the following reviewers who contributed to the development
of this special issue:

* Olivier Boissier

* Abdelghani Chibani

* Ali Chkeir

* Charles Consel

* Joëlle Coutaz

* Amal El Fallah-Seghrouchni

* Marie-Pierre Gleize

* Denis Jouvet Loria

* Abir Karami

* Maria Di Mascolo

* Olivier Simonin

BIBLIOGRAPHY

[1] A. A. ALYAFI, V.-B. NGUYEN, Y. LAURILLAU, P. REIGNIER, S. PLOIX, G. CALVARY, J. COUTAZ, M. PAL & J.-P. GUILBAUD, "From Usable to Incentive Building Energy Management Systems", *Modélisation et réalisation du contexte (Modeling and Using Context) 2* (2018), no. 1, p. 1-30.

[2] A. AWADA, P. REIGNIER, S. PLOIX, M. JACOMINO & E. ELSAFADI, "Identification Collaborative d'activités Dans Une Zone Habitée", in *IBPSA* (Reims, France), 2020.

[3] C. BECKER, C. JULIEN, P. LALANDA & F. ZAMBONELLI, "Pervasive Computing Middleware: Current Trends and Emerging Challenges", *CCF Transactions on Pervasive Computing and Interaction 1* (2019), no. 1, p. 10-23.

[4] F. BETTAHAR, W. BOURENNANE, Y. CHARLON & É. CAMPO, "HOMECARE : une plateforme technique de surveillance pour le suivi actimétrique de patients Alzheimer", in *Workshop – Alzheimer, Approche pluridisciplinaire. De la recherche clinique aux avancées technologiques*, 2013, p. 87-98.

[5] G. BOUAZIZ, D. BRULIN, H. PIGOT & É. CAMPO, "Detection of social isolation based on meal-taking activity and mobility of elderly people living alone", in *JETSAN 2021 – Colloque en Télésanté et dispositifs biomédicaux - 8e édition* (Toulouse, Blagnac, France), Université Toulouse III - Paul Sabatier [UPS], 2021.

[6] D. BOUCHABOU, S. M. NGUYEN, C. LOHR, B. LEDUC & I. KANELLOS, "Using Language Model to Bootstrap Human Activity Recognition Ambient Sensors Based in Smart Homes", *Electronics* **10** (2021), no. 20, p. 2498.

[7] M. DAHER, A. DIAB, M. EL BADAoui EL NAJjar, M. KHALIL & F. CHARPILLET, "Automatic Fall Detection System using Sensing Floors", *International Journal of Computing and Information Sciences (IJCIS) 12* (2016), no. 1, p. 75-82.

[8] E. DAMOU, "ApAM : Un environnement pour le développement et l'exécution d'applications ubiquitaires", Thèses, Université de Grenoble, 2013, https://tel.archives-ouvertes.fr/tel-00911462.

[9] A. DUBOIS & F. CHARPILLET, "Measuring Frailty and Detecting Falls for Elderly Home Care Using Depth Camera", *JAISE – Journal of Ambient Intelligence and Smart Environments* **9** (2017), no. 4, p. 469-481.

[10] K. DUCATEL, M. BOGDANOWICZ, F. SCAPOLO, J. LEIJTEN & J.-C. BURGELMAN, "Scenarios for Ambient Intelligence in 2010", Tech. report, IST Advisory Group (ISTAG), IPTS-Seville, 2001.

[11] "Edge Intelligence: low complex learning by local intelligence", https://edge-intelligence.imag.fr/.

[12] S. EK, F. PORTET, P. LALANDA & G. VEGA, "A Federated Learning Aggregation Algorithm for Pervasive Computing: Evaluation and Comparison", in *19th IEEE International Conference on Pervasive Computing and Communications PerCom 2021* (Kassel (virtual), Germany), 2021.
[13] C. ESCOFFIER, R. S. HALL & P. LALANDA, "iPOJO: an Extensible Service-Oriented Component Framework", in *IEEE International Conference on Services Computing (SCC 2007)*, 2007, p. 474-481.

[14] R. HERVY, L. ROMARY, F. CHARPILLET, J.-M. PIERREL, J.-P. THOMESSE, E. PETITJEAN, L. JEAN-PIERRE, P.-Y. DURANT & J. CHANLIAU, "System for the Remote Monitoring of Patients", 2004, Patent: US6,746,398B2.

[15] A. IDANI, "A Lightweight Development of Outbreak Prevention Strategies Built on Formal Methods and xDSLs", in *2nd ACM European Symposium on Software Engineering (ESSE 2021)* (Larissa, Greece), 2021.

[16] L. JEANPIERRE & F. CHARPILLET, "Automated Medical Diagnosis With Stochastic Models: Monitoring Chronic Diseases", *Acta biotheoretica* **52** (2004), p. 291-311.

[17] M. KHALIL, M. ESSEGHIR & L. MERGHEM-BOULAHIA, "A Federated Learning Approach for Thermal Comfort Management", *Advanced Engineering Informatics* **52** (2022), p. 101526.

[18] D. C. KHEDEKAR, A. C. TRUCO, D. A. OTEYZA & G. F. HUERTAS, "Home Automation—A Fast – Expanding Market", *Thunderbird International Business Review* **59** (2017), no. 1, p. 79-91.

[19] A. KÖNIG, R. ZEGHARI, R. GUERCHOCHE, M. D. TRAN, F. BREMOND, N. LINZ, H. LINDSAY, K. LANGEL, I. RAMAKERS, P. LEMOINE, V. BULTINGAIRE & P. ROBERT, "Remote cognitive assessment of older adults in rural areas by telemedicine and automatic speech and video analysis: protocol for a cross-over feasibility study", *BMJ Open* **11** (2021), no. 9, p. e047083.

[20] R. LUTOLF, "Smart Home concept and the integration of energy meters into a home based system", in *Seventh International Conference on Metering Apparatus and Tariffs for Electricity Supply 1992*, 1992, p. 277-278.

[21] D. MARIKYAN, S. PAPAGIANNIDIS & E. ALAMANOS, "A systematic review of the smart home literature: A user perspective", *Technological Forecasting and Social Change* **138** (2019), p. 139-154.

[22] H. B. McMAHAN, E. MOORE, D. RAMAGE, S. HAMPSON & B. A. Y. ARCAS, "Communication-Efficient Learning of Deep Networks from Decentralized Data", https://arxiv.org/abs/1602.05629, 2016.

[23] S. MENNICKEN, J. VERMEULEN & E. M. HUANG, "From today's augmented houses to tomorrow's smart homes: new directions for home automation research", in *Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing* (New York, NY, USA), UbiComp '14, Association for Computing Machinery, 2014, p. 105-115.

[24] A. MITRA, Y. NGOKO & D. TRYSTRAM, "Impact of Federated Learning On Smart Buildings", in *2021 International Conference on Artificial Intelligence and Smart Systems (ICAIS)*, 2021, p. 93-99.

[25] "Professional Builder. Being Smart About Smart Homes", 2018, https://www.probuilder.com/blog/being-smart-about-smart-homes-0.

[26] P. RUMEAU, N. VIGOUROUX, É. CAMPO, E. BOUGEOIS, F. VELLA, A. VAN DEN BOSSCHE, T. VAL & J. ANCILOTTO, "Technological Services in Shared Housing: Needs Elicitation Method from Home to Living Lab", *IRBM* **42** (2021), no. 2, p. 73-82.

[27] J. SILVA, M. AMAYRI, S. PLOIX, P. REIGNIER & C. S. SILVA, "Cooperative and Interactive Learning to Estimate Human Behaviours for Energy Applications", *Energy and Buildings* **258** (2022), article no. 111727.

[28] "Smart Buildings Alliance. À propos", https://www.smartbuildingsalliance.org/association/a-propos.

[29] "Smart house closer to graduation", https://www.chicagotribune.com/news/ct-xpm-1991-10-12-9104020288-story.html.

[30] B. K. SOVACOOL & D. D. FURSZYFER DEL RIO, "Smart home technologies in Europe: A critical review of concepts, benefits, risks and policies", *Renewable and Sustainable Energy Reviews* **120** (2020), article no. 109663.

[31] A. S. TAYLOR, R. HARPER, L. SWAN, S. IZADI, A. SELLEN & M. PERRY, "Homes That Make Us Smart", *Personal and Ubiquitous Computing* **11** (2007), no. 5, p. 383-393.

[32] L. TERRIER, A. DEMEURE & S. CAFFIAU, "CCBL: A Language for Better Supporting Context Centered Programming in the Smart Home", *PACM on Human-Computer Interaction* **1** (2017), no. EICS, article no. 14 (18 pages).

[33] J. G. TRULOVE, *The Smart House*, Harper Design, New York, 2003.
[34] N. ZOUBA, F. BREMOND & M. THONNAT, "An Activity Monitoring System for Real Elderly at Home: Validation Study", in *2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance*, 2010, p. 278-285.