**8.1 Introduction**

The RESETTING project<fnref n="1" /> aims at facilitating the shift toward more robust, circular, and eco-friendly business practices for tourism companies across Europe. Within its scope, we are building an online observatory to provide a broad view of STTs' offer in Europe, the **European STT Observatory**. This observatory requires methods and tools to assess “smartness” based on a sound definition of ST and STT and to be able to cope with technological evolution. This chapter is scoped in this effort.

To correctly understand ST (offer), a preliminary literature review on the ST concept and on STT and a broad online search on STT was performed. These early efforts revealed two significant research gaps or, rather, research uncertainties.

Although there is a common agreement in the academia that Smart Tourism (ST) refers to the use of technologies to improve visitors' and local's experiences while enabling sustainability goals, different authors consider different key aspects, from techno-, through tourist- or business-, to destination-centered definitions. Additionally, the hype around “Smart” has led to the misuse of the concept, known as “SmartWashing” (Desdemoustier et al. 2019). STTs are frequently just claims based on technological aspects provided by developers; the “smartness” of such technologies is often difficult to evaluate.

Additionally, the term “Smart” is used to describe developments fueled by cutting-edge technologies. Since technology development is constantly evolving, STTs are a moving target where a “Smart” offer quickly becomes a “Dumb” offer.

To populate the European STT Observatory, we need (a) strong definition(s) for ST(T). This effort will also allow us to create an STT smartness index to measure the smartness of the European STT supply. This chapter outlines our ongoing efforts to achieve these goals, focusing on our scientific and collaborative approach to reaching a unanimous definition of STT.

Nowadays, travel has become more accessible, convenient, and enjoyable for millions of people around the world. However, there is a digital divide between large enterprises and small and medium-sized enterprises (SMEs) in terms of their stake in the tourism industry (Minghetti and Buhalis 2010; Reverte and Luque 2020). The tourism industry's adoption of digitalization, innovation, and new technologies necessitates the use of STT. However, tourism SMEs lag behind in their ability to capitalize on the opportunity of a digital transformation of their core business due to financial and technical limitations, as well as a lack of awareness of existing STT. This is the driving force behind the creation of the European STT Observatory.

<footnote n="1">¹ Relaunching European smart and Sustainable Tourism models Through digitalization and innovative technologies.</footnote>
Thus, our research is framed in the following research questions:

* **RQ1:** What are the main criteria and methods for evaluating the smartness level of STT and comparing them across different domains and regions?

* **RQ2:** How can the participatory approach involving ST experts help reach a consensual definition of ST and STT and create a robust taxonomy for ST?

* **RQ3:** How can the European STT Observatory provide a comprehensive and up-to-date overview of the STT offer in Europe?

* **RQ4:** What are the benefits and challenges of using STT for tourism SMEs, and how can the European STT Observatory help reduce the digital gap between SMEs and large enterprises?

## 8.2 Literature Review

### 8.2.1 Smart Tourism: Evolution

The word “tourist” is believed to have originated in the eighteenth century, derived from the French word “tour,” meaning “turn” or “trip,” and referred to individuals who embarked on a journey or circuit, often for leisure or educational purposes.<fnref n="2" /> Others claim that the etymology of the term tourism is much older, tracing back to the ancient Greek word for a tool used in describing a circle. In a sense, tourism is a journey that starts and ends at the same place, home (Leiper 1983). Nomadic lifestyles have accompanied humanity since its earliest development and are responsible for the widespread distribution of human beings. Although early human migrations were probably motivated by a combination of factors, including climate change (Timmermann and Friedrich 2016), population growth, and competition for resources, the advent of religions and culture embodied in prehistoric monuments suggests that early humans visited these sites as a form of primitive tourism.

However, the modern tourism industry began in the nineteenth century with the development of transportation infrastructure such as railways and steamships, and in the 1950s, with the widespread ownership of the automobile and the arrival of charter flights, national and international tourism took off (Christou 2022; Mowforth and Munt 2016; Sharpley 2022).

In the pre-Internet era, tourism suppliers had to rely on intermediaries for their distribution functions. The emergence of Computer Reservation Systems (CRSs) in the 1970s and Global Distribution Systems (GDSs) in the late 1990s facilitated the intermediation process, but it was the development of the Internet in the 1990s, leveraged by Information and Communication Technologies (ICTs) by providing

<footnote n="2">² Still, nowadays, the Merriam-Webster dictionary defines tourism as *the practice of traveling for recreation* (Merriam-Webster 2023).</footnote>
tools for marketing, customer relationship management, yield management, quality control, and innovation, that changed both business practices and strategies and ultimately reshaped the tourism industry (Buhalis and Law 2008). As ICT evolved, new forms of tourism were shaped. The impact of new technologies is so great that it calls into question the very definition of tourism. We live in an era in which tourism has gone beyond the limits of a circular route between home and destination, and today, we can travel without leaving home (virtual tourism Verma et al. 2022), visit different times, or experience parallel realities in the areas where we travel (metaverse tourism and extended reality Yang and Wang 2023) or even aspire to visit places outside our own atmosphere (space tourism Reddy et al. 2012). With the unprecedented exponential growth we are currently witnessing, thanks to recent developments in AI, who knows what the future holds? It is in this context that ST is developing.

The evolution of Smart Tourism has been a transformative process, marked by the integration of advanced technologies into the tourism industry. The next paragraphs trace the journey from the early stages of e-tourism to the current era of Smart Tourism.

Tourism was one of the first sectors to digitalize business processes on a global scale, bringing flight and hotel booking online to become a digital pioneer. As ICT became a global phenomenon, tourism was a consistent early adopter of new technologies and platforms (Gössling 2021).

The early digitization of tourism was coined as “e-tourism” and was focused on digitizing processes within the tourism industry, changing the way organizations distributed their tourism products in the market (Buhalis and Licata 2002). This led to the development of online booking systems, e-ticketing, and other digital tools that made it easier for travelers to plan and book their trips online. Soon after, the new concept of “digital tourism” was proposed (Zambonelli et al. 2001). It is a broader term that encompasses e-tourism but also includes the use of digital tools by tourists to prepare, organize, control, and enjoy their travel experience.

In the early 2000s, the rise of mobile technology and smartphones paved the way for the development of ST. According to Li et al. (2017), the term “Smart Tourism” can be traced back to 2000 when Gordon Phillips defined it as a sustainable approach to planning, developing, operating, and marketing tourism products and businesses (Phillips 2000).<fnref n="3" /> In his opinion, ST is shaped by two types of techniques: (1) smart demand and the use of management techniques that are capable of managing demand and access and (2) smart marketing techniques that can be used to target the proper customer segments to deliver appropriate messages.

Some preliminary research on STT can be found in di Hu et al. (2008). According to Wang et al. (2013), the earliest reference to Smart Tourism Destinations (STD) was coined in the “STD initiative,” promoted by China’s State Council of Chinese Central Government in 2009, in relation to a platform on which information relating to tourists activities, the consumption of tourism products, and the status of tourism

<footnote n="3">³ This presentation in SlideShare could not be found, so this claim could not be verified.</footnote>
resources could be instantly integrated and then provided to tourists, enterprises, and organizations through a variety of end-user devices.

The concept of ST became widespread in the turn of the 2010s, inspired by the development of Smart Cities and supported by new technologies such as mobile apps, augmented reality (AR), virtual reality (VR), and the Internet of Things (IoT) to provide tourists with personalized and immersive experiences (Buhalis and Amaranggana 2014; Wang et al. 2013).

e-Tourism, digital tourism, and ST came in succession chronologically, portraying the revolutionary changes brought to the tourism industry over the past few decades, leveraged by the power of technology. However, these concepts are often used interchangeably, as they all involve the use of technology to enhance and facilitate tourism activities (Kononova et al. 2020). Yet other technology-inspired designations for tourism have been used in scientific literature, such as “Mobile tourism (M-tourism),” “Intelligent Tourism,” “Tourism 4.0,” and “Virtual Tourism” (Kononova et al. 2020).

## 8.2.2 Smart Tourism: A Fuzzy Concept

Despite all the hype around ST, there is a lack of consensus on the definition of ST (Celdrán-Bernabéu et al. 2018; Gretzel et al. 2015; Križaj et al. 2021; Li et al. 2017; Pai et al. 2020; Rodrigues et al. 2023; Roopchund 2020; Shafiee et al. 2022, 2021; Zhang and Yang 2016). Several authors argue that the development of ST is hindered by the lack of such a definition (Celdrán-Bernabéu et al. 2018; Gretzel et al. 2015; Roopchund 2020; Shafiee et al. 2022, 2021). However, producing that definition is an endeavor jeopardized by the inherent complexity of the ST ecosystem, excellently discussed in (Gretzel et al. 2015).

Celdrán-Bernabéu et al. (2018) reason that future research aimed at the theoretical-conceptual development and critical analysis of ST should fill the existing gap between knowledge and operational development of ST. Additionally, they find some regionality associated with the terminology. In Western countries, ST is not a core strategy of tourism development but is based on its contribution to sustainable development and the relationship between tourists and destinations. In East Asia, ST actions focus on policies that promote the development of technological infrastructure, while in Europe, most initiatives are identified with Smart City projects that have favored the emergence of the smart destination approach.

Other authors point out the weaknesses resulting from this lack of clarity. For instance, Rodrigues et al. (2023) point out that some existing smart tourism strategies seem to use both smart and sustainability concepts in a marginal, propagandist way, more similar to a marketing point of view. This might be due to theoretical inconsistencies, which need to be addressed in future studies, defining a reliable basis for fully understanding the smart approach in tourism. Li et al.
<table><caption>**Table 8.1** Key concepts found in Smart Tourism definition</caption><thead><tr><th>Technology and innovation</th><th>Sustainability</th></tr></thead><tbody><tr><td><ul><li>Maximizing environmental, cultural, social, and economic values through IT</li></ul></td><td><ul><li>Intelligent, meaningful and sustainable connections</li></ul></td></tr><tr><td><ul><li>Mobile digital connectivity</li></ul></td><td><ul><li>Deep civic engagement</li></ul></td></tr><tr><td><ul><li>Combined model of tourism industry and innovative technology</li><li>Data accumulation with technological means</li><li>Technology-driven innovation</li></ul></td><td><ul><li>Clean, eco-friendly, ethical, and high-quality services</li><li>Tourist experience</li><li>Individual tourist support system</li></ul></td></tr><tr><td><ul><li>Constant and systematic use of smart elements</li></ul></td><td><ul><li>Ubiquitous tour information service</li></ul></td></tr><tr><td><ul><li>Device-generated big data for monitoring behavior, tourism management, and marketing</li><li>Potential replacement of human labor through digital technologies</li></ul></td><td><ul><li>Automatic provision of suitable and precise services</li><li>Integrated efforts to find innovative ways for data accumulation and aggregation or use</li></ul></td></tr><tr><td><ul><li>Evolutionary development of traditional tourism and e-tourism</li><li>Sensors, data mining, positioning technology, SNS, and social network technology</li></ul></td><td><ul><li>Interaction with a more comfortable environment for both locals and tourists</li><li>Creation of additional travel value for the tourist</li></ul></td></tr><tr><td><ul><li>Products using technological components</li></ul></td><td><ul><li>Smart cities</li></ul></td></tr><tr><td><ul><li>Privacy preserving, context awareness, recommender systems, social media, IoT, user experience, real-time, user modeling, augmented reality, and big data</li></ul></td><td><ul><li>Inspired by the idea of smart cities</li></ul></td></tr></tbody></table>

(2017) argue that because of such practices, there is the risk that the concept of ST could be abandoned.

To highlight the different views on ST, we extracted the key concepts underlying the definitions of ST compiled by Kononova et al. (2020). The results were grouped into classes representing the broader considerations in those definitions (Table 8.1).

### 8.2.3 Related Work

One interesting approach in refining ST definition is brought by Chen et al. (2024). The authors argue that existing research on ST focuses on technological application and the opinions of users and suppliers, lacking discussion from the perspective of academic experts. Furthermore, they claim that most studies focus only on a single aspect of people, the planning process, or the technological components of ST. In this context, they interviewed 11 ST scholars to examine their views on the role of and interactions among three key components (people, process, and technology) in current ST development. The main findings (per aspect) are summarized below.
**People—Key Stakeholders**

Informants generally agreed that several stakeholders are involved in the devel-
opment of ST, including tourists, suppliers, governments, destination marketing
organizations, service providers, system providers, the technology itself, and res-
idents. However, informants revealed different levels of importance among all these
stakeholders. The development of ST is closely linked to that of the smart city and
affects the quality of life of residents, who should have a say in the planning process.

**Process—ST Planning Process**

The ST planning process requires the collaboration and effort of various stakehold-
ers, including the tourism board, government, and sectors involved. Cooperation
between academia and industry is essential to enable a strategic relationship.
Fostering innovation is critical to the success of ST planning. Implementing an
ST-related project is an essential component for a destination, as the innovative
environment ensures a smooth connection with practices.

**Technology—ST Technologies**

ST applications are being implemented in various areas such as entertainment, hotel
operation, aviation, payments, experience training, ICT-related devices or appli-
cations, medical services, small business operation, and transportation. Industries
with high interaction between tourists and suppliers are the most relevant for the
application of ST. The adaptation of ST technologies occurs depending on changes
in tourists' behaviors and service patterns.

ICT is a prerequisite component for smart ecosystem management, but ST is
not just about applying ICT. Community interest and participation are critical to
the smooth running of ST destination development. The sharing economy with
traditional businesses also raises concerns. The availability of information from end
users is an essential mediator at the operational level. ST development can evolve
into a major regional or national development strategy, and operational guidance is
needed to serve as a reference for the development of ST destinations.

The current study follows a similar approach in reaching out to the scientific
community to try to create a consensual definition of ST.

ST operationalization is supported by the development and implementation of STT, supported by different technologies and operating in diverse domains.

Križaj et al. (2021) set out to ascertain what is being implemented in self-
labeled “smart” destinations and whether most “smart” projects actually qualify
as such, using a structured operational and innovation technology adoption-based
approach, Smart Actionable Classification Model (SACM). Built on top of Perboli
et al.’s (2014) taxonomy of Smart City projects, which lacks the evaluation of recent
advanced technologies, it is adapted to the context of Smart Tourism, using Buhalis’s
definition of smart systems in tourism (Buhalis 2022)—*Smart systems use a wide
range of networks, connected devices, sensors and algorithms for big data delivery
across the IoT*—to define the **Smart Actionable** attributes used to categorize ST
projects: (i) **networked/connected devices** and applications, (ii) coordinated by
**intelligent algorithms**, and (iii) based on collected and analyzed information at
<table><caption>**Table 8.2** ST groups (from Križaj et al. 2021)</caption><thead><tr><td>id</td><td>Description</td><td>Techs</td></tr></thead><tbody><tr><td>S1</td><td>Projects dealing with large amounts of networked and connected data, with the attributes of <strong>networked/connected</strong> and <strong>big data</strong>. The use of <em>intelligent algorithms</em> is negligible</td><td>networks IoT sensors</td></tr><tr><td>S2</td><td>Projects connected only by the <strong>networked/connected</strong> attribute and using to a <em>lesser extent</em> <strong>networked/connected</strong> data. The use of <strong>intelligent algorithms</strong> is <em>less frequent</em>, and the <strong>Big Data</strong> attribute is <em>absent</em> (despite most projects using sensors, the collected data is not sufficient to classify them as big data)</td><td>sensors mobile apps</td></tr><tr><td>S3</td><td>Projects associated only with the <strong>big data</strong> attribute. They <em>rarely</em> apply <strong>intelligent algorithms</strong> in data processing, and <em>do not</em> address <strong>networked/connected goals</strong></td><td>IoT AI</td></tr><tr><td>S4</td><td>Projects that have not been assigned to any Smart Actionable. They use significantly fewer individual tech types<sup>a</sup></td><td>mobile apps sensors</td></tr></tbody></table>

a **Big Data** level. They classified 35 ST projects publicly available in Europe into the following groups based on their tech type and Smart Actionable (Table 8.2):

The authors found that *the vast majority of projects branded as “smart” predominantly pursue environmental sustainability goals, but do not feature advanced technology that meets the Smart Actionable attribute criteria, and do not address social sustainability issues to the same extent as the environmental ones.* This may suggest that there is a gap between the buzz generated by Smartness in tourism and its actual application at the destination level. These findings may also indicate that technocentric metrics alone may not be suitable for evaluating Smartness in the tourism sector. Notably, their study underscores the emphasis on sustainability objectives, with a primary focus on environmental considerations and, to a lesser degree, social aspects. Economic sustainability appears to be of less significance. The authors acknowledge that the projects under analysis, as described publicly, often employ vague technological terminology, particularly, but not exclusively, the term ICT, which impedes the classification of these projects with regard to technological intelligence. They also view their model as a preliminary milestone toward developing a stronger SACM approach, which accounts for additional terminology clarifications, as well as updated technological classification frameworks. Our work achieves a similar outcome, but our projects cannot be classified using the Perboli and colleagues’ taxonomy. This is because we do not solely focus on smart destination projects but rather mainly on independent technological solutions that can function without the structured context of a smart city or destination.

Buhalis and Amaranggana (2015) discovered three distinct moments in the personalized services expected by tourists in STD:

(1) *Before Trip*: To support the planning phase by giving all the related real-time information based on user profiling in order to make a more informed decision

<footnote>a Such projects accounted for as much as 31% of all projects in the analysis</footnote>
(2) *During Trip*: Enhanced access to real-time information to assist tourists in exploring the destination, direct personalized services, as well as a real-time feedback loop

(3) *After Trip*: Prolonged engagement to relive the experience, as well as a decent feedback system, allowing tourist to review their holistic tourism experience

## 8.3 Methodology

In this section, we first present the overall methodology used to build and operate the observatory (Fig. 8.1, where the 📄 icon represents the RESETTING Web app form). This chapter focuses on the cornerstones for building the observatory:

* Our proposed STT taxonomy, based on the operationalization of the studied STT applications and the identified technology domains

* The definition of ST and STT, from which we derive the STT Smartness Index, based on our participatory approach

The Observatory will be operated and maintained by semi-automated, AI-based Extraction, Transformation, and Loading (ETL) processes and expert-in-the-loop and is available online at http://sttobservatory.eu.

### 8.3.1 STT Operational Taxonomy

Several initiatives were developed to collect information on STTs. In Europe, we highlight the Scottish Tourism Toolkit (Gidlund and McGurren 2020), the European Commission report on ST Practices (Scholz and Friends Agenda 2022), Spain's Secretary of State for Tourism Catalogue of Technological Solutions for STD (SEGITTUR Turismo e Innovación 2022), and the Spanish Cluster of Innovative

<figure><img src="image_49.png" /><figcaption>**Fig. 8.1** Observatory methodology</figcaption></figure>
<figure><img src="image_50.png" /><figcaption>**Fig. 8.2** STT action domains and tool types</figcaption></figure>

Companies for Tourism in the Valencia Community Technological Solutions for Tourism catalog (ADESTIC 2023). We analyzed the STTs offered in these catalogs and the descriptions of STTs offered on the Internet, trying to map the areas of operation of STTs, the types of STTs available, and the types of technology most commonly used. This allowed us to identify three domains of application for STTs. In this sense, STTs can be:

* Tools as part of touristic offers

* Tools used for the marketing of touristic offers

* Tools for managing and operating touristic offers

These domains are not exclusive, i.e., some STTs operate in several domains.

Regarding the existing types of STT, we found ten different classes (Fig. 8.2). The main difference between them is not a technological consideration but rather the differences in the final objectives proposed by the developers and the domains in which they operate.

For the *Tourism Offer* action domain, we found three exclusive types of STTs:

* *Fully digital experience*—STTs that constitute tourism experiences occurring in the virtual world (e.g., virtual tours)

* *Partial digital experience*—core STTs that enable and/or enrich the tourism experience in the real world (e.g., augmented reality tours)

* *Building block*—scaffolding tools that support the enrichment of the tourist experience (e.g., audio or video content creation)

Three non-exclusive types of STTs have been identified in the *Marketing* activity domain:

* *Pre consumption*—STTs used in the stage where tourists engage in the decision-making process (e.g., selecting and booking the destination and related tourism services)

* *Consumption*—STTs used in the consumption stage, where tourists enjoy the pre-selected tourism services
* *Post consumption*—STTs used in the stage where tourists act as influencers of other users (e.g., sharing feelings raised and/or contents generated during the consumption stage)

Finally, the *Management and Operations* action domain contains four non-exclusive STT types:

* *Design of tourism products*—STTs that aim to support the creation of tourism offers (e.g., B2B platforms that support the creation of tourism packages)

* *Deployment of tourism products*—STTs that support the deployment of the tourism offer (e.g., online booking and payment platforms)

* *Operation of tourism products*—STTs that support the operation of tourism products (e.g., crowd management tools)

* *Maintenance/evolution of tourism products*—STTs that support the evaluation and decision-making regarding the existing tourism offer (e.g., data analysis tools)

In addition, we have compiled the technological areas that make up the tools analyzed. A non-exhaustive list is presented in Fig. 8.3 that tries to map the technological areas to the cognitive areas of the human brain.

The compiled domains of action cover all application areas found in the literature review and in the responses to the Web app form (see next section). They make up the backbone of the STT taxonomy that will integrate the European Observatory's Smartness Index. The latter should take into account STT's specific goals, technological and implementation aspects, potential impact on local (and global) sustainability, and stakeholder involvement.

<figure><img src="image_51.png" /><figcaption>**Fig. 8.3** Technologies most commonly found in STT</figcaption></figure>
### 8.3.2 ST(T) Definition

To achieve a consensual definition of STTs, create a robust taxonomy for ST that can cope with the technological evolution needed to populate the European STT Observatory, and create an index that can compare the smartness level of STT offers, we used a scientific collaborative approach, inviting experts to provide their views.

#### Methodology Questionnaire

We created a form-like Web app considering the following principles:

* Be intuitive and easy to answer.

* Provide help content.

* Have no mandatory answers.

* Do not collect personal data.

* Provide both open- and closed-type questions.

* Be able to capture conceptual and technological aspects from the literature review.

* Be engaging with a game-like feel to promote answering in all fields.

* Provide motivational quotes as responding progresses.

* Reward users answering all questions with a prize: a virtual tourism experience.

* Allow users to leave comments and/or personal information if desired.

* Allow users from different countries to view and respond in their native language.

Next, we briefly present the Web app's GUI (Graphical User Interface) and explain how answering is achieved.

The first question was about the focus of STT in terms of the nexus: Tourist-Destination-Tourism Operator and Sustainability-Technology. The respondents were asked to drag the icon to the desired area in the Venn diagram (Fig. 8.4).

Regarding the nexus Sustainability-Technology, we later identified an error in the formulation of the Venn diagram: the chosen approach does not allow users to select the individual intersections between Society, Technology and Environment, Economy that are hidden behind higher-level central intersections.

The second question on the technologies that make up STT asked users to rate the “smartness” of the most commonly identified technologies in STT offer (Fig. 8.5).

The respondents were asked to rank the hypotheses (e.g., A.I.) by dragging them to the appropriate box according to the desired rank levels. Users could input multiple items in each rank box, allowing for equal importance values for multiple technologies. They could also add or delete rank levels at will, creating further distance between hypotheses to best represent the different advancement states of the technologies. If a technology was not deemed to belong to the STT universe, an additional box was provided labeled *I don't think this is an STT at all!* To aid users in understanding the provided technologies, a help box labeled *Don't know a tech? Drag it here!* was created. This box led users to online content that
<figure><img src="image_52.png" /><figcaption>**Fig. 8.4** STT focus</figcaption></figure>

explains the intended technologies. Wikipedia was the chosen data source whenever possible for consistency. Additionally, users could create and rank their own custom technologies using the “*Add other*” button.

In the third question, on the basic principles of ST (Fig. 8.6), users were required to compare various concepts that underlie the definition of ST, as derived from the literature review. All functionalities, except for the help box, were similar to those found in the previous question. This meant that users could place multiple items in each rank box, add or remove rank levels at will, and create and rank their own principles for a more discriminating separation of alternatives.

For the fourth and final question, users were asked to provide their own definition of STT in an open-ended format (Fig. 8.7).

The app also featured a completeness gauge and provided motivational quotes to encourage responses (Fig. 8.7).

Respondents who achieved 100% completeness were invited to explore the depths of the earth through a virtual visit to Gralhas VII cave,<fnref n="4" /> in Portugal, provided by CEAE-LPN.<fnref n="5" />

<footnote n="4">⁴ Virtual tour (last accessed on 02/27/2024). If you wish to take the tour, please note the red arrows as they may not be very visible.</footnote>

<footnote n="5">⁵ CEAE-LPN: Centro de Estudos e Actividades Especiais-Liga para a Protecção da Natureza (en: Centre for Special Studies and Activities-League for the Protection of Nature).</footnote>
<figure><img src="image_53.png" /><figcaption>Fig. 8.5 STT technologies</figcaption></figure>
<figure><img src="image_54.png" /><figcaption>**Fig. 8.6** ST principles</figcaption></figure>

**Methodology: Questionnaire Respondents**

The population of interest for our survey is the community of ST experts. In order to maximize the chances of obtaining a large number of respondents, we searched the following databases for papers containing the term “Smart Tourism” in the title, abstract, and/or keywords and for the contact details of their authors: Scopus, SpringerLink, WebOfScience, lens.org, Dimensions, and Scielo.

In addition, since experts in a given field usually meet with their peers to validate their findings and to inspire themselves through discussion, we also retrieved the contacts of the authors presenting at these annual events focused on ST topics:

* The e-Tourism Conf. of the Int. Federation for IT and Travel & Tourism (ENTER)

* Int. Conf. of the Int. Association of Cultural and Digital Tourism (IACuDiT)

* Int. Conf. on Tourism Technology & Systems (ICOTTS)

We sent a personalized email to each of the identified ST experts, inviting them to participate in the survey.

**Methodology: Data Collection and Analysis**

In the development of our Web app, we opted to collect both categorical and open-ended data to allow for statistical analysis of the results aimed at identifying the most consensual technologies and principles underlying Smart Tourism Tools (STT), as well as identifying any new aspects and technologies not previously identified in the
<figure><img src="image_55.png" /><figcaption>Fig. 8.7 STT definition</figcaption></figure>
literature review. These two types of data underwent different analysis processes, which are explained in further detail in the next section.

The analysis results represent the weights of the sub-indices that will be aggregated to form the STT Smartness Index. In the following section, we present the finalized weights of the sub-indices, along with their rationale and mathematical formulation, where applicable. As the analysis is still ongoing, the final weights of all sub-indices are not yet available, as explained in the next section where appropriate.

## 8.4 Results: Presentation and Analysis

We received 334 expert responses worldwide, with an average completion rate of 82% (measured within the Web application itself). The Web application's built-in completeness checker measures the partial completeness of each question, meaning that if a respondent fails to rank a given variable in a ranking question, they will not receive a full score. Looking at completeness based on whether the question was answered or not, the average completeness rises to 91% (Table 8.3). Note that the average shown is based on the variables originally provided (excluding respondents' own variables) and also excludes responses to additional comments and contact information. These exclusions were also applied in the Web app's completeness checker, as these variables represent additional information to the survey.

In terms of global representativeness, we received responses from all continents, which should provide different views in relation to their regional realities. Of the 334 responses, we were only able to identify 52 countries, as the geo-referencing is based on the (optional) email domains provided, meaning that only 47% of the responses could be geo-referenced. In addition, 29 of the responses were in 10 different languages other than English—Greek, Korean, Spanish, Italian, Slovak, Chinese (simplified), Ukrainian, Portuguese, Polish, and Persian.

<table><caption>**Table 8.3** Average form completeness</caption><thead><tr><td>Question</td><td>% of responses</td></tr></thead><tbody><tr><td>Rank tool custom user STT definitions</td><td>2%</td></tr><tr><td>STT definition rank</td><td>95%</td></tr><tr><td>Rank tool custom user technologies</td><td>10%</td></tr><tr><td>Technologies rank</td><td>95%</td></tr><tr><td>STT focus: tourist, destination, company</td><td>92%</td></tr><tr><td>STT focus: sustainability, technology</td><td>93%</td></tr><tr><td>Respondent STT definition</td><td>81%</td></tr><tr><td>Respondent comments</td><td>25%</td></tr><tr><td>Email contact information</td><td>61%</td></tr><tr><td>Average completeness</td><td>91%</td></tr></tbody></table>
<figure><img src="image_56.png" /><figcaption>**Fig. 8.8** Tourist-Operator-Destination responses</figcaption></figure>

### 8.4.1 STT Focus

Figure 8.8 shows the results of the answers to the first question concerning the focus of STT regarding the nexus Tourist-Operator-Destination.

Visual analysis of the diagram reveals a consensus of the STT focus on the Tourist and the Destination to the detriment of the Tourism operator. Further analysis of the diagram seems to indicate that the users' perception of the data collection method was not consistent across the universe of STT experts. This may be the case because the data collection approach is innovative. It is reasonable to assume that the problem under study should be additive, i.e., the intersections of the groups should have higher values (or at least the same order of magnitude) than the independent groups, i.e., if one group, let's say “Tourist,” is key to STT, then intersections containing “Tourist” should retain at least some of that importance, and add the importance of the intersecting group, even though the importance of each independent group in the intersected group might be different from their importance in the independent groups. To overcome this, we performed a deeper analysis of the data and ultimately extracted indicators for the Smart Index; we analyzed the data considering each level of intersection as a separate universe. In this way, we can derive the relative importance of each of the nexus elements within each intersection universe.

Let's assume that participants in the dual intersection universe assign intrinsic importance or weight to each of the base variables—Tourist, Destination, Company—when responding. This would mean that we can calculate an average weight or ponderation for each of these variables within the dual intersection
universe. This can be expressed as:

$$Result_i = \alpha Tourist + \beta Destination + \gamma Company \quad (8.1)$$

where:

i is the group of all possible intersections in that universe. In the case of the dual intersection universe - [Destination_Tourist, Tourist_Company, Company_Destination],

$\alpha$, $\beta$, and $\gamma$ represent the weights of the base variables—Tourist, Destination, Company—constant within the intersection universe.

Using Equation 8.1, we can create a system of equations for each of the groups in a given universe. Note that for the dual intersection universe, Eq. 1 only has two terms. We can then create a matrix for the equation system—A—taking “$\frac{1}{2}$” for each of the base variables present in each dual intersection world, as we are only analyzing dual intersections.

$$A \cdot X = B \begin{bmatrix} 0.5 & 0.5 & 0 \\ 0.5 & 0 & 0.5 \\ 0 & 0.5 & 0.5 \end{bmatrix} \cdot \begin{bmatrix} \alpha \\ \beta \\ \gamma \end{bmatrix} = \begin{bmatrix} 0.50 \\ 0.29 \\ 0.21 \end{bmatrix} \quad (8.2)$$

where:

A is the systems equation matrix

X is the matrix of weights $\alpha$, $\beta$, and $\gamma$, which we are trying to find

B is the matrix containing the ratio of dual intersection i, over the total of the dual intersections universe, calculated directly from the gathered data

We can then solve for X:

$$X = A^{-1} \cdot B \quad (8.3)$$

where:

$A^{-1}$ is the inverted matrix for the equation system.

X contains the average intrinsic weights of the universe of the double intersection. We can directly estimate the average intrinsic weights for the independent universe by calculating the ratio of the entries in each independent group to the total entries for the independent universe. The average weights of the whole universe— independent and intersecting—can then be estimated using a weighted average, taking into account the dimensions of each universe in the data collected.

Table 8.4 shows the resulting overall weights for the Tourist-Operator-Destination nexus. It is clear, as identified in the visual analyses, that the Tourist is the most important focus for STT, closely followed by the Destination, and the Tour Operator (Company) is not as central to STT. We can then apply these averages to each group by adding the weights of the variables present in each group.
<table><caption>**Table 8.4** Average weights Tourist-Operator-Destination nexus</caption><thead><tr><td>Tourist</td><td>Destination</td><td>Operator</td></tr></thead><tbody><tr><td>0.50</td><td>0.42</td><td>0.08</td></tr></tbody></table>

<figure><img src="image_57.png" /><figcaption>**Fig. 8.9** Tourist-Operator-Destination results</figcaption></figure>

Figure 8.9 presents the results in a Venn diagram to facilitate comparison with the data collected. As you can see, the relationships of the original data remain, but now we have the cumulative effect required by raising the levels of intersection.

Regarding the question concerning the Sustainability-Technology nexus, the approach used to collect data in this particular Venn diagram does not allow the user to select the intersections between Society, Technology and Environment, Economy. However, it is important to note that although the user cannot enter data at the two intersections mentioned, the graphical nature of the response method leads the user to spatially locate their responses in the regions closest to their intended focus. This means that it is still possible to draw valid conclusions from the collected data.

A visual analysis of this diagram (Fig. 8.10) shows that the focus is on the environment, society, and technology and their combinations, to the detriment of the economy. Two interesting conclusions can be drawn from this. Firstly, although the term “smart” is intertwined with technology, at least in the field of SST, the emphasis seems to be primarily on the environment and society and only then on technology. Secondly, technology seems to bypass one of the pillars of the tripartite concept of sustainability, the economy.

In this case, because we have the data gaps mentioned above, we chose a simpler approach. We decided to treat the data as a whole, so we created an overall average by multiple counting the overlapping data, thus creating a new universe where
<figure><img src="image_58.png" /><figcaption>**Fig. 8.10**
Sustainability-Technology responses</figcaption></figure>

our sample data is three times larger (in the largest dataset of tested alternatives) compared to the collected data. We tested six hypotheses for extracting the average weights with different combinations of:

* Considering or not the central intersection—to test the effect of the largest group (about 50% of the total responses, when all other groups exist)

* Considering or not double intersections, replacing missing values with:

- The average size of the remaining universe of double intersections

- An average for the given variables within the remaining universe of double intersections

The alternative that best fits the data does not consider the central intersection and considers the double intersections, replacing the missing values with an average for the variables in question in the remaining universe of double intersections. The results are shown in Fig. 8.11.

### 8.4.2 Technology “Smartness”

Next, we analyze the results of the second question, in which respondents were asked to rank a list of technologies according to their state of the art. In addition, users could suggest and rank their own technology proposals. We received 72 technology suggestions, of which 49 were unique, and some were subsets of our own list.
<figure><img src="image_59.png" /><figcaption>**Fig. 8.11** Sustainability-Technology results</figcaption></figure>

The data for each technology category underwent normalization using the *Min-Max Scaling Normalization* method. This involved subtracting the minimum value found in the category from the value in each data point and dividing by the category range (i.e., the difference between the maximum and minimum values in the category). This procedure transforms every data point into a decimal between 0 and 1. The minimum values of that category are transformed into a 0, and the maximum values are transformed into a 1.

Concerning the ranking of the technologies, excluding the custom ones in this analysis, we received answers where users did not classify some technologies, although there is a box to drop technologies not considered STT. We cannot be sure if technologies not ranked were considered not STT or if this was just an issue of incomplete answering, or even a combination of the former two situations. To test the influence on the results, we created two separate average ranks, one considering the arithmetic mean and another considering the sample mean (Fig. 8.12). Figure 8.12a) shows the ranking in descending order of cutting-edge status, from left to right, taking into account the arithmetic mean rank. It does not show arithmetic averages but the ranking at equal intervals. In this way, a perfect blue spiral line is obtained, reflecting the steady descent of the ranking.

The number of ranked technologies is different for each technology (brown line in Fig. 8.12a), reflecting the different sample sizes per technology that influence the results. The behavior of the red line follows a spiral in the higher and lower rankings, but in the intermediate rankings, there are several disagreements between statistical
<figure><img src="image_60.png" /><figcaption>Fig. 8.12 Average ranks found</figcaption></figure>

approaches. If we analyze the average rank expressed in terms of the arithmetic mean actually calculated (Fig. 8.12b), the differences diminish, although they are still present. This figure expresses the actual distances between the technologies proposed by the respondents and shows that the perceived technological status is close for the divergent ranking categories. This approach also highlights the positive distances between adjacent technologies, e.g., the leap from AI to AR is much greater than the leap from the latter to VR, and the same effect can be seen between other technologies, although not as pronounced.

### 8.4.3 STT Fundamental Principles

In response to the question that asked users to rank some of the fundamental principles found in the literature review and to provide their own fundamental principles, we received 13 principle suggestions from 8 users (Table 8.5). These propositions reveal important aspects such as tourist mobility or safety.

The Web app's fundamental principles were ranked using the same strategy as the technology ranking, which was described above. This included data normalization procedures and testing of the two statistical approaches since some principles were not ranked. The ranking of the fundamental principles of STT was very consensual among the experts. Although the order of three principles varied between the two statistical approaches, their rankings were similar (Fig. 8.13).

Interestingly, the results are very similar to those obtained in the first question on the focus of STT. The tourist and the tourist experience come first, followed by the destination and environmental concerns, technological considerations, and commercial aspects.

For both questions regarding the ranking of data, we have chosen to use the sample average to construct the Smart Index because we believe it is the
<table><caption>**Table 8.5** Fundamental principles proposed by respondents</caption><thead><tr><td>User_ID</td><td>Proposed fundamental principles</td></tr></thead><tbody><tr><td>32</td><td>Help tourism destinations to be more sustainable, competitive, and resilient</td></tr><tr><td>47</td><td>Improves mobility efficiency<br>Enhance the tourist's experience</td></tr><tr><td>71</td><td>Metaverse</td></tr><tr><td>96</td><td>Creating Tourism Network</td></tr><tr><td>118</td><td>Optimize the tourist route</td></tr><tr><td>128</td><td>Bring better experience through smart chain management</td></tr><tr><td>250</td><td>It supports the improvement of tourism destination branding<br>It supports the improvement of users' Customer Satisfaction</td></tr><tr><td>318</td><td>Promote collaboration among the stakeholders of a tourist destination<br>Strengthen the governance of the destination<br>Facilitating tourist mobility around the destination<br>Increasing security in a tourist destination</td></tr></tbody></table>

<figure><img src="image_61.png" /><figcaption>**Fig. 8.13** STT fundamental principles average ranks</figcaption></figure>

most appropriate statistical approach to construct the ranking of ST fundamental principles, as it takes into account the variability and richness of the data collected. This is due to the fact that:

* The sample size is relatively large (334 responses from experts around the world) and representative of different regions and perspectives

* The design of the survey was intuitive, easy, engaging, and rewarding, which could have reduced the likelihood of non-response and incomplete answers
<figure><img src="image_62.png" /><figcaption>**Fig. 8.15** Bigram and trigram word cloud</figcaption></figure>

<figure><img src="image_63.png" /></figure>

* The survey used a rating tool that allowed multiple items in each rating box and the addition or deletion of rating levels at will, which could reflect the experts' preferences and opinions more accurately

### 8.4.4 Open-Ended STT Definition

Finally, users could write their own definition of STT in an open-ended question. We first used MAXQDA5⁶ to help analyze the responses. We started by creating two word clouds, one with just single words (Fig. 8.14) and the other with bigrams and trigrams (Fig. 8.15).

Once again, we can see the importance of the tourist, destination, and sustainability in the definitions presented. We then coded the responses according to three main categories, (i) STT objectives, (ii) STT target categories, and (iii) technology areas.

The code frequency analysis in relation to the STT objectives is shown in Fig. 8.16. Again, we see the same pattern as in the previous responses. It is worth noting that some mentions of environmental aspects could be related to broader sustainability, as these two concepts are often used interchangeably, and this was evident in our context analysis. The same could be said for inclusive tourism. The

<footnote n="6">⁶ https://www.maxqda.com/</footnote>
<figure><img src="image_64.png" /><figcaption>**Fig. 8.17** Code frequencies: STT target categories</figcaption></figure>

<figure><img src="image_65.png" /></figure>

code for this category only highlights responses that specifically mention it, but other mentions of the social aspects of sustainability could also consider this aspect.

The analysis of the code frequency regarding the key target categories is presented in Fig. 8.17 and again shows the same patterns as previously observed in relation to the STT definition. It should be emphasized that the **stakeholder** code represents the mentions where respondents broadly referred to all direct and indirect stakeholders in the tourism industry.

The code frequencies for technology mentions are shown in Fig. 8.18. The code **Technology** refers to mentions of technology as a broad field and does not distinguish between smart and mainstream technologies, i.e., these could refer to combinations of mainstream and smart technologies, the context analysis points in this direction. The **SMART** code aggregates mentions in different formats such as “innovative,” “advanced,” “new,” or “cutting edge.” In the code **Main-stream SMART**, respondents specifically mentioned the use of mainstream and/or advanced technologies.

The presented approach for the analysis of open-ended answers does not allow extrapolating meaningful indices for the Observatory smartness index. To overcome this, we tested different preprocessing techniques and natural language processing
<figure><img src="image_66.png" /><figcaption>**Fig. 8.18** Code frequencies regarding technology mentions</figcaption></figure>

(NLP) analyses. These ranged from simple approaches, such as assessing text complexity, to more complex methods, like topic modelling, a technique that identifies topics in a set of documents. We also attempted to measure data similarity in vector space, a mathematical model where words are represented as vectors in multi-dimensional space, using embeddings from small language models, simple and/or small AI models trained on text data, useful for several NLP tasks, but not highly effective on capturing context and concepts. This latter approach was also combined with a semantic search of previously extracted concepts. However, these attempts fell short of achieving the desired indices. This was largely due to the high text complexity of the definitions and their varied sentence structures. The definitions also include many examples that add redundant or extraneous information to the statistical analysis. Since LLMs (large language models, AI models trained on vast amounts of text data, able to generate humanlike text) can understand both concepts and context, they are particularly useful for treating data with such characteristics, given adequate instructions and examples. We are currently using LLMs to do Text Classification, Relationship Extraction, and Span Categorization. This approach should allow deriving a single vector of the relevant concepts for each definition that can be used to statistically determine the required indices.

## 8.5 Final Remarks

ST refers to the use of technology and data-based solutions to improve tourist experience, destination management, and sustainability. It involves the collaboration of different stakeholders, such as tourists, suppliers, governments, and residents, to create value and benefits for all. ST is inspired by the concept of smart cities but also takes into account the specific needs and characteristics of tourism and tourism destinations. STTs are the set of tools that support the development and implementation of ST. Therefore, STTs are a set of tools that:
1. apply the fundamental principles of ST, to achieve (at least some) ST goals,

2. within the specific scope envisioned by developers (application domain and specificities of the final consumer or area, ensuring concordance with ST scope),

3. considering the inputs of and the final influence on the key stakeholders,

4. taking advantage of the relationship of their specific field, the ST field (as most STT developers are not specifically working with ST, but are experts in their own field, e.g. VR), and with other interconnected knowledge areas, either technological or not, while applying techniques and technologies and

5. trying to minimize biases at an operational level (e.g. avoid search engine "commercial" algorithm influence in AI-based tourism recommendation systems) and on a conceptual level, ensuring they are indeed STT and not just digital tools.

Our results, based on the experts' opinions, corroborate that Smart Tourism's definition cannot just consider one dimension, such as the technocentric one. The results further allow us to define to what degree each key aspect of the definition should be considered when evaluating Smart Tourism offers. Our STT definition cannot yet deal with the fast pace of technological evolution. Further research is required considering different approaches, such as using the reports published yearly by the World Economic Forum, to identify and extract emerging technologies and the number of scientific publications referring to these technologies in the field of Smart Tourism, as a proxy for the technology Smartness. Additionally, this smart-metric approach means that, as technology evolves, earlier technologies do not become dumb but actually less Smart, and this can be quantified.

Our proposed consensual "current" definition for STT is that of:

> Seamlessly interconnected digital tools designed to benefit all stakeholders in the tourism industry, with a special focus on the tourist and the destination, that aim at local, and possibly global, sustainable development

At their highest level, STTs produce experiential ecstasy in the dynamic human interface with technology, enriching and not alienating people. At decreasing smartness levels, they also aim to improve the efficiency, quality, and competitiveness of tourism destinations, operations, products, and companies. They also aim to improve the quality of life of local communities and the well-being of tourists while minimizing the negative impacts of tourism on the environment and local culture. The level of STT Smartness depends more on achieving the objectives of sustainable tourism than on their technological status. Indeed, emerging technologies offer unforeseen possibilities, but so does innovation combined with conventional technology. This means that the intelligence of these tools should give higher importance to their alignment with the principles and objectives of Smart Sustainable Tourism. A full evaluation would require assessing tool deployment results, i.e., their impact on all relevant stakeholders in the tourism industry and on local (and possibly global) sustainability. This more complete Smartness assessment is beyond the
scope of our current work since the latter only considers the potential impact of STTs based on the information available, mainly in the product descriptions and then in more detailed descriptions provided by the STT developers/owners.

## 8.5.1 Future Developments

We anticipate broadening the audience to derive the ST(T) definition from other key stakeholders, such as tourists, tourism operators, and destination managers. This will allow us to derive a more representative STT Smartness Index. Finally, we plan to use the STT Smartness Index to assess the STT offer in the European market.

**Acknowledgments** This work was developed in the scope of the RESETTING project, funded by the COSME Programme (EISMEA) of the European Union under grant agreement No.101038190. We also thank all survey participants, without whom this study would not have been possible.

## References

* ADESTIC: *Soluciones Tecnológicas para el Turismo*. Tech. Rep. 2nd edition, Clúster de Empresas Innovadoras para el Turismo de la Comunitat Valenciana, Valencia, Spain (2023)

* Buhalis, D.: Smart Tourism, pp. 124–127. Edward Elgar Publishing, Cheltenham (2022). https://doi.org/10.4337/9781800377486.smart.tourism

* Buhalis, D., Amaranggana, A.: Smart tourism destinations. In: Xiang, Z., Tussyadiah, I. (eds.) Proceedings of the International Conference on Information and Communication Technologies in Tourism 2014, pp. 553–564. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-03973-2_40

* Buhalis, D., Amaranggana, A.: Smart tourism destinations enhancing tourism experience through personalisation of services. In: Tussyadiah, I., Inversini, A. (eds.) Proceedings of the International Conference on Information and Communication Technologies in Tourism, pp. 377–389. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-14343-9_28

* Buhalis, D., Law, R.: Progress in information technology and tourism management: 20 years on and 10 years after the Internet—The state of eTourism research. Tour. Manag. 29(4), 609–623 (2008). https://doi.org/10.1016/j.tourman.2008.01.005

* Buhalis, D., Licata, M.C.: The future eTourism intermediaries. Tour. Manag. 23(3), 207–220 (2002). https://doi.org/10.1016/S0261-5177(01)00085-1

* Celdrán-Bernabéu, M.A., Mazón, J.N., Ivars-Baidal, J.A., Vera-Rebollo, J.F.: Smart tourism. Un estudio de mapeo sistemático. Cuadernos de Turismo Enero-Junio (41) (2018). https://doi.org/10.6018/turismo.41.326971

* Chen, Z., Chan, I.C.C., Mehraliyev, F., Law, R., Choi, Y.: Typology of people-process-technology framework in refining smart tourism from the perspective of tourism academic experts. Tour. Recreat. Res. 49(1), 105–117. (2024). https://doi.org/10.1080/02508281.2021.1969114

* Christou, P.: The History and Evolution of Tourism. CABI (2022). https://doi.org/10.1079/9781800621282.0000

* Desdemoustier, J., Crutzen, N., Giffinger, R.: Municipalities’ understanding of the smart city concept: an exploratory analysis in Belgium. Technol. Forecasting Soc. Change 142, 129–141 (2019). https://doi.org/10.1016/j.techfore.2018.10.029. Understanding Smart Cities: Innovation ecosystems, technological advancements, and societal challenges
di Hu, L., Long, Y., yang Qian, C., Zhang, L., nian Lv, G.: Design and realization of intelligent tourism service system based on voice interaction. In: Liu, L., Li, X., Liu, K., Zhang, X., Wang, X. (eds.) Geoinformatics 2008 and Joint Conference on GIS and Built Environment: The Built Environment and Its Dynamics, vol. 7144, p. 714427. International Society for Optics and Photonics, SPIE (2008). https://doi.org/10.1117/12.812827

Gidlund, K., McGurren, M.: Scottish Tourism TOOLKIT: Technology solutions for tourism businesses in a post Covid19 Scotland. Tech. Rep. 2nd edition, Traveltech for Scotland, The University of Edinburgh, Scotland (2020)

Gretzel, U., Sigala, M., Xiang, Z., Koo, C.: Smart tourism: foundations and developments. Electron. Markets 25(3), 179–188 (2015). https://doi.org/10.1007/s12525-015-0196-8

Gretzel, U., Werthner, H., Koo, C., Lamsfus, C.: Conceptual foundations for understanding smart tourism ecosystems. Comput. Hum. Behav. 50, 558–563 (2015). https://doi.org/10.1016/j.chb.2015.03.043

Gössling, S.: Tourism, technology and ICT: a critical review of affordances and concessions. J. Sustain. Tour. 29(5), 733–750 (2021). https://doi.org/10.1080/09669582.2021.1873353

Kononova, O.V., Prokudin, D.E., Tupikina, E.: From e-Tourism to Digital Tourism. Terminologically Review. In: Proceedings of the Conference on Scientific Services & Internet (SSI-2020), vol. 2784. CEUR (2020)

Križaj, D., Bratec, M., Kopić, P., Rogelja, T.: A technology-based innovation adoption and implementation analysis of european smart tourism projects: towards a smart actionable classification model (SACM). Sustainability 13(18) (2021). https://doi.org/10.3390/su131810279

Leiper, N.: An etymology of “tourism”. Ann. Tour. Res. 10(2), 277–280 (1983). https://doi.org/10.1016/0160-7383(83)90033-6

Li, Y., Hu, C., Huang, C., Duan, L.: The concept of smart tourism in the context of tourism information services. Tour. Manag. 58, 293–300 (2017). https://doi.org/10.1016/j.tourman.2016.03.014

Merriam-Webster: Tourism (2023). Accessed 1 Oct 2023

Minghetti, V., Buhalis, D.: Digital divide in tourism. J. Travel Res. 49(3), 267–281 (2010). https://doi.org/10.1177/0047287509346843

Mowforth, M., Munt, I.: Tourism and Sustainability: Development, Globalisation and New Tourism in the Third World, 4th edn. Routledge (2016). https://doi.org/10.4324/9781315795348

Pai, C.K., Liu, Y., Kang, S., Dai, A.: The role of perceived smart tourism technology experience for tourist satisfaction, happiness and revisit intention. Sustainability 12(16) (2020). https://doi.org/10.3390/su12166592

Perboli, G., De Marco, A., Perfetti, F., Marone, M.: A new taxonomy of smart city projects. Transp. Res. Proc. 3, 470–478 (2014). https://doi.org/10.1016/j.trpro.2014.10.028. 17th Meeting of the EURO Working Group on Transportation, EWGT2014, 2–4 July 2014, Sevilla, Spain

Phillips, S.G.: The tourism industry association of Canada [EB/OL] (2000)

Reddy, M.V., Nica, M., Wilkes, K.: Space tourism: research recommendations for the future of the industry and perspectives of potential participants. Tour. Manag. 33(5), 1093–1102 (2012). https://doi.org/10.1016/j.tourman.2011.11.026

Reverte, F.G., Luque, P.D.: Digital Divide in E-Tourism, pp. 1–21. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-05324-6_109-1

Rodrigues, V., Eusébio, C., Breda, Z.: Enhancing sustainable development through tourism digitalisation: a systematic literature review. Inform. Technol. Tour. 25(1), 13–45 (2023). https://doi.org/10.1007/s40558-022-00241-w

Roopchund, R.: Mauritius as a smart tourism destination: technology for enhancing tourism experience. In: Pati, B., Panigrahi, C.R., Buyya, R., Li, K.C. (eds.) Proceedings of 3rd International Conference on Advanced Computing and Intelligent Engineering (ICACIE 2018), pp. 519–535. Springer, Singapore (2020). https://doi.org/10.1007/978-981-15-1483-8_44

Scholz & Friends Agenda: Leading Examples of Smart Tourism Practices in Europe. Tech. Rep. 2nd edition, European Commission (2022)
SEGITTUR Turismo e Innovación: Catalogue of Technological Solutions for Smart Tourist Destinations. Tech. Rep. 2nd edition, Spain's Secretary of State for Tourism, Madrid, Spain (2022)

Shafiee, S., Jahanyan, S., Ghatari, A.R., Hasanzadeh, A.: Developing sustainable tourism destinations through smart technologies: a system dynamics approach. J. Simul. **0**(0), 1–22 (2022). https://doi.org/10.1080/17477778.2022.2030656

Shafiee, S., Rajabzadeh Ghatari, A., Hasanzadeh, A., Jahanyan, S.: Smart tourism destinations: a systematic review. Tour. Rev. **76**(3), 505–528 (2021). https://doi.org/10.1108/TR-06-2019-0235

Sharpley, R.: Tourism and development theory: which way now? Tour. Plan. Dev. **19**(1), 1–12 (2022). https://doi.org/10.1080/21568316.2021.2021475

Timmermann, A., Friedrich, T.: Late pleistocene climate drivers of early human migration. Nature **538**, 92–95 (2016). https://doi.org/10.1038/nature19365

Verma, S., Warrier, L., Bolia, B., Mehta, S.: Past, present, and future of virtual tourism-a literature review. Int. J. Inform. Manag. Data Insights **2**(2), 100085 (2022). https://doi.org/10.1016/j.ijimei.2022.100085

Wang, D., Li, X.R., Li, Y.: China's "smart tourism destination" initiative: a taste of the service-dominant logic. J. Destin. Mark. Manag. **2**(2), 59–61 (2013). https://doi.org/10.1016/j.jdmm.2013.05.004. Researching Destination Experiences

Yang, F.X., Wang, Y.: Rethinking metaverse tourism: a taxonomy and an agenda for future research. J. Hosp. Tour. Res. **0**(0), 10963480231163509 (2023). https://doi.org/10.1177/10963480231163509

Zambonelli, F., Cabri, G., Leonardi, L.: Developing mobile agent organizations: a case study in digital tourism. In: Proceedings of 3rd International Symposium on Distributed Objects and Applications, pp. 270–279 (2001). https://doi.org/10.1109/DOA.2001.954092

Zhang, L., Yang, J.: Smart Tourism, pp. 862–863. Springer, Berlin (2016). https://doi.org/10.1007/978-3-319-01384-8_175

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Part III

Smart Life Applications
<figure><img src="image_67.png" /></figure>

Chapter 9

Smart-Viticulture and Deep Learning:
Challenges and Recent Developments
on Yield Prediction

Lucas Mohimont, Lilian Hollard, and Luiz Angelo Steffenel

**Abstract** Smart agriculture is a trendy topic as it has a clear impact in both productivity, ecological impact, and improvement of working conditions. Smart viticulture is one of the domains that can benefit both from wireless sensor networks and mobile devices embarked in vineyard labor tools (e.g., on a straddler tractor). One important use case is related to the yield estimation, an invaluable information to drive the harvest organization, plant management, and business's economy. Traditional methods rely on destructive sampling and manual counting, resulting in error rates sometimes greater than 30%. In this chapter, we review existing techniques for the automation of yield estimation and, focusing on deep learning methods, propose some strategies and preliminary results obtained in a production environment.

**Keywords** Computer vision · Deep learning · Viticulture · Yield prediction

**9.1 Introduction**

The development of smart agriculture has raised interesting opportunities for improved automation, logistics, and working conditions. Numerous different technologies have been studied for practical applications in agriculture. For instance, wireless sensor networks allow the collection of data from different locations in a parcel. The acquired data, such as the temperature or the humidity, are used to predict the emergence of diseases (Pérez-Expósito et al. 2017; Steffenel et al. 2021). In addition, the sensors can include embedded cameras to detect the presence of symptoms of illness or deficiencies (Lloret et al. 2011) or simply to control the maturity state of the fruits or estimate the yield. Finally, other sensors such as LIDARs (laser imaging detection and ranging) can estimate the size of the

<footnote>L. Mohimont · L. Hollard · L. A. Steffenel (✉)
Université de Reims Champagne-Ardenne, LICIIS Laboratory, Reims, France
e-mail: lucas.mohimont@univ-reims.fr; lilian.hollard@univ-reims.fr;
luiz-angelo.steffenel@univ-reims.fr</footnote>
canopy or the number of missing plants (Grocholsky et al. 2011). The advance in image processing techniques and the emergence of affordable digital cameras and embarked processors also bring new opportunities for smart agriculture, with the promise to improve the work conditions and the accuracy of forecasts as it can better humans in repetitive, time-consuming, and tedious tasks.

Among these tasks, we can cite the automation of fruit counting, a central problem in smart agriculture as it has a direct impact on yield prediction and management. Several methods have been proposed in recent years, and this concept has been applied to the detection of oranges (Maldonado and Barbosa 2016), bell peppers (Song et al. 2014), and lemons (Dorj et al. 2017).

In the case of viticulture, winegrowers use yield estimation to get decisive information for the business's economy, plant management, and harvest organization. The interest in knowing the yields early enough involves several economic, administrative, and qualitative objectives (Liu et al. 2013). For example, winegrowers can manage the wine market by estimating and controlling the crop volume on a regional or national scale. Yield forecast also helps increase the quality of the wines, by eliminating part of the harvest by thinning out the bunches, as it has been proven that too high density may hinder the quality of the grapes (Xi et al. 2018). This is an essential element in some appellations such as Champagne (INAO), where annual production quotas in kg/ha are defined and any excessive production cannot be transformed in Champagne.

Traditionally, winemakers estimate their production by performing samplings on selected land plots, using a simple formula such as Eq. 9.1 for a given area.

$$ 
\text{Yield (kg/ha)} = \frac{\text{nb plants} \times \text{nb grapes} \times \text{avg grape weight}}{\text{surface}} \quad (9.1) $$

This procedure relies on both grape counting and weighting. Grapes are harvested among a random sample allowing an estimation of the number of grapes by vine, the number of berries per grape, and the weight of the berries. Furthermore, grape counting is mostly performed by a human operator, leading to uncertainties on the precision and dubious repeatability. Weighting of the grapes, on the other hand, requires them to be harvested prematurely, a destructive process that cannot be performed extensively. Some works use historical data to limit grape weighting, but significant variations from year to year can skew the predictions. As a result, a variation of 30% can be found between the estimations and the reality (Dami and Sabbatini 2011). Reducing this error is therefore essential for the organization of the harvest.

The moment yield forecasts are performed also impacts the solutions. Indeed, most winegrowers prefer to obtain estimations at least 1 month before harvest, which usually falls before the *veraison*, i.e., the start of the maturing period where grapes' colors change. This complicates the yield estimation using non-intrusive or non-destructive methods, as the color contrast between the leaves of the vine and the green grapes is reduced (Di Gennaro et al. 2019). In addition, the solutions shall
adapt to the winegrowers' working hours, being tolerant to different light intensities during the day and variable weather conditions (Zhang et al. 2022).

In this work, we focus on the analysis of works where deep learning models are used for grape and berry counting, the first step toward an accurate yield performance forecast. Recent works demonstrate that deep learning contributes to automatically count grapes in the vineyard (Santos et al. 2017; Heinrich et al. 2019) showing more robustness and accuracy than other computing vision methods based on signal processing or traditional machine learning (Diago et al. 2015; Dunn and Martin 2004; Liu et al. 2018).

We also present some results obtained by our team in an effort to develop a yield performance forecast model for the Champagne vineyards. Since 2019, we work on data collection and labeling, as well as the development and training deep learning models for smart-viticulture. Today, our focus is on the development of lightweight and robust models that can later be integrated in edge or embarked devices, allowing on-site real-time processing even in remote locations with no network coverage.

The remainder of the chapter is structured as follows: Sect. 9.2 introduced the literature on grape detection and counting, pointing out traditional and deep learning models as well as the remaining challenges and opportunities. Similarly, Sect. 9.3 focuses on berry counting and weighting. In Sect. 9.4, we introduce our works and some of our preliminary results. Finally, Sect. 9.5 concludes this work.

## 9.2 Grape Detection and Counting

The main component for automatic yield estimation is to detect and count grape clusters (also called *bunches of grapes* or simply *grapes*). In this case, images with grapes are sent to the detection algorithm, whose output includes the number of grapes present in the image and eventually an image containing the location of the grapes as well (see Fig. 9.1).

<figure><img src="image_68.png" /><figcaption>**Fig. 9.1** Example of grape cluster detection (Chardonnay, before veraison) (Mohimont et al. 2022)</figcaption></figure>
This is an important step because it enables several practical applications besides
yield estimation, for example, automatic harvesting using a robot (Reis et al. 2012;
Luo et al. 2016, 2018), the automatic spraying of growth hormones, Berenstein et al.
(2010), the characterization of phenotyping (Rose et al. 2016), or even the detection
of water stress to optimize irrigation (Kang et al. 2023).

Multiple obstacles shall be met during the creation of such an algorithm. First of all, the acquisition method is important: drone/UAV (unmanned aerial vehicle) images are good for general analysis on the canopy (water stress, diseases) but may lack the resolution to identify grape details (Ariza-Sentís et al. 2023). Indeed, precise grape counting and characterization often require ground-based image acquisition solutions, like a camera embedded in a vehicle. Furthermore, several variability factors (lighting, angle, distance, complex background) are present in natural images, and the algorithm must account for (ii) occlusions created by the foliage, and often there is (iii) color confusion between the grapes and the foliage.

A simple approach to detect grapes consists in using thresholding segmentation.
Thresholding segmentation is a simple approach that was first used by Dunn and
Martin (2004) to evaluate the potential of image processing for yield estimation.
Here, one or more thresholds (e.g., color or brightness filters) are applied to
the images to keep the areas that correspond to the fruits only. Thresholding
segmentation algorithms usually have short execution times and are easy to develop.
However, these algorithms have several drawbacks that limit their use in the field
without partial control of the environment. The main issue is that strict thresholds are
not robust to the color variations caused by natural lighting and the background. The
thresholds must be determined in each situation, and the grapes must have attained
the maturity to discriminate their color from the ground or the leaves (Di Gennaro
et al. 2019; Torres-Sánchez et al. 2021). For this reason, most works limit its usage
to nighttime with a lighting source, in order to erase the background elements (sky,
ground, and further rows). Indeed, threshold segmentation has been applied on the
field on red and white varieties but only during the night, with artificial lighting
(Reis et al. 2012).

A different approach relies on traditional machine learning (ML) to make the
detection of grapes more robust to lighting variations. It segments the image by
using small pixel neighborhoods, or blocks, as inputs for a classification model.
Examples of pixel neighborhoods are illustrated in Fig. 9.2.

The pixel classification model produces a binary output (grape or non-grape) that is applied to the central pixel or the entire input block. Classical ML techniques cannot be directly applied to the raw images, as features must be extracted first for each block. A simple feature is the mean values of the R, G, and B channels to produce a vector with only three components (Chamelat et al. 2006). Similarly, the mean value of the RGB (red, green, blue) channels was used in several studies (Casser 2016; Luo et al. 2016; Font et al. 2015). A genetic algorithm was suggested to select the best color channels among several possible color spaces: RGB, HSV (hue, saturation, value), and CIELAB (Behroozi-Khazaei and Maleki 2017). Nonetheless, pixel classification suffers from several limitations, including sensitivity to color (variety) and potentially long execution time, and it heavily
<figure><img src="image_69.png" /><figcaption>**Fig. 9.2** Examples of pixel neighborhoods extracted from a vine image (Pinot Noir, during veraison) (Mohimont et al. 2022)</figcaption></figure>

depends on the choice of extractors because the quality of the features will impact
the final results.

9.2.1 Using Deep Learning

Deep learning has recently been applied to the problem of detecting and count-
ing grapes. A naive approach uses the same pixel-wise classification discussed
previously, but with a neural network that combines both feature extraction and
classification (Lopes et al. 2017; Cecotti et al. 2020; Milella et al. 2019).

The use of convolutional neural networks (CNN) simplifies the detection step
because the model will learn the best features from the data. However, this method
is always limited by the small size of the blocks and the long calculations required
to segment one image. Several CNNs have been studied to predict the mass of
the grapes automatically (Silver et al. 2019). The error is relatively low, 11% in
controlled conditions with an artificial background, but the proposed method has
many drawbacks. The distance between the grapes and the camera varies, impacting
the results. This limit could be overcome using a depth sensor, but this approach
should also be evaluated on several rows and with different varieties to better
appreciate its potential.

Several popular object detection models, Faster R-CNN (Ren et al. 216), R-CNN (Girshick et al. 2014), R-FCN (Dai et al. 2016), and SSD (Liu et al. 2016), have been tested on the task of grape detection and counting using videos (Heinrich et al. 2019). SSD was applied to grapes detection at two stages (0.5 cm berries and 1.2
cm berries) in natural conditions and with real-time hardware acceleration (TPU) (Aguiar et al. 2021). The Mask R-CNN model (He et al. 2018), which detects objects and segments them, has also been applied to the detection of grapes (Santos et al. 2020). The counting of grapes from videos is corrected with a *structure-from-motion* method that estimates the 3D position along the camera path. The 3D position is used as an identifier to avoid counting one grape twice. Mask R-CNN and multiple Yolo models (Redmon et al. 2016) were compared on the dataset published by Deng et al. (2020), Santos et al. (2020). Yolo models were also compared on red grape detection from smartphone images (Li et al. 2021). The Faster R-CNN model was applied similarly with a tracking algorithm to process videos of Riesling and Pinot Noir vines taken at night with artificial lighting (Jaramillo et al. 2021). Mask R-CNN was studied in multiple recent works Barbole and Jadhav (2021), Ghiani et al. (2021), Yin et al. (2021). The authors of Barbole and Jadhav (2021) compared the performances of Mask R-CNN to other models like U-Net and Yolov3 (Redmon et al. 2018) (they also used the WGISD benchmark published by Santos et al. 2020). They found better precision with U-Net and better recall with Yolov3. The authors of Ghiani et al. (2021) applied Mask-RCNN to the GrapeCS-ML dataset published by Seng et al. (2018). It contains images of different varieties taken in natural conditions at different stages. One limitation of this dataset is that most images only contain one grape cluster, so the resulting model cannot process images with many clusters.

Mask R-CNN was also applied with stereo images to detect and reconstruct 3D models of the grapes for automated harvesting (Yin et al. 2021). The Yolov4 (Bochkovskiy et al. 2020) model was applied to low-resolution images of white grapes to measure the correlation between grape counting and fruit weight (Sozzi et al. 2021). A low correlation was found between the number of detected clusters and the actual number of clusters (0.24 R²). The correlation between the number of detected clusters and fruit weight was better (0.59 R²), indicating a potential application for yield estimation in future works. Similarly, YOLOv5s was successfully used by Shen et al. (2023) to perform real-time detection and counting in a defoliated dataset, reaching a 79% accuracy after applying filtering for motion-induced errors.

The authors of Chen et al. proposed a modified PSPNet (Zhao et al. 2016) model for grape segmentation (Chen et al. 2021). It was applied on white and red grapes after veraison. They reached good segmentation performances with an average of 87.42% IoU (Intersection over Union). Similar results were obtained on both red and white grapes. However, the main limitation of semantic segmentation models is their inability to separate overlapped clusters. This results in inaccurate grape counting. Similarly, the authors of Peng et al. (2021) applied a DeepLabV3+ model to multiple varieties of red, green, purple, and black colors, with or without spherical berries. They reached an IoU of 88.44% with an inference of 60 ms/image, which allows an automatic harvesting deployment. The authors are well aware of the difficulty of separating overlapping grapes and proposed the use of a depth sensor (Intel Realsense D435 stereo camera) to solve this problem (Peng et al. 2021).
Their results, 85.6% recall and 87.1% precision, show the viability of their solution. However, this method should be compared to object detection models.

Finally, a generative model was proposed to adapt images to different lighting conditions (Fei et al. 2021). A CycleGAN (Zhu et al. 2020) model was used to translate images taken in daylight to images taken at night with artificial lighting. This step can be useful to increase the size of existing datasets and make the models robust to the varying environmental conditions.

### 9.2.2 Remaining Challenges

Although effective, most of the methods presented in the previous section only have been applied to controlled environments and are seldom adapted to the processing of real-time video streams. For this reason, looking for more performing solutions offering advanced computing and AI capabilities is interesting, and several works now explore the interaction between different layers (cloud-fog-edge) (Vermesan et al. 2021), including Edge-AI capable devices.

Developing real-time Edge AI also helps improve the autonomy of the devices, as one of the main challenges in smart agriculture is the difficulty to deploy solutions capable of handling complex actions in environments subjected to high network constraints (absence of network coverage, high latency) and the limited performance of embarked devices.

Finally, another challenge comes from the complexity of the natural environments in which these devices are supposed to work. For example, Kumrai et al. (2020) demonstrate the difficulty of recognizing human activity according to the angle, capturing and extracting information from the captured images for a camera mounted on a robot. This is somehow similar to the problems faced when detecting and counting grapes on the vineyard, as the angle and other plant elements (leaves, branches) lead to image occlusions that affect the identification of the grapes or the variance of luminosity during the day. Hence, robust deep learning models shall withstand such environmental variance all while delivering accurate results.

## 9.3 Berry Counting and Weighting

If counting the grapes is the first step toward an automatic yield prediction, it is often insufficient to provide accurate results. Indeed, a complementary step is required to reduce the variability caused by different grape bunch sizes in the plot: counting and weighting the berries. Indeed, according to the work from Clingeleffer et al. (2001), the number of grapes per vine, the number of berries per grape, and the weight of the berries make up, respectively, 60%, 30%, and 10% of the yield variance in a model using these three factors.
<figure><img src="image_70.png" /><figcaption>**Fig. 9.3** Example of berry counting (Chardonnay, before veraison) (Mohimont et al. 2022)</figcaption></figure>

As this method is inherently destructive, the sample size is often reduced in traditional estimation methods, leading to important biases. As a consequence, several studies propose methods for automatic counting of berries through computer vision. These methods consist mainly in the detection of visible berries and the further estimation of the number of grape bunches. As stated before, these two factors account for up to 90% of the variance in the yield estimation. An example of automatic berry counting is illustrated in Fig. 9.3.

The problem with berries counting can therefore be formulated as a circle detection or local maxima detection task. Most signal processing solutions use the specular reflection caused by the light on the surface of the berries, producing a pattern that follows a Gaussian distribution. The berries appear as bright little spheres that make them more easily distinguishable from the background of the image. One of the first methods exploiting specular reflection used a Gaussian kernel to process images taken with a smartphone with flash (Grossetete et al. 2012), but it was limited to close-up images of one or several grapes to limit confusion caused by the background.

Several other image analysis algorithms have been used to detect berries: the h-maxima transform (Aquino et al. 2017, 2018), the fast radial symmetry transform (Nuske et al. 2014; Luo et al. 2021), and the Hough transform (Murillo-Bracamontes et al. 2012; Diago et al. 2015; Liu et al. 2015, 2020; Roscher et al. 2014; Rahman et al. 2014; Keresztes et al. 2018). The fast radial symmetry transform detects berry candidates rapidly, whereas the Hough transform potentially has higher memory and computing power requirements. An alternative to the h-maxima transform, named invariant maxima detector, is proposed by (Nuske et al. 2014) to detect berry candidates with artificial lighting. Indeed, these works show that detectors based on local maximums are too sensitive to variations in natural lighting, and the use of a flash or lamp creates a uniform specular reflection on the surface of the berries, making the detection easier.
This classical approach is therefore complex because it requires several algo-
rithms that need to be fine-tuned to (1) detect the berry candidates, (2) extract the
features of each candidate, and (3) filter the false positives with a classifier.

Of course, counting grapes and berries may not be enough to predict the
yield, as the final weight/volume also depends on other factors such as the
average grape/berry size or weight and the weather conditions prior to the harvest
(Anastasiou et al. 2023).

9.3.1 Deep Learning Strategies

Deep learning can simplify berry counting by processing raw images. CNNs can be used as a feature extractor and as a classifier at the same time to filter berry candidates. This technique has been implemented on a Raspberry Pi (Keresztes et al. 2018) for real-time detection. CNNs have also been shown to be more accurate than support vector machines (SVM) (Škrabánek 2018). Furthermore, CNNs have been adapted to object counting with density map prediction. The areas with strong density correspond to the berries’ location (Coviello et al. 2020). This method manages to count berries with an error rate of approximately 10%. CNNs have also been adapted to image segmentation. These CNNs have shown very promising results. A fully convolutional neural (FCN) model was proposed to count the berries with two classes of labeling, the first for the inside of the berries and the second for the edges (Zabawa et al. 2020). The model is faster and better than Mask R-CNN (He et al. 2018) and UNet (Ronneberger et al. 2015). This difference is explained by the structure of the models as Mask R-CNN uses a complex structure to detect and segment objects and by the small size of the database. Deng et al. (2020) used the Hough Transform for circle counting after grape detection with the Yolov4 model (it is similar to the method proposed by Rudolph et al. for flower counting Rudolph et al. 2019). One limitation of this algorithm is its sensibility to berries’ apparent size (the radius must be known).

Miao et al. proposed to use edge detection to solve the problem of counting overlapping berries (Miao et al. 2021). The proposed methodology is complex with a holistically nested edge detection model for berry edge segmentation, a Yolo model for berry detection, and a RANSAC algorithm for sphere fitting. It was evaluated on images of individual grape clusters in different conditions (laboratory or field conditions). A simpler object detection model based on RetinaNet (Lin et al. 2018), modified with a counting section, was evaluated on three plants for counting bananas-per-bunch, spikelets-per-wheats-spike, and berries-per-grape-cluster in natural conditions (Khoroshevsky et al. 2021).

Palacios et al. (2022a,b) proposed a three-step segmentation process, with SegNet models, for grapes, berries, and the canopy features extraction from images taken at night with lighting. The goal is to use segmentation to measure traits with predictive power for actual berry counting (visible + hidden) with regression models.
### 9.3.2 Current Challenges

A problem arises from deploying the berry detection algorithms in the field. They must be applied to image sequences to cover entire rows of vines. Doing this introduces the risk of counting the same grapes twice. Therefore, more or less complex solutions have been applied to avoid redundancy during the counting. A simple one reconstructs the full image of the row using a video and then keeps the highest counts in the overlapping areas (Nuske et al. 2014). It is also possible to follow the detected grapes from one image to the next based on the distance between the grapes in consecutive images (Heinrich et al. 2019). Another mechanical solution consists in taking an image based on the distance covered by a vehicle (one every two meters, for instance) (Aquino et al. 2018).

Another challenge is related to the estimation of the yield volume. The classic modeling approach estimated the total number of berries with a regression model, generally linear. The total estimation number is then converted into mass using the historical average weight of the berries or by extracting samples from the field. This method used by Aquino et al. (2018) with a linear model predicts the total weight based on the visible fruits. An average error rate of 12.83% was achieved on five varieties over 30 segments of three vines.

A recent method uses new techniques to count berries and estimate the total number of berries in a grape (Liu et al. 2020). A 3D reconstruction method that only needs a single 2D image is built by positioning the berries to fill the estimated profile, estimated from the edges, of a single grape bunch. Grape compactness is used as a parameter to simulate different varieties and growth stages. Therefore, this method directly estimates the number of berries in a grape bunch. Although it has only been applied to partially controlled conditions with images of individual grapes with an artificial background, a yield estimation was done by combining this method with another one proposed by the same authors Liu et al. (2017) based on the counting of the shoots. The error rates of predicted yield on three parcels were 3%, 6%, and 16%, which is comparable to or better than other methods.

Finally, the phenological stage also influences the results. For instance, berries in early stages are harder to count due to their reduced size, and their color is similar to the foliage before ripening (Fig. 9.4). Good results were observed in practice by Nuske et al. in 2014 up to 75 days before the harvest (Nuske et al. 2014). The counting can be more difficult once the grapes close (the berries touch each other). The work of Liu et al. shows similar yield estimation performances before and after the ripening (Liu et al. 2020).

Also, we must account for the grapes hidden by the foliage, one of the main problems currently limiting yield prediction performances. Contrarily to several works conducted in laboratory or controlled environments, a winemaker cannot trim his vines. A possible solution is to use a blower to remove the foliage temporarily, but it has not been quantitatively evaluated (Nellithimaru and Kantor 2019). An additional modeling step is hence necessary, and it could benefit from additional variables such as the porosity of the canopy (0.82 R² correlation between the canopy's porosity and the percentage of visible grapes) (Victorino et al. 2019).
<figure><img src="image_71.png" /><figcaption>Stage 27-29 - Young fruits begin to swell, berries small</figcaption></figure>

<figure><img src="image_72.png" /><figcaption>Stage 31 - Berries pea-sized</figcaption></figure>

<figure><img src="image_73.png" /><figcaption>Stage 33 - Begin of berry touch</figcaption></figure>

<figure><img src="image_74.png" /><figcaption>Stage 35 - begin of berry ripen; if it applies, loss of green color (veraison)</figcaption></figure>

<caption>**Fig. 9.4** Different stages of a vine</caption>

Recent studies have shown that the fruit area might be more robust to occlusion than berry counting (Victorino et al. 2020). This work was extended by comparing yield estimation from images to the classical manual sampling approach on six parcels (Victorino et al. 2022). They used artificial background and manual segmentation of the images to extract meaningful features (such as visible bunch area, canopy porosity, etc.) for yield estimation. In this manner, ideal performances of image analysis methods were evaluated against the current yield estimation process. An average error rate of 8% was found with image analysis compared to 31% for manual sampling. It is also noted that the error increases with canopy density. Although fruit area prediction potential is still highly impacted by foliage, moderate defoliation can help obtain a better correlation (Iñiguez et al. 2021). In practice, the conditions in the vineyard are more difficult: smaller green grapes, high canopy occlusion, natural background, lighting, etc.

Features related to occlusion and fruit area were used as yield predictors in the work of Palacios et al. (2022a,b). An error rate, including six varieties, of 29.77% NRMSE (Normalized root-mean-square error, in kg per vine, 0.83 R²) was reported with images taken 66 days before harvest. Error rates ranged from 16.47% to 39.17% depending on the variety. Zabawa et al. (2020), Zabawa et al. (2022) reported an MAE rate of 26% for 70 vines of Riesling. They only used berry detection and a simple equation (estimated number of berries multiplied by the berry weight).

A possible solution to solve the hidden-berries problem was proposed by Kierdorf et al. (2021). A CycleGAN generative model was used to predict the
location of hidden berries from pairs of vine images before and after defoliation. As a result, better counting performances were obtained, 0.88 $R^2$ compared to 0.72 $R^2$ on the images without defoliation. However, it has not yet been applied to more complex vine images with a natural background. More experiments are needed to determine the practicality of this new method for yield prediction.

## 9.4 Contributions

As illustrated above, several grape detection methods are using deep learning and image processing for grape detection as well as berry counting. These methods, although effective, do not focus on speed of execution or the use of real-time video streams. We are therefore faced with a problem confronting artificial intelligence models with high prediction performance against models capable of combining high prediction and execution time performance in the edge. Indeed, the ability to optimize and compress deep learning models leads toward the convergence of solutions between robust models in the cloud and stand-alone models with deep learning inference directly at the edge.

In order to achieve such objectives, we have conducted several experiences since 2019 for the European projects H2020 AI4DI (Mohimont et al. 2021) and KDT JU EdgeAI, using different cameras and methods, for example, with a GoPro fixed on a picket or embarked on a tractor (Fig. 9.5a). For instance, one of the models we developed was an alleged UNet encoder-decoder with a ResNet-34 backbone with 24M parameters and sufficiently optimized to be deployed in embedded computing boards like a Nvidia Jetson Nano 2GB. As the dataset is limited (approximately 600 photos were taken in the 2019 campaign, from which 386 photos were labeled), there is a risk of overfitting. To avoid this problem, we applied transfer-learning from a ResNet-34 backbone pre-trained on the ImageNet database. Transfer learning allow us to speed up the training process and, as the original model was trained in a larger variety of images, reduces the risk of overfitting. Another optimization we

<figure><img src="image_75.png" /><figcaption>Fig. 9.5 (a) Camera attached to the vehicle, (b) defoliated vine</figcaption></figure>
have made is to avoid processing the whole images but, instead, to proceed them by
patches of 512×512 pixels. In this way, we greatly increase the size of the training
base. A mini-batch size of 16 is then used, and about 30% of the database is used for
model validation. To speed up the training phase, we have used the DGX-1 server
from the ROMEO Supercomputing Center, a dedicated deep learning machine with
eight Nvidia Tesla V100 GPUs.

After training, this model presents an IoU score of 0.69 and an F1 score of 0.8 (the IoU is limited due to the lack of precision in the labeling). Nonetheless, the model allows detecting nearly 100% of the grapes with a false-positive rate near 0%. In this work, we calibrated our deep learning algorithms for automated yield using a systematic tracking of several rows in the vineyard car (Rossignon 2020). This tracking, performed over four rows (200 vines) at different phenological stages, has included the counting of classic organs (vine, flowers/grapes) using traditional strategies (random counting or by the sampling of the parcels) as well as sampling the berries to estimate their volume and ripeness.

The counting of the grapes has also been done by unveiling hidden grapes by defoliation. Hence, the operator first counts the visible grapes in the plant and, after defoliation, takes a second picture (Fig. 9.5b). Around 30 images were taken in this way and labeled and used to help identify partially hidden grapes. The model can therefore be used to simply count grapes as seen in Fig. 9.6a or segment the image for further analysis (berry counting), like in Fig. 9.6b.

Thanks to the manual and automated counting data, a linear regression model
has been generated for each row, followed by a cross-examination of each model
using the three other rows (Fig. 9.7). Although the error rate varies from 0% to
31%, depending on the model and the row, we obtain an average error rate of 14%.
This is better than current error rates with the traditional approach and can be further
improved. Hence, the improvement of this analysis is based on a better distinction
between grapes in the foreground and grapes in the background, as well as using
non-linear regression models and other variables such as the porosity of the canopy
(to account for occlusions).

<figure><img src="image_76.png" /><figcaption>**Fig. 9.6** Examples of image analysis with deep learning. (a) Grape counting. (b) Image segmentation for further berry counting</figcaption></figure>
<figure><img src="image_77.png" /><figcaption>**Fig. 9.7** Correlation between visible grapes and the total number of grapes</figcaption></figure>

<figure><img src="image_78.png" /><figcaption>**Fig. 9.8** (a) Prototype device installed in a tractor. (b) Biomass chart obtained with Physiocap</figcaption></figure>

As this is an ongoing project, we regularly iterate with our partner Vranken-Pommery on how to effectively deploy the solution according to their needs. The optimization efforts made to execute the AI algorithm in edge devices allow us to run most of the steps in an embarked node, as illustrated in Fig. 9.8a. Processing most information onsite allows to reduce the amount of data to store and favor the deployment on areas with reduced or unavailable Internet connection.

We are now preparing the prototype devices to be tested in preparation to the Summer 2023 harvest, based on a Jetson Nano device equipped with a camera and a RTK GPS receiver, making it an affordable solution (less than 500 for the complete equipment). Grape count shall be performed from the middle of June 2023, and together with geolocation data, we aim at producing detailed grape density maps similar to the biomass maps from Fig. 9.8b produced by the Physiocap sensor,<fnref n="1" /> allowing a precise harvest planning (workers, material, logistics).

Berry counting is also being introduced in our workflow, as illustrated in Fig. 9.9. Additional work is required to extrapolate the total number of berries in the grape

<footnote n="1">¹ https://www.physiocap.eu/</footnote>
<figure><img src="image_79.png" /><figcaption>**Fig. 9.9** Examples of berry detection</figcaption></figure>

(3D counting) from a 2D image (visible berries) but also to allow real-time counting in an embedded device.

**9.5 Conclusion**

Deep learning has introduced several innovations on the domain of smart agriculture. Contrarily to traditional signal-processing and image analysis methods, deep learning seems to be more robust to natural variations, and pretraining allows easy transfer to similar problems. This is indeed the case of grapevine yield performance forecast, a smart-viticulture use case that can heavily benefit from automation. Generally speaking, counting fruits is a difficult task because the aspect of the plant can vary enormously depending on the point of view and development stages.

In this chapter, we focus on grape yield forecast strategies that use deep learning to count grapes or berries, providing a survey on existing works. We can observe that many works present good performances, but these results shall be put in perspective as most authors rely in controlled scenarios or specific lightning conditions that are not compatible with field deployment at scale. Nonetheless, they point toward some interesting techniques that can be further improved using more robust models and richer datasets.

We also presented some preliminary results from our team, which focus on grape detection and counting under natural conditions. The good performances we obtained show the interest of deep learning and encourage the pursuit of the works, even though additional modeling steps are still needed to account for the non-visible part of the fruits hidden by leaves or the number of berries per cluster and their average weight. Furthermore, a compromise between accuracy and inference speed is necessary for a practical implementation allowing a real-time and autonomous usage. We expect that future improvements in our models and more comprehensive data sources will allow the creation of better yield estimation models.
**Acknowledgments** This work has been started as part of the project AI4DI "Artificial Intelligence for Digitizing Industry" (ECSEL JU grant agreement No 826060) and continues as part of the EdgeAI project "Edge AI Technologies for Optimised Performance Embedded Systems" under grant agreement No 101097300. The EdgeAI project is co-funded by grants from Austria, Belgium, France, Greece, Italy, Latvia, Luxembourg, Netherlands, Norway, and the European Key Digital Technologies Joint Undertaking (KDT JU).

We want to thank Vranken-Pommery Monopole for allowing image collection in their vineyards. We also thank the ROMEO Computing Center<fnref n="2" /> of Université de Reims Champagne-Ardenne, whose Nvidia DGX-1 server allowed us to accelerate the training steps and compare several model approaches.

## References

Aguiar, A.S., Magalhães, S.A., dos Santos, F.N., Castro, L., Pinho, T., Valente, J., Martins, R., Boaventura-Cunha, J.: Grape bunch detection at different growth stages using deep learning quantized models. Agronomy **11**(9) (2021). https://www.mdpi.com/2073-4395/11/9/1890

Anastasiou, E., Templalexis, C., Lentzou, D., Biniari, K., Xanthopoulos, G., Fountas, S.: Do soil and climatic parameters affect yield and quality on table grapes? Smart Agric. Technol. **3**, 100088 (2023). https://doi.org/10.1016/j.atech.2022.100088

Aquino, A., Diago, M.P., Millán, B., Tardaguila, J.: A new methodology for estimating the grapevine-berry number per cluster using image analysis. Biosyst. Eng. **156**, 80-95 (2017). https://doi.org/10.1016/j.biosystemseng.2016.12.011

Aquino, A., Millan, B., Diago, M.P., Tardaguila, J.: Automated early yield prediction in vineyards from on-the-go image acquisition. Comput. Electron. Agric. **144**, 26-36 (2018). https://doi.org/10.1016/j.compag.2017.11.026

Ariza-Sentís, M., Vélez, S., Valente, J.: Dataset on UAV RGB videos acquired over a vineyard including bunch labels for object detection and tracking. Data Brief **46**, 108848 (2023). https://doi.org/10.1016/j.dib.2022.108848

Barbole, M.D., Jadhav, D.P.: Comparative analysis of deep learning architectures for grape cluster instance segmentation. Inform. Technol. Ind. **9**(11), 344-352 (2021). https://doi.org/10.17762/itii.v9i1.138

Behroozi-Khazaei, N., Maleki, M.R.: A robust algorithm based on color features for grape cluster segmentation. Comput. Electron. Agric. **142**, 41-49 (2017). https://doi.org/10.1016/j.compag.2017.08.025

Berenstein, R., Shahar, O.B., Shapiro, A., Edan, Y.: Grape clusters and foliage detection algorithms for autonomous selective vineyard sprayer. Intell. Service Robot. **3**(44), 233-243 (2010). https://doi.org/10.1007/s11370-010-0078-z

Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: Optimal speed and accuracy of object detection (2020)

Casser, V.: Using feedforward neural networks for color based grape detection in field images. In: CSCUBS 2016—Computer Science Conference for University of Bonn Students, pp. 23-33 (2016). https://doi.org/paper/Using-Feedforward-Neural-Networks-for-Color-Based-Casser/139ccba0b3a00565f61febcc62f98c6c44cca990

Cecotti, H., Rivera, A., Farhadloo, M., Pedroza, M.A.: Grape detection with convolutional neural networks. Expert Syst. Appl. **159**, 113588 (2020). https://doi.org/10.1016/j.eswa.2020.113588

<footnote n="2">² https://romeo.univ-reims.fr</footnote>
Chamelat, R., Rosso, E., Choksuriwong, A., Rosenberger, C., Laurent, H., Bro, P.: Grape detection by image processing. In: IECON 2006—32nd Annual Conference on IEEE Industrial Electronics, pp. 3697–3702 (2006). https://doi.org/10.1109/IECON.2006.347704

Chen, S., Song, Y., Su, J., Fang, Y., Shen, L., Mi, Z., Su, B.: Segmentation of field grape bunches via an improved pyramid scene parsing network. Int. J. Agric. Biol. Eng. **14**(6), 185–194 (2021)

Clingeleffer, P.R., Martin, S.R., Dunn, G.M., Krstic, M.P.: Crop development, crop estimation and crop control to secure quality and production of major wine grape varieties: a national approach. Adelaide, Grape and Wine Research and Development Corporation (2001)

Coviello, L., Cristoforetti, M., Jurman, G., Furlanello, C.: Gbcnet: In-field grape berries counting for yield estimation by dilated CNNs. Appl. Sci. **10**(1414), 4870 (2020). https://doi.org/10.3390/app10144870

Dai, J., Li, Y., He, K., Sun, J.: R-FCN: Object detection via region-based fully convolutional networks (2016)

Dami, I., Sabbatini, P.: Crop Estimation of Grapes. The Ohio State University Fact Sheet (2011)

Deng, G., Geng, T., He, C., Wang, X., He, B., Duan, L.: Tsgye: Two-stage grape yield estimation. In: Yang, H., Pasupa, K., Leung, A.C.S., Kwok, J.T., Chan, J.H., King, I. (eds.) Neural Information Processing, pp. 580–588. Springer, Cham (2020)

Di Gennaro, S.F., Toscano, P., Cinat, P., Berton, A., Matese, A.: A low-cost and unsupervised image recognition methodology for yield estimation in a vineyard. Front. Plant Sci. **10**, 559 (2019). https://doi.org/10.3389/fpls.2019.00559

Diago, M.P., Tardaguila, J., Aleixos, N., Millan, B., Prats-Montalban, J.M., Cubero, S., Blasco, J.: Assessment of cluster yield components by image analysis. J. Sci. Food Agric. **95**(66), 1274–1282 (2015). https://doi.org/10.1002/jsfa.6819

Dorj, U.O., Lee, M., seok Yun, S.: An yield estimation in citrus orchards via fruit detection and counting using image processing. Comput. Electron. Agric. **140**, 103–112 (2017). https://doi.org/10.1016/j.compag.2017.05.019

Dunn, G.M., Martin, S.R.: Yield prediction from digital image analysis: a technique with potential for vineyard assessments prior to harvest. Aust. J. Grape Wine Res. **10**(33), 196–198 (2004). https://doi.org/10.1111/j.1755-0238.2004.tb00022.x

Fei, Z., Olenskyj, A., Bailey, B.N., Earles, M.: Enlisting 3D crop models and GANs for more data efficient and generalizable fruit detection (2021)

Font, D., Tresanchez, M., Martínez, D., Moreno, J., Clotet, E., Palacín, J.: Vineyard yield estimation based on the analysis of high resolution images obtained with artificial illumination at night. Sensors **15**(4), 8284–8301 (2015). https://doi.org/10.3390/s150408284

Ghiani, L., Sassu, A., Palumbo, F., Mercenaro, L., Gambella, F.: In-field automatic detection of grape bunches under a totally uncontrolled environment. Sensors **21**(11) (2021). https://doi.org/10.3390/s21113908

Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation (2014)

Grocholsky, B., Nuske, S., Aasted, M., Achar, S., Bates, T.: A camera and laser system for automatic vine balance assessment. In: American Society of Agricultural and Biological Engineers Annual International Meeting 2011, ASABE 2011, vol. 7 (2011). https://doi.org/10.13031/2013.38146. Journal Abbreviation: American Society of Agricultural and Biological Engineers Annual International Meeting 2011, ASABE 2011

Grossetete, M., Berthoumieu, Y., Da Costa, J.P., Germain, C., Lavialle, O., Grenier, G.: Early estimation of vineyard yield: site specific counting of berries by using a smartphone. In: International Conference on Agriculture Engineering (AgEng) (2012)

He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN (2018). arXiv:1703.06870 [cs]

Heinrich, K., Roth, A., Breithaupt, L., Möller, B., Maresch, J.: Yield prognosis for the agrarian management of vineyards using deep learning for object counting. Wirtschaftsinformatik 2019 Proceedings p. 15 (2019)

INAO: AOC champagne—conditions de production
Íñiguez, R., Palacios, F., Barrio, I., Hernández, I., Gutiérrez, S., Tardaguila, J.: Impact of leaf occlusions on yield assessment by computer vision in commercial vineyards. Agronomy **11**(5) (2021). https://doi.org/10.3390/agronomy11051003

Jaramillo, J., Vanden Heuvel, J., Petersen, K.H.: Low-cost, computer vision-based, prebloom cluster count prediction in vineyards. Front. Agron. **3**, 8 (2021). https://doi.org/10.3389/fagro.2021.648080

Kang, C., Diverres, G., Achyut, P., Karkee, M., Zhang, Q., Keller, M.: Estimating soil and grapevine water status using ground based hyperspectral imaging under diffused lighting conditions: addressing the effect of lighting variability in vineyards. Comput. Electron. Agric. **212**, 108175 (2023). https://doi.org/10.1016/j.compag.2023.108175

Keresztes, B., Abdelghafour, F., Randriamanga, D., Da Costa, J.P., Germain, C.: Real-time fruit detection using deep neural networks. In: 14th International Conference on Precision Agriculture (2018)

Khoroshevsky, F., Khoroshevsky, S., Bar-Hillel, A.: Parts-per-object count in agricultural images: solving phenotyping problems via a single deep neural network. Remote Sensing **13**(13) (2021). https://doi.org/10.3390/rs13132496

Kierdorf, J., Weber, I., Kicherer, A., Zabawa, L., Drees, L., Roscher, R.: Behind the leaves—estimation of occluded grapevine berries with conditional generative adversarial networks (2021)

Kumrai, T., Korpela, J., Maekawa, T., Yu, Y., Kanai, R.: Human activity recognition with deep reinforcement learning using the camera of a mobile robot. In: 2020 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp. 1–10 (2020). https://doi.org/10.1109/PerCom45495.2020.9127376

Li, H., Li, C., Li, G., Chen, L.: A real-time table grape detection method based on improved yolov4-tiny network in complex background. Biosyst. Eng. **212**, 347–359 (2021). https://doi.org/10.1016/j.biosystemseng.2021.11.011

Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection (2018)

Liu, S., Cossell, S., Tang, J., Dunn, G., Whitty, M.: A computer vision system for early stage grape yield estimation based on shoot detection. Comput. Electron. Agric. **137**, 88–101 (2017). https://doi.org/10.1016/j.compag.2017.03.013

Liu, S., Li, X., Wu, H., Xin, B., Tang, J., Petrie, P.R., Whitty, M.: A robust automated flower estimation system for grape vines. Biosyst. Eng. **172**, 110–123 (2018). https://doi.org/10.1016/j.biosystemseng.2018.05.009

Liu, S., Marden, S., Whitty, M.: Towards automated yield estimation in viticulture. In: Proceedings of the Australasian Conference on Robotics and Automation, p. 9 (2013)

Liu, S., Whitty, M., Cossell, S.: A lightweight method for grape berry counting based on automated 3 d bunch reconstruction from a single image. In: ICRA, International Conference on Robotics and Automation (IEEE), Workshop on Robotics in Agriculture (2015)

Liu, S., Zeng, X., Whitty, M.: A vision-based robust grape berry counting algorithm for fast calibration-free bunch weight estimation in the field. Comput. Electron. Agric. **173**, 11 (2020). https://doi.org/10.1016/j.compag.2020.105360

Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. Lecture Notes in Computer Science pp. 21–37 (2016). https://doi.org/10.1007/978-3-319-46448-0_2

Lloret, J., Bosch, I., Sendra, S., Serrano, A.: A wireless sensor network for vineyard monitoring that uses image processing. Sensors **11**(6), 6165–6196 (2011). https://doi.org/10.3390/s110606165

Lopes, C., Torres, A., Guzman, R., Graça, J., Monteiro, A., Braga, R., Barriguinha, A., Victorino, G., Reys, M.: Using an unmanned ground vehicle to scout vineyards for non-intrusive estimation of canopy features and grape yield. In: 20th GiESCO International Meeting (2017)

Luo, L., Liu, W., Lu, Q., Wang, J., Wen, W., Yan, D., Tang, Y.: Grape berry detection and size measurement based on edge image processing and geometric morphology. Machines **9**(10), 233 (2021)
Luo, L., Tang, Y., Lu, Q., Chen, X., Zhang, P., Zou, X.: A vision methodology for harvesting robot to detect cutting points on peduncles of double overlapping grape clusters in a vineyard. Comput. Ind. **99**, 130–139 (2018). https://doi.org/10.1016/j.compind.2018.03.017

Luo, L., Tang, Y., Zou, X., Wang, C., Zhang, P., Feng, W.: Robust grape cluster detection in a vineyard by combining the AdaBoost framework and multiple color components. Sensors **16**(1212), 21 (2016). https://doi.org/10.3390/s16122098

Luo, L., Tang, Y., Zou, X., Ye, M., Feng, W., Li, G.: Vision-based extraction of spatial information in grape clusters for harvesting robots. Biosyst. Eng. **151**, 90–104 (2016). https://doi.org/10.1016/j.biosystemseng.2016.08.026

Maldonado, W., Barbosa, J.C.: Automatic green fruit counting in orange trees using digital images. Comput. Electron. Agric. **127**, 572–581 (2016). https://doi.org/10.1016/j.compag.2016.07.023

Miao, Y., Huang, L., Zhang, S.: A two-step phenotypic parameter measurement strategy for overlapped grapes under different light conditions. Sensors **21**(13) (2021). https://doi.org/10.3390/s21134532

Milella, A., Marani, R., Petitti, A., Reina, G.: In-field high throughput grapevine phenotyping with a consumer-grade depth camera. Comput. Electron. Agric. **156**, 293–306 (2019). https://doi.org/10.1016/j.compag.2018.11.026

Mohimont, L., Alin, F., Rondeau, M., Gaveau, N., Steffenel, L.A.: Computer vision and deep learning for precision viticulture. Agronomy **12**(10) (2022). https://doi.org/10.3390/agronomy12102463

Mohimont, L., Roesler, M., Rondeau, M., Gaveau, N., Alin, F., Steffenel, L.A.: Comparison of machine learning and deep learning methods for grape cluster segmentation. In: Boumerdassi, S., Ghogho, M., Renault, É. (eds.) Smart and Sustainable Agriculture, pp. 84–102. Springer, Cham (2021)

Murillo-Bracamontes, E.A., Martinez-Rosas, M.E., Miranda-Velasco, M.M., Martinez-Reyes, H.L., Martinez-Sandoval, J.R., Cervantes-de Avila, H.: Implementation of Hough transform for fruit image segmentation. Proc. Eng. **35**, 230–239 (2012). https://doi.org/10.1016/j.proeng.2012.04.185

Nellithimaru, A.K., Kantor, G.A.: Rols : Robust object-level slam for grape counting. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2648–2656 (2019). https://doi.org/10.1109/CVPRW.2019.00321

Nuske, S., Wilshusen, K., Achar, S., Yoder, L., Singh, S.: Automated visual yield estimation in vineyards. J. Field Robot. **31**(55), 837–860 (2014). https://doi.org/10.1002/rob.21541

Palacios, F., Diago, M.P., Melo-Pinto, P., Tardaguila, J.: Early yield prediction in different grapevine varieties using computer vision and machine learning. Precision Agriculture pp. 1–29 (2022)

Palacios, F., Melo-Pinto, P., Diago, M.P., Tardaguila, J.: Deep learning and computer vision for assessing the number of actual berries in commercial vineyards. Biosyst. Eng. **218**, 175–188 (2022). https://doi.org/10.1016/j.biosystemseng.2022.04.015

Peng, Y., Wang, A., Liu, J., Faheem, M.: A comparative study of semantic segmentation models for identification of grape with different varieties. Agriculture **11**(10) (2021). https://www.mdpi.com/2077-0472/11/10/997

Peng, Y., Zhao, S., Liu, J.: Segmentation of overlapping grape clusters based on the depth region growing method. Electronics **10**(22), 2813 (2021)

Pérez-Expósito, J.P., Fernández-Caramés, T.M., Fraga-Lamas, P., Castedo, L.: Vinesens: An eco- smart decision-support viticulture system. Sensors **17**(33), 26 (2017). https://doi.org/10.3390/s17030465

Rahman, A., Hellicar, A.: Identification of mature grape bunches using image processing and computational intelligence methods. In: 2014 IEEE Symposium on Computational Intelligence for Multimedia, Signal and Vision Processing (CIMSIVP), pp. 1–6 (2014). https://doi.org/10.1109/CIMSIVP.2014.7013272

Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection (2016)

Redmon, J., Farhadi, A.: Yolov3: An incremental improvement (2018)
Reis, M.J.C.S., Morais, R., Peres, E., Pereira, C., Contente, O., Soares, S., Valente, A., Baptista, J., Ferreira, P.J.S.G., Bulas Cruz, J.: Automatic detection of bunches of grapes in natural environment from color images. J. Appl. Logic **10**(44), 285–290 (2012). https://doi.org/10.1016/j.jal.2012.07.004

Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks (2016)

Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Lecture Notes in Computer Science, pp. 234–241. Springer, Berlin (2015). https://doi.org/10.1007/978-3-319-24574-4_28

Roscher, R., Herzog, K., Kunkel, A., Kicherer, A., Töpfer, R., Förstner, W.: Automated image analysis framework for high-throughput determination of grapevine berry sizes using conditional random fields. Comput. Electron. Agric. **100**, 148–158 (2014). https://doi.org/10.1016/j.compag.2013.11.008

Rose, J.C., Kicherer, A., Wieland, M., Klingbeil, L., Töpfer, R., Kuhlmann, H.: Towards automated large-scale 3d phenotyping of vineyards under field conditions. Sensors **16**(1212), 2136 (2016). https://doi.org/10.3390/s16122136

Rossignon, L.: Vers une méthode optimale d'estimation du rendement de la vigne basée sur l'intelligence artificielle. Master's thesis, AgroParisTech (2020)

Rudolph, R., Herzog, K., Töpfer, R., Steinhage, V.: Efficient identification, localization and quantification of grapevine inflorescences in unprepared field images using fully convolutional networks. J. Grapevine Res. **58**(3) (2019). https://doi.org/10.5073/vitis.2019.58.95-104

Santos, T., Bassoi, L., Oldoni¹, H., Martins, R.: Automatic grape bunch detection in vineyards based on affordable 3D phenotyping using a consumer webcam. XI Congresso Brasileiro de Agroinformática (SBIAgro 2017) (2017)

Santos, T.T., de Souza, L.L., dos Santos, A.A., Avila, S.: Grape detection, segmentation, and tracking using deep neural networks and three-dimensional association. Comput. Electron. Agric. **170**, 105247 (2020). https://doi.org/10.1016/j.compag.2020.105247

Seng, K.P., Ang, L., Schmidtke, L.M., Rogiers, S.Y.: Computer vision and machine learning for viticulture technology. IEEE Access **6**, 67494–67510 (2018). https://doi.org/10.1109/ACCESS.2018.2875862

Shen, L., Su, J., He, R., Song, L., Huang, R., Fang, Y., Song, Y., Su, B.: Real-time tracking and counting of grape clusters in the field based on channel pruning with yolov5s. Comput. Electron. Agric. **206**, 107662 (2023). https://doi.org/10.1016/j.compag.2023.107662

Silver, D.L., Monga, T.: In vino veritas: Estimating vineyard grape yield from images using deep learning. In: Meurs, M.J., Rudzicz, F. (eds.) Advances in Artificial Intelligence, Lecture Notes in Computer Science, pp. 212–224. Springer, Berlin (2019). https://doi.org/10.1007/978-3-030-18305-9_17

Škrabánek, P.: Deepgrapes: Precise detection of grapes in low-resolution images. IFAC-PapersOnLine **51**(66), 185–189 (2018). https://doi.org/10.1016/j.ifacol.2018.07.151

Song, Y., Glasbey, C., Horgan, G., Polder, G., Dieleman, J., van der Heijden, G.: Automatic fruit recognition and counting from multiple images. Biosyst. Eng. **118**, 203–215 (2014). https://doi.org/10.1016/j.biosystemseng.2013.12.008

Sozzi, M., Cantalamessa, s., Cogato, A., Kayad, A., Marinello, F.: Grape yield spatial variability assessment using yolov4 object detection algorithm. pp. 193–198 (2021). https://doi.org/10.3920/978-90-8686-916-9_22

Steffenel, L.A., Langlet, A., Hollard, L., Mohimont, L., Gaveau, N., Copola, M., Pierlot, C., Rondeau, M.: Ai-driven strategies to implement a grapevine downy mildew warning system. In: Industrial Artificial Intelligence Technologies and Applications, pp. 177–187. Rivers Publishers (2022)

Torres-Sánchez, J., Mesas-Carrascosa, F.J., Santesteban, L.G., Jiménez-Brenes, F.M., Oneka, O., Villa-Llop, A., Loidi, M., López-Granados, F.: Grape cluster detection using UAV photogrammetric point clouds as a low-cost tool for yield forecasting in vineyards. Sensors **21**(9) (2021). https://doi.org/10.3390/s21093083
Vermesan, O., Martinsen, J.E., Kristoffersen, A., Bahr, R., Bellmann, R.O., Hjertaker, T., Breiland, J., Andersen, K., Sand, H.E., Rahmanpour, P., Lindberg, D.: Optimisation of soybean manufacturing process using real-time artificial intelligence of things technology. In: Vermesan, O., John, R., Luca, C.D., Coppola, M. (eds.) Artificial Intelligence for Digitising Industry Applications, pp. 301–325. Rivers Publisher (2021)

Victorino, G., Braga, R.P., Santos-Victor, J., Lopes, C.M.: Comparing a new non-invasive vineyard yield estimation approach based on image analysis with manual sample-based methods. Agronomy **12**(6) (2022). https://doi.org/10.3390/agronomy12061464

Victorino, G., Maia, G., Queiroz, J., Braga, R., Marques, J., Lopes, C.: Grapevine yield prediction using image analysis – improving the estimation of non-visible bunches. European Federation for Information Technology in Agriculture, Food and the Environment (EFITA), p. 6 (2019)

Victorino, G.F., Braga, R., Santos-Victor, J., Lopes, C.M.: Yield components detection and image-based indicators for non-invasive grapevine yield prediction at different phenological phases. OENO One **54**(4), 833–848 (2020). https://doi.org/10.20870/oeno-one.2020.54.4.3616

Xi, X., Zha, Q., Jiang, A., Tian, Y.: Stimulatory effect of bunch thinning on sugar accumulation and anthocyanin biosynthesis in Shenhua grape berry (Vitis vinifera × v. labrusca). Aust. J. Grape Wine Res. **24**(2), 158–165 (2018). https://doi.org/10.1111/ajgw.12323

Yin, W., Wen, H., Ning, Z., Ye, J., Dong, Z., Luo, L.: Fruit detection and pose estimation for grape cluster-harvesting robot using binocular imagery based on deep neural networks. Front. Robot. AI **8**, 163 (2021). https://doi.org/10.3389/frobt.2021.626989

Zabawa, L., Kicherer, A., Klingbeil, L., Töpfer, R., Kuhlmann, H., Roscher, R.: Counting of grapevine berries in images via semantic segmentation using convolutional neural networks. ISPRS J. Photogramm. Remote Sensing **164**, 73–83 (2020). https://doi.org/10.1016/j.isprsjprs.2020.04.002

Zabawa, L., Kicherer, A., Klingbeil, L., Töpfer, R., Roscher, R., Kuhlmann, H.: Image-based analysis of yield parameters in viticulture. Biosyst. Eng. **218**, 94–109 (2022). https://doi.org/10.1016/j.biosystemseng.2022.04.009

Zhang, C., Ding, H., Shi, Q., Wang, Y.: Grape cluster real-time detection in complex natural scenes based on yolov5s deep learning network. Agriculture **12**(8) (2022). https://doi.org/10.3390/agriculture12081242

Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network (2016). https://doi.org/10.48550/ARXIV.1612.01105

Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks (2020)

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
<figure><img src="image_80.png" /></figure>

Chapter 10
# How Do You Ride an Elevator?
Passenger In-Cabin Behavior Analysis
on a Smart-Elevator Platform

**Tarmo Robal** and **Uljana Reinsalu**

**Abstract** Modern elevators came into wide use some 150 years ago. With the advancement of technology, the main task of elevators has remained the same—transport people and goods in between floors—yet elevators have become more sophisticated with a trend toward smart-elevators. While increasing passenger comfort and travel experience, these systems also allow to get an insight into elevator passengers' behavior during their travels. Have you ever wondered why some people always stand in the same place, or what is your favorite spot to stand in an elevator? This chapter focuses on exploring passengers' in-cabin behavior while they travel between floors, enabled by a smart-elevator platform. To interpret collected real passengers' data, a general passenger location analysis model and evaluation methods are developed and several scenarios designed to validate system readiness for behavioral studies. The latter three enablers are used to analyze real elevator passengers' data to understand how passengers behave in cabins. The results show that passengers tend to choose their favorite position inside the cabin, permitted by the situation, and smart-elevators as a platform can be efficiently used to study passengers' behavior to enhance cabin environment.

**Keywords** Smart-elevators · Socio-cyber-physical systems · Behavioral studies · Elevator environment · Elevator users · Privacy

**10.1 Introduction**

The history of passenger elevators goes back to the beginning of the 1800s when steam and hydraulic power were introduced for lifting. The first passenger elevator powered by a steam engine was installed in 1857 in New York in a department store. Despite the advancement in technology at these times, the elevator received a cold reception from passengers with refusal to accept it (Bernard 2014). Nowadays,

<footnote>T. Robal (✉) · U. Reinsalu
School of Information Technologies, Tallinn University of Technology, Tallinn, Estonia
e-mail: tarmo.robal@taltech.ee; uljana.reinsalu@taltech.ee</footnote>
passenger elevators are a norm for modern commercial and residential buildings, providing an alternative mobility option to stairs. In high-rise buildings, which are typically equipped with several elevators to accommodate peoples' moving needs, elevators are the easiest way for people to move from one floor to another. For handicapped persons, elevators are key enablers of mobility for ensuring accessibility to buildings. Today, new buildings are typically equipped with elevators, and many older buildings are retrofitted. In Estonia (since 2019) and many other European Union countries, new buildings with five or more floors must be fitted with an elevator, as required by the law.

The advancements in technology and computerization of systems have also shaped the development of elevators, which today can be referred to as cyber-physical-social systems (CPSS) (Cassandras 2016; Dressler 2018; Lee and Seshia 2016; Zhuge 2014), where the “social” part incorporates human aspects of otherwise technical cyber-physical systems. Elevator systems have become complex, including rope-free and side-ways moving cabin systems (e.g., ThyssenKrupp Multi elevator). The trend is toward smart-elevators equipped with various sensors allowing to monitor, sense, and interpret the operational context of the elevator and deliver better service, safety, lower operational and maintenance cost, and improved user experience (UX) and comfort for passengers. Still, the need for passenger elevators and the underlying mode of exploitation has stayed the same—passengers place a call, wait for the cabin to arrive, and take a ride to a desired floor. However, the extended capabilities of smart-elevator platforms also provide an opportunity to study the exploitation of elevators closer. With the emergence of smart cities, the social aspects of elevators are also becoming more important, and in the future, smart-elevators can be addressed as socio-cyber-physical systems (SCPS) (Calinescu et al. 2019), which are complex systems where human and technical aspects are massively interconnected. Adding smart features to elevators, new or existing, allows to improve their exploitation and make our everyday lives more convenient. For example, imagine entering the elevator with hands full, and you can just ask by voice to be taken to a desired floor. This on the other hand drives the need to better understand human involvement and behavior in such systems and, through modelling, data analysis, and exploration, enable engineers to come up with improvements and enhancements for such SCPS.

Considerable amount of research is available for CPSS, and still, smart-elevators as SCPS have received little attention with the main focus on decreasing waiting time and energy consumption (Bamunuarachchi and Ranasinghe 2015; Bharti et al. 2017; Chou et al. 2018; Fernandez and Cortes 2015; Fujimura et al. 2013; Wang et al. 2011), thereby reducing carbon footprint, optimal parking in group elevator control (Brand and Nikovski 2004), use of floor sensors and RFID technology for elevator scheduling (Kwon et al. 2014), and the use of mobiles phones to improve flow of people (Turunen et al. 2013). A thorough overview of elevator control systems is provided in Fernandez and Cortes (2015); Ge et al. (2018) and passenger behavioral patterns while using elevators discussed in Liang et al. (2013).

In this chapter, we focus on elevator passengers' behavior inside the elevator cabin during their mobility. We take advantage of the existing smart-elevator plat-
form and study passengers' preferences and behavior for standing and movement in the cabin and the dependencies on cabin occupancy. The knowledge on passenger behavior can be used for movement prediction (e.g., using machine learning), improved layout of information panels (including advertisements), touch buttons, and sensors of a smart-elevator or even for improving user experience and enabling personalized travel service through passenger recognition in future smart-elevators. To the best of our knowledge, human behavior inside an elevator cabin has not yet been studied in the context of and using the equipment of a smart-elevator.

For the studies, we use the smart-elevator system (SES) (Leier et al. 2021; Reinsalu et al. 2020; Robal et al. 2020) set up at Tallinn University of Technology (TalTech) campus in the Information and Communication Technologies (ICT) building. The main aim of the smart-elevator system project was to investigate the applicability of common hardware and software to bring smart features to everyday passenger elevators and help users navigate in the ICT building—thus making elevators smarter by adding features of floor pre-selection (Leier et al. 2021; Robal et al. 2020), floor prediction (Reinsalu et al. 2020; Robal et al. 2020), and voice commands (e.g., set destination floor or even ask for weather forecast) and information services on staff working in the building (Leier et al. 2021). The SES also facilitates passenger identification and anonymous profiles of which we advantage in our passenger behavior research, based on the use of real passengers' travel data. In this chapter, we investigate passengers' in-cabin behavior and their attitude toward smart-elevators to establish a general framework for investigating and understanding elevator usage in the context of SCPS through the following research questions:

RQ1: *How to model and analyze the location of elevator passengers inside the cabin during their travels?* We hypothesize that the two-dimensional elevator floor model can be divided into meaningful parts for passenger location-based behavior analyses. We establish a general model to analyze passengers' in-cabin locations based on captured coordinates. We further apply this model on our smart-elevator platform and use real passengers' data to explore their in-cabin behavior.

RQ2: *What are the most preferred standing locations of passengers in an elevator cabin while travelling alone or with fellow travellers?* We hypothesize that with single occupancy, passengers prefer to stand near the doors or in the middle of the cabin, whereas in a situation with multiple passengers, distance is kept.

RQ3: *What are the reported preferred standing locations travellers indicate?* We hypothesize that travellers declare a preference to stand in the elevator either in the middle of the cabin or closer to the doors. We are also eager to see if the reported preferences map with the data captured from actual passengers.

RQ4: *How likely is an elevator passenger to choose the same standing location for successive travels?* The hypothesis is that each traveller tends to have a preferred location(s) to stand in the elevator cabin.
RQ5: *What are the in-cabin movement path patterns passengers follow during their travels, if any?* We hypothesize that each passenger tends to follow a certain path in the elevator cabin environment while entering and exiting.

RQ6: *What is the perceived invasion of privacy passengers have toward different devices installed into a smart-elevator?* We hypothesize that cameras will be considered the most invasive devices regarding privacy, whereas other devices that are not likely to capture (facial) images are considered less invasive.

The results of passengers' in-cabin behavior studies indicate that passengers tend to have their preferred standing locations and they favor to stand in the middle of the elevator cabin or in the first half of a rectangular-shaped elevator cabin.

The main contributions of this chapter are (i) a section-based model for passenger location analysis in elevator cabin, transferable to other contexts, (ii) scenarios for system validation tests of passengers' in-cabin behavior analysis, (iii) methods for passengers' position data analysis, and (iv) studies on passenger in-cabin behavior and preferences.

The rest of the chapter is organized as follows: Sect. 10.2 is dedicated to related work, while Sect. 10.3 presents the Smart-Elevator System platform. In Sect. 10.4, we address RQ1 and establish the general location analysis model for passengers' behavior analysis in the cabin, which we will validate through ground-truth study and specially designed experiments. In Sect. 10.5, we take up on the established model and methods to answer the rest of the research questions using collected real passengers' data. Finally, Sect. 10.6 provides discussion, and Sect. 10.7 draws conclusions.

## 10.2 Related Work

The research on human behavior regarding elevators has mainly focused on passenger arrival at elevator lobbies (Sorsa et al. 2013, 2021), passenger flow influence on lift control systems (Lin et al. 2016), finding patterns in usage (Liang et al. 2013), or exploring evacuation models (Heyes and Spearpoint 2012; Ronchi and Nilsson 2013).

Liang et al. (2013) gathered real-world traces of human behavior data from 12 elevators in an 18-story office building, showing that elevator usage patterns vary depending on the layout of the building, i.e., whether the stairs as an alternative are available next to the elevator, or not, and the proximity of stairs to elevators, and the function of the building (e.g., in hospitals and hotels, most of the vertical movement is done using elevators). Their study also indicated high-rise buildings with multiple elevators benefit more than low-rise buildings from human behavioral patterns on elevator usage, since waiting time in smaller buildings is minimal as the elevator can reach all floors quickly.
Ronchi and Nilsson (2013) investigated the capabilities of evacuation models in high-rise buildings and showed that the use of elevators can reduce the evacuation time in a non-fire emergency, while for fire events, the elevator was less valuable due to the layout of the particular building used in the study. Heyes and Spearpoint (2012) explored evacuation behavior of building residents in case of a fire, to develop parameters that could be used for designing an evacuation system that uses elevators. Their study shows that the number of building occupants that are likely to use the elevator as an evacuation method was increasingly dependent on the floor level. The primary factor whether to choose stairs or the elevator is the prediction of how much time it takes to reach the ground level via each evacuation route.

Sorsa et al. (2013, 2021) rejected the assumption that passengers arrive at the elevator lobbies separately, showing that in multi-story office, hotel, and residential buildings, people tend to arrive in batches of variable size (typically two or more persons) depending on the time of day. Considering passengers’ batch arrival helped to improve elevator group performance by reducing car loading, round-trip time, and passenger waiting times by 30–40%.

Susi et al. (2004) explored the effect of human behavior in a simulation case of the elevator traffic flow by studying the effect of passenger behavior. The work describes a model of human decision-making in a transportation system, which can be used for more accurate elevator traffic flow simulations to achieve realistic results by including passenger behavior into simulations. They find that simulation results are affected by characteristics and behavior of passengers.

Chou et al. (2018) used cameras and deep learning to minimize the average waiting time for passengers and at the same time decrease energy consumption by rescheduling elevator movements. The cameras were placed in front of elevators outside the cabin, and region-based convolutional neural network (R-CNN) was used to detect the number of passengers queuing for an elevator and dispatch elevators according to the detected demand such that the elevator with the smallest energy consumption was serving the waiting passengers.

To the best of our knowledge, human movement behavior inside an elevator car has not yet been studied in the context of and using the equipment of a smart-elevator. Liang et al. (2013) studied human behavioral patterns in the context of exploiting modern elevators through indicators such as elevator load factor, the number of floors travelled, and the doors-opened events to describe general behavioral patterns of office-building inhabitants, using the data (logs) generated by the elevator system itself, we in contrary focus not on the external events caused by the passengers but the passengers’ behavior in traveling situation inside the elevator cabin, and for this, we use real data from real passengers collected using the smart-elevator platform.

Our previous work on the smart-elevator platform (Leier et al. 2021) has focused on profiling passengers for travel behavior characterization and destination floor prediction (Reinsalu et al. 2020; Robal et al. 2020) to enhance travel experience in elevators. We have also investigated elevator passengers’ behavior regarding cabin usage, i.e., preferred locations (Basov et al. 2022), typical movement paths followed in the cabin, and whether there are recurring patterns (Robal et al.
2022). In connection of the SES, we have also proposed a design for a low-power touchless remote elevator call button (Reinsalu and Robal 2023). This chapter is a continuation of our work on the smart-elevator platform and contributes to fill the gap in the existing literature for social research in the context of elevator usage by providing a general model and methods to study and evaluate passengers' in-cabin behavior during their travels and applying the method in real-life scenario using real-world passenger ride data. The model and methods could be applied to any passenger elevator or transferred to other domains (e.g., public transportation) for user behavior studies.

## 10.3 Smart-Elevator Platform

A smart-elevator system can be considered as a CPSS and SCPS advantaging of data mining and artificial intelligence (AI) (Russell and Norvig 2009), e.g., facial image recognition (Robal et al. 2018; Silva et al. 2018; Stark 2019; Zhao et al. 2003) and human speech recognition (Allen 2003; Goetsu and Sakai 2019; Ketkar and Mukherjee 2011; Ross et al. 2004). Our studies on passengers' in-cabin behavior are carried out using the SES (Leier et al. 2021) developed at the School of Information Technologies and installed at TalTech in ICT building, which resides at the campus and hosts offices for university staff, labs, computer classes, and offices for private and start-up companies. Thus, the passengers of the elevator are the employees having offices in the ICT building and students accessing classrooms and working places. The smart-elevator operates through all the eight floors of the building, with a load limit of 1000 kg referring to a maximum of 13 persons.

The ICT building is a typical office building with eight floors (0 to 7) with the main entrance at level 1. The building has two elevators from KONE (global company for elevators and escalators), one on each side of the building. Each elevator is a single car running in its allocated elevator shaft. The elevator on the north side of the building is equipped with additional common hardware and software to deliver the features of the smart-elevator (Leier et al. 2021)—a RGB camera (Basler acA2040-25gc) for facial recognition, four depth cameras (Intel Real Sense D435) to detect and track passengers' location within the cabin, a speakerphone (Senheiser SP20) enabling voice commands, and a mini-PC (Intel NUC Mini PC NUC5i7RYB) for processing sensor data. The RGB camera is installed in the back corner of the cabin on the side of the button panel at the height of 1.55 meters to capture the video from the elevator entrance and cabin to ensure facial detection. The elevator can be called to a floor by up/down travel buttons located at each floor next to the elevator entrance and operated by push-button controls inside the cabin. Figure 10.1 describes the smart-elevator system cabin context and setup.

For the passenger in-cabin behavior studies, we take advantage of the four depth cameras installed in the elevator ceiling (Fig. 10.1) to capture passengers' location and movement data in the cabin during their rides. In general, the number of needed
<figure><img src="image_81.png" /><figcaption>**Fig. 10.1** Smart-elevator system and the cabin context. With the black labels, the devices for smart-elevator platform used in this study are shown, while white labels indicate the locations of common controls</figcaption></figure>

depth cameras and their placement depends on the elevator cabin layout (length, width, and height) and the field of view (FoV) of the chosen depth cameras (Leier et al. 2021). The current setup of the SES uses four depth cameras, as a single camera would not be able to provide full coverage of the cabin—to detect passengers’ position, it is needed to capture relatively flat areas in the upper half of the cabin. The cabin height of our elevator is 2.2 meters, which means that a single Intel RealSense D435 depth camera placed in the middle of cabin ceiling would only be able to capture objects in the cabin at the height of 1.3 meters—making it impossible to capture taller passengers, especially those standing close to the walls. Therefore, to cover the whole cabin at maximum height up to 2.0 m, our smart-elevator uses four depth cameras. The SES positioning system merges the results from each camera and maps them to global coordinate system reflecting the elevator coordinates layout (Fig. 10.2).

While the location of passengers inside the cabin is tracked using the depth cameras, the boarding of new passengers is also registered by the RGB camera, placed such that all incoming passengers could be detected and assigned a tracking ID. Further, the detection results are used to identify known frequent passengers through facial recognition against existing profiles. The profiles are stored as anonymous profiles with only a vector of facial features and numeric IDs (no facial photo/video is stored in the system), making it impossible to match these profiles to identifiable persons. Passengers of the elevator are notified of the existence of the cameras within the elevator cabin by a sign outside the cabin, and in case they disagree, they are instructed to use the second elevator of the ICT building. The advantage of the facial recognition camera is that passengers are not required to make any additional efforts to be known by the system as it can be fully automized. An alternative to the use of facial recognition camera to identify and distinguish frequent passengers would be to use contactless cards (e.g., either as anonymous cards or staff cards) with radio-frequency identification (RFID) or near-
<figure><img src="image_82.png" /><figcaption>**Fig. 10.2** Elevator cabin context and the coordinates system of the SES positioning service (SES-PS)</figcaption></figure>

field communication (NFC) technology. In this study, we exploit passenger profiles as anonymous numerical IDs to distinguish frequent travellers re-taking elevator while addressing RQ4 and RQ5.

To continue, let us now define some terms. A *travel* is a ride an elevator passenger undertakes between departure and destination floors such that it starts with a passenger entering the elevator cabin and ends with exiting the cabin. Each travel is assigned a new travel identification number (travel ID) in the SES stored alongside with track data. A *track* is the location of the passenger inside the elevator cabin attributed with travel ID, position coordinates (x, y), and timestamp. *Single travel* is a travel with only one passenger in the cabin, while *crowded travel* describes situations where at least two or more passengers have occupied the cabin for a travel.

The passenger position data is gathered using the SES positioning service (SES-PS), which allows to collect position data for each individual passenger by tracking and retrieving sensor data from the four depth sensors located in the ceiling of the elevator cabin. Detection of travellers is done using an image processing algorithm (Leier et al. 2021). The algorithm delimits the heads of passengers in the elevator cabin regardless of their height. The system starts to locate heads at the height of 120 cm and, with every iteration, increases the detection height by 5 cm. Every height layer is run through until the ceiling height is reached. All the detected heads' movements will be monitored throughout the travel and track data stored periodically (every 200 ms). Whenever a passenger with an existing profile (the passenger has travelled previously) is identified, the track is also associated with the profile ID in the SES. The coordinates in the SES-PS are expressed in the metric system using centimeters as the unit (Fig. 10.2). The coordinates for the x-axis range from 0 to 220 cm and for the y-axis from 0 to 110 cm, reflecting the central position of the detected object position at 120 cm or higher above the floor. Figure 10.2 outlines the elevator cabin context with the equipment and the coordinates system used by the SES-PS.
# 10.4 Passenger Location Analysis Model and Methods

As learned in Sect. 10.3, the SES-PS collects data about passengers’ location and movement in the cabin, using a global coordinate system laid out on the floor area (Fig. 10.2). In this section, we first discuss how to establish a model for passenger location interpretation in the elevator cabin by projecting captured coordinates to meaningful passenger locations and, by this, address RQ1. We then proceed to validate the model and establish methods of data analysis through a ground-truth study.

## 10.4.1 Passenger Location Analysis Model

To analyze passenger location in elevator cabin, we propose to divide the elevator-floor-area two-dimensional model into ideally equally sized squared sections. We reason that from top-down view, a person in an elevator cabin will take up an elliptical space, which we transform into a circle, as we have no knowledge about the direction of the person inside the elevator. For the diameter of the circle, we use the approximate shoulder length of humans. A study by Randall et al. (1946) for Army Air Force reports that 95% of cadets have biacromial shoulder width of 42.9 cm. Considering that not every passenger is in the ideal shape of a “cadet” and that humans prefer to have a personal space around them, we extend this area to approximately $d_{section} = 50$ cm. We do not recommend to decrease this value; however, extending the personal space could be applied. As the person in the elevator is represented by a circular area, we therefore opt for the squared sections in the model, as it is the best way to fit a circle representing a person. Figure 10.3 describes the establishment of one section for our passenger location analysis model.

Next, having established a model for a single section, we scale the section-based location model to elevator floor area. For this, we find the number of possible sections $s = n_l \times n_w$ over the length $l$ and width $w$ of the elevator floor area, where

<figure><img src="image_83.png" /><figcaption>**Fig. 10.3** Development of the passenger location section model (a view of one section)—from biacromial shoulder-width to section</figcaption></figure>
<table><caption>**Table 10.1** Examples of elevator cabin dimension conversion into sections using the passenger location section model</caption><thead><tr><th rowspan="2">Elevator #</th><th colspan="2">Cabin dimensions</th><th colspan="2"># Sections</th><th rowspan="2">Total # sections</th><th colspan="2">Section dimensions</th></tr><tr><th>l</th><th>w</th><th>n<sub>l</sub></th><th>n<sub>w</sub></th><th>l<sub>s</sub></th><th>w<sub>s</sub></th></tr></thead><tbody><tr><td>1</td><td>110</td><td>100</td><td>2</td><td>2</td><td>5</td><td>55</td><td>50</td></tr><tr><td>2</td><td>135</td><td>130</td><td>2</td><td>2</td><td>5</td><td>68</td><td>65</td></tr><tr><td>3</td><td>214</td><td>168</td><td>4</td><td>3</td><td>13</td><td>54</td><td>56</td></tr><tr><td>4*</td><td>220</td><td>110</td><td>4</td><td>2</td><td>9</td><td>55</td><td>55</td></tr><tr><td>5</td><td>267</td><td>173</td><td>5</td><td>3</td><td>16</td><td>53</td><td>58</td></tr></tbody></table>

$n_l$ and $n_w$ are the number of possible sections over the dimensions $l \in Z$ and $w \in Z$ correspondingly. After studying several elevator cabin size configurations ranging from 1.0 to 2.7 meters in dimensions (Table 10.1 exemplifies some of them), we find that to identify the number of sections, the dimensions should be rounded up toward the nearest decimal and then the lowest fit applied; thus, we use the floor function to settle the number of sections over a dimension. Equation 10.1 provides the general formula for finding the number of sections $N$ over a cabin dimension $Z$.

$$N = \left\lfloor \frac{\lceil Z \rceil}{d_{section}} \right\rfloor \qquad (10.1)$$

In our passenger location analysis model, we have one more additional section representing the center of the elevator. The center of this section matches the central point of the elevator floor area. We use this section to identify passengers who prefer to stand in the middle of the elevator (e.g., while travelling alone). Therefore, the total number of sections is $s_{total} = (n_l \times n_w) + 1$.

In reality, the actual cabin dimensions (height, width, depth) vary by design, producer, possibilities to install an elevator in a building, etc. Thus, depending on the actual elevator cabin dimensions, it may not be always possible to retain the ideally modelled squared sections in the two-dimensional-floor model. In case squared sections cannot be achieved, section dimensions should be adjusted (stretched) in the dimension (length/width) where otherwise full sections cannot be established. The section dimension is then determined by cabin dimension divided by the number of sections $N$ for that cabin dimension (Eq. 10.1). An alternative would be to define partial sections. In Table 10.1, one can observe section dimension variability considering different common cabin sizes.

The designed location analysis model is not connected to the elevator capacity characteristics (maximum and normal loading), which are typically calculated based on weights or a combination of weights and elevator speed, together with a recommendation for allocated passenger area. The latter for normal loading is ca. 0.21 m² and for maximum ca. 0.14 m². In our model, the area allocated per passenger is 0.25 m², allowing sufficient personal space (around 5–10 cm) or space for belongings (e.g., rucksack). Fitting the elevator cabin with the maximum number

<footnote>* The smart-elevator used in this study</footnote>
<figure><img src="image_84.png" /><figcaption>**Fig. 10.4** The model for passenger location analysis with nine sections (1–9) and the context of the elevator cabin environment. Letters A–D mark the stand-location points used in the ground-truth study. Dash-dot-dot colored lines mark the movement path for the continuous location detection study: Route#1, blue; Route#2, green; Route#3, orange; and Route#4, magenta</figcaption></figure>

of passengers comes with the cost of the loss of personal space and being pressed together with fellow passengers.

We now turn to our smart-elevator platform. Considering our smart-elevator cabin dimensions ($l \times w$) are 220 × 110 cm, we detect that for the passenger location analysis, the number of sections is nine. The application of the model divides the elevator floor area into eight equal square-sized sections with an additional section of the same size overlaying the sections in the middle of the cabin. Figure 10.4 describes the context of the elevator cabin environment (e.g., the position of doors sliding open from left to right while standing in the cabin and facing the doors, location of floor selection buttons, etc.) and the division of the floor area to nine sections, each of which identifies a potential location of an elevator passenger. The center point of a passenger in a section is at ca. 22 cm according to the average passenger’s shoulder width, which correlates to 27.5 cm in the section of the location model for our smart-elevator. As usually passengers do not stand against the wall, there is a high probability that the passenger center point aligns well with the section center point in the model. The elevator producer KONE has limited the maximum number of people for the elevator type we use to 13 passengers.

Next, we validate the established zone-based location model and SES-PS, and establish methods for data analysis to be able to analyze passenger in-cabin location behavior based on real travel data (Sect. 10.5) used to answer RQ2, RQ4, and RQ5.
### 10.4.2 Model Validation and Methods Establishment

To validate the location analysis model and to be able to properly interpret the real elevator passenger data collected by SES, we carried out a ground-truth study consisting of a series of validation experiments, based on the developed passenger location analysis model. As a result, we establish methods for data analysis according to the location analysis model and answer RQ1.

On April 6, 2021, late evening (after 20:00), a ground truth study was carried out by a single test passenger to evaluate and validate SES setup and precision in detecting a passenger's location in the elevator cabin based on the location analysis model (Sect. 10.4.1). We chose a late hour to have minimum disturbance for other potential travellers, as well as for the continuity of the experiments. Through a series of experiments, the performance and accuracy of the SES-PS were validated, whereas the location of the test passenger inside the elevator cabin was known in advance. Multiple locations inside the cabin were selected (Fig. 10.4) with the goal to determine the accuracy of passenger position detection in any of the given section according to the location analysis model and to identify any variability. The study assumed that the test subject was always standing in the middle of the designated section of the location analysis model, and for this, to aid the test passenger, a grid was marked down on the elevator floor with white paint tape (Fig. 10.5). The grid,

<figure><img src="image_85.png" /><figcaption>**Fig. 10.5** Performing the ground-truth study to validate the analysis model and SES-PS: sections of the location analysis model marked down on the elevator floor with paint tape (left), and the test passenger carrying out the experiments standing in Section 9 in the middle of the elevator cabin, facing the doors (right)</figcaption></figure>
placed to help the test passenger stick to pre-planned elevator usage scenarios, was removed immediately after the experiments, which lasted for 3 hours.

Two different experiment series, (i) *static stand positions* and (ii) *continuous position detection for pre-determined movement path*, were carried out to validate the SES positioning service against the location analysis model. Although the total number of planned test travels was 60 (10 for each static stand position and 5 for every continuous position detection), the actual number of travels captured was 61 due to a counting error made by the test passenger. Four additional travel records appeared in the captured data as other passengers entered the cabin in the middle of one of the experiment trips. These travels were removed from the analysis data, leaving thereby a dataset of 59 travels and location tracks data of 37,044 coordinate pairs. The collected data was analyzed using general-purpose programming language Python (ver. 3.7) with *Psycogp<fnref n="1" />* PostgreSQL database adapter and *XlsxWriter*,<fnref n="2" /> *DateTime*,<fnref n="3" /> *numpy*,<fnref n="4" /> and *Matplotlib*<fnref n="5" /> packages.

### 10.4.2.1 Static Stand Positions

First, a series of static stand-still experiments was carried out with four different specifically selected stand-positions *A-D* (Fig. 10.4) in the elevator cabin. The position *A* represents a zone right in front of the elevator doors on the opening side (doors open in the direction from Section 5 to 1). Position *B* in Section 4 marks the back corner of the elevator cabin in front of the mirror while *C* (Section 9) the middle of the elevator and *D* on the border of Sections 6 and 7 an ambiguous multi-section area in front of the elevator floor buttons, which can be reached for pressing from Sections 6, 7 or 9 in an approximate reach radius of 55 cm (Fig. 10.4).

Each position *A-D* was tested with a series of ten travels (except *C* for which 1 series appeared invalid due to accidental additional passengers and was removed) between two floor levels (e.g., floor 3–5), with a travel lasting about 20 seconds. The SES-PS captures passenger position tracks with depth cameras at a rate of 200 ms—roughly 100 position data points for each travel in the experiment. The collected data indicated 104 tracks in average per experiment travel (min 88 and max 112). To ensure data completeness, for each travel, a new elevator call was made. It was required that the test passenger follows the same route to enter the elevator and stands in the center of the agreed experiment position (*A*, *B*, *C*, or *D*), turning around to face the doors once having reached the position.

Standing in each position *A-D* was analyzed separately. To evaluate the SES-PS accuracy of measuring the static stand location, the data coordinates describing the

<footnote n="1">¹ https://pypi.org/project/psycogp2/</footnote>

<footnote n="2">² https://pypi.org/project/XlsxWriter/</footnote>

<footnote n="3">³ https://pypi.org/project/DateTime/</footnote>

<footnote n="4">⁴ https://pypi.org/project/numpy/</footnote>

<footnote n="5">⁵ https://pypi.org/project/matplotlib/</footnote>
<figure><img src="image_86.png" /><figcaption>**Fig. 10.6** Visualization of the captured passenger location data before data filtering and after (callout with a dashed line) movement information has been filtered out, for example, location point C in Section 9 of the location analysis model</figcaption></figure>

<table><caption>**Table 10.2** Experiment results for the static stand position study</caption><thead><tr><th rowspan="2">Position</th><th colspan="2"># track coordinates</th><th colspan="2">M<sub>x,y</sub><sup>a</sup></th><th colspan="2">E<sub>x,y</sub><sup>b</sup></th><th colspan="2">Average deviation</th></tr><tr><th>Experiments</th><th>Filtered</th><th>x</th><th>y</th><th>x</th><th>y</th><th>x</th><th>y</th></tr></thead><tbody><tr><td>A</td><td>983</td><td>893</td><td>27.5</td><td>82.5</td><td>32</td><td>76.5</td><td>4.9 ± 4.4</td><td>-7.3 ± 4.2</td></tr><tr><td>B</td><td>1156</td><td>759</td><td>192.5</td><td>27.5</td><td>170.4</td><td>28.5</td><td>-21.3 ± 4.8</td><td>1.4 ± 3.3</td></tr><tr><td>C</td><td>962</td><td>778</td><td>110</td><td>55</td><td>100.9</td><td>52.6</td><td>-9.4 ± 2.9</td><td>-2.8 ± 1.5</td></tr><tr><td>D</td><td>1042</td><td>873</td><td>110</td><td>82.5</td><td>103.9</td><td>80.9</td><td>-7.1 ± 4.5</td><td>-2.8 ± 4.9</td></tr><tr><td>Avg</td><td>1036</td><td>826</td><td>n/a</td><td>n/a</td><td>n/a</td><td>n/a</td><td>-8.2 ± 4.2</td><td>-2.9 ± 3.5</td></tr><tr><td>Sum</td><td>4143</td><td>3303</td><td>n/a</td><td>n/a</td><td>n/a</td><td>n/a</td><td>n/a</td><td>n/a</td></tr></tbody></table>

movement into the required position A–D were filtered out as follows: first, any coordinate outside of the planned stand position section was deemed as movement into the section and eliminated; additionally, the movement points from the section edge to the center of the planned stand position section were removed until five consecutive points (ca. 1 second) were captured at the planned stand position section to assure a safe margin for reaching a stand position. Similar filtering action was carried out for data describing exiting a section and the elevator. Figure 10.6 visualizes the experiment data before and after filtering for position C located in Section 9.

The results of the static stand positions experiment series are outlined in Table 10.2. On average, 210 location points for each series were deemed to describe moving to location and thus removed from the analysis—more for positions further away from the doors (e.g., position B). We noted that for position B, the deviation for determining the x-coordinate differs from all other findings, which could be the misalignment of the positioning service or the effect of the back-wall mirror reflections. In further analysis, this blind-spot area has been accounted for.

<footnote><sup>a</sup> Central section coordinates in the model, [cm]</footnote>

<footnote><sup>b</sup> Median central coordinates of section in experiments, [cm]</footnote>
The results of static stand position experiments indicate that the coordinates captured by the SES-PS system are slightly off-centered on the x-axis, while the y-coordinates are rather accurate. This misalignment may be due to the test passenger's posture and height (1.90 m) but also an alignment shift in the positioning service or noise from artificial lightning and reflections. From the validation experiments, we conclude that the SES-PS is able to locate the passenger in a section center with a deviation of 8 cm on x-axis and 3 cm on the y-axis, which, considering the technical setup and the passenger location analysis model, is satisfactory.

With this, we conclude that it is feasible to model passengers' location through location sections as meaningful parts of the two-dimensional floor model (RQ1), confirming our hypothesis. The experiment forms a benchmark for interpreting the collected real passenger travel track data used to answer RQ2 and RQ4.

### 10.4.2.2 Continuous Position Detection for a Movement Path

To validate the detection of passenger movement path inside the cabin with the SES-PS, we designed four different routes starting and ending at the cabin doors in between the positions A–D as given on Fig. 10.4:

* Route#1: $A \rightarrow C \rightarrow B \rightarrow A$, a scenario where a passenger enters the elevator, moves to the middle, reaches the floor buttons, and moves to the back corner having a good view on the indicator panel and the cabin for the duration of travel.

* Route#2: $A \rightarrow D \rightarrow B \rightarrow A$, a scenario similar to Route#1, except the floor buttons are reached right in front of these at the position D.

* Route#3: $A \rightarrow D \rightarrow C \rightarrow A$, where after pressing the floor buttons the passenger proceeds to stand in the middle of the elevator at the position C.

* Route#4: $A \rightarrow D \rightarrow A$, where the passenger reaches the button panel and then immediately steps back to the closest position to doors on the opening side.

The test passenger followed each route for five times during the experiments. Each travel was through four floor levels (e.g., floor 1 to 5) with an average travel duration of 25 seconds, during which, in average, 112 track points for each travel were captured by SES-PS. At each point (A–D), the test passenger made a short stop, yet keeping the movement as natural as possible. While exiting the cabin, the shortest path through the opening side of the doors (position A) was taken.

For the analysis, the captured location data was filtered and compressed as follows: section and location coordinates were added to the movement path whenever five consecutive points were captured by SES in the same section (at rate of 200 ms), provided that the section was an adjacent section (in the location analysis model) to the previous one in the movement path list. The path was constructed as a sequence of sections (detected from the track coordinates) passed by the passenger, where each section is sequentially counted only once, forming, for example, a path 5-6-9-6-5. Figure 10.7 illustrates the detected paths for Route#2 in the experiments.

From the data analysis, we notice that there is a certain blind-spot area of approximately 15 cm from the wall into which no coordinates fall. First of all,
<figure><img src="image_87.png" /><figcaption>**Fig. 10.7** Movement path construction with applied data filtering for Route#2 in the experiments. The dashed black line indicates the planned route for the experiment</figcaption></figure>

the SES-PS estimates the center point of the passenger, and second, even if a passenger stands against the wall, the detected center point of the person would still be ca. 15 cm away from it. The validation experiments with continuous location detection confirmed that the SES positioning service can sufficiently enough track the passenger movement inside the cabin environment throughout the travel. The analysis of collected experiment data allowed to establish a path construction method with data point reduction for real passenger data analysis using the location analysis model. The results will be applied to answer RQ5.

## 10.5 Passenger In-Cabin Behavior Analysis

In this section, we apply the methods established and knowledge gained from the ground-truth study to answer the research questions about passenger movement behavior in elevator cabin environment. For this, we use the track data of real passengers collected through 61 days (2 months, April-May 2021) and consisting of 11,731 travels, out of which 67.9% of travels were made with a single passenger, and in 32.1% of cases, there were multiple passengers in the cabin. The period of data collection matches with the enforced COVID-19 restrictions (2+2 rule and facial mask mandate), which affects the available number of travels as well as the ability of the SES to differentiate between known travellers through facial recognition. In addition, the ICT building was partially closed for students due to the pandemics, and we also noticed that people preferred stairs over the use of the elevators. Travels performed by the test passenger for the ground-truth study have been excluded. Table 10.3 characterizes the real passenger data we used.
<table><caption>**Table 10.3** Smart-elevator passenger data used for the behavior analysis</caption><thead><tr><th></th><th># Travels</th><th># Tracks</th><th>ntp<sub>avg</sub><sup>a</sup></th><th>d<sub>avg</sub><sup>b</sup></th></tr></thead><tbody><tr><td>Total (count)</td><td>11,731</td><td>1,414,740</td><td>120</td><td>24</td></tr><tr><td>Travelling alone in cabin (count)</td><td>7790</td><td>1,034,162</td><td>130</td><td>26</td></tr><tr><td>Travelling in a crowded cabin (count)</td><td>3761</td><td>380,578</td><td>101</td><td>20</td></tr></tbody></table>

<figure><img src="image_88.png" /><figcaption>**Fig. 10.8** Preferred standing positions by sections for single travel and with multiple passengers (PAX) in the cabin. The column *Multi* indicates more than one passenger in the cabin during a travel</figcaption></figure>

### 10.5.1 RQ2: Passengers' Preferred Standing Locations

We start passenger in-cabin behavior analysis by considering the most preferred standing locations while travelling alone or with fellow passengers (crowded) in the elevator cabin. To answer *RQ2*, we look at all track data collected on real passengers during the study period, regardless whether the passengers had an existing user profile in the SES or not. The data is analyzed according to the location analysis model (Sect. 10.4.1) and the method described in Sect. 10.4.2. For each position track coordinate x-y pair, a location section 1–9 representing passenger’s location is found.

The analysis (Fig. 10.8) reveals that while travelling alone in the cabin, the most favorable position to stand is in the middle of the elevator (Section 9). However, if multiple travellers are in the elevator, other sections become more favorable (Fig. 10.9), and we can observe a dramatic drop of more than two times for Section 9 in occupancy compared to single travels. The less favorable locations are Sections 6–8, while Section 8 in the back corner in front of the facial recognition camera is the least occupied among all travels. We also observe that passengers

<footnote>a Avg. tracks per travel per passenger</footnote>

<footnote>b Avg. travel duration per passenger [s]</footnote>
<figure><img src="image_89.png" /><figcaption>**Fig. 10.9** Density maps for elevator passengers' in-cabin positions trough travels: (a) single passenger in cabin, (b–e) 2–5 passengers 2–5 in cabin, (f) overall density with multiple passengers in cabin. Blue, least density; red, highest density. Single-density maps created using Matplotlib (Hunter 2007)</figcaption></figure>

prefer to stand on the opposite side to the button panel and floor level indicator (compare Sections 2 and 3 to Sections 6 and 7), which we believe is due to have a better view over the travel status from the floor indicator but also not to block other passengers from (de-)boarding. It would be interesting to see if this distribution changes when an additional panel is installed on the opposite wall or there are no floor buttons present in the cabin at all.

The density maps for preferred in-cabin standing positions (Fig. 10.9) reveal that a central location is preferred while travelling alone. Yet, when other passengers are also in the cabin, distance is kept, and standing locations begin to form around the opposite wall to the button panel and floor indicator (Sections 1–3) on the closing
side of the doors but also just opposite to the opening side of the door (Section 5). Overall, we see that with single occupancy, passengers prefer to stand in the middle of the cabin or closer to the doors, whereas in a situation with multiple passengers in the cabin, passengers tend to keep distance, confirming our initial hypothesis.

Although these results are directly bound to a particular elevator type we used, we still observe the following general findings: (i) the established location analysis model performs well in distinguishing passenger locations, (ii) typically passengers try not to block (de-)boarding fellow passengers and distance is kept while at the same time having a good view of the travel status indicators, and (iii) for single travels, a central location is preferred. For another type of an elevator, we expect these findings to hold; however, in case of larger elevators with multiple button panels (i.e., panels on both sides of the elevator doors), this is likely to change, and single travellers may also opt for a location closer to a wall. It would be interesting to carry out this experiment on different elevator layouts, equipment permitted. Unfortunately, still, today, it would require significant installation work of equipment.

### 10.5.2 RQ3: Passengers' Reported Standing Locations

We compare the results of the detected standing preferences of passengers (Sect. 10.5.1) to their explicit preferences reported back through a survey on elevator usage habits. The survey, consisting of 19 questions, was sent to potential elevator users of the ICT building through internal mailing lists. We selected to carry out the survey on employees working in the building as they are potentially using the elevators on a daily basis, are familiar with the elevators, and are likely represented in the collected passenger travel data we analyzed. Out of the 180 potential users, 63 (35%) responded to our survey. Eighty-six percent of the respondents reported that they had used either both of the elevators or the smart-elevator, while 11% had used only the regular elevator, and two respondents (3%) had never used the elevators in the ICT building.

In the survey, we presented the participants with the smart-elevator of the ICT building and the floor plan with identifiable spots for standing (Fig. 10.10). We asked the participants to indicate their explicit preference toward a single spot in the cabin during the travel and evaluate all the spots regarding the likelihood of occupying a spot for two scenarios: travelling alone and with fellow passengers in the cabin. The response rate for these questions was 98%. For the spots (Fig. 10.10), we selected all the nine sections of the location analysis model (Sect. 10.4.1), plus a few extra of interest (F in front of the button panel and J in the middle at the back wall in front of the mirror).

The preferred standing positions explicitly reported by users are presented on Fig. 10.11. We observe the same trends as for RQ2 for Sections 1, 3, 5, 6, 8, and 9, whereas for Sections 2, 4, and 7, an opposite trend is observed compared to the findings from the passengers' location behavior (Fig. 10.8), in line with our
<figure><img src="image_90.png" /><figcaption>**Fig. 10.10** Elevator floor plan with preferred standing spots A–K presented to survey respondents to collect their standing location preferences regarding the elevator usage</figcaption></figure>

<figure><img src="image_91.png" /><figcaption>**Fig. 10.11** Reported explicit preferences for standing locations for single and multi-user travels. The spots presented in the user survey have been transferred to sections of the location analysis model</figcaption></figure>
hypothesis. Interestingly, the preference to stand in the middle of the elevator is superseded by preferring Section 3. For the additional spot *F* (Fig. 10.10) in the survey, we do not find any reported preference (0%) while travelling alone, and only one respondent (2%) reported it back as a preference while travelling with fellow passengers. The spot *J* is reported to be preferred in 8–10% of cases correspondingly.

### 10.5.3 RQ4: Preferred Standing Position for Successive Travels

Next, we consider the likelihood of a passenger to choose the same standing location in the elevator cabin for re-occurring travels, i.e., do passengers have their favorite standing positions they choose in the cabin each time they ride? To answer RQ4, we analyze only the track data of passengers known (profiled) by SES. Profiles are created, and passengers identified by SES automatically using facial recognition (Leier et al. 2021; Robal et al. 2020) with an identification success rate of 98.2%. Unfortunately, during the study, COVID-19 restrictions (including the mask mandate) were effective, which significantly reduced the travel data available for such known passengers due to system inability to recognize all known passengers. Thereby, we obtain only 793 travels with 305 distinct profiles, which we again separate into two groups: travelling alone (single, 30.0%) or in a crowded cabin (two passengers, 42.1%; three, 21.7%; four and more, 5.9%). We reject any profile that has less than three travels associated. Further, we organize travels of each profile in chronological order, split the set into two, and use the first 2/3 to determine the preferred standing position and the last 1/3 to verify the hypothesis of choosing mostly the same standing location. This leaves us with 361 travels (46%), 83 as single and 278 as crowded cabin travels, with 17 and 60 profiles correspondingly. To determine the preferred location, we apply the same approach as for RQ2 based on the location analysis model, determining the preferred standing position as one of the nine sections.

For each passenger, we find over her travels a list of standing positions as sections of the location analysis model (Section 1–9) in a decreasing order of occupancy and use the top-two items (*Top 1* and *Top 2*) of this list as the most likely standing positions. We then compare these to the remaining 1/3 of data (Table 10.4). Based on the small sample of data we have, we see that while travelling alone in the cabin, passengers tend to choose (as hypothesized) the same standing position as they have

<table><caption>**Table 10.4** Probability of choosing the same standing position for re-occurring travels</caption><thead><tr><th>Cabin situation group</th><th># Profiles</th><th># Travels</th><th>Top 1</th><th>Top 2</th><th>Top 1 or Top 2</th></tr></thead><tbody><tr><td>Single travel</td><td>17</td><td>83</td><td>60%</td><td>31%</td><td>91%</td></tr><tr><td>Crowded travel</td><td>60</td><td>278</td><td>26%</td><td>9%</td><td>35%</td></tr></tbody></table>
<table><caption>**Table 10.5** Characterization of constructed movement paths and re-occurrence match</caption><thead><tr><th></th><th># Travels/profile</th><th>L<sub>path</sub><sup>a</sup></th><th>L<sub>pax-path</sub><sup>b</sup></th><th>M<sup>c</sup> [%]</th><th>M<sub>partial</sub><sup>d</sup> [%]</th></tr></thead><tbody><tr><td>Avg</td><td>5 ± 2</td><td>6</td><td>6.4 ± 1.0</td><td>13%</td><td>62.3%</td></tr><tr><td>Min</td><td>3</td><td>3</td><td>5.2 ± 1.3</td><td>75%</td><td>100.0%</td></tr><tr><td>Max</td><td>12</td><td>11</td><td>7.8 ± 1.4</td><td>0%</td><td>0.0%</td></tr></tbody></table>

a Path length over all travels

b Avg. path length per passenger

c Exact path match rate

d Partial path match rate on the first three positions

chosen previously, while in the situation of a crowded cabin, no favorable position is chosen, as it may already be taken, and a random open standing location is occupied.

### 10.5.4 RQ5: Recurring Movement Flow Patterns

We also study passengers' in-cabin behavior regarding their movement in the cabin environment and explore whether they tend to choose the same path through the cabin during their travels. To answer RQ5, we analyze the successive travels of profiled passengers for the single travel situation only and use the same dataset as for RQ4. From RQ4, we already know that passengers tend to choose the same standing location in 60% of cases when travelling alone. We decide not to analyze movement routes while travelling in a crowded elevator as the path would greatly depend on the occupancy of the elevator cabin and locations of the fellow passengers.

For each travel associated with a profile in the SES, we construct a movement path based on the sections the traveller has been found to be present in (passing or standing) using the method described in Sect. 10.4.2.2, producing a path (e.g., 5-6-9-1-5) for each travel. Table 10.5 characterizes the movement paths for the 83 single travels available for the 17 profiled passengers.

Based on the small sample of data we were able to collect, we do not find that passengers would in an identifiable way follow the same movement path when entering, standing, and exiting the elevator for travelling in between floors. The average path consists of four to eight sections of movement with a maximum of 11 sections. The same path is followed only in 13% of travels observed. In this regard, we fail to confirm our hypothesis. However, for two profiled travellers, we notice the exact path match to be 75% and 67%. Analyzing the first three positions of a path, we interestingly find that the same passenger who had the exact match rate at 75% had a movement path match by first three positions at 100%, whereas the general rate for all single-travelling profiled passengers was at 62%. This is somewhat expected, as passengers usually enter the elevator cabin from Section 5, move to Section 6 to press a floor button, and then to Section 9 to stand in the middle of the elevator cabin. With a larger set of data available and for a longer duration, there might be some interesting findings to be uncovered.
<table><caption>**Table 10.6** Perceived invasion of privacy imposed by devices of a smart-elevator platform</caption><thead><tr><th>Device</th><th>Do not know</th><th>No</th><th>Somewhat</th><th>Definitely</th></tr></thead><tbody><tr><td>Facial recognition camera</td><td>17%</td><td>11%</td><td>29%</td><td>33%</td></tr><tr><td>Microphone</td><td>8%</td><td>12%</td><td>29%</td><td>33%</td></tr><tr><td>Infrared camera</td><td>17%</td><td>19%</td><td>22%</td><td>21%</td></tr><tr><td>Motion sensor</td><td>33%</td><td>31%</td><td>8%</td><td>3%</td></tr><tr><td>Weight measurement</td><td>25%</td><td>27%</td><td>12%</td><td>10%</td></tr></tbody></table>

### 10.5.5 RQ6: Perceived Invasion of Privacy in Smart-Elevator Cabin

In our survey on elevator usage habits (Sect. 10.5.2), we also asked participants to rate the devices installed in the smart-elevator cabin considering the perceived invasion of privacy on the scale of *No*, *Somewhat*, and *Definitely*. The question we presented in the survey was as follows: *Do you feel your privacy is threatened when using a smart-elevator equipped with the following devices?* We presented the following list of devices: (i) facial recognition camera; (ii) microphone; (iii) infrared camera, referring to the used depth cameras; (iv) motion sensor; and (v) weight measurement.

The majority of respondents (62%) identified that they felt their privacy being invaded by facial recognition camera and microphone in the elevator cabin, while 17% could not decide, and 11% did not see a problem with these devices being installed into the elevator cabin. This confirms our initial hypothesis that cameras are perceived as the most privacy-invasive devices, together with microphones. Interestingly, a fifth of respondents also considered weight measurement, present in every elevator, as a threat to their privacy. Table 10.6 reports the responses.

## 10.6 Discussion

Although advancements in technology have enabled to equip, for example, common passenger elevators with features of a smart-elevator, there is still a long way to go toward smarter and assistive elevator travel. In comparison to traditional passenger elevators, smart-elevators allow to deliver additional features and value to make passengers' daily travels more convenient and improve through it UX. For example, inclusion of voice user interfaces (VUIs) would allow to select destination floor without the need to touch buttons, a highly assistive service for persons with visual impairment, or for anyone in case hands are loaded with things. To improve user travel experience, it is important to collect knowledge on the actual exploitation of elevators, including passengers' behavior in the cabin and its dependency on the number of passengers during their travels. This could, for instance, be used as an input for cabin layout design. Adding smartness to elevators will in the long run
enable to establish smart-elevator systems that are able to learn from and adapt to its daily users and the habits of the building residents, either on a personal or aggregated level, targeting smooth and convenient travel with least waiting time and power consumption.

The present SES, initiated just as an innovation project to bring smart features to common elevators, has found its way into research as a platform to study user behavior. This chapter focused on passenger in-cabin behavior based on the location detection made possible by this smart-elevator platform. Elevators, compact enough, are just one of the possible type of objects, where user behavioral studies could be beneficial. For example, the approach described in this chapter could be transferred to study the distribution of passengers in public transport systems, e.g., city buses and bus stops, to either find better cabin layout, improve usability, or passenger flow.

On top of allowing to study user behavior in a closed space through a short period, such as the elevator travel is, the smart-elevator platform equipped with passenger position detection can provide valuable information for cabin interior (re-)design, e.g., placement of interactive screens (including building guide, important information boards, or screens with commercials in department stores), placement of security cameras, etc. based on actual exploitation of elevators by its passengers, and thereby contribute to increased levels of UX. For instance, our studies showed that passengers prefer to stand close to the button panel in the center of the cabin or opposite to it in this particular elevator type. The SES platform in combination of additional sensors could also be used to detect emergency situations, for example, in elderly homes, where the system could automatically notify the operator that a person has fallen and needs attention. In other cases of emergency, e.g., fire, the platform could provide immediate feedback on the number of passengers in the cabin. The system could also be used for vandalism detection based on fast movements in the cabin environment.

Elevators are able to detect changes in the weight of the cabin. Yet there is no way to distinguish whether this change came from passengers or cargo. The SES discussed in this chapter would allow to add this knowledge to the weight change with a precision of a passenger count. This would allow to establish better usage models and improve maintenance planning, which today is planned as a periodic event over some time, but could be transformed into aperiodic and usage-based, being more economical and allowing to save on cost.

Like with every new technology (recall that the first passenger elevator was rejected by the users), there are also challenges with smart-elevators. Although the technology may allow to provide a better service and novel features, this may come on the cost of privacy (e.g., cameras) and may not be well received by users—passengers may feel discomfort or annoyed and discontinue the use. Even if the system is technically secure, convincing passengers of it and winning their trust may take time. After the setup of the smart-elevator in the ICT building, we noticed a rise in the use of stairs instead of an elevator, even up to higher floors. Also, technical challenges have arisen. The traditional elevator technology is proven and rarely fails, whereas the smart features and their combination are rather experimental and therefore require a DevOps person or a team to continuously be involved and
monitor the system and the status of its various components and services, for proper
system exploitation.

As for future work, we have planned to continue to maintain the smart-elevator platform and to further investigate novel possibilities applicable for smart travelling with elevators. For example, a project of a “touchless-button” was initiated (inspired by the pandemics), allowing passengers to place elevator calls without touching any buttons and set the destination floor already when placing the elevator call (Reinsalu and Robal 2023). The latter will allow to optimize operations and can be considered a step toward energy-saving and greener deal delivered by smart solutions. In the future, it would be interesting to investigate our approach on several different elevator types, preferably in high-rise buildings, should this become possible. With the existing SES, we would like to continue behavior studies and explore how individual passengers react to (de-)boarding in sense of their position or how passengers change their behavior when their destination floor is about to be reached.

**10.7 Conclusions**

Advances in technology have delivered the possibility to set up smart-elevators as cyber physical-social systems. These systems, based on their innovative features, enable a platform to study passengers’ behavior—once used to be possible only with surveillance cameras and manual work—to improve elevator systems, user experience, quality of service (QoS), and passenger travel experience and further enhance the concept of smart-elevators toward smarter life.

This chapter discussed a model and methods for elevator passengers' in-cabin behavior analysis, which were benchmarked and validated against the smart-elevator positioning service. This allowed to investigate real passengers' in-cabin behavior using an existing smart-elevator platform built using common hardware and software. Although our study was only limited to one type of an elevator cabin, the established section-based location analysis model and methods could be applied to any other elevator type able to carry more than ten persons at a time, being therefore large enough by floor area. Such elevators are typically found in large commercial buildings, shopping malls, hotels, hospitals, etc. With the model, methods, and experiments, we have contributed to fill the gap in existing literature regarding studies on human behavioral patterns in the context of elevator travels. The results can be used for smart-elevator cabin layout design, including sensors, but also improve the quality of service by knowing how passengers take advantage of the existing elevator in real-life situations—all in all, small things matter!

The scenarios designed for the ground-truth study to validate the elevator system positioning service can be used on any elevator (or other system) type for a similar purpose. As for the passengers' behavior, we explored whether there are favorable standing places for elevator passengers in general, finding that passengers tend to prefer to stand in the middle of the elevator while being the only occupant of the cabin, which is not the case when multiple travellers occupy the cabin. Further,
we took advantage of the smart-elevator system profiling capabilities to identify repeatedly travelling passengers to investigate whether passengers tend to have personal preferred standing locations and if they follow certain movement paths in the cabin. The analysis revealed that indeed passengers tend to choose the same standing location quite often (60%) in case they are the only occupant of the cabin. We however failed to find confident results that passengers would always follow the same movement path while entering, travelling, and exiting the elevator. The small sample of data we collected due to COVID restrictions intervening our study was not sufficient to draw any solid conclusions.

With the advancement of technology over time driving the elaboration of smart cities for smarter everyday life, smart-elevators will also become a standard of every building, delivering us improved and more convenient travel service for different situations, considering the operational context and our needs at a particular moment in time and space.

## References

* Allen, J.: Speech Recognition and Synthesis, pp. 1664–1667. John Wiley and Sons, GBR (2003)

* Bamunuarachchi, D.T., Ranasinghe, D.N.: Elevator group optimization in a smart building. In: 2015 IEEE 10th International Conference on Industrial and Information Systems (ICIIS), pp. 71–76 (2015)

* Basov, K., Robal, T., Reinsalu, U., Leier, M.: Elevator passenger in-cabin behaviour – a study on smart-elevator platform. In: Ivanovic, M., Kirikova, M., Niedrite, L. (eds.) Digital Business and Intelligent Systems, pp. 3–18. Springer, Cham (2022)

* Bernard, A.: Lifted: A Cultural History of the Elevator. NYU Press, New York (2014)

* Bharti, H., Saxena, R.K., Sukhija, S., Yadav, V.: Cognitive model for smarter dispatch system/elevator. In: 2017 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM), pp. 21–28 (2017)

* Brand, M., Nikovski, D.: Optimal parking in group elevator control. In: IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, vol. 1, pp. 1002–1008 (2004)

* Calinescu, R., Cámara, J., Paterson, C.: Socio-cyber-physical systems: models, opportunities, open challenges. In: 2019 IEEE/ACM 5th Intl Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS), pp. 2–6 (2019)

* Cassandras, C.G.: Smart cities as cyber-physical social systems. Engineering 2(2), 156–158 (2016)

* Chou, S., Budhi, D.A., Dewabharata, A., Zulvia, F.E.: Improving elevator dynamic control policies based on energy and demand visibility. In: 2018 3rd International Conference on Intelligent Green Building and Smart Grid (IGBSG), pp. 1–4 (2018)

* Dressler, F.: Cyber physical social systems: Towards deeply integrated hybridized systems. In: 2018 International Conference on Computing, Networking and Communications (ICNC), pp. 420–424 (2018)

* Fernandez, J.R., Cortes, P.: A survey of elevator group control systems for vertical transportation: a look at recent literature. IEEE Control Syst. Mag. 35(4), 38–55 (2015)

* Fujimura, T., Ueno, S., Tsuji, H., Miwa, H.: Control algorithm for multi-car elevators with high transportation flexibility. In: 2013 IEEE 2nd Global Conference on Consumer Electronics (GCCE), pp. 544–545 (2013)