# Off-Line Evaluation of Indoor Positioning Systems in Different Scenarios: The Experiences From IPIN 2020 Competition

Francesco Potorti, Joaquin Torres-Sospedra, Darwin Quezada-Gaibor, Antonio Ramon Jimenez, Fernando Seco, Antoni Perez-Navarro, Miguel Ortiz, Ni Zhu, Valerie Renaudin, Ryosuke Ichikari, et al.

► To cite this version:

Francesco Potorti, Joaquin Torres-Sospedra, Darwin Quezada-Gaibor, Antonio Ramon Jimenez, Fernando Seco, et al. Off-Line Evaluation of Indoor Positioning Systems in Different Scenarios: The Experiences From IPIN 2020 Competition. IEEE Sensors Journal, 2022, 22 (6), pp.5011-5054. 10.1109/JSEN.2021.3083149. hal-04500584

HAL Id: hal-04500584

https://hal.science/hal-04500584v1

Submitted on 14 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Off-Line Evaluation of Indoor Positioning Systems in Different Scenarios: The Experiences From IPIN 2020 Competition

Francesco Potorti<sup>ID</sup>, Member, IEEE, Joaquín Torres-Sospedra<sup>ID</sup>, Darwin Quezada-Gaibor, Antonio Ramón Jiménez<sup>ID</sup>, Fernando Seco<sup>ID</sup>, Antoni Pérez-Navarro<sup>ID</sup>, Member, IEEE, Miguel Ortiz<sup>ID</sup>, Ni Zhu<sup>ID</sup>, Valerie Renaudin<sup>ID</sup>, Member, IEEE, Ryosuke Ichikari<sup>ID</sup>, Ryo Shimomura, Nozomu Ohta, Satsuki Nagae, Takeshi Kurata<sup>ID</sup>, Dongyan Wei<sup>ID</sup>, Member, IEEE, Xinchun Ji, Wenchao Zhang<sup>ID</sup>, Sebastian Kram<sup>ID</sup>, Maximilian Stahlke<sup>ID</sup>, Christopher Mutschler<sup>ID</sup>, Antonino Crivello<sup>ID</sup>, Paolo Barsocchi<sup>ID</sup>, Michele Girolami<sup>ID</sup>, Filippo Palumbo<sup>ID</sup>, Ruizhi Chen<sup>ID</sup>, Yuan Wu, Wei Li, Yue Yu<sup>ID</sup>, Shihao Xu<sup>ID</sup>, Lixiong Huang<sup>ID</sup>, Tao Liu, Jian Kuang<sup>ID</sup>, Xiaoji Niu<sup>ID</sup>, Takuto Yoshida<sup>ID</sup>, Yoshiteru Nagata<sup>ID</sup>, Yuto Fukushima<sup>ID</sup>, Nobuya Fukatani, Nozomi Hayashida<sup>ID</sup>, Yusuke Asai, Kenta Urano, Graduate Student Member, IEEE, Wenfei Ge<sup>ID</sup>, Nien-Ting Lee, Shih-Hau Fang<sup>ID</sup>, Senior Member, IEEE, You-Cheng Jie<sup>ID</sup>, Shawn-Rong Young<sup>ID</sup>, Ying-Ren Chien<sup>ID</sup>, Senior Member, IEEE, Chih-Chieh Yu<sup>ID</sup>, Chengqi Ma<sup>ID</sup>, Bang Wu<sup>ID</sup>, Wei Zhang<sup>ID</sup>, Yankun Wang, Yonglei Fan<sup>ID</sup>, Stefan Poslad<sup>ID</sup>, David R. Selviah<sup>ID</sup>, Member, IEEE, Weixi Wang<sup>ID</sup>, Hong Yuan<sup>ID</sup>, Yoshitomo Yonamoto, Masahiro Yamaguchi, Tomoya Kaichi<sup>ID</sup>, Baoding Zhou<sup>ID</sup>, Xu Liu<sup>ID</sup>, Zhining Gu, Chengjing Yang, Zhiqian Wu, Doudou Xie, Can Huang, Lingxiang Zheng, Ao Peng<sup>ID</sup>, Member, IEEE, Ge Jin<sup>ID</sup>, Qu Wang<sup>ID</sup>, Haiyong Luo<sup>ID</sup>, Member, IEEE, Hao Xiong, Linfeng Bao, Pushuo Zhang, Fang Zhao<sup>ID</sup>, Chia-An Yu<sup>ID</sup>, Chun-Hao Hung<sup>ID</sup>, Leonid Antsfeld, Boris Chidlovskii<sup>ID</sup>, Member, IEEE, Haitao Jiang, Ming Xia<sup>ID</sup>, Dayu Yan<sup>ID</sup>, Yuhang Li, Yitong Dong, Ivo Silva<sup>ID</sup>, Cristiano Pendão<sup>ID</sup>, Filipe Meneses<sup>ID</sup>, Member, IEEE, Maria João Nicolau<sup>ID</sup>, António Costa<sup>ID</sup>, Member, IEEE, Adriano Moreira<sup>ID</sup>, Member, IEEE, Cedric De Cock<sup>ID</sup>, David Plets<sup>ID</sup>, Member, IEEE, Miroslav Opiela<sup>ID</sup>, Jakub Džama, Liqiang Zhang<sup>ID</sup>, Graduate Student Member, IEEE, Hu Li, Boxuan Chen, Yu Liu<sup>ID</sup>, Seanglidet Yean<sup>ID</sup>, Bo Zhi Lim<sup>ID</sup>, Wei Jie Teo<sup>ID</sup>, Bu Sung Lee<sup>ID</sup>, Senior Member, IEEE, and Hong Lye Oh<sup>ID</sup>

**Abstract**—Every year, for ten years now, the IPIN competition has aimed at evaluating real-world indoor localisation systems by testing them in a realistic environment, with realistic movement, using the EvAAL framework. The competition provided a unique overview of the state-of-the-art of systems, technologies, and methods for indoor positioning and navigation purposes. Through fair comparison of the performance achieved by each system, the competition was able to identify the most promising approaches and to pinpoint the most critical working conditions. In 2020, the competition included 5 diverse off-site off-site Tracks, each resembling real use cases and challenges for indoor positioning. The results in terms of participation and accuracy of the proposed systems have been encouraging. The best performing competitors obtained a third quartile of error of 1 m for the Smartphone Track and 0.5 m for the Foot-mounted IMU Track. While not running on physical systems, but only as algorithms, these results represent impressive achievements.

**Index Terms**—Indoor positioning and navigation, evaluation, smartphone-based positioning, foot-mounted IMU, positioning in industrial scenarios and factories, vehicle-positioning.

<figure><img src="image_2.png" /></figure>

## I. INTRODUCTION

Manuscript received April 2, 2021; accepted May 6, 2021. Date of publication May 24, 2021; date of current version March 14, 2022. The associate editor coordinating the review of this article and approving it for publication was Prof. Chan Gook Park. (Corresponding authors: Joaquín Torres-Sospedra; Antonino Crivello.)

Please see the Acknowledgment section of this article for the author supporting information and affiliations.

Digital Object Identifier 10.1109/JSEN.2021.3083149

THE International conference on Indoor Positioning and Indoor Navigation (IPIN), born in 2010, has been a reference for researchers and practitioners interested in systems, methods, techniques and technologies for indoor positioning and indoor navigation. In fact, estimating the location of a mobile target still represents a challenging task in indoor
environments. While solutions based on Global Navigation Satellite System (GNSS) are successfully used outdoor, pinpointing the location of an indoor target requires the adoption of technologies that most often cannot exploit satellites because indoor obstacles, walls and, most of all, ceilings are all factors that significantly reduce the strength of satellite signals. Indoor localisation systems, be they targeted at personal navigation or other purposes, heavily rely on the use of a wide variety of sensors. This is in sharp contrast with outdoor localisation, which relies only on GNSS radio signals, at least as far as consumer-grade applications are concerned.

Since its inception, IPIN's core topics have been low-level hardware and software techniques for positioning and navigation. In the last few years a growing interest has been observed in topics regarding system evaluation, standardisation and interoperability. In fact, reaching a wide consensus on the evaluation metrics for these systems is a fundamental step towards filling the gap between prototypes and commercial systems. In this paper we are particularly interested in testing and evaluation of systems and, as a showcase, we fully describe the IPIN 2020 competition. While the competition usually benefits from the attendance of the congress, in 2020 the competition was a solo event which nonetheless attracted 95 attendees to the final event, which was held online.

Past editions of the IPIN competitions were organised by hosting two different kinds of Tracks, namely on-site and off-site. In the on-site Tracks, competitors demonstrate their system by performing an assigned test in a given place. An actor carries the competing system while walking in and between multi-floor buildings. The system shall provide position estimates in real time using local data processing on opportunistic signals, without any ad hoc infrastructure. In off-sites Tracks competitors calibrate their algorithms in advance using a ground-truth reference database provided by the committee, and compete using new unreferenced data. Due to worldwide travel restrictions, the 2020 competition only hosted off-site tracks for active indoor positioning systems.

This paper contains organisational aspects and highlights the choices taken by the organisers. The core part of the paper is the description of the competing systems. This edition provided five off-site Tracks: *Smartphone, Foot-mounted IMU, xDR in manufacturing, On-vehicle smartphone and Channel impulse response*. Each Track is explained in a dedicated section which also contains contributions and system descriptions authored by the competitors. As a follow-up to [1], [2], this work provides a unique overview on the state of the art of systems, technologies and methods for indoor positioning and navigation purpose. Through a fair comparison, the performance achieved by each system in a real-world scenario helps understanding which are the most promising approaches, under which working conditions. Comparison is performed according to the Evaluating Ambient Assisted Living (EvAAL) framework [3].

The paper is structured as follows. Section II summarises the history of the IPIN competition and highlights possible future directions for the next editions. Section III is an overview of the five Tracks and their commonalities, which are founded on the EvAAL framework. Sections IV to VIII report

the characteristics and final results for each Track and the detailed descriptions of most competing systems. Section IX is an attempt at identifying lessons to be learned from the practical experience of competitors: even if the competition was off-site, the algorithms used were stressed in a challenging and competitive environment, which offered insight to both competitors and attendees.

## II. PAST, PRESENT AND FUTURE DIRECTIONS

Research in the area of indoor positioning and navigation in the last decade has elicited a strong interest from both academic and industrial communities. We expect indoor Location-based services (LBS) to experience significant growth and evolution and to be commonly available on commercial devices in the future.

Although impressive advances in the field of algorithms for indoor localisation and tracking have been achieved, evaluation frameworks were missing. Under this respect, the EvAAL framework was a pioneer initiative devoted to compare, with a rigorous methodology, the performance of indoor localisation systems in real-world, non-trivial settings. Here we summarise the 10-year-long journey of the EvAAL framework, from the first EvAAL competition in 2011 to the recent IPIN 2020 competition, and we give a look at the next edition of IPIN scheduled for late November 2021.

The EvAAL framework has been designed to test and compare the performance of indoor localisation systems, following a rigorous approach. It consists of four core criteria plus four extended criteria, the latter being desirable ones which should be applied as far as possible [3]. The core criteria, which are necessary to define a competition as conformant to the EvAAL framework, are:

1) *Natural movement of an actor:* an actor walks with natural speed and attitude.

2) *Realistic environment:* the walking path is set in a realistic setting; EvAAL competitions were done in a living lab, IPIN competitions in wider settings, like a congress centre, a university building, a shopping mall.

3) *Realistic measurement resolution:* final error measurements below 50 cm in space and 0.5 s in time should be considered as null, when indoor people's movement are considered; when the actor walks, the test should be considered adequate if his/her time and space errors when passing on the test points are not greater than the above figures, which is easy for a trained person.

4) *Third quartile of point Euclidean error:* the accuracy score is based on the third quartile of the point error.

Applicability of extended criteria to IPIN 2020 is discussed in Section III.

Table I is an overview of the size of past competitions. While the IPIN competitions aim to compare systems based only on their accuracy performance, the early EvAAL editions were characterised by a richer set of goals, including the deployment complexity of the solution; the time required to calibrate and configure the system; the impact of the system in terms of the end-user's perception. These indicators were mainly driven by the Ambient Assisted Living (AAL) application scenarios to which EvAAL was inspired [4].
<table><caption>TABLE I
NUMBER OF TRACKS AND NUMBER OF COMPETITORS FOR ON-SITE AND OFF-SITE INDOOR LOCALISATION COMPETITION TRACKS</caption><thead><tr><th>Edition</th><th>Tracks</th><th>Competitors Real-time</th><th>Competitors Off-line</th></tr></thead><tbody><tr><td>EvAAL 2011</td><td>1</td><td>7</td><td>-</td></tr><tr><td>EvAAL 2012</td><td>1</td><td>8</td><td>-</td></tr><tr><td>EvAAL 2013</td><td>1</td><td>7</td><td>-</td></tr><tr><td>IPIN 2014</td><td>2</td><td>7</td><td>-</td></tr><tr><td>IPIN 2015</td><td>3</td><td>6</td><td>4</td></tr><tr><td>IPIN 2016</td><td>4</td><td>14</td><td>5</td></tr><tr><td>IPIN 2017</td><td>4</td><td>7</td><td>9</td></tr><tr><td>IPIN 2018</td><td>4</td><td>14</td><td>19</td></tr><tr><td>IPIN 2019</td><td>5</td><td>8</td><td>15</td></tr><tr><td>IPIN 2020</td><td>5</td><td>-</td><td>21</td></tr></tbody></table>

In 2014, the EvAAL competition met the IEEE IPIN conference, giving birth to the IPIN competition [5]. Such partnership was the result of two complementary communities: on the one hand, experience from the EvAAL competitions provided a well-established evaluation framework; on the other hand the IPIN provided a vibrant community composed of academic and industrial attendants who every year share advances in the area of indoor navigation and positioning. The birth of the IPIN competition series extended the range of potential competitors. Indeed, the IPIN competition has seen a consistent increased of the number of competition Tracks, each of which focused on specific constraints and objectives, as shown in Table I. Tracks are split between on-site and off-site. The on-site Tracks take place during the IPIN conference and competitors do a live test of their solutions. Off-site Tracks are performed remotely. For the latter ones, competitors are required to test their solution by following rules and data sets provided by the organisers.

During the last 7 IPIN competition editions, competitors have had the opportunity to test their systems in shopping malls, conference halls, university campus and large research centres. Such a variety of locations is the distinguishing feature of IPIN competitions with respect to similar initiatives. In fact, the confluence of EvAAL into the IPIN conference refined the methodology adopted to assess the performance, adding the following characteristics: no additional instrumentation allowed, non-overlapping competition Tracks, highly representative competition areas, easy-to-understand measurement statistics to define the final ranking of the tested systems.

Appreciation of this format by competitors (both from academy and industry) and sponsors is reflected in the consistently growing attendance to the competition.

So far so good, but what's next for the IPIN competition? Organisers are looking at two growing trends:

* the increasing performance and diffusion of sensing units available with commercial devices;

* the wide adoption of learning methodologies with a never-seen-before statistical power.

As far as sensing is concerned, new short-range Radio-Frequency (RF) technologies such as Wi-Fi Time-of-Flight (TOF) measurements, Ultra-Wide Band (UWB) and Bluetooth 5.0 are the next obvious target to include in testing by augmenting the existing Tracks or creating new ones. In the future, medium- and long-range RF technologies 5G

and 6G may become drivers for localisation technology, but currently it is not easy to set up a representative testbed: telecommunication providers might play a crucial role for indoor localisation; the IPIN competition is open to testing and experiencing such disrupting technologies.

As far as the increasing pervasiveness of machine learning is concerned, our prospect is to support such evolution by offering always-more challenging data sets to the competitors, in order to assess the performance of their systems, as it has been done with the off-site Tracks. Under this respect, we consider heterogeneity as one of the most challenging properties of such data sets. Heterogeneity refers to the different nature of data that can be simultaneously analysed, to improve the performance of ML-based algorithms. Fingerprint data sets, based on of Wi-Fi Received Signal Strength Indicator (RSSI) readings, can be enriched with context information derived from Bluetooth beacons, environmental or physiological sensor readings, giving rise to unexpected possible correlations. In turn, such data sets can be used as non-structured inputs to multi-layer neural networks (e.g. Recurrent Neural Network (RNN) based on Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) layers) to solve classification and regression problems applied to indoor localisation. This trend has been in place in both on-site and off-site Tracks in the last years, and it is going to continue.

Another interesting topic where the IPIN competition could promote new challenges is the adoption of a more accurate and meaningful metric for computing the positioning error. In fact, one of the objectives of IPIN is to define standard procedures for evaluating positioning systems, in an effort to improve over the recent ISO/IEC 18305 standard [6], [7]. Discussion is underway about using an alternative or additional criterion for computing the error, that is, the distance from each reference point in the ground truth to the position estimated by the competing system. Currently, the IPIN competition series defines the point error as the horizontal distance plus a fixed penalty of 15 m per each wrong floor. Now times appear to be mature for the adoption of a “real-world” distance, that is, the length of the path that a person would need to travel from the reference point to the estimated point. This is the same as the Euclidean distance if the two points are in line of sight, that is on the same floor and in the same room, but can be very different otherwise. A complete discussion on the benefits of this new method and of the possible algorithms to use, complete with code, is available at [8].

A final consideration about the future of the IPIN competition refers to the integration of multiple indoor localisation systems. More specifically, we envision a future where different indoor localisation services coexist in the same area. Such systems will require to be integrated and orchestrated so that to reproduce, as much as possible, the well-assessed user experience of navigation in outdoor environments [9]. We consider two key challenges:

* to standardise Application Programming Interfaces (APIs) designed to discover, access and use an indoor localisation systems with a commercial device;

* to regulate the privacy consents asked of end-users in order to provide location-based services in accordance
with the EU General Data Protection Regulation (GDPR) regulation framework.

### III. ORGANISING AN OFFSITE COMPETITION WITH MULTIPLE TRACKS

In the 2014–2019 editions the systems competing in on-site Tracks were tested during the IPIN conference, in the same or a nearby site. This way, competitors were able to both compete with their system and attend the conference, and conference attendees could come and look at how the competition was done. Since the conference was cancelled in 2020, only off-site Tracks were organised for this edition, under the supervision of competition chairs Francesco Potortì and Sangjoon Park.

The institutions involved were the Institute of Information Science and Technologies (ISTI) of the National Research Council (CNR, IT), UBIK Geospatial Solutions S.L. (ES), the GEOTEC laboratory of Universitat Jaume I (UJI, ES), Consejo Superior de Investigaciones Científicas (CSIC, ES), the IN3 of Universitat Oberta de Catalunya (UOC, ES) the GEOLOC Team, University Eiffel (FR), the National Institute of Advanced Industrial Science and Technology (AIST, JP), University of Tsukuba (JP), Aerospace Information Research Institute, (CAS, CH), IIS Fraunhofer (DE) and the Electronics and Telecommunications Research Institute (ETRI, KR).

#### A. Preparing the Competition Areas

In contrast with the two previous editions where most on-site and off-site Tracks took place in the same large area (a shopping mall in 2018 [1] and a research centre in 2019 [2]), in 2020 travel restrictions made it impractical to gather together and take measurements in the same place, so all Tracks were independent.

The set of evaluation scenarios cover a university library building, the shopping mall from IPIN 2018 competition [1], a manufacturing site, road-based tracks with different satellite view conditions (including indoors) and an environment resembling an industrial setting.

All the Tracks complied with the EvAAL framework [3] by adopting its four distinguishing core criteria (described in Section II):

1) Natural movement of an actor

2) Realistic environment

3) Realistic measurement resolution

4) Third quartile of point Euclidean error

Additionally, all the Tracks were compliant with most of the extended criteria defined by the EvAAL framework, as detailed below.

1) Secret Path: The final path is disclosed immediately before the test starts, and only to the competitor whose system is under test. This prevents competitors to design systems exploiting specific features of the path. This criterion is always respected in all Tracks given the way the off-site competition Tracks are set up: competitors are provided with training sets, ground truth and, in some Tracks, a map. When they have finished tuning their systems, they ask the organisers for a path without ground truth, and submit their estimate. The ground truth is published only after the competition is finished.

2) Independent Actor: The actor is an agent not trained to use the localisation system. This criterion is always respected, given the way the off-site Tracks are set up.

3) Independent Logging System: The competitor system estimates the position at a rate of twice per second.... This criterion is respected or exceeded in all Tracks.

... and sends the estimates on a radio network provided by the committee. This prevents any malicious actions from the competitors. The source code of the logging system is publicly available. This criterion is not respected, because the competitors may retry and further tune their systems while trying to guess the correct ground truth. To avoid this, the committee should ask the competitors to provide their code, and run it locally in a real-time fashion, or provide a real-time APIs. This is feasible, in principle, but would require a non-trivial software infrastructure to be in place, and a non-trivial additional effort from the competitors to comply with it.

4) Identical Path and Timing: The actor walks along the same identical path with the same identical timing for all competitors, within time and space errors smaller than the above defined measurement resolution. This is a natural consequence of the fact that the same data are provided to all competitors.

#### B. Competition Results

For each submitted trial, the error was computed by comparing the estimated coordinates with the ground truth, that is, reference coordinates of the key points marked on the ground along the path. This metric combines the floor detection accuracy and the horizontal positioning error.

$$ \epsilon = \|\mathbf{P}_R - \mathbf{P}_E\| + p \cdot |f_R - f_E| \quad (1) $$

where

* $\mathbf{P}_R$ is the vector with the ground truth horizontal (2-D) coordinates

* $\mathbf{P}_E$ is the vector with the horizontal coordinates estimated by the competitors

* $\|\mathbf{P}_R - \mathbf{P}_E\|$ is the horizontal error, and it is computed as the Euclidean distance between the ground truth and the estimated position provided by the competitor in the 2D space.

* $p$ is the base floor estimation error penalty and is set to 15 m.

* $|f_R - f_E|$ is the absolute difference between the actual floor number and the estimated one.

The point error $\epsilon$ is computed for all key points marked on the ground that define the path of a specific challenge. The “accuracy score” $s$ is given by the third quartile of $\epsilon$:

$$ s = 3^{\text{rd}}\text{quartile}\{\epsilon\} \quad (2) $$

The team with the lowest score wins the challenge. Note that each competitor had the opportunity to submit the results for multiple trials. Table II shows the scores for all the five Tracks. Some additional metrics included in the ISO/IEC 18305 standard are also reported in the table. Fig. 1 depicts the cumulative distributions of the accuracy score $s$ for the winners and runners-up of the five Tracks.
<table><caption>TABLE II
RESULTS FOR ALL TRACKS. THE FIRST COLUMN IS THE COMPETITION SCORE (EQUATION 2), WHILE THE REMAINING COLUMNS SHOW OTHER COMPLEMENTARY RELEVANT METRICS (MEAN, RMSE. MEDIAN, 95th PERCENTILE AND FLOOR HIT RATE (IF AVAILABLE). WE ALSO INCLUDE A REFERENCE TO THE SECTION WHERE THE SYSTEM IS DESCRIBED</caption><thead><tr><th>Track</th><th>Team</th><th>Score 3rd quartile [m]</th><th>Mean [m]</th><th>RMSE [m]</th><th>Median [m]</th><th>95th percentile [m]</th><th>Floor detect rate [%]</th><th>Section reference</th></tr></thead><tbody><tr><td rowspan="14">Track 3<br>Smartphone-based<br>Pedestrian Positioning</td><td>WHU-FIVE</td><td>0.98</td><td>0.86</td><td>1.05</td><td>0.76</td><td>1.97</td><td>100.00</td><td>IV-C.1</td></tr><tr><td>IOT2US</td><td>1.72</td><td>1.26</td><td>1.46</td><td>1.14</td><td>2.59</td><td>100.00</td><td>IV-C.2</td></tr><tr><td>XMU.ATR</td><td>1.85</td><td>1.28</td><td>1.45</td><td>1.12</td><td>2.53</td><td>100.00</td><td>IV-C.3</td></tr><tr><td>Naver Labs Europe</td><td>1.95</td><td>1.57</td><td>2.54</td><td>1.18</td><td>3.99</td><td>98.78</td><td>IV-C.4</td></tr><tr><td>UMinho</td><td>2.72</td><td>1.93</td><td>2.40</td><td>1.57</td><td>5.07</td><td>100.00</td><td>IV-C.5</td></tr><tr><td>imec-Waves</td><td>2.81</td><td>2.19</td><td>3.62</td><td>1.22</td><td>5.17</td><td>97.56</td><td>IV-C.6</td></tr><tr><td>Yai</td><td>4.73</td><td>6.75</td><td>14.23</td><td>2.80</td><td>49.13</td><td>92.68</td><td>IV-C.7</td></tr><tr><td>WiMap</td><td>5.41</td><td>3.65</td><td>4.49</td><td>2.94</td><td>7.50</td><td>98.78</td><td>-</td></tr><tr><td>Indora</td><td>6.85</td><td>7.02</td><td>10.96</td><td>3.80</td><td>27.19</td><td>100.00</td><td>IV-C.8</td></tr><tr><td>TJU</td><td>7.22</td><td>5.11</td><td>6.09</td><td>3.80</td><td>10.91</td><td>98.78</td><td>IV-C.9</td></tr><tr><td>Next-Newbie Reckoners</td><td>23.71</td><td>20.09</td><td>21.54</td><td>19.22</td><td>34.66</td><td>93.90</td><td>IV-C.10</td></tr><tr><td rowspan="4">Track 4<br>Foot-mounted IMU</td><td>WHU-GNSS</td><td>0.50</td><td>0.36</td><td>1.10</td><td>0.05</td><td>1.07</td><td>100.00</td><td>V-C.1</td></tr><tr><td>AIR</td><td>5.83</td><td>3.87</td><td>4.85</td><td>3.13</td><td>10.16</td><td>100.00</td><td>V-C.2</td></tr><tr><td>Free-Walking</td><td>65.17</td><td>47.65</td><td>52.30</td><td>49.13</td><td>82.08</td><td>13.43</td><td>V-C.3</td></tr><tr><td>BHSNIP</td><td>89.93</td><td>58.39</td><td>68.86</td><td>42.04</td><td>119.82</td><td>13.43</td><td>V-C.4</td></tr><tr><td rowspan="2">Track 5<br>Pedestrian<br>Dead-Reckoning</td><td>Kawaguchi Lab (whole)<br>(ave.of scores)</td><td>6.96<br>6.83</td><td>5.34<br>5.21</td><td>6.23<br>6.03</td><td>4.85<br>4.77</td><td>10.80<br>10.31</td><td>na<br>na</td><td>VI-C.1</td></tr><tr><td>yonayona (whole)<br>(ave.of scores)</td><td>14.75<br>12.36</td><td>11.40<br>10.35</td><td>12.80<br>11.26</td><td>10.42<br>10.23</td><td>22.07<br>18.68</td><td>na<br>na</td><td>VI-C.2</td></tr><tr><td rowspan="2">Track 5<br>Vehicle<br>Dead-Reckoning</td><td>Kawaguchi Lab (whole)<br>(ave. of scores)</td><td>24.82<br>27.05</td><td>19.82<br>20.41</td><td>22.22<br>22.97</td><td>18.26<br>20.07</td><td>39.31<br>36.04</td><td>na<br>na</td><td>VI-C.1</td></tr><tr><td>yonayona (whole)<br>(ave. of scores)</td><td>60.87<br>60.23</td><td>47.99<br>44.91</td><td>59.65<br>54.03</td><td>35.75<br>34.63</td><td>125.13<br>95.51</td><td>na<br>na</td><td>VI-C.2</td></tr><tr><td rowspan="3">Track 6<br>Smartphone-based<br>Vehicle Positioning</td><td>WHU-Autonavi</td><td>7.02</td><td>6.75</td><td>11.32</td><td>2.97</td><td>31.33</td><td>na</td><td>VII-C.1</td></tr><tr><td>SZU</td><td>18.49</td><td>11.76</td><td>17.02</td><td>6.96</td><td>32.45</td><td>na</td><td>VII-C.2</td></tr><tr><td>YAI</td><td>236.6</td><td>174.87</td><td>194.56</td><td>177.76</td><td>315.1</td><td>na</td><td>VII-C.3</td></tr><tr><td>Track 7<br>Channel Impulse Resp</td><td>YAI</td><td>1.38</td><td>1.14</td><td>1.66</td><td>0.72</td><td>3.60</td><td>na</td><td>VIII-C.1</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>Fig. 1. Cumulative distributions of point errors (Equation 1).</figcaption></figure>

## IV. TRACK 3: SMARTPHONE

### A. Track Description

The goal of Track 3 is to evaluate the performance of different integrated navigation solutions based on a regular smartphone sensor fusion (magnetometer, barometer, wireless

communications, Attitude and Heading Reference System (AHRS) or micro-electro-mechanical systems (MEMS), among others) in an off-site context. As done in the 2016–2019 editions [1], [2], [10], [11], the same data collection strategy and evaluation procedure has been followed.
<table><caption>TABLE III
INFORMATION OF THE SENSORS IN THE SAMSUNG GALAXY A5 2017 (SM-A520F)</caption><thead><tr><td>Sensor</td><td>Model & Manufacturer</td><td>SamplingFreq.(Hz)</td></tr></thead><tbody><tr><td>Accelerometer</td><td>STM - K6DS3TR</td><td>200</td></tr><tr><td>Gyroscope</td><td>STM - K6DS3TR</td><td>200</td></tr><tr><td>Magnetometer</td><td>AKM - AK9916</td><td>100</td></tr><tr><td>Barometer</td><td>STM - LPS25H</td><td>5</td></tr><tr><td>Light Sensor</td><td>AMS - TMD3725</td><td>5</td></tr><tr><td>Proximity Sensor</td><td>AMS - TMD3725</td><td>2</td></tr><tr><td>AHRS</td><td>Samsung</td><td>100</td></tr><tr><td>GNSS</td><td>GNSS/Network</td><td>na</td></tr><tr><td>Wi-Fi</td><td></td><td>~0.2</td></tr><tr><td>Sound</td><td></td><td>2</td></tr></tbody></table>

The competition data set was collected by the same actor using a Samsung Galaxy A5 2017 (SM-A520F) phone with Android 8.0. Despite being three years old, this model has the advantage of having being used in Track 3 for 2018 and 2019 competitions. The main features of the embedded sensors, including the maximum sampling frequency, are summarised in Table III.

As in the previous competitions, we have used the Android app “GetSensorData” [12] to record and store the smartphone sensors data into a single text file, i.e. into a logfile. The data set is split into three subsets, namely training, validation and evaluation:

* The first set is devoted to calibration purposes and covers most of the evaluation area; it contains 18 short single-floor tracks (collected 4 times each), 4 long trajectories across bookshelves and 2 floor transition tracks. We placed key points at every relevant location, i.e. initial/final locations, significant turns in the tracks and the last step to arrive at a new floor. A total of 78 training logfiles were provided to competitors.

* The second set is devoted to validation purposes, allowing competitors to have an initial assessment of the positioning system, and contains 13 multi-floor long tracks. The number of key points is arbitrary and significantly lower than in the training set. A total of 13 validation logfiles were provided to competitors.

* The last set is devoted to evaluation purposes, allowing competitors to have an independent evaluation without ground truth data, and contains just 1 multi-floor very long track. In contrast to the systematic data collection done in the training files, the evaluation logfile included realistic movements (e.g. simulating a user that was messaging or attending a phone call) and stops. Only 1 unlabelled logfile was provided to competitors.

We set the maximum allowed sampling frequency in “Get-SensorData” for all sensors to record as much as possible data. Additionally, the smartphone was not connected to any cellular or Wi-Fi network as, for instance, the Wi-Fi sampling frequency significantly drops when the phone is connected to a Wi-Fi network. The logfiles and supplementary materials are available in [13]. This package complements the ones from the previous editions [14]–[17].

## B. Competition Area

The environment selected for Track 3 is a modern multi-storey library building located at Universitat Jaume I

<figure><img src="image_4.png" /><figcaption>Fig. 2. Floor plan of UJI's Library.</figcaption></figure>

(Castellón, Spain) and includes a small outdoor area near the main entry. This environment covers the use case for a smartphone application guiding students and staff to find the location of a book.

Before collecting data, the library building was visually inspected to identify the most challenging parts where competitors could find it difficult to obtain accurate positioning. We finally selected the main hall entry (≈300 m²), the second floor (≈1000 m²), the third floor (≈900 m²), the floating fourth floor (≈200 m²) and the fifth floor (≈700 m²) to collect data. We discarded common areas on the first floor and zones with restricted access. The library is composed by two interconnected blocks, we mainly collected data in the first block, except for the fifth floor, where part of the second block was finally surveyed.

For the evaluation path, we considered a walk done by a student that was doing some homework in the library. The student starts sitting in his/her work place (on the third floor), the student stands up and looks for a book, attends a phone call –despite it not being allowed–, comes back to the main workplace and stops for a while. Then, the student needs additional materials that is on the fifth floor, and goes directly there. The book is not in the place it was supposed to be, and the student asks a mate. The book seems to be in the new bookshelves located in the same floor but in the second block (left side in Fig. 2). On the way to the second block, the student meets a friend in the floating fourth floor (which was not mapped). Our student gets the book, returns to the work place, but the computer and other materials are gone. The student has not realised being instead on the second floor and starts to look around desperately. The student goes out to notify the security staff about this event. When the student goes back to the workplace on the third floor, he/she realises everything is there and sits to continue working after 20 minutes walking. The path goes through 82 key points for a total length of approximately 1000 m.

We have used geo-referenced indoor maps (ArcGIS engine) and the ArcMap tool to calibrate all the reference points used in the data set. We performed on-site local measurements using a laser distance meter with respect to representative points, such as walls, pillars and doors, which were already well represented in the indoor maps. The inaccuracies deriving from this procedure might be considered irrelevant as all the information and indoor maps are provided to all competitors.
<figure><img src="image_5.png" /><figcaption>Fig. 3. Tasks during the training stage of WHU-FIVE team system.</figcaption></figure>

### C. Indoor Positioning Solutions Provided by Competitors

1. **Team WHU-FIVE:** Just like many indoor positioning systems, the WHU-Five system includes two stages, the training stage and the testing stage. In the training stage, the system attempts to build fingerprints, train the models and extract some information about the positioning environment. In the testing stage, it deals with the testing data to form the final trajectory, leveraging the trained information and fusion algorithm.

a) **Training stage:** In the training stage, four tasks are performed that can be seen in Fig. 3. The first one is the feature map construction. In this task, the Wi-Fi fingerprint and geo-magnetic fingerprints map is built, and extract the book-shelf layout from the book-shelf training data set. The second task is the training of the step length model. The third one is the extraction of the stair-steps' number of each two layers from the Floor-Transition data set. The last task is training the motion pattern model to recognize some motion types, such as up/down stairs, in/out doors, in-situ steering motion and phone-holding posture.

In order to build the feature maps, the position of every sampled signal features must be known. The reference points are used in the training sets to optimize the Pedestrian Dead Reckoning (PDR) algorithm to obtain a highly accurate trajectory estimations. The model between PDR and reference points can then be built. The model is optimized with the Levenberg-Marquardt (LMA) optimal algorithm. With the optimal PDR, the feature maps for Wi-Fi and geomagnetic can be built. Also, the book-shelf layouts can be extracted. As for the step length model, a leverage linear regression is used to train the model parameters. For the stair-step number between each layer, the average steps of each stair on the stair training data set is used. The motion recognition is performed by training a multi-layer perceptron neural network model. The time domain and frequency domain characteristics are extracted from the original data of the Inertial Measurement Unit (IMU) and barometer sensors, and then are input to the neural network. The output of neural network is the motion types set.

b) **Testing stage:** In the testing stage, first, a Wi-Fi fingerprint positioning is used to find out the initial 3-dimensional position. Then, PDR is fused with the building map and geomagnetic fingerprint positioning result to estimate the 2-dimensional trajectory. Meanwhile the motion recognition

<figure><img src="image_6.png" /><figcaption>Fig. 4. Main algorithm of WHU-FIVE system.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 5. IOT2US system overview.</figcaption></figure>

result is used to revise some error estimation of PDR. Lastly, the motion on stairs is used to estimate the layer id. Combined with the layer id, the 2-dimensional trajectory can be built up to 3-dimensional trajectory.

For the final testing trajectory estimation, the IMU is used to provide the original PDR. The recognized in-situ steering motion is used to weed out the corresponding steps and the phone-holding posture is used to revise the heading of PDR. Then the revised PDR is fused with the building map and book-shelf layout in particle filtering algorithm to obtain further trajectory estimation. The geomagnetic fingerprint matching positioning result is then fused with the previously estimated trajectory in Kalman Filter (KF) algorithm to obtain the final 2-dimensional trajectory estimates. Combining the layer estimation result and the 2-dimensional estimates, the final 3-dimensional trajectory is obtained. The algorithm of testing stage of the system as shown in Fig. 4.

2. **Team IOT2US:** IOT2US team system includes four main stages: 1) the floor and region decisioning based on Wi-Fi; 2) the mobility mode detection; 3) the landmark detection; and 4) the PDR algorithm and information fusion. Six types of sensor data were fused in the whole processing of the track reconstruction and each part involved some of them, as can be seen in Fig. 5. In the following paragraphs every stage is explained in detail.

a) **Wi-Fi:** According to the training data, there are five floors involved in this Track. To determine the floor information and get the rough location of the user, Wi-Fi Received Signal Strength (RSS) information is used to decide the floor and region. Here region is defined as the area that each training logfile covers.
There are two phases in the Wi-Fi process. The RSS fingerprints which contains time, floor, region and key-value pair of MAC and RSS, are built from the training data set during the offline phase. The RSS fingerprint can be represented as a vector of (time, floor, region, mac, rss). And during the online phase, the Wi-Fi RSS information in the evaluation logfile is compared with the RSS fingerprints to compute the most suitable location. More specifically, there are two steps in the online phase.

First, the floor and the region of estimation of the point are determined with respect to the wireless Access Point (AP) availability. A coefficient (λ) is defined to indicate the possibility to estimate a point appearing in the region to which the RSS fingerprint belongs,

$$ \lambda = 1 - \frac{n}{n_e + n_r} \qquad (3) $$

where, the $n_e$ is the number of APs detected at the estimation point, the $n_r$ is the number of APs detected at the RSS fingerprint. The $n$ is the number of APs detected at the estimation point and the RSS fingerprint at the same time. This coefficient $\lambda$ is calculated for every RSS fingerprint that belongs to the same region. The region with the minimum sum $\lambda$ is the estimated region to which the estimation point belongs to. Then the floor based on this estimated region can be obtained.

Second, to get the rough location of the estimation point, the RSS information is compared against all RSS fingerprints. The Euclidean distance in the signal space between the estimation point and every RSS fingerprint is calculated as:

$$ d = \sqrt{\sum_{i=1}^{m} (rss_e^i - rss_r^i)} \qquad (4) $$

where, the $rss_e^i$ and $rss_r^i$ are the RSS values of the i-th AP detected at the estimation point and the RSS fingerprint, respectively.

b) Movement modes recognition: Different modes of mobility can be detected using machine learning or deep learning algorithms using multi-sensors data. For this Track, four categories of motion modes were introduced: normal walking, turning, climbing (stairs), descending (stairs). The process chain mainly includes data segmentation, labelling, feature extraction and classification. In IOT2US team system, accelerometer, gyroscope, magnetic field, and pressure are used for motion modes classification. Some statistical characteristics (e.g., mean, max, derivative) of these time series in time-domain are extracted as features. Decision tree and Support Vector Machine (SVM) are investigated to classify these motion modes.

c) Landmark detection: Map information is one of the most important clues that can help to correct the trajectory. Traditional map matching trends to make use of the structure of the rooms, corridors and tunnels to restrict the trajectory. However, this request too many details of the map and sometimes to measure the building structure in detail is a heavy workload. Hence, some landmarks are identified that activities can only happen at certain places as Correction Reference Points (CRP) to correct the trajectory.

<figure><img src="image_8.png" /><figcaption>Fig. 6. Light intensity determined CRPs in IOT2US system.</figcaption></figure>

Through analyzing the relationship between the real environment and sensors data, three types of CRP can be identified, which are determined by Barometer, Ambient light sensor, and door-crossing activity: 1) the activity of climbing and descending stairs can only happen at stairs, and hence, with a rough location using Wi-Fi can give at least two CRPs. 2) Light intensity measured by Ambient light sensor also has relationship with activities. As an example of walking across bookshelf activity, as shown in Fig. 6, the activity of walking out of the bookshelf and turning backing is recorded as a peak in light intensity. The light determined CRPs also can be determined at place of crossing doors, enter/leave the building and approaching to a window. 3) The third type of CRP determined by door-crossing is similar with the first one that requires recognize the door-crossing activity and Wi-Fi to find a rough location to determine the position of a door.

d) PDR and Information Fusion: Step counting, step/stride length estimation and heading determination are three crucial processes for PDR positioning systems. Peak detection [18] is used to count steps, the Weinberg method [19] to estimate step length and complementary filter [20] for heading determination. To overcome this the traditional PDR algorithm issue of error accumulation over elapsed time, information is fused, including movement modes, floors, regions and landmarks. First, as the sensor data may show distinctive characteristics when a user performs different activities, movement modes are used to finely tune the parameters of step counting and step length estimation. For example, when a user is climbing or descending stairs, the parameters of step counting and step length estimation algorithms is accordingly adjusted to improve their performance. Second, before calculating locations using PDR, Wi-Fi is used to determine the absolute floor and the region information where the user roughly locates. This process assisted IOT2US to provide absolute location to help calculating the PDR trajectory and determining a CRP. And third, if the user is detected as reaching at a CRP, but the calculation result diverges from it, the heading is adjusted, step length and the previous track, to drive the trajectory back to the CRP.

3) Team XMU_ATR: XMU_ATR proposed XMU_PDR system. It is a multi-source indoor positioning system using information obtained from an Inertial Measurement Unit (IMU),
<figure><img src="image_9.png" /><figcaption>Fig. 7. Framework of XMU_ATR team system: XMU_PDR.</figcaption></figure>

Wi-Fi, magnetometer, barometer and indoor maps, jointly. Fig. 7 shows the framework of the proposed system. There are four main functional modules described as in the following paragraphs.

a) *Inverted pendulum model based Pedestrian Dead Reckoning (PDR)*: A raw trajectory is obtained from the inertial data using an inverted pendulum model based PDR algorithm [21] by estimating every pedestrian step length and heading angle directly. The proposed system uses an inverted pendulum model to calculate the step length. To achieve 3-D positioning, the barometer is used to estimate the transition of floors. In the PDR system, heading errors are one of the main factors when estimating positions which will lead to a decrease in positioning accuracy over time, other information needs to be introduced to correct the trajectory.

b) *Wi-Fi fingerprints matching*: Wi-Fi fingerprints matching is a reliable way to obtain absolute indoor position. Since the ground truth of some reference points in the training set is provided, the Wi-Fi Received Signal Strength (RSS) is extracted at the reference points as fingerprints to build up the fingerprints database. Weighted *k*-Nearest Neighbor (WKNN) algorithm is used to match the Wi-Fi Access Point (AP) between fingerprints database and the RSS from evaluation data. Since the initial point is unknown, the trajectory obtained by PDR can only be presented in a temporary navigation frame automatic defined by the dead-reckoning system. Therefore, the result of Wi-Fi positioning can be used to estimate the transformation relationship between the navigation frame and the geographic frame, including coordinate translation and rotation.

c) *Magnetic fingerprints matching*: The indoor magnetic field can be treated as a time invariance distribution in spatiality. The accuracy of spatial resolution can achieve centimeter level in a small area. The magnetic fingerprints matching is used for trajectory refinement. Since the trajectory

to be evaluated may have some overlap with the training set, the observations of magnetometer in the training set are also used as labeled fingerprints. A modified dynamic time warping algorithm is used in this part which can deal with the matching problem between two sequences with different directions. A matching threshold is set to decide whether the matching is successful. If there is a trajectory in the evaluation set match to part of trajectories from the training set, this track can be located on the map.

d) *Map matching*: To reduce positioning error accumulated from the noise of inertial observations, the map information is used for trajectory calibration. The optimal estimation under the map constraint especially the track at specific locations such as walls, doors, and stairs are realized. The floorplan is presented in the form of grids, and a loss function is defined to adjust trajectory actions referring to walls and some specified behavior patterns.

4) *Naver Labs Europe (NLE) Team*: Naver Labs Europe (NLE) Team system is based on extending the localization pipeline developed for IPIN 2019 [1] challenge with new components. The pipeline is a sensor fusion framework deploying smartphone inertial sensors, Wi-Fi measurements, magnetic field data and landmarks. The main components are described in the following paragraphs:

a) *Floor detection*: Floor is detected using a standard *k*-Nearest Neighbor (KNN) classifier trained on barometer and Wi-Fi data.

b) *Activity detection*: User's activity (walking, standing, going up or down the stairs) is identified by applying spectral analysis on the accelerometer data and simple thresholds.

c) *PDR*: User steps are first identified by applying peaks detection to accelerometer data. Then, the acceleration features are extracted and a model is trained for the step length/speed in a given sliding window. Together with the orientation sensor, a first order approximation PDR of user's track is built.
<figure><img src="image_10.png" /><figcaption>Fig. 8. Magnetic field based localization from the NLE team system. Magnetic field data captured by the mobile phone are transformed in time series. Encoded into 2D images, they form an input for Convolutional neural network (CNN) and Recurrent Neural Network (RNN) trained to predict the user's position.</figcaption></figure>

d) *Semi-supervised Variational Auto-Encoder (VAE) for Wi-Fi:* A radiomap is constructed from the Wi-Fi data provided in the training and the validation data sets. Recorded data provides a Wi-Fi scan reading every 4 seconds approximately, but without an exact position where this scan was taken. Using the inertial sensor data, the approximate position can be inferred by using the semi-supervised Variational Auto-Encoder (VAE) [22].

e) *Magnetic field based localization:* This component for indoor localization uses magnetic field data captured by the mobile phone. In indoor environment, magnetic anomalies are created by different ferromagnetic objects. To benefit from their presence, the state of the art landmark-based classification [23] is extended. Once the magnetometer captures changes of the Earth's magnetic field due to indoor magnetic anomalies, they are transformed in multi-variate times series. Temporal patterns are then converted in visual ones by using 1D convolutions with Recurrent plots, Gramian Angular Fields and Markov Transition Fields (see Fig. 8). This represents magnetic field data as image sequences and permits to deploy convolutional layers to associate magnetic patterns with particular places. A deep regression is trained on the user's position and combined convolutional and recurrent layers in the deep network [24].

f) *Deep PDR:* PDR is processed by applying deep learning. Acceleration and orientation sensor data streams are pre-processed and represented as 2D images analogously to the magnetic field data. CNN and RNN are used to extract underlying hidden correlations between different sensors and modalities to learn a model of user local displacement. This allows coping with sensor noise and replaces the manual feature extraction which is frequently a subject to data noise and sophisticated thresholding, including tuning to different pedestrian profiles, depending on gender, age, height etc. The deep PDR model is learned to predict relative (x, y) displacements. The relative displacement model is trained using the regression loss on available annotated data.

The deep PDR model is locally accurate but accumulates errors over time. This PDR drift is compensated by using global localization components based on Wi-Fi and magnetic field based localization.

g) *Landmarks and pseudo labels:* CNN/Deep Neural Network (DNN) models require large-scale training data. However, genuine ground truth annotations are sparse and available for a limited number of landmarks. On the other hand, raw sensor data are massively generated at a high rate. So, sensor data is annotated with pseudo labels and a large annotated set

for training CNN/DNNs is generated. Pseudo labeling is based on simpler tasks of user walking and landmark detection and an interpolation of user's behaviour between the landmarks using the first order approximation PDR.

h) *Prediction fusion and map projection:* Relative predictions provided by deep PDR and absolute predictions provided by Wi-Fi and magnetic field data are combined using an Extended Kalman Filter (EKF). The output of the filter is then fine-tuned, by projecting it on the paths that were traversed while collecting training and validation sets, to make sure that the final result lies within the navigation space in the building.

5) *Team UMinho:* The UMinho team approach for the 2020 competition (Fig. 9) was based on a Particle Filter (PF) to fuse Wi-Fi fingerprinting positioning with motion, heading and atmospheric pressure data. In a calibration or initialization phase, the positioning system is prepared by creating a space model (floor plan) and a Wi-Fi radio map created using the training data. The radio map and the floor plan are then used in the Validation and Evaluation phases by the PF to estimate the trajectory using PDR obtained from motion and heading data.

a) *Creating the Wi-Fi Radio Map:* The Wi-Fi radio map was created using the provided Training data sets. To obtain a higher quality radio map, correction techniques are applied to the trajectories obtained through PDR (module 2). The correction approach uses the ground truth points included in the Training data sets (POSI) to correct the distance and heading for each training trajectory. The description of the training data sets specifies that the user travels along a straight path between two consecutive POSIs, making it possible to estimate the travel distance error and the heading error between two POSIs. The segment between the two POSIs is then corrected proportionally by re-doing the PDR between the two POSIs, adjusting each step length and heading value so that the estimated position at the second POSI matches the ground truth. This approach works for the Training trajectories, except the ones including stairs. The corrected trajectories from module 2 are fused with Wi-Fi, pressure and POSI data in module 3 to estimate the floor and (x, y) position for each Wi-Fi sample in the Training data set. The radio map is obtained from this process.

b) *Floor plan integration:* The floor plan is integrated by processing the provided bitmaps or vector images for each floor. Some processing is required to incorporate the floor plan into the PF (module 1). The first step is to remove some elements that may prevent the particles from moving freely
<figure><img src="image_11.png" /><figcaption>Fig. 9. Overview of the UMinho team approach.</figcaption></figure>

between the existing spaces, such as doors represented in the floor plan. Then the floor plan is converted to binary format.

c) *Estimating the final trajectory:* The final trajectory is estimated by processing the Evaluation data set which is provided with the same types of data available in the Training data sets, except for the ground truth POSIs. Module 4 is responsible for estimating the displacement using the accelerometer measurements. The movement displacement is assessed by an algorithm that estimates the user's steps and corresponding length (algorithm also applied in module 3). The displacement and heading information for the Evaluation trajectory are obtained from module 4, necessary to perform PDR, which is integrated into the PF (module 6). Wi-Fi and pressure samples are combined in module 5 to estimate the floor and the z coordinate for each Wi-Fi sample. The radio map, created in the initialization phase, is used to perform Wi-Fi fingerprinting and improve the floor estimation. These enhanced Wi-Fi samples are then fused with the displacement and heading samples in the PF. The PF (module 6), based on the solution presented in [25], performs sensor fusion of Wi-Fi fingerprinting with PDR (displacement and heading). Particles are created around the initial position, which is estimated using Wi-Fi fingerprinting. Particles states follow a PDR motion model considering noise in the heading and displacement. Particles' weights are updated based on Wi-Fi fingerprinting, using a distance function to convert the distance between the particle and the Wi-Fi position estimate into a weight. Higher weights are assigned to particles closer to the Wi-Fi position estimate. To reduce errors from Wi-Fi fingerprinting, a partial radio map is used, considering only Wi-Fi samples that are in the neighbourhood of the PF estimated position. Particles with lower weights, including those that hit walls or obstacles, are resampled based on the multinomial resampling method. The floor changes provided by module 5 allow the PF to adjust the motion model when a user changes floors, reducing the step length. The PF also performs adjustments when a floor transition is detected to ensure that all particles are moved into

the current floor. The final pose is obtained from the particles' positions and headings weighted average.

6) *Team Imec-WAVES:* imec-WAVES team positioning system consists of 4 parts: Pedestrian Dead Reckoning, RSS fingerprinting, floor (transition) detection and PF. Fig. 10 shows a flowchart of the system. These steps are introduced in detail in the following paragraphs.

a) *Pedestrian dead reckoning:* The PDR algorithm fuses the data of accelerometers, gyroscopes and magnetometers to estimate the trajectory. It consists of step detection, heading estimation and step length estimation. Step detection and heading estimation are based on [26]. For step length estimation, an adaptive Weinberg model is used [27]. The phone carrying mode is determined for each step by a KNN-classifier. The features used are average and variance of both roll and pitch from the AHRS data during one step. The competition training data include one carrying mode: holding the smartphone in front of the body. However, the competition introduction document mentions realistic movements (e.g. phone call). Therefore, additional training data was created by the team where an actor walked for several minutes while pretending to make a phone call. If a phone call is detected in the evaluation data, the heading is flipped 180°.

b) *RSS fingerprinting:* The radiomap is constructed by interpolating between the known coordinates in the training data [1], [2]. The Euclidean distance metric is used to match RSS vectors in the validation/evaluation data with RSS vectors in the radio map. The metric can only be applied to a subset of RSS values for each vector, depending on the APs the vectors have in common. A penalty is added to the Euclidean distance for each AP that is not in the subset. This prevents a (false) good match when the RSS vectors have only a few APs in common but with similar RSS values. The weighted centroid of the three best matches is selected as the estimated position.

c) *Floor (transition) detection:* The barometer is used to estimate height changes. The average pressure during each detected step is converted to a height difference relative to
<figure><img src="image_12.png" /><figcaption>Fig. 10. System flowchart of imec-WAVES team.</figcaption></figure>

the first step [28]. If the absolute height change is more than 1.5 m within a window of 15 consecutive steps, these steps are labeled as ‘on stairs’. However, the pressure changes with time regardless of the actual height difference. Therefore, if the time between two steps is more than 20 s, the height difference between these steps is removed to prevent false stair detections when standing still for a long time. The height difference during each sequence of ‘on stairs’ labels is then used to estimate the floor difference. The middle step of each ‘on stairs’ sequence is labeled as ‘transition’. The barometer method can accurately detect transitions, but can have errors in the estimated floor differences because the true floor height is unknown. The visited floor itself can be determined by matching RSS measurements with the radiomap, but sometimes the wrong floor is matched for a short time interval. The Viterbi algorithm, which was used for the previous competition to perform localization [29], is now used to find the most likely sequence of visited floors by fusing these RSS matching and barometer methods.

d) Particle filtering: A PF is used to fuse the output of previous parts and to perform map matching. To enable the latter, the provided floor plan images are converted to XML files containing the locations of each wall, staircase and elevator. This is done with the WHIPP tool [30], [31]. The locations of bookshelves in the evaluation environment are deduced from the training data and regarded as walls. The implemented PF is the Backtracking Particle Filter (BPF) [32]. During each iteration, the BPF uses new information to update current and previous states. At initialization, thousands of particles are generated uniformly over the floor plan. The amount of particles is drastically reduced during the next iterations when the filter starts to converge. The length and heading of the detected steps are used to propagate the particles. Artificial Gaussian noise added to the heading depends on the detected carrying mode. The floor plan is used to remove all particles that crossed a wall during propagation, as this is physically not possible. If RSS measurements are available for the current step, the position is estimated and a Gaussian curve is used to weigh the particles based on their distance to the estimated position. If the step is labeled as ‘on stairs’, the weights of particles outside of staircases are decreased. If the step is

<figure><img src="image_13.png" /><figcaption>Fig. 11. The membership functions used by the YAI team system for the received RSS values.</figcaption></figure>

labeled as ‘transition’, the next floor plan and radi are loaded.

7) Team YAI: One of the most popular positioning methods for indoor positioning is the fingerprinting method. However, the accuracy of positioning results suffers from the definition of distance among the Received Signal Strength (RSS) values and the fingerprinting table. Because the unit of the received Wi-Fi RSS value is dBm, the similarity calculation is prone to errors when performing fingerprinting positioning. YAI team proposes a fuzzy-based pre-processing method so that the RSS entries in the fingerprinting database can be converted into the corresponding defuzzification values. In the following paragraphs the system is introduced in detail.

a) Fuzzy-based pre-processing: The membership function used is the bell-shape membership function, which could be expressed as follows:

$$ \mu(x) = \left( 1 + \left| \frac{x-c}{a} \right|^{2b} \right)^{-1} \quad (5) $$

where the parameters a, b, and c would affect the width and slope associated with the bell-shape. Using the training data, the parameters in (5) are obtained. Fig. 11 illustrates the membership functions used in this competition.

The defuzzification method used is based on the weighted average formula as follows:

$$ y^* = \frac{\sum_{i=1}^{N} y_i \mu_i(x)}{\sum_{i=1}^{N} \mu_i(x)} \quad (6) $$
<figure><img src="image_14.png" /><figcaption>Fig. 12. The defuzzification process used by the YAI team system.</figcaption></figure>

<table><caption>TABLE IV
AN EXEMPLARY FINGERPRINT MAP WITH THE PROPOSED FUZZY-BASED PRE-PROCESSING OF YAI TEAM SYSTEM</caption><thead><tr><th colspan="6">Original fingerprint map FP<sub>i</sub> with the values of RSS in dBm</th></tr><tr><th>Record</th><th>Location</th><th>AP<sub>1</sub></th><th>AP<sub>2</sub></th><th>AP<sub>3</sub></th><th>AP<sub>4</sub></th></tr></thead><tbody><tr><td>FP<sub>1</sub></td><td>(x<sub>1</sub>, y<sub>1</sub>)</td><td>-65</td><td>-77</td><td>-73</td><td>na</td></tr><tr><td>FP<sub>2</sub></td><td>(x<sub>2</sub>, y<sub>2</sub>)</td><td>-76</td><td>na</td><td>-82</td><td>na</td></tr><tr><td>FP<sub>3</sub></td><td>(x<sub>3</sub>, y<sub>3</sub>)</td><td>-72</td><td>-64</td><td>na</td><td>-88</td></tr><tr><td>FP<sub>4</sub></td><td>(x<sub>4</sub>, y<sub>4</sub>)</td><td>-54</td><td>-48</td><td>na</td><td>-70</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr></tbody></table><table><thead><tr><th colspan="6">Fuzzy-processed fingerprint map <span style="text-decoration: overline;">FP<sub>i</sub></span></th></tr><tr><th>Record</th><th>Location</th><th>AP<sub>1</sub></th><th>AP<sub>2</sub></th><th>AP<sub>3</sub></th><th>AP<sub>4</sub></th></tr></thead><tbody><tr><td><span style="text-decoration: overline;">FP<sub>1</sub></span></td><td>(x<sub>1</sub>, y<sub>1</sub>)</td><td>2</td><td>1.13</td><td>2</td><td>0</td></tr><tr><td><span style="text-decoration: overline;">FP<sub>2</sub></span></td><td>(x<sub>2</sub>, y<sub>2</sub>)</td><td>1.31</td><td>0</td><td>0.61</td><td>0</td></tr><tr><td><span style="text-decoration: overline;">FP<sub>3</sub></span></td><td>(x<sub>3</sub>, y<sub>3</sub>)</td><td>2</td><td>2</td><td>0</td><td>0</td></tr><tr><td><span style="text-decoration: overline;">FP<sub>4</sub></span></td><td>(x<sub>4</sub>, y<sub>4</sub>)</td><td>3.07</td><td>4</td><td>0</td><td>2</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr></tbody></table>

where $x$ is the received RSS values, $\mu_i(x)$ denotes the degree of membership for the $i$-th rule, $y_i$ is the weight of the $i$-th rule, and $N = 5$ is the number of rules used. Fig. 12 shows the mapping for the defuzzification process used. Using the mapping function as shown in Fig. 12, a fuzzy-processed fingerprint map is obtained. An example to explain the main idea is illustrated in Table IV. Note that “na” denotes the RSS value and it is below the receiving sensitivity of a Wi-Fi module on a smartphone. In this case, the demapping function maps “na” to 0, which means the RSS value is very weak. Then this fuzzy-processed fingerprint map can be leveraged to get the location with the conventional localization method.

b) **Simulation results:** To evaluate the performance of the proposed positioning algorithm the competition files were used. Using the testing data, the resulting Cumulative Distribution Function (CDF) curves (see Fig. 13) of the errors in positioning are obtained. It can be seen that when locating using only Wi-Fi, the third quartile of the positioning errors are 3.539 m. After improving the fuzzy-based pre-processing, the value could further drop to 2.189 m, thus significantly reducing the positioning errors.

c) **Team Indora:** The positioning method of Indora team system is designed for the smartphone users in the scenario with known map floor plans, smartphones equipped with sensors, and with no additional infrastructure installed in the building, which is in accord with the competition Track rules. The

<figure><img src="image_15.png" /><figcaption>Fig. 13. The resulting CDF of the positioning errors with respect to the testing data set with the YAI's system.</figcaption></figure>

main research focus is on the Bayesian filtering component, especially the comparison of grid-based approaches with the particle filter present in various solutions including systems introduced by other competitors.

a) **System description:** The proposed positioning system [33] consists of multiple components merged together using the Bayesian filtering. A floor transition is detected using barometer measurements. However, these transitions were also identified using the Wi-Fi fingerprinting method as the incorrect floor detection has significant effect on the overall performance in the competition. The Bayes filter calculation corresponds to the movement on a single floor. The filter probabilistically estimates a current state which is defined as the 2D position on the selected floor. A calculation (consisting of the transition phase and the evaluation phase) is triggered when a step is detected. The transition phase is performed using PDR, i.e., the step direction is obtained from inertial sensors and together with the expected step length introduces a new estimation calculated from the prior position. Noisy sensor measurements, incorrect step length model, and other aspects are resolved by the filter which increases the uncertainty of the current estimation. The uncertainty is reduced during the evaluation phase utilizing the map. The map model (based on the tessellation) is generated from annotated floor plans using a custom tool. This framework was applied on the IPIN 2018 [1] and IPIN 2019 [2] competitions using the centroid grid filter as the Bayesian filtering implementation. Another method for the positioning was derived from the existing system by replacing the grid filter with the particle filter, improved step length calculation, and using Wi-Fi fingerprinting as an additional method for the evaluation phase of the filtering. The position is chosen among particles (or grid cells in the former approach) with the maximum belief value, i.e., the position within the building with the highest assigned probability.

b) **Competition strategy:** The proposed framework is designed for the real-time positioning. This aspect was taken into account during the off-site competition. Every trial was simulated on the smartphone in the same manner as for the on-site localization. However, the input sensor file was split for the processing convenience to individual files with consecutive steps on a single floor, i.e., the floor transition detection method was performed on raw sensor measurements and all parts were processed separately.
First attempt was computed using the particle filter and the Wi-Fi fingerprinting method. Second attempt consists of the method output using the centroid grid filter. The main focus was on the parameter configuration in this submission. The centroid grid filter was applied on IPIN 2018 [1] and IPIN 2019 [2] competitions. In [33], the method was compared with other grid-based approaches and the recommended configuration was discussed. Multiple parameter settings were explored based on the previous observations leading to the selection of the final competition submission. Third result was obtained as a combination of both approaches. The particular method was chosen for each floor individually according to the author's consideration based on visualized trajectories.

c) Results analysis: The third attempt was a combination of both applied methods. The centroid grid filter was selected on a segment corresponding to the second floor and the particle filter approach was chosen for all other floors. However, the achieved result did not outperform other attempts. Official third quartiles of errors are 6.85 m (particle filter), 8.39 m (grid filter), and 7.02 m (mixed). The analysis with known ground truth positions revealed the incorrect time shift of the first segment on the third floor. Corrected positions obtained the third quartile of errors 3.86 m, 4.46 m, and 3.89 m. For better understanding of the system performance, the first approach was replayed without the Wi-Fi fingerprinting method. The method with the particle filter component resulted in 4.38 m. This result is similar to the centroid grid filter (4.49 m). These two systems differs only in the applied Bayesian filtering implementation. The results supported former observations as the grid approach is more stable in the prediction (2.7 m mean of errors) and the particle filter provides a lightweight approach for the computation but it requires additional approach to reduce outliers (4.3 m mean of errors). The Wi-Fi fingerprinting erased large errors on some checkpoints, e.g., seven consecutive positions with errors above 14 m including three positions above 24 m were corrected with the Wi-Fi to reasonable values with the maximum 7.2 m and minimum under 1 m.

9) Team TJU: Fig. 14 shows the block diagram of the approach proposed by TJU team. It consists of four stages: 1) the PDR, 2) magnetic fingerprinting, 3) the floor recognition and 4) the trajectory fusion. The proposed approach uses the magnetic fingerprinting and PDR to separately generate the trajectory of the device, and fuses the two estimated trajectories to produce precise trajectory by Kalman Filter (KF). In addition, the current floor is detected relying on Wi-Fi database for each floor, and the transitions between floors depends on air pressure measurements. The details of the four stages are described in the following paragraphs.

a) PDR backbone: The PDR backbone is implemented following two steps:

* step detection using peak detection method;

* adaptive position updating strategies according to peak values:

$$ \left\{ \begin{array}{l} \mathbf{r}_p = \mathbf{r}_{p-1}, \\ \mathbf{r}_p = \mathbf{r}_{p-1} + SL[\cos\psi \quad \sin\psi], \quad \|\mathbf{f}_p\| \le \gamma \\ \|\mathbf{f}_p\| > \gamma \end{array} \right. \qquad (7) $$

<figure><img src="image_16.png" /><figcaption>Fig. 14. Flowchart of the heading estimation algorithm proposed by TJU team system.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Fig. 15. Flowchart of the heading estimation algorithm proposed by TJU team.</figcaption></figure>

where **r** refers to 2D position; *p* is the timestamp of a specific force norm peak; *SL* refers to a constant step length; *ψ* refers to estimated heading; $\|\mathbf{f}_p\|$ refers to the peak value of specific force norm; and *γ* is the threshold to recognize still state and motion.

Generally, the orientation of the device is estimated using accelerometer and gyroscope readings. The magnetometer was rarely used indoors due to the distortion of the magnetic field. In spite of the drawbacks of indoor magnetic data, filtered magnetic field can still enhance the orientation estimation [34]. Inspired by TJU team previous work [35], the original Madgwick algorithm [36] is improved to achieve high-accuracy and robust heading estimation. Fig. 15 shows the flowchart of the proposed heading estimation algorithm. When the agent is still, its attitude is frozen to avoid introducing errors. Otherwise, MadgwickAHRS algorithms based on different data streams are triggered according to quality of magnetic fields. The improved method is immune to magnetic disturbance but long-term motion in magnetically disturbed environment may degrade the performance of the improved heading method.

b) Magnetic fingerprinting: Magnetic fingerprinting can be formulated as a classification problem. Namely, a classification model can be trained using magnetic features labeled by reference position indexes. Instead of using magnetometer readings $\mathbf{m}^b$ as an observation directly, they are transformed into magnetic vector $\mathbf{m}^n = \{X, Y, Z\}$ under navigation frame (n-frame). Since X and Y are changed with directions, horizontal intensity, $H = \sqrt{X^2 + Y^2}$, vertical intensity, Z, and
total intensity, *F*, are used as features to train a machine
learning-based model. The benefit is that *H*, *Z* and *F* are
immune to sensor orientations and there is no need to collect
the data of all directions at a reference point.

During data collection, a participant stays at a reference
point for a short while or walks slowly through a reference
point. Therefore, the data in 0.5 s before and after the reference
point timestamp is considered as that at the point, and extract
all data for entire training data set. As a consequence, a raw
data set labeled by reference point indexes is built. To enhance
the data set, a one-element sliding window with a size of *N* is
used to extract the data of each raw 1 s fragment for each ref-
erence point. Then, the extracted data in the sample window is
transformed into the feature {Hᵢᵢ+N₋₁, Zᵢᵢ+N₋₁, Fᵢᵢ+N₋₁}.
Finally, several machine learning-based models are trained,
such as KNN, SVM, Naive Bayes and ensemble models. After
10-fold cross validation, it can be found that the 1-NN model
achieves the highest F1-score of 0.9949.

c) Floor recognition: In this stage, the training data are used to create a radio map for each floor using received signal strength (RSS) measurements. RSS values in existing MAC address list are used to train a random forest model to determine the floor ID. However, since the Wi-Fi data is collected with a very low frequency of approximately 0.25 Hz, it cannot provide enough time resolution to determine the precise transitions between floors. Therefore, barometer data with a higher frequency of 5 Hz is used to detect the floor transitions. A mean filter was used to smooth the barometer data before calculating the data difference in successive timestamps. Then, the start and end of the transition between floors can be clearly identified. Wi-Fi data assisted with barometer data can estimate the vertical trajectory of a user well.

d) Fusion using Kalman filter: The PDR system outputs high-frequency 2D positions, while reference points with a lower frequency are recognized by magnetic fingerprinting model. A Kalman filter acts as a bridge to relate two systems. The filter outputs the corrected path using the relative estimations from the PDR model (time update) and absolute position estimations from the magnetic positioning system (measurement update).

10) Team Next-Newbie Reckoners (NNReckoners): NNReckoners team's method focused on two main parts: position prediction using Random Forest prediction model and IMU position estimate using Wi-Fi propagation and PF (Fig. 16). In other words, the team approached the competition data set with 2 challenges in mind: 1) increasing the volume of Wi-Fi data to train the prediction model, and 2) leverage the IMU to improve position estimate by Wi-Fi.

In order to build the RSS data set to train the prediction
model, enough data is required. Hence, data augmentation
was adopted to improve the model performance. With the data
extrapolation, modifying the RSS at detected AP by increasing
the signal strength by 5, the original Wi-Fi data set was
enlarged by 30 times [37].

Complementing the Wi-Fi prediction, PDR was applied by
incorporating the step count, stride length and orientation
calculated from the IMU sensor. The idea of Wi-Fi propagation

<figure><img src="image_18.png" /><figcaption>Fig. 16. NNReckoners team system overview.</figcaption></figure>

was to trust the Wi-Fi prediction and recursively estimate
the position using the step counts and bearings obtained in
between.

Additionally, a particle filter was used to identify the best possible route taken with respect to the derived PDR algorithm. A set of particles was first distributed within the bounds of the map as provided by the competition organisers. Subsequently, a simulation is started from the starting timestamp to the ending timestamp where each particle was moved in accordance to the step data and bearing obtained from the PDR algorithm. Each particle contained a weight and a history of coordinates. If a Wi-Fi position estimate is available, the weight of particles within a radius from the estimated position and its estimated distance error is further increased. It was set due to the fact that Wi-Fi prediction was observed to produce shorter distance errors than PDR. At the end of the simulation, the particles with the highest weight are selected, and the final route is determined by looping through and averaging the history of coordinates for all selected particles.

V. TRACK 4: FOOT-MOUNTED IMU-BASED POSITIONING

A. Track Description

Track 4 was dedicated to foot-mounted inertial and GNSS
navigation in an *off-site* context. Data were collected with the
PEdestrian Reference SYstem (PERSY) sensor (see Table VI)
developed by the GEOLOC team at University Gustave Eif-
fel. Track chairs collected the data by walking through the
competition area over a 1.2 km walk path spanning four
different floors, using lifts, escalators and travelators. Also a
few outdoor parts were included as shown in Fig. 17, as well
as some breaks of various duration. Track 4 followed the
same data collection strategies as the off-site competitions
organised in previous years [1], [2]. In contrast with all the
other Tracks, where competitors were provided with a detailed
map beforehand and could make use of that information,
competitors in Track 4 could not use of any map information.

Two data sets were given to competitors. *Data set num 1*
was taken on a single static location for several hours, and
was meant to be used for sensor calibration, by enabling
competitors to compute noise and measurement bias of inertial
sensors (Allan variance). *Data set num 2* was the data recorded
on the Atlantis shopping mall area following 6 different steps,
as shown in *Table V* and in *Fig. 18*.
<figure><img src="image_19.png" /><figcaption>Fig. 17. PERSY and description of Track 4 over Atlantis shopping mall.</figcaption></figure>

<table><caption>TABLE V
STEPS DESCRIPTION COMPOSING Data Set Num 2</caption><thead><tr><td>Step</td><td>Duration</td><td>Description</td></tr></thead><tbody><tr><td>Step1</td><td>10 s</td><td>hand-held static phase</td></tr><tr><td>Step2</td><td>60 s</td><td>magnetometer calibration</td></tr><tr><td>Step3</td><td>10 s</td><td>hand-held static phase</td></tr><tr><td>Step4</td><td>≈2 min</td><td>PERSY setup on the foot</td></tr><tr><td>Step5</td><td>60 s</td><td>static phase with PERSY on the foot</td></tr><tr><td>Step6</td><td>≈30 min</td><td>evaluation Track including key points from 1 to 68</td></tr></tbody></table>

<figure><img src="image_20.png" /><figcaption>Fig. 18. Temporal view of steps composing Data set num 2.</figcaption></figure>

The competitors' objective was to re-build the trajectory realised by the Track chairs. The evaluation was done by comparing 2D position and floor level estimated by each team to the coordinates of 67 reference points (key points). To do so, a Table containing timestamps of expected key points was shared, and competitors had to provide the corresponding coordinates.

<table><caption>TABLE VI
INFORMATION ABOUT EMBEDDED SENSORS INSIDE PERSY</caption><thead><tr><td>Sensor</td><td>Model & Manufacturer</td><td>Sampling Freq. (Hz)</td></tr></thead><tbody><tr><td>Accelerometer</td><td>STIM300 - Sensoror</td><td>160</td></tr><tr><td>Gyroscope</td><td>STIM300 - Sensoror</td><td>160</td></tr><tr><td>Magnetometer</td><td>HMC5983 - Honeywell</td><td>160</td></tr><tr><td>GNSS</td><td>NEO-M8T - Ublox</td><td>5</td></tr></tbody></table>

Data Set and supplementary materials –e.g. data sheet of sensors embedded in PERSY– were provided to competitors of Track 4. These contents and the ground truth location for evaluation are now available for further benchmarking in [38]. This package complements the ones from the previous editions [39] and [40].

## B. Competition Area

For IPIN 2020, due to Covid-19 situation, Track4 competition was held in “Atlantis Le centre”, a large shopping mall close to Nantes - France. This site has already been used in IPIN 2018 for all competition Tracks, and a very accurate survey was realised. This has eased the design of the ground truth (see [1] for details on the survey). There were multiple difficulties when surveying such a big shopping mall: wide areas, lifts, escalators, and even a carousel, as illustrated in Fig. 19. Complexity related to the Covid-19 also led the Track chairs to make loops on the path in order to respect the direction of travel, as shown in Figure 20.

## C. Indoor Positioning Solutions Provided by Competitors

1) Team WHUGNSS: The classic zero-velocity update algorithm (Zero-velocity update (ZUPT)) based foot-mounted pedestrian dead reckoning consists of a strap-down inertial navigation algorithm, a stance phase detection algorithm, and an error state Kalman filter. However, the classic ZUPT-based Foot-PDR [41], [42] cannot overcome the influence of the complex motion of the pedestrian. The WHU-GNSS team system is based on several schemes designed to improve navigation performance, as shown in Fig. 21.

The core algorithm is the strap-down inertial navigation algorithm. On this basis, a zero-speed detection method with adaptive threshold setting is used to adapt to different users. Next, the motion pattern recognition algorithm is used to distinguish whether the user is walking normally or taking the escalator and elevator, and uses constant speed, Zero-velocity update (ZUPT), Zero angular rate update (ZARU), Improved heuristic drift elimination (iHDE), linear trajectory and height constraints to improve the position estimation accuracy according to the discrimination results. In addition, the magnetic field will be used to detect whether the user has returned to the place where they have walked, so as to correct the current navigation state with the historical estimated position. And when the user comes to an outdoor scene, the GNSS signal will be used to improve the final positioning performance.

a) The multi-constraint algorithms: The classic Generalized Likelihood Ratio Test (GLRT) method is one of the most common algorithms for detecting the stance phase [43], [44].
<figure><img src="image_21.png" /><figcaption>Fig. 19. Track4 difficulties over the path.</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>Fig. 20. Left part: imposed direction of travel. Right part: multi-floor environment.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>Fig. 21. Block diagram of multi-constraints-based foot-mounted PDR algorithm of WHU-GNSS team system.</figcaption></figure>

In the WHU-GNSS team system, an improved adaptive threshold is used instead of the fixed threshold method to detect the stance-phase in each gait cycle. The adaptive threshold method is adaptable to different gait frequencies in dynamic motion. Once the stance phase is detected, a zero velocity vector is used to estimate and correct the navigation error [41].

The heading angle error and the z-axis gyroscope bias of the ZUPT algorithm are unobservable. Thus, the following methods are used to constrain the error divergence of the heading angle. First, the Zero angular rate update (ZARU) algorithm is employed to estimate the gyroscope bias and heading angle error [45]. Compared with the stance phase detection algorithm, a stricter fixed threshold is used and a more extended continuous period to determine the update chance of ZARU. Second, when a pedestrian is determined to be walking in a straight-line path or the corridor's primary

orientation, the Improved heuristic drift elimination (iHDE) and the straight-line constraint algorithms are applied to estimate the heading angle and the z-axis gyroscope bias [46], [47]. These algorithms can effectively improve the performance and reliability of pedestrian navigation.

The height error divergence is also a significant problem in Foot-PDR, especially for multi-floor navigation and positioning applications. In the absence of a barometer, an effective height constraint algorithm is adopted to reduce the error drift along the vertical channel. When pedestrians go up and downstairs, the slope angle can be considered constant in most cases [48]. In the WHU-GNSS solution, the stride length and the slope angle between adjacent footsteps are used to determine whether the pedestrian is walking on a plane or going up and downstairs. Then the slope-based or plane-based height constraint algorithm is used to improve the estimated height accuracy in Foot-PDR.

The other extreme scenario is the escalator or lift. Usually, escalators run at a constant speed. When a pedestrian stands relatively static on the escalator, the specific forces measured by the foot-mounted IMU are almost all derived from local gravity. The gravity information can be fused in a tightly coupled manner in the WHU-GNSS solution, so the drifting error can be constrained even when a pedestrian stands still on an escalator. Moreover, when the pedestrian takes a lift, the specific forces (i.e., the accelerations) will exhibit clear acceleration motion and deceleration motion process. The vertical (up or down) velocity information of the pedestrian can be estimated using acceleration and deceleration motions. Thus, the vertical velocity can be as observation information to improve the performance and stability of the Foot-PDR.

Many ferromagnetic materials exist in indoor building structures. So, magnetometers cannot be used to determine the heading angle in Foot-PDR directly. Yet, combined with a rough position, the magnetic field signals can recognize similar areas when the pedestrians return to places they have walked before. This meaningful information can help improve the robustness of Foot-PDR in practical application.

The Foot-PDR is integrated with GNSS signals in a loosely-coupled manner [42]. Satellites with small elevations should be discarded to avoid the gross error as much as possible. Besides, some measurements with low quality judged
<figure><img src="image_24.png" /><figcaption>Fig. 22. The scheme of foot-mounted PDR system based on multi-constraint algorithms proposed by AIR team.</figcaption></figure>

by the innovation vector's magnitude and covariance need to be rejected in the Kalman Filter (KF). Furthermore, the adaptively robust filtering algorithm is used to control the effects of inaccurate measurements in the WHU-GNSS solution and improve system accuracy.

The optimal inertial sensor parameters (i.e., the bias instability of gyroscopes and accelerometers, the angular random walk, and the velocity random walk) are determined through the provided long-term static data. The magnetometer is also calibrated through the classic ellipsoid fitting method.

2) Team AIR: The pedestrian foot-mounted PDR system proposed by AIR team is shown in Fig. 22.

In the above framework, five constraint algorithms are included in the middle modules: Stance & Still Phase Detection, the Heuristic Drift Elimination (HDE), the Height Update Algorithm (HUPT), the Zero-velocity update (ZUPT), and the Earth Magnetic Yaw. Meanwhile, the Stance & Still Phase Detection includes two components: the Generalized Likelihood Ratio Test (GLRT) detector algorithm used under the condition of the slow and normal pedestrian gait speed, and the Hidden Markov Model (HMM) detector algorithm used under the condition of the dynamic and fast pedestrian gait speed. After that, using the improved HDE and HUPT method to estimate current position errors, ZUPT is used to estimate the velocity error, while Earth Magnetic Yaw based on quasi-Static Magnetic Field (QSMF) method is used to estimate the heading error.

a) The multi-constraint algorithms: A gait or a walk cycle consists of two phases: the swing and stance phase. In the swing phase, the foot is not in contact with the ground. In contrast, the foot contacts the ground in the stance phase. GLRT algorithm has obvious advantages for zero speed detection of stable pedestrian gait velocity, while HMM algorithm has a good effect for zero speed detection of dynamic and fast pedestrian gait speed. Thus, the two methods are combined to achieve the dynamic human stance & still phase detection [49].

The Real Pedestrian Indoor Trajectory (Walking Straight)

<figure><img src="image_25.png" /><figcaption>Fig. 23. Revise the current step's inertial recursive position with the position calculated from the stride heading in the AIR team system.</figcaption></figure>

When the Stance & Still Phase Detection detects the stance and swing phases of human foot gait from the data from IMUs, ZUPT method is used to constraint the velocity divergence [50].

HDE algorithm is a very useful method to constraint the system's heading drift, if the indoor reference heading can be known in advance. In the AIR team method, the initial heading is used to calculate several possible reference directions of pedestrian walking [48]. Then, unlike the existing HDE method, which mainly corrects inertia recursive heading, the closest reference direction is used to calculate the estimate position at the current footstep, then uses the position error between the estimate position and the inertia recursive position to restrain the position divergence. The procedure is shown in Fig. 23.

Height divergence is a major problem in Inertial Navigation System (INS)-based foot-mounted PDR system in multi-story positioning. If a pedestrian is walking on a plane, the slope of the current stride is approximately zero degree, if that, keep the height always unchanged. While walking on a staircase, the method proposed uses the actual slope of the stairs (usually 20 45 degrees) to calculate the height change of the current stride, which can be used to constrain the height divergence of the current stride [48]. If pedestrian is on an elevator or escalator, it mainly can be effectively determined by analyzing the characteristics of acceleration, especially the acceleration in the vertical direction.

The magnetic field is very useful to estimate the heading of the system, but the magnetic disturbance has a severely effect on the estimation. In AIR team system, an improved QSMF method combined with a compass filter is used to estimate the heading in the perturbed magnetic field [51]. In addition, in areas where pedestrians repeatedly walk, a series of magnetic sequence information is used for pedestrian trajectory matching to improve the effect of heading constraint.

3) Team Free-Walking: The positioning system proposed by team Free-Walking is shown in Fig. 24.

The Free-Walking system combines data pre-processing, motion mode recognition, INS mechanization, adaptive zero velocity detection, ZUPT-aided Kalman Filter (KF) and altitude constraint. The data pre-processing includes sensor calibration, filtering and Coordinate system transformation. After
<figure><img src="image_26.png" /><figcaption>Fig. 24. System architecture of proposed pedestrian inertial navigation based on motion mode recognition proposed by Free-Walking team.</figcaption></figure>

pre-processing, motion mode recognition algorithms are used to help adaptive threshold ZUPT detection. Then a ZUPT-based KF is used to get position information. Meanwhile, motion mode results is also used to constraint height error.

a) *Error-constraint method based on walking mode:* For pedestrian positioning, the human motion modes describe the overall movement of pedestrians. The pedestrian motion modes are particularly important for pedestrian navigation, while the pedestrian motion modes are variable during the procedure of pedestrian navigation. Therefore, a walking mode classifier is designed (see Fig. 25) based on the stacked denoising autoencoder [52] and temporal Convolutional neural network (CNN) with attention to recognize eight pedestrian motion modes [53], [54].

ZUPT-aided INS has ability to suppress navigation errors. Free-Walking team uses the periodic gait-cycle window to divide the pedestrian movement into discrete gait cycles; then, the minimum value in each gait cycle is taken as the zero-speed state point. The time length of the gait cycle is different under different motions. The gait-cycle duration is adaptively adjusted based on the classification result of walking mode to adapt to various pedestrian motions [55]. Compared to the existing methods, the proposed method does not need to set the zero-speed detection threshold, and performs well for zero-speed interval detection under various pedestrian movements. The stationary state of the foot during the stance phase is taken and feeds the zero-velocity information (pseudo-measurement) into KF to compensate for the velocity, the position and the attitude errors.

The height errors in Strapdown Inertial Navigation System (SINS) solution will grow without boundary and cannot be eliminated by ZUPT measurements. When a user walks on the same floor, the altitude does not change. The altitude changes only when the user goes up and down stairs. Therefore, the vertical displacement of pedestrian is constrained by two

factors: stair height and motion mode. If the height of each stair in a multi-floor building is fixed, the height of each gait cycle is determined by the number of walking stairs in that gait cycle. Therefore, the classification result of walking mode is used to constrain the height error.

4) *Team BHSNIP:* The Pedestrian Navigation System (PNS) based on Inertial navigation system–extended Kalman filter–zero velocity update (IEZ) –also referred as INS-EKF-ZUPT– is widely used in complex environments without external infrastructure owing to its characteristics of autonomy and continuity. However, due to the poor observability of heading errors to ZUPT and the instability of vertical inertial channels, further corrections of the estimated trajectories under the IEZ framework are still needed to obtain higher positioning accuracy.

In order to achieve high performance for PNS in terms of accuracy and robustness, BHSNIP team integrates the Micro-Electro-Mechanical Systems–Inertial Measurement Unit (MEMS-IMU) and Global Positioning System (GPS) as shown in Fig. 26. In this scheme, MEMS-IMU provides the 3-axis accelerometer, 3-axis magnetometer, and 3-axis gyroscope readings which are [fₓ, fᵧ, fᵧ], [magₓ, magᵧ, magz], and [ωₓ, ωᵧ, ωz] in the body frame, respectively. The main work has the following features:

1) Aiming at the weakly observability of heading drift for MEMS-IMU, the iHDE algorithm is proposed. The algorithm has the following three steps: First, heading information is extracted from pedestrian’s straight-line motion track, which is used to construct four or eight datum directions of the building; second, building heading information is utilized to estimate yaw errors of trajectories that satisfy specified rules; and third, these yaw errors are utilized as the EKF observation to estimate the state error of the navigation parameters.

2) In order to deal with the problem that the inertial vertical channel is unstable under the traditional IEZ framework, which makes it impossible to locate the floor by SINS solutions, the improved step height equidistant (ISHE) is exploited. At the beginning, the adaptive network-based fuzzy inference system (ANFIS) is used to identify different vertical modes including elevator, escalator and staircase (walking upstairs, horizontal movement, and walking downstairs). Then, the floor information or altitude is estimated by ISHE.

3) To detect the stance phase accurately, adaptive-ZUPT algorithm is used based on backward neural network. In conventional researches, positioning performance is easily affected by the ZUPT with fixed threshold, because it is difficult to determine ZUPT conditions for jump, fast walking, running.

4) GPS is fused with MEMS-IMMEMS-IMU through Robust Extended Kalman Filter (REKF), which can remove the contaminated points of GPS signal. What is more, GPS can provide global coordinates.

Fig. 27 shows the horizontal trajectory. The estimated track starts from the red circle and the blue line represents the moving trail of the pedestrian based on the proposed method. The positive direction of abscissa and longitudinal represents
<figure><img src="image_27.png" /><figcaption>Fig. 25. Pedestrian walking mode recognition based on the stacked denoising autoencoder and temporal convolutional network with attention in the Free-Walking team system.</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>Fig. 26. Scheme of BHSNIP team positioning system.</figcaption></figure>

<figure><img src="image_29.png" /><figcaption>Fig. 27. Estimated horizontal trajectory by Team BHSNIP for Track 4 of IPIN Competition 2020.</figcaption></figure>

<figure><img src="image_30.png" /><figcaption>Fig. 28. Estimated 3D trajectory by Team BHSNIP for Track 4 of IPIN Competition 2020.</figcaption></figure>

East and north respectively. The track in the figure is shown in relative coordinates that will be transformed into the WGS84 coordinate system.

Fig. 28 illustrates the three-dimensional trajectory. The estimated track also starts from the red circle and the blue line represents the moving trail of the pedestrian based on the proposed method. The x-axis, y-axis and z-axis of the

coordinate system represent east, north and up respectively. The relative coordinates representing the track in Fig. 28 will be transformed to the WGS84 coordinate system.

## VI. TRACK 5: XDR CHALLENGE IN MANUFACTURING 2020

### A. Track Description

The purpose of Track 5 is to evaluate the practical performance of indoor localisation methods under realistic industrial scenarios. Indoor localisation competitions have been held, named “PDR Challenge” or “xDR Challenge” as the official competitions or the relevant event in past IPIN conferences. Track 5 is a sequel of the PDR/xDR Challenge series competition, which is named as “xDR Challenge in Manufacturing 2020”. In this year’s competition, the competitors are asked to estimate the trajectory of employees working in the factory and forklifts driven in the factory.

As specific industrial scenarios, the target for PDR Challenge 2017 and xDR Challenge 2018 were picking operation in a warehouse [56], while for xDR Challenge 2019 it was
serving in a restaurant and manufacturing operations in a
factory. The scenario of the competition for 2020 was manu-
facturing operations in a factory. Competitors were required
to estimate operators' trajectory and forklifts' trajectory in
the factory by utilising indoor localisation methods based on
dead reckoning algorithm, positional correction methods with
Bluetooth Low Energy (BLE) beacons and other information
provided.

Characteristics of Track 5 can be summarised as follows:

1) *Utilising the Data Actually Used in the Operation:* Similar to other Tracks, Track 5 aims to compare practical performance of indoor localisation methods or systems under realistic industrial scenarios. Its most remarkable characteristic is that the data provided to competitors is obtained from by an analysing system for manufacturing operation which was used during real operation [57], after approval of provision of the data actually used.

The operators are carrying Android devices which measure
sensor data for the analysis based on indoor localisation.
This means that in Track 5 data is not provided by an actor
following a predetermined path, but by real operators doing
their daily job. This adds significant difficulty in estimating
the trajectory with respect to other Tracks, mostly because
the target movements include various types of motion during
the manufacturing operations, rather than simply walking at
constant speed and staying still for a while.

As the data set, we provided measured sensor data that
include angular velocity, acceleration, magnetism, atmospheric
pressure, and RSSI of BLE beacons. Also, partial ground
truth positions are provided for correcting the position. These
ground truth data are assumed to be available from the record
of the operations and required for long-term estimation by
indoor localisation. The lengths of the data are in units of
working hours. The lengths per data are about 2 hours to
7 hours.

2) *Evaluating Dead Reckoning Methods for Various Types of Moving Objects:* The PDR/xDR Challenge series competitions deal with indoor localisation methods based on various types of the dead reckoning methods. Dead reckoning for vehicle is called Vehicle Dead-Reckoning (VDR). The term “xDR” is used to indicate various types of dead reckoning. The target of the Track 5 is not only operators working in the factory, but also forklifts driven in the factory. Dead reckoning of the vehicle such as the forklifts is a quite challenging topic. Thus, there are two separated sub-Tracks for PDR and VDR.

3) *Multi-Faceted Evaluation of Performance for Indoor Localisation Methods:* In order to evaluate practical performance under industrial scenarios, multi-faceted evaluation metrics has been used. The evaluation metrics in the PDR/xDR Challenges has been revised. As the evaluation metrics for this year's competition, a three-evaluation indicators and three-negative check criteria were adopted as follows:
Evaluation indicators about error

* Absolute error - Circular Error (CE): absolute 2D positional error compared with ground truth position.

* Error distribution bias - Circular Accuracy (CA): evaluating degree of bias of error distribution in 2D error space.

* Error accumulation gradient (EAG): evaluating speed of error accumulation caused by relative tracking with dead reckoning.

Negative Checks

* Requirement of moving velocity: checking if local moving speeds in the trajectory are less than a defined threshold.

* Requirement of validity of trajectory: checking the incursion of the trajectory into un-walkable area.

* Coverage ratio: check if each evaluation point has corresponding submitted results.

Each evaluation indicator and criterion are converted into
evaluation indexes up to 100 and weighted summed for
calculating the integrated index which determines the winner
of the competition. We adopted median of CEs (CE50) as
an indicator of the absolute error. The error accumulation is
the one of main concerns in relative tracking method such
as xDR. In order to evaluate the error accumulation, BLE
signals in the data set have been intentionally and partially
deleted [56]. Partial ground-truth position is provided for error
correction and for evaluating the speed of error accumulation
from the correction points where the ground truth position
is provided. Competitors are required to deal with these
unique characteristics of the data set. CE75 has not been
used for determining the winner, but only for comparison
according to the EvAAL framework. However, CE75 can be
easily calculated by using our evaluation script for calculating
evaluation indicators and negative checks. Please refer to the
script shared on the GitHub for further details [58].

B. Competition Area

The target field for the PDR subtrack is shown in Fig. 29.
The target field of the VDR subtrack is shown in Fig. 30.
We provided some examples as sample data sets. In the figures,
examples of the movements of an operator and a forklift
are shown in blue dots. The yellow dots represent examples
of the partial ground truth data for correcting the positional
errors. The black coloured areas represents the un-walkable
areas. Competitors are able to avoid the incursion into the
un-walkable area by using map matching techniques. BLE
beacons are arranged in the target area for absolute localisation
and positional correction. According to the demands of the
factory for maintenance, solar-powered BLE beacons, Fujitsu's
PulsarGum, are used. The interval of signal emission is 1.26 s
at minimum, but it is not guaranteed and varies in proportion to
the amount of generated electricity. Competitors are required
to deal with this characteristic of the beacon.

C. Indoor Positioning Solutions Provided by Competitors

1) Team KawaguchiLab: Team KawaguchiLab has studied IMU-based indoor localization using smartphone. In the 2020 competition, the challenge was to integrate KawaguchiLab IMU-based research with non-IMU sensor based system (BLE, map information), and to build a robust indoor positioning system. KawaguchiLab system is simple because it makes no complex assumptions. Therefore, even in a Track 5 environment where there are few movement constraints, it works
<figure><img src="image_31.png" /><figcaption>Fig. 29. The target area of PDR subtrack in Track 5.</figcaption></figure>

<figure><img src="image_32.png" /><figcaption>Fig. 30. The target area of VDR subtrack in Track 5.</figcaption></figure>

<figure><img src="image_33.png" /><figcaption>Fig. 31. The scheme of three steps indoor localization system of KawaguchiLab team system.</figcaption></figure>

robustly, although there is a trade-off for some loss of accuracy. Fig. 31 shows overview of KawaguchiLab team system. It consists of three phases: denoising, dead reckoning, and compensation.

a) **Denoising phase:** Gyroscopes have an offset that depends on the inherent characteristics of the sensor and temperature. It causes a serious cumulative errors in dead reckoning. Hence, they are removed using real-time offset removal algorithm: First, whether the sensor is stationary or moving is obtained with an Fast Fourier Transform (FFT) based method; second, the offset by averaging the angular velocity while stationary is calculated. Finally, the angular velocity is calibrated using the latest updated offset.

b) **Dead reckoning phase:** In speed estimation, Deep Neural Network (DNN) base method is used [59]–[61]. Deep neural network architecture consist of Long Short-Term Memory (LSTM) and full-connected layer LSTM extract time series features of 3-axis acceleration by sliding window and full-connected layer converts time series features to speed. This approach gains robustness to noisy data and work with various gaits.

In heading estimation, first, gravity direction $\hat{\mathbf{g}}^{DCS}$ is estimated using Multiplicative Extended Kalman Filter (MEKF) [62]. DCS represents the device coordinate system. Secon, angular velocity $\omega^{DCS}$ is projected to gravity to get the horizontal angular velocity $\hat{\omega}_{z}^{GCS}$. GCS represents the global coordinate system. Projection process is as follows:

$$ \hat{\omega}_{z}^{GCS} = - \frac{\omega^{DCS} \cdot \hat{\mathbf{g}}^{DCS}}{\|\hat{\mathbf{g}}^{DCS}\|} \qquad (8) $$

Finally, the heading is calculated by integrating time-series horizontal angular velocity. Integration process is as follows:

$$ \hat{h} = \sum \hat{\omega}_{z}^{GCS} dt \qquad (9) $$

c) **Compensation phase:** Pseudo reference position from BLE signal is generated to compensate trajectory. BLE signals are searched using sliding window for about 10 s. Then, the distance from BLE beacon to subject is estimated using three or more BLE signals. A pseudo reference position by using these distance.

Similarity transformation model [63] is used to compensate the trajectory using the true reference position and pseudo reference position. The parameter of this model is updated using similitude ratio. The similitude ratio $s$ is calculated by using actual moving distance $d$ and estimated moving distance $d_e$.

$$ s = \frac{d}{d_e} \qquad (10) $$

The parameter $\alpha$ is updated by multiplying similitude ratio ($\alpha_0 = 1$).

$$ \alpha_k = s\alpha_{k-1} \qquad (11) $$

Finally, the $\alpha$ is multiplied to the estimated position change.

The path is generated using map image as physical constraints to avoid obstacles. The shortest path from one reference point to the next one is calculated and with astar algorithm the next reference point is searched.

2) **Team YONAYONA:** YONAYONA team indoor positioning technology is implemented in two stages: absolute position determination using BLE signals and map matching using map information. Using the acceleration and angular velocity measured by the IMU is a relative positioning approach, which often causes drifting errors. Therefore, YONAYONA system first efficiently estimates the location based on the RSSI, position, and signal strength parameters, and then corrects for the natural behavior of the person's walking speed and direction. A major challenge for this algorithm is to deal with the situation when the number of observed BLE beacons is not enough or when there is a wall between the previous predicted position and the next predicted position.
a) Absolute position determination: Since the number of BLE beacons observed is not constant, absolute positioning is calculated by selecting three beacons with high RSSI at a certain time (every 0.5 s in this implementation). The distance between the observer and the beacons can be computed by RSSI and Ptx (measured RSSI 0.1 m away from the beacon). As a result of trying various approaches to estimate the position based on the distance data, such as trilateration, position averaging, and position averaging with power value weighting, the average-weighted method, which has the least error, is applied in this algorithm.

b) Map matching: In this section, an algorithm is build to predict realistic human movement based on the map data provided by the competition organizers. In cases where a line connecting two points estimated by absolute surveying would encroach into a wall, an inaccessible point with a nearby accessible one is replaced. Then, the estimated points are connected with each other in a smooth trajectory so that the walking speed can be kept within a sensible range.

c) Problem: This algorithm relies on absolute position estimations, which makes it difficult to deal with situations where there is a large error in the value of the signal received from the beacons, or where the number of signals received is not sufficient. In this implementation, the estimation accuracy within the Absolute Localization Inapplicable Period (ALIP) time set at a specific time was reduced, resulting in a larger error. In fact, there were not enough time to build an algorithm that also implemented PDR and VDR by the competition deadline, so it is not possible to refer to relative positioning. A possible improvement to this technique is to design a robust system using the Kalman Filter (KF) from two estimates, one for absolute positioning and one for relative positioning.

# VII. TRACK 6: SMARTPHONE-BASED VEHICLE POSITIONING WITHOUT ADDITIONAL EQUIPMENT

## A. Track Description

The goal of Track 6 is to evaluate the performance of different integrated navigation solutions based on the sensors of vehicle-mounted smartphone, such as GNSS, MEMS and magnetometer, etc. A Huawei mate20 smartphone was used to record raw multi-sensor data in the vehicle scene and a reference system based on Differential Global Navigation Satellite System (DGNSS) and Fiber Optic Gyro Inertial Navigation System (FOG-INS) with an expected accuracy of 5 cm at 1 Hz provided the ground truth. Two data sets were provided. The first one containing the ground-truth reference was used for sensor and algorithm calibration. The second one was for the calculation of the coordinates and accuracy evaluation.

## B. Competition Area

The test route of Track 6 (see Fig. 32) includes an outdoor scenario with unobstructed satellite view, an attenuation scenario with partially obstructed view and an indoor scenario without satellite view. In the test process (see Fig. 33), there were several long interruptions of GNSS signal and an irregular test route was adopted. Besides the navigation measurements derived from the sensors installed in smartphone, there

<figure><img src="image_34.png" /><figcaption>Fig. 32. The test route and GNSS condition of Track 6.</figcaption></figure>

were no external aid information and no prior knowledge of the test route. The competitors could only rely on smartphone to calculate the vehicle position.

The test area of Track 6 was selected in Haidian airport and surrounding areas, Beijing. The whole test route was about 19 km and consisted of two phases: the initial alignment phase and the final evaluation phase. The initial alignment phase was carried out in an open sky scene. It can be specifically divided into the sensor calibration stage (traverse the posture states, about 3 minutes), the static initial alignment (about 5 minutes), and the dynamic alignment (several running, stop and turn around, about 15 minutes). The evaluation stage was carried out in the scene of GNSS signal obstruction and simulated interruption. It can be specifically divided into three stages:

1) frequent GNSS signal attenuation stage: obstructed buildings, tree shades, etc. – about 25 minutes;

2) simulated GNSS absent signal stage: completely interrupted, simulated by turning off the Mobile phone GNSS positioning function;

3) indoor parking stage – about 3 minutes.

Following the EvAAL evaluation criteria, the 75% horizontal positioning error of competitors output points was evaluated.

## C. Indoor Positioning Solutions Provided by Competitors

1) Team WHU&AutoNavi: WHU&Autonavi Team system uses GNSS/INS integrated positioning as the basic algorithm and focus on making full use of vehicle motion constraint information and magnetometer observations to provide stable positioning services. Fig. 34 shows the flowchart of the vehicle integrated positioning algorithm based on smartphone built-in sensors. And the algorithm can be divided into 3 parts: 1) GNSS/INS integrated positioning algorithm (the red dotted part), 2) the vehicle motion model constraints (the orange part), and 3) magnetic heading constraint (the green part).

a) GNSS/INS integrated positioning algorithm: GNSS/INS integrated positioning is the most basic and backbone algorithm in vehicle positioning scenarios. INS is used as a bridge to correlate all available observations, and GNSS, as the only available absolute positioning method in the offline mode of the smartphone, determines the positioning performance of the system.

INS mechanization is employed to integrate the gyros and accelerometer output. Due to the low performance of the
<figure><img src="image_35.png" /><figcaption>Fig. 33. The test process of Track 6.</figcaption></figure>

<figure><img src="image_36.png" /><figcaption>Fig. 34. Flowchart of the vehicle integrated positioning algorithm based on smartphone built-in sensors of the WHU&Autonavi Team system.</figcaption></figure>

smartphone built-in sensors, the influence of the angular rate and sculling effect caused by the rotation of Earth and motion speed can be ignored [64, 65]. Therefore, the rigorous INS mechanization can be simplified to achieve more efficient calculations.

An Extended Kalman Filter (EKF) is employed to fuse GNSS and INS for reducing the error caused by non-linearity. And the 20-dimensional navigation error state includes position, velocity, attitude, gyroscope bias, accelerometer bias, misalignment angle (the angular difference between the smartphone built-in sensor and the vehicle coordinate system), and the lever arm parameters (the offset of the sensor measurement center to the center of the vehicle coordinate system). To maximize the navigation performance of the sensor, the performance parameters of the gyroscope and accelerometer are adjusted according to the three sets of training data given by the competition.

For smartphones, the distance between the GNSS antenna and the IMU measurement center is very close (e.g., several centimeters), and the GNSS position accuracy in single-point positioning mode is at the meter level, so the GNSS antenna and the IMU measurement center can be considered to overlap. Besides, since the standard deviation cannot accurately determine the true positioning accuracy of the GNSS position, the chi-square test is used to eliminate the gross errors in the GNSS position to ensure the reliability of the filtering [66].

b) Vehicle motion model constraints: To deal with scenarios where GNSS signals are interfered in a complex environment, the vehicle motion constraint model is fully used to improve the relative positioning capability of the system. WHU&Autonavi system simply divides the vehicle motion state into stationary and moving by using the raw output of gyroscope and accelerometer.

Stationary state: When the vehicle is judged to be stationary, it can be considered that the speed of the vehicle is zero, that is, Zero-velocity update (ZUPT). ZUPT is an effective means to control the accumulation of velocity error. At the same time, the heading of the vehicle should remain unchanged, and all heading errors can be considered to be caused by sensor errors. The WHU&Autonavi system stores the heading angle at the initial moment of the stationary period and constructs a virtual heading angle observation value, so as to achieve the purpose of effectively controlling the accumulation of heading angle error, called Zero Integrated Heading Rate (ZIHR) [67].

Motion state: For the normal driving behavior of ordinary users, the vehicle will only move forward or backward. Based on such objective facts, it can be assumed that the lateral and vertical speeds in the vehicle coordinate system (that is, the v system) are always zero [67]. However, the forward speed of the vehicle still cannot be accurately obtained. WHU&Autonavi Team system uses rtic1 supervised learning method to train the vehicle forward speed prediction model [68], and the error can be controlled within 0.5 m s⁻¹.

Due to the random disassembly and reinstallation of the smartphone, the problem of the installation angle and lever arm parameters is not fixed. At this time, traditional direct setting or pre-calibration methods do not have the conditions for implementation. Automatic calibration of the installation angle and lever arm parameters can make the vehicle motion constraint algorithm more applicable.

c) Magnetic heading constraints: The magnetic interference caused by the vehicle shell can be equivalent to the magnetometer bias. So, the heading angle calculated based on the magnetometer observations can still accurately reflect the true heading angle change after the calibration and deduction of the magnetometer bias. Besides, the quasi-Static Magnetic Field (QSMF) is employed for avoiding environmental magnetic interference [69].

2) SZU-Mellivora Capensis: The data collection of Track 6 is located near the Beijing Haidian Airport. Its goal is to evaluate
the performance of vehicle navigation solutions based on the integration of different sensors such as GNSS, MEMS, and magnetometers on in-vehicle smartphones. This test is under typical urban road conditions. The smartphone is fixed inside the vehicle, and data is collected through the phone sensor. A single test process lasts about 1 hour and the test route consists of static initial alignment phase (about 5 minutes), open environment phase (about 20 minutes), obstructed environment phase where the GNSS signal is attenuated or blocked by the surrounding buildings or trees (about 25 minutes, during which the GNSS positioning results will be frequently interrupted) and no GNSS signal phase (underground parking lots about 10 minutes, with no GNSS positioning results). The driving process of the test vehicle includes going straight, left/right turning, reversing and parking.

To get the update of the vehicle's position, SZU-Mellivora Capensis team system obtain its velocity and heading. As for the velocity update, the system uses accelerometer and gyroscope, extract their data and align the coordinates, and then train them through proposed Deep Neural Network (DNN) to get the predicted velocity. The same is true for the heading prediction, but raw data used comes from the gyroscope, the magnetometer and the AHRS. Based on the prediction of velocity and heading, the relative displacement of the vehicle can be inferred. Then, the federated filter is used for data fusion. The weight factor is modified through observability to improve the filter and achieve high-precision localization. Finally, a smoothing filter is applied in this method.

The traditional inertial dead-reckoning mentioned above to estimate the motion of the vehicle is a challenging problem. To reduce this unavoidable inertial drift, a data-driven approach is used to inertial tracking. Referring to the network structure on IONet, the motion state of the vehicle is predicted by a trained deep Recurrent Neural Network (RNN). The RNN maintains the local hidden state within a time window, and then extracts the potential features of the time series. These features affect the state output at the next moment, thus enabling an effective recovery of the potential connection between data features and vehicle motion. The time window size is chosen as 1 s (50 frames). The data within the window are (n × 3 × 50) dimensional long-term dependent feature vectors constructed by stacking aligned n sensors. The changes of Δ_v and Δ_h in 1 s can be predicted by Equation 12:

$$ (v, \Delta h) = RNN((a_i, w_i, m_i, g_i)_t^T) \quad (12) $$

Unlike previous data-driven-inertial tracking work, the regression of the displacement vector is split into two separate parts: velocity estimation and heading estimation. The division of the regression task reduces the impact of extraneous sensors on prediction accuracy. In the velocity estimation part, input data are the 3-axis accelerometer and 3-axis gravity sensor data for a one-second period, which are corrected for the coordinate system alignment described above. The output is the average velocity over this time period, based in the two-dimensional plane. In the heading estimation section, input data are the 3-axis gyroscope and 3-axis magnetometer data during the time period, and the output is the sum of the heading changes in one second. The

<figure><img src="image_37.png" /><figcaption>Fig. 35. The RNN framework of the proposed method by ZU-Mellivora Capensis team.</figcaption></figure>

above input data is the best combination of sensors after the experiments performed.

Fig. 35 shows the RNN framework proposed in this system. A two-layer Long Short-Term Memory (LSTM) is used as the core module to solve the gradient explosion and vanishing problem of traditional RNNs, and it can effectively exploit the long-term dependence of time series. Each LSTM layer has 256 hidden nodes well above the dimensionality of the input data. This is in order to give enough inputs to the LSTM units so that the LSTM can fully utilize its function of selecting useful information. To avoid the overfitting problem, a dropout layer is added after each LSTM layer to increase the orthogonality between the features in each layer. Finally, a fully connected layer is placed to regress the velocity and heading changes, respectively. The loss function is defined in terms of the mean square error between the motion parameters and the ground truth. The ADAM optimizer is chosen to minimize this loss value and learn to obtain the best parameters within the RNN.

After obtaining the velocity and heading, the trajectory points can be expressed as:

$$ \left\{ \begin{aligned} x &= x_0 + vdt \cdot \cos(h_0 + \Delta h) \\ y &= y_0 + vdt \cdot \sin(h_0 + \Delta h) \end{aligned} \right. \quad (13) $$

3) Team YAI: YAI Team system uses three types of sensor data in this competition, namely ACCE, AHRS, GNSS. In the data pre-processing part, the ACCE and AHRS data were averaged per second to obtain data with a frequency of 1 Hz, while the missing GNSS were marked. First, the displacement of the vehicle per second is obtained by adding the initial velocity of the original GNSS to the ACCE data. Then, the YAW angle data of AHRS is initialized. After setting the initial direction angle, the angle ranges from minus 180 to 180 degrees.

The proposed framework used Kalman Filter (KF) for tracking. Fig. 36 shows the flow chart of the proposed tracking framework. The prepossessed data was introduced into the Kalman filter and GNSS to get KF gain to correct the error. Because the KF relied on the previous path to calculate, it does not work well during the missing section and may
<figure><img src="image_38.png" /><figcaption>Fig. 36. Flow chart for the proposed tracking framework of YAI team system.</figcaption></figure>

<figure><img src="image_39.png" /><figcaption>Fig. 37. The estimated driving direction during the testing period, where the horizontal axis is time with the unit second and the vertical axis is the estimated angle.</figcaption></figure>

produce significant cumulative errors, especially when the device encounters a large-scale angle variation. Fig. 37 shows the estimated driving direction during the testing period. The angle variation is obtained by the difference among a short period, 5 s in this competition. The framework calculated the missing sections of each route depending on the degrees of the angle variation with a threshold of 90 degree. For the small-scale angle variation, the above method was used to compensate for the missing section. For the large-scale angle

variation, the KF was changed to one-dimensional to address this issue. To be more specific, KF gain is let to not change the lateral displacement, while only affecting one-dimensional displacement. Afterwards, the AHRS is directly multiplied instead to estimate the direction of travel.

# VIII. TRACK 7: CHANNEL IMPULSE RESPONSES

## A. Track Description

Environments with complex Radio-Frequency (RF) propagation conditions such as indoor, urban or industrial environments have been a challenge for the RF positioning community for a long time. Especially in industrial environments, the abundance of metal objects causing absorption, reflection, diffraction and scattering of the signals leads to highly complex signal propagation that is hard to model analytically. Therefore, classic RF positioning methods relying on multi-angulation or multi-lateration are difficult to apply.

Received Signal Strength (RSS) based positioning exploit- ing the spatial significance of the propagation conditions has been used in these environments for many years. However, recently the use of Channel Impulse Responsess (CIRs), containing information on the whole signal propagation path, including Multipath Component (MPC) has been proposed. While the specialised hardware and firmware components used to obtain these signals are not yet available in mass user prod- ucts like smartphones, the introduction of Ultra-Wide Band (UWB) technology into newer generations of devices means that CIR-based positioning is a promising possibility even for low-cost applications in the near future. Since CIRs contains a variety of spatial and environment-related information, it has been used for positioning in three different ways:

* *Model error mitigation* [70], [71]: CIRs are used to classify propagation conditions like a missing line-of-sight (LoS) link or to estimate model errors caused by multipath components. The goal is to use CIRs to enhance classic positioning methods.

* *Fingerprinting* [72], [73]: CIRs is assumed to be spatially significant and the relation of the signal propagation to the environment is exploited for positioning by implicit modelling using a set of pre-recorded training data.

* *Multipath-SLAM* [74]–[77]: CIRs is used to jointly esti- mate the position of virtual anchors (i.e. characteristic reflection points) or other significant features and the user positions.

Since, in recent years, many research groups have been working on CIR-based positioning in adverse environments, a data set to compare different approaches under a common evaluation framework is highly beneficial to the community. Hence, a robotic scenario data set has been introduced using the popular Decawave DW1000 UWB chip. For this, an indus- trial environment in a testing hall has been reconstructed, equipped with state-of-the-art positioning reference systems.

## B. Competition Area

The environment resembles an industrial setting: it includes metal shelves, industrial vehicles and other objects that
<figure><img src="image_40.png" /><figcaption>Fig. 38. Measurement setup. The industrial environment consists of metal shelves and industrial vehicles. The Receiver/Anchor tags are highlighted with red boxes, the transmitter/mobile node is highlighted with a green circle.</figcaption></figure>

<figure><img src="image_41.png" /><figcaption>Fig. 39. Data distribution of the recorded data set. The objects in the environment are indicated in green: Metal shelves filled with goods (1) and (3); industrial vehicles (2) and (4) and a large metal box (5). The anchor node positions are depicted as red squares.</figcaption></figure>

influence the radio signal propagation in the environment. The measurement setup is depicted in Fig. 38: the stationary anchor nodes (highlighted with red boxes) are placed around the area and the mobile node (highlighted with a green circle) is attached to a wooden table to ensure a constant height. The measurement setup was such that the mobile node was configured as a transmitter and the stationary anchor nodes were configured as receivers. The wooden table was moved throughout the environment at constant speed (as best as possible). Hence, the environment and data resembles a robotic scenario.

Fig. 39 shows the distribution of the acquired data within the environment. The trajectories of the transmitter nodes are in-between the various objects; the size of the area is approximately 1 m × 20 m. In total, about 300,000 channel impulse responses were captured over a time period of approximately 1.5 h. The sampling interval of the data was about 10 Hz. For clearance, a constant sampling interval is not available, as straightforward re-sampling of CIR data is not possible because of the complexity of the signals. Of these, a temporally coherent set of 230,000 CIRs is available for training purposes, while another coherent set of 70,000 data points is used for testing/evaluation. This corresponds to exactly 20 minutes of recording time. A detailed description of the file format and the system specifications as well as downloadable links for the anchor/node configuration and the training and test data is available at http://evaal.aaloa.org/images/2020/ta7-v4.pdf.

## C. Indoor Positioning Solutions Provided by Competitors

1) Team YAI: First, the proposed system by team YAI converts the real and imaginary parts of the Channel Impulse

<figure><img src="image_42.png" /><figcaption>Fig. 40. The network architecture of the used DNN for CIR-based positioning of YAI team proposed system.</figcaption></figure>

Responses (CIR) into the magnitude, where the phase information is removed [78], [79]. All CIRs collected were organised by each receiver at the same location and at the same time.

After processing the CIR signals, the proposed system utilizes deep learning to build a mapping between the CIR magnitude and location. In the indoor environment, the size of 1 m² is taken as a grid, and then the position where the existing data appears is divided into 15 × 19 grid cells, numbered 0–284. Then, each grid is regarded as a class and each classifier is trained for a reviewer. This way, the positioning problem can be viewed as a classification problem. A typical machine learning DNN follows, to train a classifier for a receiver. In the learning procedure, the temporal magnitude CIR is directly regarded as a static feature vector to learn the grid information by DNN. The DNN network architecture used in the experiments include 8 layers, 465,373 neurons, and the activation function is softmax. A categorical-crossentropy follows to train the network parameters. Fig. 40 shows the network architecture of the used DNN. Finally, six receivers contain six independent classifiers that are able to convert the CIR magnitude into a grid.

Finally, the proposed system uses an ensemble approach to combine the estimation results from the six classifiers. In order to make the final answer more precise, the voting method is used first. That is, the final result is obtained by the majority
<figure><img src="image_43.png" /><figcaption>Fig. 41. Flow chart of the system proposed by YAI team for CIR-based positioning.</figcaption></figure>

voting from the six receivers. If voting from the six receivers results in a tie, the proposed system will select the receiver that contains the strongest CIR magnitude. This is under the assumption that the strongest CIR magnitude encounters less deviation and interference. The answer reported by that strongest receiver is viewed as the final estimation. If all six receivers have different answers, the proposed system will choose the one having the best performance at the offline training stage. Fig. 41 shows the flow chart of the proposed framework while Fig. 1 shows the final results.

## IX. RESULTS AND LESSONS LEARNED

In **Track 3 Smartphone** 11 teams were registered and submitted their final results. 9 out of 11 teams scored under 7 m, with the best one at 1 m. Despite the challenges imposed by the evaluation scenario, with some open areas and a few unmapped locations, 4 teams achieved a positioning error lower than 2 m. Competitors had a few months to process the pre-collected data. The key to success is the sensor fusion approach, which was adopted by all systems which obtained a score below 3 m. Kalman Filter (KF) and all its variants are very popular for this task.

In **Track 4 Foot-mounted IMU** 5 teams were registered, of which 4 accepted to publish their results. There were two final scores under 7 m. What was really amazing is the score reached by the winner: 0.5 m on such a scenario is impressive. The key of success is an excellent implementation of ZUPT. Techniques seem to be really up to scratch now, at least on scenarios with a constant walking pattern. For IPIN 2021,

Track 4 plans to add a running pattern for a more challenging competition.

In **Track 5 xDR in manufacturing** 4 teams registered and 2 submitted their final results. Final scores (CE75) of the Track 5's winners is higher than in other Tracks. Considering the fact that the data are measured in actual industrial situation, the achieved results in PDR-subtrack can to be regarded as positive.

The results in the VDR sub-track are worse than expected, exceeding 7 m. One possible reason is the lack of BLE beacons in the area of the VDR sub-track. Another one may be the lack of awareness surrounding VDR methods: maybe educational campaigns are needed to spread knowledge about VDR methods among researchers and practitioners.

In **Track 6 On-vehicle smartphone** 3 teams were registered and submitted their final results. Two final scores were under 30 m, with the best one at 7 m. The key of success is the perfect use of vehicle motion constraint information and magnetometer observations including ZUPT, ZIHR, Non-Holonomic Constraints (NHC) and magnetic heading. Considering the long interruptions of GNSS signal in the test data, more to the point is to maintain the vehicle heading accurate.

In **Track 7 Channel impulse response** only one team participated in the competition, reaching an evaluation score of 1.4 m. We expect future editions to see more widespread participation in this Track which is focused on a still little-known, leading-edge method.

### A. Lessons Learned

Maybe the most important results of the IPIN competition are comments made by competitors and observations made by Track chairs about the competing systems performance. Here we summarise the most important ones.

#### 1) Competitor Observations:

* Some competitors reported that the heading estimation is a critical step for systems based on sensor fusion. It seems that it only might work well when the smartphone is held in front of the body. This was mostly the case in the evaluation trajectory, except for one relatively short time interval. To improve accuracy, it is necessary to use a PDR algorithm that can better handle realistic movement and different walking modes. Thus, more phone carrying modes are required both at calibration and evaluation phases to handle realistic scenarios.

* Floor plans are essential for map-based localisation approaches. It may be beneficial for such competitors to have access to additional information about the building, i.e., pictures, videos, etc. The better understanding of the building with its specifics leads to more informed decisions regarding the system components and parameter configurations.

* Some competitors used visual inspection to choose the final submitted trajectories. This approach is not straightforward when comparing two trajectory candidates without knowing the ground truth locations of the evaluation points. Although it was quite feasible to identify entry and exit positions for the floor transition in the estimated trajectories, the path accuracy especially in larger open
areas was difficult to rate. After the competition, know-
ing the ground truth location of the evaluation points,
competitors realised that more rigorous methods for trials
comparison are needed instead of relying in simple visual
consideration.

* Deep learning requires large scale annotation to train accurate models. As the challenge data is sparsely annotated, one can proceed by pseudo-labelling non-annotated sensor data. This weak annotation works well in narrow corridors where the user's position can be accurately approximated. However, in open spaces, like in Track 3 *Smartphone*, weak annotations are harder to get straight. Approximation of user's position is less accurate and this added noise hurts the performance of deep learning models. This raises the issue of finding alternate ways to densely annotate sensor data.

* In Track 4 *Foot-mounted IMU*, there are some special scenes such as carousels, frequent stair, elevator switching as well as frequent pedestrian walking modes switching. Therefore, designing an accurate real-time algorithm to recognise localisation environment and walking mode is essential for foot-mounted pedestrian positioning.

* Some competitors for Track 4 *Foot-mounted IMU* learned how to analyse the foot-mounted IMU's signal characteristics in two unique scenes (escalator or lift) as well as the pedestrian positioning algorithms in these two unique indoor environments. Because the IMU noise is different between dynamic and static conditions, it was possible to fine-tune the sensor parameters based on the Allan variance. The adjustment method is to meet the optimal result of zero-speed correction under long-term static data. Moreover, some re-visits of the trajectory were found and made use of such valuable opportunities to correct the drift of foot-INS through close loop adjustment (smoothing like Simultaneous Localization and Map (SLAM)). How to improve the stability and reliability of Foot-INS is also an important issue.

## 2) Track Chairs Observations:

* Regarding Track 3 *Smartphone*, this is the fifth year in a row using the same data collection strategy and format to store the data. Despite that, it does not lose interest from the research community and has achieved gradual improvements in results year after year, showing that Track 3 is very competitive, and research teams are still interested in participating. Some teams have reported that this Track, with the collected data sets, have allowed them to improve their systems year by year.

* In Track 4 *Foot-mounted IMU*, this year we witnessed a wide range of resulting performance, with the winner doing much better than the other competitors. That means that in the future Track 4 competition will have to be more competitive and at the same time will have to pay attention to let the doors open to new competitors. One possible solution would be to add complex pattern like running or jumping; organisers also envisage to use a novel sensor delivering barometer data in addition to GNSS, IMU and magnetometer signals.

* For Track 5 *xDR in manufacturing*, we have kept challenging new trails in evaluation. Due to the difficulty in the realistic scenarios and the novelty of the competitions, some teams gave up before submitting the results; as a result, the number of the participants was less than expected. In order to attract more competitors, it is better to provide chances for using the evaluation framework and evaluation indicators. Sharing the evaluation scripts on GitHub will help promoting the evaluation framework. Moreover, development of the VDR method should be encouraged for boosting competitions of VDR.

* This was the debut year for Track 6 *On-vehicle smartphone*. Performance of competing system was widely varied, with the top two teams at about 10n, which is within the expected range. In IPIN 2021 an odometer sensor could be added to make it closer to the vehicle scene. At the same time, considering that more and more mobile phones can support differential positioning, differential positioning results will be provided to improve positioning accuracy. In addition, changing the posture of the mobile phone during the test will be considered, as this is a typical case in the real scene.

* Track 7 *Channel impulse response* is based on CIR, which is a novel topic and as such has not attracted many competitors. Since, in terms of RF signalling, most of the indoor positioning community is focusing on RSS- or range-based methods, a more detailed description of CIR- and other channel-based methods, including an extensive reference to recently published related approaches could have made the implied positioning task clearer. Furthermore, an example processing pipeline could have provided more guidance in handling the data. The amount of training data could have been reduced to allow for less computationally demanding computations. In the end, we believe that interest in this area is bound to grow with time, given its great promises.

# X. CONCLUSION

The IPIN Competition has been highly relevant for the indoor positioning community since the first edition held in Busan (Korea) in 2014. Based on the EvAAL framework, the purpose of IPIN competitions is to evaluate positioning solutions from academy and industry in challenging environments, using realistic procedures on a level field.

For the first time, the 2020 edition did not host on-site Tracks (Tracks 1 and 2), because of worldwide travel restrictions. However, the number of off-site Tracks was a record-high. The evaluation areas included a library building, a shopping mall, an indoor-outdoor road Track and industrial-oriented environments.

Of the 21 teams competing online in 2020 in Tracks 3–7, 20 accepted to contribute to this paper and concisely described their algorithm workflow. This collection is arguably the best description we can get today of state-of-the-art in personal indoor localisation systems at the algorithmic level.

21 competing teams and 95 attendants to the final online event witness a vibrant activity in the personal positioning field. This activity is focused on creating an environment
where indoor localisation systems can provide general and
cheap ways to position, track and navigate people indoors as
easily as GNSS does outdoors.

Most competitors of smartphone-based systems have employed as many sources of information as possible for positioning, making it clear that not only good sensing capabilities are relevant for positioning, but also rich context information (maps, images, videos) plays a key role in enhancing positioning accuracy.

In the previous section we have presented a short overview
of results. We find them impressive, especially with respect to
what was available just few years ago.

Yet, these observed results highlight a significant gap between the accuracy reported in the literature and the results obtained in the competition. It is far too easy to find accuracies reported in the literature which are unrealistically good with respect to what we observe in on-site Tracks. What is more uncomfortable is finding the same even with respect to off-site Tracks, which generally provide far better results.

This is mostly due to insufficient test and evaluation procedures, as the vast majority of papers in the literature present results obtained by simulation or trial in a small lab. Some papers present results obtained in larger areas (usually one floor of an office or university building) with an actor walking at a natural pace. Still, very few papers that we know of consider testing in unfamiliar areas, thus minimising the effect of building a system tuned to the laboratory environment.

Indoor localisation and seamless location-based services are enablers for an enormous market that will develop in the near future. The IPIN competitions have played an essential role in the academic and industrial research in this field; as far as we can tell they are going to play it for the foreseeable future as well.

CONTRIBUTIONS

The IPIN 2020 Competition was organized by ISTI in Pisa, an institute of the CNR (National Research Council) of Italy. Francesco Potortì chaired the competition and was responsible for the final overall review of the paper.

The team managing Track 3 (smartphone-based) was composed by Joaquín Torres-Sospedra, Darwin Quezada-Gaibor, Antonio Ramón Jiménez, Fernando Seco, and Antoni Pérez-Navarro.

The team managing Track 4 (foot-mounted IMU-based positioning) was composed by Miguel Ortiz, Ni Zhu, and Valerie Renaudin.

The team managing Track 5 (xDR Challenge in Manufacturing 2020) was composed by Ryosuke Ichikari, Ryo Shimo-mura, Nozomu Ohta, Satsuki Nagae, and Takeshi Kurata.

The team managing Track 6 (smartphone-based vehicle positioning without additional equipment) was composed by Dongyan Wei, Xinchun Ji, and Wenchao Zhang.

The team managing Track 7 (channel impulse responses) was composed by Sebastian Kram, Maximilian Stahlke, and Christopher Mutschler.

The team managing the online event competition was composed by Antonino Crivello, Paolo Barsocchi, Michele Girolami, and Filippo Palumbo.

All teams represented in this article have contributed with the description of their system and their invaluable feedback about their experiences and lessons learned in the competition:

• Track 3

- Team WHU-Five is composed by Ruizhi Chen (Corresponding author, email: ruizhi.chen@whu.edu.cn), Yuan Wu, Wei li, Yue Yu, Shihao Xu, and Lixiong Huang.

- Team IOT2US is composed by Chengqi Ma, Bang Wu (Corresponding author, email: bang.wu@qmul.ac.uk), Wei Zhang, Yankun Wang, Yonglei Fan, Stefan Poslad, David R. Selviah, and Weixi Wang

- Team XMU_ATR is composed by Lingxiang Zheng (Corresponding author, email: lxzheng@gmail.com), Ao Peng, and Ge Jin.

- Team NLE is composed by Leonid Antsfeld (Corresponding author, email: leonid.antsfeld@naverlabs.com) and Boris Chidlovskii.

- Team UMinho is composed by Ivo Silva, Cristiano Pendão, Filipe Meneses, Maria João Nicolau, António Costa, and Adriano Moreira (Corresponding author, email: adriano.moreira@algoritmi.uminho.pt).

- Team imec-WAVES is composed by Cedric De Cock (Corresponding author, email: cedric.decock@ugent.be) and David Plets.

- Team YAI is composed by Chia-An Yu, Shih-Hau Fang Fang, Ying-Ren Chien (Corresponding author, email: yrchien@niu.edu.tw), Chun-Hao Hung, and Chih-Chieh Yu.

- Team Indora is composed by Miroslav Opiela (Corresponding author, email: miroslav.opiela@upjs.sk) and Jakub Džama.

- Team TJU is composed by Liqiang Zhang (Corresponding author, email: zhangliqiang@tju.edu.cn), Boxuan Chen, Hu Li, Yazhen Liao, Qingyuan Gong, and Yu Liu.

- Team Next-Newbie Reckoners is composed by Sean-glidet Yean (Corresponding author, email: seanglidet.yean@ntu.edu.sg), Bo Zhi Lim, Wei Jie Teo, Bu Sung Lee, and Hong Lye Oh.

• Track 4

- Team WHU-GNSS is composed of Jian Kuang, Tao Liu, and Xiaoji Niu.

- Team AIR is composed of Wenchao Zhang, Dongyan Wei, and Hong Yuan.

- Team Free-Walking is composed of Qu Wang, Haiyong Luo, Hao Xiong, Linfeng Bao, Pushuo Zhang, and Fang Zhao.

- Team BHSNIP is composed of Ming Xia, Dayu Yan, Yuhang Li, Yitong Dong, and Haitao Jiang.

• Track 5

- Team KawaguchiLab is composed by Takuto Yoshida, Yoshiteru Nagata, Yuto Fukushima, Nobuya Fukatani, Nozomi Hayashida, Yusuke Asai, and Kenta Urano.
- Team YONAYONA is composed by Yoshitomo Yonamoto (Corresponding author, email: yonayona@keio.jp), Masahiro Yamaguchi, and Tomoya Kaichi.

• Track 6

- Team WHU&AutoNavi is composed by Jian Kuang, Wenfei Ge, Zhi Dou, Aipeng Tang, and Xiaobing Zhang.

- Team SZU-Mellivora Capensis is composed by Baoding Zhou, Xu Liu, Zhining Gu, Chengjing Yang, Zhiqian Wu, Doudou Xie, and Can Huang.

- Team YAI is composed by Chia-An Yu, Shih-Hau Fang Fang, Ying-Ren Chien (Corresponding author, email: yrchien@niu.edu.tw), Chun-Hao Hung, and Chih-Chieh Yu.

• Track 7

- Team YAI is composed by Nien-Ting Lee, Shih-Hau Fang (Corresponding author, email: shfang@saturn.yzu.edu.tw), Ying-Ren Chien, You-Cheng Jie, Shawn-Rong Young, and Chih-Chieh Yu.

ACRONYMS

<table><tr><td>AAL</td><td>Ambient Assisted Living</td></tr><tr><td>AHRS</td><td>Attitude and Heading Reference System</td></tr><tr><td>ALIP</td><td>Absolute Localization Inapplicable Period</td></tr><tr><td>ANFIS</td><td>adaptive network-based fuzzy inference system</td></tr><tr><td>AP</td><td>Access Point</td></tr><tr><td>API</td><td>Application Programming Interfaces</td></tr><tr><td>APIs</td><td>Application Programming Interfaces</td></tr><tr><td>BLE</td><td>Bluetooth Low Energy</td></tr><tr><td>BPF</td><td>Backtracking Particle Filter</td></tr><tr><td>CA</td><td>Circular Accuracy</td></tr><tr><td>CDF</td><td>Cumulative Distribution Function</td></tr><tr><td>CE</td><td>Circular Error</td></tr><tr><td>CIR</td><td>Channel Impulse Responses</td></tr><tr><td>CNN</td><td>Convolutional neural network</td></tr><tr><td>CRP</td><td>Correction Reference Points</td></tr><tr><td>DGNSS</td><td>Differential Global Navigation Satellite System</td></tr><tr><td>DNN</td><td>Deep Neural Network</td></tr><tr><td>EAG</td><td>Error accumulation gradient</td></tr><tr><td>EKF</td><td>Extended Kalman Filter</td></tr><tr><td>EvAAL</td><td>Evaluating Ambient Assisted Living</td></tr><tr><td>EKF</td><td>Extended Kalman Filter</td></tr><tr><td>EvAAL</td><td>Evaluating Ambient Assisted Living</td></tr><tr><td>FFT</td><td>Fast Fourier Transform</td></tr><tr><td>FOG-INS</td><td>Fiber Optic Gyro Inertial Navigation System</td></tr><tr><td>GDPR</td><td>EU General Data Protection Regulation</td></tr><tr><td>GLRT</td><td>Generalized Likelihood Ratio Test</td></tr><tr><td>GNSS</td><td>Global Navigation Satellite System</td></tr><tr><td>GPS</td><td>Global Positioning System</td></tr><tr><td>GRU</td><td>Gated Recurrent Units</td></tr><tr><td>HDE</td><td>Heuristic Drift Elimination</td></tr><tr><td>HMM</td><td>Hidden Markov Model</td></tr><tr><td>HUPT</td><td>Height Update Algorithm</td></tr></table>

<table><tr><td>IEZ</td><td>Inertial navigation system-extended Kalman filter-zero velocity update</td></tr><tr><td>iHDE</td><td>Improved heuristic drift elimination</td></tr><tr><td>IMU</td><td>Inertial Measurement Unit</td></tr><tr><td>INS</td><td>Inertial Navigation System</td></tr><tr><td>IPIN</td><td>Indoor Positioning and Indoor Navigation</td></tr><tr><td>ISHE</td><td>improved step height equidistant</td></tr><tr><td>KF</td><td>Kalman Filter</td></tr><tr><td>KNN</td><td>k-Nearest Neighbor</td></tr><tr><td>LBS</td><td>Location-based services</td></tr><tr><td>LMA</td><td>Levenberg-Marquardt</td></tr><tr><td>LoS</td><td>line-of-sight</td></tr><tr><td>LSTM</td><td>Long Short-Term Memory</td></tr><tr><td>MEKF</td><td>Multiplicative Extended Kalman Filter</td></tr><tr><td>MEMS</td><td>micro-electro-mechanical systems</td></tr><tr><td>MEMS-IMU</td><td>Micro-Electro-Mechanical Systems-Inertial Measurement Unit</td></tr><tr><td>MPC</td><td>Multipath Component</td></tr><tr><td>NHC</td><td>Non-Holonomic Constraints</td></tr><tr><td>PDR</td><td>Pedestrian Dead Reckoning</td></tr><tr><td>PERSY</td><td>PEdestrian Reference System</td></tr><tr><td>PF</td><td>Particle Filter</td></tr><tr><td>PNS</td><td>Pedestrian Navigation System</td></tr><tr><td>QSMF</td><td>quasi-Static Magnetic Field</td></tr><tr><td>REKF</td><td>Robust Extended Kalman Filter</td></tr><tr><td>RF</td><td>Radio-Frequency</td></tr><tr><td>RNN</td><td>Recurrent Neural Network</td></tr><tr><td>RSS</td><td>Received Signal Strength</td></tr><tr><td>RSSI</td><td>Received Signal Strength Indicator</td></tr><tr><td>SINS</td><td>Strapdown Inertial Navigation System</td></tr><tr><td>SLAM</td><td>Simultaneous Localization and Map</td></tr><tr><td>SVM</td><td>Support Vector Machine</td></tr><tr><td>TOF</td><td>Time-of-Flight</td></tr><tr><td>UWB</td><td>Ultra-Wide Band</td></tr><tr><td>VAE</td><td>Variational Auto-Encoder</td></tr><tr><td>VDR</td><td>Vehicle Dead-Reckoning</td></tr><tr><td>WKNN</td><td>Weighted k-Nearest Neighbor</td></tr><tr><td>ZARU</td><td>Zero angular rate update</td></tr><tr><td>ZIHR</td><td>Zero Integrated Heading Rate</td></tr><tr><td>ZUPT</td><td>Zero-velocity update</td></tr></table>

ACKNOWLEDGMENT

Track 3 organizers were supported by the European Union's Horizon 2020 Research and Innovation programme under the Marie Skłodowska Curie Grant 813278 (A-WEAR: A network for dynamic WEarable Applications with pRivacy constraints), MICROCEBUS (MICINN, ref. RTI2018-095168-B-C55, MCIU/AEI/FEDER UE), INSIGNIA (MICINN ref. PTQ2018-009981), and REPNIN+ (MICINN, ref. TEC2017-90808-REDT). We would like to thank the UJI's Library managers and employees for their support while collecting the required datasets for Track 3.

Track 5 organizers were supported by JST-OPERA Program, Japan, under Grant JPMJOP1612.

Track 7 organizers were supported by the Bavarian Ministry for Economic Affairs, Infrastructure, Transport and
Technology through the Center for Analytics-Data-
Applications (ADA-Center) within the framework of
“BAYERN DIGITAL II.”

Team UMinho (Track 3) was supported by FCT—Fundação para a Ciência e Tecnologia within the R&D Units Project Scope under Grant UIDB/00319/2020, and the Ph.D. Fellowship under Grant PD/BD/137401/2018.

Team YAI (Track 3) was supported by the Ministry of Science and Technology (MOST) of Taiwan under Grant MOST 109-2221-E-197-026.

Team Indora (Track 3) was supported in part by the Slovak Grant Agency, Ministry of Education and Academy of Science, Slovakia, under Grant 1/0177/21, and in part by the Slovak Research and Development Agency under Contract APVV-15-0091.

Team TJU (Track 3) was supported in part by the National Natural Science Foundation of China under Grant 61771338 and in part by the Tianjin Research Funding under Grant 18ZXRHSY00190.

Team Next-Newbie Reckoners (Track 3) were supported by the Singapore Government through the Industry Alignment Fund—Industry Collaboration Projects Grant. This research was conducted at Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU), which is a collaboration between Singapore Telecommunications Limited (Singtel) and Nanyang Technological University (NTU).

Team KawaguchiLab (Track 5) was supported by JSPS KAKENHI under Grant JP17H01762.

Team WHU&AutoNavi (Track 6) was supported by the National Key Research and Development Program of China under Grant 2016YFB0502202.

Team YAI (Tracks 6 and 7) was supported by the Ministry of Science and Technology (MOST) of Taiwan under Grant MOST 110-2634-F-155-001.

Francesco Potorti, Antonino Crivello, Paolo Barsocchi, Michele Girolami, and Filippo Palumbo are with the Information Science and Technologies Institute, National Research Council, 56124 Pisa, Italy (e-mail: antonino.crivello@isti.cnr.it).

Joaquín Torres-Sospedra is with UBIK Geospatial Solutions S.L., 12006 Castellón, Spain (e-mail: jtorres@uji.es).

Darwin Quezada-Gaibor is with the Institute of New Imaging Tech-
nologies, Unviersitat Jaume I, 12006 Castellón, Spain, and also with
the Electrical Engineering Unit, Tampere University, 33100 Tampere,
Finland.

Antonio Ramón Jiménez and Fernando Seco are with the Centre for Automation and Robotics (CSIC-UPM), 28500 Arganda del Rey, Spain.

Antoni Pérez-Navarro is with the Faculty of Computer Sciences, Multimedia and Telecommunication, Universitat Oberta de Catalunya, 08860 Barcelona, Spain, and also with the Internet Interdisciplinary Institute (IN3), Universitat Oberta de Catalunya, 08860 Barcelona, Spain.

Miguel Ortiz, Ni Zhu, and Valerie Renaudin are with the AME-GEOLOC, Université Gustave Eiffel, IFSTTAR, 44344 Bouguenais, France.

Ryosuke Ichikari is with the National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8560, Japan.

Ryo Shimomura and Nozomu Ohta are with the National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8560, Japan, and also with the Department of Intelligent Interaction Technologies, University of Tsukuba, Tsukuba 305-8573, Japan.

Takeshi Kurata is with the National Institute of Advanced Industrial
Science and Technology (AIST), Tsukuba 305-8560, Japan, and also
with the Faculty of Engineering, Information and Systems, University of
Tsukuba, Tsukuba 305-8573, Japan.

Satsuki Nagae is with the Department of Intelligent Interaction Tech-
nologies, University of Tsukuba, Tsukuba 305-8573, Japan.

Dongyan Wei, Xinchun Ji, and Wenchao Zhang are with the Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China.

Sebastian Kram, Maximilian Stahlke, and Christopher Mutschler are with the Fraunhofer Institute for Integrated Circuits IIS, 90411 Nürnberg, Germany.

Ruizhi Chen, Yuan Wu, Wei Li, Yue Yu, Shihao Xu, and Lixiong Huang are with the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan 430072, China.

Tao Liu, Jian Kuang, and Wenfei Ge are with the GNSS Research Center, Wuhan University, Wuhan 430072, China.

Xiaoji Niu is with the GNSS Research Center, Wuhan University, Wuhan 430072, China, and also with the Collaborative Innovation Center of Geospatial Technology, Wuhan University, Wuhan 430072, China.

Takuto Yoshida, Yoshiteru Nagata, Yuto Fukushima, Nobuya Fukatani, Nozomi Hayashida, Yusuke Asai, and Kenta Urano are with the Department of Information and Communication Engineering, Graduate School of Engineering, Nagoya University, Nagoya 464-8601, Japan.

Nien-Ting Lee, Shih-Hau Fang, You-Cheng Jie, Shawn-Rong Young, Chia-An Yu, and Chun-Hao Hung are with the Department of Electrical Engineering, Yuan Ze University, Taoyuan 32003, Taiwan, and also with the MOST Joint Research Center for AI Technology and All Vista Healthcare, Taipei 10617, Taiwan.

Ying-Ren Chien and Chih-Chieh Yu are with the Department of
Electrical Engineering, National Ilan University, Yilan 26047, Taiwan.

Chengqi Ma and David R. Selviah are with the Electronic and Electrical Engineering Department, University College London, London WC1E 7JE, U.K.

Bang Wu, Yonglei Fan, and Stefan Poslad are with the Electronic Engineering and Computer Science Department, Queen Mary University of London, London E1 4NS, U.K.

Wei Zhang, Yankun Wang, and Weixi Wang are with the Department of Research Institute for Smart Cities, Shenzhen University, Shenzhen 518060, China.

Hong Yuan is with the Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China.

Yoshitomo Yonamoto and Tomoya Kaichi are with the Faculty of Science and Technology, Keio University, Yokohama 223-8522, Japan.

Masahiro Yamaguchi is with the Nippon Electric Company Ltd., (NEC), Kawasaki 211-8666, Japan.

Baoding Zhou and Xu Liu are with College of Civil and Transportation Engineering, Shenzhen University, Shenzhen 518060, China.

Zhining Gu is with the School of Architecture and Urban Planning, Shenzhen University, Shenzhen 518060, China.

Chengjing Yang, Zhiqian Wu, Doudou Xie, and Can Huang are with the College of Civil and Transportation Engineering, Shenzhen University, Shenzhen 518060, China.

Lingxiang Zheng, Ao Peng, and Ge Jin are with the Department of Informatics and Communication Engineering, Xiamen University, Xiamen 361005, China.

Qu Wang is with the School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China.

Haiyong Luo and Linfeng Bao are with the Beijing Key Labora-
tory of Mobile Computing and Pervasive Device, Institute of Com-
puting Technology, Chinese Academy of Sciences, Beijing 100190,
China.

Hao Xiong, Pushuo Zhang, and Fang Zhao are with the School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing 100876, China.

Leonid Antsfeld and Boris Chidlovskii are with Naver Labs Europe, 38240 Meylan, France.

Haitao Jiang, Ming Xia, Dayu Yan, Yuhang Li, and Yitong Dong are with the School of Electronic and Information Engineering, Beihang University, Beijing 100083, China.

Ivo Silva, Cristiano Pendão, Filipe Meneses, Maria João Nicolau, António Costa, and Adriano Moreira are with the ALGORITMI Research Center, University of Minho, 4800-058 Guimarães, Portugal.

Cedric De Cock and David Plets are with the imec-WAVES Group, Department of Information Technology, Ghent University, 9052 Ghent, Belgium.

Miroslav Opiela and Jakub Džama are with the Faculty of Science, Institute of Computer Science, Pavol Jozef Šafárik University in Košice, 04001 Košice, Slovakia.
Liqiang Zhang, Hu Li, Boxuan Chen, and Yu Liu are with the School of Microelectronics, Tianjin University, Tianjin 300072, China.

Seanglidet Yean, Bo Zhi Lim, Wei Jie Teo, Bu Sung Lee, and Hong Lye Oh are with the School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798.

REFERENCES

[1] V. Renaudin et al., "Evaluating indoor positioning systems in a shopping mall: The lessons learned from the IPIN 2018 competition," *IEEE Access*, vol. 7, pp. 148594-148628, 2019.

[2] F. Potortì et al., "The IPIN 2019 indoor localisation competition—Description and results," *IEEE Access*, vol. 8, pp. 206674-206718, 2020.

[3] F. Potortì et al., "Comparing the performance of indoor localization systems through the EvAAL framework," *Sensors*, vol. 17, no. 10, p. 2327, Oct. 2017.

[4] P. Barsocchi, S. Chessa, F. Furfari, and F. Potortì, "Evaluating ambient assisted living solutions: The localization competition," *IEEE Pervasive Comput.*, vol. 12, no. 4, pp. 72-79, Oct./Dec. 2013.

[5] F. Potortì, A. Crivello, and F. Palumbo, "The EvAAL evaluation framework and the IPIN competitions," in *Geographical and Fingerprinting Data to Create Systems for Indoor Positioning and Indoor/Outdoor Navigation*. Amsterdam, The Netherlands: Elsevier, 2019, pp. 209-224.

[6] *Information Technology—Real Time Locating Systems—Test and Evaluation of Localization and Tracking Systems*, Standard ISO/IEC 18305:2016, 2016.

[7] F. Potortì, A. Crivello, P. Barsocchi, and F. Palumbo, "Evaluation of indoor localisation systems: Comments on the ISO/IEC 18305 standard," in *Proc. Int. Conf. Indoor Positioning Indoor Navigat. (IPIN)*, Sep. 2018, pp. 1-7.

[8] G. M. Mendoza-Silva et al., "Beyond Euclidean distance for error measurement in pedestrian indoor location," *IEEE Trans. Instrum. Meas.*, vol. 70, pp. 1-11, 2021. [Online]. Available: https://ieeexplore.ieee.org/document/9186638

[9] F. Furfari et al., "Discovering location based services: A unified approach for heterogeneous indoor localization systems," *Internet Things*, vol. 13, Mar. 2021, Art. no. 100334.

[10] J. Torres-Sospedra et al., "The smartphone-based offline indoor location competition at IPIN 2016: Analysis and future work," *Sensors*, vol. 17, no. 3, pp. 1-17, 2017.

[11] J. Torres-Sospedra et al., "Off-line evaluation of mobile-centric indoor positioning systems: The experiences from the 2017 IPIN competition," *Sensors*, vol. 18, no. 2, p. 487, Feb. 2018.

[12] A. R. Jiménez, F. Seco, and J. Torres-Sospedra, "Tools for smartphone multi-sensor data registration and GT mapping for positioning applications," in *Proc. Int. Conf. Indoor Positioning Indoor Navigat. (IPIN)*, Sep. 2019, pp. 1-8.

[13] J. Torres-Sospedra, D. Quezada-Gaibor, A. R. Jiménez, F. Seco, and A. Perez-Navarro, "Datasets and supporting materials for the IPIN 2020 competition track 3 (smartphone-based, off-site)," Zenodo, 2020, doi: 10.5281/zenodo.4314992.

[14] A. R. Jiménez et al., "Datasets and supporting materials for the IPIN 2019 competition track 3 (smartphone-based, off-site)," Zenodo, 2019, doi: 10.5281/zenodo.3606765.

[15] A. R. Jiménez et al., "Datasets and supporting materials for the IPIN 2018 competition track 3 (smartphone-based, off-site)," Zenodo, 2018, doi: 10.5281/zenodo.2823964.

[16] A. R. Jimenez, G. M. Mendoza-Silva, F. Seco, and J. Torres-Sospedra, "Datasets and supporting materials for the IPIN 2017 competition track 3 (smartphone-based, off-site)," Zenodo, 2017, doi: 10.5281/zenodo.2823924.

[17] A. R. Jimenez, G. M. Mendoza-Silva, R. Montoliu, F. Seco, and J. Torres-Sospedra, "Datasets and supporting materials for the IPIN 2016 competition track 3 (smartphone-based, off-site)," Zenodo, 2016, doi: 10.5281/zenodo.2791530.

[18] H.-J. Jang, J. Kim, and D.-H. Hwang, "Robust step detection method for pedestrian navigation systems," *Electron. Lett.*, vol. 43, no. 14, pp. 749-751, 2007. [Online]. Available: https://digital-library.theiet.org/content/journals/10.1049/el_20070478

[19] H. Weinberg, "Using the ADXL202 in pedometer and personal navigation applications," Analog Devices, Norwood, MA, USA, Appl. Note AN-602, Feb. 2002, vol. 2.

[20] E. R. Bachmann, R. B. McGhee, X. Yun, and M. J. Zyda, "Inertial and magnetic posture tracking for inserting humans into networked virtual environments," in *Proc. ACM Symp. Virtual Reality Softw. Technol. (VRST)*. New York, NY, USA: Association for Computing Machinery, 2001, pp. 9-16, doi: 10.1145/505008.505011.

[21] L. Zheng, Z. Wu, W. Zhou, S. Weng, and H. Zheng, "A smartphone based hand-held indoor positioning system," in *Frontier Computing*, J. C. Hung, N. Y. Yen, and K.-C. Li, Eds. Singapore: Springer, 2016, pp. 639-650.

[22] B. Chidlovskii and L. Antsfeld, "Semi-supervised variational autoencoder for WiFi indoor localization," in *Proc. Int. Conf. Indoor Positioning Indoor Navigat. (IPIN)*, Sep. 2019, pp. 1-8.

[23] N. Lee, S. Ahn, and D. Han, "AMID: Accurate magnetic indoor localization using deep learning," *Sensors*, vol. 18, no. 5, p. 1598, May 2018.

[24] L. Antsfeld, B. Chidlovskii, and D. Borisov, "Magnetic sensor based indoor positioning by multi-channel deep regression," in *Proc. Conf. Embedded Netw. Sensor Syst. (SenSys)*, Nov. 2020, pp. 707-708.

[25] I. Silva, A. Moreira, M. J. Nicolau, and C. Pendão, "Floor plan-free particle filter for indoor positioning of industrial vehicles," in *Proc. Int. Conf. Localization (ICL-GNSS)*, Tampere, Finland, 2020, pp. 14-30. [Online]. Available: http://ceur-ws.org/Vol-2626/paper2.pdf

[26] W. Kang and Y. Han, "SmartPDR: Smartphone-based pedestrian dead reckoning for indoor localization," *IEEE Sensors J.*, vol. 15, no. 5, pp. 2906-2916, May 2015.

[27] N.-H. Ho, P. Truong, and G.-M. Jeong, "Step-detection and adaptive step-length estimation for pedestrian dead-reckoning at various walking speeds using a smartphone," *Sensors*, vol. 16, no. 9, p. 1423, Sep. 2016.

[28] B. Li, B. Harvey, and T. Gallagher, "Using barometers to determine the height for indoor positioning," in *Proc. Int. Conf. Indoor Positioning Indoor Navigat.*, Oct. 2013, pp. 1-7.

[29] J. Trogh, D. Plets, L. Martens, and W. Joseph, "Advanced real-time indoor tracking based on the Viterbi algorithm and semantic data," *Int. J. Distrib. Sensor Netw.*, vol. 2015, pp. 1-11, Oct. 2015.

[30] WiCa Heuristic Indoor Propagation Prediction Tool (WHIPP). Accessed: Jan. 1, 2021. [Online]. Available: https://www.waves.intec.ugent.be/exposure-tool/expert-edition

[31] D. Plets, W. Joseph, K. Vanhecke, E. Tanghe, and L. Martens, "Coverage prediction and optimization algorithms for indoor environments," *EURASIP J. Wireless Commun. Netw.*, vol. 2012, no. 1, pp. 1-23, Dec. 2012.

[32] Widyawan, M. Klepal, and S. Beauregard, "A novel backtracking particle filter for pattern matching indoor localization," in *Proc. 1st ACM Int. Workshop Mobile Entity Localization Tracking GPS-Less Environ.*, 2008, pp. 79-84.

[33] M. Opiela and F. Galčík, "Grid-based Bayesian filtering methods for pedestrian dead reckoning indoor positioning using smartphones," *Sensors*, vol. 20, no. 18, p. 5343, Sep. 2020.

[34] W. H. K. de Vries, H. E. J. Veeger, C. T. M. Baten, and F. C. T. van der Helm, "Magnetic distortion in motion labs, implications for validating inertial magnetic sensors," *Gait Posture*, vol. 29, no. 4, pp. 535-541, Jun. 2009.

[35] L. Zhang, Y. Liu, and J. Sun, "A hybrid framework for mitigating heading drift for a wearable pedestrian navigation system through adaptive fusion of inertial and magnetic measurements," *Appl. Sci.*, vol. 11, no. 4, p. 1902, Feb. 2021. [Online]. Available: https://www.mdpi.com/2076-3417/11/4/1902

[36] S. O. Madgwick, A. J. Harrison, and R. Vaidyanathan, "Estimation of IMU and MARG orientation using a gradient descent algorithm," in *Proc. IEEE Int. Conf. Rehabil. Robot.*, Jun./Jul. 2011, pp. 1-7.

[37] R. S. Sinha, S.-M. Lee, M. Rim, and S.-H. Hwang, "Data augmentation schemes for deep learning in an indoor positioning application," *Electronics*, vol. 8, no. 5, p. 554, May 2019.

[38] M. Ortiz, A. Ramesh, N. Zhu, and V. Renaudin, "Datasets and supporting materials for the IPIN 2020 competition track 4 (foot-mounted IMU based positioning, off-site)," Zenodo, 2021, doi: 10.5281/zenodo.4668618.

[39] M. Ortiz, J. Perul, J. Torres-Sospedra, and V. Renaudin, "Datasets and supporting materials for the IPIN 2019 competition track 4 (foot-mounted IMU based positioning, off-site)," Zenodo, 2020, doi: 10.5281/zenodo.3937220.

[40] M. Ortiz, J. Perul, J. Torres-Sospedra, and V. Renaudin, "Datasets and supporting materials for the IPIN 2018 competition track 4 (foot-mounted IMU based positioning, off-site)," Zenodo, 2019, doi: 10.5281/zenodo.3228012.

[41] E. Foxlin, "Pedestrian tracking with shoe-mounted inertial sensors," *IEEE Comput. Graph. Appl.*, vol. 25, no. 6, pp. 38-46, Nov. 2005.

[42] S. Godha and G. Lachapelle, "Foot mounted inertial system for pedestrian navigation," *Meas. Sci. Technol.*, vol. 19, no. 7, Jul. 2008, Art. no. 075202.

[43] I. Skog, P. Handel, J. O. Nilsson, and J. Rantakokko, "Zero-velocity detection—An algorithm evaluation," *IEEE Trans. Biomed. Eng.*, vol. 57, no. 11, pp. 2657-2666, Nov. 2010.
[44] Z. Wang, H. Zhao, S. Qiu, and Q. Gao, "Stance-phase detection for ZUPT-aided foot-mounted pedestrian navigation system," *IEEE/ASME Trans. Mechatronics*, vol. 20, no. 6, pp. 3170–3181, Dec. 2015.

[45] A. R. Jiménez, F. Seco, J. C. Prieto, and J. Guevara, "Indoor pedestrian navigation using an INS/EKF framework for yaw drift reduction and a foot-mounted IMU," in *Proc. 7th Workshop Positioning, Navigat. Commun.*, Mar. 2010, pp. 135–143.

[46] J. Borenstein and L. Ojeda, "Heuristic drift elimination for personnel tracking systems," *J. Navigat.*, vol. 63, no. 4, p. 591, 2010.

[47] A. R. Jiménez, F. Seco, F. Zampella, J. C. Prieto, and J. Guevara, "Improved heuristic drift elimination (iHDE) for pedestrian navigation in complex buildings," in *Proc. Int. Conf. Indoor Positioning Indoor Navigat.*, Sep. 2011, pp. 1–8.

[48] W. Zhang, D. Wei, and H. Yuan, "The improved constraint methods for foot-mounted PDR system," *IEEE Access*, vol. 8, pp. 31764–31779, 2020.

[49] W. Zhang, X. Li, D. Wei, X. Ji, and H. Yuan, "A foot-mounted PDR system based on IMU/EKF+HMM+ZUPT+ZARU+HDR+compass algorithm," in *Proc. Int. Conf. Indoor Positioning Indoor Navigat. (IPIN)*, Sep. 2017, pp. 1–5.

[50] W. Zhang, D. Wei, and H. Yuan, "Novel drift reduction methods in foot-mounted PDR system," *Sensors*, vol. 19, no. 18, p. 3962, Sep. 2019.

[51] W. Zhang, D. Wei, P. Gong, and H. Yuan, "The PDR system based on improved QSF+ map matching algorithm," in *Proc. China Satell. Navigat. Conf.* Singapore: Springer, 2018, pp. 753–764.

[52] Q. Wang et al., "Personalized stride-length estimation based on active online learning," *IEEE Internet Things J.*, vol. 7, no. 6, pp. 4885–4897, Jun. 2020.

[53] Q. Wang et al., "Pedestrian dead reckoning based on walking pattern recognition and online magnetic fingerprint trajectory calibration," *IEEE Internet Things J.*, vol. 8, no. 3, pp. 2011–2026, Feb. 2021.

[54] X. Gao, H. Luo, Q. Wang, F. Zhao, L. Ye, and Y. Zhang, "A human activity recognition algorithm based on stacking denoising autoencoder and LightGBM," *Sensors*, vol. 19, no. 4, p. 947, Feb. 2019.

[55] C. Yu, L. Haiyong, Z. Fang, W. Qu, and S. Wenhua, "Adaptive Kalman filtering-based pedestrian navigation algorithm for smartphones," *Int. J. Adv. Robot. Syst.*, vol. 17, no. 3, May 2020, Art. no. 172988142093093.

[56] R. Ichikari, K. Kaji, R. Shimomura, M. Kourogi, T. Okuma, and T. Kurata, "Off-site indoor localization competitions based on measured data in a warehouse," *Sensors*, vol. 19, no. 4, p. 763, Feb. 2019, doi: 10.3390/s19040763.

[57] T. Kurata et al., "IoH technologies into indoor manufacturing sites," in *Proc. Int. Conf. Adv. Prod. Manage. Syst.*, vol. 567. Cham, Switzerland: Springer, 2019, pp. 372–380, doi: 10.1007/978-3-030-29996-5_43.

[58] *xDR-Challenge-2020-Evaluation*. Accessed: Mar. 28, 2021. [Online]. Available: https://github.com/PDR-benchmark-standardization-committee/xDR-Challenge-2020-evaluation

[59] T. Yoshida, J. Nozaki, K. Urano, K. Hiroi, T. Yonezawa, and N. Kawaguchi, "Gait dependency of smartphone walking speed estimation using deep learning (poster)," in *Proc. 17th Annu. Int. Conf. Mobile Syst., Appl., Services (MobiSys).* New York, NY, USA: Association for Computing Machinery, Jun. 2019, pp. 641–642, doi: 10.1145/3307334.3328667.

[60] T. Yoshida et al., "Sampling rate dependency in pedestrian walking speed estimation using dualCNN-LSTM," in *Proc. UbiComp/ISWC*. New York, NY, USA: Association for Computing Machinery, Sep. 2019, pp. 862–868, doi: 10.1145/3341162.3343765.

[61] N. Kawaguchi, J. Nozaki, T. Yoshida, K. Hiroi, T. Yonezawa, and K. Kaji, "End-to-end walking speed estimation method for smartphone PDR using dualCNN-LSTM," in *Proc. CEUR Workshop*, vol. 2498, F. Potorti, V. Renaudin, K. O'Keefe, and F. Palumbo, Eds., 2019, pp. 463–470. [Online]. Available: http://ceur-ws.org/Vol-2498/short60.pdf

[62] A. Manos, I. Klein, and T. Hazan, "Gravity direction estimation and heading determination for pedestrian navigation," in *Proc. Int. Conf. Indoor Positioning Indoor Navigat. (IPIN)*, Sep. 2018, pp. 206–212.

[63] J. Nozaki, K. Hiroi, K. Kaji, and N. Kawaguchi, "Compensation scheme for PDR using sparse location and error model," in *Proc. ACM Int. Joint Conf. Pervasive Ubiquitous Comput., ACM Int. Symp. Wearable Comput.* New York, NY, USA: Association for Computing Machinery, Sep. 2017, pp. 587–596, doi: 10.1145/3123024.3124406.

[64] J. Kuang, X. Niu, and X. Chen, "Robust pedestrian dead reckoning based on MEMS-IMU for smartphones," *Sensors*, vol. 18, no. 5, p. 1391, May 2018.

[65] Q. Zhang, X. Niu, H. Zhang, and C. Shi, "Algorithm improvement of the low-end GNSS/INS systems for land vehicles navigation," *Math. Problem Eng*, vol. 2013, Jun. 2013, Art. no. 435286.

[66] B. Li et al., "Robust Kalman filtering based on chi-square increment and its application," *Remote Sens.*, vol. 12, no. 4, p. 732, Feb. 2020.

[67] E. H. Shin, "Estimation techniques for low-cost inertial navigation," Univ. Calgary, Calgary, AB, Canada, UCGE Rep. 2021, 2005.

[68] M. A. Esfahani, H. Wang, K. Wu, and S. Yuan, "AbolDeepIO: A novel deep inertial odometry network for autonomous vehicles," *IEEE Trans. Intell. Transp. Syst.*, vol. 21, no. 5, pp. 1941–1950, May 2020.

[69] Y. Li, "Integration of MEMS sensors, WiFi, and magnetic features for indoor pedestrian navigation with consumer portable devices," Ph.D. dissertation, Univ. Calgary, Calgary, AB, Canada, 2016, doi: 10.11575/PRISM/26584.

[70] M. Stahlke, S. Kram, C. Mutschler, and T. Mahr, "NLOS detection using UWB channel impulse responses and convolutional neural networks," in *Proc. Int. Conf. Localization GNSS (ICL-GNSS)*, Jun. 2020, pp. 1–6.

[71] K. Bregar and M. Mohorcic, "Improving indoor localization using convolutional neural networks on computationally restricted devices," *IEEE Access*, vol. 6, pp. 17429–17441, 2018.

[72] A. Niitsoo, T. Edelhäußer, E. Eberlein, N. Hadaschik, and C. Mutschler, "A deep learning approach to position estimation from channel impulse responses," *Sensors*, vol. 19, no. 5, p. 1064, Mar. 2019.

[73] S. Kram, M. Stahlke, T. Feigl, J. Seitz, and J. Thielecke, "UWB channel impulse responses for positioning in complex environments: A detailed feature analysis," *Sensors*, vol. 19, no. 24, p. 5547, Dec. 2019.

[74] E. Leitinger, F. Meyer, F. Hlawatsch, K. Witrisal, F. Tufvesson, and M. Z. Win, "A belief propagation algorithm for multipath-based SLAM," *IEEE Trans. Wireless Commun.*, vol. 18, no. 12, pp. 5613–5629, Dec. 2019.

[75] M. Ulmschneider, C. Gentner, T. Jost, and A. Dammann, "Rao-blackwellized Gaussian sum particle filtering for multipath assisted positioning," *J. Electr. Comput. Eng.*, vol. 2018, pp. 1–15, Apr. 2018.

[76] R. Karásek and C. Gentner, "Stochastic data association for multipath assisted positioning using a single transmitter," *IEEE Access*, vol. 8, pp. 46735–46752, 2020.

[77] E. Leitinger, P. Meissner, C. Rüdisser, G. Dumphart, and K. Witrisal, "Evaluation of position-related information in multipath components for indoor positioning," *IEEE J. Sel. Areas Commun.*, vol. 33, no. 11, pp. 2313–2328, Nov. 2015.

[78] S.-H. Fang, W.-H. Chang, Y. Tsao, H.-C. Shih, and C. Wang, "Channel state reconstruction using multilevel discrete wavelet transform for improved fingerprinting-based indoor localization," *IEEE Sensors J.*, vol. 16, no. 21, pp. 7784–7791, Nov. 2016.

[79] S.-H. Fang, C.-C. Li, W.-C. Lu, Z. Xu, and Y.-R. Chien, "Enhanced device-free human detection: Efficient learning from phase and amplitude of channel state information," *IEEE Trans. Veh. Technol.*, vol. 68, no. 3, pp. 3048–3051, Mar. 2019.

Francesco Potorti (Member, IEEE) has been working in satellite and terrestrial communications with the Information Science and Technologies Institute, National Research Council, Pisa, Italy, since 1989, where he is a Senior Researcher. He has organized the 2011–2013 EvAAL competitions; defined the EvAAL framework; organized the IPIN competitions from 2014 to 2017; and chaired the tenth edition of the IPIN conference and the sixth edition of the IPIN Competition in 2019. He has coauthored more than 80 peer-reviewed scientific articles. His current research interests include RSS-based indoor localization, interoperability, and evaluation of indoor localization systems. He is a member of the IPIN Steering Board.

<figure><img src="image_44.png" /></figure>

Joaquín Torres-Sospedra was born in Castelló, Spain, in 1979. He received the Ph.D. degree in ensembles of neural networks and machine learning from the Universitat Jaume I in 2011. He joined the Institute of New Imaging Technologies (INIT), Universitat Jaume I, in April 2013, where he led indoor positioning projects. Since January 2020, he has been the Scientific Coordinator of UBIK Geospatial Solutions and still collaborates with the INIT, as well as other international research institutions. He has supervised five master's and two Ph.D. students. He is supervising six Ph.D. students. He has authored more than 120 articles in journals and conferences. His current research interests include indoor positioning solutions based on Wi-Fi and BLE, machine learning, and evaluation. He is the Chair of the IPIN International Standards Committee and the IPIN Smartphone-Based Off-Site Competition.
<figure><img src="image_45.png" /></figure>

**Darwin Quezada-Gaibor** received the bachelor's degree in mechatronic engineering from the Universidad Tecnológica América, Ecuador, in 2013, and the master's degree in radio engineering—GNSS receivers: hardware and software from Samara National Research University, Russia, in 2017. He is currently pursuing the Ph.D. degree with the Universitat Jaume I, Spain, and Tampere University, Finland. He is an Early Stage Researcher (ESR) with the Universitat Jaume I and Tampere University. His main

research interests include VoIP, cloud computing, networking, servers, and open-source software.

<figure><img src="image_46.png" /></figure>

**Miguel Ortiz** received the M.Sc. degree in mechanics, automation, and engineering from the Ecole Nationale Supérieure d'Arts et Métier in 2001. He joined the lab after spending six years in a company where he managed systems architecture for automotive applications. He has 12 years of experience in the GNSS domain as a research engineer. Since 2019, he has been the Head Deputy of the GEOLOC Laboratory, Université Gustave Eiffel, Nantes Campus. He is a Research Engineer with the GEOLOC

Laboratory, Université Gustave Eiffel (ex-IFSTTAR). As an Expert in embedded electronic systems, his research interests focus on software and hardware developments for both intelligent transport systems (ITS) and pedestrian navigation research field. Since 2017, he has been the Convenor of the CEN/CENELEC TC5-WG1 named "Navigation and Positioning Receivers for Road Applications."

<figure><img src="image_47.png" /></figure>

**Antonio Ramón Jiménez** was born in Sanander, Spain, in 1968. He received the degree in physics and computer science and the Ph.D. degree in physics from the Universidad Complutense de Madrid, Madrid, Spain, in 1991 and 1998, respectively. Since 1993, he has been with the Centre for Automation and Robotics, Spanish Council for Scientific Research, Madrid, where he holds a research position. He has authored more than 100 articles in journals and conference proceedings. His current research interests

include local positioning solutions for indoor/ GPS-denied localization and navigation of persons and robots, signal processing, Bayesian estimation, and inertial-ultrasonic-RFID sensor fusion. He is a reviewer for many international journals and projects in the field.

<figure><img src="image_48.png" /></figure>

**Ni Zhu** received the degree in aeronautic telecommunications engineering from the Ecole Nationale de l'Aviation Civile (ENAC) in 2015, and the Ph.D. degree in science of information and communication from the University of Lille in 2018. She is a Research Fellow with the GEOLOC Laboratory, Université Gustave Eiffel (ex-IFSTTAR). Her research interests include GNSS channel propagation modeling in urban environments, integrity monitoring for terrestrial applications, and multi-sensory fusion techniques for indoor/outdoor pedestrian positioning.

<figure><img src="image_49.png" /></figure>

**Fernando Seco** was born in Madrid, Spain, in 1972. He received the degree in physics from the Universidad Complutense of Madrid, Madrid, in 1996, and the Ph.D. degree in physics from the Universidad Nacional de Educación a Distancia, Madrid, in 2002, with a dissertation about the magnetostrictive generation of ultrasonic waves applied to a linear position sensor.

Since 1997, he has been with the Center for Automation and Robotics, Spanish Council for Scientific Research, Arganda del Rey, Madrid, where he holds a research position. His current research interests include the design and development of indoor local positioning systems, especially those based on ultrasonic and radio-frequency technologies in signal processing for CDMA-based localization systems and Bayesian estimation.

<figure><img src="image_50.png" /></figure>

**Valerie Renaudin** (Member, IEEE) received the M.Sc. degree in geomatics engineering from ESGT in 1999, and the Ph.D. degree in computer, communication, and information sciences from the EPFL in 2009. She was a Technical Director with Swissat Company, Samstagern, Switzerland, developing real-time geopositioning solutions based on a permanent global navigation satellite system (GNSS) network, and a Senior Research Associate with the PLAN Group, University of Calgary, Canada. She is a

Research Director (eq. Full Professor) with Université Gustave Eiffel (ex-IFSTTAR), France, where she is also leading the Geopositioning Laboratory (GEOLOC), where she built a team specializing in positioning and navigation for travelers in multimodal transport. Her research interests include outdoor/indoor navigation using GNSS, and inertial and magnetic data, particularly for pedestrians to improve sustainable personal mobility. Since 2013, she has been a member of the IEEE Society. She is the Steering Committee of the International Conference on Indoor Positioning and Indoor Navigation. She was a recipient of the European Marie Curie Career Integration Grant for her project smartWALK.

<figure><img src="image_51.png" /></figure>

**Antoni Pérez-Navarro** (Member, IEEE) received the bachelor's and Ph.D. degrees in physics from the Universitat Autònoma de Barcelona (UAB), in 1995 and 2000, respectively. He is currently the Director of the Technological Observatory, EIMT department, the Director of the technical collection of books of editorial Ediuoc, the Deputy Director of Research with the eLearn Center, Universitat Oberta de Catalunya (UOC), and a Lecturer with the Computer Science, Multimedia and Telecommunication Department (EIMT

department). He also works with the Escola Universitària Salesiana de Sarrià (EUSS) where he gives classes in industrial engineering grades. He has published several articles in international journals on all these topics. His main research interests include indoor positioning and the prevention of diseases via smartphones. He is a part of the Technical Program Committee of the IPIN. He is one of the organizers of the symposium "Challenges of Fingerprinting in Indoor Positioning and Navigation," Barcelona, in 2015 and the main Chair of the IPIN Conference in 2021. He acts as a reviewer of several journals.

<figure><img src="image_52.png" /></figure>

**Ryosuke Ichikari** received the Ph.D. degree in engineering from Ritsumeikan University in 2010. He is a Senior Researcher with the Human Augmentation Research Center, National Institute of Advanced Industrial Science and Technology (AIST), Japan. His research interests include indoor localization, virtual/mixed reality, and assistive technology for people with disabilities.

<figure><img src="image_53.png" /></figure>

**Ryo Shimomura** received the master's degree from the University of Tsukuba, Japan, in 2018, where he is currently pursuing the Ph.D. degree.
<figure><img src="image_54.png" /></figure>

**Nozomu Ohta** received the master's degree from the University of Tsukuba, Japan, in 2020, where he is currently pursuing the Ph.D. degree.

<figure><img src="image_55.png" /></figure>

**Wenchao Zhang** received the B.S. degree in surveying engineering from the China University of Mining and Technology (CUMT) in 2013, the M.S. degree in surveying engineering from Information Engineering University in 2016, and the Ph.D. degree in signal and information processing from the University of Chinese Academy of Sciences (UCAS) in 2020. He is currently an Assistant Researcher with the Aero Information Research (AIR) Institute, Chinese Academy of Sciences (CAS). His research interests include the multi-information fusion method, integrated navigation algorithm, and pedestrian autonomous positioning algorithm.

<figure><img src="image_56.png" /></figure>

**Satsuki Nagae** is currently pursuing the master's degree with the University of Tsukuba, Japan.

<figure><img src="image_57.png" /></figure>

**Sebastian Kram** received the master's degree in electrical and communication engineering from the Friedrich Alexander University of Erlangen-Nuremberg (FAU), Germany, in 2017. He joined the Locating and Communication Systems Department, Fraunhofer Institute for Integrated Circuits IIS in 2017. Since 2020, he has been working with the Navigation Group at the Chair for Information Technology (Communication Electronics), FAU. His research interests include tracking algorithms, machine learning, and sensor data fusion. He focuses on radio signal-based adaptive cooperative positioning in adverse environments using both model- and data-driven methods.

<figure><img src="image_58.png" /></figure>

**Takeshi Kurata** received the B.E., M.E., and D.E. degrees from the University of Tsukuba, Japan. Since 1996, he has been working as a Researcher with AIST, where he is currently the Deputy Director of the Human Augmentation Research Center. From 2003 to 2005, he was a Visiting Scholar with the HIT Laboratory, University of Washington, USA. From 2011 to 2014, he was a Ph.D. Co-Supervisor with Joseph Fourier University (UJF), Grenoble, France. Since 2020, he has been with the ISO/IEC JTC 1/SC 24 HoD, Japan. From 2018 to 2020, he was the Group Leader of the IoT R&D Center, Sumitomo Electric Industries, Ltd. He is also a Professor with the Faculty of Engineering, Information and Systems, University of Tsukuba (Cooperative Graduate School Program). His current research interests include indoor positioning, service research, assistive technology, the loH, and XR. He received the FY2016 AIST President Award (research). From 2014 to 2017, he was the PDR Benchmark Standardization Committee Chair.

<figure><img src="image_59.png" /></figure>

**Maximilian Stahlke** received the master's degree in electronic and mechatronic systems from the Institute of Technology Georg Simon Ohm, Germany, in 2020. Since 2020, he has been working with the Machine Learning and Information Fusion Group, Locating and Communication Systems Department, Fraunhofer Institute for Integrated Circuits IIS. His research interest includes hybrid positioning for radio-based localization systems with a focus on model- and data-driven sensor fusion.

<figure><img src="image_60.png" /></figure>

**Dongyan Wei** (Member, IEEE) received the B.S. degree in communication engineering from the University of Electronic Science and Technology of China (UESTC) in 2006, and the Ph.D. degree in signal and information processing from the Beijing University of Post and Telecommunication (BUPT) in 2011. He is currently a Research Fellow of the Aerospace Information Research (AIR) Institute, Chinese Academy of Sciences (CAS). He is the author of one book, more than 30 articles, and more than 20 inventions. His research interests include indoor position, multi-sensor fusion, and positioning in wireless networks. He is a TPC Member of the IPIN 2019 and the Deputy Chair of the IPIN 2022.

<figure><img src="image_61.png" /></figure>

**Christopher Mutschler** received the Diploma and Ph.D. degrees in computer science from Friedrich-Alexander-University Erlangen-Nuremberg (FAU) in 2010 and 2014, respectively. From 2017 to 2019, he was the Head of the Machine Learning and Information Fusion Group, where he was the Chief Scientist. He is the Head of the Precise Positioning and Analytics Department, Fraunhofer Institute for Integrated Circuits IIS, Nuremberg, Germany. He is a part-time Scientific Staff with Friedrich-Erlangen-Nuremberg (FAU), offering courses on machine learning. His research interests include machine learning and hybrid sensor fusion for radio-based locating systems.

<figure><img src="image_62.png" /></figure>

**Xinchun Ji** received the B.S. and M.S. degrees in guidance navigation and control (GNC) from the Beijing University of Aeronautics and Astronautics (BUAA) in 2010 and 2013, respectively. He is currently pursuing the Ph.D. degree in electronic information with Northwestern Polytechnical University (NWPU). He is currently a Senior Engineer with the Aerospace Information Research (AIR) Institute, Chinese Academy of Sciences (CAS). His research interests include multi-sensor fusion and geomagnetic matching for vehicle navigation applications.

<figure><img src="image_63.png" /></figure>

**Antonino Crivello** received the Ph.D. degree in information engineering and science from the University of Siena, Italy, in 2018. He is a Researcher with the Information Science and Technology Institute, National Research Council (ISTI-CNR), Pisa, Italy. His research interests include indoor positioning and ambient assisted living.