IoPT: A Concept of Internet of Perception-aware Things
Yuki Matsuda

yukimat@is.naist.jp
Nara Institute of Science and Technology
Ikoma, Nara, Japan…



Lv.570 dB



 …

 

90 dB



Noisy!!
Itʼs annoying...

…
Here is soLively!

Lv.2

…

 
  



Figure 1: The difference between the worlds sensed by IoT and perceived by human

ABSTRACTKEYWORDS

The Internet of Things (IoT) is undergoing remarkable technologicalinnovation, it is expected uncountable number of IoT devices will
be installed everywhere and enrich our daily life in near future.
There is a technical challenge that is the physically accurate data
does not always match the “experience” of people, because the“perception” of people will be easily biased by various stimulations
from surrounding environments. This paper presents a concept ofInternet of Perception-aware Things (IoPT), which aims to fill thegap in perception between IoT and human. Through the case study
targeting subjective crowdedness, we have confirmed perception
data have huge deviations though there are correlations between
sensor data and perception data, and perception will be biased dueto the environmental conditions.

Internet of Things, Perception, Human-in-the-loop System.

CCS CONCEPTS• Human-centered computing → Ubiquitous computing.
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
IoT ’22, November 7–10, 2022, Delft, Netherlands© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9665-3/22/11. . . $15.00
https://doi.org/10.1145/3567445.3571108

ACM Reference Format:
Yuki Matsuda. 2022. IoPT: A Concept of Internet of Perception-aware Things.
In Proceedings of the 12th International Conference on the Internet of Things(IoT ’22), November 7–10, 2022, Delft, Netherlands. ACM, New York, NY, USA,
4 pages. https://doi.org/10.1145/3567445.3571108

1INTRODUCTION

The Internet of Things (IoT) is undergoing remarkable technological innovation, and it is expected that countless IoT devices will
be installed in our living environment and carried by the general
people in the near future. This will be even more pronounced inurban environments where large numbers of people live, and where
mass IoT devices will be installed, continuously generating dataabout cities. In providing smart city services (e.g., sightseeing navigation) utilizing data obtained from such various IoT devices, howcan we provide services that are compassionate to the “people”
who actually use them? This is an important question in gaining
acceptance of the smart city concepts among general people. Until
today, several researches for realizing compassionate smart city
services are proposed, such as the psychologically safe road sensing
using a wearable device and social media [8], the memorable route
recommendation by detecting effective landmarks [9], the tourists’satisfaction level estimation in sightseeing spots [5], and mappingof the smellscapes using social media [7].

IoT ’22, November 7–10, 2022, Delft, Netherlands

Yuki Matsuda±  
    
       

 
 
   
y1

x2

y2

x3

y3

xnym   

…

…

…

x1        
Figure 2: A concept of Internet of Perception-aware Things (IoPT)

Here, the challenge is the physically accurate data does not always match the “experience” of people, because the “perception” ofpeople will be easily biased by various stimulations from surrounding environments. Figure 1 shows the typical example of mismatch
between IoT and human. Simply comparing noise level (dB) between a crowded temple and a crowded shopping mall, the latter
is absolutely noisier, however, people sometime perceive the otherway round depends on the situation. This situation might cause thedeterioration in the subjective quality of context-aware services,e.g., the recommendation without considering time, place, and occasion (TPO) will be much further from our expectations. This paper
presents a concept of Internet of Perception-aware Things (IoPT),
which aims to fill the gap in perception between IoT and human.

2INTERNET OF PERCEPTION-AWARE
THINGS (IOPT)In this section, we design the Internet of Perception-aware Things(IoPT), a concept for filling the gap in perception between IoT andhuman. Figure 2 shows the overview of the IoPT concept. The key
idea is that we employ participatory sensing [1], not as a method
to collect input data for direct analysis of urban environments, but
as a method to collect data for training models of IoT.
The IoPT consists of three core components: (1) participatory
sensing, (2) IoT sensor network, and (3) IoPT model.
Participatory sensing The data about the world perceived byhuman (perception data) will be collected by using a participatory sensing framework, e.g., requesting general people

to report their current feelings/perspectives via smartphone
apps. The perception data will be represented as a distribution form because people might perceive in different ways
even environment is the same completely.
IoT sensor network The data about the world sensed by IoT(sensor data) will be collected by using a IoT sensor network.
The various IoT, including IoT installed in certain places andmobile IoT carried by general people or vehicles etc., can beutilized as the sensing node.
IoPT model The model which allows the IoT sensor network
to simulate the perception of human (IoPT model), will be
trained with perception data as ground-truth and sensor data
as input.
Eventually, the IoT sensor network, which was tuned to acquirephysically accurate data at first, can be equipped with the capability
to understand the “experience” of people. It means that smart city
services considering people’s perceptions can be provided without
relying on continuous participatory sensing.

3

CASE STUDY AND DISCUSSIONAs the preliminary case study to confirm the IoPT concept, wesurveyed how the world is perceived by human by recruiting 2,200
general people on the Yahoo! Crowdsourcing platform in May 2021.
The overview of the case study is shown in Figure 3. As the target
which can be investigated with an online questionnaire, we selected
the subjective crowdedness of the city. To collect perceptions frompeople, we showed photos of cityscape including various numbers

IoPT: A Concept of Internet of Perception-aware Things

RGBT Crowd Counting
dataset

IoT ’22, November 7–10, 2022, Delft, Netherlands

Large

number of people

Answer subjective crowdednessby considering they are there.

pick 1 photo up
from every 25 percentile

randomselectI felt here isvery crowded

60 photos

sorted bynumber of people
in photo

I felt here isnot crowded

4 photos
x15 groups
Small

Crowd worker

number of people

Figure 3: Procedure of case study• the sensor data and perception data have correlations however perception contains huge deviations,• the surrounding environment (e.g., COVID-19 situation)
gives significant effects on the perception of people.
It suggests there is a use-case that our IoPT concept could apply.
Also, it suggests that the perception of people might be changed
over time affected by the alteration in customs, changes in population distributions, the happening of historic crises, etc. This change  ­ ­­







   ­­­   ­   of pedestrians (10–120 people), and asked them to answer subjective crowdedness degree on an 8-level Likert Scale (not crowded: 0 –very crowded: 7) by considering that they are there. We selected 60
photos of the first-person views from the RGBT Crowd Counting
dataset [3], and sorted them by the number of people in the photo.
Then, pick one photo up from every 25 percentiles randomly, andcreate 15 groups with 4 photos. One of these groups will be provided to one person, and the answer will be collected. To exclude
careless responses, the screening process of checking straightlining(selecting the same options) will be applied.
In addition, we divided people into two groups and requested
them to imagine different situations: the before-COVID-19 situationand the after-COVID-19 situation. In total, we got valid answers
from 834 people (3,336 samples), and 603 people (2,412 samples)
respectively.
The result is shown in Figure 4. Bubble charts and Histograms
show the relationship between the actual number of people in photos and the perceived crowdedness degree, and frequency of thelatter, respectively. To confirm whether the perception significantly
changes depending on the situation of the before- and after-COVID19 situations (Figure 4(a) and (b)), we conducted statistical testing.
First, random sampling is applied to both datasets for arranging
the number of samples, consequently, 2,412 samples are acquiredfor each. We applied the Kolmogorov-Smirnov Test with a 5% significance level, then it is confirmed that both distributions are notfollowing a normal distribution. Finally, we have confirmed a significant difference between the distribution of both conditions with
the Mann-Whitney U Test with a 5% significance level. From theresult, we have confirmed:  ­ ­­  Figure 4: Result of experiment focusing crowdedness of thecitywill cause a performance decrement in the IoPT model, i.e., conceptdrift [4]. To adjust the IoPT model with these changes, we mightneed to employ online learning and incremental learning.

4

CONCLUSION AND FUTURE PERSPECTIVE

In this paper, we proposed the IoPT concept, and designed architecture to fill the gap in perception between IoT and human. Throughthe case study, we confirmed there is a case in which the IoPTIoT ’22, November 7–10, 2022, Delft, Netherlands

concept might be applied. Also, we confirmed the technical challenges to realizing this concept. As a next step, we will build theIoPT model by collecting data with a participatory sensing framework [6] and a crowdedness estimation system for fixed-routed
buses [2], and evaluate whether the IoT sensor can simulate theperception of people.

ACKNOWLEDGMENTS
This study was supported in part by JST PRESTO under Grant No.
JPMJPR2039.

REFERENCES[1] Jeffrey A. Burke, Deborah Estrin, Mark Hansen, Andrew Parker, Nithya Ramanathan, Sasank Reddy, and Mani B. Srivastava. 2006. Participatory sensing. InFirst Workshop on World-Sensor-Web (WSW ’06). ACM, 1–5.
[2] Yuji Kanamitsu, Eigo Taya, Koki Tachibana, Yugo Nakamura, Yuki Matsuda, Hirohiko Suwa, and Keiichi Yasumoto. 2022. Estimating Congestion in a Fixed-RouteBus by Using BLE Signals. Sensors 22, 3 (2022), 1–15. https://doi.org/10.3390/s22030881[3] Lingbo Liu, Jiaqi Chen, Hefeng Wu, Guanbin Li, Chenglong Li, and Liang Lin.
2021. Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT

Yuki Matsuda

Benchmark for Crowd Counting. In 2021 IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR ’21). IEEE/CVF, 4821–4831. https://doi.org/10.1109/
CVPR46437.2021.00479[4] Jie Lu, Anjin Liu, Fan Dong, Feng Gu, João Gama, and Guangquan Zhang. 2019.
Learning under Concept Drift: A Review. IEEE Transactions on Knowledge andData Engineering 31 (2019), 2346–2363.
[5] Yuki Matsuda, Dmitrii Fedotov, Yuta Takahashi, Yutaka Arakawa, Keiichi Yasumoto, and Wolfgang Minker. 2018. EmoTour: Estimating Emotion and Satisfaction of Users Based on Behavioral Cues and Audiovisual Data. Sensors 18, 11(2018), 1–19. https://doi.org/10.3390/s18113978[6] Yuki Matsuda, Shogo Kawanaka, Hirohiko Suwa, Yutaka Arakawa, and KeiichiYasumoto. 2022. ParmoSense: Scenario-based Participatory Mobile Urban Sensing
Platform with User Motivation Engine. Sensors and Materials 34, 8 (2022), 3063–3091. https://doi.org/10.18494/SAM3961[7] Daniele Quercia, Rossano Schifanella, Luca Maria Aiello, and Kate McLean. 2015.
Smelly Maps: The Digital Life of Urban Smellscapes. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM ’15). AAAI, 327–336.
[8] Bernd Resch, Anja Summa, Günther Sagl, Peter Zeile, and Jan-Philipp Exner.
2015. Urban Emotions—Geo-Semantic Emotion Extraction from Technical Sensors,
Human Sensors and Crowdsourced Data. In Progress in Location-Based Services
2014. Springer, 199–212. https://doi.org/10.1007/978-3-319-11879-6_14[9] Shoko Wakamiya, Hiroshi Kawasaki, Yukiko Kawai, Adam Jatowt, Eiji Aramaki,and Toyokazu Akiyama. 2016. Lets Not Stare at Smartphones While Walking: Memorable Route Recommendation by Detecting Effective Landmarks. In Proceedings ofthe 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing(UbiComp ’16). ACM, 1136–1146. https://doi.org/10.1145/2971648.2971758