Seeing the Wind: An Interactive Mist Interface for Airflow
Input
TIAN MIN, Keio University, JapanCHENGSHUO XIA∗ , Guangzhou Institute of Technology, Xidian University, China
TAKUMI YAMAMOTO, Keio University, Japan
YUTA SUGIURA, Keio University, Japan

444
Fig. 1. By installing ToF sensors near the nozzle of the humidifier, it can be transformed into a device capableof detecting airflow within a certain space. The mist will capture the disturbances generated by motions andreflect them in the ToF readings through changes in light scattering. This system can detect various types of
motions, including (b) blowing, (c) gestures, (d) interactions with objects, and (e) exercises.
Human activities can introduce variations in various environmental cues, such as light and sound, which canserve as inputs for interfaces. However, one often overlooked aspect is the airflow variation caused by these
activities, which presents challenges in detection and utilization due to its intangible nature. In this paper, wehave unveiled an approach using mist to capture invisible airflow variations, rendering them detectable byTime-of-Flight (ToF) sensors. We investigate the capability of this sensing technique under different types of
mist or smoke, as well as the impact of airflow speed. To illustrate the feasibility of this concept, we created a
prototype using a humidifier and demonstrated its capability to recognize motions. On this basis, we introduce
potential applications, discuss inherent limitations, and provide design lessons grounded in mist-based airflow
sensing.
∗ Corresponding Author. Second Affiliation: Keio University, Japan.

Authors’ addresses: Tian Min, welkinmin@keio.jp, Keio University, Yokohama, Japan; Chengshuo Xia, chengshuoxia@xidian.
edu.cn, Guangzhou Institute of Technology, Xidian University, Guangzhou, China; Takumi Yamamoto, imuka06x17@keio.jp,
Keio University, Yokohama, Japan; Yuta Sugiura, sugiura@keio.jp, Keio University, Yokohama, Japan.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and thefull citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee. Request permissions from permissions@acm.org.
© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.
2573-0142/2023/12-ART444 $15.00
https://doi.org/10.1145/3626480
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:2Min et al.

CCS Concepts: • Human-centered computing → Interaction techniques.
Additional Key Words and Phrases: mist, interaction, motion recognition
ACM Reference Format:
Tian Min, Chengshuo Xia, Takumi Yamamoto, and Yuta Sugiura. 2023. Seeing the Wind: An Interactive Mist
Interface for Airflow Input. Proc. ACM Hum.-Comput. Interact. 7, ISS, Article 444 (December 2023), 22 pages.
https://doi.org/10.1145/3626480

1

INTRODUCTION

When designing interactive devices or sensing technologies, researchers often draw inspirationfrom human interaction with the surrounding objects. Previous efforts have been made using
various sensors and modalities to enable motion recognition and thus create innovative interaction
paradigms [23, 51, 52]. In addition to these commonly exploited physical properties, airflow, whichis pervasive in our surroundings, holds significant potential for interactive applications. Airflow isa perceivable and dynamic medium that humans can perceive, enabling its use as a conveyor of
information and thus a tactile display [25]. At the same time, it can be generated through motions
such as blowing, breathing, and mid-air gestures, sending information about the intensity anddirection of these motions. Furthermore, many daily motions, such as opening a door, closing
a book, or standing up from a chair, could naturally affect the surrounding airflow, presenting
an opportunity for utilizing airflow as an input for implicit interaction. Therefore, capturing andleveraging airflow properties opens up new possibilities for the interaction between humans anddevices.
However, detecting airflow, an intangible and invisible entity, presents certain challenges. Conventional approaches like anemometers (instruments measuring the airflow speed), mass flow
sensors, or pressure sensors are commonly employed in integrated systems and specialized domains. These techniques rely on the principle that the airflow generates pressure on object surfaces.
Although effective, these methods often prove less accessible for interaction purposes. Nevertheless, alternative solutions have emerged to address this limitation by indirect measurements[43].
One promising approach involves exploiting wind noise, which produces a distinct broadband
frequency response that exhibits particularly high amplitudes for low-frequency components when
captured by a microphone. Leveraging this characteristic, researchers have developed remarkable
and cost-effective blowable interfaces [7, 8, 30, 31]. However, microphone-based airflow detection
methods often focus on blowing motions. This is because blowing can generate the most noticeablewind noise with relatively high-speed airflow. In contrast, detecting other motions that generate
weaker airflow, such as waving or exercising, which do not produce sufficient sound or fast airflow,
can be challenging with microphones. It either becomes difficult to detect or requires additional
consideration for background noise separation.
Our key observation is that the mist exhibits variations in shape and volume as it drifts throughthe air, influenced by wind or airflow. Due to the composition of numerous tiny particles, themist has the unique potential and possibility to capture finer and more subtle changes in airflow.
This distinctive characteristic makes it stand out in sensing applications. From an interactive
perspective, the mist can provide natural visual feedback during interactions. It enables us to see
airflow variations that often go unnoticed in daily life but can be utilized as a means of sensing.
Previous studies have primarily focused on using the light scattering and obstruction or reflection
characteristics of the mist to create mid-air displays [22, 26, 46]. However, utilizing mist as an
input bridges the gap between intangible air disturbances and motion recognition. Traditional
anemometers are primarily designed for meteorological or industrial purposes and are not easily
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

Seeing the Wind: An Interactive Mist Interface for Airflow Input

444:3

integrated into interactive systems that rely on motion recognition as their main objective. Acquiring
information about airflow could significantly expand the possibilities in interactive design.
Based on these principles, we present a novel method for measuring airflow by employing mist
as the medium and recognizing the motions. Our proof-of-concept prototype consists of Time-ofFlight (ToF) sensors composed of an infrared (IR) diode, a phototransistor, and a mist-generating
device, such as a humidifier. The ToF sensors are positioned near the humidifier nozzle in an upwardorientation to detect the emitted mist. In an idle state, the particles in the mist obstruct the IR lightand reflect the part of the light, and thus the sensor can detect it. However, when the surrounding
airflow disperses the mist, the sensor’s measurements fluctuate depending on the intensity of the
airflow. Thus, according to sensing the airflow with the mist, we could be able to recognize thecorresponding motions that cause airflow with various intensities. Since we can use the common
mist generator (e.g., humidifier), this device can be ubiquitously placed in various indoor locations.
Users can easily replicate and modify the proposed sensing technique and apply it in their desired
scenarios. For example, when placed on a desktop, it can be used to recognize gestures or breath
input for controlling other electronic devices. During physical activities, it can count the numberof squats or spot jumps the user performs. When positioned in an entryway, it can detect door
openings or the presence of individuals passing by.
In this paper, we investigated and developed a mist-based interface for sensing human motions.
An interface consists of three ToF sensors and achieves a small, lightweight design that caneffectively applied to various types of daily mist-generating devices, such as humidifiers. The
system could sense human behavior more implicitly, which provides great potential in the naturalinteraction paradigm. The contributions of this paper are:
• The ToF sensor-based mist sensing capability was tested under different mist types andsituations.
• A proof-of-concept design for mist-based interface sensing the human motions and thecorresponding evaluations.
• Design lessons summarized for mist-based interface design and example applications presented.
2
2.1

RELATED WORK
Mist for Interfaces

Mist can be characterized by its formless nature and consists of numerous particles. It could obstruct
and scatter the light in the surroundings and allow mid-air display to become possible [33, 34]. Fog[26], water drops [2], and airborne particles [36] have been used to scatter the light. Previous efforts
focus on obtaining a better display experience. Tokuda et al. [46] designed MistForm, a shapechanging mist display that accommodates multiple viewers. They solved challenges such as image
distortion and uneven brightness on dynamically curved surfaces through machine learning-based
screen characterization.
There are also attempts to create novel interaction methods based on the mist display. Plasencia
et al. [26] proposed an interaction paradigm between a traditional tabletop surface and a seethrough and reach-through mist display. They utilized a perspective technique to merge the two
displays seamlessly, establishing a cohesive and immersive 3D interactive space. In addition to
enabling reach-through interactions, researchers are actively exploring methods to enhance thereconfigurable capabilities of mist screens and integrate them into the interactive experience.
Currently, there exist various approaches to manipulate the form and shape of mist in the air,offering new ways for visualization design. For example, movable nozzles [22], mechanical actuators[46] or aerosol [45]. Norasikin et al. [29] suggested a cost-effective and off-the-shelf solution for
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:4Min et al.

creating reconfigurable mid-air displays using Bessel beams. Airflow was taken as a medium fortransmitting sound waves, and Bessel beam generators were positioned around the nozzle. Thisapproach gave examples of aligning the projected content with the shape of the mist, thereby
introducing novel possibilities for interactive visualizations.
When examining the modules for detecting human activities in mist displays above, it becomes
evident that camera-based motion capture devices like Kinect or OptiTrack were predominantly
employed. Previous research has predominantly focused on employing mist as a medium for
projection or holography. In this paper, we innovatively regarded the variation in mist as an input
form. It might expand the design space for mist-based interaction if the sensing techniques basedon the intrinsic properties of the mist itself could be applied.
2.2

Sensing the Human Motion Indirectly

Human motion recognition has been successfully achieved by directly using the human body as thedetection object. The mainstream sensing methods include wearable IMU [19, 47], EMG [10, 18],
or non-wearable vision sensors [20, 49], Radar sensors [1, 12] and so on. This explicit sensing
method has been further explored for its algorithm and performance since it directly captures
the motion-related signals of the human body. However, another type of detection using implicit
sensing is emerging as well. This type of detection does not directly sense the human but acts on
the changes in other physical variables caused by the human motions, thus inferring the motions
performed.
This indirect sensing method provides more opportunities to combine the recognition systemwith daily items, thus forming a more natural and ubiquitous interface. To capture the user’s
behavior with a home-based item, Sugiura et al. [44] embedded the photoreflective distance sensor
module into a soft item and detect the distance variation between the sensor module and item
surface, which user’s behavior can cause. It therefore created a soft item-based interface to sense
the user’s behavior interacting with the cushion, toy, and other elastic objects. Similarly, Sugiura etal. [43] also deployed the photoreflective sensor to detect the stretching state of the elastic textile
because it would have different light transmission properties in different stretching states. It can beused to measure the tangential force applied by users as a stretchable interface. For sensing the user’s
force, Pei et al. [32] introduced a non-contact method named Forcesight, using laser speckle imaging.
Their key observation is that the presence of force induces deformations on object surfaces. Despite
being subtle, these deformations result in noticeable and distinguishable shifts in laser speckle
patterns, which could be exploited to detect the applied force. The corresponding idea employing
laser speckle imaging has also been used to detect the user’s behavior with household appliances[54], human motion [55], ground detection [48] and so on. In addition, detecting the lightness
variation caused by users could also create a ubiquitous interface. Optosense [50] presented asystem that employed the photodiode to build a self-powered interface. It utilizes the ambient lightto power the system itself and is able to sense the energy fluctuations led by lightness variation,and thus recognize the ambient situation and user’s motion. To employ the ambient energy change
for sensing the human, Sozu [53] introduced a toolkit design that allows the sensing system to be
deployed in various environments capable of generating energy, such as taps, outdoor, heating,doors, and so on. Through the energy harvesters, it can convert those energies into electricity
and thus sense the user’s motion to make any variation of energies, for example, opening thetap. Besides the light, the audio is a significant element in sensing human motion. For example,
SAMoSA [27] showed a smartwatch-based low power-consumption motion recognition system. It
senses the sound around the user by sub-sampling to recognize the corresponding motions.
Thus, it is believed that the indirect sensing method for human motion could build a morepervasive and ubiquitous interface for users [4, 37]. In this paper, we observe that the airflow that
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

Seeing the Wind: An Interactive Mist Interface for Airflow Input

444:5

causes the ubiquitous mist variation can also be employed for a potential indirect human motion
sensing method.
2.3

Airflow Used for Interaction Design

Airflow or wind can be produced through blowing, serving as an input modality, while its tactile
perception by humans enables it to function as an output method [42]. This interplay has built thebasis for using wind as a communication medium and the concept of blowable interfaces [38].
BLUI [31] is a hands-free interactive system that supports blowing at a laptop or computer screen
to directly control applications. This system captures the wind noise with a single microphone
and uses a k-nearest neighbors classifier to distinguish the area where the user is blowing withspecific wind noise. Given the widespread inclusion of microphones in various commercial products,
researchers have investigated the integration of breath input into wearable devices [7, 11, 30]. As acomplement, there is a growing interest in exploring design paradigms for breath-based interaction[5]. Kusabuka and Indo [21] proposed a system with a breath and IMU sensor on a mask, anddesigned a series of breath gestures including swiping, circling, dragging & dropping, and so on.
Blowing interaction is also commonly associated with healthcare or rehabilitation for respiratoryrelated diseases. For example, Larson et al. [24] presented a low-cost smartphone application forhome spirometry; Nikkila et al. designed a breath-controlled game to encourage medical adherence
for children with asthma.
The exploration of airflow as a potential medium for haptic feedback and its application as anaugmentation tool for enriching immersive experiences has also garnered more and more researchinterest these years. Lee et al. [25] conducted a study on the utilization of airflow as a non-contact
wearable display. Their system incorporated multiple micro-fans positioned near the cheeks, neck,upper arms, wrists, and ankles. The researchers examined the perception threshold of airflow
intensity and duration for tactile stimulation. Building upon this work, Shim et al. [41] further
advanced the concept. They used wind as an extra feedback modality to address the limited channel
capacity problem resulting from the small contact area when applying a tactile display to the backof a smartwatch. In the field of virtual reality (VR), airflow is widely acknowledged as a crucialcomponent in simulating real-world experiences, and numerous studies have been conducted to
optimize its impact on enhancing user immersion [8, 15, 17, 28, 35]. Moreover, the incorporation ofwind-driven wafting odor adds another dimension to the interaction of airflow, particularly in thecontext of olfactory display [14]. Existing airflow-based system designs primarily focus on blowing.
However, it is noteworthy to recognize that airflow could be generated in various contexts
beyond blowing and the input method could make more extensive use of the airflow itself. This
paper positions itself as adding a new perspective of input to the design space of the "atmosphericinterfaces" [3].
3 MIST SENSING TECHNIQUE
To create a mist-based interface, our idea is to capture the variations in the mist caused by the
airflow generated by human motion and recognize it. Utilizing the mist variation could have awider detection range than normal airflow or wind noise detection, and it also maintains greatinteraction and visualization features.
Therefore, we focus on the techniques related to mist sensing. The classical way of detecting
smoke is usually with chemical sensors or photoelectric sensors, which are often used for specifictypes of smoke detection, such as nephelometers or aethalometers for safety purposes to detect thepresence of fires, specific chemical leaks, environmental monitoring, and so on [13]. However, fordaily interaction purposes, such expensive devices are not well suited for common mist detection,such as mist from humidifiers. Since the nature of mist lies in its constituent particles, different
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:6Min et al.

particles cause the reflection of light. Therefore, we used the ToF sensor to capture the change inmist. The emitter on the ToF sensor transmits an infrared light, which, upon encountering smoke
particles, undergoes absorption, reflection, or penetration. The reflected light is then received by
the photodetector on the sensor. Thus, in this section, we investigated the capability of mist sensing
with the proposed method, including the sensing capability of different types of common mist, theinfluence of airflow direction, and velocity.

Fig. 2. Experiment devices: (a) VL53L1X ToF sensor. (b) WT87A anemometer. (c) LKC-1000S+ 2nd air monitor.
(d) Small fan; Source of smoke/ mist generation: (e) Humidifier. (f) Dry ice. (g) Vaporizer. (h) Incense. (i) Hot
water. (j) Cigarette.

3.1

Apparatus

The equipment used in these studies is illustrated in Figure 2 (a)-(d). We used the VL53L1X ToF
sensor (Strawberry Linux Co., Ltd.), which has a 940 nm wavelength infrared laser to detect
distances ranging from 100 mm to 4000 mm. We employed a sampling frequency of 30 Hz. For
the instrument that measures wind speed, we utilized the WT87A anemometer (manufactured by
Wintact), capable of detecting wind speeds from 0.3 m/s to 30 m/s, with an accuracy range of 5%.
Additionally, we employed an Arduino Uno R3 to record the data and a common small fan as thesource of airflow generation.
3.2 Sensing Common Mist Types
To investigate the impact of different mist types on the light scatter effect for the ToF sensor, westudied six objects capable of generating mist or smoke (Figure 2 (e)-(j)). As shown in Figure 3 (a),to clarify the effects of mist, a flat barrier was placed at a distance of 15 cm from the ToF sensor tocompare the effect of mist with normal distance detection. The mist was generated at a distanceof 5 cm from the sensor. In order to ensure the vertical ascent of the smoke, the experiment was
conducted indoors in a controlled environment devoid of airflow.
As shown in Figure 3 (b), the sensor data was applied to Gaussian filtering. Smaller amounts of
mist/smoke, such as hot water, incense, and cigarettes, are influenced by the surrounding airflow
caused by the heat generated during their burn, resulting in greater fluctuations. Other objects
like humidifiers, capable of generating a stable and substantial volume of smoke, exhibit lesser
fluctuations. Additionally, all tested mist types demonstrated the ability to be sensed by the ToF
sensor. As shown in Figure 3 (c), to further investigate the properties of different types of mist,
we used an LKC-1000S+ 2nd air monitor (Elitech Technology, Inc) to measure the particle density.
Based on part of the samples such as Incense, cigarettes, and Vaporizer, it can be observed thathigher particle concentrations result in a higher efficiency of blocking the infrared light.
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

Seeing the Wind: An Interactive Mist Interface for Airflow Input

444:7

Fig. 3. (a) A smoke source is placed along the straight line between the sensor and the barrier. (b) The datafrom the sensor when different types of smoke obstruct the IR laser and (c) the number of particles (per/L)
measured by a particle sensor. The colors of the curves correspond to the colors of categories in the bar chart.

3.3

Sensing Airflow Direction

Fig. 4. When two sensors are placed on the sides of the humidifier nozzle, the sensor data and corresponding
schematic diagram when airflow blows from the right side. The colors of the curves correspond to the colorsof the sensors, while the x-axis and y-axis represent the frame number in sequence and sensor readings,
respectively.

We then tested how the sensor data can reflect the direction of airflow. Two ToF sensors with thelight emission source upwards were placed at a horizontal distance of 1cm and closed to the nozzle
on both sides, and an airflow was applied from the right side (Figure 4). Under idle conditions,the mist generated upward from the nozzle could simultaneously cover the detection range ofboth sensors. Once the airflow was blown from the right side, it initially dispersed the mist above
the right sensor, resulting in a peak in the curve represented by the green line. Subsequently, themist above the left sensor was dispersed, forming another peak on the blue line. By increasing thenumber of sensors near the nozzle, it is possible to detect airflow direction in a 360-degree range.
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:83.4

Min et al.

Sensing Airflow Velocity

Fig. 5. (a) The anemometer was positioned in proximity to the mist-covered sensor. Specified wind speed
values were achieved by adjusting the rotational speed of the fan and the distance between the fan andthe anemometer. (b) The distribution of sensor readings across 1,500 frames of data is depicted. Each row
represents the distribution of readings corresponding to airflow velocity and the absence of mist. The color ofeach cell reflects the sensor reading.

Different airflow velocities are expected to cause varying dispersion effects on mist. To further
examine this phenomenon, we placed an anemometer close to the sensor and the nozzle to measure
the airflow velocity arriving at the mist’s source (Figure 5 (a)). By adjusting the speed of a fan andits distance from the anemometer, we generated a continuous horizontal airflow near the nozzle at
a designated velocity. We conducted a total of 11 conditions, including the absence of mist, withdata recorded at intervals of 0.5 m/s ranging from 0 m/s to 4.5 m/s. Once the anemometer readings
stabilized, we initiated the recording of sensor data for a duration of 1,500 frames.
We organized the sensor data into a heat map shown in Figure 5 (b). The color of each grid cellreflects the magnitude of the sensor data. By analyzing the proportions of different colors withineach row, we can evaluate the impact of continuous airflow on mist dispersion with velocity changes.
Notably, we observed a clear nonlinear relationship between airflow velocity and mist dispersal.
When the airflow velocity is below 1 m/s, an increase in airflow velocity does not effectively disperse
the mist. However, within the range of 1 m/s to 2 m/s, increasing the airflow velocity demonstrates
the most pronounced impact. This also suggests that the sensitivity of sensing mist changes is better
when the airflow speed of the generated airflow is in the interval of 1-2 m/s. During this interval,
users can visibly see changes in the mist pattern. However, in practice, the airflow affecting themist is also affected by the distance between the airflow source and the mist source. For example,
an airflow source with a specific speed will have a diminishing effect as it moves away from themist. Therefore, in the design, it is necessary to balance the velocity of the generated airflow andthe distance between the airflow source and the mist.
4
4.1

SYSTEM DESIGN
HardwareOur proof-of-concept prototype consists of a mist generator humidifier, three ToF sensors, anda bracket (Figure 6). The humidifier is a 3oz Ultrasonic Aroma Diffuser manufactured by MUJI,
which is in a cylindrical shape with a base diameter of 78 mm and a height of 140 mm. The nozzle
is 20 mm in width. The humidifier can spray mist in front of the nozzle with an elevation angle
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

Seeing the Wind: An Interactive Mist Interface for Airflow Input

444:9

Fig. 6. Proof-of-concept prototype.

of 45 degrees. Due to the narrow nozzle, the mist from the humidifier covers a cone shape with aslender tail. This mist-covered space has a length of approximately 15 cm and a maximum width ofabout 10 cm.
In order to maintain an appropriate value on the sensors and quickly reset once the disturbance isover, we aim for a stable presence of mist in specific regions, which is the area near the nozzle for thishumidifier. If the sensors’ distance from the nozzle is too far, even without apparent disturbances,the mist itself may dissipate or drift due to gravity. When airflow-carrying mist has a relatively
high velocity, it helps maintain the shape and integrity of the mist. For example, at the nozzle ofa humidifier, we can observe the mist being sprayed out like a ribbon. Background disturbances
caused by factors like air circulation in the environment would have minimal impact on the mistat this location. Therefore, we positioned one sensor directly beneath the nozzle. As mentioned
in Section 3.3, placing sensors on both sides of the nozzle could detect airflow direction throughthe time difference of peaks in sensor readings, we positioned a sensor on each side, closer to thenozzle. Since the humidifier we used as the prototype sprays mist diagonally upwards, the sensors
on both sides are slightly tilted to ensure they can be covered by the mist in an idle state.
As shown in Figure 6 (b), Middle Sensor is positioned below the nozzle at a distance of 15 mm,
while Left Sensor and Right Sensor are tilted at an angle of approximately 50 degrees and placed ata horizontal distance of approximately 22 mm from the nozzle. Three VL53L1X ToF sensors are set
to a sampling rate of 40 Hz. The sensors are powered by a 3.3V power supply from an Arduino Uno
R3 and connected via the IC2 interface for data recording.
Due to the narrow shape of the mist at the nozzle, if we want to add more sensors and ensure
their complete coverage by the mist simultaneously, they should be positioned along the direction
where the mist is sprayed. For example, based on the current configuration, we could add more in
the direction where the mist is sprayed. However, as the sensor is positioned away from the nozzle,
the sensors are more susceptible to disturbances and slower resetting. Considering the potential
lack of useful information and features in the data with further increases in sensor quantity andthe risk of model overfitting, we chose to use three sensors to capture the mist variation close to
the source (i.e., nozzle) in this prototype.
4.2

Machine Learning

Considering the complexity of the airflow patterns, the manually selected features would bechallenging and time-consuming. Thus, we employed the ROCKET [9], an exceptionally fast andProc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:10

Min et al.

Fig. 7. The input data is applied with random kernels, and then sent to a linear classifier.

accurate time series classification method using random convolutional kernels. For a given input
time series, ROCKET generates a vast number (10,000) of random convolutional kernels at various
positions and with different lengths. Within each kernel, features are extracted and a feature map
is generated. Rather than relying on complex neural networks, simple linear classifiers are utilized
to perform the classification task (illustrated in Figure 7). In this study, the default setting (10,000
kernels, 4 features per kernel) of ROCKET classifier is used.
5

EVALUATION

We evaluated using the designed prototype to validate the performance of mist induced by airflow
as a method for motion recognition. Additionally, we investigated the effects of training dataset
size, the sensor number and placement, and the performance of universality. Additionally, we also
conducted a study to explore the performance of the interface further to recognize the different
motions that cause similar airflow.
5.1

Tested Motions

Referred to the related work [5, 21], we tested several blowing gestures, including Blow, Double
Blow, Long Blow, and extended them by variations from different directions with respect to thedevice. Additionally, we explored the use of common interactive objects that can generate airflow,
resulting in two motions of: Flip Book (From Left) and Wave with Paper (In Front with a half-folded
A4 paper). We also chose two indoor fitness exercises: Jumping Jack and Squat, bringing the total
number of motions to 10 (illustrated in Figure 8).
5.2

Dataset Collection

We recruited 12 participants (3 females), all right-handed, with a mean age of 22, from a university.
Participants were compensated with 10$ for their involvement. The device is set on a desk infront of participants in their reach. After a brief introduction of the device and the experiment’s
purpose, data collection started. The device was placed on a common office desk, with the nozzle
approximately 85 cm above the ground. Participants were invited to sit on a regular chair around
50 cm in height, which was placed in front of the device, and perform the motions in Figure 8 (a)to (h), followed by participants stood up at the desk and performed motions (i) and (j). The datacollection followed the order of blowing (motion (a) to (f)), interacting with objects (motion (g) and(h)), and exercises (motion (i) and (j)).
For motions (a) to (f), participants proceeded from left to right, completing all motions in that
direction before transitioning to the next. Participants were not constrained by specific motion
intensity and were instructed to perform the motions naturally and comfortably. For each motion,
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

Seeing the Wind: An Interactive Mist Interface for Airflow Input

444:11

Fig. 8. 10 motions tested: (a) Blow Left. (B) Blow Front. (c) Blow Right. (d) Double Blow Left. (e) Long Blow
Front. (f) Double Blow Right. (g) Flip the Book on the Left. (h) Wave with Paper in Front. (i) Jumping Jack. (j)Squat. The blue arrows in the figure represent the expected direction of airflow. The green arrows in (g) and(h) represent the direction of the movement. The white rectangular planes in (i) and (j) represent the desk
where the device is set.
Table 1. Classification accuracy when using different numbers and combinations of sensors.

AccuracySD

Middle

Left

Right

Left andMiddleRight and Left andMiddleRight

All Sensors

62.61%
2.12

75.50%
5.07

71.83%
6.75

80.86%
4.64

77.56%
5.06

91.17%
4.22

86.43%
3.46

participants performed the task upon receiving instructions. We recorded ten sets of data with 160
frames. This procedure resulted in 3,600 * 160 time series data (12 participant × 10 motions × 10
rounds × 3 sensors × 160 frames).
5.3

Results

5.3.1 Within-User Result with Different Sensor Numbers. We conducted tests to evaluate the classification accuracy of our classifier for 10 different motion recognition. The results of classification withdata from three, two, and single sensors and different combinations respectively are summarized inTable 1 and shown in Figure 9 (10-fold validation used).
When using data from all three sensors (Figure 9 (a)), the accuracy reached 91.17% (SD=4.22).
When using data from the sensors set on each side of the nozzle (Figure 9 (b)), the accuracy is
86.43% (SD=3.46). The confusion among directions increases when removing the middle sensor.
Classification results from left and middle, right and middle sensors are 80.86% (SD=4.64) and 77.56%(SD=5.06), respectively. This finding provides evidence to some extent for the significance of asymmetric sensor configuration. Furthermore, we observed that these results further amplify thebias and false positive of the model towards directions (Blow Left and Double Blow Right). Whenusing data from a single middle, left, or right sensor (Figure 9 (e)-(g)), the accuracy dropped to62.61% (SD=2.12), 75.50% (SD=5.07) and 71.83% (SD=6.75), respectively. From the results, we foundthat the layout with a single sensor led to lower accuracy. Among them, using the middle sensor
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:12Min et al.

Fig. 9. Confusion matrices when using data from different numbers and combinations of sensors: (a) All
3 sensors. (b) 2 sides sensors. (c) Left and middle sensors. (d) Right and middle sensors. (e) Single middle
sensor. (f) Single left sensor. (g) Single right Sensor. The middle-right corner of each confusion matrix is notedwith an overhead view of the device, where the green color represents the enabled sensor data from thecorresponding position(s).

resulted in the lowest accuracy. Using a single sensor on the left or right side, the recognition
accuracy of motions from different directions was not significantly affected by the position of thesingle sensor.
We believe that one of the main characteristics of sensor data generated by mist is the presence ofsignificant noise. When using low-dimensional data as the training set, overfitting may occur. If themodel parameters are relatively excessive compared to the available data, it can lead to learning thenoise or outliers present in the training set. As the data dimension increases, particularly throughsensor configuration allowing for richer pattern generation, the increase in training data helps
balance the bias observed in low-dimensional settings and improves the accuracy. However, it isimportant to consider the limitations imposed by the system design and sensor configuration, suchas the need for stable sensor coverage by mist in the proposed system. Based on the results, itshows that employing a symmetric configuration with three sensors achieves an accuracy of over
90% represents one of the promising solutions.
5.3.2 Amount of Training Dataset. Since the system would be considered a daily interface, itis necessary to understand the requirement of training data to initialize a robust classifier. We
conducted an experiment to examine the impact of training set size on accuracy. We divided all thedata into 10 groups, with each group containing 120 data samples. Then, using a random selection
approach, we sequentially selected one, two, three, and up to nine groups for training. During
each model training iteration, we randomly chose a group of data that was not included in theProc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

Seeing the Wind: An Interactive Mist Interface for Airflow Input

444:13Fig. 10. (a) The impact of the number of training groups on the classifier’s performance. (b) Within-user andcross-user accuracy of 3 sensors and 2 side sensors on 10 motions.

training set as the testing set. Given our findings from the within-user test, we observed comparable
accuracy using data from the two-side sensor configuration compared to using all 3 sensors. And
examined the influence of training data size on this two-side sensor configuration too.
The results are shown in Figure 10 (a). In general, it needs at least 7 groups of training data toobtain accuracy around 90% for a 3-sensor configuration. As for the two-side sensor configuration,
it can only be close to 85% accuracy with around 8 groups of training data. It turns out that theclassifier still requires a considerable amount of training data to maintain and stabilize at highaccuracy, covering the randomness of airflow inputs with sample quantity and diversity.
5.3.3 Cross-user Test. To evaluate the system’s universality, a cross-user experiment was conducted.
The results will indicate whether the system can be smoothly used by users who have not beenrecorded by the model. We employed data from 11 participants as the training set and used thedata from the twelfth participant as the testing set, iterating this process for all participants. Finally,
we recorded the average results and plotted them in Figure 10 (b). For the configuration with 3
sensors, the cross-user test achieved an accuracy of 79.30% (SD=10.48); for the configuration with 2
sensors, the accuracy was 72.08% (SD=8.52). This demonstrates the system’s degree of universality
given the current training data scale and motion set.
By observing the confusion matrix of each participant, we noticed that for 3 sensors configuration,
Flip Book, Jumping Jack and Squat can maintain a relatively high accuracy of around 84% during
the test. Furthermore, the middle sensor indeed enhanced the robustness of the model from theresults under the cross-user circumstance.
5.3.4 Recognizing the Motions Causing Similar Airflow. To further test the performance of oursystem, we conducted an experiment on gesture recognition to push the limit of the mist-sensing
technique further. Most existing blowable interface designs that based on microphone input areconcentrate on detecting blowing [21, 31, 38]. We believe that mid-air gestures, such as waving andclapping, can also generate airflow similar to the tested motions. Considering that the proposedsystem essentially detects variations in mist formation, it is possible for users to generate airflow
with similar characteristics with different motions during the interaction. Therefore, in this section,
we incorporate gestures into the classifier. By doing so, we aim to test how the mist-sensing technique handles confusion when confronted with similar inputs and a higher number of classification
categories.
We tested 4 different motions, namely Wave and Clap, performed on both sides of the device(Figure 11). Participants performed Wave on both sides using their respective Left and Right hands.
Other experiment settings were consistent with the previous study.
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:14

Min et al.

Fig. 11. Gesture set tested in supplementary study: (a) Wave Left. (b) Wave Right. (c) Clap Left. (d) Clap
Right. The blue arrows in the figure represent the expected direction of airflow. The green arrows in (a) and(b) represent the direction of the movement.

Fig. 12. (a) Confusion matrix between Blow, Wave and Clap. (b) Confusion matrix of 14 motions combining
10 motions set in the previous study and gestures.

As shown in Figure 12 (a), when testing between 6 motions of Blow, Wave and Clap from bothsides, the accuracy is 69.38% (SD=6.89). We observed a significant decrease in model accuracy and abias towards right-sided motions when these motions were combined together. Additionally, there
was noticeable confusion between the left-sided Wave and Clap. One possible reason could be that
the ROCKET classifier used in this study was selected to handle the complex noise and randomness
generated by mist with a larger set of motions. In the case of limited samples and similar airflow,
the three-dimensional data may be insufficient compared to the number of model parameters.
When bringing the gestures into the 10 motions tested in the previous study, the accuracy is80.28% (SD=3.94). Compared to the 10 motions set in the previous study, an increase in the numberof categories can lead to a greater overlap of features among different classes. This feature conflict
can increase the model’s confusion, making it more challenging for the model to find effective
decision boundaries. In this case, the cross-user accuracy dropped to 70.25% (SD=8.05), which isinadequate for an interactive interface. For this prototype based on a humidifier, which usually
serves as a stationary desktop device, using a few motions to control the humidifier’s functionality
or surrounding IoT devices is sufficient. If there is a need to recognize more complex airflow
patterns, richer data features and a larger number of samples would be needed.
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

Seeing the Wind: An Interactive Mist Interface for Airflow Input

444:15The selection of the number and types of motion categories would be a crucial factor affecting
the accuracy of the mist sensing-based detection task. When motions can generate similar airflow
while the number of categories is relatively small, the performance of the model is more likely
to decrease. In such cases, using lighter machine-learning methods might be more effective inmanually observing and designing features. On the other hand, when the number of categories islarge, it is advisable to increase the data dimension or sample size, providing the model with moremeaningful features. In real-world scenarios, for example, if the device is placed on the right sideof a table, removing the motions related to right-side directions or merging several directions of
the same motion can enhance the recognition performance of the model.
6

EXAMPLE APPLICATIONS

Fig. 13. The example applications of mist sensing: (a) Lights can be turned on by detecting the airflow
generated by the user when sitting down. (b) Users can use predefined mid-air gestures to replace hotkeys tocontrol the computer, like screenshots. (c) Lights can be turned off automatically when the user closes a book.
(d) Users can be greeted when entering a room. (e) The airflow generated by motion can be used to time orcount the exercise training. (f) Users can use real-world mist to control or shape virtual objects. The whitearrows in the figure represent airflow directions.

Mist sensing is based on the variations in airflow, which can be generated in both explicit andimplicit activities, regardless of whether the user interacts with the system with intention or not[39]. Additionally, due to the composition of mist as tiny particles, it can capture finer and smaller
interference. This enables it to support various activities, including blowing, waving, jumping,walking by, or sitting down within a certain spatial area. In this section, we provide 6 usage scenariosof the prototype to demonstrate the concept of mist sensing with airflow.
Typically placed on a desk, the system can sense input through the mist to control computers orother IoT devices. For example, users can switch desktops by waving their hands or quickly takescreenshots by clapping (Figure 13 (b)). The proposed system can enable intriguing interactions
by employing gestures that share common characteristics [16]. Furthermore, in the absence ofintentional input, the mist can also capture other activities of the user at the desk. For example,
when a user sits down and generates subtle disturbances in airflow that usually go unnoticed, theProc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:16

Min et al.

system can automatically turn on the lights (Figure 13 (a)). Similarly, the airflow produced when
the user stands up or closes a book after reading can be captured to turn the lights off once again(Figure 13 (c)).
When the system is placed in the hallway or entrance area, it supports indoor traffic sensing. For
example, when placed on a cabinet or small table near the entrance, the mist can detect the airflow
generated when the door is opened or closed (Figure 13 (d)). Since the entrance is not a primaryarea of indoor activity, it could exclusively capture the disturbances caused by people entering andexiting. This makes the system support counting the number of people in an office or laboratory. It
could also be used to automate motions such as turning on the lights when the door is opened or
having a smart voice assistant greet the visitor.
When the system is located on the table in the living room or the floor in spacious areas, it canmonitor the user’s exercises. Fitness exercises like squats make it hard to generate airflow in aclear and concentrated direction. Instead, they create ambiguous and complex airflow around the
body during exercise. These airflows are difficult to capture by microphones because they are too
subtle to create evident wind noise. However, they can be detected by mist due to the presence oftiny particles susceptible to interference. For example, when the system is placed on a table, it canmeasure the duration or quantity of squats or jumping Jacks performed by a person in front of thetable (Figure 13 (e)). Similarly, when placed on the floor, it can detect the duration and quantity ofpush-ups or sit-ups performed by a person.
In addition, the system also has the potential to be used for various and unique purposes. Mist
sensing can create tangible interfaces where users can physically manipulate the shape of mist tointeract with digital content or virtual environments. This can include shaping or molding the mist
to control objects. For example, when a computer graphics designer is creating smoke for visual
effect, the designer can use a desktop humidifier as an input method to control the general shapeof the smoke in the virtual environment based on real-world mist and refine the details further(Figure 13 (f)).
7

DESIGN LESSONS

From the design and evaluation of the proposed method of sensing the mist, the system is able torecognize the mist variation caused by human motions (10 motions tested) and has the potential
capability to interact with daily objects, exercise, and hand gestures. Such an implicit mist-based
interface has great potential in daily life because human motion can change airflow and thus cause
a variation in mist. However, according to our exploration, designing a mist-based interface is noteasy. Since the core of sensing is the airflow change, the airflow normally follows a propagation
process from the motion to the mist and is affected by various factors, and as a freeform object, thevariation of the mist is also affected by various elements. Therefore, in this section, we summarize
the lessons that need to be considered in the development and design of using mist as an interface.
7.1

The Distance between the Mist Source and Motion

The core reason affecting the variation of mist is the intensity of airflow. As tested in Section 3.4,
when the airflow velocity is too low (e.g., less than 1 m/s), the airflow change does not cause thevariation of mist very well, and the ability to sense will decrease drastically. When the airflow
velocity is in a specific interval, the airflow will bring a more obvious variation to the mist, and thedetection capability of the system will be greatly improved at this time. Therefore, in the design, inorder to bring better performance to the system, we normally need to ensure that the intensityof airflow affecting mist is in an acceptable range (for example, the wind speed of 1-2m/s givenin this paper). The airflow intensity will be affected by distance and velocity, so the design needs
to be balanced between the mist source and the motion source. For example, after determining
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

Seeing the Wind: An Interactive Mist Interface for Airflow Input

444:17the airflow velocity that the motion can generate, the mist source needs to be placed within areasonable distance so that the system has a good perception capability.
7.2

Design with Reasonable Variety of Motions

The motion leads to an instantaneous change in airflow, which then undergoes propagation anddecay. However, the change from motion mapping to airflow to detectable mist is a high-dimensional
to low-dimensional process. The change of motion is various, while the variation of mist is limited.
Therefore, in this paper, we choose 10 motions to demonstrate the classification capability of thesystem and test the performance using different numbers of sensors. As a result in Section 5.3.1,
the performance of the system improves as the number of sensors increases. This is due to theincrease in valid information as well as useful features due to the increase in data dimensionality,
but this rise is definitely limited. Therefore, in order to ensure the performance of the system, we
believe that the number of designed target recognition motions should not be too many, and about
5 is a suitable range from the result in Section 5.3.1. In this case, the designed interface system canbe a blowable interface, a desk-based object interaction detector, or a simple home fitness recorder.
7.3

Practical Dataset Collection

We rely mainly on the machine learning technique to design and develop a prototype for theinterface. However, the requirement for datasets has always been a concern for such systems. In
this paper, we tested the performance of training datasets with different amounts on the system.
During the data collection, each user performed each motion for about the 40s. With such data
length, the classifier can achieve a relatively good performance. And we also suggested before thatthe recognition objects of such interfaces need not be too many. Considering the classifier designof low-dimensional time series, compared with other types of recognition systems the dataset canbe kept at a modest size in the mist-interface design.
7.4

Avoid the False Positive by other Motions

As mentioned before, the core of mist change is the change of airflow. Section 5.4.3 tests therecognition capability of the system with motions that generate similar airflow. The results showthat the performance of the system is challenged but still has some recognition capability. Therefore,
we believe there is still a need to consider a combination of factors in the design to minimize theinterference of other motions to the system, which can cause false positive problems. Such aconsideration can be reflected in the choice of the system location, which can be placed as far as
possible in places with less airflow interference, avoiding places like near air conditioners, next towindows, corridors with high pedestrian traffic, and so on.
8

DISCUSSION

The underlying principle of the mist sensing proposed is based on the variation in ToF sensor
values caused by the airflow. It could seemingly perform a similar capability of motion recognition
with proximity sensors such as radar. Integrating mist with proximity sensors could redefine theirdetection range and capabilities. At the same time, users benefit from natural visual feedback
due to the visibility of the mist. They can perceive the mist’s location and changes, aiding theirunderstanding of the interaction’s effects. As mentioned in Section 7.1, different motions can resultin similar airflow changes and result in false positives. However, this characteristic of mist sensing
allows for affording users greater flexibility and freedom to interact with the system in ways thatsuit their preferences. In practical applications, instead of defining specific motions, users could
be prompted to make the mist drift to the left or disperse the mist for three seconds. Finally, thistechnique opens up possibilities for new application scenarios, especially those involving mid-air
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:18

Min et al.

interaction, such as olfactory interfaces [6, 40] or mist displays [29, 46]. These domains can leverage
mist sensing to introduce innovative input methods.
On the other hand, the proposed method has its limitations in terms of applicable scenarios. Firstis precision and accuracy. Mist sensing may lack the precision of traditional proximity sensors.
Another concern would be the mist sensitivity. The sensitivity of mist sensing is a complicatedissue related to the speed of airflow, location of sensors, and type of mist/smoke as we discussed in
Section 3. Although it could also be directly adjusted by threshold or machine learning model, themist’s characteristic causes it to be highly sensitive to airflow variations, making it less suitablefor environments with strong wind or air turbulence. Implementation and maintenance of the
hardware, encompassing sensors and misting systems, can be more intricate compared to traditional
proximity sensors [12, 44]. Users may require some adaptation time to familiarize themselves withthis novel interaction method, as they need to comprehend how to interact effectively with mist.
In summary, mist sensing holds the potential to inspire innovative interaction designs in specificcontexts, such as the indoor installations demonstrated in our prototype. However, when consideringreal-world applications, its complexity becomes apparent. Mist sensing is influenced by numerous
factors, including the type of mist or smoke, nozzle geometry, nozzle orientation, initial spray
velocity, the quantity and spatial distribution of sensors, and the maximum effective range ofinteractions, among others. These challenges encompass the need for robust mist or smoke control,
optimal sensor placement and configuration, and versatile algorithms capable of interpreting noisy
and low-dimension sensor data effectively. Additionally, addressing issues related to scalability anduser acceptance will be important in practical applications.
9

LIMITATIONS AND FUTURE WORK

The paper presented a ToF sensor-based method detecting the mist variation caused by humanmotions and thus could be employed as an interface to recognize human motions with different
application scenarios. The airflow acts as the medium, transmitting the movement from the humanto the variation of mist. However, we did not pay close attention to the airflow investigation inthis paper. As we tested in Sections 3 and 5, we only focused on the ultimate mist variations. In
fact, studying the airflow characteristics could be clearer to figure out the ability of mist sensing
and helpful for design. For example, it could be useful to understand how intensive the airflow isnormally generated when conducting the exercise. Such a quantitative description would be moreaccurate in controlling the mist source’s placement.
When we designed the system, each sample length was selected as 4s. It is longer than a normal
human motion recognition system. This is because we have to consider the airflow propagationand decay process, particularly for the motion of blowing. It is necessary to capture the useful
information as much as possible. We argued that it still fits naturally into the user’s habits. Thisis because when doing this type of blowing motion, it usually lasts for a longer period of time.
Nevertheless, for other exercise motions, it is able to decrease the sample length to obtain a rapid
response of classification, as these motions would be quicker in general.
In addition, we only adopted the humidifier’s mist as the main source. It is still interesting totest other types of common mist or smoke in daily life to be sensed by different types of motions.
Based on this, studying and designing an easily configurable interface system could be useful to beadapted to many types of mist or smoke. However, the density of mist is still a significant factor
that needs to be considered. A low density of mist would be difficult to capture and prone to beaffected by airflow. A high density of mist tends to cause discomfort in an enclosed space. Thus,
creating a flexible system to fit more common mist could be the next step to making a more practicalinteractive paradigm.
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

Seeing the Wind: An Interactive Mist Interface for Airflow Input

444:19In this paper, the performance of sensing and recognition capability was mainly studied. We did
not conduct a user study to collect the qualitative result from the user-end, for example, the likelevel and agreement score. As a daily system, recruiting and conducting a detailed user study isvaluable to figure out any human factors-related findings. Since we employed a ubiquitous item
(humidifier) to create the system, it could be the next step to allow the users to use our system intheir home or office to interact with the humidifier as an exercise counter or electronics controller.
10

CONCLUSION

This paper demonstrates a technique for mist sensing, which is able to be employed to recognize
human motion input. The airflow caused by human motion could affect the patterns of mist. Since
the mist is able to reflect the emitted light, the variation of mist can be sensed by ToF sensors.
Therefore, we first examined the capability of a ToF sensor-based mist sensing system against thedifferent mist types and airflow characteristics. Based on this, we developed a prototype design
with a common humidifier to evaluate the performance of mist-based motion recognition. The
result showed that our system could achieve a 91.17% accuracy within-user, and 79.30% across users
in classifying 10 motions, including various types of blowing from different directions, interactingwith objects (flipping the book, waving the paper) and exercises (jumping-jacks and squats). We
also presented several application examples and design lessons to create and use such a mist-based
interface system. Using a very implicit and ubiquitous approach, we believe that the mist is able tobuild a more natural and interesting interface in users’ daily lives.
ACKNOWLEDGMENTS
This work was partially funded by JST PRESTO (Grant Number JPMJPR2134) and Fundamental
Research Funds for the Central Universities (Grant Number XJSJ23109).
REFERENCES[1] Karan Ahuja, Yue Jiang, Mayank Goel, and Chris Harrison. 2021. Vid2Doppler: Synthesizing Doppler radar data fromvideos for training privacy-preserving activity recognition. In Proceedings of the 2021 CHI Conference on Human Factors
in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/
3411764.3445138[2] Peter C. Barnum, Srinivasa G. Narasimhan, and Takeo Kanade. 2010. A Multi-Layered Display with Water Drops. ACM
Trans. Graph. 29, 4, Article 76 (jul 2010), 7 pages. https://doi.org/10.1145/1778765.1778813[3] Jessica Broscheit, Qi Wang, Susanne Draheim, and Kai von Luck. 2021. Towards Atmospheric Interfaces. In Proceedingsof the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction (Salzburg, Austria) (TEI ’21).
Association for Computing Machinery, New York, NY, USA, Article 52, 7 pages. https://doi.org/10.1145/3430524.3442458[4] Ulrich Burgbacher and Klaus Hinrichs. 2014. An Implicit Author Verification System for Text Messages Basedon Gesture Typing Biometrics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 2951–2954. https://doi.org/10.1145/2556288.2557346[5] Lisa Anneke Burr, Julius Šula, Julia Mayrhauser, and Alexander Meschtscherjakov. 2023. BREATHTURES: A First Step
Towards Breathing Gestures as Distinct Input Modality. In Extended Abstracts of the 2023 CHI Conference on Human
Factors in Computing Systems (Hamburg, Germany) (CHI EA ’23). Association for Computing Machinery, New York,NY, USA, Article 60, 6 pages. https://doi.org/10.1145/3544549.3585787[6] Yan Yan Cao and Naohito Okude. 2015. Scented Pebbles: Interactive Ambient Experience with Smell and Lighting. In
Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (Stanford, California,
USA) (TEI ’15). Association for Computing Machinery, New York, NY, USA, 409–410. https://doi.org/10.1145/2677199.
2690873[7] Wei-Hung Chen. 2015. Blowatch: Blowable and Hands-Free Interaction for Smartwatches. In Proceedings of the 33rd
Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI EA’15). Association for Computing Machinery, New York, NY, USA, 103–108. https://doi.org/10.1145/2702613.2726961[8] Yuxuan Chen, Yi Jin, Zhaoning Zeng, Shiyu Chen, Yang Zhou, Xinchi Xu, Jie Liu, and Guihuan Feng. 2023. VisaudiBlow:
Fine-Grained Blowing Interaction Based on Visual and Auditory Detection. In Extended Abstracts of the 2023 CHI
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:20

Min et al.

Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI EA ’23). Association for Computing
Machinery, New York, NY, USA, Article 311, 6 pages. https://doi.org/10.1145/3544549.3585829[9] Angus Dempster, François Petitjean, and Geoffrey I. Webb. 2020. ROCKET: exceptionally fast and accurate timeseries classification using random convolutional kernels. Data Mining and Knowledge Discovery 34, 5 (01 Sep 2020),
1454–1495. https://doi.org/10.1007/s10618-020-00701-z[10] Ethan Eddy, Erik J Scheme, and Scott Bateman. 2023. A Framework and Call to Action for the Future Development
of EMG-Based Input in HCI. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.
Association for Computing Machinery, New York, NY, USA, 1–23. https://doi.org/10.1145/3544548.3580962[11] Jackson Feijó Filho, Wilson Prata, and Thiago Valle. 2012. Breath Mobile: A Low-Cost Software-Based Breathing
Controlled Mobile Phone Interface. In Proceedings of the 14th International Conference on Human-Computer Interaction
with Mobile Devices and Services Companion (San Francisco, California, USA) (MobileHCI ’12). Association for Computing
Machinery, New York, NY, USA, 157–160. https://doi.org/10.1145/2371664.2371697[12] Tamil Selvan Gunasekaran, Ryo Hajika, Yun Suen Pai, Eiji Hayashi, and Mark Billinghurst. 2022. RaITIn: Radar-Based
Identification for Tangible Interactions. In CHI Conference on Human Factors in Computing Systems Extended Abstracts.
Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/3491101.3519808[13] Roy Harrison, David Beddows, Alan Jones, Ana Calvo, Célia Alves, and Casimiro Pio. 2013. An evaluation of someissues regarding the use of aethalometers to measure woodsmoke concentrations. Atmospheric Environment 80 (122013), 540–548. https://doi.org/10.1016/j.atmosenv.2013.08.026[14] Keisuke Hasegawa, Liwei Qiu, and Hiroyuki Shinoda. 2017. Interactive Midair Odor Control via Ultrasound-Driven
Air Flow. In SIGGRAPH Asia 2017 Emerging Technologies (Bangkok, Thailand) (SA ’17). Association for Computing
Machinery, New York, NY, USA, Article 8, 2 pages. https://doi.org/10.1145/3132818.3132833[15] Juro Hosoi, Yuki Ban, Kenichi Ito, and Shin’ichi Warisawa. 2021. VWind: Virtual Wind Sensation to the Ear by
Cross-Modal Effects of Audio-Visual, Thermal, and Vibrotactile Stimuli. In SIGGRAPH Asia 2021 Emerging Technologies(Tokyo, Japan) (SA ’21 Emerging Technologies). Association for Computing Machinery, New York, NY, USA, Article 18,
2 pages. https://doi.org/10.1145/3476122.3484848[16] Masoumehsadat Hosseini, Tjado Ihmels, Ziqian Chen, Marion Koelle, Heiko Müller, and Susanne Boll. 2023. Towards a
Consensus Gesture Set: A Survey of Mid-Air Gestures in HCI for Maximized Agreement Across Domains. In Proceedingsof the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for
Computing Machinery, New York, NY, USA, Article 311, 24 pages. https://doi.org/10.1145/3544548.3581420[17] Chiao-En Hsieh, Yu-Ta Chen, Hao-Chen Wang, Yang-Sheng Chen, Ping-Hsuan Han, and Yi-Ping Hung. 2020. Utilizing
Handheld Wind Feedback to Expand the Perception of Environmental Wind from Stationary Device. In ACM SIGGRAPH
2020 Posters (Virtual Event, USA) (SIGGRAPH ’20). Association for Computing Machinery, New York, NY, USA, Article
28, 2 pages. https://doi.org/10.1145/3388770.3407406[18] Frederic Kerber, Pascal Lessel, and Antonio Krüger. 2015. Same-side hand interactions with arm-placed devices using
EMG. In Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems.
Association for Computing Machinery, New York, NY, USA, 1367–1372. https://doi.org/10.1145/2702613.2732895[19] Sven Kratz and Maribeth Back. 2015. Towards accurate automatic segmentation of imu-tracked motion gestures. In
Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. Association
for Computing Machinery, New York, NY, USA, 1337–1342. https://doi.org/10.1145/2702613.2732922[20] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. 2011. HMDB: A large video database for human motion
recognition. In 2011 International Conference on Computer Vision. 2556–2563. https://doi.org/10.1109/ICCV.2011.6126543[21] Takahiro Kusabuka and Takuya Indo. 2020. IBUKI: Gesture Input Method Based on Breathing. In Adjunct Proceedingsof the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20 Adjunct).
Association for Computing Machinery, New York, NY, USA, 102–104. https://doi.org/10.1145/3379350.3416134[22] Miu-Ling Lam, Yaozhun Huang, and Bin Chen. 2015. Interactive Volumetric Fog Display. In SIGGRAPH Asia 2015
Emerging Technologies (Kobe, Japan) (SA ’15). Association for Computing Machinery, New York, NY, USA, Article 13,
2 pages. https://doi.org/10.1145/2818466.2818488[23] Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity
Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology(Tokyo, Japan) (UIST ’16). Association for Computing Machinery, New York, NY, USA, 321–333. https://doi.org/10.
1145/2984511.2984582[24] Eric C. Larson, Mayank Goel, Gaetano Boriello, Sonya Heltshe, Margaret Rosenfeld, and Shwetak N. Patel. 2012.
SpiroSmart: Using a Microphone to Measure Lung Function on a Mobile Phone. In Proceedings of the 2012 ACM
Conference on Ubiquitous Computing (Pittsburgh, Pennsylvania) (UbiComp ’12). Association for Computing Machinery,
New York, NY, USA, 280–289. https://doi.org/10.1145/2370216.2370261[25] Jaeyeon Lee and Geehyuk Lee. 2016. Designing a Non-Contact Wearable Tactile Display Using Airflows. In Proceedingsof the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). Association for
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

Seeing the Wind: An Interactive Mist Interface for Airflow Input

444:21

Computing Machinery, New York, NY, USA, 183–194. https://doi.org/10.1145/2984511.2984583[26] Diego Martinez Plasencia, Edward Joyce, and Sriram Subramanian. 2014. MisTable: Reach-through Personal Screens
for Tabletops. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario,
Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 3493–3502. https://doi.org/10.1145/
2556288.2557325[27] Vimal Mollyn, Karan Ahuja, Dhruv Verma, Chris Harrison, and Mayank Goel. 2022. SAMoSA: Sensing Activities with
Motion and Subsampled Audio. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6,
3 (2022), 1–19. https://doi.org/10.1145/3550284[28] Taeyong Moon and Gerard J. Kim. 2004. Design and Evaluation of a Wind Display for Virtual Reality. In Proceedings of
the ACM Symposium on Virtual Reality Software and Technology (Hong Kong) (VRST ’04). Association for Computing
Machinery, New York, NY, USA, 122–128. https://doi.org/10.1145/1077534.1077558[29] Mohd Adili Norasikin, Diego Martinez-Plasencia, Gianluca Memoli, and Sriram Subramanian. 2019. SonicSpray: A
Technique to Reconfigure Permeable Mid-Air Displays. In Proceedings of the 2019 ACM International Conference on
Interactive Surfaces and Spaces (Daejeon, Republic of Korea) (ISS ’19). Association for Computing Machinery, New York,NY, USA, 113–122. https://doi.org/10.1145/3343055.3359704[30] Makoto Oka and Hirohiko Mori. 2018. Proposal of Interaction Using Breath on Tablet Device. In Human Interface andthe Management of Information. Interaction, Visualization, and Analytics, Sakae Yamamoto and Hirohiko Mori (Eds.).
Springer International Publishing, Cham, 267–278.
[31] Shwetak N. Patel and Gregory D. Abowd. 2007. Blui: Low-Cost Localized Blowable User Interfaces. In Proceedings ofthe 20th Annual ACM Symposium on User Interface Software and Technology (Newport, Rhode Island, USA) (UIST ’07).
Association for Computing Machinery, New York, NY, USA, 217–220. https://doi.org/10.1145/1294211.1294250[32] Siyou Pei, Pradyumna Chari, Xue Wang, Xiaoying Yang, Achuta Kadambi, and Yang Zhang. 2022. ForceSight: NonContact Force Sensing with Laser Speckle Imaging. In Proceedings of the 35th Annual ACM Symposium on User Interface
Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA,
Article 25, 11 pages. https://doi.org/10.1145/3526113.3545622[33] Ismo Rakkolainen. 2008. Mid-Air Displays Enabling Novel User Interfaces. In Proceedings of the 1st ACM International
Workshop on Semantic Ambient Media Experiences (Vancouver, British Columbia, Canada) (SAME ’08). Association for
Computing Machinery, New York, NY, USA, 25–30. https://doi.org/10.1145/1461912.1461919[34] Ismo Rakkolainen and Antti Sand. 2013. A Movable Immaterial Volumetric Display. In SIGGRAPH Asia 2013 Posters(Hong Kong, Hong Kong) (SA ’13). Association for Computing Machinery, New York, NY, USA, Article 2, 1 pages.
https://doi.org/10.1145/2542302.2542305[35] Michael Rietzler, Katrin Plaumann, Taras Kränzle, Marcel Erath, Alexander Stahl, and Enrico Rukzio. 2017. VaiR:
Simulating 3D Airflows in Virtual Reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing
Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 5669–5677.
https://doi.org/10.1145/3025453.3026009[36] Deepak Ranjan Sahoo, Takuto Nakamura, Asier Marzo, Themis Omirou, Michihiro Asakawa, and Sriram Subramanian.
2016. JOLED: A Mid-Air Display Based on Electrostatic Rotation of Levitated Janus Objects. In Proceedings of the 29th
Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). Association for Computing
Machinery, New York, NY, USA, 437–448. https://doi.org/10.1145/2984511.2984549[37] Munehiko Sato, Rohan S. Puri, Alex Olwal, Yosuke Ushigome, Lukas Franciszkiewicz, Deepak Chandra, Ivan Poupyrev,
and Ramesh Raskar. 2017. Zensei: Embedded, Multi-Electrode Bioimpedance Sensing for Implicit, Ubiquitous User
Recognition. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado,USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 3972–3985. https://doi.org/10.1145/
3025453.3025536[38] Erika Sawada, Shinya Ida, Tatsuhito Awaji, Keisuke Morishita, Tomohisa Aruga, Ryuta Takeichi, Tomoko Fujii,
Hidetoshi Kimura, Toshinari Nakamura, Masahiro Furukawa, Noriyoshi Shimizu, Takuji Tokiwa, Hideaki Nii, Maki
Sugimoto, and Masahiko Inami. 2007. BYU-BYU-View: A Wind Communication Interface. In ACM SIGGRAPH 2007
Emerging Technologies (San Diego, California) (SIGGRAPH ’07). Association for Computing Machinery, New York, NY,
USA, 1–es. https://doi.org/10.1145/1278280.1278282[39] Albrecht Schmidt. 2000. Implicit human computer interaction through context. Personal Technologies 4, 2 (01 Jun 2000),
191–199. https://doi.org/10.1007/BF01324126[40] Sue Ann Seah, Diego Martinez Plasencia, Peter D. Bennett, Abhijit Karnik, Vlad Stefan Otrocol, Jarrod Knibbe, Andy
Cockburn, and Sriram Subramanian. 2014. SensaBubble: A Chrono-Sensory Mid-Air Display of Sight and Smell. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14).
Association for Computing Machinery, New York, NY, USA, 2863–2872. https://doi.org/10.1145/2556288.2557087[41] Youngbo Aram Shim, Jaeyeon Lee, and Geehyuk Lee. 2018. Exploring Multimodal Watch-Back Tactile Display Using
Wind and Vibration. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC,
Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.

444:22Min et al.

Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.
3173706[42] Craig Shultz and Chris Harrison. 2022. LRAir: Non-contact Haptics Using Synthetic Jets. In 2022 IEEE Haptics Symposium(HAPTICS). IEEE, 1–6. https://doi.org/10.1109/HAPTICS52432.2022.9765565[43] Yuta Sugiura, Masahiko Inami, and Takeo Igarashi. 2012. A Thin Stretchable Interface for Tangential Force Measurement.
In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (Cambridge, Massachusetts,USA) (UIST ’12). Association for Computing Machinery, New York, NY, USA, 529–536. https://doi.org/10.1145/2380116.
2380182[44] Yuta Sugiura, Gota Kakehi, Anusha Withana, Calista Lee, Daisuke Sakamoto, Maki Sugimoto, Masahiko Inami, andTakeo Igarashi. 2011. Detecting shape deformation of soft objects using directional photoreflectivity measurement. In
Proceedings of the 24th annual ACM symposium on User interface software and technology. Association for Computing
Machinery, New York, NY, USA, 509–516. https://doi.org/10.1145/2047196.2047263[45] Ippei Suzuki, Shuntarou Yoshimitsu, Keisuke Kawahara, Nobutaka Ito, Atushi Shinoda, Akira Ishii, Takatoshi Yoshida,
and Yoichi Ochiai. 2017. Gushed Light Field: Video Showcase of Aerosol-Based Fog Display. In Proceedings of the 2017
CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI EA ’17).
Association for Computing Machinery, New York, NY, USA, 453. https://doi.org/10.1145/3027063.3049774[46] Yutaka Tokuda, Mohd Adili Norasikin, Sriram Subramanian, and Diego Martinez Plasencia. 2017. MistForm: Adaptive
Shape Changing Fog Screens. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems(Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4383–4395. https://doi.org/10.1145/3025453.3025608[47] Chengshuo Xia, Xinrui Fang, Riku Arakawa, and Yuta Sugiura. 2022. VoLearn: A Cross-Modal Operable MotionLearning System Combined with Virtual Avatar and Auditory Feedback. Proceedings of the ACM on Interactive, Mobile,
Wearable and Ubiquitous Technologies 6, 2 (2022), 1–26. https://doi.org/10.1145/3534576[48] Zihan Yan, Yuxiaotong Lin, Guanyun Wang, Yu Cai, Peng Cao, Haipeng Mi, and Yang Zhang. 2023. LaserShoes:
Low-Cost Ground Surface Detection Using Laser Speckle Imaging. In Proceedings of the 2023 CHI Conference on
Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–20. https://doi.org/10.1145/3544548.3581344[49] Hui-Shyong Yeo, Erwin Wu, Daehwa Kim, Juyoung Lee, Hyung-il Kim, Seo Young Oh, Luna Takagi, Woontack Woo,
Hideki Koike, and Aaron John Quigley. 2023. OmniSense: Exploring Novel Input Sensing and Interaction Techniques
on Mobile Device with an Omni-Directional Camera. In Proceedings of the 2023 CHI Conference on Human Factors
in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–18. https://doi.org/10.1145/
3544548.3580747[50] Dingtian Zhang, Jung Wook Park, Yang Zhang, Yuhui Zhao, Yiyang Wang, Yunzhi Li, Tanvi Bhagwat, Wen-Fang
Chou, Xiaojia Jia, Bernard Kippelen, et al. 2020. OptoSense: Towards ubiquitous self-powered ambient light sensing
surfaces. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1–27.
https://doi.org/10.1145/3411826[51] Dingtian Zhang, Jung Wook Park, Yang Zhang, Yuhui Zhao, Yiyang Wang, Yunzhi Li, Tanvi Bhagwat, Wen-Fang Chou,
Xiaojia Jia, Bernard Kippelen, Canek Fuentes-Hernandez, Thad Starner, and Gregory D. Abowd. 2020. OptoSense:
Towards Ubiquitous Self-Powered Ambient Light Sensing Surfaces. Proc. ACM Interact. Mob. Wearable UbiquitousTechnol. 4, 3, Article 103 (sep 2020), 27 pages. https://doi.org/10.1145/3411826[52] Yang Zhang and Chris Harrison. 2015. Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture
Recognition. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte,NC, USA) (UIST ’15). Association for Computing Machinery, New York, NY, USA, 167–173. https://doi.org/10.1145/
2807442.2807480[53] Yang Zhang, Yasha Iravantchi, Haojian Jin, Swarun Kumar, and Chris Harrison. 2019. Sozu: Self-powered radio tags
for building-scale activity sensing. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software andTechnology. Association for Computing Machinery, New York, NY, USA, 973–985. https://doi.org/10.1145/3332165.
3347952[54] Yang Zhang, Gierad Laput, and Chris Harrison. 2018. Vibrosight: Long-range vibrometry for smart environment
sensing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. Association for
Computing Machinery, New York, NY, USA, 225–236. https://doi.org/10.1145/3242587.3242608[55] Jan Zizka, Alex Olwal, and Ramesh Raskar. 2011. SpeckleSense: fast, precise, low-cost and compact motion sensing
using laser speckle. In Proceedings of the 24th annual ACM symposium on User interface software and technology.
Association for Computing Machinery, New York, NY, USA, 489–498. https://doi.org/10.1145/2047196.2047261

Received 2023-07-01; accepted 2023-09-22

Proc. ACM Hum.-Comput. Interact., Vol. 7, No. ISS, Article 444. Publication date: December 2023.