Smatable: A System to Transform Furniture into
Interface using Vibration Sensor
Makoto Yoshida1 , Tomokazu Matsui1 , Tokimune Ishiyama1 , Manato Fujimoto2 , Hirohiko Suwa1 , Keiichi Yasumoto1
1

Nara Institute of Science and Technology, Ikoma, Nara 630-0192, Japan
E-mail: {yoshida.makoto.yn3, matsui.tomokazu.mo4, ishiyama.tokimune.io3, h-suwa, yasumoto}@is.naist.jp
2
Osaka Metropolitan University, Osaka, Osaka 558-8585, Japan
E-mail: manato@omu.ac.jp

Abstract—Recently, with the spread of smart houses, thesmartness of housing equipment and home appliances has progressed, and the functionality and usability of interfaces between
people and equipment and between people and home appliances
have become important factors. Currently, the main interfaces
are remote controls, smartphone applications, and even voicerecognition. Furthermore, research is also being conducted oninterfaces that can be operated without having the device at
hand using cameras and radio waves. However, special equipment
must be installed for operation, and compatibility with roomdesign has become an issue. In this research, we proposed asystem to transform existing furniture into an interface rather
than providing a new interface. The proposed system focused onvibration sensors that are small, inexpensive, and can be attached
to existing furniture or hidden from view. To evaluate the proposed system, an experiment was conducted to transform existing
furniture into an interface for swiping by simply attaching thevibration sensor to the existing furniture. Specifically, the system
attaches four vibration sensors with synchronized output signals
to a table and uses a CNN to learn the vibration data obtained
from the sensors to predict the direction of the swipe. As a result,when the table and person swiping were fixed, the system could
predict the swipe with an accuracy of over 0.86.
Index Terms—Interfaces, Vibration sensors, Deep learning

I. I NTRODUCTION
Recently, there has been an active trend toward the useof smart home equipment and home appliances. A factor
that determines the usability of smart home appliances is the”interface between humans and smart home appliances.” Inparticular, the interface of smart home appliances used in daily
life must be easy to use, simple to use, fun to use, and improve
the quality of life. The interaction provided by these interfaces
is a key elemental technology that determines whether smart
appliances will be integrated into our daily lives. Currently,
the mainstream interfaces between smart home appliancesand humans include remote controls, portable devices suchas smartphone applications, and voice recognition. Although
voice recognition has the advantage of being non-contact, it iscommand-based, with one command for each action, and when
repeated operations are performed, such as turning pages, theuser must also repeat the pronunciation, which can be stressful.
In addition, efforts are being made to embed highly functioned
interfaces in home appliances themselves, and products suchas electrostatic touch sensors and touch panels that function asinterfaces are being developed. Embedding interface functions
have the advantage of being intuitive and natural. However,home appliances and furniture are specially designed for thispurpose, and the scale becomes large, resulting in costs otherthan for the original function and design. On the other hand,
non-contact interfaces are not only systems that use voicerecognition, but also those that use gestures such as fingers
and arms. For example, there is an interface that uses a motion
sensor to operate the CG on the screen when arranging threedimensional objects in the initial stage of work, such as design,
although it is not for domestic use [1].
This method requires a grasping motion in space, so amonitor is necessary, and the motion sensor must be installed
so that the hand can be seen, which limits the design offurniture when applied in the home. Additionally, there arecamera-based systems that use special gloves to detect human
hand movements [2], and systems that use an infrared lighton the palm to capture images with an infrared camera
to recognize arm and hand movements and input data [3].
Moreover, a camera that captures the movement of a hand on
a GUI projected by a projector and inputs commands havebeen studied [4], [5].
Since cameras can easily obtain a large amount of information and identify moving objects, they can be used not only
for gesture recognition as an interface but also for various
applications such as human behavior recognition and anomaly
detection in the home. However, the amount of information
handled is large, and there are concerns about privacy invasion,
especially in the home. Therefore, the camera’s presence inresidence causes psychological stress. Recently, methods using
radio waves and vibration sensors have been proposed as
moving object recognition methods less likely to infringe on
privacy than cameras [6]–[14]. Table 1 shows a comparisonof various sensors that can detect human movements. Vibration sensors, in particular, have the advantage of being easy
to install, cost-effective, and have little concern about the
infringement of privacy. Additionally, it is characterized byits concealability, which allows the sensor to be placed in an
invisible position, and is a great advantage when considering
its use as an interface for devices used in the home, where
design is important.
In this research, we propose a method that uses this feature

TABLE IS ENSING M ETHOD C OMPARISON TABLE

CameraMicrophone
Laser
InfraredRadio wave
Vibration

Ease ofprivacy
considerations

Small
installation
size

Abundanceof
information

Lo light

Sensorcovertness

cost

×
×
✓
✓
✓

×
✓
×
×
×
✓

✓
×
-

×
✓
✓
✓
✓
✓

×
×
×
✓

×
✓
×
×
✓

of vibration sensors to connect smart home appliances andusers simply by simply attaching sensors without changing
the design of existing furniture and use it as a furnitureinterface that blends into our lives. In this paper, we took atable as one piece of furniture and examined the detectionof swiping motions on the table using a vibration sensor
as a basic function for making it an interface. Section IIof this paper introduces existing studies and issues related
to this research. Section III describes the proposed system
configuration, Section IV discusses the signals output from thevibration sensor, Section V describes the swipe identification
method using CNN, its evaluation results and discussion, andfinally, Section VI provides a summary and future work.
II. RELATED WORKThis section introduces several existing studies related tothis research, summarizes their problems, and describes theissues to be addressed in this research and the proposed
methodology.
A. Tangible user interface
Patten et al. [13] studied and proposed a system called“Tangible User Interface.” This system uses an object on atable to which a knob called a sensor puck is attached. The
object is equipped with a coil, which is used to make anelectromagnetic coupling with the table to track the object’s
state and position. In addition, an image reflecting the stateand position of the knob is projected onto the table by aprojector, allowing for simultaneous input and feedback fromthe user. This research is excellent in that it recognizes theuser’s manipulation of the object and reflects that feedback
as a projected image, thus providing a direct manipulation of
the object and a more intuitive interface. However, it requires
dedicated equipment for both the sensor pack on the table andthe table itself. Therefore, it is not possible to use the interface
in the way we want to achieve it, which is to simply attach asimple sensor to an existing piece of furniture.
B. Research on gesture recognition using camera indoors
In recent years, some research has been conducted using
camera-based gesture recognition as an interface. Goto etal. [4] used a camera with a pan-tilt function and a projector
to project GUI images on a table, wall, or other desired
location in the home. By implementing a combination of
background subtraction and skin color extraction from video,
hand gestures are detected as if they were touch panels.

Simone et al. [5] have constructed an interface by combininga camera, a projector, and an infrared laser. This interface
not only recognizes human actions on the projected image
but also performs object recognition. In the example of theuse case of cooking support, the projector projects a cooking
recipe and also realizes recognition of food ingredients. In
addition, multi-touch is also realized using infrared lasers.
However, these methods are not only large in scale but also
require the installation of a camera in a position where theprojected surface and hands are within the angle of view,and nothing obstructs the camera so that the camera canrecognize the projected GUI and hands, which limits the
positioning relationship with furniture and other objects. In
addition, the camera-based method not only raises privacy
concerns by installing a camera inside a room but also creates
psychological concerns even when the camera is not in use,since the camera is installed in a position where it is easily
visible to the user due to the angle of view. Furthermore, froma practical standpoint, the amount of light stored in the image
sensor is reduced in many cases where the lighting in thehome is darkened, resulting in blurred images and difficulty
in increasing the frame rate, which increases the difficulty ofidentifying moving objects. In terms of operation, the amountof information handled at the time of data acquisition is large,
making pre-processing computationally expensive.
C. Recognition of handwritten input using acoustic features
Other interfaces use acoustic signals to detect human writing
actions. There is research on a pen interface that uses a
microphone attached to the body of a pen to pick up acoustic
signals generated on a desk during writing and recognize thecharacters in printed type [15], and research on a pen interface
that uses a microphone to record writing sounds generated neara table from a distance, and combines them with character andlanguage models estimated from the writing sound data [16],and others. These methods are superior in terms of cost andhave a small installation scale, but they are easily affected byambient noise, and there is a concern that the recognition rate
may decrease under noise conditions.
D. Gesture recognition using electromagnetic waves
Kim et al. [14] used millimeter waves to perform gesturerecognition by machine learning using the amplitude variationof the peak component of the impulse response and decision
trees. This method is superior in that it does not take privacy
information. However, it requires the installation of radio wave
absorbers and horn antennas to suppress the effects of reflected
electromagnetic waves, which is large-scale. In addition, there
is concern that the use of metal furniture may change thepropagation of radio waves.
E. Positioning of this study with respect to existing researchissuesWhen considering the method of moving object recognition used as an interface in the home, infrared rays, lasers,
electromagnetic waves, and vibration sensors are candidates,Fig.2

Fig.1

Sensor mounting position
Piezoelectricelement

Bottom View

Magnet18.5
Steel cover

150

75

42

3

CH4
CH1

Piezoelectricelement

AmplifierCH1

X-axis
CH1
CH2
AMP

CH2

CH2

45

2.5

CH3

CH3

3

CH4

Units:[cm]PCM
Recorder4ch
synchronousrecorder
CH1
CH2
CH3
CH4

Y-axisCH3
CH4
AMP

Y-axisVibration sensor

CH3

CH2

CH1

X-axisCH4

Fig.3

Fig. 2. block diagram
Fig2.

3

Dynamic range extension low noise circuit＋

Fig. 1. Sensor Structure
Fig.1

3

except for cameras and microphones due to privacy concerns.
However, infrared rays and lasers require sensors to be placedin a position with good visibility, which limits the degreeof freedom in arranging furniture. In addition, since electromagnetic waves are reflected, furniture using electromagnetic
waves is easily affected by the surrounding environment, andthere is a problem that the materials used for furniture arelimited. On the other hand, the vibration sensor does notprovide as rich information as the camera, so privacy canbe preserved. However, it has the disadvantage that it cannot
obtain information when it is stationary, but it can be said that
it is suitable for detecting moving objects because it contains
a lot of information about moving objects. In addition, it has
the advantage of being low cost and easy to arrange in large
numbers.
In addition, it has the greatest feature that it can be usedeven when the sensor is hidden as long as it is in a place
where vibration is transmitted, which is a great advantage insensing at home where design is important. From the above,
we believe that vibration sensors are suitable as interface
devices for smart homes. In this research, we will develop andpropose a vibration sensor that can be used as an interface with
furniture just by attaching it without changing the design ormaterial of the furniture. In this research, we used CNN, which
is often used in speech recognition and image recognition.
The reason is that the band of the signal recorded by thevibration generated when swiping the table is the voice band,and it was confirmed that the spectrogram clearly shows thecharacteristics of the swipe. The swipe detection range was
set to 4 directions on 2 axes, and detection was limited tothe axial direction only. Specifically, we aimed to detect four
actions: swiping up (UP), swiping down (DOWN), swiping
left (LEFT), and swiping right (RIGHT).
III. S ENSOR S YSTEM
A. Sensor structure
Fig. 1 shows the structure and installation of the table
and sensors used to detect swipes. Two pairs of vibration
sensors are aligned in a straight line on the back of the table
and mounted in orthogonal directions in the X and Y axes,

X-axis and Y-axis
independent amplifiers－

For dynamic
range expansion

For low noise

Positive and negative
power supplies

Signal addition
For low noise

CH1,2 X-axis

For low noise

CH3,4 Y-axis

Shield Structure

Floating power supply

Fig3

4

Fig. 3. Developed sensor amp system

respectively, for a total of four sensors to capture the vibrationthat occurs when a finger swipes across the table. The sensor
is attached to the back of the table so that the sensor cannot beseen from the table so as not to impair the design. In previous
studies, when fabricating vibration sensors, a weight of acertain mass was placed on top of the piezoelectric element toincrease the sensitivity of the electrical signal generated when
the object is subjected to acceleration. However, weight cannot
be used when the piezoelectric element is mounted on the backside of a table. As shown in Fig. 1, the piezoelectric elementis placed on a steel-covered plate attached to the table sidewith double-sided tape and fixed with a magnet, thus utilizing
magnetic force as a substitute for the gravity obtained with aweight.
B. Sensor system block
Fig. 2 shows a block diagram of the developed sensor
system. Since there are four sensors in total, each signal isamplified by a dedicated amplifier and recorded synchronously
by a multi-channel recorder.
C. Sensor amplifierFig. 3 shows the 4-channel amplifier system fabricated.
Vibration signals are known to have a very large dynamic
range, i.e., the difference between strong and weak signals.
The amplifiers were designed to have rail-to-rail outputs using
both power supplies to ensure a dynamic range. The amplifier
circuit and substrate are designed to reduce noise and ensure
the signal-to-noise ratio by using parallel synthesis for the
input stage amplifier.

Fig.5

Fig.4

3.5

CH3
CH1

3.0

■ Waveform

CH4■ Swipe direction →Time[s]→Time[s]
CH3

CH3

CH1

CH2

CH4

(a)

CH1

CH2

CH4

(b)

39

Amplitude Amplitude Amplitude Amplitude

CH3

Down

Amplitude Amplitude Amplitude Amplitude

Amplitude Amplitude Amplitude Amplitude

Amplitude Amplitude Amplitude Amplitude

CH2

Up

3

4

5

13

14

15

16

17

37
3.0

CH1

26
38

27
39

28
40

29
41

CH3

CH3

CH1

6

7

8

18

19

20

CH3

31

30

43

44

9

10

11 12

21

22

23 24

CH2

34

33

32

CH4

42

45

46

48

Units:[cm]Vibration sensor
position

Center of the table

Vibrator

35 36
47

150
Legend:→Time[s]

CH2

2

25

→Time[s]

CH1

1
13

CH4

Left

Right
CH1

3.5

13

CH2

45

Waveform of the vibration sensor signal when a swipe

Excitation
position

CH2

Fig. 5. Measurement environment for table vibration transfer characteristics

CH4

CH4

(c)

(d)

Fig6

6

TABLE II

Fig. 4. Time axis data of vibration sensor output during the swipe

In addition, the amplifier boards for the X-axis and the
Y-axis are housed in the housing as a set of two channels.
Additionally, it has a shield structure using an aluminum caseto avoid the influence of radiation noise from electric powerlines and other electronic devices when installed indoors.
Furthermore, the power supply has a floating power supply
structure using batteries, and separate power supplies are installed for the X-axis and the Y-axis, thereby avoiding induced
noise from power lines. These systems are for experiments andare large, but they can be made smaller by using chip partsand laminated substrates. The sample rate of data acquisitionis 44.1 kHz, and quantization is performed with 16Bit PCM.
The data to be recorded is recorded as one file for every twochannels and recorded as two stereo WAV files, one each forCH1 and CH2 for the X-axis and CH3 and CH4 for the Y-axis.
IV. A NALYSIS OF VIBRATION SENSOR SIGNAL DURING
SWIPING

A. Confirmation of time domain waveform of vibration signal
Fig. 4 shows the time-axis waveforms obtained from thevibration sensor by swiping a finger on the table. The fourchannels of the sensor signal, CH1, CH2, CH3, and CH4, aresynchronized in time. In Fig. 4, (a) is the time axis waveformwhen swiping to the right from the left, (b) is the time axis
waveform when swiping to the left from the right, (c) is thetime axis waveform when swiping down from the top, and (d)is the time axis waveform when swiping up from the bottom.
Swipes were made near the center of the table.
At first, we thought that the signals of the sensor near thestart position of the swipe and the sensor near the end positionof the swipe would change as follows. We expected that theamplitude of the sensor near the start position would gradually
decrease as the swiping finger moved, and conversely, thesignal from the sensor near the end position would gradually increase. Contrary to this, we could not find a clear
increase/decrease change in the amplitude of the time domain
waveforms on each axis.
B. Transmission characteristics of vibration in table
To investigate the relationship between the distance fromthe vibration source and the output of the vibration sensor

E XCITATION FREQUENCY.

100

200

400

Frequency[Hz]
1K

2k

4k

8K

10K

in more detail, we conducted the following experiment. As
shown in Fig. 5, the table surface was divided into sections of
130 mm in length and width at equal intervals, and 48 points
of intersection were vibrated by a vibrator at constant power,
one at a time, with sine waves of 8 different frequencies asshown in Table 2, and the sensor output level was recorded.
For the data collected at 48 locations, the distance wascalculated from the relationship between the excitation position and the sensor position, and the relationship between the
distance and the sensor output level was checked. Initially, we
thought that the sensor output level was inversely proportional
to the distance. Fig. 6 shows the results for CH4 excitation at
400 Hz. Contrary to expectations, the amplitude level showed
no inversely proportional relationship to the distance between
the sensor and the exciter. This indicates that it is difficult to
detect the direction of the hand at the time of swiping onlyby the amplitude level.
C. Auditory evaluation of vibration sensor signals
Since the recorded vibration data was recorded in PCM
and its frequency range is the same as the audible range, we
confirmed the vibration signal by hearing it as shown in Fig. 7.
We confirmed the vibration signals by placing CH1 on the left
ear and CH2 on the right ear on the X-axis, and were able torecognize them as sound and clearly feel the movement of thefingers. Similarly, when the vibration signal was checked byplacing CH3 in the left ear and CH4 in the right ear on the
Y-axis, the movement of the fingers could be clearly felt.
D. Confirmation by spectrogram of vibration signal
From the results of the auditory confirmation experiment
in section IV-C, we thought that when the vibration signal
was converted to sound, humans would be able to perceive
movement as changes in pitch and tone. Therefore, in thisresearch, instead of using the amplitude information as itis, we considered performing movement detection using thespectrogram as a feature amount. Fig. 8 shows the spectrogram
calculated using FFT (Fast Fourier Transform) after recording

Distance between vibration source and sensor VS sensor output【CH4】
Fig.8

0.0

■ Spectrogram

Frequency[Hz] →Frequency[Hz] →The recorded vibration signals can be heard by humans. This is because-50.0
signals recorded at a sampling rate of 44.1kHz are in the audible range.
→Time[s]Frequency[Hz] →Frequency[Hz] →Frequency[Hz] →Frequency[Hz] →Frequency[Hz] →Frequency[Hz] →Frequency[Hz] →Frequency[Hz] →Frequency[Hz] →CH4Frequency[Hz] →

-40.0

CH3

Down

Up

Frequency[Hz] →

400Hz

-30.0

Frequency[Hz] →CH2Frequency[Hz] →-20.0

Frequency[Hz] →Sensor output RMS [dBFs]

CH1

Fig.9

Left

Right

-10.0

→Time[s]→Time[s]→Time[s]■Swipe direction

CH3

CH3

-60.0
0.0

20.0

40.0

60.0

80.0

CH1

100.0

CH2

CH1

CH3

CH3

CH2

CH1

CH4

CH1

CH2

CH2

CH4

CH4

Distance[cm]
When the vibration
data of the opposing sensors (CH1, CH2
(c)
(d)
(b)
(a) and CH3,
CH4)
corresponding
to
the
X
and
Y
axes
were
playedbackassound,
I
Fig. 6. Distance between vibration source and sensor VS sensor output[CH4]Fig. 8. Spectrogram of the signal at the swipe
could feel the movement of the hand swiping on the table.
CH4

CH3

CH3

Up
CH2

CH1

Distance between vibration source and sensor VS sensor output
CH1
CH2
【CH4】

Left

Right

CH4

Down

Collect vibration data when swiping a table

fig7

10

Preprocessing

Up, Down, Left, Rightlabeled in 4 directions

STFT processed data ×4ch

CH4

fig8

Fig. 7. Auditory evaluation of vibration sensor signals

Data for 2 seconds per swipe

CH1

CH2

CH3

CH4

STFT

Four signals are
acquired synchronously

Fig.9(501,178)(501,178)(501,178)(501,178)

concatenate

Fig10.

CH1+2+3+4
X(1002, 356)

11

11

Fig. 9. Preprocessing of sensor signal data

the signals of CH1, CH2, CH3, and CH4 in synchronization.
Swipes were made near the center of the table.
In Fig. 8, (a) is swiped from left to right (hereafter referred to as RIGHT), (b) is when swiped from right to left
(hereafter referred to as LEFT), and (c) is swiped from top
to bottom (hereinafter referred to as DOWN), and finally(d) is the spectrogram when swiping from the bottom to the
top (hereinafter referred to as UP). The obtained spectrogramshows that, for horizontal swipes, the signal from the sensor
at the starting point of the swipe shows a faint upward-sloping
striped pattern, and the sensor at the ending point shows a faintdownward-sloping striped pattern. On the other hand, for thedata in the vertical direction, the change in the density of the
striped pattern was small, and it was difficult to distinguishvisually, but it was confirmed that a similar relationship wasmaintained in some data.
These stripes are thought to correspond to changes in thefundamentals and harmonics, i.e., what is perceived by theauditory sense as changes in pitch and timbre. Based on
the above, we thought of using a pattern that combines four
spectrograms as input and identifying the swipe direction using
the temporal changes in timbre and pitch as features. Since
temporal changes in pitch and timbre are expressed as changes
in the pattern when the spectrogram is captured as an image,
we investigated classification using CNN (Convolutional Neural Network), which is used in image recognition.

V. S WIPE DIRECTION ESTIMATION USING CNN
A. Data acquisition and preprocessing
When we checked the signals obtained from the four sensors
in section IV-D as a spectrogram, we found that a characteristic
pattern appeared (Fig. 8), and furthermore, the pattern differeddepending on the direction of swiping. Therefore, we decided
to combine the four spectrograms into one, as shown below,
and treat them as a single image-like matrix connected to theswipe direction. Fig. 9 shows the details of the preprocessing
performed before deep learning. Data is cut for each swiping
direction (UP, DOWN, LEFT, RIGHT) for two seconds, whichcontains one swipe, and converted into a spectrogram by STFT
(window function is Han window, frame size is 1000 samples).
Since signal data is obtained synchronously from the 4-channel
sensors, the same process is performed for each of the fourchannels, and then the resulting four matrices for the fourchannels are combined as a single matrix, labeled and storedwith the swiped direction (UP, DOWN, LEFT, RIGHT).
B. Acquisition of training and validation data
Data acquisition for training and validation consisted of
10 swipes per session for each swipe direction (up, down,
left, right), and nine sessions of data were acquired in eachdirection. Swipes were made near the center of the table. In

998×352×64

499×176×64

9 Sessions

Conv2D(3×3)+relu

MaxPooling(2×2)

Doropout Flatten Dense Doropout(0.25)(0.25)

4

128

128

128

499×176×64
Conv2D(3×3)+relu

5620736

UP

CH1+2+3+4

Input(1002×356)

Quadratureclassification

499×176×64

Up

DOWN
LEFT

Down

RIGHT
Dense

Outputsoftmax

Right

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

10

Up

1 session

1000×354×64

5620736…

1002×356×1

Down

10
Right

Fig. 10. Layer structure of the CNN

14

10

Left

Fold1

addition, when he checked the signal level of the obtained 2second swipe data, it was lower than expected, so he uniformly
performed a numerical amplification (multiplication) of six
times (15.6 dB).

3-Fold cross validation

1sesstion

Fig11.

Left

Training data set
6 sessions

Fold2

Validation data set
3 sessions

Fold3

Fig.12

As mentioned above, recognizing patterns in a spectrogram
grouped into a single matrix is similar to image recognition,
so in this research, we used CNN, which is often used inimage recognition, for deep learning. Fig. 10 shows the layer
structure of the CNN used. The obtained matrix is used as
an input and trained by a CNN consisting of 2 convolution
layers with 64 filters, a pooling layer, 2 dropout layers, and2 fully connected layers. This configuration was created withreference to the configuration used for character discrimination. Specifically, the array size of the input layer is an arrayof (1002, 356) obtained by synthesizing four channels into one
two-dimensional array using the STFT result of one channel.
Therefore, the number of arrays was adjusted accordingly.
The convolution size of the subsequent convolution layer is(3 × 3), and the activation function is ReLU. The numberof convolution filters is 64, and two convolution layers areconnected. The pooling layer provided after the convolutional
layer has a (2 × 2) configuration with 64 layers. After that,
one dropout layer with a dropout ratio of 0.25 is added,
followed by flattening through a fully connected layer, again
with a dropout ratio of 0.25, followed by flattening through afully connected layer, and finally four outputs corresponding
to the directions (UP, DOWN, LEFT, RIGHT) with softmax
functions outputs.
D. Learning methods
As shown in Fig. 11, 6 of the 9 sessions of swipe datawith 10 swipes per session obtained in section 5.2 wereused for training, and the remaining 3 sessions were used asvalidation data to perform 3-Fold cross-validation and confirm
recognition accuracy. In the learning using CNN, the batch
size was set to 10, and the learning model was constructed by
performing 24 epochs of learning. In constructing the learningmodel, the loss function is set to “categorical crossentropy”
and the optimization algorithm “adm” is used. Fig. 12 shows
the learning curve and loss curve for one of the 3Folds, aswell as the confusion matrix resulting from predicting a 4-way
swipe of the data for validation using the resulting training
model.

• 3foldのうちFig. 11.1つの学習・ロスカーブ・混同行列を示す
3-fold cross-validation using 9 sessions of data

True label

C. CNN layer configuration

Fig.13

Right

28

0

2

Left

0

28

0

2

Down

0

0

30

0

UP

15

0

0

0

0

30

Right

Left

Down

UP

Predicted label

Fig. 12. An example of a learning curve and loss curve and a confusion
matrixE. Comparison by person and by table

Fig.13

Fig.12

Next, in order to confirm whether the swipe direction
can be identified even with different tables, as shown in
Fig. 13(a), we prepared two additional tables and conducteda similar experiment. The table on which the experiment was
first performed was A, and the added tables were B and C,
respectively. Of the two tables, Table C chose a low table thatwas completely different from the other two. All the people
performing the swipe motion were the same in the experiment,
and the accuracy was confirmed by 3-fold cross-validation as
in the experiment described earlier. The results are shownin Table 3. In all cases, the accuracy was higher than 0.9,
although there was a slight decrease in accuracy in some cases
when the table was changed.
Next, as shown in Fig 13(b), verification was performed
when different people swiped the same table. As in theprevious experiment, the results confirmed for each personby three-part cross-validation showed that the lowest accuracy
was 0.86.
F. Accuracy in models trained on other tables and participants
As shown in Fig. 14 (a), we created a learning model using
swipe data from two of the three tables that were not subject toestimation, and conducted experiments to estimate the swipe
direction for the remaining one table. Fig. 14 (b), shows theresults of an experiment in which a learning model is createdusing data from two non-targets and tested with the remaining

14

Fig.14

Fig.13training data

A

Table A

CH4

Table B

Trained
model

CH3
CH4

CH1

CH3

A

Trainedmodel

Person A

CH2

A

CH4

Person A

Person B

Person C(b)(a)
Fig.1315
Fig. 13. Evaluation using three types of tables and three participants

TABLE III
ACCURACY AND LOSS WHEN USING A PER - PERSON , PER - TABLELEARNING MODEL (3- FOLD CROSS - VALIDATION )Table A
Table B
Table C
Person A
Person B
Person C

Accuracy
0.98
0.94
0.90
0.98
0.86
0.92

Loss
0.05
0.32
0.60
0.04
0.61
0.47

Fixed conditions
Person A
Person A
Person A
Table A
Table A
Table A

one. The result is as shown in Table 4, and the result is a large
drop in accuracy.
G. Verification of accuracy by adding data for one session of
the relevant table and participants
In order to consider how to improve the accuracy, we createda learning model by adding the data from the table that tries
to predict the swipe direction for one session to the learning
data as shown in Fig. 15(a), and predicted the swipe direction.
This is a study with a view to making it practically usable as
an interface by adding a little unknown data.
Additionally, as shown in Fig. 15(b), we added the data of
the person whose swipe direction is to be guessed for one
session to the learning data, created the learning model again,
and then Predicted the swipe direction. Results are shownin Table 5. In a per-participant experiment, all participants
improved to an accuracy of 0.70 or better. In the table-bytable experiment, Table C, which is a very different type oftable, showed only limited improvement in accuracy, but allother tables improved to 0.71 or better.
VI. CONCLUSIONS AND FUTURE WORKIn this paper, on a table, which is a familiar piece offurniture in the home, the identification of human swipe
operations was performed using only the vibration transmitted
to the table without using a camera. The system can be
implemented simply by attaching a simple vibration sensor,
and there is a possibility that it can be used as an instantinterface anywhere, not just on the table.
The prediction was performed using a CNN, inputtingchanges in the vibration signals received from four of his

FooterA

B

AC

Trained
model

C

B

Trainedmodel

C
Predict

Predict A

B

CB

Validation data

Predict
Trainedmodel

Predict B
Trained
model

C

C

Table CAC

Table A
A

CH2

CH1

CH
2

B

training data

Predict C

CH3
CH1

Validation data

B

A

B

Predict

A

A

Participants are
fixed at A

The table isfixed at A

(b)

(a)

Fig. 14. Evaluation of learning models using data from different tables andparticipants

TABLE IV
ACCURACY AND LOSS OF THE TRAINING MODEL USING DATA EXCLUDINGTABLES AND PERSONS TO BE PREDICTED

Table A
Table B
Table C
Person A
Person B
Person C

Accuracy
0.56
0.25
0.22
0.50
0.39
0.67

loss
1.30
2.59
5.03
2.89
3.95
0.88

Training data
Table B,C
Table A,C
Table B,A
Person B,CPerson A,C
Person A,Bproprietary synchronized synchronous vibration sensors as
combinations of patterns obtained from the spectrogram. In
the experiment, three persons each swiped a table for multiple
sessions, and the table vibration was recorded using four
vibration sensors. The recorded data was used to determine
the direction of the swipe, and when the vibration data of
only the person who swiped the table was used for 9 sessions
and confirmed by 3-fold cross-validation, the classification
accuracy was higher than 0.86
Additionally, we created a learning model using 6 sessionsof swipe data, 3 sessions each for the other 2 persons,excluding the person who swiped, and determined the swipe
direction for the 3 sessions, but the accuracy decreased toless than 0.39. When data from the person who swiped wasadded for one session, the accuracy improved, and predicts
were made with an accuracy of more than 0.71.
Similarly, We created training data using 6 sessions out of
9 sessions of vibration data on the table to be predicted andconfirmed the prediction of the swipe direction for the remaining 3 sessions of vibration data by 3-fold cross-validation. As
a result, although the accuracy varied depending on the table,
we were able to make predictions with high accuracy of 0.90
or more.
In addition, we created a learning model using 6 sessions ofswipe data collected from two other tables, excluding the tablefor which we wanted to determine the swipe direction. Using
this learning model, the swipe direction was determined for3 sessions of data in the swipe direction determination target
table. As a result, the accuracy decreased to 0.22 or less.
Therefore, when we added the data of the table swiped forCBTrained
model

A

C

B

Trainedmodel

Predict

Add only
1 session

A

AC

Trainedmodel

Add only
1 session

A

B

C

A

Trainedmodel

A
Predict

B

Predict

Add only
1 session

B

A

B

C

FooterTrainedmodel

Add only Predict
1 session

B

C

A

B

Predict

Add only
1 session

C(a)

Trainedmodel

Add only
1 session

C
Predict(b)

Fig. 15. Evaluation on a learning model with one additional session of data
to be predicted

TABLE V
ACCURACY AND LOSS WHEN ADDED TO ONE SESSION TRAINING DATA

Table A
Table B
Table C
Person A
Person B
Person C

Accuracy
0.73
0.90
0.43
0.75
0.72
0.76

loss
0.78
0.36
4.66
1.85
1.30
0.73

Training data
Table B,C +A(1Session)
Table A,C +B(1Session)
Table B,A +C(1Session)
Person B,C +A(1Session)
Person A,C +B(1Session)
Person A,B +C(1Session)

only one session, the accuracy improved for the two tables,
excluding the table with a significantly different shape, and it
became possible to estimate with an accuracy of 0.73 or more.
In this paper, we examined the possibility of using vibration
data during swiping as an interface, but the swipe position was
limited to the inside of the square connecting the four sensors,
and the swipe was performed roughly in the middle. Therefore,
it is necessary to verify the accuracy when there is a large bias
in the swipe position. Additionally, at this time, the collectionof training data for each table and each person operating thetable is necessary for stable use, which poses a challenge.
We plan to continue the following efforts so that it can beused as an interface with only simple learning in the future. In
this system, we have concluded that replaying the vibrations
as sound could clearly sense movement with human hearing,so we think that improving the deep learning architecture will
likely improve the detection accuracy. For this reason, we willconsider acquiring generalizability in the CNN layer structure
by reviewing the layer structure and architecture, such assynthesizing after performing convolution independently foreach channel. For practical use as an interface, we will plan
to use ensemble learning to improve accuracy and support
retries in cases of low confidence.
ACKNOWLEDGEMENTS
This work was supported in part by the Japan Society for the
Promotion of Science, Grants-in-Aid for Scientific Researchnumber JP20H04177.
R EFERENCES[1] N. Shunsuke, K. Norihiro, “Development of motion-gesture interface
to support 3d spatial modeling,” in japanese, THE 65th ANNUAL
CONFERENCE OF JSSD, 2018, Vol. 65, pp. 332–333.
[2] A. Kimura, F. Shibata, T. Tsuruta, T. Sakai, M. Oniyanagi, H. Tamura,“Design and Implementation of Minority-Report-Style Gesture,” injapanese, IPSJ Journal, 47 (4),2006, pp. 1327–1339.
[3] G. Yamamoto, K Sato, “PALMbit:A Body Interface Utilizing Light
Projection onto Palms,” in japanese, The Journal of The Institute of
Image Information and Television Engineers, Vol. 61, No. 6, pp. 797–
804, 2007.
[4] H. Goto, Y. Kawasaki and A. Nakamura, “Development of an information projection interface using a projector-camera system,” 19th International Symposium in Robot and Human Interactive Communication,
2010, pp. 50–55.
[5] S. Pizzagalli, D. Spoladore, S. Arlati, M. Sacco and L. Greci, “HIC:
An interactive and ubiquitous home controller system for the smart
home,” 2018 IEEE 6th International Conference on Serious Games andApplications for Health (SeGAH), 2018, pp. 1–6.
[6] S. Pan, T. Yu, M. Mirshekari, J. Fagert, A. Bonde, O. J. Mengshoel,
H. Y. Noh, P. Zhang, “ Footprintid: Indoor pedestrian identification
through ambient structural vibration sensing,” Proceedings of the ACM
on Interactive, Mobile, Wearable and Ubiquitous Technologies 1 (3),2017, pp. 1–31.
[7] J. Clemente, W. Song, M. Valero, F. Li and X. Liy, “Indoor Person
Identification and Fall Detection through Non-intrusive Floor Seismic
Sensing,” 2019 IEEE International Conference on Smart Computing(SMARTCOMP), 2019, pp. 417–424.
[8] S. Akiyama, M. Yoshida, Y. Moriyama, H. Suwa and K. Yasumoto,“Estimation of Walking Direction with Vibration Sensor based on
Piezoelectric Device,” 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops),
2020, pp. 1–6.
[9] K. Umakoshi, T. Matsui, M. Yoshida, H. Choi, M. Fujimoto, H. Suwa, K.
Yasumoto, “Non-contact person identification by piezoelectric-based gait
vibration sensing, in: Proceedings of the 35th International Conference
on Advanced Information Networking and Applications (2021), 2021,
pp. 1–14.
[10] Y. Kashimoto, M. Fujimoto, H. Suwa, Y. Arakawa and K. Yasumoto, “Floor vibration type estimation with piezo sensor toward
indoor positioning system,” 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 2016, pp. 1–6, doi:
10.1109/IPIN.2016.7743667.
[11] S. Pan, N. Wang, Y. Qian, I. Velibeyoglu, H. Y. Noh, P. Zhang, “Indoor
person identification through footstep induced structural vibration,” in:Proceedings of the 16th International Workshop on Mobile Computing
Systems and Applications (HotMobile’15), 2015, pp. 81–86.
[12] M. Mirshekari, J. Fagert, A. Bonde, P. Zhang, and H. Y. Noh, “Human
Gait Monitoring Using Footstep-Induced Floor Vibrations Across Different Structures. In Proceedings of the 2018 ACM International Joint
Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (UbiComp ’18). Association
for Computing Machinery, New York, NY, USA, 2018, pp. 1382–1391.
[13] J. Patten, H. Ishii, J. Hines, and G. Pangaro, “Sensetable: a wirelessobject tracking platform for tangible user interfaces. In Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems (CHI’01). Association for Computing Machinery, New York, NY, USA, 2001,
pp. 253–260.
[14] M. Kim,S. Moriyama, “Hand Gesture Recognition using Millimeterwave Channel Characteristics, ” in japanese, IEICE Communications
Society Conference.Vol. 1, No. BS-8-1, 2018, pp. S74–S75.
[15] M. Schrapel, M.Ludwig Stadler, and M. Rohs. 2018, “Pentelligence:
Combining Pen Tip Motion and Writing Sounds for Handwritten Digit
Recognition. In Proceedings of the 2018 CHI Conference on Human
Factors in Computing Systems (CHI ’18). Association for Computing
Machinery, New York, NY, USA, Paper 131,pp. 1–11.
[16] H. Yin, A. Zhou, G. Su, B. Chen, L. Liu, and H. Ma. 2020, “Learning
to Recognize Handwriting Input with Acoustic Features. Proc. ACM
Interact. Mob. Wearable Ubiquitous Technol. 4, 2, Article 64 (June2020), 26 pages.