# SkInteract: An On-body Interaction System Based on Skin-Texture Recognition

Manuel Prätorius, Aaron Scherzinger, Klaus Hinrichs

► To cite this version:

Manuel Prätorius, Aaron Scherzinger, Klaus Hinrichs. SkInteract: An On-body Interaction System Based on Skin-Texture Recognition. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.425-432, 10.1007/978-3-319-22723-8_34. hal-01610824

HAL Id: hal-01610824

https://inria.hal.science/hal-01610824

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# SkInteract: An On-Body Interaction System Based on Skin-Texture Recognition

Manuel Prätorius, Aaron Scherzinger, Klaus Hinrichs

Visualization and Computer Graphics Research Group (VisCG)
University of Münster, Einsteinstraße 62, 48149 Münster, Germany
{manuelpraetorius, scherzinger, khh}@uni-muenster.de

**Abstract.** In this paper we propose *SkInteract*, a system for on-body interaction utilizing the diverse texture of the human skin. We use an area fingerprint sensor to capture images and locate the corresponding area within a previously created map of the skin surface. In addition to the location of the sensor it is possible to calculate its orientation with respect to the reference map. This allows to assign arbitrary semantics to areas of the user's skin and to use the rotation as an additional input modality. In order to evaluate the feasibility of *SkInteract* a user study with a preliminary prototype was conducted. We propose two different interaction concepts which are based on either attaching a fixed sensor to a wearable device or using a moveable sensor, for instance attached to a pen, to perform on-body input.

**Keywords:** mobile, input, sensors, fingerprint, skin, on-body, smartring, smartwatch, biometrics

## 1 Introduction

Miniaturization has fostered the research and development of small mobile and wearable interaction devices. The diminishing interaction space of these devices, which comes at the cost of their mobility, has become a major challenge in the field of human computer interaction.

Recently an increasing number of mobile devices have integrated small fingerprint sensors as a means of user authentication. In this paper we propose the *SkInteract* system which utilizes those sensors to expand the limited interaction space of wearables to the skin surface. With our approach especially area type sensors provide the opportunity to design novel interaction techniques based on biometric features of the skin texture by creating a map of the skin surface and recognizing the position and orientation of sensor images within the mapped regions. Specific user interfaces can then be realized by assigning semantics to distinct areas of the map and utilizing the rotation as an additional modality. Because of a user's proprioception the interaction on the skin can be performed eyes-free which is advantageous especially in a mobile context.
## 2 Related Work

Several input systems have been proposed which are based on recognizing the body part that is used for interaction. This allows to perform user authentication, map different semantics to the body parts, select sets of input metaphors, or increase the precision of the interaction [1, 2, 3].

Harrison et al. [4] have presented a system which is able to recognize different objects and parts of the finger such as the pad, tip, knuckle, and nail by analyzing the acoustic signal produced while tapping on a touch surface. Another system by Harrison et al. [5] appropriates the forearm and the hand as an input surface by analyzing the vibrations caused by finger taps on those areas. The enhancement of expressiveness in multi-touch interaction by utilizing fiduciary-tagged gloves is presented in [6]. A finger-worn device which is equipped with a camera to recognize the texture the user is touching is presented in [7]. The device is able to distinguish 22 different materials, e.g., wood, jeans, and skin.

The first user interface based on fingerprint recognition has been proposed by Sugiura and Koseki [8]. Their approach is to assign commands to the user's fingertips and perform the corresponding action once a fingerprint is recognized.

The user performance in a scenario where a push button is able to identify the user's fingers was evaluated in the work of Wang and Canny [9]. In order to recognize the fingers they used colored markers attached to the fingertips. They compared several input situations considering input speed between multi-finger and single finger tapping.

In contrast to previous work on user interaction on the human skin we propose the novel approach to create a map of the skin texture which enables us to recognize the location and orientation of a sensor. This technique appropriates the skin for both discrete and continuous input.

## 3 Prototype System Outline

To evaluate the capabilities of the *SkInteract* approach we have created a simple hardware prototype and software implementation which is capable of both generating maps of specific skin texture areas and detecting the position and rotation of input images within those areas.

### 3.1 Hardware Setup

Since capacitive area sensors have already been integrated into mobile devices such as Apple's iPhone we have chosen to use this type of sensor for our first prototype. We use an FPC1011F3 model which has a resolution of 363 dpi. The resulting size of the recorded images is 152 x 200 pixels for an area of 10.64 x 14.00 mm. We read data from the sensor using the SPI interface of a RaspberryPi and transfer each complete image via Ethernet to a standard laptop with an Intel Core i7-2670QM CPU which performs the expensive image processing and recognition tasks. Using this
setup, which is shown in Figure 1a, we are able to retrieve four images per second from the sensor.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** (a) Preliminary *SkInteract* prototype setup. (b) A user capturing images of the skin with the area fingerprint sensor. (c) An example of a position and orientation as indicated during the user study. The arrows depict the eight orientations of the sensor which were tested for each position.</figcaption></figure>

## 3.2 Scanning and Mapping the Hand

To create a map of a region of the skin surface a simplified version of the image stitching method proposed in [10] is applied to the images captured by the sensor. For both the image stitching as well as the later recognition and classification of the input images our method utilizes scale-invariant feature transform (SIFT) without the need for any preprocessing [11]. Although more sophisticated feature extraction and image enhancement methods have been proposed for the use of fingerprint images in user authentication applications, SIFT features work well in our context as they provide a more generalized method which is also applicable to regions of the palm and the wrist.

In the stitching process new images are added sequentially to an intermediate stitching result by computing matches between the SIFT features of the two images and calculating a homography matrix using random sample consensus (RANSAC) [12]. For robustness we perform several checks regarding the determinant of the matrix as well as the transformed bounding box of the input image.

For the prototype system evaluation presented in section 4 we have focused on creating separate sub-maps of the skin surface as shown in Figure 2 instead of a single global map, since the stitching of larger numbers of images can lead to accumulated distortion effects caused by the projection of non-planar soft tissue into two-dimensional space, depending on the particular region of the skin. Although this problem could be alleviated to a certain degree by applying an additional global optimization to minimize the accumulated errors, continuous interaction could be realized more efficiently by specifying transitions between the separate sub-maps, thus narrowing down the recognition to such areas.
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Several examples of stitched skin textures. The green squares mark the areas that were used for the user study.</figcaption></figure>

## 3.3 Region Detection

For a new input image we first compute the SIFT features. We then iterate over the list of previously recorded areas of the fingers, palm, and the wrist (which are each represented by a stitched reference image) and match the features of the input image and the reference image. Feature matching is performed using an exhaustive brute-force approach in order to obtain the best matching results. For each feature we compute the two best matches and only keep matches where the ratio of the distance between the best and the second-best match is better than a threshold *t*. Our experiments have shown that *t* = 0.8 generally seems to be a good value for our prototype implementation.

For the region with the largest number of good matches we perform RANSAC to calculate a homography matrix *H* that transforms our input image with respect to the reference image. If we cannot find enough good matches or the determinant of *H* is near zero we reject the input image. Otherwise we apply *H* to transform the bounding box of the input image, and use the vertices and edges of the transformed image to compute the rotation and translation with respect to the reference image (see Figure 3). This allows us to not only recognize the region of the finger or palm but also to determine the orientation of the sensor which can be used as an additional interaction modality.

# 4 System Evaluation

To evaluate the feasibility of *SkInteract* we have conducted a user study and performed some additional tests to characterize the system parameters. We have focused
on three main aspects of the system which are the recognition rate, runtime duration of the recognition, and the required size of the sensor for use in a potential user interface.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Transformation of the sensor image</figcaption></figure>

## 4.1 Recognition Rate

In order to quantify the recognition rate of our prototype system we have conducted a user study with a total of 8 participants from our department, 7 male and one female. For each participant we have captured reference images for 11 different skin regions (green squares in Figure 2) with the stitching method outlined in section 3.2. Examples for the stitched reference images are given in Figure 2. For each of the reference regions we have captured 8 test images with different orientations of the sensor (gray arrows in Figure 1c). Image capturing has been triggered manually once the sensor was placed appropriately and the preview of the image did not show significant artifacts caused by the oiliness and the moisture of the skin (see section 4.4). The acquisition of test images was randomized regarding both region and orientation. Each test image was then matched using the algorithms described in section 3.3. From the total of 704 test images 698 were classified correctly, resulting in a recognition rate of 99.15%. From the total of 6 recognition failures 5 were rejected as non-matchable. One test image was captured at the wrong position but has correctly been recognized and thus resulted in a false positive. Other than that no false positive matches occurred in the user study.

## 4.2 Computational Cost of the Area Recognition

We have tested the duration of the area recognition and homography matrix calculation using OpenCV's standard implementations of SIFT, brute force feature matching, and RANSAC. For a small number of reference regions (corresponding to a small area of the skin surface) the computation of the SIFT features for the input image limits the processing speed, while for larger areas the matching of the features becomes the limiting factor. For our tests we have gradually increased the area A of the
scanned reference region in each testing cycle by adding an additional image of the
sensor, each corresponding to an area of 1.49 cm² and not overlapping with any of the
other reference images. For A > 12 cm² the influence of the feature computation step
becomes negligible and the required processing time t for the matching step and
homography can be estimated by t = A·f with f = 0.03 s/cm². Although a reduction of
matching time could presumably be achieved by implementing a more sophisticated
matching strategy such as the Locally Sensitive Hashing algorithm proposed in [13],
the impact on the recognition rate of the system would have to be evaluated to consid-
er its applicability for *SkInteract*. Since no optimization has been applied, the pro-
cessing time has to be considered an upper limit. More adapted features and feature
selection may reduce the feature set, and thus give a significant speed-up.

4.3 Required Size of the Input Region

To evaluate the required sensor size we have gradually decreased the input area size
by cropping the images captured by our sensor from their center. In each of several
testing cycles conducted with a single test person we have reduced the image size by
12 pixels in both x- and y-dimension, starting with an image size of 152 x 152 pixels
which corresponds to a surface area of 10.64 x 10.64 mm². In each testing cycle we
have tested 10 input images against the same set of 5 reference regions. The recogni-
tion rate in our tests remained constant at 100% up to an input image size of 56 x 56
(which corresponds to a surface area of 3.92 x 3.92 mm²) and decreased drastically
afterwards. For an input image size below 32 x 32 pixels or 2.24 x 2.24 mm² the sys-
tem could not perform the recognition at all. The minimum area size required for a
robust recognition could potentially be decreased even further by using a fingerprint
sensor with a higher resolution compared to our 363 dpi model.

4.4 Drawbacks of the Capacitive Fingerprint Sensor

During the user study we have observed that the moisture and oiliness of the skin
have a considerable impact on the quality of the images captured by the sensor, mak-
ing it necessary for some users to occasionally clean the sensor. While this was not
necessary in most cases it may restrict a continuous interaction for some users. Other
types of sensors like ultrasonic or optical sensors, however, may not be affected by
this problem.

Additionally, capacitive fingerprint sensors have a short sensing range and there-
fore need a good contact to the skin to capture images of appropriate quality, which
makes it difficult to sense areas around knuckles and joints.

5 Design Space and Applications

*SkInteract* can be used as a basic platform for several different user interface applica-
tions. These applications can be divided into two main categories, the first being a
system with a fixed sensor and the second a system with a moveable sensor.
An example of a system with a fixed sensor is a smartwatch or a smartring which allows to interact with an integrated sensor using the fingers and palm of the other hand (see Figure 4a). In addition to mapping different functions onto the selected interaction areas of the skin surface, rotating could be used as a gesture for additional functions, for instance controlling the volume in a music player. An alternative would be to map distinct functions to discrete states defined by the relative orientation between the user's hand and the sensor. However, it would be necessary to perform an in-depth user study to evaluate the usability and comfort of potential interaction areas on the hand under the constraints of a smartwatch or a smartring scenario.

As the required size of the input area of the sensor in the current setup is only 3.92 x 3.92 mm² we hypothesize that an appropriate sensor can be integrated into a small moveable device. Such a device could be a pen as shown in Figure 4b or a thimble with the sensor attached to its tip. Besides the discrete selection of functions by tapping on the respective areas, the continuous motion of a moveable sensor on the skin surface could be used to provide gestural interaction.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** (a) Fixed sensor setup integrated into a smartwatch. (b) Moveable sensor attached to the tip of a pen.</figcaption></figure>

To alleviate the cognitive load of remembering all of the input areas and the function mapping they could be visualized by an augmented reality head-mounted display.

Furthermore, privacy issues regarding both the potentially required data transfer as well as the storage of the captured images would have to be taken into account to avoid any exploitation of the biometric information included in the data.

# 6 Conclusion and Future Work

In this paper we have presented *SkInteract*, a novel system that appropriates the skin as an interaction surface. A device which can sense the texture of the skin can be utilized to generate maps of the skin which allows to recognize the position and the
orientation of such a sensor. We have conducted a user study to evaluate the feasibil-
ity of the approach with a prototype which is based on a capacitive area fingerprint
sensor.

In future work we plan to test several setups with different sensors and to design
actual interfaces to explore the interaction space.

7 References

1. Holz, C., & Baudisch, P. (2013, October). Fiberio: A touchscreen that senses fingerprints. In *Proceedings of the 26th annual ACM symposium on User interface software and technology* (pp. 41-50). ACM.

2. Harrison, C., Ramamurthy, S., & Hudson, S. E. (2012, February). On-body interaction: armed and dangerous. In *Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction* (pp. 69-76). ACM.

3. Holz, C., & Baudisch, P. (2010, April). The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (pp. 581-590). ACM.

4. Harrison, C., Schwarz, J., & Hudson, S. E. (2011, October). TapSense: enhancing finger interaction on touch surfaces. In *Proceedings of the 24th annual ACM symposium on User interface software and technology* (pp. 627-636). ACM.

5. Harrison, C., Tan, D., & Morris, D. (2010, April). Skinput: appropriating the body as an input surface. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (pp. 453-462). ACM.

6. Marquardt, N., Kiemer, J., & Greenberg, S. (2010, November). What caused that touch?: expressive interaction with a surface through fiduciary-tagged gloves. In *ACM International Conference on Interactive Tabletops and Surfaces* (pp. 139-142). ACM.

7. Yang, X. D., Grossman, T., Wigdor, D., & Fitzmaurice, G. (2012, October). Magic finger: always-available input through finger instrumentation. In *Proceedings of the 25th annual ACM symposium on User interface software and technology* (pp. 147-156). ACM.

8. Sugiura, A., & Koseki, Y. (1998, November). A user interface using fingerprint recognition: holding commands and data objects on fingers. In *Proceedings of the 11th annual ACM symposium on User interface software and technology* (pp. 71-79). ACM.

9. Wang, J., & Canny, J. (2004, April). FingerSense: augmenting expressiveness to physical pushing button by fingertip identification. In *CHI'04 Extended Abstracts on Human Factors in Computing Systems* (pp. 1267-1270). ACM.

10. Brown, M., & Lowe, D. G. (2007). Automatic panoramic image stitching using invariant features. *International journal of computer vision*, 74(1), 59-73.

11. Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. *International journal of computer vision*, 60(2), 91-110.

12. Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. *Communications of the ACM*, 24(6), 381-395.

13. Auclair, A., Cohen, L. D., & Vincent, N. (2008). How to use SIFT vectors to analyze an image with database templates. In *Adaptive Multimedia Retrieval: Retrieval, User, and Semantics* (pp. 224-236). Springer Berlin Heidelberg.