# SoPhy: Smart Socks for Video Consultations of Physiotherapy

Deepti Aggarwal, Thuong Hoang, Weiyi Zhang, Bernd Ploderer, Frank Vetere, Mark Bradford

<figure><img src="image_1.png" /></figure>

## To cite this version:

Deepti Aggarwal, Thuong Hoang, Weiyi Zhang, Bernd Ploderer, Frank Vetere, et al.. SoPhy: Smart Socks for Video Consultations of Physiotherapy. 16th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2017, Bombay, India. pp.424-428, 10.1007/978-3-319-68059-0_44. hal-01679830

HAL Id: hal-01679830

https://inria.hal.science/hal-01679830

Submitted on 10 Jan 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# SoPhy: Smart Socks for Video Consultations of
Physiotherapy

Deepti Aggarwal¹, Thuong Hoang¹, Weiyi Zhang¹,
Bernd Ploderer¹,², Frank Vetere¹, Mark Bradford³

¹Microsoft Research Center for SocialNUI, University of Melbourne, Australia

²Electrical Engineering and Computer Science, Queensland University of Technology, Australia

³Department of Anaesthesia and Pain Management, Royal Children's Hospital, Australia

## ABSTRACT

While physiotherapists are increasingly organizing video consultations, assessment of lower body movements over video remains a challenge. We present a wearable technology, SoPhy that captures and presents information related to three key aspects of lower limb movements - range of foot movement, weight distribution and foot orientation. SoPhy consists of a pair of socks embedded with sensors for the patients to wear, and a web-interface that displays the captured information to physiotherapists in real-time. The objective of this demonstration is to offer first-hand experience of SoPhy and to create conversations around designing technologies for supporting bodily communication in video consultations.

## 1 Introduction

Over the last decade, physiotherapists have started using video conferencing tools to offer diagnostic and therapeutic advice to patients living in remote or rural areas [2,3]. While video consultations offer several benefits to patients and clinicians, our recent study [2] highlights the limitations of video technology in communicating the essential bodily information that physiotherapists need to formulate their assessment. The study showed that the physiotherapists found it difficult to understand the subtle differences in exercises such as depth of squats, and range of movements over video, particularly, for lower limb movements. The absence of the essential bodily information reduced clinician's confidence in assessing their patients and consequently, they offered less specific treatment over video than face-to-face consultations.

The importance of communicating rich information to clinicians is always emphasized [5], as it helps clinicians in making informed decisions and thereby, making the overall consultation effective. However, lesser attention has been paid to understand how new technologies could enhance clinician's ability to assess and treat patients during video consultations. We are interested in lower limb movements as they are more challenging to communicate over video. Video conferencing tools are typically configured to support talking heads conversations, and have little consideration for the observation of full body movements [4]. Moreover, video technology possess
<figure><img src="image_3.png" /><figcaption>**Figure 1:** Setup of SoPhy during a video consultation: (a) In left, a physiotherapist is using SoPhy web-interface to understand patient's movements. (b) In right, a patient is performing lower body movements by wearing SoPhy socks.</figcaption></figure>

limited visual acuity that further reduces clinician's ability to discern subtle changes in exercises e.g., point of balance and depth of squats [2].

This is the first attempt that extends video consultations beyond audio-video medium and explores the potential of a wearable technology to support the tasks of physiotherapists. We designed *SoPhy* that provides information of key aspects of lower limb movements during video consultations (refer Figure 1). We conducted a laboratory study to investigate how *SoPhy* helps physiotherapists in assessing patients over video [1]. The findings showed that *SoPhy* increased their confidence in assessing low limb exercises like squats, and they required fewer repetitions of exercises to assess patients with *SoPhy*. Next we illustrate the design of *SoPhy*.

## 2 SoPhy: Our Proposed System

*SoPhy* stands for ‘socks for physiotherapy’. *SoPhy* is a novel wearable technology consisting of (1) a pair of socks embedded with three pressure sensors and one Inertial Measurement Unit (IMU) that patients wear while performing lower body movements (Figure 2a); and (2) a web-interface that visualizes information about weight distribution, range of foot movement and foot orientation to physiotherapists in real-time (Figure 2b).

We have followed user-centered approach to iteratively design *SoPhy*, and collaborated with a senior physiotherapist (last author) at a leading hospital in Australia. To this end, the selection of different sensors and visualisation of the data are the results of multiple lab trials and feedback from the collaborating physiotherapist. Below we describe the bodily information supported by *SoPhy*.

**Weight Distribution:** Weight distribution describes the amount of weight a person bears in different areas of the foot e.g., on toes, balls and heel. While a healthy person distributes equal weight on each foot (leg), the pattern changes in case of an injured. For instance, if a person has injured his big toe, he may bear more weight on the outside of the foot. *SoPhy* captures the pattern of weight distribution on the balls and heel of the foot, through pressure sensors stitched on the *SoPhy* socks. The web-interface displays weight distribution on the feet sketches showing the feet from underneath
<figure><img src="image_4.png" /><figcaption>**Figure 2:** *SoPhy* consists of two parts: (a) a pair of socks with sensors embedded, and (b) a web-interface that visualizes the information related to foot orientation, range of movement and weight distribution.</figcaption></figure>

(refer Figure 2b). For each sensor, the pressure values are presented with a number on a scale of 0-30 and on a color spectrum of yellow to red.

*Foot Orientation:* Foot orientation refers to the alignment of foot in four directions. First is the dorsiflexion, which occurs when the person bears weight on the heel of the foot with toes lifted up in the air. Second is the plantarflexion where the weight is on the balls and toes of the foot with heel lifted up in the air. Third is the medial orientation where the weight is on the inside of the foot and the person lifts the outside of the foot up in the air. And final orientation is lateral orientation where the person bears weight on the outside of the foot and lifts the inside of the foot up in the air. In SoPhy, foot orientation is captured by the IMU sensor sewed on the socks on the bridge of the foot. We have used multiple foot sketches to present this information on the web-interface: three each for dorsiflexion and plantarflexion, and two each for medial and lateral orientation (refer Figure 2b).

*Range of Movement:* Range of movement refers to the magnitude of the foot orientation across four directions described above. The range is defined on a scale of 1 to 10 and is calculated from the IMU data. On the web-interface, this value is represented as a ‘Flex degree’ under each foot (refer Figure 2b).

# 3 Engaging with SoPhy

Figure 1 shows the setup of SoPhy during a video consultation. The patient wears the SoPhy socks before starting the video consultation with physiotherapist. During a video consultation, physiotherapist will ask the patient to perform different lower body exercises, e.g., dorsiflexion and plantarflexion, squats, and heel raises (refer Figure 3). As the patient performs these exercises, the socks capture data about foot movements. This data is then sent to the web interface, where the physiotherapist can see the movement data in real-time. We designed a mobile app to support data communication between the socks and web interface via a Bluetooth shield attached on the SoPhy socks.

At the venue, visitors will be able to try out SoPhy in solo or in pairs. They can wear the SoPhy socks and check the visual feedback on the web-interface for different lower body movements. Alternatively, they can role-play as a physiotherapist and
<figure><img src="image_5.png" /><figcaption>Figure 3: Examples of lower limb exercises: (a) Dorsiflexion and Plantarflexion, (b) Squats (c) Single leg heel raises.</figcaption></figure>

patient, to mimic a clinical consultation related to lower limb assessment. We anticipate that audience will get the following insights by interacting with SoPhy:

1. They will get an awareness of the role of different bodily cues related to lower body exercises.

2. They will become familiar with the emerging practice of video consultations and the challenges of video technology in communicating lower limb movements.

3. They will gain insights on the challenges faced by physiotherapists in assessing patients during video consultations, and how SoPhy can help them in their decision-making process.

4. They will get a first-hand experience of SoPhy, which will particularly, be of interest to DIY med-tech community.

5. Finally, it will potentially seed new interests and conversations around the development of future video consultation systems that provide essential bodily information to clinicians.

## 4 REFERENCES

1. Aggarwal, D., Zhang, W., Hoang, T., Ploderer, B., Vetere, F., and Bradford, M. 2017. SoPhy: A Wearable Technology for Lower Limb Assessment in Video Consultations of Physiotherapy. To appear in *Proceedings of the 35th Annual ACM Conference on Human Factors in Computing Systems (CHI'17)*.

2. Aggarwal, D., Ploderer, B., Vetere, F., Bradford, M. and Hoang, T. 2016. Doctor, Can You See My Squats?: Understanding Bodily Communication in Video Consultations for Physiotherapy. In *Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS '16)*. ACM, 1197-1208.

3. Ekeland, A.G., Bowes, A. and Flottorp, S. 2010. Effectiveness of telemedicine: A systematic review of reviews. *International Journal of medical informatics* 2010, 79(11), 736-771.

4. Licoppe, C. and Morel, J 2012. Video-in-interaction: "Talking heads" and the multimodal organization of mobile and Skype video calls. *Research on Language & Social Interaction* 2012, 45(4), 399-429.

5. Stewart, M.A. 1995. Effective physician-patient communication and health outcomes: a review. In *Journal of Can Med Assoc.*, 1423-1433.