# Comparing Fatigue When Using Large Horizontal and Vertical Multi-touch Interaction Displays

Shiroq Al-Megren, Ahmed Kharrufa, Jonathan Hook, Amey Holden, Selina

Sutton, Patrick Olivier

► To cite this version:

Shiroq Al-Megren, Ahmed Kharrufa, Jonathan Hook, Amey Holden, Selina Sutton, et al. Comparing Fatigue When Using Large Horizontal and Vertical Multi-touch Interaction Displays. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.156-164, 10.1007/978-3-319-22723-8_13. hal-01610843

HAL Id: hal-01610843

https://inria.hal.science/hal-01610843

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Comparing Fatigue when Using Large Horizontal and Vertical Multi-Touch Interaction Displays

Shiroq Al-Megren<fnref n="1" />, Ahmed Kharrufa<fnref n="2" />, Jonathan Hook<fnref n="3" />, Amey Holden<fnref n="2" />, Selina Sutton<fnref n="2" />, and Patrick Olivier<fnref n="2" />

**Abstract.** We report on a user study that compared muscle fatigue experienced when using a large multi-touch display in horizontal and vertical configurations over a one-hour period. Muscle fatigue is recognized as the reduction in a muscle's capacity to generate force or power output and was measured objectively and subjectively before and after a puzzle-solving task. While subjective measures showed a significant level of overall arm muscle fatigue after the task for both configurations, objective measures showed a significant level of muscle fatigue on the middle deltoids and the non-dominant extensor digitorum for the vertical configuration only. We discuss the design implications of these findings and suggest relevant future areas of investigation.

**Keywords.** Large displays; interaction; tabletops; fatigue; ergonomics

## 1 Introduction

Muscle fatigue is the transitory decrease in a muscle's capacity to contract and perform actions, which leads to the development of musculoskeletal imbalance if it persists for longer stretches of time and overruns the body's recovery system [1, 2]. Previous work has reported on subjective perceptions of fatigue [3, 4], which are deemed unreliable for assessing physical demands [5], and objective measures of muscle fatigue [6, 7] on tablet sized vertical touchscreens. Past research has also explored how display angle affects a range of different aspects of users' interaction with large interactive displays (e.g. [8, 9]). However, relatively little attention has been paid to the impact that display angle has on users' physiology over time and consequential musculoskeletal damage. The few studies that have investigated this issue have required subjects to perform a constrained task repeatedly for a relatively short time, rather than exploring the typical varied and longitudinal usage that designers envisage, such as teachers' interaction with interactive boards (e.g., [10] who looked at vertical

<footnote n="1">¹ School of Computing, University of Leeds, Leeds, United Kingdom
scsam1@leeds.ac.uk</footnote>

<footnote n="2">²Culture Lab, Newcastle University, Newcastle Upon Tyne, United Kingdom
{ahmed.kharrufa, selina.sutton, patrick.olivier}@ncl.ac.uk
ameyholden3@gmail.com</footnote>

<footnote n="3">³Department of Theatre, Film and Television, University of York, York, United Kingdom
jonathan.hook@york.ac.uk</footnote>
boards only). Other studies, such as [11, 12], have reported informal and conflicting insights for longitudinal usage of large interactive displays.

We present a study that measured muscle fatigue objectively and subjectively for both horizontal and vertical large interactive displays using an unconstrained task, which required users to perform gestures commonly used in multi-touch interaction. Our results suggest that when designing multi-touch applications that will be used for extended periods, designs that consider fatigue should, in general, favor horizontal over vertical displays, and in both cases should aim at locating interactions at closely accessible locations on the display whenever possible. Our results also suggest that designers should pay close attention to objective measures of fatigue when evaluating the appropriateness of their designs to users' physiology, rather than relying on subjective measures alone.

## 2 Study Design

We performed the study on a Microsoft PixelSense multi-touch tabletop with a 30" rear projected display (21" high, 27" deep, and 42.5" wide). The task was to complete a series of 25-piece puzzles, using the tabletop's Puzzle application, which was adapted to include scaling of pieces. Pieces could be translated, scaled and rotated, or an integrated combination of two or more of these actions, using one or more fingers, uni- or bi-manually. Initially, the pieces of each puzzle were displayed at random orientations, sizes and locations. Two pieces could only connect to form one larger piece if they were a similar size (within 20%) and orientation (within 30°). Once completed the puzzle was replaced with another. This exercise was repeated for one-hour. The puzzles were displayed in a random order and subjects completed five puzzles on average. This task was chosen because it required participants to complete common types of interactions used when performing different tasks using large interactive displays, along with intermittent periods of no activity.

## 3 Procedure

Surface Electromyography (SEMG) was used to monitor three muscles on each arm before, during and after the task. These muscles are: middle deltoid, bicep brachii, and extensor digitorum muscles (see Figure 1a). Lozano et al. [7] suggested these muscles are appropriate sites for measuring gestural interaction and fatigue on a multi-touch tablet when using SEMG. The deltoid placement was altered, from anterior to middle, to consider both shoulder abduction and flexion. To locate the placement of the SEMG sensor for each muscle, a specific location on the arm was palpated as the subject performed a movement activating the target muscle. The skin was then shaved, wiped with alcohol and the sensors placed parallel to their prospective muscle at 2cm apart. We followed Criswell's guidelines [13] to ensure correct placement.

Two measures were then collected: maximal voluntary isometric contractions (MVIC) and ratings using the Borg scale [14] both before and after the task (as described below). The subjects were then asked to sit and familiarize themselves with
short period. To do this our subjects were asked to remain seated and maintain a posture for 10 seconds while holding a 2.5kg weight. The postures were; i) Middle deltoid: arm elevated at 90° in the frontal plane; ii) Bicep brachii: arm held close to the body while elevating the forearm at 90° in the sagittal plane; iii) Extensor digitorum: the forearm resting on a desk with the wrist resting on the edge of the table. A MVIC is a baseline and so future readings may be greater than 100%.

**Localized Fatigue.** For fatigue indexing, we adopted the median power frequency (MPF) of the MVIC in the time domain as a reference point [20]. Essentially, a decreasing MPF signal indicates that muscle fatigue is increasing. Previous research corroborates the reliability and consistency of this method of analysis (e.g. [21]) unlike using the amplitude in the time domain, where the literature reports significant contradictions (e.g. [7], [21]).

**Force.** 'Force' quantifies a muscle's electrical activity during contraction and is described as a percentage of a MVIC. To extract force information we integrated a rectified EMG, a long-established technique due to their linear relationship [22].

The SEMG used was ZeroWire, a wireless system with six surface channels, with Biosense's bio-logic disposable press-stud electrodes (Ag-AgCl). This system operates using light autonomous signal processing and power transmission units, each weighing 10gm. Each channel provides a bandwidth of 10-1000 Hz for a signal sampled at 2000 sample/sec. The transmitters wirelessly transfer the signals captured with the electrodes to the main unit, which is directly connected to a computer running the ZeroWire software suite. This minimizes the restriction of a user's movements. Matlab and Microsoft Excel were used to process the EMGs and analyze the results.

## 4.1 Localized Fatigue Analysis

The 10-second MVIC collected before and after the task was divided into 10 segments with a 50% overlap. This led to 3000 data points for each segment from the original 2000. The last segment was excluded from the analysis because of the overlap. Each segment was then rectified and passed through a low pass fourth-order Butterworth filter with a cut-off frequency of 500Hz. A fast Fourier transform was then performed to calculate the power spectrum of each segment from which the MPF is obtained. Matlab was used to process the EMG using functions provided by the Signal Processing Toolbox (DSP) and the Biomechanics et al. Toolbox (BEAT) [23].

For normalization (to eliminate variations between the subjects such as age and muscle mass) the data recorded before the task was averaged and used as a reference value for the data collected after the task. These values were averaged to produce singular values representing the MPF of each muscle. A paired t-test was then carried out between the two data sets.

## 4.2 Localized Fatigue Results

**Horizontal.** While some muscles did show a decrease of MPF indicating some level of fatigue, none of them showed a significant evidence for increased fatigue. The dominant and non-dominant extensor digitorum showed a decrease to 80% and 63%
respectively, while the dominant and non-dominant bicep brachii showed a decrease to 88% and 95% respectively. Moreover, while the dominant middle deltoid also showed evidence of significant increase of MPF indicating decreased fatigue (119% and t(17)=1.74, p=0.01), no change was noted for the non-dominant middle deltoid.

**Vertical.** The middle deltoid showed a significant decrease of MPF indicating increased fatigue (63% and t(17)=1.74, p=0.003 for the dominant, and 71%, t(17)=1.74, p=0.02 for the non-dominant side). As for the extensor digitorum only the non-dominant side showed a significant decrease of MPF indicating increased fatigue (75% and t(17)=1.74, p=0.01) with no noted decrease for the dominant hand. The biceps brachii showed non-significant decrease of MPF for the non-dominant hand to 79% and no-decrease for the dominant hand.

## 4.3 Force Analysis

Ten 1-minute EMGs were collected during the experimental task at 5-minute intervals. Each EMG was passed through a low-pass fourth-order Butterworth filter with a cut-off frequency of 500Hz. The averaged root mean square was then calculated using BEAT, which was then averaged. Normalization was carried out using the averaged MVICs collected before the start of the task. The normalized values were then averaged to represent the muscle activity as a percentage of the MVIC (see Figure 2). A one-way repeated measure analysis of variance (ANOVA) was then carried out.

<figure><img src="image_2.png" /><figcaption>Fig. 2 Force results for 1-minute readings collected every 5 minutes for both configurations</figcaption></figure>

## 4.4 Force Results

**Horizontal.** A one-way repeated ANOVA for the 10 measures showed that only the dominant extensor digitorum showed significant inconsistency of muscle activity throughout the task (a change of 13% and F(17, 9)=3.09, p=0.002). No significant evidence was found against the other muscles proving their activity to be consistent. Nevertheless, the non-dominant extensor digitorum showed an increase and decrease of up to 8% and the bicep brachii showed a change of up to 10% and 14% for the dominant and non-dominant sides respectively. The middle deltoids' activity proved relatively stable with activations ranging from 8-11% for both sides.

**Vertical.** A one-way repeated ANOVA for the 10 measures showed significant inconsistency of muscle activity throughout the task for the dominant extensor digi-
torum (a change of 13% and $F(17, 9)=2.85, p=0.004$). No significant evidence was found against the other muscles proving their activity to be consistent. Nevertheless, both biceps brachii showed an increase and decrease in activation of up to 13%, while both middle deltoid muscles showed relatively consistent activation of up to 18%.

Table 1 marks the muscles that showed significant evidence of increase in level of fatigue after the trial session for both configurations. It also marks significant inconsistencies in activation of force during task interactions.

<table><caption>**Table 1.** Objective measures summary of significant increase in fatigue and force inconsistency</caption><thead><tr><th rowspan="3"></th><th rowspan="3"></th><th colspan="4">Significant Evidence</th></tr><tr><th colspan="2">Horizontal</th><th colspan="2">Vertical</th></tr><tr><th>Dom.</th><th>Non-Dom.</th><th>Dom.</th><th>Non-Dom.</th></tr></thead><tbody><tr><td rowspan="2">Middle Deltoid</td><td>Fatigue</td><td>×</td><td>×</td><td>✓</td><td>✓</td></tr><tr><td>Force</td><td>×</td><td>×</td><td>×</td><td>×</td></tr><tr><td rowspan="2">Bicep Brachii</td><td>Fatigue</td><td>×</td><td>×</td><td>×</td><td>×</td></tr><tr><td>Force</td><td>×</td><td>×</td><td>×</td><td>×</td></tr><tr><td rowspan="2">Extensor digitorum</td><td>Fatigue</td><td>×</td><td>×</td><td>×</td><td>✓</td></tr><tr><td>Force</td><td>✓</td><td>×</td><td>✓</td><td>×</td></tr></tbody></table>

## 5 Subjective Measures

When examining fatigue, subjective ratings are commonly used to assess perceived physical exertion and supplement objective measures. We used the Borg CR100 scale [14], which is a fine graded ratio scaling method that estimates the level of exertion and determines the ratio relationship between perceptual responses. The scale contains subjective dynamic ranges with values ranging from 0 to 120 annotated with verbal anchors ranging from “nothing at all” to “absolute maximum”. This scale was chosen as the most appropriate after critical assessment and comparison of four other scales. The subjects were asked to rate their perceived muscle exertion after the MVIC had been recorded before and after the experiment.

The average of the scores collected was analyzed for statistical significance using a paired t-test. For the horizontal configuration, the averaged value of the ratings was significantly greater after the task ($t(17)=1.74, p=0.0003$). Similarly for the vertical configuration, the averaged value was significantly greater after the task ($t(17)=1.74, p=0.0005$). This indicates an overall increased perceived level of fatigue for both configurations (see Table 2).

<table><caption>**Table 2.** Subjective mean and standard deviation results for the horizontal and vertical configurations</caption><thead><tr><th>Configuration</th><th>Before</th><th>After</th></tr></thead><tbody><tr><td>Horizontal</td><td>3.97 (±7.96)</td><td>15.44 (±14.93)</td></tr><tr><td>Vertical</td><td>3.17 (±5.40)</td><td>12.44 (±11.92)</td></tr></tbody></table>
# 6 Discussion and Conclusion

The main aim of the study was to compare the muscle fatigue experienced when using large multi-touch interactive displays in horizontal and vertical configurations, over a one-hour period.

Our findings from the objective measures provide statistically significant evidence for the presence of potentially damaging fatigue for the middle deltoids for the vertical case only and not the horizontal. This can be due to the elevation required of the arms when interacting with the vertical display (i.e. gorilla-arm effect [24]) and the employment of larger force activation percentages. The presence of fatigue was also noted for the non-dominant extensor digitorum despite consistent activation of force. This can be due to the regular use of the non-dominant muscle (either uni- or bi-manually) in the vertical configuration, as can be derived from the significantly inconsistent activation of the dominant extensor digitorum. The inconsistent activation of dominant extensor digitorum suggests that the users were alternating between dominant and non-dominant sides. Unlike the objective measures, the results of the subjective measures showed that the subjects perceived the presence of fatigue in both the vertical and horizontal configurations with statistical significance. This emphasizes the unreliability of a person's perception of fatigue, where objective results found significant levels of muscle fatigue that can be damaging only in the vertical case.

These results have clear implications on interaction design for large interactive displays. Vertical interactive displays, while suitable for intermittent use over short periods of time, are not as suitable for frequent longer use – our study showed presence of fatigue for a one-hour task, but it might occur in shorter durations. Prolonged activities leading to muscle fatigue, which has previously been reported anecdotally, have the potential to lead to musculoskeletal disorders [1, 2]. For tasks that require frequent interactions in the range of one or more hours, designers should choose a horizontal display where possible to minimize damage to the musculoskeletal system, modify the design of interaction techniques to reduce the need for continuous interaction, or to change the location of distant interactions when possible to more accessible locations on the display. Moreover, subjective measures showed a significant level of fatigue for both horizontal and vertical cases, indicating that for either case, designers should locate points of interaction in spaces easily accessible to the user whenever possible. As a simple example, for large displays, the use of contextual commands that are located based on users' interaction location is recommended over the traditional desktop model of showing commands on upper or side toolbars and menus. However, identifying the display space that helps reduce fatigue, whether on horizontal or vertical displays, is the subject of further investigation.

This study has several limitations that point towards future work. We studied fatigue for only male participants to overcome physiological differences. Additional studies with female participants would give insight on sex differences and its effect on measured fatigue when interacting with large interactive displays. Furthermore, our study only considered a small subset of the muscles' activated when interacting with
the display and only two angles of interaction. In the future, we plan to assess the fatigability of other muscles (e.g. capitis muscles) at additional angles of interaction.

## References

1. Vøllestad, N.K.: Measurement of human muscle fatigue. J Neurosci Methods 74, 219-227 (1997)

2. Wahlström, J.: Ergonomics, musculoskeletal disorders and computer work. Occ Med 55, 168-176 (2005)

3. Meyer, S., Cohen, O., Nilsen, E.: Device comparisons for goal-directed drawing tasks. CHI'95 Conf Companion, pp. 251-252. ACM, Boston (1994)

4. Sears, A., Shneiderman, B.: High precision touchscreens: design strategies and comparisons with a mouse. Int J Man-Mach Stud 34, 593-613 (1991)

5. Barriera-Viruet, H., Sobeih, T.M., Daraiseh, N., Salem, S.: Questionnaires vs observational and direct measurements: a systematic review. TIES'06 7, 261-284 (2006)

6. Young, J.G., Trudeau, M.B., Odell, D., Marinelli, K., Dennerlein, J.T.: Wrist and shoulder posture and muscle activity during touch-screen tablet use. WORK 45, 59-71 (2013)

7. Lozano, C., Jindrich, D., Kahol, K.: The impact on musculoskeletal system during multitouch tablet interactions. ACM CHI'11, pp. 825-828. ACM, Vancouver (2011)

8. Ichino, J., Isoda, K., Hanai, A., Ueda, T.: Effects of the display angle in museums on user's cognition, behavior, and subjective responses. CHI'13, pp. 2979-2988. ACM, Paris (2013)

9. Muller-Tomfelde, C., Wessels, A., Schremmer, C.: Tilted tabletops: In between horizontal and vertical workspaces. TABLETOP'08 pp. 49-56 (2008)

10. Zerpa, C., Lopez, N., Przysucha, E., Sanzo, P.: The effect of common teaching tools on upper extremity muscle activity. Education 4, 160-166 (2014)

11. Morris, M.R., Brush, A.J.B., Meyers, B.R.: A field study of knowledge workers'; use of interactive horizontal displays. TABLETOP'08 pp. 105-112 (2008)

12. Wigdor, D., Perm, G., Ryall, K., Esenther, A., Chia, S.: Living with a tabletop: Analysis and observations of long term office use of a multi-touch table. TABLETOP'07, pp. 60-67 (2007)

13. Criswell, E.: Cram's Introduction to Surface Electromyography. Jones and Jones (2011)

14. Borg, G., Borg, E.: A new generation for scaling methods: Level-anchored ratio scaling. Psychologica 28, 15-45 (2001)

15. Cioni, R., Giannini, F., Paradiso, C., Battistini, N., Navona, C., Starita, A.: Sex differences in surface EMG interference pattern power spectrum. J Appl Physiol 77, 2163-2168 (1994)

16. Hincapi-Ramos, J.D., Guo, X., Moghadasian, P., Irani, P.: Consumed endurance: a metric to quantify arm fatigue of mid-air interactions. CHI'14, pp. 1063-1072. ACM, Toronto (2014)

17. Al-Mulla, M.R., Sepulveda, F., Colley, M.: A review of non-invasive techniques to detect and predict localised muscle fatigue. Sensors 11, 3545-3594 (2011)

18. Peres, S.C., Nguyen, V., Kortum, P.T., Akladios, M., Wood, S.B., Muddimer, A.: Software ergonomics: relating subjective and objective measures. CHI'09 EA, pp. 3949-3954. ACM, Boston (2009)

19. Sherwood, L.: Human Physiology: From Cells to Systems. Cengage Learning (2012)

20. Phinyomark, A., Thongpanja, S., Hu, H., Phukpattaranont, P., Limsakul, C.: The Usefulness of Mean and Median Frequencies in Electromyography Analysis (2012)

21. Murata, A., Ishihara, H.: Evaluation of shoulder muscular fatigue induced during mouse operation in a VDT task. IEICE - Trans Inf Syst E88-D, 223-229 (2005)

22. Lawrence, J.H., De Luca, C.J.: Myoelectric signal versus force relationship in different human muscles. J Appl Physiol Respir Environ Exerc Physiol 54, 1653-1659 (1983)

23. NISMAT http://www.nismat.org/software/beat/doc/beat.html

24. Gorilla arm http://www.computer-dictionary-online.org/index.asp?q=gorilla+arm