ITE Trans. on MTA Vol. 5, No. 2, pp. 57-66 (2017)

Copyright © 2017 by ITE Transactions on Media Technology and Applications (MTA)

Paper
Design of Assistive Tabletop Projector-Camera System
for the Elderly with Cognitive and Motor Skill Impairments
Jaakko Hyry †,†† , Max Krichenbauer † , Goshiro Yamamoto †,††† ,Takafumi Taketomi † ,Christian Sandor † ,Hirokazu Kato † ,Petri Pulli ††Abstract

Projector-camera (ProCam) systems have a potential to become popular and aﬀordable as they can create

interactive surfaces for example on tabletops, walls, household items or on a palm of a hand. The possibility that thesesystems will be used at homes in the future is increasing. The elderly living alone at home often need assistance in their
daily tasks as the likelihood of cognitive and motor skill related impairments increases with age. ProCam systems could
be used for guidance due to easy to manipulate large interaction surfaces, but research on its suitability for elderly users is
scarce. Our research focus is on elderly users and examining their characteristics as potential users of ProCam systems andthe implications for interaction design. We conducted a user study with a mixed impairments group of elderly aged 82-94 toinvestigate how a personalized and skill-suited user interface should be designed. In our qualitative approach, we discoveredthat the combinations of both cognitive and motor skill deﬁciencies of the elderly prohibit one-for-all designs so the userinterface design should be adapted to each individual’s interaction skills. Lastly, we make suggestions for designing ProCam
interaction for elderly.

Key words: projector-camera systems, elderly, interaction, cognitive impairments, motor skill impairment

Associated Memory Impairment (AAMI) require lesssupport and would be a better target for assistive technology. If the elderly are more capable at home themselves, it will reduce the need for more caregivers in the
future1) . From this perspective, technology designed forelderly use is a necessary and an important challenge.
There are several additional challenges when developing supporting technologies for the elderly. The acceptance and use of new technology are often diﬃcult
due to the convoluted guides and structures3) . Designs should take into consideration the abilities elderly
have and what kind of physical or mental changes aging
causes for these individuals4) .
The problem is that the elderly are often less computer literate and that there is a gap in technology skills
between the young and the old5) as the elderly preferpreviously learned methods and have trouble learning
newer technologies6) . Younger users also often adopt
new technologies faster and use a multitude of devices,such as laptops, tablets and smartphones eﬀortlessly.
Motor skill impairments aﬀect speed, precision and theability to manipulate small devices, while cognitive impairments aﬀect memory functions for learning.

1. Introduction
The rapidly aging population is posing an increasing
burden on many countries around the world to keep up
with the rising cost of healthcare and lack of professional caregivers for the elderly1) . While the advancements in information technology oﬀer a number of solutions to ease the burden of personal caregivers, fewer
applications are available to support the independent
living of the elderly so that they can continue living at
home as long as possible2) .
The elderly with severe Alzheimer’s need constantsupport from caregivers and cannot be assisted withtechnology at home. However, the elderly with earlyor mild stages of cognitive impairments, such as AgeReceived June 13, 2016; Revised January 10, 2017; Accepted March 2,
Received; Revised; Accepted2017† Interactive Media Design Lab, Nara Institute of Science and Technology(Nara, Japan)†† University of Oulu
(Oulu, Finland)††† Kyoto University Hospital(Kyoto, Japan)

57

ITE Trans. on MTA Vol. 5, No. 2 (2017)

Considering these factors, projection-based user interfaces have a number of advantages over traditional
indirect input methods, or touchscreens. Firstly, they
allow projection on top of objects and surfaces in a realworld environment and secondly they provide large projections that are easier targets for interaction purposes.
To the best of our knowledge, there are no previous
studies on the best practices for user interface design
for a projector-camera system (ProCam UI design) for
elderly users. Thus, we examine how the elderly behavewith our ProCam system, as it is not clear how the individual characteristics, cognitive and physical impairments may aﬀect the use and design for a projection
tabletop systems. Therefore, we are interested in thefollowing research questions: 1) To what degree does
area pointing selection methods suit impaired elderly?
2) In what way does motor skill impairments aﬀect interaction? In this paper, we conducted a user study
with elderly participants using our ProCam tabletop
interface. Our results inform the design of projectionbased tabletop interfaces for elderly users with varying
degrees of cognitive and physical impairments.

ceptance. The common problems the elderly have withtechnology are generalized in references3)13)14) :( 1 ) Lack of knowledge: Use metaphors of new UI’s
is less known, so learning new is diﬃcult compared to younger( 2 ) Hard to learn: Indirect manipulation is cognitively challenging, as it requires more mental
processing( 3 ) Motor skills limitations: Movements becomeless precise with old age so small targets are
harder to manipulate( 4 ) Eyesight limitations: Worse eyesight makes
small screens diﬃcult to read and manipulate.
These various factors aﬀect use and are more common
the older a person gets for example mouse manipulation and target acquisition are harder to perform15)16) .
While these studies use traditional, indirect input devices, some have also studied the use of touchscreeninterfaces for the elderly17)18) . Interaction in general for
elderly users have some suggestions19) :( 1 ) Use direct manipulation to reduce mental load
( 2 ) Direct manipulation results in faster and moreaccurate target acquisition( 3 ) Larger screens and icons reduce input errors
and enhance readability
Common user interface selection methods, such as
tapping, for touch interaction can be hard for motor
skill impaired users, so some alternative methods havebeen studied such as area pointing or crossing over an
area20)21) . All of these approaches are more suitable fornovel users, which the elderly often are, and thus anassistive system should take into account these factors.

2. Related Work
2. 1 Prior Research on ProCam Systems
Recently, ProCam systems have become less costly
than before and oﬀer various solutions for task assistance, often work support or navigation assistance foryounger users. As examples, cooking support with projection assistance7) , reliable interaction detection using
a depth sensor8) and multiple surface projections withseveral input methods demonstrated with the OmniTouch system9) . These previous studies show the potential of using ProCam system in assistive tasks.
Studies using ProCam technology for elderly assistance are increasing, such as ambient environment that
conduct everyday task scenarios10) or have remote caregiver assistance using on-site projection7)11) . There aresome prior studies on wearable ProCam systems that
display information on wall surfaces. Yamamoto etal.12) present an assistive system for the elderly thatprojects icons on a wall with tapping interaction capability. ProCam systems show promise for assistance but
more research on how elderly interact with these systems and how interaction should be designed from anempirical point of view is needed.
2. 2 Interaction Methods
The suitability of ProCam systems for the elderly
needs focus on user’s interaction performance and ac-

3. Our Projection Tabletop System
The overall vision behind our system is to project
information in an elderly user’s home and help themconduct daily tasks. However, ﬁrst we need to examine
what type of interaction methods would be the mostsuitable considering the cognitive and motor skill limitations for a tabletop ProCam system. For motor skill
impaired users, a device occupying a hand limits theinteraction possibilities to a single hand and holding adevice for long periods at a time is tiresome. To overcome this limitation our projection is displayed from aﬁxed device installation in the ceiling.
To reduce memory load for memory-impaired users,
interacting with objects is enhanced by augments
graphics directly on top of or next to the objects and not
through a device screen such as smartphones, tablets58

Paper » Design of Assistive Tabletop Projector-Camera System for the Elderly with Cognitive and Motor Skill Impairments

prevents ethical concerns of the elderly regarding video
monitoring as the IR-camera can only see light reﬂectedfrom the markers without seeing the user. The calibration is done instantly by projecting and capturing acheckerboard pattern with the camera in RGB mode.
Any projected information at the location of the corresponding markers is based on geometrical calibration
between the camera and the projection area. By limiting the interaction on the tabletop surface, we can calculate the homography between the camera coordinate
system and the projection area.
For tracking and displaying the camera is switched to
IR-mode. We use standard ARToolKit markers22) for
estimating the coordinates of the objects (e.g. menu
card, a pillbox or a calendar) and the input paddle.
Several markers can be tracked at the same time. To
make the markers less visible to the users but easily
visible in the IR-space we use retroreﬂective material.
Using the homography, graphics for the menu-card canbe projected onto the exact location and updated inreal-time if the card is moved. The same can be doneto any object with a marker on it. The system canhandle pointing interaction with the paddle since location of the paddle can be estimated in each frame. For
the ring, we use a diﬀerent detection approach, as an
AR marker on a tiny object is not practical. Instead,
we track the ring from the IR-image by using the same
retroreﬂective material on the ring’s surface. To detect
where a user points at with the ring, the IR-image ismeasured in eccentricity and ratio between width andheight of a bounding rectangle along the axis of theorientation. The ﬁngertip position is estimated as 5-6
centimeters from the ring’s location and adjusted manually for each user if necessary.
3. 2 User Interface Designs
While elderly users can be eﬀective with technology,
younger users adapt new devices and accumulate UIinteraction methods in a faster pace creating a larger
knowledge base. The elderly rely on existing mental
models from past experiences that also apply to technology use. But the lack of experience with new technology like touchscreen devices or hand gestures creates
obstacles for user interaction, so we studied how various selection methods would work in case of projectorcamera systems. Home environment surfaces can act
as projection displays, which is why in our test cases atable was used as an interactive surface. It also enables
haptic properties, as it is a physical object that can betouched. The large interaction area also makes manip-

Projector(b)
CameraComputer

Retrorefletive
Material(a)Interactive Surface
Fig. 1

(c-1)

(c-2)

System overview; (a) ProCam installation, (b)user interface, (c) interactive tools.

or smart glasses. The digital information can be associated directly on real-world objects to enhance atask. Projected graphical elements also do not block
real world objects in comparison to a head-mounted
display through which the user can only see graphicalelements inside the device.
3. 1 System Structure
Our system consists of PC (Fujitsu-Siemens 13.3 inch,
i3-M380M 2.53 GHz Dual core laptop), camera (OptiTrack FLEX V100:R2, 640×480, 120 Hz), a projector(Optoma EW1691e DLP, 3000 lumens, 1280×800) andseparate speakers for audio feedback as shown in Fig.
1 (a). The projector and camera are installed above atable pointing downwards from 140 cm and 80 cm, respectively. The projector displays the UI on the tablesurface with an area of 90 cm×56 cm, and the camera
detects within an area of 60 cm×45 cm the location of
the target objects placed on it; a menu-trigger card andinput device options, a ﬁnger-worn ring and a handheld
paddle (4 cm×4 cm), as shown in Fig. 1 (c-1) and (c-2).
The paddle was thought to be easy for the elderly to
associate the device with user interaction, although it
might turn out diﬃcult for people with hand motor skill
impairments. The ring enables user interaction similar
to touchscreens and lets the user to hold the hand ona table for support, which reduces negative impact ofhand tremors or stiﬀness.
For ProCam calibration and marker detection with
one camera, we chose an OptiTrack FLEX: V100:R2
camera because it has two alternative modes: a color
mode (RGB) for initial scene calibration and an infrared mode (IR) for tracking the markers. Using an
IR-camera in combination with retroreﬂective markers
59

ITE Trans. on MTA Vol. 5, No. 2 (2017)highlighting

MENU

MENU
MENU

MENU

movingarrow(a)activation area appears

progress bar for time counting
MENU

move to“Activation area”
Fig. 2

(b)

MENU
MENU

MENU

MENU

slide tokeep the location
for specified time “Activation area”
for “Activation” (c)Selection methods; (a) two-step, (b) hover, (c) slide.

ulation easier as there is no need for ﬁne movements
whereas small touchscreen need precision. The designs
for the user interface are aimed at elderly based on thefollowing considerations on existing guidelines23)24) . 1)
We limited the number of icons to a maximum of eight
so that the user does not have too many choices to interact with. 2) We considered how the layout for hand interaction should be performed with easy access to icons
for the users. Based on average human arm ergonomics,
a 50 cm length for a tabletop surface interaction was
calculated to be suﬃcient and transferred as our detection capture area size limitation. Multiple rows of icons
were unnecessary due to the reduced amount of icons
and there is less chance of the user blocking information
accidentally by having the interaction space as an arc.
An arc shape also follows the arms natural movements
in tabletop interaction.
The elderly often have a diminished ﬁeld of view10) ,so we placed the icons in the center of the table. 3) We
created larger, dynamic sized icons 1.25 times the sizeof the paddle (the physical size of it was approximately
5 cm×5 cm), with big fonts spread out evenly with anicon-icon distance of 1 cm. This also limits the amountof icons you can use on the current projection surface
as seen in Fig. 1 (b) while keeping the user’s focus in
the same area. Research by Bakaev also support usinglarger icons for elderly users as it increases their interaction speed25) . 4) To help the user understand where
the system tracks their ﬁnger’s location a red circle isprojected right in front of the input ﬁnger’s tip as a
cursor.
We wanted the users to clearly verify their choiceof an icon because completion of critical tasks suchas medication intake by a memory impaired individual should be forwarded to a doctor or to family members. To enable menu selection, we chose two interaction methods that have an additional veriﬁcation step
as shown in Figure 2 (a) and (c). But to reduce thenumber of interaction steps, we also chose a common

time-based hover method as shown in Fig. 4 (b) forcomparison. Tapping is a viable method for ProCam
system but because motor skill impaired users haveproblems with tapping and accuracy26) in this study
we implemented methods that require the user to only
move over a target area for selection.
The two-step method (Fig. 2 (a)) uses a central
verifying icon. The selection works by 1) ﬁrst choosing
a desired icon e.g. ‘make a call’ 2) then moving theﬁnger to the conﬁrmation icon ‘SELECT’ appearing in
the center of the UI. This activation step was createdin order to avoid unconscious or accidental selection. It
also provides a single activation icon at a ﬁxed location
under the assumption that the same location is easy to
see and learn. We added a highlight box around theicons when hovered over and an arrow to point to theactivation area for clariﬁcation for the user. Dwell timeon input was immediate. Distance to activation icon
was 12 cm.
The hover selection (Fig. 2 (b)) is a common technique where selection is done, by holding a ﬁnger above
an icon for a short period of time. We expected thismethod to have less errors compared to the other two,as the user does not move their ﬁnger from the selection spot. Additionally, touching a single icon might bea more intuitive for interaction. The speed of selection
can also be customized for each user, as we do not know
which speeds are suitable for the elderly participants.
Dwell time on input was one second and four seconds.
In the slide method (Fig. 2 (c)) the user places
their ﬁnger on top of an icon and the activation area
‘SELECT’ appears immediately above it. We assumed
the elderly focusing on an icon would more aware of
the activation icon as it appears in their point of focus.
This method was expected to alleviate the problem of anarrowing ﬁeld of view often presented by old age andreduce large movements of the hand. Dwell time on
input was immediate.

60Paper » Design of Assistive Tabletop Projector-Camera System for the Elderly with Cognitive and Motor Skill Impairments

there either permanently or in an interval care stay period. From the available pool, the elderly were chosen
based on their need for assistance on Instrumental Activities of Daily Living (IADL) and their Mini-Mental
State Exam (MMSE) scores between 30-12. In summary, a score between 30-27 is normal aging, 26-24 iseither normal or mild cognitive impairment (MCI), 2318 is mild dementia, 17-12 is moderate dementia and11-0 severe dementia. Eight elderly users had memory
impairments, four with AAMI and four with mild tomoderate Alzheimer’s. Motor skill limitations ranged
from severe arthritis to slight hand tremors and joint
stiﬀness on nearly all of the elderly. None of the elderly
participants had used smartphones with a touchscreenor any type of tablet device. Only two elderly users had
some experience with computers, one for work and one
for writing at home.
The participants had short pre-interview on theirtechnology experience (PC, tablet, smartphone, older
mobile phone), impairment status (medical records)anda longer post-test interviews for test feedback. The participants ﬁlled out a questionnaire during and after the
test. Table 1 details the elderly participant and general
impairments based on the medical records and questionnaires.
Task: Making a video call task was chosen because
elderly often need to communicate with family members
or caregivers. Task itself is self-explanatory to everyone
and thus should require less learning but with new interaction methods. The steps for the task were:( 1 ) Place the menu launcher card on the table toopen the main menu
( 2 ) Select ‘Make a call’ icon from the available ﬁve
options( 3 ) Select the right person from six possible alternatives (shown as photos and name labels)( 4 ) Close the call by selecting the ‘End call’ icon
( 5 ) Return to main menu by selecting ‘Return’icon
Procedure: The ProCam system was installed onlocation at the university for the students and in thetwo care-homes for the elderly. Before the tests, thesystems main purpose as an assistive tool for the elderly was explained, and the large projection area on
the table was identiﬁed as the usable interaction area.
Finger pointing and the paddle were explained as the
input interaction tools. No guidance was given to theusers during the test. Test administrator (TA) assisted
if there was a system error or if the user felt they could

4. User Study
After initial pilot tests, we conducted a user study
with a task of performing a video call. We wanted toobserve the UI interaction capabilities of the elderly
while using touch-based selection methods on a ProCam system. We studied how inexperience with newtechnology, how cognitive and motor skill impairments
and individual’s characteristics aﬀect the use. In addition we also wanted to compare younger (Y) and elderly (E) users’ performance diﬀerences, and if a purposefully slow dwell-time-on-input diﬀered between theelderly and younger users. We used a 4-second delayon our hover method, which normally had a 1-second
dwell-time-on-input. Our hypotheses were that 1) the
elderly would perform slower and make more errors due
to aging impairments, 2) there is some tradeoﬀ between
using a method with and without veriﬁcation steps on
a ProCam system, and 3) system responsiveness aﬀects
younger users more than the elderly. We wanted to examine the exact nature of the tradeoﬀ. It is important
to note that normally testing user interface for example
when using Fitt’s Law23) , the assumption is that eachuser is cognitively and physically on the same level. In
the case of the elderly, however, individual impairmentsof memory or motor skills aﬀect this premise, so qualitative analysis is required. Due to the scarce availability of similarly skilled and impaired elderly users forthe tests, especially in the case of memory impaired
individuals, our study is limited in scale and scope.
Agreements for tests, data collection and use in publications from the elderly users was gathered before thetest. In the case of more severe mental impairment,
the agreements was given by a family member as a legal guardian, but these cases were not used in our data
analysis. In the experiments, we collected quantitative
data to discover trends, which can then be explained using qualitative data from interviews and observations.
Before testing, we also consulted an ethical committee
who declared an approval was not required in our case.
4. 1 Selection Methods Comparison
Participants: A total of 23 users participated in
the test. Five elderly participants had to be excluded
before and during the test due to severely diminishedcognitive and physical capabilities. Therefore, a total of
18 participant’s results were analyzed. Nine were randomly chosen students from University of Oulu, Finland(mean age 26.6) and nine were elderly users (mean age
89.1) available from two care-homes in Finland living
61

ITE Trans. on MTA Vol. 5, No. 2 (2017)

Time [s]

No.
1
2
3
4
5
6
7
8
9

300
250
200
150
100
50
0

Hover
Slide
Two-step

E1

E2

Count

Count

Fig. 3Count

Table 1 Characteristics of the elderly participants based on medical records.
Age Sex
Cognitive
Physical
87
F
None
Stiﬀness in hand/wrist, reduced hearing
88
F
AAMI
Slight hand tremors, hearing ok
94
F
AAMI
Stiﬀness in hand/wrist, reduced hearing
90
F
AAMI
No motor skill impairments
87
F
AAMI
Arthritis: limited hand/wrist movement
91
M
Mod. Alz.
Hand tremors, reduced hearing, left eye surgery
91
M
Mod. Alz.
Hand tremors, reduced hearing and corrected eyesight92
M
Mild/Mod. Alz.
Stiﬀness in hand/wrist, slowness in movements
82
F
Mild Alz.
No index ﬁnger in dominant hand, little motor skill problems

E3

E4

E5

E6

E7

E8
E9
Y1
Y2
Y3
Elderly and younger users

Y4

Y5

Y6

Y7

Y8

Y9

Comparison of completion time in each method: (a) hover, (b) slide, and (c) two-step methods.
(E=Elderly, Y=Younger.)

6
5
4
3
2
1
0
6
5
4
3
2
1
0

(a) Hover method

6
5
4
3
2
1
0

(c) Two-step method

Mistakes
Non-registered presses
Advice given(b) Slide method

E1

E2

E3

E4

E5

E6

E7

E8

E9

Y1

Y2

Y3

Y4

Y5

Y6

Y7

Y8

Y9

Elderly and younger users
Fig. 4Comparison of mistakes, non-registered pressed, and advice requested in each method: (a) hover,(b) slide, and (c) two-step methods. (E=Elderly, Y=Younger.)ances in the two groups were not equal. As hypothesized the elderly (E) were slower in all methods
compared to younger (Y). The independent t-test revealed statistically signiﬁcant diﬀerences (p<.05) between groups for two-step t(16)=3.4, slide t(16)=5.7
and hover t(16)=4.7.
Figure 4 shows the elderly (E1-9) and the younger(Y1-9) users’ mistakes, non-registered presses, and advice requests, using all three methods. We used theTwo-sample Kolmogorov-Smirnov test to compare mistakes between the young and the old separately foreach three methods. Statistically signiﬁcant diﬀerences(p<.05) were found between the groups for slide method(p=.037) and two-step method (p=.009). We also performed non-parametric tests since the size of the sample

not proceed. Each user completed a video-calling task
twice to diﬀerent targets while using all the three interaction methods (slide, hover, two-step). We crosscompared all methods with each participant. The participants were told to think-out-loud while using thesystem as the sessions were video recorded for closer
analysis. A pre-and post-test video interview and questionnaire was also conducted to gather feedback in moredetail. The interaction data was logged automatically.
The methods were within subjects counterbalanced to
avoid interaction-learning bias and a comparison between the younger users and elderly users was done.
Result: The task completion time variables werenormally distributed as shown in Fig. 3. Levene’s
test for equality of variances showed that the vari62

Paper » Design of Assistive Tabletop Projector-Camera System for the Elderly with Cognitive and Motor Skill Impairments

Fig. 5

Fig. 6

User taps with the ﬁnger even when he was supposed to use a paddle to select.

data is small. Comparing all three methods between
users with AAMI and Alzheimer’s, statistical signiﬁcant diﬀerences were not found. On the other hand,when comparing each method to one another using the
Wilcoxon signed-rank test, we found a slight tendency
to produce less mistakes between two-step and hoverfor users with AAMI (p=.059). Users with Alzheimer’susers did not have (p=.317) the same tendency. While
there was no signiﬁcant diﬀerence regarding mistakes
between all of the methods among the elderly users,
there was a tendency (p=.053) for hover method using the Mann-Whitney test. For the elderly, the hover
method (H) produced the least number of mistakes
compared to slide (S) and two-step (T) methods, H: 9,
S: 23 and T: 21 mistakes. The elderly also made more
mistakes (E: 58, Y: 1) compared to the young. Dueto the intentionally slowed down selection speed of thehover method (4-second delay) in one test, the younger
users felt more frustrated compared to the elderly users,
which was expected. In contrast, the elderly users felt
the slowed down selection made them feel more in control and they had more time to comprehend the userinterface. This also reduced the number of accidental
selections in comparison to the other two methods.
Cognitive impairments resulted in three users callingto a wrong target person as a result of forgetting thetask description. Both groups preferred the ring (16users) and the reasoning by the elderly were e.g. “it’snatural”, “I cannot lose it”, “this is easy” or “it felt
comfortable”.
When presented with a paddle for interaction andnot the ring, we still observed many users tapping theicons with their ﬁnger instinctively even without guidance (Fig. 5).
Several repeating problems occurred where the users
mistakenly tried to interact with the tooltips, guiding
text boxes or users obscured icons accidentally (Fig. 6).
While using the ring was natural for most, some tried
to use the ring itself for selecting instead of using theirﬁngertip (Fig. 7).

Fig. 7User obscuring an icon with the hand.

Trying to select and icon by pressing with thering instead of the ﬁngertip.

Using a guiding cursor resulted in users hoveringover the interaction area instead of touching the icons
straight. We also observed the elderly often reading
the menu or the instructions out loud by pointing at
the icon or text.
During the hover method, many elderly users did not
hold their ﬁnger above the icon long enough to successfully register a selection if they did not understand thetime-bar concept. Younger users noticed the selection
progressing with the time-bar icon.
4. 2 Grouping based on impairments
A person’s age is not a signiﬁcant factor in how wellthe elderly can use new technology. Instead various impairments vary from individual to individual although
age might increase the likelihood of some form of impairment over time. We divided the elderly into diﬀerent groups based on their cognitive or physical capabilities: People with A) normal or very mild cognitive impairment B) with cognitive impairment C) motor skill
impairment D) visually impaired. We looked at theoverall performance with the given tasks of the diﬀerent groups and noticed the following. Group A without
cognitive deﬁciencies needed less time with the interaction and could adjust to various selection methods
eﬀortlessly. Group B with cognitive problems needed
more time and less complex input methods. The more
options they had, the more likely they were to makemistakes. Group C with motor skill issues required direct input methods with the ring and could not properly
hold objects steadily in their hands due to diminishedgrip strength, arthritis (Fig. 8) or hand tremors.
Their impairment also slowed down interaction speed
and accuracy for pointing actions. Group D had people
63

ITE Trans. on MTA Vol. 5, No. 2 (2017)

Fig. 8ties with touchscreens, but overlaying on top of objects
is not possible with a touchscreen device. While theusers felt that interacting with real-world objects beneﬁtted from overlaying assistive graphics on top of objects, some users blocked UI elements with their body.
Thus, interaction space needs to be clearly indicated sothat the user knows its borders. When the projection
is on top of objects, the association between the object and the graphical elements is easy to make as theobject itself creates a perceivable border. More importantly objects themselves can be the interactive surface
the user manipulates. With a touchscreen it is possible to point to the objects placed on its surface but assoon as you remove them from the screen, any interaction becomes indirect and presumable more diﬃcult for
elderly users to comprehend19) .
Regarding selection, we tested three methods for elderly use but did not ﬁnd one single method that was
the most suitable for all of them. As expected, our hypotheses conﬁrmed the elderly’s slow performance andhigh error count. While the hover method was the least
error-prone for everyone, again preferences and impairments aﬀect which method is the most suitable for eachuser.
For future research, it would be beneﬁcial to look
into how sequential tasks could be assisted with projection. In our study, the amount of guidance needed foreach user was inconclusive. We would need to clarify
if, for example, worse MMSE scores aﬀect the amountof information that should be oﬀered to the user at one
time. Notably, the current system is not suitable forusers with more severe forms of dementia, as these users
need a more reﬁned solution that is not possible withthe current level of features. This study showed thatcurrently, the design approach is more suitable for normal or slightly memory-impaired elderly users. In thelong term, we aim to extend ProCam-based systems towards intelligent adaptive elderly care technology thatwould be able to track and analyze potential decline ofperformance in daily tasks. This would allow e.g. displaying more guidance when a person loses focus andstops a task suddenly. Or it would enable caregivers
to target their support to the most necessary situations
or even automatically analyze the impact of improved
medication plans. Furthermore, as a limitation our system does not take into account the three-dimensional
plane so all tracked objects were registered on a ﬂat surface. Also the veriﬁcation of a selection should be tested
with tapping functionality in addition to area-selection

A user suﬀering from arthritis wearing the ring
was faster and more precise than with a paddle
input tool.

with eyesight deﬁciencies in both or one eye. The group
could interact with the UI well, as the icon sizes werelarge enough for all the users. However, age associateddiminished ﬁeld of view aﬀected icons on the edge of the
UI. Some users did not always see a change happening in
the UI as it was located outside if their eyes peripheral
vision. As an additional group E reluctance to use technology was created for some users. This group of users
managed to use the UI interaction with varying degrees
of success, but mostly the reasons for failure or slowness
was a lack of conﬁdence. They needed encouragement
to proceed. Feelings of accomplishment through successful use of the UI were beneﬁcial the further a test
proceeded. Encouragement from other elderly on theirsuccesses was also a positive factor to get the reluctant
users to do the experiments. It is important to note thatmany of the users are within several diﬀerent groups,
so interaction should be designed around a single user’s
personal capabilities. An adaptive UI is the most obvious solution to the various problems the elderly users
might have.

5. Discussion
Our user study focused on ﬁnding out how ProCam
systems should be designed for elderly users. The elderly participants we had in our user study had diﬀerent preferences for selection methods due to the various physical and cognitive impairments so ﬂexibility isneeded. The commonly used and well researched tapping detection is a viable method for elderly technology
use18) , but as discussed by26) , tapping can be problematic for motor skill impaired users. We avoided thisproblem by testing interaction methods that are basedon area pointing. Yet, observations suggest that evenarea pointing has diﬃculties derived from age-related
impairments. We cannot clearly say which method is
the most suitable for each user even if we know their
impairment levels. This is most likely due to the small
sample sizes used in our test.
Interaction design for ProCam’s has some similari64

Paper » Design of Assistive Tabletop Projector-Camera System for the Elderly with Cognitive and Motor Skill Impairments

methods.
In general, projection next or on top of objects does
not confuse elderly users as using various projection
types were clear for the elderly: 1) The elderly treated
the table surface as interactive area naturally. 2) Assistive projections (menu card) were understood as being
related to each other. Yet, we observed various unexpected challenges in the use of our system especiallywith memory impaired. We therefore suggest that ProCam systems should be adapted for each person’s interaction capabilities. Based on the user study and on
the user behaviors with the system (Fig. 9), we recommend taking into account the following points when
designing similar systems for elderly users to increase
the likelihood of better results:( 1 ) A cursor-like element may confuse some users,
young and old, as the basic interaction is alreadyunderstood as being ﬁnger or a separate controlobject operated. Our added red pointer only
added to the confusion and caused the interaction to turn from direct to indirect interaction.
( 2 ) Decline in cognitive skills will most likely result in reduced understanding of the interaction.
In these cases it might be suitable to slow down
the interaction speed and reduce the amount ofelements for the users.
( 3 ) Several selection methods should be available
for diﬀerent impairments or preferences of theusers. In the case of motor skill impairments,
some approaches may obstruct or even prevent
using just one method. Each of the selection
methods we tested (slide, two-step and hover)
was seen as the best method by some of the participants.
( 4 ) Motor skill impairments may prohibit use of
more complex ways of interaction. For example holding an object or rotating the wrist was
shown to be diﬃcult for many elderly. This stiﬀness of joints or weak grip limits the UI to simple
pointing by ﬁnger or open hand interactions.
( 5 ) Visually impaired users need graphical indicators for the UI if the elements are at the edgeof their vision. The same feature would be also
beneﬁcial for regular users as diminished ﬁeld orview is common among the elderly population10) .
This is a tradeoﬀ between using a large projection and the users ability to see or realize thelocation of all of the graphical elements( 6 ) Encouragement to the use of an assistive sys-

Fig. 9User groups and the suggested solutions to theirvarious impairments.

tems should come from other elderly to create
trust for the device. When a user interacts withthe system successfully, it should also create feelings of accomplishment that will also increase
the likelihood of repeated use.

6. Conclusion
We argued that the increasing amount of new technology is diﬃcult to use for the elderly due to inexperience and is most often not designed speciﬁcally forthem. We studied if ProCam would be usable as assistive technology when the elderly have cognitive andphysical impairments.
The ﬁndings from our user study suggest that ProCam is a viable technology for assistance if the interaction and the usability is properly adapted to eachindividual’s skills. Each user’s individual combinationsof both cognitive and motor skill deﬁciencies prohibited
one-for-all interaction design in our study. But in principle, the use of projection can be an eﬀective way topresent information as the elderly ﬁnd the graphical elements projected onto a table and next to objects easy tocomprehend. We suggested practical design approaches
that might improve the creation, testing, eﬀectivenessand the likelihood of acceptance of ProCam UIs for theelderly. We encourage research on the eﬀects of cognitive problems in interaction, but also suggest that motor skill impairments will be a factor in almost all of
the user study cases.
This study was limited by the small amount of available elderly participants, so in our future research we
aim to conﬁrm the proposed design suggestions in aquantitative study, test the guidance of sequential tasks
and how adaptable UI’s could beneﬁt the elderly users.
Especially impact of diﬀerent levels of cognitive impairments would require further studies in order to makefuture ProCam-based assistive systems practical.
65

ITE Trans. on MTA Vol. 5, No. 2 (2017)

References
1) H. Niemelä and K. Salminen: “Kansallisten eläkestrategioiden
muotoutuminen ja Euroopan unionin avoin koordinaatiomenetelmä”(2009)
2) Centers for Disease Control (CDC). The state of aging andhealth in America 2013. Atlanta, GA: Centers for Disease Control
and Prevention, US Department of Health and Human Services(2013)
3) D. Hawthorn: “Interface Design and Engagement with Older
People”, Behavaviour & Information Technology 26(4): pp.333–
341 (2007)
4) T. Broady, A. Chan, and P. Caputi: “Comparison of older andyounger adults’ attitudes towards and abilities with computers:
Implications for training and learning”, British Journal of Educational Technology 41(3): pp.473–485 (2010)5) N. Charness and W. R. Boot: “Aging and Information Technology Use: Potential and Barriers”, Current Directions in Psychological Science 18(5) pp.253–258 (2009)6) K. Mikkola and R. Halonen: “Nonsense? - ICT perceived by
the elderly”, The European, Mediterranean & Middle Eastern
Conference on Information Systems (2011)7) S. Ikeda, Z. Asghar, J. Hyry, P. Pulli, A. Pitkänen, and H. Kato:“Remote assistance using visual prompts for demented elderly
in cooking”, In Proc. International Symposium on Applied Sciences in Biomedical and Communication Technologies, pp.46:1–
46:5 (2011)8) A. D. Wilson: “Using a depth camera as a touch sensor”, In Proc.
International Conference on Interactive Tabletops and Surfaces:
pp.69–72 (2010)9) C. Harrison, H. Benko, and A.D. Wilson: “OmniTouch: wearable
multitouch interaction everywhere”, In Proceedings of the 24th
annual ACM symposium on User interface software and technology, pp.441–450 (2011)10) B. Sekuler, P.J. Bennett, and M. Mamelak: “Eﬀects of aging
on the useful ﬁeld of view”, Experimental aging research, 26(2),pp.103–120 (2000)
11) G. Yamamoto, Z. Asghar, Y. Uranishi, T. Taketomi, C. Sandor,
T. Kuroda, P. Pulli, and H. Kato: “Grid-pattern indicating interface for ambient assisted living”, In Proc. International Conference on Disability, Virtual Reality and Associated Technologies,
pp.405–408 (2014)12) G. Yamamoto, T. Kuroda, D. Yoshitake, S. Hickey, J. Hyry, K.
Chihara, and P. Pulli: “PiTaSu: wearable interface for assisting senior citizens with memory problems”, IJDHD 2011, 10(4):pp.295–300 (2011)13) Y. Asano, H. Saito, H. Sato, L. Wang, Q. Gao, and P.-L. P. Rau:
“Tips for designing mobile phone web pages for the elderly”, In
Proc. HCII 2007, pp.675–680 (2007)
14) C. L. Kelley and N. Charness: “Issues in training older adults
to use computers”, Behavior & Information Technology, 14(2),pp.107–120 (1995)
15) C. J. Ketcham, R. D. Seidler, A. W. V. Gemmert, and G. E.
Stelmach: “Age-related kinematic diﬀerences as inﬂuenced bytask diﬃculty, target size, and movement amplitude”, Journal ofGerontology: Psychological Sciences, 57B(1) pp.54–64 (2002)16) M. W. Smith, J. Sharit, and S. J. Czaja: “Aging, motor control,
and the performance of computer mouse tasks”, Human Factors,
41(3) pp.389–396 (1999)
17) L. Findlater, J. E. Froehlich, K. Fatal, J. O. Wobbrock, and
T. Dastyar: “Age-related diﬀerences in performance with touchscreens compared to traditional mouse input”, In Proc. SIGCHI
2013: pp.343–346 (2013)
18) Z. X. Jin, T. Plocher, and L. Kiﬀ: “Touch screen user interfaces
for older adults: button size and spacing”, In Proc. international
conference on Universal access in human computer interaction:
coping with diversity, pp.933–941 (2007)
19) W. K. English, D. C. Engelbart, and M. L. Berman: “Displayselection techniques for text manipulation”, IEEE Transactions
on Human Factors in Electronics, HFE-8(1), pp.5–15 (1967)20) N. Vigouroux, P. Rumeau, F. Vella, and B. Vellas: “Studying
point-select-drag interaction techniques for older people with cognitive impairment”, In Universal Access in HCI, Part I, HCII
2009, LNCS 5614: pp.422–428 (2009)21) L. Findlater, A. Jansen, K. Shinohara, M. Dixon, P. Kamb, J.
Rakita, and J. O. Wobbrock: “Enhanced Area Cursors: Reducing Fine Pointing Demands for People with Motor Impairments”,

66

Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology, No. 10, pp. 153–162 (2010)22) ARToolKit:
http://www.hitl.washington.edu/artoolkit/ Retreived 21.11.201523) R. W. Soukoreﬀ and I. S. MacKenzie: “Towards a standard forpointing device evaluation, perspectives on 27 years of Fitts’ law
research in HCI”, International journal of human-computer studies, 61(6), pp.751–789 (2004)
24) R. Hyman: “Stimulus information as a determinant of reaction
time”, Journal of Experimental Psychology, Vol 45(3), pp. 188–
196 (1953)25) M. Bakaev: “Fitts’ law for older adults: considering a factor ofage”, In Proceedings of the VIII Brazilian Symposium on Human
Factors in Computing Systems, pp. 260–263 (2008)
26) J. O. Wobbrock and K.Z. Gajos: “A comparison of area pointing and goal crossing for people with and without motor impairments”, In Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility, pp.3–10) (2007)

Jaakko Hyry received his M.Sc. and Ph.D.
in Information processing science from Universityof Oulu in 2008 and 2015 respectively. He was adoctoral student at Nara Institute of Science andTechnology between 2012-2014. He is going to work
as a post-doc researcher at Yokohama National University from April 2017.

Max Krichenbauer is a doctoral student
at Nara Institute of Science and Technology, Graduate School of Information Science, Interactive Media Design Lab. He received his B.Sc. from the University of Applied Sciences Upper Austria in 2010and his M.Sc. from Nara Institute of Science andTechnology in 2014. His research focus is 3D user
intefaces for media production.

Goshiro Yamamoto received his B.E.,
M.E., and Ph.D. in Engineering degrees from Osaka University, Japan in 2004, 2006, and 2009 respectively. Since 2016, he has been a senior lecturer
in Kyoto University Hospital. Before joining Kyoto
University Hospital, he had been an assistant professor in Nara Institute of Science and Technology
for ﬁve years.

Takafumi Taketomi received his B.E.
degree in National Institute for Academic Degrees
and University Evaluation in 2006. He received
his M.E. and Ph.D. degrees in Engineering degrees from Nara Institute of Science and Technology (NAIST) in 2008 and 2011, respectively. He has
been an assistant professor at NAIST since 2011.

Christian Sandor is an Associate Professor at Nara Institute of Science and Technology,
Japan. He previously worked at Canons LeadingEdge Technology Research Headquarters inTokyo, Japan. Before joining Canon, he obtained adoctor-ate in computer science from the Technische
Universität München, Germany.

Hirokazu Kato received his B.E., M.E.
and Dr. Eng. degrees from Osaka University, Japan
in 1986, 1988 and 1996, respectively. Since 2007, hehas been a professor in the Graduate School of Information Science of Nara Institute of Science andTechnology. He received the Virtual Reality
Techni-cal Achievement Award from IEEE VGTCin 2009, and Lasting Impact Award at the 11th
IEEE Inter-national Symposium on Mixed andAugmented Re-ality in 2012.

Petri Pulli received his M.E. with the highest highest honours at the University of Oulu in
1981 where he also received his Ph.D. in 1991. He
has been working as a professor from 1998 and currently also as the dean of education at the University of Oulu.