Title
Sub Title
AuthorPublisher
Publication year
JtitleJaLC DOI
Abstract
Notes
Genre
URL

Research on virtual touch panel interface for handicapped people陸, 灝(Lu, Hao)
小木, 哲朗(Ogi, Tetsuro)
慶應義塾大学大学院システムデザイン・マネジメント研究科
2012

修士学位論文. 2012年度システムエンジニアリング学 第116号
Thesis or Dissertation
https://koara.lib.keio.ac.jp/xoonips/modules/xoonips/detail.php?koara_id=KO40002001-000020120062

慶應義塾大学学術情報リポジトリ(KOARA)に掲載されているコンテンツの著作権は、それぞれの著作者、学会または出版社/発行者に帰属し、その権利は著作権法によって保護されています。引用にあたっては、著作権法を遵守してご利用ください。
The copyrights of content available on the KeiO Associated Repository of Academic resources (KOARA) belong to the respective authors, academic societies, or
publishers/issuers, and these rights are protected by the Japanese Copyright Act. When quoting the content, please follow the Japanese copyright act.

Powered by TCPDF (www.tcpdf.org)

Master’s Dissertation2012

Research on Virtual Touch Panel Interface
for Handicapped People
LU, Hao（Student ID Number：81133668）

Supervisors:
Prof. OGI, Tetsuro
Associate Prof. TATEYAMA, Yoshisuke
Associate Prof. LEE, Hasup

March 2013

Graduate School of System Design and Management, Keio
University
Major in System Design and Management

II

SUMMARY OF MASTER’S DISSERTATION
Student
Identification
Number81133668

Name

LU, Hao

Title

Research on Virtual Touch Panel Interface for Handicapped
PeopleAbstract
Nowadays, the touch panel interface has been used more and more widely. But thehandicapped people are facing difficulties when use the normal touch panel interface. In
this research, we present a novel input method that allows the user to operate a systemwith hand gestures instead of only with their fingers. This system allows the user tooperate instruments without contact. Instead, it can be operated from a distance. Thissystem is designed to help handicapped people who are finding it difficult to reach theoperation button of instruments or are finding it difficult to do actions which can beeasily be done by normal people. In this paper, the concept of a virtual touch panel is
explained. The composition and structure of this system is introduced. The design of ahandicapped people friendly UI is discussed.

Key Word:
Virtual Touch Panel, Non-contact Input Method, Human Interface, Design for
Handicapped People, Kinect

III

IV

TABLE OF CONTENTS
Summary of Master’s Dissertation .......................................................................... III
List of Figures ......................................................................................................... IX
List of Tables ........................................................................................................... XI
1

Introduction ...................................................................................................... 1
1.1

Background ............................................................................................... 1

1.1.1 The Fast Developing Technology and Electronic Controlled World ..... 1
1.1.2 The Great Amount of Handicapped People ........................................... 5
1.2

Requirement Analysis ............................................................................. 12

1.3

Research Purpose .................................................................................... 15

1.3.1 Target User........................................................................................... 15
1.3.2 Target Function of the System ............................................................. 16
1.3.3 Scope of the System ............................................................................ 17
2

3

Related Works ................................................................................................. 18
2.1

Voice-Controlled Elevator ....................................................................... 18

2.2

Non-Contact Touch Display .................................................................... 19

Concept ........................................................................................................... 21
3.1

Composition ............................................................................................ 21

3.2

Key Technology - Kinect Sensor............................................................. 22

V

3.3
4

Data Flow ................................................................................................ 26

GUI Design ..................................................................................................... 28
4.1

The Differences and Attention Points of Design ..................................... 28

4.1.1 Distance cannot be Directly Known .................................................... 28
4.1.2 Relation between Cursor Movement and Button Activation ............... 29
4.2

Traditional Matrix Design ....................................................................... 30

4.2.1 Disadvantages ...................................................................................... 31
4.3

Circular Design ....................................................................................... 34

4.3.1 Advantages .......................................................................................... 37
4.3.2 The Best Length of Radius .................................................................. 40
4.4

Compare between Matrix Design and Circular Design .......................... 43

4.4.1 Average Operation Time ...................................................................... 45
4.4.2 Accuracy .............................................................................................. 47
5

Operation Engine Design................................................................................ 49
5.1

Algorithm of Operation Engine............................................................... 49

5.2

Noise Cancellation .................................................................................. 50

6

Logging Function ........................................................................................... 54

7

Verification and Validation ............................................................................. 56
7.1

Verification Test ...................................................................................... 56

7.2

Validation ................................................................................................ 58VI

7.2.1 Interview with Ms. Qiu........................................................................ 58
7.2.2 Evaluation by Handicapped User ........................................................ 59
8

Discussion....................................................................................................... 61
8.1

Compare between Buttons ...................................................................... 61

8.2

Compare with Physical Touch Panel ....................................................... 64

8.2.1 Fitts’ Law ............................................................................................. 64
8.2.2 Equation for Virtual Touch Panel Interface ......................................... 65
8.2.3 Equation for Physical Touch Panel Interface....................................... 69
8.2.4 Comparison.......................................................................................... 70
9Conclusion ...................................................................................................... 72
9.1

Summary ................................................................................................. 72

9.2

Future Work ............................................................................................. 72

Published ................................................................................................................. 73
Acknowlegment ....................................................................................................... 74
References ............................................................................................................... 76

VII

VIII

LIST OF FIGURES
Figure 1: Re-Imagination of Note Taking in Nowadays ........................................... 2
Figure 2: Number of Internet Users and Penetrate Rate in Japan ............................. 3
Figure 3: The Electricity Usage in Japan .................................................................. 5
Figure 4: Proportion of People with Physical Disability ........................................... 7
Figure 5: History Numbers of People with Disability ............................................... 8
Figure 6: History Data of People with Physical Disability ....................................... 9
Figure 7: The Historical Data and Future Forecast of Aged People........................ 11
Figure 8: Examples of Control Panels of Electronic Controlled Machine .............. 12
Figure 9: A Handicapped Panel in Elevator ............................................................ 13
Figure 10: Handicapped People Facing Difficulties Operating Elevator ................ 15
Figure 11: Non-contact Touch Screen ..................................................................... 19
Figure 12: Virtual Touch Panel Display ................... Error! Bookmark not defined.
Figure 13: The Composition of the Virtual Touch Panel DisplayError! Bookmark
not defined.
Figure 14: Infrared Line Illuminator in Virtual Touch Panel Display .............. Error!
Bookmark not defined.
Figure 15: Composition of the Virtual Touch Panel ................................................ 22
Figure 16: A Kinect Sensor ..................................................................................... 23
Figure 17: The Construction of Kinect Sensor ........................................................ 24
Figure 18: The Interaction Range of Kinect Sensor ................................................ 25
Figure 19: The Data Flow of the System................................................................. 26
Figure 20: A Matrix Design of Button ..................................................................... 30
Figure 21: Example of Differ between Start and End ............................................. 33
Figure 22: The Reachable Range of the User’s Hand ............................................. 34
Figure 23: A Circular Design of Button .................................................................. 35
Figure 24: A Working Example of Circular Design ................................................ 36
Figure 25: The Hand Cover Range and the Size of the Button ............................... 38
Figure 26: The Environment of Test on Different Radius ....................................... 41
Figure 27: The Result of the Test on Different Radius ............................................ 43
IX

Figure 28: A Tester is Operating the System ........................................................... 44
Figure 29: The Average Operation Time on Different Design ................................ 47
Figure 30: The Movement Direction and the Operation ......................................... 50
Figure 31: Real Hand Movement and Hand Position Got by Kinect Sensor .......... 51
Figure 32: Noise Reduction ..................................................................................... 52
Figure 33: The environment of the Verification Test............................................... 56
Figure 34: Accuracy by Different Heights .............................................................. 57
Figure 35: Average Operation Time by Different Heights ...................................... 58
Figure 36: The Average Operation Time of Different Buttons ................................ 62
Figure 37: Accuracy between buttons ..................................................................... 63
Figure 38: A tester is operating the system in the experiment ................................. 66
Figure 39: Test for Fitts’ Law .................................................................................. 67
Figure 40: Result of Test for Fitts’ Law .................................................................. 69

X

LIST OF TABLES
Table 1: The Number of Disabled People in Japan ................................................... 6
Table 2: The Interview of Ms. Qiu .......................................................................... 14
Table 3: Kinect Sensor Components and Specifications ......................................... 24
Table 4: The Average Operation Time of Different Radius ..................................... 42
Table 5: The Result of ANOVA on Operation Time by Different Radius ............... 42
Table 6: The Average Operation Time of Different Design .................................... 45
Table 7: The Result of Student-t Test on Operation Time by Different Design ...... 46
Table 8: The Result of Operation on Matrix Design ............................................... 47
Table 9: The Result of Operation on Circular Design ............................................. 48
Table 10: The Result of Student-t Test on Accuracy of Different Design ............... 48
Table 11: The Second Interview of Ms. Qiu............................................................ 59
Table 12: The Average Operation Time of Different Buttons ................................. 61
Table 13: The Result of ANOVA on Operation Time by Different Buttons ............ 62
Table 14: The Accuracy of Different Buttons.......................................................... 63
Table 15: The Result of ANOVA on Accuracy by Different Buttons ...................... 64
Table 16: Result of Test for Fitts’ Law .................................................................... 68

XI

XII

Graduate School of System Design and Management, Keio University

1 INTRODUCTION
1.1 Background1.1.1 The Fast Developing Technology and Electronic Controlled World
Nowadays, science and technology develop very fast. No matter you want to admit
it or not, we actually have stepped in a brand new era of the Information Age, or commonly
known as the Digital Age, since the early 1970s on the grounds that the technology ofcomputer microminiaturization began to be broken through.
After that, personal computer and internet sprung up all around the world in the next
two decades. From the year 1990 till now, the pervasive adoption of such technology by
the public has brought about a great rapid evolution of technology in our daily life. It is amass of breakthroughs in information processing, storage, and transmission that makes theapplication of Information Technology possible in virtually all corners of society.
Let’s give an example. We used pencil and notepad to jot down something reminder
in the past. While now it’s the notebook software in computer or cell phone that obliges,
being always synced by web cloud. More than that, pictures and audios in the software areenabled and all these stuff saved is searchable.

1Graduate School of System Design and Management, Keio University

Figure 1: Re-Imagination of Note Taking in Nowadays
However, this instance is just a tip of the iceberg. The electronic technology of theInformation Age has an overwhelming and comprehensive impact affecting every humanbeing on Earth in every aspect of his or her life. Look around the world we live in.
Newspaper has become electronic news in portal website, twitter and so forth.
Newsweek, the second largest and popular weekly magazine in United States, has
stopped its printed version and transition to all-digital version from the year 2013 [1].
Ecommerce challenges traditional stores. Paper diary book has been threatened bypersonal blog with multimedia, location-aware and share function enabled etc. So many
manual tools around us are almost replaced completely by electronic products.
At present, Japan is also the electronic controlled society without exception. The
growth of internet users in Japan is one proof of it. Two indicators that are often used asstatistic factors measure the extent of informatization in a given region by International
Telecommunication Union: Internet Penetration Rate and the Number of Internet Users [2].
The internet users here are persons using the Internet from any device including
mobile phones. Internet penetration rate refers to the number of active internet users within2Graduate School of System Design and Management, Keio University
a specific population. Figure 2 shows number of Internet Users & Penetration Rate inJapan from 2001 to 2011 from the statistic data from household surveys and Internet
subscription data by Japanese government [3].

Figure 2: Number of Internet Users and Penetrate Rate in JapanIn Figure 2, the blue bar shows that the number of internet users by year in the periodof 2001 to 2011. The internet users increased from 55.93 million in 2001 to 96.10 million
in 2011, the percentage of increment is over 70 percent in the decades. The red lines shows
the Internet Penetration Rate increases from 46.3 percent in 2001 to 79.1 percent in 2011.
Over 30 percent had been increased in the decades. The statistic data tells a story about theastonishing growth of electronic appliance usage in Japan. They have spread over in ourlife without our attention.
Another data support for the extensive usage of electronic appliance is the growth of3

Graduate School of System Design and Management, Keio University
electricity demand in Japan.
Speak of electricity, we still remember nearly 2 centuries ago, on an August day in
1831, young Michael Faraday, tinkering in a British laboratory, placed two wires on a ringof soft iron, discovered the secrets of electromagnetism and blazed the way to the miracle– Electric Era [4].
After that electricity almost developed at the speed of light. There appeared hundredsof thousands of uses of electricity, commercial and scientific, such as light bulb, telegraph,
telephone, television, electronic mobile, electric bus, microwave, cell phone, too numerous
to enumerate. Electricity penetrates into our life anywhere and everywhere in summary byusing for lighting, heating, motive power, telecommucation and the basic of electronic
technology. The most miracle gift electricity brings to us is that it’s the No. 1 convenient
way to transform energy. At present, electricity seems like blood of human society.
Community would become wan without electricity even for a short time.
Figure 3 below unfolds that the output of power in Japan fluctuated during 1900 to
2010, but the main trend increased stably [5].

4Graduate School of System Design and Management, Keio University

Figure 3: The Electricity Usage in JapanIn Figure 3, the blue line is the electricity usage for lighting use, the green line is theelectricity usage for electricity power and the red line is the total usage of electricity inJapan during the year 1965 to the year 2010. The unit in the graph is billion kilo watt hours.
The figure shows an unquestionable fact that the usage of the electricity both in lighting
and electricity power increase at a stable pace.

1.1.2 The Great Amount of Handicapped People
Handicapped people in Japan have the same rights as people without disabilities.
They deserve inherent respect for their human worth and dignity. And in many instances
there is a need for updated infrastructures and services for them to embody the attentions,concerns and respects from non-disabled people. Promoting accessibility including built
5Graduate School of System Design and Management, Keio University
environment, public facilities, social welfare, etc. to handicapped people is an effective
way to improve their quality of life, their feeling of happiness and the sense of social
belonging.
1.1.2.1 Existing Circumstances of handicapped people
According to the latest census from Japan Cabinet Office (CAO), the number ofhandicapped people currently in Japan is 7,443,000. Among them, 3,663,000 is the countof people with physical diability. The number of people with cognitive diability is 547,000.
And people with mental diability is 3,233,000. Compare to the current Japanese population,
among 1000 people, 58 people is disabled. Among them, 29 people are physically disabled,
4 people are cognitivity disabled and 25 people are mentally disabled. [6]Table 1: The Number of Disabled People in JapanThe data above is from the the last census which has already published its result. The
census takes every 5 years, and the data above is from the census in 2006.
Commonly, physical disability refers to impairment which limits the physicalfunction of fine or gross motor ability. Here in our data, physical disability includes visual
impairement, hearing impairment, dysphonia, limb impairment and internal impairment.
6

Graduate School of System Design and Management, Keio University
According to CAO’s explanation, internal impairment, as opposed to other types of
impairment displayed above, is about organ impairment or several chronic disorders, suchas diabetes, AIDS, etc. Hence, people with limb impairment and internal impairment are
the most possible group which requires wheelchairs or medical walkers to help them move.
The census result shows that 8.8 percent of physical disability is visual disability, 10.1
percent is hearing disability, limb disability is 50.6 percent and internal disability is 30.5.
So our target group comprises 81.1 percent of physical disability. That’s equal to about 24
people of one thousand people.

Figure 4: Proportion of People with Physical Disability

7

Graduate School of System Design and Management, Keio University
1.1.2.2 Growth of handicapped people
We want to know how the number of target group—people with limb and internal
disability changes in the future since we have to make decision about how much
infrustructures we should design for them. In order to find this answer, we ought to figureout what factors influence the change of the number.
Figure 5 below is about the history numbers of people with disability censused by
CAO every five years from 1976 to 2006. It indicates that the number of people withdisability above 65 years old during this time increased year after year from 442,000 to
3,576,000. But the part of people with disability under 65 years old did not change verymuch. Its population basically kept a stable number between 872,000 and 1,237,000. So
we conclude that the number of people with disability rose because of growth of people
with disability above 65 years old.

Figure 5: History Numbers of People with Disability
8

Graduate School of System Design and Management, Keio University*Note: the number of people under 17 years old was not censused in 1981.
The reason why the number of old people with disability grew over time can be
explained in Figure 6.

Figure 6: History Data of People with Physical Disability
Figure 6 evinces explicitly that the amounts of people with limb disability, hering
disability, dysphonia or visual disability holded stably during 30 years. However, thenumbers of people with internal disability grew rapidly from 1, 408, 000 in 1976 to 3,
576, 000 in 2006. It tells the reason for growth of people with disability is the growth ofinternal disability. Generally, old people is easier to suffer internal illness than young
9Graduate School of System Design and Management, Keio University
people.
Consequently, syntherizing Figure 5 and Figure 6, we mke a conclusion that target
people consist of people with limb disability and people with internal disability. Both ofthem mostly need wheel chair or medical walker to help them move. While the number ofpeople with limb disability is basically constant, the number of people with internal
disability is variable. The reason of this change is the growth of number of old people who
suffer internal diseases. The radical reason is the growth of old people due to aging
problem in Japan. Therefore, old people is the potential people who is high risk to expand
the population of handicapped people.
1.1.2.3 Prediction about the number of handicapped people
Japan is haunted by the problem of aged society for a long time. The statistic data thatJapanese Ministry of Internal Affairs and Communications publishedindicated that Japan
had the highest percentage of people over the age of 65 in the world in 2007.
According to the newly updated statistic data from the Japan Statistic Bureau, 24.0%
of population is 65 years or older. The number amounts to that 30.58 million people inJapan are 65 years old or older at August 2012 [7].
Figure 7 shows historical statistic data and the future forecast about demographic oldpeople in Japan. The historical data is get by Japan Statistic Bureau. The census data of
population is taken every five years from 1960 to 2010. The future forecast is the forecast
according the historical data till the year 2060 [8].

10Graduate School of System Design and Management, Keio University

Figure 7: The Historical Data and Future Forecast of Aged People
In Figure 7, the green bar indicates the population of people include and under 19
years old, the red bar indicates the population of people between 20 and 64 years old, theblue bar indicates the population of people between 65 and 74 years old and the pink bar
indicates the population of people include and over 75 years old. The number on the top towhole bars is the total population in Japan. The red line indicates the percentage of
population of people over 65 years old on the total population.
Figure 7 reveals clearly that the number of 65 years old or older people in Japan
increased stably in the history. Due to the change of population has hysteretic quality, thepopulation of old people will keep its growth in a very long future time. The University ofDenver once modeled the growth of old people in Japan given the specific birth rate anddeath rate, presumably the population policy would not change greatly in the future. From
the figure, we can know that percentage of Japanese 65 years old or older would not stop
increasing even at the year 2060.
As we previously discussed, the aging people has a direct relationship with the11Graduate School of System Design and Management, Keio University
number of handicapped people or people who sitting in the wheelchairs, the data shows
the number of handicapped people will keep increasing.

1.2 Requirement Analysis
The fact we showed here in this section shows it is clear that now we are inInformation Era and the machines use electricity power is all around our world and daily
life. Normally, the electronic controlled machines used a button or panel for the operation.
The buttons or panel is normally allocated on the wall or on the surface of the machine.
Even if the buttons transform from switch of light to touchpad with computer to touch
screen on iPad, they still need to be touched to control.

Figure 8: Examples of Control Panels of Electronic Controlled Machine12

Graduate School of System Design and Management, Keio University
For the normal people like us, we can use the panels without inconvenient. But forhandicapped people sit in the wheelchair, they may find a problem that they cannot use thepanel because they cannot reach it. Normally the manufacturer set a handicapped-peopleused panel in a lower place.

Figure 9: A Handicapped Panel in Elevator
But is the panel enough for handicapped to use? Are every handicapped people
satisfy with the panel and won't face any problem when operating the machine?
About this question, we interviewed the Ms. Qiu, Director of Accessibility Supervise
Department from Shanghai Disabled People Assistive Devices Center, one of the official
accessible devices support center of China Disabled Persons’ Federation, about thesituation of the handicapped people operating the panel devices like elevator. Thisobjective of this department is to supervise the accessibility devices in the public area tomake the city more accessible for handicapped people to use. She is working for the13

Graduate School of System Design and Management, Keio University
department for years and now as a director, she knows very well about the accessible
devices and handicapped people’s situation.
Table 2: The Interview of Ms. Qiu
Q Is the handicapped people facing difficulty when operating the machines by buttons
or panels like elevator?
A A lot of place have accessible devices like the lower panel in the elevator. The
situation of the handicapped people can’t operate the panel has been improved.
Q For the people who can move his/her arms and hands but face problem of movehis/her body, will they be able to reach the button and panels?
A This is a very severe conditions, the people who can only move his/her arms andhands will face a problem that the button is in front of him/her, but they may not beable to reach it. This is also a problem that we want to solve.
Q If there is a device that can allow the user to operate the button or panel only needmoving his/her hand without needing to touch or reach any button or panel, will thishelp the handicapped people?
A Yes, that’s true. It will help a lot.

From Table 2, we can know that although the panel is in a lower place, it is also notvery easy for the handicapped people to use if the handicapped people can’t move his/herbody but only can move his/her arm. They cannot reach the button if they moves towards
the button and the button is in front of them even if their arms and hands can move freely.
To investigate the situation, we did an experiment. We let a tester sit in a wheelchair
and let him try to reach the handicapped panel without moving his body to see if the14Graduate School of System Design and Management, Keio University
handicapped people can or cannot reach the buttons.

Figure 10: Handicapped People Facing Difficulties Operating Elevator
From Figure 10, we can see clearly that there is still a distance between the user’s
hand and the panel. If the handicapped people cannot move forward his body, he will not
be able to reach the button and operate it.
From the interview, we know that the requirement of the handicapped people is a
device that only need to move his/her hand without needing to touch or reach the button or
panel.

1.3 Research Purpose
In this research, we want to help these handicapped people or people sit in thewheelchair to meet their requirement: operate the system without needing reach or touchthem.

1.3.1 Target User
There is a lot kinds of people who need to sit in a wheelchair. One of them is totally
physically disabled and can’t move most part of his/her body. Normally the people can’teven operate the wheelchair by himself/herself and often have campanions to help them15Graduate School of System Design and Management, Keio University
doing things that he/she want to do.
The target people who we need to help is: people need to use a wheelchair to move in
the ground and have the ability to use his/her arm freely. These people have the ability tooperate buttons by themselves if the button is not out of reach.t
As we discussed previously, the disability investigated is catagerized into three types,
physical disability, cognitive disability and mental disability by CAO. However, not allthese people are the target people on our research. The mental and cognitive disabled
people normally will find it difficult to know what he/she should do to operate the systemor what he/she shouldn’t do. And normally those people have campanions who help themmoving and living. Those people can help them with the operating the normal system
normally. The target user of the system is the people who needs wheelchaires to movearound but with no problem of move his/her arms and knowing that he/she should moving
his/her arms. Normally, the target user is physical disabled people and the aged people
who need wheelchair to move.

1.3.2 Target Function of the System
The main target function of the system is to let the user can operate the buttons
without needing them to touch the buttons. The system shall let the user operate the buttons
when the user is still have a distance with the system, normally 1 or 2 meters. In thisdistance, the user can see the system and put out his hand, moving his hand in the air andoperating the buttons.
Another function is allow the user to operate the buttons with his/her whole handinstead of his/her finger. The finger move is a much complex move than moving the wholehand. When move the whole hand, only the joint of elbow, shoulder and wrist need to16Graduate School of System Design and Management, Keio University
move, but when moving the specific finger, the joint of elbow, shoulder and wrist still needto move, the joint in the hand and the finger also need to move. Some handicapped people
may not be able to complete such a complex movement.

1.3.3 Scope of the System
Theoretically, the system can be used in any system that need the user to operate thebutton. For example, elevators, security doors that need password, call panel in the hall of
the apartment etc. The system can work as an extra application, works simultaneously withthe existing physical buttons.

17Graduate School of System Design and Management, Keio University

2 RELATED WORKSFirstly, let’s pick up some related work on the virtual touch panel interface and see
what is the benefit and the disadvantages for the virtual touch panel for handicapped people
to use.

2.1 Voice-Controlled Elevator
Recently, companies also find out the problem of the handicapped people can’t reachand operate the operation panels in elevator. Some non-contact or touch-less elevator isdeveloped. One of the product is voice-controlled elevator. Mitsubishi Electric Corporation
developed its new model of voice-controlled elevator in 2011 [9].

Figure 11: Voice Controlled Elevator
This products works fine but the voice recognition part is a problem for the use.
Normally, the voice recognition relays on a pattern dictionary and use pattern recognition
to recognize the word. The system in this Mitsubishi elevator also use this technology.
In the Mitsubishi elevator, the system can recognize the phrase like “Floor 1”, “Floor
2” etc. in Japanese. The problem of the pattern recognition is user have accents, different
18Graduate School of System Design and Management, Keio University
accents will be treated as different pattern in the sound wave. Also, the way of the user
make command may be different. Some users may even cannot speak Japanese. All theseuser will find a difficult of operating this elevator. But the problem is if we want to makethis system better as we want to make the recognition more accurate and widely, we willneed a huge dictionary with all of the patterns. This leads to 2 problem that first we needto build up a huge dictionary and second we need to link this huge dictionary into theelevator. Both will cause huge cost and time.

2.2 Non-Contact Touch Display
Xiang et al. from University of Electronic Science and Technology of China
designed a non-contact touch display [10].
The basic concept of the system is using a laser pointer to point the button on thedisplay, with using a camera to get the video and image of the display and detecting theposition of the laser pointer to determine the touch operation.

Figure 12: Non-contact Touch Screen

19Graduate School of System Design and Management, Keio University
The technology of the system is good and the accuracy of the touch position is veryhigh. The problem is the system cannot be implemented in a public place like elevator andsecurity door as users usually don’t carry a laser pointer with them. If we put the laser
pointer some places near the system, we don’t need this system as if the user can reach andtake the laser pointer, we can just put the operation button in the same places where we aregoing to put the laser pointer as the user can reach and use it.

20Graduate School of System Design and Management, Keio University

3 CONCEPTThe goal of virtual touch panel interface is to provide an input method which canallow the user to operate a system by using their whole hands instead of moving theirfingers, which are very complicated actions that many handicapped people even a smallpart of non-disabled people cannot handle very well. Also, we want to give users a way to
doing operations from a distance, which can allow the user to do the action in a more
comfortable way, with no functional absence compared with a normal touch panel.

3.1 Composition
In order to build such a system, we need 3 fundamental parts to help us:Display panel



Kinect sensor

PC

The display panel in this system shows the virtual panel with the visual button andshows the current hand position of the user. It will help the user with visualized information
to lead the user finish the operation just like the display in many other systems do.
The Kinect sensor is a COTS machine produced by Microsoft Corporation. Having
a Kinect sensor in our system can help us in detecting the user’s hands and getting the realtime data of hand position. Using this data, we can do the hand tracking and detecting theoperation by users [11]. (See detail in Section 3.2)
The PC is a necessary part. The PC is used for running the program and processing
the data for the reaction of the operation. And also the PC is being used in debugging to
improve the system.
21

Graduate School of System Design and Management, Keio University

Figure 13: Composition of the Virtual Touch Panel
The display panel and the Kinect sensor is direct in front of the user, while the PC isin front of the operator or monitor of the system if needed. If the system is fully developed
and the debugging is unnecessary, we can simplify the system to only PC and Kinect sensor,
with PC is using for both processing data and showing the panel.

3.2 Key Technology - Kinect Sensor
A Kinect sensor (also called a Kinect) is a motion sensing input device developed byMicrosoft. It enables the users to control and interact with the system without need to touch
buttons or keys [11].

22Graduate School of System Design and Management, Keio University

Figure 14: A Kinect Sensor
The Kinect sensor is a horizontal bar connected to a small base with a motorized
pivot and is designed to be positioned lengthwise above or below the video display. Inside
the sensor case, a Kinect sensor contains: An RGB camera that stores three channel data in a 1280x960 resolution. This
makes capturing a color image possible.
 An infrared (IR) emitter and an IR depth sensor. The emitter emits infrared lightbeams and the depth sensor reads the IR beams reflected back to the sensor. The
reflected beams are converted into depth information measuring the distance
between an object and the sensor. This makes capturing a depth image possible.
 A multi-array microphone, which contains four microphones for capturing sound.
Because there are four microphones, it is possible to record audio as well as findthe location of the sound source and the direction of the audio wave.
 A 3-axis accelerometer configured for a 2G range, where G is the acceleration
due to gravity. It is possible to use the accelerometer to determine the current
orientation of the Kinect.
23

Graduate School of System Design and Management, Keio University

Figure 15: The Construction of Kinect Sensor
The detail specification of the Kinect is list in Table 3:
Table 3: Kinect Sensor Components and Specifications

KinectArray Specifications

Viewing angle

43°vertical by 57°horizontal field of view

Vertical tilt range±27°

Frame rate (depth
and color stream)

30 frames per second (FPS)

Audio format

16-kHz, 24-bit mono pulse code modulation (PCM)

Audio input
characteristics

A four-microphone array with 24-bit analog-to-digitalconverter (ADC) and Kinect-resident signal processing
including acoustic echo cancellation and noise suppression

24Graduate School of System Design and Management, Keio University

Accelerometercharacteristics

A 2G/4G/8G accelerometer configured for the 2G range,
with a 1°accuracy upper limit.

The interaction space is the area in front of the Kinect sensor where the infrared andcolor sensors have an unblocked view of everything in front of the sensor. If the lighting
is not too bright and not too dim, and the objects being tracked are not too reflective, you
should get good results tracking human skeletons. While a sensor is often placed in frontof and at the level of a user's head, it can be placed in a wide variety of positions.
The interaction space is defined by the field of view of the Kinect cameras. The
sensor supports tilt function by using the built-in tilt motor. The tilt motor supports anadditional +27 and -27 degrees, which greatly increases the possible interaction space infront of the sensor.

Figure 16: The Interaction Range of Kinect Sensor

25

Graduate School of System Design and Management, Keio University

3.3 Data Flow
In this system, we use the Kinect sensor’s IR depth sensor to get the real-time depth
map of the real world. Depth map is an image that contains information relating to the
distance of the surface of objects from the viewpoint. The depth map is sending to PC forprocessing.

Figure 17: The Data Flow of the System
In the PC, we use the program written with Microsoft Kinect SDK for Windows toprocessing the depth map sent by the Kinect sensor. From the depth map, we can get thegeneral outline of the object in the sensing range of the camera. We judge the outline ofobjects one by one to recognize the user among all the objects.
After we recognized the user, we attach the skeleton model of the user to recognize
the hands of the user. The system will first recognize the start position when the user stop
his/her hand for 1 second and record the start position and shows a cursor in the center ofthe display. After that the system will recognize the operation by the operation engine in26

Graduate School of System Design and Management, Keio University
the system to recognize the operation by the user’s hand movement.
Beside the cursor, the current selected button will highlight in the display and a
textbox on the display will show the result of the user’s operation to give user the feedback.

27

Graduate School of System Design and Management, Keio University

4 GUI DESIGN
The most important part of a human interface is the GUI (Graphic User Interface).
This part is directly seen by users, and the good or bad aspects of the GUI have a direct
impact on the usability of the system.

4.1 The Differences and Attention Points of Design
The virtual touch panel interface is a novel interface that different to the physicalinterface and current human-machine interface. So before discuss about the detail design,
we shall first discuss about the differences and points we need to pay attention in theinterfaces design of virtual touch interface.

4.1.1 Distance cannot be Directly Known
In a physical user interface. The operation is directly. User use his/her hand directly
touch the panels and the buttons. The position of the buttons is at the same place where theuser see where they are. The user can know exactly how much they need to move to reachthe target button. So normally the user won’t find any problem to reach and operate thephysical panels or buttons.
But in a virtual touch panel interface, the user cannot direct operate the button by
his/her hand. The user need to operate the system by operate the cursor by moving his
hand. Because the cursor only a relatively relationship with user’s hand, user can know
exactly which direction the cursor is going to move before the user move his/her hand, butwill not know exactly how much the cursor is going to move.
User don't exactly know how much he/she need to move his/her hand in order to get

28Graduate School of System Design and Management, Keio University
to the target button because the user doesn’t know the speed of the cursor until the user
pays his/her learning time to get used to this system.

4.1.2 Relation between Cursor Movement and Button Activation
Normally in a user interface, the cursor movement and button activation operation iscontrolled by different module. Like when the user is operating a computer with a mouse,
left button is used to activate the button while move the cursor use the optical sensor or
trackball. But in a virtual touch panel interface, both the cursor movement and button
activation requires the user to move his/her hand.
If we consider the X axis as left to right, Y axis as down to up, Z axis as behind to
forward. The value of X and Y will be used to decide the cursor’s position and Z value will
be used to decide if the user is doing the button activation operation. When the user wantto move a cursor, the ideal situation of the operation is to change the X and Y value andnot change the Z value. And ideal situation of operation is to change the Z value while notchanging the X and Y value.
But such operation is impossible to be done as when the user move his/her hand, thedirection is always not absolute parallel to the axis and normally even not in a line. And
because when the user operates the system, the user’s hand is moving in the 3D space
without anything to be used as reference object for the axis. The user’s feeling of the axis
may have a degree to the real axis of the system. The hand movement done by the user
may deviate from the direction the user want to move. Also the Kinect sensor is a machine
which have machine error that the hand position recognized by the sensor is not absolutely
accurate. This will lead the hand movement detecting by the system is not a perfect line
parallel to the XY plane when moving the cursor or along the Z axis when activating the29Graduate School of System Design and Management, Keio University
button. Normally the movement detect by the system is a curve line along the user feelingaxis.
This may lead the system misjudge the current cursor movement operation or part ofthe operation as a button activation operation and vice versa. In order to make the system
recognize the correct operation, we need to pay attention and make special design in every
part of the system.

4.2 Traditional Matrix Design
Traditionally, we align the buttons in rows and columns. A sample of the matrix
design of number input system is shown in Figure 18.

Figure 18: A Matrix Design of Button
The matrix design is a very common human-machine interface. It is being widelyused both in the graphic user interface in the display and operation buttons or panels in thereal world. It works very well when the panel is a real panel in a flat plane and the user
operates it in 2D interface. But according to the previous section, we can find that thematrix design are not very good in the virtual touch panel interface. Here are several

30

Graduate School of System Design and Management, Keio University
disadvantages.

4.2.1 Disadvantages
4.2.1.1 Disadvantage 1: The distance between buttons is not easy to know.
In the matrix design, the buttons is a limit-sized shape, either the user move thecursor too less or too more won’t let the cursor stops on the target button. As we previous
discussed, the user operating a virtual touch panel interface cannot easily know about the
distance, because the user doesn’t know the speed of the cursor until the user pays his/herlearning time to get used to this system.
The same problem happens when the user uses the mouse to operate the cursor in thecomputer’s display. It takes time to get used to it. The time seems very little because
normally we operate the computer in hours and normally we have experiences. But for anovel system like the virtual touch panel and the normal operation time is in seconds. The
learning time costs too much.
4.2.1.2 Disadvantage 2: May pass through other buttons when moving from current
position to target button.
In matrix design, if the user want to move the cursor from a button to another button,
he may pass through another button. Like in Figure 18, if the user wants to move frombutton 4 to button 0, he/she must pass through button 7 or other buttons if he/she use
another route.
As we previous discussed, in this system, both the cursor movement and button
activation requires the user to move his/her hand. When the user want to move a cursor,
the ideal situation of the operation is to change the X and Y value and not change the Z31

Graduate School of System Design and Management, Keio University
value. But such operation is impossible to be done as when the user move his/her hand, thedirection is always not absolute parallel to the axis. The hand movement done by the user
often is a curve and had a big change of Z value. Also the Kinect sensor is a machine whichhave machine error that the hand position recognized by the sensor is not absolutely
accurate. This may lead the system misjudge the current cursor movement operation as abutton activation operation. If there are buttons between the current position and the target
button, the action that the user takes to move the cursor may end up with the wrong
activating operation and activate the wrong button.
4.2.1.3 Disadvantage 3: The button is surround by lots of other buttons.
In matrix design, each of the buttons is surround by other buttons. If we called theother buttons surrounding the current button as “neighborhood button”, the number of theneighborhood buttons is at least 3 when the current button is in the corner and at most 8
when the current button is not on any side of the panel.
As we discussed previously, in this system, both the cursor movement and button
activation requires the user to move his/her hand. When the user want to activate such abutton, the ideal situation of the operation is to change the Z value and not change the X
and Y value. But when the user move his/her hand forward, the direction is always not
absolute forward. There are some changes of the X and Y even the user doesn’t want to.
This will cause a problem that when the user want to activate such a button, his/her handmoving cause the cursor make movements too. This may leads this action end up withactivate the neighborhood button instead of the target button.

32

Graduate School of System Design and Management, Keio University

Figure 19: Example of Differ between Start and End
This possibility of wrong operation increases as the number of neighborhood button
number increases. As the matrix design have a lot of neighborhood button of the target
button, it doesn’t performance very well in this situation.
4.2.1.4 Disadvantage 4: The distance between each button is not equal. Maximum
distance may be too much for user to reach.
In matrix design the distances between the buttons is different. Like in Figure 18, thedistance the user need to move from button 4 to button 7 is not equal to the situation he/sheneed to move from button 4 to button 9.
In this system, the user move his hand in front of the camera, but not all the place ofthe sensing range of the camera can be reached by the user. Only a limited range can bereached by the user.
Figure 20 shows the user in the sensing range of the camera. The whole image is theinfrared image get by the Kinect sensor, where the pink is the user detected and recognizedby the system. The red dot indicates the current position of the user’s left hand. The big
33

Graduate School of System Design and Management, Keio University
red circle is the approximately reachable range of the user’s hand.

Figure 20: The Reachable Range of the User’s Hand
Because of the distance between each button is not equal. The distance between thetarget button and the current position too large. It may cause the operation time being
enlarged and there is a possibility that the distance is too large that the user cannot use hishand to move such a distance. As long as the button increases in the touch panel, thedistance to size ratio will keep get greater as the number of buttons in rows and columns
increases. The possibility of the user cannot reach the target button will be greater if there
is more button.

4.3 Circular Design
As long as the matrix design is not fit for the virtual touch panel user interface. We
considered and designed a new design that can fit for the virtual touch panel user interface.
The main concept of the design is to avoid pass through other button when move34

Graduate School of System Design and Management, Keio University
from the start position to the target button, reduce the importance of the distance andreduce the number of the button surround the target button. Finally, we make up a designthat make all the buttons in a circular ring around the start position. A sample of circular
design is shown in Figure 21.

Figure 21: A Circular Design of Button
Because the button is next to each other, if we just use a cursor to indicate the current
position, the user may get confused when the cursor is just on the border of button.
Therefore we highlighted the current button that the cursor is on. A working sample ofcircular design is shown in Figure 22.

35

Graduate School of System Design and Management, Keio University

Figure 22: A Working Example of Circular Design
The characteristic of circular design is that the buttons is on a ring which center isthe start position. The button shares and separate the full 360° from the start position.
When the user move his/her hand to the specific direction, the cursor will be on the specificbutton on the same direction the user’s hand moves. In the case of the situation, the rangeof the direction for a button is an angle of 30°. If we treat the positive direction of y axis
as 0°, ±15° will be button “1”, 15° ~ 45° will be button “2” etc.
In a circular design, the border of range of the panel is to the outside border of eachbutton. The cursor cannot move outside the rings. Therefore, no matter how far the user
moves his hand away from the start point, the cursor will be on the buttons.
36Graduate School of System Design and Management, Keio University

4.3.1 Advantages
Compares to the traditional matrix design, the circular design have several
advantages in a virtual touch panel interface.
4.3.1.1 Advantage 1: Only direction is needed to select a specific button.
As we previous discussed, the user operate a system indirectly with a cursor, the user
cannot handle the distance of the cursor’s movement at the very first time because the user
doesn’t know or use to the speed of the cursor. But the direction of the cursor’s movement
can be handled in the very first time because the cursor will move to the same direction of
the hand’s movement and user can control the direction of his/her hand movement.
In circular design, the buttons is allocated in a ring which center is the start position.
When the user move his/her hand to the specific direction, the cursor will be on the specificbutton on the same direction the user’s hand moves.
Therefore, the user can operate a virtual touch panel interface with the circular design
without needing any previous experience to the system. The user can operate the systemwith circular design more easily than a system with the matrix design in the very first time
because there is no need for the user to handle how much distance the user needs to movehis/her hand.
4.3.1.2 Advantage 2: The size of the button can be treated as unlimited.
In the circular design, the border of the panel is the outside border of each button.
The cursor cannot move outside of the button. Therefore, no matter how much the user
move his hand far from the start position, as long as the direction to the start position is in
the range of the target button, the cursor is on the button. Therefore, the size of the button
can be treated as unlimited within the sensing range of the camera or the reachable zone of37

Graduate School of System Design and Management, Keio University
the user’ hand. Figure 23 shows the size of each button in the sensing range of the camera.
The whole image is the infrared image get by the Kinect sensor, where the pink is the user
detected and recognized by the system. The place of the empty button in the center is theuser’s left hand position detected by the system. The big red circle is the approximately
reachable range of the user’s hand. The red block separated by the blue line is the size ofeach button.

Figure 23: The Hand Cover Range and the Size of the Button
The user can move the cursor to the button more easily than the traditional matrix
design.
4.3.1.3 Advantage 3: Won’t pass through other buttons when moving to the target
button.
As we previous discussed, in this system, both the cursor movement and button
activation requires the user to move his/her hand. When the user want to move a cursor,
38

Graduate School of System Design and Management, Keio University
the ideal situation of the operation is to change the X and Y value and not change the Zvalue. But such operation is impossible to be done as when the user move his/her hand, thedirection is always not absolute parallel to the axis. The hand movement done by the user
often is a curve and had a big change of Z value. This may cause a problem that there is achance the system will misjudge the cursor moving operation to a button activation
operation. Therefore if there are buttons between the current position to the target button.
The system may activate the wrong button by miss.
In a circular design, the button is specified in each direction, there won’t be any
button to be passed by when the user moves the cursor from the start position to the target
button as long as the user doesn’t move the cursor to the wrong direction. If we compare
the system with the circular design to the traditional matrix design, we can see that also
the machine error still affect and cause the system to activate the wrong button, thepossibility of activating wrong button when the user is doing cursor moving operation issignificantly reduced.
4.3.1.4 Advantage 4: The button is only surround by 2 buttons.
As we previous discussed, in this system, both the cursor movement and button
activation requires the user to move his/her hand. When the user want to push his/her handforward and activate the target button, the direction is always not absolute parallel to theaxis and the movement is a curve. The hand movement done by the user often had a bigchange of X and Y value. This may cause a problem that after the hand movement, thecursor is outside the target button. If there are buttons surround the target button, there is achance that the cursor is end up in another button and the system recognize the wrong
button. The possibility of this problem happening is positive related to the number of the39

Graduate School of System Design and Management, Keio University
buttons surround to the target button.
In the circular design, the number of the neighborhood button is 2, which means thenumber of neighborhood button is reduced as

1
4of the number in matrix design. The

possibility of problem that the system activate the wrong button when user is doing the
button activation operation is significantly reduced.
4.3.1.5 Advantage 5: The distance between each buttons is same and can be tweaked。
In the circular design, different to the traditional matrix design, the distance between
the start button and each button are the same. The average operation time can be controlled.
Although the distance of pixels between the buttons in the display is a fixed valuethat depends on the resolution of the display. The distance that the user need to movehis/her hand can be tweaked by tweaking the cursor speed of the system.
So after the appropriate adjustment, there won’t be button that will be out of reachof the user if the user can reach the distance of the radius of the ring, the user can reach all
the buttons. Also the distance won’t be too small that the cursor is too sensitive that makethe system difficult to operate.

4.3.2 The Best Length of RadiusIn the circular design, the radius of the menu in the display is a fixed value that thepanel in the display will be a fixed size. The real radius between the buttons that the userneed to move his/her hand can be tweaked by tweaking the cursor speed. If the radius is
set too big, the user will use more time to reach it. If the radius is set too small, the cursor
will become too sensitive and the user will be difficult to operate it.
In order to know the best length of the radius of the system, we make an experiment40

Graduate School of System Design and Management, Keio University
to compare between different radiuses.
In this experiment, we set the Kinect and a display on a table, the height of Kinect(camera’s center) to the ground is 80cm. The height of the display (display’s center) to theground is 96cm. We let the user sit in a chair, 1.8m away from the center of the Kinect.

Figure 24: The Environment of Test on Different RadiusIn the experiment, the user need to operate the system which radius of the menu is
set in different value. The user need to input the button 0-9 in a randomized order generated
by the system. After the user sit in the chair in front of the camera and put up his/her hand,
Kinect will recognize the user and feedback the position of the user’s hand to the system.
The system will show a cursor in the start position and show the number of the target
button which the user need to activate and the timer is start at the same time. After user
pushed a button, no matter the button is right or wrong, the timer is stopped. Then thesystem reset, and recognize the user, then shows a different target button. The process willlooped until all button has been test. The operation time will be recorded as data.
We tested the radius from 10cm to 25cm, at 5 cm steps. 8 tester participate theexperiment, the result of average operation time of 1 button by different tester and different
radius is shown in Table 4.
41

Graduate School of System Design and Management, Keio University
Table 4: The Average Operation Time of Different Radius
Radius 10cm 15cm 20cm 25cm
Tester 1
2.74 2.99 3.36 3.17
Tester 2
3.49 2.54 3.18 2.72
Tester 3
2.48 2.55 2.78 3.13
Tester 4
3.02 2.73 1.70 2.63
Tester 5
1.95 1.86 2.57 3.23
Tester 6
2.38 2.16 2.48 3.81
Tester 7
2.14 2.29 2.45 3.63
Tester 8
2.06 2.25 2.25 3.27
Average 2.53 2.42 2.60 3.20
(Unit: Second)
We analyzed the data by ANOVA to see if the value of radius is a factor that can takeeffect on the average operation time of the system.
Table 5: The Result of ANOVA on Operation Time by Different Radius
Source of Variation
SS
df
MS
F
P-value
F critBetween Groups
2.89577 3 0.965257 4.625222 0.009475 2.946685
Within Groups
5.843434 28 0.208694
Total

8.739204 31

The P-value is 0.00947, which means the possibility of the value of the radius isn’ta factor of the average operation time is 0.0947%. From the result, we can know that thevalue of the radius is a factor of the average operation time, the value of the radius doestake effect on the average operation time of the virtual touch panel interface.
Figure 25 is the chart we plot the average and the standard deviation of the result
into a line chart.

42

Graduate School of System Design and Management, Keio University

Average Operation Time5
4.5

Time (s)

4
3.20

3.5
3

2.53

2.42

10cm

15cm

2.60

2.5
2
1.5
1
20cm

25cm

RadiusFigure 25: The Result of the Test on Different Radius
From the result, we can see that in the range of 15cm to 25cm, the less the radius,
the less the average operation time and fluctuation. But both the average operation time
and the standard deviation of 10cm is greater than the ones in 15cm. The reason can beconsider as the Kinect has machine error. When the radius is getting too small, the effectof the machine error will affect the system and cause the user harder to operate.
According to the result of the result, we can make a conclusion that the best radius
for the circular design in virtual touch panel interface is 15cm.

4.4 Compare between Matrix Design and Circular Design
We discussed the disadvantages and advantages of the traditional matrix design andthe circular design. The circular design is more proper to the virtual touch panel than thematrix design in our theory. We need to know how both the design’s real performance inthe system to know if the circular design is really better.
We make another experiment to compare the 2 designs. The environment is the same43

Graduate School of System Design and Management, Keio University
as the experiment we do in Section 3.3.2 in order to know the best radius of the circular
design. We set the Kinect and a display on a table, the height of Kinect (camera’s center)to the ground is 80cm. The height of the display (display’s center) to the ground is 96cm.
We let the user sit in a chair, 1.8m away from the center of the Kinect. (See Figure 24)

Figure 26: A Tester is Operating the System
The user need to operate the system in different design. The operation of the user isthe same as the experiment in Section 3.2.2. The user need to input the button 0-9 in arandomized order generated by the system. After the user sit in the chair in front of thecamera and put up his/her hand, Kinect will recognize the user and feedback the positionof the user’s hand to the system. The system will show a cursor in the start position and44

Graduate School of System Design and Management, Keio University
show the number of the target button which the user need to activate and the timer is startat the same time. After user pushed a button, no matter the button is right or wrong, thetimer is stopped. Then the system reset, and recognize the user, then shows a different
target button. The process will looped until all button has been test. The operation time
will be recorded as data.
Additionally in this experiment, we want to know if the circular design has moreaccuracy on the operation. We mark the operation with tags. If the operation ends up withactivated the correct target button in less than 5 seconds, the operation will be marked“Correct”. If the operation ends up with activated the wrong button, the data will be marked“Incorrect”. If no button is activated in 5 seconds, the operation will be marked “Fail”.
Both of the design is set in a 30cm x 30cm square from the start position. In thecircular design, we set the radius of the menu as 15cm. In the matrix design, we allocate
the button equidistantly as the space between each button is 1/5 of the button’s width.

4.4.1 Average Operation Time
5 Tester participated the experiment, the average operation time of 1 button bydifferent tester and different design is shown in Table 6.
Table 6: The Average Operation Time of Different Design

1
2
3
4
5

MatrixDesign
3.07
3.16
2.74
2.84
2.52

Circular
Design
2.45
2.78
2.48
2.36
1.86

Average

2.86

2.39

Tester(Unit: Second)

45

Graduate School of System Design and Management, Keio University
We analyzed the data by student-t test to see the average operation time of matrix
design and circular design are different in the system.
Table 7: The Result of Student-t Test on Operation Time by Different Design
Matrix Design Circular DesignMean
2.86435
2.385032198
Variance
0.067443
0.11181169
Observations
5
5
Pearson Correlation
0.87444
Hypothesized Mean Difference0
df
4
t Stat6.476706
P(T<=t) one-tail0.001464t Critical one-tail2.131847
P(T<=t) two-tail0.002929
t Critical two-tail2.776445

The P-value is 0.00293, which means the possibility of the average operation time
are same in different design is 0.0293%, which can be treat as impossible. From the result,
we can know that average operation time on matrix design and circular design are different
in virtual touch panel interface.

46

Graduate School of System Design and Management, Keio University

Average Operation Time
4.5
4
3.5

Time (s)

3
2.5
2
1.5

2.86
2.39

1
0.5
0
MatrixCircularFigure 27: The Average Operation Time on Different Design
Figure 27 is the result of average operation time plot in a bar chart. From the chart
we can see that average operation time of the circular design use 16.4% less time than the
traditional matrix design.

4.4.2 Accuracy
On each design, the user need to activate 10 buttons, the count of operation ended up
with different result by different tester on matrix design is shown in Table 8.
Table 8: The Result of Operation on Matrix Design
Tester1
2
3
4
5

Correct
877710Incorrect
1
1
2
2
0

Fail
1
2
1
1
0

Accuracy
80%
70%
70%
70%
100%

Average

7.8

1.2

1.0

78%

The result on the circular design is shown in Table 9.
47

Graduate School of System Design and Management, Keio University
Table 9: The Result of Operation on Circular Design
Tester1
2
3
4
5

Correct
10
10
9
8
10

Incorrect
0
0
0
2
0

Fail
0
0
1
0
0

Accuracy
100%
100%
90%
80%
100%

Average

9.4

0.4

0.2

94%

We analyzed the data with student-t test on the accuracy of matrix design and circular
design to see if the accuracy of the different design are different.
Table 10: The Result of Student-t Test on Accuracy of Different Design
MatrixCircular
DesignDesign
0.78
0.94
0.017
0.008
5
5
0.514496

Mean
Variance
Observations
Pearson Correlation
Hypothesized Mean
Differencedft StatP(T<=t) one-tailt Critical one-tailP(T<=t) two-tailt Critical two-tail04-3.13786
0.01746
2.131847
0.03492
2.776445

From the result, we can see that the P-value is 0.03492, which means the possibilityof the average operation time are same in different design is only 3.492%, which can betreat as impossible. From the result, we can know that accuracy on matrix design andcircular design are different in virtual touch panel interface.

48

Graduate School of System Design and Management, Keio University

5 OPERATION ENGINE DESIGN
From Kinect, we can get a time series of hand position, but we cannot recognize
what the user is doing unless we have a detection engine. The detection engine needs analgorithm to analyze the time series of position, and output the current operation or user
status.
In this system, the detection engine works in detecting whether the current operation
is moving cursor, or selecting the button and activate it.

5.1 Algorithm of Operation Engine
A very simple algorithm is monitoring the Z value. If the Z value reduced such avalue in a short time, we can consider the user is “pushing”.
But as we discussed before, in a virtual touch panel interface, the direction of thehand movement is not directly along the absolutely axis. When the user want to move thecursor horizontally and vertically, the hand movement also may cause a change of the Zvalue.
And in an extreme situation when user is too desire to push the button that he moves
his hand so fast, it may sometimes meet the requirement of “Z value reduced such a value
in a short time”, and lead the system to a misdetection.
We need a constraint to limit the direction of push action. The most proper parameteris the angle between the moving direction and the direction of Z axis. When a person tried
to move his hand forward, the hand actually doesn’t go absolutely the direction of Z axis,
there is an angle between moving direction and the Z axis. In this system, we consider 45°

49

Graduate School of System Design and Management, Keio University
is the borderline between both actions. When the angle is less than 45°, we consider theuser is “pushing forward”, otherwise we consider the user is “moving between buttons”.

Figure 28: The Movement Direction and the Operation
Also, we need a constraint to limit the speed of push action. If the user is moving
his hand but the speed is too slow, we should consider this operation as an unconsciously
hands movement other than a push operation. In this system, we use 0.3m/s as the boundary.
If the user move his/her hand over 0.3m/s and the angle between the direction of the hand
movement and the Z axis is less than 45 degree, the operation will be considered as a push
operation.

5.2 Noise Cancellation
The Kinect is a tracking device. As all the tracking devices do, the position feedback
by the devices is not 100% accurate. The inaccuracy include: misdetection of hand byother body part like arms, lose detection of the user etc. which can be treated as noise in
the system.
In order to remove the noise, firstly, we need to remove the inaccurate data which isthe position of the other part of the user due to misdetection.
Normally, the time of the misdetection is short and the inaccurate data is an isolated
50

Graduate School of System Design and Management, Keio University
point in the time series. There is several mathematical method to judge it. But in order tomake it as simple as we can and know it in real time without affecting the performance ofthe system, we use the distance between current position and the average of previous frameand next frame to judge if the frame is inaccurate. If the distance is too big that no one canmove that by hand in 0.1s, which is the time of 3 frames, we consider this point as aninaccurate point, and replace the data of the frame with the average value of previous frameand next frame.
The second part of reduce difference between the real hand movement direction andthe direction detected by Kinect sensor because of the machine error.
Because of the machine error, the track of the hand movement we get from Kinect
sensor is not a smooth line. Normally, it is a polylines. Figure 29 shows an example of thereal hand movement and the data got by Kinect sensor.

Figure 29: Real Hand Movement and Hand Position Got by Kinect Sensor
In Figure 29, the dash line is the track of the hand movement by the user. P0~P5 isthe hand position got by Kinect sensor per every frame. We connect the hand position witharrows. From Figure 29, we can see that the direction of hand movement got by Kinect
sensor is an unsmooth polylines and the direction of the movement in each frame has a51

Graduate School of System Design and Management, Keio University
significant difference to the real direction of the user’s hand movement. Therefore, weneed a noise cancellation method to reduce the error.
We have a lot of way to reduce the noise and remove data if the data is inaccurate,
such as judging an isolated point in the time series. But as our system is running real timeand all the data need to be processed with no delay of the user’s hand movement, we prefera simple method that don’t effect the performance of the system.
From Figure 29, we can see that, although the point of the hand position we got fromthe Kinect sensor is not that accurate, but it is still with an error range of the real position.
The error range is in a limitation according to the specification of the Kinect sensor. If wethe start point and end point of an direction have a big distance between each other, theaffection of the machine error will be reduced, the angle between the real direction and the
direction we get from the sensor will get smaller.
If we draw an arrow from P0 to P5 in Figure 29 (See the blue arrow in Figure 30), we
can see that the direction is much closer to the real direction, the noise is reduced.

Figure 30: Noise Reduction
In this system, we calculate the direction between the current position and the one52Graduate School of System Design and Management, Keio University
we get 5 frames ago to reduce the noise. This method doesn’t need any iteration andcomplex calculation and doesn’t have delay to reduce the noise.

53

Graduate School of System Design and Management, Keio University

6 LOGGING FUNCTION
In this system, there still are spaces for the accuracy and operation time to be
improved. In order to make the further optimization possible, we want to log some datathat we can use it for future improvement. We need to log a large amount of data for all thetime the system is working.
Microsoft, the producer of Kinect, provides a product called Kinect Studio which
can record all the data got by Kinect. The data can be used for playback in the Kinect
program. The product is fully functioned, but there is a very big problem that it will record
all the data get from the Kinect in order to playback with fully function of Kinect. So thedata is too large. It will take approximately 30MB for just 1 second, 108GB for an hour,which is impossible to be used by taking the log for a long time. So obviously, the Kinect
Studio is out of our option.
In this system, we only need the video or image data taken from camera, along withthe detection data of user’s hand. Therefore, it is able to have a log function to taking all
the data we need and do not use a lot of spaces.
We made the logging function ourselves. For the data from camera, we log 1 jpeg
image for each 5 frames, which means 6 files per second. This will take approximately
200kB for 1 second, 720MB for an hour.
For the hand detection data, we only log the 3 value of X, Y and Z coordinates. It
will only take 7MB for an hour.
Our logging function only takes less than 1G for an hour, which is less than 1% of
the data recorded by Kinect Studio. Therefore we can use this to record the data of the54

Graduate School of System Design and Management, Keio University
system when it is being used and make further optimization to the system by these data.

55

Graduate School of System Design and Management, Keio University

7 VERIFICATION AND VALIDATION
7.1 Verification Test
In order to know if our system is useable and the accuracy of the detection algorithm,
and which height of the camera is best. We conducted experiments to test the system on
different heights of the system.
Normally, when a person put out his hand, the height is around the height of ahuman’s chest. When sit on a chair of 40cm, the height of chest is around 80cm to 90cm.
According to a building regulation of accessible toilets, the height of washing basin forunisex wheelchair use is 72cm-74cm [12]. In this experiment, we are going to test thesystem with the height of the Kinect sensor is 75cm, 80cm, 85cm and 90cm. A 15” display
is set as the height of the display is 96cm to the ground. The user sits in a chair with adistance of 1.8m between front wheels to the center of Kinect.

Figure 31: The environment of the Verification Test

56

Graduate School of System Design and Management, Keio University
In the test, the user need to operate the system which radius of the menu is set indifferent value. The user need to input the button 0-9 in a randomized order generated by
the system. After the user sit in the chair in front of the camera and put up his/her hand,
Kinect will recognize the user and feedback the position of the user’s hand to the system.
The system will show a cursor in the start position and show the number of the target
button which the user need to activate and the timer is start at the same time. After user
pushed a button, no matter the button is right or wrong, the timer is stopped. Then thesystem reset, and recognize the user, then shows a different target button. The process willlooped until all button has been test. The operation time will be recorded as data.
11 tester participate this test and the average accuracy by different heights is shownin Figure 32.

Accuracy
100

95

95

90

Accuracy (%)

90

86

85
80
75

74

70
65
60
75

80

85

90

Height (cm)Figure 32: Accuracy by Different Heights
Figure 33 shows the average and standard deviation of operation time on each button
by different heights.
57

Graduate School of System Design and Management, Keio University

Average Operation Time
5

Time (s)

4
3

2.60
2.26

2.24
1.74

2
1
0
75

80

85

90

Height (cm)Figure 33: Average Operation Time by Different Heights
From the result, we can see that 75cm is too low that the accuracy for the Kinect
sensor is not very good. The accuracy for the other height is good enough to be used as an
input method. The average operation time is around 2 seconds which is acceptable as an
input method by the user. This performance is shown that this system reaches the standards
that can be used.

7.2 Validation
7.2.1 Interview with Ms. Qiu
We interviewed Ms. Qiu again to ask if the system can help handicapped people as
an input method. We described our system to Ms. Qiu and show her our verification result.
The detail of interview is in Table 11.

58

Graduate School of System Design and Management, Keio University
Table 11: The Second Interview of Ms. Qiu
Q The last time we interviewed you, you mentioned about if there is an input methodthat allows the user to operate button without touching it, it should help thehandicapped people, right?
A Yes, that’s true.
Q We have made some research on such method, now we designed a system. The
system is called Virtual Touch Panel. The detail of the system is ……. (Minutes ofdescription of the system). How do you think of this system?
A This system looks cool. It should help the handicapped people I think if the accuracy
and operation time is acceptable. Have you done some test on how the performanceof the system?
Q Yes, after the verification test, the accuracy can be up to 95%, the average operation
time is about 2 seconds.
A That’s cool, I think it should help the handicapped people. Thank you for the
research. We actually hope more student like you can make research on the helping
the handicapped people. Thank you again.

From the interview to Ms. Qiu, we can know that the system meet the requirementof the handicapped people.

7.2.2 Evaluation by Handicapped User
In order to know if the handicapped user will find it easier and more comfortable touse this system than the ordinary input method, we contacted Mr. Hokari. Mr. Hokari had
a traffic accident decades ago and cannot move any part below his chest anymore since59Graduate School of System Design and Management, Keio University
then.
Mr. Hokari is a wheelchair user and he find it not very easy to use the ordinary touch
panel as he need to use minutes to try to reach the panel then use minutes to adjust thewheelchair’s location to move on. We introduced him about our system’s concept and let
him tried a prototype of our system.

Figure 34: Evaluation by Mr. Hokari
After trying this system, Mr. Hokari thought this system should work well as no
additional thing is needed in this system and the accuracy and average operation time isacceptable. He thought this system should help handicapped people as a novel input
method.

60

Graduate School of System Design and Management, Keio University

8 DISCUSSION
Still, there is some interesting thing we need to discuss about the system.

8.1 Compare between Buttons
In the verification test, we let the user operate the button in a randomized order tocalculate the time and the accuracy of the system in different height.
The button is in the same distance to the start position with different directions, thedifficulty of operation each button should be the same theoretically. But in the real world,
because the system is operated by human’s hand and the human’s body is connected byjoints, the difficulty of move the hand in different direction may be different.
In order to know if the direction of the button take effect on the average operation
time and accuracy of the system, we grouped the test result of verification test by different
buttons. The result of the average operation time of different buttons by different heights
is shown in Table 12.
Table 12: The Average Operation Time of Different Buttons`
75cm 80cm 85cm 90cm Average
2.51
2.50
1.41
Button 1 2.24
2.16
2.56
2.32
2.08
Button 2 2.91
2.47
2.64
2.18
1.56
Button 3 2.43
2.20
2.31
1.92
1.82
Button 4 1.86
1.98
2.18
2.07
1.77
Button 5 3.17
2.30
2.27
2.55
2.15
Button 6 3.00
2.49
1.92
1.96
1.52
Button 7 3.09
2.12
2.29
2.58
2.08
Button 8 2.95
2.47
2.19
2.23
2.14
Button 9 2.46
2.25
1.77
2.08
1.98
Button 0 1.86
1.92

(Unit: Second)

61

Graduate School of System Design and Management, Keio University
We plot the data into a line chart, with the standard deviation as the error range. The
figure is shown in Figure 35.

Time (s)Average Operation Time
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0

2.27

2.45

2.35

2.28
1.93

2.56

2.55
2.20

2.24
1.86

Figure 35: The Average Operation Time of Different Buttons
From Figure 35, we can see that there is clearly some difference between eachbuttons. But in order to know if the average operation time between each buttons aredifferent in statistical. We analyzed the data with ANOVA, the result of the ANOVA isshown in Table 13.
Table 13: The Result of ANOVA on Operation Time by Different Buttons
Source of Variation
SS
df
MS
F
P-value
F critBetween Groups
1.46261 9 0.162512 0.894234 0.542014 2.210697
Within Groups
5.452005 30 0.181733
Total

6.914615 39

The P-value is 0.54, which means the possibility of the average operation time aresame between different buttons is 54%. We can say that the average operation time is thesame. The direction of the button doesn’t take effect on the average operation time of the62Graduate School of System Design and Management, Keio University
system.
The result of the accuracy of different buttons by different heights is shown in Table
14.
Table 14: The Accuracy of Different Buttons`
75cm 80cm 85cm 90cm Average
Button 1 73% 100% 100% 86%
90%
Button 2 55% 82% 100% 86%
81%
Button 3 55% 91% 100% 71%
79%
Button 4 91% 91% 100% 86%
92%
Button 5 82% 73% 91% 86%
83%
Button 6 73% 73% 91% 71%
77%
Button 7 82% 82% 91% 100%89%
Button 8 55% 91% 100% 86%
83%
Button 9 91% 91% 91% 71%
86%
Button 0 82% 91% 91% 86%
87%

We plot the data into a line chart, with the standard deviation as the error range. The
figure is shown in Figure 36.

Accuracy (%)

Accuracy
100
95
90
85
80
75
70
65
60
55
50

92

90
81

89
83

79

83
77

Figure 36: Accuracy between buttons

63

86

87

Graduate School of System Design and Management, Keio University
From Figure 36, we can clearly see that there is difference between the different
buttons. But in order to know if the accuracy between each buttons are different instatistical. We analyzed the data with ANOVA, the result of the ANOVA is shown in Table
15.
Table 15: The Result of ANOVA on Accuracy by Different Buttons
Source of Variation
SS
df
MS
F
P-value
F critBetween Groups
0.08587 9 0.009541 0.559125 0.818986 2.210697
Within Groups
0.511933 30 0.017064
Total

0.597803 39

The P-value is 0.82, which means the possibility of the accuracy are same between
different buttons is 82%. We can say that the average operation time is the same. The
direction of the button doesn’t take effect on the average operation time of the system.
From the result, we can know that although the accuracy and time seems to havesome difference between each buttons, there is no significant difference in statistical. The
direction of the button doesn’t take effect on the performance of the system.

8.2 Compare with Physical Touch Panel
8.2.1 Fitts’ Law
Fitts’ law is a model of human movement primarily used in human-computerinteraction and ergonomics that predicts that the time required to rapidly move to a target
area is a function of the distance to the target and the size of the target. [13] [14]Fitts’ law has been formulated mathematically in several different ways. Here we use
the common form, the Shannon formulation published by Dr. MacKenzie. [15] [16]64Graduate School of System Design and Management, Keio University
𝑇 = 𝑎 + 𝑏 ∗ 𝐼𝐷

(1)

In this equation, T is the average time taken to complete the operation. ID (Index of
Difficulty) is a measure of the difficulty. a is the initial time for the devices, include start
time and reaction time before the user start to operate. b is the inherent speed of the device,or “slope”. The a and b is constant for each devices and by comparing the 2 constant, we
can compare the performance of different input method.
As we can imagine, the longer the distance the user need to move, the smaller thesize the button is, the larger the difficulty, which need more time to operate. According toFitts and MacKenzie, the equation of difficulty is shown in Equation 2.
𝐼𝐷 = log 2 (1 +

𝐷
)
𝑊

(2)

In Equation 2, D is the distance from the starting point to the center of the target. W
is the width of the target measured along the axis of motion.
We can compare different input method and devices by comparing the a and b value
in Equation 1.

8.2.2 Equation for Virtual Touch Panel Interface
In order to know the value of a and b in Equation 1 of the virtual touch panel interface.
We made an experiment.
In this experiment, we set the Kinect and a display on a table, the height of Kinect(camera’s center) to the ground is 90cm. The height of the display (display’s center) to theground is 96cm. We let the user sit in a chair, 1.8m away from the center of the Kinect.

65Graduate School of System Design and Management, Keio University

Figure 37: A tester is operating the system in the experiment
The task of the User is operate a cursor by moving their hands in front of the Kinect
to activate the button in different size, different direction and different distance to the startpoint in the display. There are 3 kinds of sizes of the button: 10cm, 12.5cm and 15cm.
There are 8 kinds of direction: up, down, left, right and 4 diagonally. There are 4 kinds 4
kinds of distance (from the button’s center to the start point):15cm, 20cm, 25cm and 30cm.
Each of the data above refer to the data in the real space, which is the distance that thehand need to move. The cursor speed in the display is 20 dpi. The tester need to move theirhand toward the camera, which can be also treated as “pushing forward”, to activate thebutton.
After the tester sit in front of the system and put up their hands, the Kinect sensor
will recognize him and feedback his hands position to the system. The system will markthe current hand position as start position and then shows a cursor (a red dot) to indicate

66Graduate School of System Design and Management, Keio University
the current hand position, which is in the center at first. The user will then move his hand
until the cursor is over the button, and then push forward his hand to activate it. The user
have to move his hand back to the start position and then the next button will be shown inthe display. The tester shall redo the process above to activate the button until all the button
has been activated.

Figure 38: Test for Fitts’ Law
The time between cursor in the start position and button has been activated will berecorded as operation time. The first button’s operation time is started at the time when theuser’s hand is being recognized and the cursor is shown, the remaining buttons’ operation
67

Graduate School of System Design and Management, Keio University
time is started when the cursor moved back to start position and the next button shows.
8 testers participate the experiment, the average operation time of different size andtime is shown in the table below.
Table 16: Result of Test for Fitts’ Law
Distance (m)
0.15
0.15
0.15
0.2
0.2
0.2
0.25
0.25
0.25
0.3
0.3
0.3

Size (m)
0.1
0.125
0.15
0.1
0.125
0.15
0.1
0.125
0.15
0.1
0.125
0.15

Time (s)
2.18
1.77
1.55
2.11
1.98
2.09
2.74
2.34
2.03
2.66
2.45
2.61

If we adapt this data in Equation. The distance between start position and the centerof the button will be the D in Equation 2, the size of the button will be the W in the
Equation 2. By these data, we can get the ID of in Equation 1 and the time will be the T in
Equation 1.
We set the ID as X axis, the T as Y axis and plot the result data in a scatter chart. The
chart is shown in Figure 39.

68

Graduate School of System Design and Management, Keio University

Fitts' Law on Virtual Touch Panel3
y = 1.1223x + 0.5436
2.5

Axis Title

2
1.5
1
0.5
0
0

0.5

1

1.5
ID

2

2.5

3

Figure 39: Result of Test for Fitts’ Law
From the chart, we can see that the data follow a linear relationship. According to
the data, we can know that the operation time in this system is following the Fitts’ Law。
If we adapt the data in Equation 1. We can get the equation of the expected operation timeof the system.
𝑇 = 0.54 + 1.12 ∗ I𝐷

(3)

8.2.3 Equation for Physical Touch Panel Interface
Parhi et al. researched on the operation time of several design on a physical touch
panel devices with a 3.5” touch display [17].
The result of the operation time on different difficulty is in Figure 40.

69Graduate School of System Design and Management, Keio University

Fitts' Law on Physical Touch Device
3
2.5

T (s)

2
1.5
y = 0.4815x - 0.1663
1
0.5
0
0

0.5

1

1.5

-0.5

2

2.5

3

ID

Figure 40: Fitts’ Law on Physical Touch Device
The equation of the Fitts’ Law is on Equation 4.
𝑇 = −0.17 + 0.48 ∗ 𝐼𝐷

(4)

8.2.4 Comparison
From Equation 3 and Equation 4. We can make the comparison between the physicaltouch panel and the virtual touch panel.
From the equations, we can know that the initial time of the virtual touch panel is
0.7s late than the physical touch panel. According to the user experience in the experiment,
we find that the delay of the time has 2 reasons. First, user need time to find and confirm
the cursor in virtual touch panel while the time isn’t needed in the physical touch panel.
Second, user need some time to take the action of pushing forward to activate the button
in virtual touch panel while the action isn’t need in the physical touch panel.
From the equations, we can know that the slope of the virtual touch panel is 2.3 times
more than the physical touch panel. The reason can be considered as the all the user

70

Graduate School of System Design and Management, Keio University
operating our system is the first time user without any experience, the user tend to slow
down his/her operation to make sure the operation he/she done is correct.

71

Graduate School of System Design and Management, Keio University

9 CONCLUSION
9.1 Summary
In this research, we presented a novel input method that allows the user to operate asystem with their hand gesture instead of only with their fingers. We discussed the detaildesign of GUI, operation interface and operation engine for a virtual touch panel interface.
From the result of the verification test, we showed this system is practical andfeasible. We also provide a way to take the log of system and use it for further optimization.
We get the positive feedback from the representative of official organization for thehandicapped people that the system can help the handicapped people with operating themachines.

9.2 Future WorkAs we discussed previously in Section 8.2, we compare our system with the physicaltouch panel and find out the performance of the virtual touch panel is a little bit worse thanthe physical touch panel. The further research on reducing the gap between the virtual
touch panel and the physical touch panel is necessary.
Also, the research on adjusting and implementing this system in the real world is stillneed to be done in the future.

72

Graduate School of System Design and Management, Keio University

PUBLISHED
H. Lu, T. Ogi, Y. Tateyama and H. Lee, "Virtual Touch Panel for Handicapped
People," in ASIAGRAPH Forum 2012, Tokyo, 2012.
H. Lu, T. Ogi, Y. Tateyama and H. Lee, "Interface Design of Virtual Touch Panel for
Handicapped People," in 2012 Asian Conference on Design and Digital Engineering(ACDDE 2012), Niseko, 2012.

73

Graduate School of System Design and Management, Keio University

ACKNOWLEGMENT
Firstly, I want to thank my supervisor Prof. Ogi, without his guidance and assistance,
this research won’t be able to be finished. Every week, Prof. Ogi opens a seminar andinvite every student from the laboratory. In this seminar, we share the progress and the
difficult we faced in the last week and get the advices from Prof. Ogi and other lab mates.
This help me a lot when I face problems and when my method and direction was wrong.
Secondly, I want to thank Prof. Tateyama. Prof. Tateyama is a specialist in computer
programming, visual simulation and human interface. He gave me a lot of useful advices
and introduced me a lot of new method and theory in this area.
I also want to thank Prof. Lee. Prof. Lee is an assistant professor from Keio
University Global COE Project. He help me to get the Global COE Project support the
research and give me a lot of advices and encouraged me when I could barely makeprogress.
Prof. Nakano and Prof. Kohtake also gave me a lot of advices and help me getthrough a lot of difficulties.
I want to thank Ms. Qiu, Mr. Hokari and other people that gave me important idea
and advices in this research, without them, the system won’t get this much progress.
I can’t forget the people who take their time to participate the test and experiment in
my research. My research and this system made important progress with their data andfeedback.
I also want to thank Keio University Global COE Project (Center for Education andResearch of Symbiotic, Safe and Secure System Design, Keio University) to support this74

Graduate School of System Design and Management, Keio University
research.

75

Graduate School of System Design and Management, Keio University

REFERENCES[1]

T. Brown and B. Shetty, "A Turn of the Page for Newsweek," The
Newsweek/Daily Beast Company LLC, 18 October 2012. [Online]. Available:
http://www.thedailybeast.com/articles/2012/10/18/a-turn-of-the-page-fornewsweek.html. [Accessed 21 January 2013].

[2]

International Telecommunication Union, "Definitions of World
Telecommunication/ICT Indicators," 2010.

[3]

総務省, “平成 24 年版 情報通信白書,” 2012.

[4]

A. W. Hirshfeld, The Electric Life of Michael Faraday, Walker and Company,
2006.

[5]

経済産業省資源エネルギー庁, “エネルギー白書 2012,” 2012.

[6]

内閣府, “平成 24 年版 高齢社会白書,” 2012.

[7]総務省統計局, “人口推計－平成 25 年 1 月報 －,” 2013.

[8]

内閣府, “平成 24 年版 障害者白書,” 2012.

[9]三菱電機株式会社, “タッチレスコール技術を開発 操作ボタンに触れずに利用できるエレベーター,” 3 March 2011. [オンライン]. Available:
http://www.mitsubishielectric.co.jp/news/2011/pdf/0303-b.pdf. [アクセス日: 4
February 2013].

[10] 向斌宾, 蒋向东, 王继岷 和 李建国 , “非接触式触摸屏系统的设计,”76Graduate School of System Design and Management, Keio University

计算机工程与应用, 卷 32, pp. 71-74, 2011.
[11] MSDN, "Kinect for Windows SDK Documents," Microsoft, [Online]. Available:
http://msdn.microsoft.com/en-us/library/hh855347.aspx. [Accessed 12 Janurary
2013].
[12] LABC, "Building Regulations in Practice - Accessible Toilets," [Online].
Available:
http://www.charnwood.gov.uk/files/documents/accessible_toilet_diagram_and_gui
dance/. [Accessed 27 January 2013].
[13] P. M. Fitts, "The information capacity of the human motor system in controllingthe amplitude of movement," Journal of Experimental Psychology, vol. 47, no. 6,
pp. 381-391, June 1954.
[14] P. M. Fitts and J. R. Peterson, "Information capacity of discrete motor responses.
Journal of Experimental Psychology," Journal of Experimental Psychology, vol.
67, no. 2, pp. 103-112, Feburary 1964.
[15] I. S. MacKenzie and T. Kauppinen, "Fitts' law as a research and design tool inhuman–computer interaction," in ACM CHI 2001 Conference on Human Factors
in Computing Systems, New York, 2001.
[16] I. S. MacKenzie and W. A. S. Buxton, "Extending Fitts' law to two-dimensionaltasks," in ACM CHI 1992 Conference on Human Factors in Computing Systems,
New York, 1992.
[17] P. Parhi, A. K. Karlson and B. B. Bederson, "Target Size Study for One-Handed
Thumb Use on Small Touchscreen Devices," in ACM MobileHCI 2006 8th77Graduate School of System Design and Management, Keio University
Conference on Human-Computer Interaction with Mobile Devices and Services,
New York, 2006.
[18] S. Totilo, "Natal Recognizes 31 Body Parts, Uses Tenth of Xbox 360 “Computing
Resources”.," Gawker Media, 7 Janurary 2010. [Online]. Available:
http://kotaku.com/5442775/natal-recognizes-31-body-parts-uses-tenth-of-xbox360-computing-resources. [Accessed 12 Janurary 2013].
[19] A. Murata, "Extending effective target width in Fitts' law to a two-dimensional
pointing task," International Journal of Human-Computer Interaction, vol. 11, no.
2, pp. 137-152, 1999.
[20] A. Murata and H. Iwase, "Extending effective target width in Fitts' law to a threedimensional pointing task," Human Movement Science, vol. 20, pp. 791-805, 4September 2001.
[21] L.-W. Chan, T.-T. Hu, J.-Y. Lin, Y.-P. Hung and J. Hsu, "On top of tabletop: A
virtual touch panel display," in 3rd IEEE International Workshop on HorizontalInteractive Human-Computer Systems, 2008.

78