SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 3(2); September (2022), pp. 102–112
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

Restaurant Menu with Gesture Recognition
Ian Christian Susanto, Kasthuri Subaramaniam, Abdul Samad bin Shibghatullah
Institute of Computer Science & Digital Innovation, UCSI University, UCSI Heights, 1 Jalan Puncak Menara Gading, Kuala Lumpur, WP Kuala Lumpur
56000, Malaysia

ARTICLE INFO
Article History
Received 22 December 2021
Accepted 06 November 2022
Keywords
Digital restaurant menu
Hand gesture recognition
Hand pose estimation
Human-computer interaction

ABSTRACTThe COVID-19 pandemic has brought back concerns of microorganism contamination into the
public consciousness. Efforts have been made to ensure minimal chances of virus transmission
by using movement control orders, encouraging social distancing, obligation to wear masks, andencouraging washing of hands so that it will reduce the chance of transmission. Indirecttransmissions through surface contact are more difficult to prevent. There are many places inwhich people touch potentially infectious surfaces in the public, for example, ATM or
touchscreen menu ordering at restaurants. Businesses and the general public are still looking forways to minimize surface contact transmission. There have been efforts to minimize the chancesof transmission through surface contact using an antiviral coating but it may be inadequate. The
antiviral coating requires time to be effective. Repeatedly coating surfaces with sanitizer has highoperational costs. These concerns are unlikely to disappear after the pandemic either, other viruses
can be transmitted through this method. There needs to be a permanent solution. One such solution
is using touchless technology such as hand gesture recognition. However, developing a hand
gesture recognition algorithm can be a difficult task. The goal of the hand gesture recognition
library is to facilitate the implementation of future applications. Gesture recognition is an example
of a message window, icons, menus, pointer (WIMP) user interface that has poor functionality inthe real world, especially when used in digital food restaurants. In this article, we will look atcolorful hand gesture detection styles and finish evaluating the hand camouflage of our system.
We describe a contactless digital restaurant menu system and find that the system passes stoner
acceptance tests.
© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

1. Introduction
The evolution of user interface design consists of long
periods with small changes followed by short periods withlarge changes. It can be categorized the history of userinterface design into four periods [1]. In the early 1950s to
1960s, computers had no user interfaces. Instructions had
to be inputted using punch cards. The outputs were printed
through line printers. This gives very limited debugging
capabilities. Few people had the capabilities to perform
debugging with switches and lights. Others weren’t solucky; they had to make sure their programs were errorfree before inserting the punch card. Starting in the early

1960s up to the early 1980s, computers started to havealphanumeric displays. This means users can now interact
by typing commands into a terminal. This type of interface
still survives today through command-line shells in many
operating systems such as Windows, Mac OS X, and Linux.
In the 1970s, user interfaces using windows, icons, menus,
and pointers became popular. This type of interface isreferred to as a WIMP interface. It was made popular in1986 with the release of Macintosh and was adapted byWindows and Motif. WIMP interfaces are still prevalent,even today, due to their ease of learning, ease of use, andease of knowledge transfer. User interface design trends
correspond of long ages with small changes followed by

Corresponding authors’ E-mail: 1001954676@ucsiuniversity.edu.my, kasthurisuba@ucsiuniversity.edu.my, abdulsamad@ucsiuniversity.edu.my
URL: www.ucsiuniversity.edu.my102

short ages with large changes. Van Dam [1] categorizes
these trends into four ages Punch cards, command line
interfaces, WIMP( window, icons, menus, pointer), andpost-WIMP. Post-WIMP interfaces calculate on relations
other than menus, forms, or toolbars to achieve further
intuitive gests . An illustration of a post-WIMP interface ishand gesture recognition. Hand gesture recognition has
many advantages, videlicet sterility as the stoner no way
touches the device, availability to people with physicaldebits, and effectiveness of using three confines [2].
Computer vision and wearable technology are the twomain types of hand gesture recognition. Although theformer is easier to wear, it can stand out and requires a
fresh outfit [3].
Due to the COVID-19 pandemic, people have been
more cautious of their hygiene. While there exists
evidence of a low risk of transmission through surface
contact, other variants of SARS-CoV-2 may have higher
risks of fomite transmission [4]. Therefore, caution would
still be preferred. Shin and Kang [5] found reduced
perceived risks when technological alternatives areprovided to customer-staff interactions. The usage oftouchless technology, such as hand gesture recognition,
will further reduce the risks of transmission, as no surface
contact is initiated.
In this paper, we propose a digital restaurant menu
system using hand gesture recognition. We discuss theanalysis, design, implementation, and evaluation of theapplication. The paper will be structured as follows;
Section 1 discusses the problem statement, research
objectives, and report structure; Section 2 discusses
different methods of performing gesture recognition andseveral digital restaurant menus systems and compare
different systems that use gesture recognition; Section 3
discusses the research methodology of the project, inparticular the software development life cycle used and themethod of gathering requirements. Results of therequirements gathering and the design of the system areelaborated in Section 4. Section 5 gives theimplementation of the application, details on the hand
gesture recognition module, and an evaluation of thefunctionalities of the application through tests. Section 6 isthe conclusion of the experiments.

2.1. Hand Gesture Recognition
There are seven categories of vision based hand gestureaccording to Oudah et al [3]:• Color-based recognition. Gestures are determinedbased on the color of each pixel. Wang and Popović [6]used the nearest-neighbor approach to track a colored
glove. The drawback of this method is the need for an
additional glove [7]..
• Appearance-based
recognition.
Gesturesaredetermined by extracting features from the shape of thehand. One way to do this is through edge detection [8].
• Motion-based recognition. The gesture is determinedby detecting changes between frames [9].
• Skeleton-based recognition. The image is used to
extract the geometric characteristics of the hand,including joint orientation, joint spacing, skeleton joint
position, and joint angle [10].
• Depth-based recognition. One benefit of using depth
information to extract motions is the algorithm nolonger needs to take lighting, shadow, and color into
account [11].
• 3D model-based recognition. The algorithm comparesa 2D input to a 3-dimensional model. Ge et al [12].
produce a more detailed output by constructing a 3Dmodel of the hand using Graph CNN.
• Deep learning-based recognition. Despite the laborintensive process of gathering datasets, deep learning
can be effective for gesture recognition [13].
In this project, gesture recognition is achieved withpose estimation. The advantage of this method is the easeof modifying the list of gestures to detect. Pose estimation
is typically done in two steps: first, the location of the handis determined. The initial image is cropped and resizedaccording to the location. Next, pose estimation is
performed on the image containing only the hand [14]. In asystem proposed by Malik et al [15], the hand joint position,
hand mesh vertices, and structural constraint are obtainedby feeding a depth map into a convolutional neural network.
To improve the precision of a pose estimation system andenable weakly supervised training, Spurr et al. [16] propose
the introduction of biomechanical constraints. A real-time
hand posture estimate solution with RGB input is presented
by Zhang et al [17].

2. Literature Review
1032.2. Digital Restaurant Menus

3. Research Methodology

The most basic form of restaurant menu is paper-based.
However, the consumer must wait to place their food order,
which is a negative. Additionally, it wastes paper because
each consumer needs an additional piece of paper to order[18]. To address these issues, some digital restaurant menu
systems have been developed.
Digital restaurant menus are classified by Şahin [19]into two categories: touchscreen systems do not require
staff interaction, while non-touchscreen menu systems
require staff interaction. An Android-based digitalordering system was created by Bhargave et al [18]. in
which each table has a tablet where customers can viewthe menu, allowing for self-service and a decrease in paper
waste. A mobile application-based digital restaurant menu
is suggested by Rusdi et al [20].

3.1. System Development Life Cycle

2.3. Comparison of Hand Gesture Recognition
Systems
We compare various existing systems that use hand
gesture recognition. Desai and Desai [21] proposed a home
automation system operated with hand gesture recognition.
Microsoft Kinect is used as input for recognition. Suriya
and Vijayachamundeeswari [22] used hand gesturerecognition to control the mouse on a desktop computer.
Skin color detection is applied, then the convex hullalgorithm is utilized to find the fingers. Using Gestix [23],a hand gesture-based interface, doctors can accesselectronic medical record (EMR) databases without
touching a computer. Al et al [24]. created a Universal
Robot (UR3) arm that recognizes hand gestures. The hand
is located using proximity sensors. Zhang et al [25]. uses
hand gesture recognition to control the movement of aplayer in a VR game. The input is gathered using the Leap
Motion controller mounted in front of an Oculus Riftheadset. Out of the system described above, Gestix bears
the most similarities due to only using color and motion
data to control a graphical user interface. The systemdescribed in [21] is analogous in that each gesture is
assigned to a certain action. In contrast to [23] or [25],which interact with a robot arm or a VR player,
respectively, gestures are also utilized to navigate a menu.

The system development life cycle (SDLC) is the process
by which a system is designed, built and delivered to its
users so that they understand how the information system
helps achieve business needs [26]. A typical SDLC can bedivided into four main phases: Planning, where the project
team decides how and why it will be developed; Analysis
that examines currently existing systems, collects
requirements and develops a system concept; Design inwhich details of system operation such as hardware,
software, network infrastructure, user interfaces, databases,
etc. are defined more precisely than in the analysis phase;
The application where the system is built [27]. This
includes system build, installation and support plan. The
prototype model was chosen as the SDLC model for thisproject. This model is based on an iterative process ofgathering requirements, building a model and verifying it.
The process is best suited for systems where the system
details are not yet clearly defined. This can happen inseveral ways, for example, the requirements are not clearly
defined, the efficiency of the algorithm or the systemarchitecture is unknown [28] [29].
3.2. Questionnaire
Requirements gathering is done using a questionnaire.
Respondents consist of 29 UCSI University undergraduate
and master’s students. The questionnaire is shared throughthe UCSI Course Networking page. To understand therequirements deeper than through multiple choice, somelong-form questions were included. Topics of interest
include the respondent’s opinions on digital restaurant
menus, desirable features, concerns about the hygiene ofconventional menus, and opinions on the advantages anddisadvantages of using hand gesture recognition.
4. Analysis and Design
4.1. Analysis
We go over the results of the survey that was developed inPart 3 in this section. 2. The first query concerns the use ofdigital menus in eateries. Using a 5-level Likert scale,
respondents selected their responses. Figure 1. shows theend outcome. 15 people selected "4" (agree), while theoverall average was 4.17. We get to the conclusion that the104

majority of individuals feel that restaurants should use
digital menus.

Fig. 1. Number of responses on digital menus.
The next two questions are regarding features needed
in a digital restaurant menu. The first question is divided
into several sub-questions, where respondents rate features
on a Likert scale, ranging from 1 (very undesirable) to 5
(very desirable). In Table 1, we show the average scores ofeach feature. In the next question, respondents may write
other features they desire that are not yet listed.
Features requested by respondents include dishpricing, special offers, special sets, digital payment
methods, shopping cart systems, waiter notification
systems, and ingredient freshness descriptions.

capacity to maintain the hygienic standards of their
touchscreen surfaces. Those respondents who don't have
any issues with cleanliness are confident in the restaurant's
hygiene policies.
The next question asks respondents if they think gesturerecognition can help with this issue. This is supported by 19
replies, the majority of which point to the absence of
surface contact. Three respondents express no opinion.
Four responses indicated skepticism toward the technology
and the need for additional testing to determine its viability.
The final question concerns possible flaws with thesystem. Fifteen replies are concerned about the recognition
system’s potential for poor accuracy and sluggish
performance. Seven of the replies expressed concern about
the learning curve. Two comments raised the subject ofmaintenance. Two responses address the potential for error.
Five of the respondents express no worries about any
problems.

Table 1. Average score of each feature
Avg.
Score

Feature

Avg.
Score

4.52

Non-allergy warnings

4.59

4.34

Recommended

4.38

3.76

List of ingredients

4.17

Dish variations

4.14

Estimated preparation
time

3.93

Allergywarnings

4.59

FeatureImage of thedishDescription of
the dishNutritional
content

The respondents are then asked about their worries
regarding the cleanliness of touchscreen digital restaurant
menus. Concerns about the hygiene of touchscreen digitalmenus were raised by 17 respondents. The fact thatnumerous individuals use the device and could potentiallyleave bacteria or viruses behind is one reason for the worry.
Some people expressed doubt about the restaurant'sFig. 2. Use case diagram of the system.

4.2. Design
In this section, we look at the system’s design created tofulfill the requirements [30]. The use case diagram asshown in Figure 2 consists of two main parts: a customeroriented department and an employee-oriented department.
Figure 3, the class diagram, shows what types of data
will be needed. The gesture recognition system and the105actual digital menu do not share any data other than tocontrol the application, therefore we can split the diagram
into two. The Server part includes all classes that are usedin the digital menu and the server that stores them, while
the Gesture Recognition part includes all classes needed in
the recognition system.

Fig. 5.A screenshot shows the cart being filled.

Fig. 3. The system class diagram

The activity diagram shows the user’s activity as
shown in Figure 4.

Fig. 4. Users interaction shown in the activity diagram

5. System Implementation and Evaluation
In this section, we look at the system's list of features, how
gestures are recognized, and the tests that are run to makesure the system is functioning properly

camera view as feedback. Below it, the detected gesture isshown. Next, the list of items in the current cart is shown,which includes which items are picked, and how many ofeach item are added. Finally, the current total price is shown.
By making a peace sign, the user can switch between
viewing the recommended dishes and all available dishes.
To prevent accidental swaps, the gesture must be detected
for one second. To scroll the menu, the user can hold a “fist”
gesture and move their hand to the left or right.
Adding an item to the cart requires the user to perform
a “point up” gesture. To add a specific item, the user must
aim the cursor at the image, name, or description of themenu. The user can redo or hold the gesture to add morethan one item. To remove the item, conversely, the user canperform a “point down” gesture.
The “thumb up” gesture acts as a confirmation forvarious purposes. To finish editing the cart, the user canperform a “thumb up” gesture, which opens a confirmation
dialog. The user can do the same gesture again to confirm
their order or perform a “thumb down” gesture to cancel. If
the user confirms the order, a thank you dialog box isdisplayed along with the total price and order number.
Finally, if the previous user did not clear their cart, a newuser may do the “thumbs down” gesture to reset the cart.
The staff can manage orders through web pages. As
shown in Figure 6, the left part of the screen shows thecurrent order list. Each order shows the order ID, the list ofitems being ordered, and the quantities of each item. If theorder is done and ready to be served, the staff can click the“Ready” button on the bottom left of each order.

5.1. Implementation
First, we will discuss the customer-facing application, as
shown in Figure 5. Most of the screen is taken up by themenu. Other information is located on the right side of thescreen. On the top right of the screen, the user is shown the106

acceptable range of the condition. While this approach
worked for our use case, other use cases may require moresophisticated methods, such as support vector machines.
Figure 8 shows how the application interacts with thehand gesture recognition module.

Fig. 6. Screenshot of the order management page.

On the right side, the ready orders can be seen. If theitem is claimed by a customer, the staff can click the“Taken” button. The order is then removed from the list.
In the case where the order is stated as ready by mistake,the staff can click the “Undo” button.
After each action, a confirmation message will appear
at the top of the page. This will say “Operation successful!”
if the action succeeds or “Operation failed!” if the action
cannot be performed for one reason or another.
Fig. 8. Activity diagram of how the application interacts with thehand gesture recognition module.

5.3. Evaluation

Fig. 7. A screen showing the list of ready orders.

Finally, the ready orders could be viewed on anotherpage. This page is meant to be viewed by customers
waiting for their orders. A screenshot is shown in Figure 7.
Ready orders are gathered by polling the server.

To determine whether the system is working as expected,
some evaluation should take place. This section will
elaborate on the tests that were created as requirements fora satisfactory system. This includes unit tests, module tests,
integration tests, and user acceptance tests.
Firstly, unit tests are performed. A unit test verifies thata single component is functioning as expected. In Table 2,
Table 3, Table 4, and Table 5, we describe the unit tests
performed on each module.
Table 2. Unit test for the user interface.

5.2. Hand Gesture Recognition
No.

Hand gesture recognition is done by first estimating thepose using Mediapipe Hands [17]. As shown in Figure 3,
gestures are defined as the curl of each finger and its
direction on the image plane.
The curl of a finger is the average angle between eachfinger bone (the vector between two key points on thefinger), multiplied with a constant such that 0 is “fullystraight” and 1 is “fully curled”. The direction of a finger
on the image plane is determined by the angle between the
X axis and the vector from the base to the tip of the finger.
To determine the gesture, for each GestureCondition
object, we check whether the user’s pose is within the

1

2
3
4
5

107

Description
Item widget: An item widget is created.
The widget should have the correct
layout.
Item widget: Feed dummy data into thewidget. The data should be displayedcorrectly.
Main application: The top right partshould show what the camera is seeing.
Main application: Each part should be in
the correct position.
Main application: Feed dummy data into
the widget. The data should be displayedcorrectly.

Result
Pass

Pass
Pass
Pass
Pass

6

7

8

9

Order management menu: Feed dummy
data into the page. The layout should
display correctly.
Order management menu: Go to the URL
that shows the confirmation alert. The
confirmation message should appear for a
few seconds before disappearing.
Ready orders view: Feed dummy data
into the page. The layout should display
correctly
Ready orders view: Modify the data on
the page. The ready orders should update
automatically.

Table 4. Unit test for the gesture recognition
module.

Pass
No.
Pass
1
Pass
2
Pass
3

Table 3. Unit test for the networking module.
No.
1

2

3

4

5

6

7

8

9

10
11

Description
REST API: Go to the URL for therecommended menu. The server should
return JSON data containing all relevant
information.
REST API: Go to the URL for the entire
menu. The server should return JSON
data containing all relevant information.
REST API: Go to the URL for adding anorder. The server should return JSON
data that confirms the success along withthe order ID. The new entry should exist
in the database.
REST API: Go to the URL for viewing
the list of warnings. The server should
return JSON data containing all relevant
information.
Order management menu: Insert dummyorders into the database. Go to the menu
page. The page should show all ordersthat were inserted.
Order management menu: Click “Ready”
on one of the orders. The order should
now be marked as ready in the database.
Order management menu: Click “Undo”
on one of the orders on the ready list.
This should unmark the order as ready.
Order management menu: Click “Taken”
on one of the orders on the ready list.
This should remove the order from thedatabase.
Ready orders view: Go to the order
viewing URL. The page should show all
ready orders.
Ready orders view: Performing tests 7
and 8, this page should change
accordingly.
Main application: Given JSON data,
output the data in Qt Map representation.

Result
4
Pass
5
Pass

6
Pass

7

Pass

Pass
8
Pass

Description
Module initialization: The initialization
function should initialize the Python
module correctly. This is indicated when
the Python module prints out a successmessage.
Keypoint detection: Given an image, thePython module should produce a list of
21 key points.
Curl calculation: Given a list of five key
points that describe a curled finger, thecurl function should output a high number(close to 1).
Curl calculation: Given a list of five key
points that describe a straight finger, thecurl function should output a low number(close to 0).
Curl calculation: Given a list of five key
points that describe a semi-curled finger,
the curl function should output a medium
number (close to 0.5).
Direction calculation: Given a list of fivekey points that describe a finger, thedirection function should output its
direction correctly.
Gesture comparison: Given a
GestureCondition instance and a Gestureinstance that suits it, the comparison
function should return true. Challenging
cases such as angles that wrap from 360
degrees to 0 degrees should be tested aswell.
Gesture comparison: Given a
GestureCondition instance and a Gestureinstance that does not suit it, thecomparison function should return false.

Result

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Table 5. Unit test for the app control module.
Pass

No.
1

Pass

2

Pass

Pass

3

Pass

4

108

Description
Camera data capture: Give the signal toperform camera capture. An image should
be outputted.
Camera data capture: An image in YUV
color space is given. The conversion
function should output the same image in
RGB.
Control: A dummy sequence of the“thumbs up” signal is given. The module
should send a “confirm” signal.
Control: A sequence of “thumbs down”
signals is given. The module should senda “cancel” signal.

Result
Pass

Pass

Pass

Pass

5

6

7

Control: A sequence of “point up” signals
is given, where the coordinate of the handis changing. The module should send thesame signal. The “point down” signal
should behave similarly.
Control: A sequence of “peace” signals is
given. The module should send a “swap”
signal.
Control: A sequence of “fist” signals is
given, where the coordinate of the hand ischanging. The “move” signal should be
sent.

Pass
10

Pass

A series of integration tests are performed. The purposeof the integration test is to check that the interaction
between multiple modules works as intended. Table 7
shows the list of integration tests performed.
Table 7. The integration tests along with each result.
No.
1

Table 6. Module tests along with each result.

1

2

3

4

5

6

7

8

9

Description
User interface: Start up the application.
The layout of the application should be as
specified in Figure 5.1.
User interface: Go to the URL forordering data. The layout of the data
should be as shown in Figure 5.9.
User interface: Go to the URL forviewing ready orders. The layout of thedata should be as shown in Figure 5.11.
Networking: Using the application, send arequest to the server for the recommended
menus. After receiving the data, theapplication should output the data in Qt
Map representation. Request for the entire
menu and the list of warnings should
behave similarly.
Networking: Using the application, send arequest to the server to order an item.
After receiving the data, the application
should return what the order ID is.
Networking: Open the URL for the ordermanagement menu. The page should
request order data from the database andshow it accordingly.
Networking: Go to the URL for viewing
ready orders. The page should request thelist of ready orders every 10 seconds.
Gesture recognition: Initialize the moduleand add a list of gestures to check. Send aseries of images with different gesturesand one for none of the checked gestures.
The module should be able to correctly
determine which gestures are being
performed.
App control: Start up the app control
module. The module should request the

Pass

Pass

Next, we perform module tests. These tests are meant
to test each module separately before the interactions
between different modules are implemented. Table 6
shows the list of module tests performed.

No.

camera for a new image every frame andoutput the RGB values of the image.
App control: Send a sequence of different
gestures. The signals that are being
outputted should correspond to thegestures performed.

Result

2

Pass
3
Pass
4

Pass

5
Pass
6

Pass

7

Pass

8

Pass

9

10
Pass
11
Pass

109

Description
Start up the application. It should
immediately request a list ofrecommended items and show it as in
Figure 5.2.
Using the app control module, send a“swap” signal. The application should
request a list of the entire menu andrefresh the screen accordingly.
Using the app control module, send aseries of “move” signals. The list of items
should move left and right accordingly.
Using the app control module, send a few
instructions to “add item to cart” and“remove item from cart”. The application
should behave accordingly.
Using the app control module, scroll theentire menu and perform Test 4 again.
The items being added and removed
should be different.
Using the app control module, send two“thumbs up” signals. The interface should
ask for confirmation and send a request to
create a new order.
Add a few items to the cart, then using
the app control module send a “thumbs
down” signal. The cart should be reset.
Using the app control module, send a“thumbs up” signal and a “thumbs down”
signal. The interface should ask for
confirmation and then cancel it.
Initialize the hand gesture recognition
module and the app control module. The
camera should start up and the module
should be correctly initialized.
The tester moves their hand on camera.
The coordinates should be sent as a signalby the app control module.
The tester performs a thumbs up, a
thumbs down, and a peace sign. Eachgesture should be sent as a signal by theapp control module.

Result
Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

12

13

14

15

16

17

The tester points up at one point, andagain at another. The app control module
should send a signal to “add items to cart”
at those two coordinates. Similar behavior
should apply to the “point down” gesture.
The tester moves their hand while
performing a “fist” gesture. The list ofitems should scroll correctly.
Go to the order management menu. The
page should load the list of orders on theleft and the list of ready orders on theright.
Open the order viewing page. Click“Ready” on one of the orders on the ordermanagement page. This order should
move to the ready list and be updated on
the order viewing page.
Click “Undo” on one of the ready orders
on the order management page. Thisorder should go back to the unfinishedorders list and be removed from the order
viewing page.
Perform Test 15 and now click “Taken”
on the ready order. This order should be
removed from the ready orders list andthe order viewing page.

6.

Pass

Pass

Pass

Pass

Pass

Pass

Conclusion

The need for touchless technology is surfaced due tothe pandemic. People are more cautious of the hygiene ofsurfaces. While the responsibility of maintaining hygiene
has always been on the business, this type of awarenesspushes businesses to be even more thorough inmaintaining it. Touchless technology will be able toeliminate the need to touch surfaces, in hope of reducing
maintenance costs and increasing efficiency while still
being hygienic.
To reduce this worry, we propose a digital restaurant
menu that uses touchless technology, in particular hand
gesture recognition. We look at the hand gesturerecognition techniques that currently exists. The methodthat is selected for this project is hand pose estimation due
to being more versatile. We gather requirements by using
a questionnaire. Questions that were asked include
whether the respondent thinks digital menus should be
used at all, what features should be present, and whether
hand gesture recognition would be an improvement uponthese systems. We find out that most respondents agree
restaurants should be using digital menus, and that gesturerecognition would improve upon it.

The design of such system follows. The system isdivided into two main parts: the application and the server.
The application consists of all customer-facing
functionalities, such as viewing menus and ordering. The
server hosts all staff-facing functionalities, such as editing
the menu, managing orders, and showing ready orders.
Implementation of the system follows. The application
is implemented using Qt library on C++. The gesturerecognition module is written in Python, but the functions
were called with C++. The server uses Django for backend
and staff pages. The frontend does not use any libraries, but
is implemented with HTML, CSS, and JavaScript. Thissystem is evaluated through several unit tests, module tests,
integration tests, and user acceptance tests, and was deemedsatisfactory.
The customers can interact with the application without
ever touching the surface. Therefore, the main purpose of
the system is satisfied. However, there were some missing
features that could be subject to future research.
References
1. A. Van Dam, “Post-WIMP user interfaces”, Communications of
the ACM vol. 40(2), pp.63-67, 1997.
2. J.P. Wachs, M. Kölsch, H. Stern, and Y. Edan, “Vision-basedhand-gesture applications”, Communications of the ACM, vol.
54(2), pp.60-71, 2011.
3. M. Oudah, A. Al-Naji, and J. Chahl, “Hand Gesture Recognition
Based on Computer Vision: A Review of Techniques”, Journalof Imaging, vol. 6(8), p.73, 2020.
4. T. Chen. "Fomites and the COVID-19 pandemic: An evidence
review on its role in viral transmission." Vancouver, BC:
National Collaborating Centre for Environmental Health, 2021.
5. H. Shin, J. Kang. “Reducing perceived health risk to attract hotelcustomers in the COVID-19 pandemic era: Focused ontechnology
innovationforsocialdistancing
andcleanliness.” International
Journalof
Hospitality
Management, 91, p102664, 2020.
6. R.Y. Wang, J. Popović, “Real-time hand-tracking with a color
glove”, ACM Transactions on Graphics (TOG), vol. 28(3), pp.18, 2009.
7. M. Perimal, S.N. Basah, M.J.A. Safar, H. Yazid, “Hand-Gesture
Recognition-Algorithm based on Finger Counting”, Journal of
Telecommunication, Electronic and Computer Engineering(JTEC), vol.10(1-13), pp.19-24, 2018.
8. V.S. Kulkarni, S.D. Lokhande, “Appearance Based Recognitionof American Sign Language Using Gesture Segmentation”,
International Journal on Computer Science and Engineering,vol.2(03), pp.560-565, 2010.

110

9. J. Molina, J.A. Pajuelo, J.M. Martínez, “Real-time Motion- 24. G.A. Al, P. Estrela, U. Martinez-Hernandez, “Towards anbased Hand Gesture Recognition from Time-of-Flight Video”,
intuitive human-robot interaction based on hand gesture
Journal of Signal Processing Systems, vol.86(1), pp.17-25, 2017.
recognition and proximity sensors”, in 2020 IEEE International
10. D. Konstantinidis, K. Dimitropoulos, P. Daras, “Sign language
Conference on Multisensor Fusion and Integration for Intelligent
recognition based on hand and body skeletal data”, in 2018Systems (MFI), pp.330-335, 2020.
3DTV-Conference: The True Vision-Capture, Transmission and 25. F. Zhang, S. Chu, R. Pan, N. Ji, L. Xi, “Double hand-gesture
Display of 3D Video (3DTV-Con), pp.1-4, 2018.
interaction for walk-through in VR environment”, in 2017
11. X. Ma, J. Peng, “Kinect Sensor-Based Long-Distance Hand
IEEE/ACIS 16th International Conference on Computer andGesture Recognition and Fingertip Detection with Depth
Information Science (ICIS), pp.539-544, 2017.
Information”, Journal of Sensors, 2018.
26. N.Pepin, A.S. Shibghatullah, K.Subaramaniam, R.A. Sulaiman,
12. L. Ge, Z. Ren, Y. Li, Z. Xue, Y. Wang, J. Cai, J. Yuan, “3D
Z.A. Abas, S.Sarsam, "A Reusable Product Line Asset in Smart
Hand Shape and Pose Estimation from a Single RGB Image”, inMobile Application: A Systematic Literature Review", (2022)
Proceedings of the IEEE/CVF Conference on Computer VisionInternational Journal of Advanced Computer Science andand Pattern Recognition, pp.10833-10842, 2019.
Applications, Vol. 13, No. 9, pp. 53-60, 2022.
13. X.Y. Wu, “A hand gesture recognition algorithm based on DC- 27. A.A.N.Akhla, T.C. Ling, A.S.Shibghatullah, C.S. Mon, A.K.
CNN”, Multimedia Tools and Applications, pp.9193-9205,
Cherukuri, C.L. Yen, L.C. Yi, "Impact of Real-Time Information
2019.
for Travellers: A Systematic Review", Journal of Information
14. C. Zimmermann, T. Broz, “Learning to Estimate 3D Hand Poseand Knowledge Management,2022.
from Single RGB Images”, in Proceedings of the IEEE 28. A.S.Shibghatullah, A. Jalil, M.H.A. Wahab, J.N.P. Soon, K.
International Conference on Computer Vision, pp.4903-4911,
Subaramaniam, T. Eldabi, "Vehicle Tracking Application Based2017.
on Real Time Traffic", International Journal of Electrical and
15. J. Malik, A. Elhayek, F. Nunnari, D. Stricker, “Simple andElectronic Engineering and Telecommunications, Vol. 11, No. 1,
effective deep hand shape and pose regression from a single
pp. 67-73, 2022.
depth image”, Computers & Graphics, vol.85, pp.85-91, 2019. 29. M.S.B. Haini, C.S.Mon, A.S.B. Shibghatullah,A.B.,Jalil,
16. A. Spurr, U. Iqbal, P. Molchanov, O. Hilliges, J. Kautz,
K.A.P.Subaramaniam, A.A.A.Hussin, "An investigation into
“Weakly Supervised 3D Hand Pose Estimation via
requirement of mobile app for apartment residents", International
Biomechanical Constraints”, in Computer Vision-ECCV 2020:
Journal on Advanced Science, Engineering and Information
16th European Conference, Glasgow, UK, August 23-28, 2020,
Technology, Vol. 9, No.6, pp. 1841-1848, 2019.
Proceedings, Part XVII 16, pp.211-228, 2020.
30. A.F.N.A. Rahman, A.S. Shibghatullah, Z.A. Abas, T. Eldabi, C.S.
17. F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung,
Mon, A.A.A.Hussin, "Solving late for relief event in bus crew
C.L. Chang, M. Grundmann, “Mediapipe Hands: On-device
rescheduling using multi agent system", Journal of Engineering
Real-time Hand Tracking”, 2020.
Science and Technology, Vol. 15, No. 3, pp. 1972-1983, 2020.
18. A. Bhargave, N. Jadhav, A. Joshi, P. Oke, S.R. Lahane, “Digital
Ordering System for Restaurant Using Android”, International
Journal of Scientific and Research Publications, vol.3(4), 2013.
Authors Introduction
19. E. Şahin, “An Evaluation of Digital Menu Types and Their
Advantages”, Journal of Tourism and Gastronomy Studies,
Ian Christian Susantovol.8(4), pp.2374-2386, 2020.
Ian is currently an undergraduate
20. J.F. Rusdi, N.A. Abu, N. Agustina, M. kchouri, S. Dewi,
student pursuing his BSc (Hons) in“Software Development Stages of Mobile Computing
Computing studies at the Institute of
Implementation in Restaurant Food Ordering”, SciTech
Computer Science and Digital
Framework, vol.1(1), pp.24-33, 2019.
Innovation (ICSDI) at UCSI
21. S. Desai, A. Desai, “Human Computer Interaction Through
University, Malaysia.
Hand Gestures for Home Automation using Microsoft Kinect”,
in Proceedings of International Conference on Communication
and Networks, pp.19-29, 2017.
22. R. Suriya, V. Vijayachamundeeswari, “A survey on hand
gesture recognition for simple mouse control”, in International
Conference in Information Communication and Embedded
Systems (ICICES2014), pp.1-5, 2014.
23. J. Wachs, H. Stern, Y. Edan, M. Gillam, C. Feied, M. Smith, J.
Handler, “Gestix: A Doctor-Computer Sterile Gesture Interface
for Dynamic Environments”, in Soft Computing in Industrial
Applications, vol.39, pp.30-39, 2008.

111

Asst. Prof. Ts. Kasthuri Subaramaniam
Dr Kasthuri is currently an assistantprofessor at the Institute ofComputing and Digital Innovation(ICSDI), UCSI University, Kuala
Lumpur, Malaysia. She holds a
Bachelor's degree in Computer
Science and a Master's degree in
Computer Science from the
University of Malaya. She has
supervised many college students as
principal supervisor and co-supervisor. She has
publications in Scopus-Indexed and Web of Science. She
is also a co-investigator of the Pioneer Science Incentive
Fund (PSIF) in the field of augmented reality. His researchinterests include human-computer interaction, humanpersonality types, augmented reality, e-learning, mobilecommerce, and e-commerce.
Assoc. Prof. Dr. Abdul Samad bin Shibghatullah
He obtained a Bachelor of Accounting
from Kebangsaan University Malaysia,
Bangi, Malaysia in 1999, a Master ofComputer Science from Teknologi
University Malaysia, Skudai, Malaysia in
2002 and a Ph. degree in Computer
Science from Brunel University in
Uxbridge, UK. He is currently Associate
Professor at the Institute of Computing
and Digital Innovation at UCSI
University in Kuala Lumpur, Malaysia.
His current research interests include optimization, modeling, andscheduling.

112