URVoice: An Akl-Toussaint/ Graham- Sklansky Approach towards Convex Hull Computation for Sign Language Interpretation
Authors: Madhumitha V
Santhi Natarajan
Bharathi Malarkeddy A
Venue: Submitted to ICLR 2023 (2023)
Keywords: Communication disorder
computational geometry
convex hull
sign language
URVoice
vocalizer
computer vision
deep learning

Abstract: We present URVoice, a vocalizer for the communication impaired, based on the Indian Sign Language Notations. Contemporary psychological theories consider language and speech as devices to understand complex psychological processes and deliver them as cultural products of ideas and communication. Sign and gesture language, offering an intelligent co-ordination of eye-and-hand and ear-and-mouth, has evolved as an intelligent manifestation of speech for the impaired. However, they have very limited modality and iconicity in accommodating a greater range of linguistically relevant meanings. URVoice is an Augmentative and Alternative Communication (AAC) device, which currently features a pipeline of forward communication from signer to collocutor with a novel approach shouldered on convex hull using vision-based approach. The solution achieves real time translation of gesture to text/voice using convex hull as the computational geometry, which follows Akl-Toussaint heuristic and Graham-Sklansky scan algorithms. The results are weighed against our other solutions based on conventional Machine Learning and Deep Learning approaches. A futuristic version of URVoice, with voice translated to sign language gestures, will be a complete solution for effectively bridging the cognitive and communication gap between the impaired and the abled lot.

Under review as a conference paper at ICLR 2023
AN AKL-TOUSSAINT/ GRAHAM- SKLANSKY APPROACH TOWARDS CONVEX HULL COMPUTATION FOR SIGN LANGUAGE INTERPRETA- TION
URVOICE:
Anonymous authors authors Paper under double-blind review
ABSTRACT
We present URVoice, a vocalizer for the communication impaired, based on the Indian Sign Language Notations. Contemporary psychological theories consider language and speech as devices to understand complex psychological processes and deliver them cultural products of ideas and communication. Sign and ges- ture language, offering an intelligent co-ordination of eye-and-hand mand and ear-and- mouth, has evolved However, they have very limited modality and iconicity in accommodating a greater range of linguistically relevant meanings. URVoice is an Augmentative and Alternative Communication (AAC) device, which currently features a pipeline of forward communication from signer to collocutor with a novel approach shoul- dered on convex hull using vision based approach. The solution achieves real time translation of gesture to text/voice using convex hull as the computational geome- try, which follows Akl-Toussaint heuristic and Graham-Sklansky scan algorithms. The results are weighed against our other solutions based on conventional Ma- chine Learning and Deep Learning approaches. A futuristic version of URVoice, with voice translated to sign language gestures, will a complete solution for effectively bridging the cognitive and communication gap between the impaired and the abled lot.
intelligent manifestation of speech for the impaired.
INTRODUCTION
The truism of language and scientific linguistics emerged from man's reinforcing vocal behaviour as a response to the evolving social and natural circumstances. The human species grew to evolve as the singular evolutionary group with a unique neurological organization support language. Speech and language, that started off as a mapping of meanings to sounds, has now grown to mapping of complex representational intelligence complicated cognitive communication systems. They have further succeeded in understanding the structural compositions of language in terms of underlying mental expressions.
A communication disorder in human information processing system adversely affects a person's 's ability to talk, understand, read, and write. Individuals with speech and language impairments lack sufficient representational and communication intelligence and leave them with very less choice to express the forbiddingly abstract levels of subtilities human communication. Speech and language impairments are considered high-incidence disability BrainFactt (2012) Francisco (2017) Yukiko & Kiyoshi (2004).
Sign and gesture language, attributed to an intelligent co-ordination of eye-and-hand and car-and- mouth, evolved as an intelligent manifestation speech for the impaired. There is no universal sign language used around the world. There are about 138 to 300 different types of sign languages used around the globe tocha/iichar (2018). few most widely used sign languages are discussed in detail in TI
However, they have very limited modality and iconicity in accommodating a greater range lin- guistically relevant meanings. They also fail to cover the verbal spectrum of temporal and spacial

---

Under review as a conference paper at ICLR 2023
characteristics of communication. Bridging this gap shall strengthen their mental thoughts, avoid reliance on interpreters, and shall also provide access to new technologies.
1.1 TECINOLOGICAA/MEDICA SOLUTIONS FOR THE IMPAIRED: STATE OF THE ART
The existing solution includes text-to-speech and sign-to-speech software enabling the speech im- paired and the deaf and mute to "speak". These Augmentative and Alternative Communication (AAC) devices (listed in Table A2) range from a simple picture hoard to a computer program that synthesizes speech from text.
1.2 PRESENTING URVOICE, OUR SOLUTION
Considering the limitations of existing AACs, study focused designing cheap, compact and portable vocalizer involving vision-based approaches. We present URVoice, a vocalizer based on the Indian Sign Language Notations. It provides simpler and more intuitive way of communication and makes it possible for remote communication.
URVoice achieves real time translation of gesture text/voice using convex hull as the compu- tational geometry, which follows Akl-Toussaint heuristic and Graham-Sklansky scan algorithms. The results are weighed against our other solutions based on conventional Machine Learning and Deep Learning approaches. A futuristic version of URVoice, with voice translated to sign language gestures, will be complete solution for effectively bridging the cognitive and communication gap between the impaired and the abled lot.
AN ARCHITECTURAL OVERVIEW
2.1 URVOICE: ARCHITECTURE
URVoice converts gestures visual input into audio/ text output for collocutor or relays as text message to a computer. Similarly, it takes in audio as input from the collocutor/ computer and converts it into gesture/ text output for the signer. This duplex communication model shall run on an accelerator hardware for optimal performance in real-time communication. A block diagram of the functioning of the vocalizer is presented in the Fig. This paper features pipeline for way communication in URVoice, i.e, visual input to audio/ text output. The gestures are captured in real-time and processed to give audio/ as output.
2.2 CONVEX HULL OPTIMIZATION: A NOVEL APPROACH IN URVOICE
The novelty of URVoice the of convex hull as the computational geometry method for recog- nition of gestures involving use of threshold technique for image segmentation and extraction of various mathematical features from the convex hull. The recognition accomplished by a simple
Vocalloer Desice
Cohoutte Computer
Speaker
Audio outgus
Sign
capture
Dicply
Textouoput
Computer
reloved
Text
Acceberator
card
Voice Text
Signol processingg
DSP prccessor
Gesture detection
Sperch recagnttinn
Pre-proonsing each frams
Feature cxtraction
Gesture
Text
Computer
Collacttrr
Test/ Gesture output
Ausio incorded
microphone
Audiinout
Figure I: Functional overview the Vocalizer
2

---

Under review as a conference paper at ICLR 2023
Miciion is cagitured using Depth Canaera
Gestrre crogped each ftame
Image processing is applied oa gestices cropped:
Geussian Blur IBSV Bizary Mask - Dilnton Eresioa- Grussinn Blar Threshold
Find Coatonus
Find the contocr
dedeets
Using the dececcts, find the leagea. sagle and ummbet of dedects
MUNDESS
ALPIARBTTS
WORLLES
lengil mumlerr defects nuie prodictions
Using saulos defiets make pendictions
Using mumber dedects maize predictions
Texs/ Aedio Ouegut
Figure 2: Overview of the algorithm
and efficient rule-based classifier. An overview of the algorithm for hand gesture recognition is described in Fig. 2 This algorithm predicts only static gestures of both single and double hands.
Table 1: Convex Hull Literature
Algorithm
Proposed by
Complexity (Inlogn)
Description
Graham's 's method (1972)
Graham
To compute the convex hull of points on the plane based on polar angles
Linear time algo- rithm (1972)
Sklansky (1822)
Sklansky
((n)
It was short and elegant to find the convex hull of simple polygon lincar time.
Gift method
wrapping
Chand and Kapur Chand
((nh)
It determines the left most point, which is a vertex of the convex hull, then searches for the point that his on one of the line from the current vertex it. All the vertices of the convex hull can be found by repeating this procedure.
& Kapur (1970. Jarvis
Jarvis
Graham sean
Graham Graham 1972
(nlogn)
A modified version of Gift wrapping method. Starting from the lowest puint, all the points are sorted in increasing order of the angle of the lowest point made with the x-axis, and then efficient scarching scheme can be performed.
Monotone
chain
Andrew Andrew 1979
(nlogg)
variant of the Gram scan. sorts points their coordinates
algorithm
instead of angles.
N-dimensional
Barber, Huhdanpaa
Dobkin and
(mllggr)
uses divide and conquer approach like that of quicksort.
Quickhull
Barber
Hoare (1661)
Divide and con- quer
Preparata and Hong Preparata (977))
(nlogm)
This algorithu is applicable to 3D case.
& Hong
Incremental con-
Kallay Kallay (1984)
(nllgn)
used to develop 3D convex hull algorithms.
vex hull algorithm
Ultimate
pla- hull
Kirkpatrick and Seidel Kirkpatrick 1986)
(nllohh)
The first optimal senut--ennitive algorithm. is a modifica- tion of the divide and conquer algorithm by using the technique of mariigg-heffor-coonqees and low- oow-dimnssiona linear program- ming.
convex
Seidel
algorithm
Chan's 's algorithm
Chan Chan 1996
(nloghh)
A simpler optimal output-senstive algorithm. wrapping with the execution of an algorithm with ((noogn) (such Graham scun) small subsets of the input.
It combines gift
Akl-TTuussain
Selim Akl and T. Tou-
((n)
often used as the first step in implementations convex hull algorithm to improve the performance.
heuristic
ssaint Bhattacharva
&
Toussant [1983] AKI
loussunl
1978)
Graham-Sk Sklansky
Preparata
Sharos
((n)
is similar to the Sklamsky algorithm. Often used after Akl- Toussaint heuristic
1985
3

---

Under review as a conference paper at ICLR 2023
(a) Captured image
(b) Cropped image with histogram
(c) Blur image with his- togram
600
200
200
180
(d) HSV image with his-
(e) Masked
image
with his-
(l) Dilated image
with
his-
togram
togram
togram
(g) Eroded
image
with his-
(h) Blurred togram
image with his-
(i) Thresholded image with his-
togram
togram
Figure 3: Image Progression in URVoice
AN ALGORITHMIC OVERVIEW
3.1 URVOICE CONVEX HULL OPTIMIZATION
URVoice's central algorithm mainly features convex hull optimization it provides the necessary keypoints in gestures for their recognition. The computation of the convex hull is central problem in computational geometry. has been vastly studied, not only because of its practical applications, such as computer graphics and statistics, but also because other computational geometry problems start with the computation of a convex hull. The various proposed convex hull algorithms are dis- cussed in detail in the Ta
3.1.1 THE
TOUSSAINT HEURISTIC MODEL
The Graham-Sklanske scan is important technique in computational geometry which was inde- pendently proposed by Graham to compute the convex hull of a sorted set of points and by Sklansky compute the convex hull of simple polygon. Whereas the Sklansky scan fails for simple poly- gons, it succeeds for star-shaped polygons, fact upon which the correctness of the Graham scan relies. The idea of the Graham scan is make single scan through a sorted list of points. At each step in the scan, either a point is deleted or retained based on the test. So, if there are points, maximum of points can deleted. Thus, the algorithm takes O(n) time. Since its intro- duction the Graham-Sklan Sklansky scan has found widespread application to other problems. The idea of Akl-Toussain heuristic to quickly exclude many points that would not be part of the convex hull Xianshu et al. (2003)
Thus, the convex hull method follows Akl- Touussustt heuristic, which results in a monotone polygon, followed by Craham-Sklanske
to obtain the points of convex hull.
3.2 URVOICE COMPUTATIONAL PIPELINE
The pipeline presented consists of the following stages :-

---

Under review as a conference paper at ICLR 2023
1. Image Acquisition
2. Pre-proe pre-processing
3. Feature Extraction
4. Gesture Prediction
It is discussed in detail in the following subsections. The mathematical model for the pipeline is described A.1
3.3 IMAGE ACQUISITION
An RGB image of gesture is captured during run-time with the help of an external depth camera, as shown in Fig. B(a.. The camera used in this research is Orbbec Astra Pro with a resolution of 1280*720 @30fps.
3.4 PRE-PROCESSING
The region of interest is cropped out of the captured image having a size of (200 200) pix- els for number gestures and (400 x 200) pixels for alphahet and word gestures, and is stored as prop_image as shown in Fig. 3b) along with its histogram. The image is pre-processed using the
Table 2: Pre-processing stages
Stage Name
Description
of filter
Gaussian
Gaussian blur is applicd first on crop_ jmpaam to smoothen the pixel values of the hand so as to create some amount of uniformity. Here, a Gaussian filter convolves over this image and thereby blurs it. The worktlow for obtaining the Gaussian filter is presented in Algorithm A.2 From this algorithm, the value obtained 0.8 and the Gaussian filter obtained
Blur
0.06292
0.124998
0.06292
0. 1.129998
0.248326
0.124998
0.06292
0.124998
0.06292
After convolution, blur image is obtained, shown Fig. c). and stored as blur with BGR pixel format. The histogram of Fig. B(e) shows minor changes in the peaks and valleys of the histogram. The BGR format of the image is then converted HSV format mainly focus the intensity component of the image which helps in eliminating the background noise and shadows.
IIV
HIS color space most suitable for boser- image segmentation and it is and it is hence directly applied on blur with BGR image format. The Algorithm 2 presents the workflow for obtaining the hav values. The bgr values of the blurred image are then replaced by the hsv values after applying 2|
stored hav. The resultant image obtained as shown in Fig. (d along with its histogram. It can be observed from the image that the region of interest, ic.. the hand, 18 now covered green pixels. Thus, mask
applied on to focus the region interest.
3
Mask
A mask is applied on hso to crop out the gesture within the image. The workflow for the same is resultant image obtained is as shown in Fig. ((e) along with its histogram which now has two peaks.
presented in Algorithm 3A222
binary mask is returned and stored in mask.
Dilation
Dilation is applied on mask to overcome the disfigured regions and also assist in joining broken parts of the image. It observed that the thickness of the foreground object has increased. This operation consists of convolving the image (mask) with kemel (5 x unit matrix). The kernel has defined anchor point which is the center. As the kernel is scanned over the image, the maximal pixel value overlapped by the kernel is computed and this replaces the image pixel at the anchor point position. This process takes place once thrughouu the image. The final image ohtained, as shown in Fig. 3 along with its histogram, is stored in dilation.
Erosion
Erosion is applied on dilation eliminate noise if any. is observed the thickness of the fore- ground object has decreased. This operation is similar to dilation. As the kernel [5x5 unit matrix] scanned over the image. computes the minimal pixel value overlapped by the kemel and this minimal value replaces the image pixel under the anchor point. This process takes place once throughout the image. The final image ubtained, as shown Fig. gg) along with its histogram, is stored in erosion. Now, the resultant image contains rough edges whech has to be fixed by blurring
6
Gaussian Blur
Gaussian blur is again applied the edges of the applied on crosion to smoothen the image. This helps to remove the deformities gesture formed due erosion. The procedure followed same in Point under Pre- tre-prucessing and the resultant image obtained, as shown in ())), is stored in filtered. The histogram in the Fig. h)) shows appearance small new peaks between the earlier obtained peaks. This is considered noise and hence has be removed by thresholding the image.
Threshold
Thresholding is applicd on filtered to ensure uniformity of higher-grade gray pixel values the image. The workflow for thresholding the image presented in Algorithm| 44.. The resultant hinary image obtained, as shown in Fig. (3 is stored in thresh. The small peaks in the histogram of the blur image (Fig. Bi) erased in the histogram of the thresholded image (Fig. Bi)).
5

---

Under review as a conference paper at ICLR 2023
Table 3: Gesture Features
Feature
Description
Contours
Contours algorithm processes the arbitrary binary image threah and returns a vector detected contour points. It uses CV RETR TREE mode retrieve all the contours and reconstruct full hicrarchy of nested contours along with CV DVVCHAINN APPROX SIMPLE method to compress horizontal, vertical, and diagonal segments and leaves only their end points. Contour retrieval follows Suzuki's algorithm which is explained f(x,y) v) denotes the value of the pixel at location (2.y), The uppermost row, the lowermost row, the rightmost column, and
follows: The function
leftmost column the picture composes the frame. Then, unique number is assigned border and it is denoted by NBD. The criteria for checking the outer border or hole border is shown in Fig. 0 Ha). The NBD of the frame is assumed to be Other borders are numbered sequentially. The information of the parent any border is saved in LNBD last NED. The workflow to find the contour points is presented in Algorithm [|2.2] The retrieved tree of list of points are stored in contours. s. Using contours, s,oors, the largest contour in terms of area is Isolated and stored as contour. Fig. b) shows the contour line (as green line) joining all the points in contour. The area of contour is calculated using the formula presented in Algorithm (|222
every new
Convex Hull Convexity Defects
The smallest convex polygon, that encloses all the of contour points in contour, is found by convex hull method which follows Aki- Tousassst hcuristic followed by Graham-Sklanne rithm returns
and
scan as presented in Algorithm [ This algo- sequence of stack of vertices of the convex hull and stored in hull. Fig.
shows the convex hull lines (as red lines) joining all the points in hulf, Considering H [n] as convex hull points and C'nl] as contour points, for each pair of adjacent hull points (8. H(t + I]), defining one edge the convex hull, the distance from the edge for each point on the contour C' that lies between H [] and H|: calculated. The worktlow for calculating the maximum distance is presented in Algorithm A.2 If the distance is greater than zero, then defoct considered (n) of the contour point where the maximum is located and the maximum distance is recorded in de fects as list containing: start point, end point, farthest point, approximate distance farthest point
(excluding Clnl Hli 1 is
present. When defoct is present. the value i 1, the index
Length and Angle
For each defect in de fed.s, the coonlinates of start point, end point and farthest point is stored in start, end and far respectively. The workflow for calculating lengths angle formed at the vertex of the farthest point is presented in Algorithm Algorithm8jA.2
the sides of the triangle formed by joining these points and depiction of these distances and angle with respect to fingers having convex defect is shown in Fig. (c). If the arigle calculated is less than 90", then variable count de.efeeetttt fects incremented by one. The final image depicting the contours green line and defects red dots is shown in Fig. dd.
1-1
Outer
Hale
(a) Outer and hole bor- der check
(b) Contour and Convex hull
(c) Sides and angle w.r. to a Convex defect
(d) Final image
Figure 4: Convex Hull Optimization in URVoice
Table 4: Gesture prediction
Gesture
Methodology
Number
When a defect with its vertex angle less than 90 is encountered. variable is incremented with the value of the distance a, else, a variable B is incremented with the value of the distance b. These variables A and B represents the sum of distances between the fingers and the sum lengths the fingers. Finally, the predictions for the number gestures are obtained by using the values count de fects. A and B presented in Algorithm
Alphabet
The alphabet gestures are predicted with the help of count_ de fects and area values as presented in Algorithm 10
Word
The word gestures are predicted with the help of count de fects and area values as presented in Algorithm l A.2
OpenCV library in Python to extract the necessary features. The pre pre-processing stages explained
in Table
3.5 FEATURE EXTRACTION
By employing the fully segmented image thresh, required features are extracted and assembled for prediction. The requisite features for prediction are presented in Table

---

Under review as a conference paper at ICLR 2023
Algorithm 1:
Convex Hull(conlour)
1: // Purpose: To find the coordinates of conves hull of the contour
2: Ioput: (sr; 5) coordinates
3: // //utput:
coordinates of boundary points of convex bull
4: Parameters: I Top point at the top H Stack, TopLess- point just below T TopMe ost
5:
6: PosinX point with lowest X-coordinate
7: Pmons x puánt highest X-coordinate
8:
point with lowest Y-coordinate
9: PmexY point highest Y-coordinate
10:
Detine the bounding box and quadrant of PminX, Pmaxx Pminy and PmaxY
the points inside the quadrant then
12:
Discard the points
13:
else
14:
Select point lying regions
hounding box and quadrant
15:
end if
each corner region
17:
minif(a Y)=
=
Discard points lying inside the triangle (pmii X P1: Ppmaxy)
lie and ahove the line I.pmo X. Pmoxx then 21: Sort them in ascending order of their X-coardinates and stack in P
20: the non-discarded
22:
23: Sort them descending order of their X courdinates and stack in
24:
25: Apply Graham- Graham-Sklansky procedure the obtained monotone polygon:
Initialize empty HullStack
27: P[N
P[I]
28: Push (P[1), HullStack)
29: Push (P(2), StlSaakk)
30:
for Ind := 3
do
31:
while (HullStackk stacks more one element) and (P[Ind) the L(TopLess, TopMost) do Stack) [discard T Mopt end while
32:
33:
34:
[asb(P[[nd], HullSteck)
35:
end for
36:
pop(H Stack)
Table S: Gestures studied
Table 6: Time taken to process the code
Gesture
Names
Gesture
Mid-range of time taken (in seconds)
Numbers
I. 1.2.3.4.5.66..888
Text output
Audio output
Alphabets
B. B.C.L.P.RR. W.T Doctor, This, Your, Mine
Numbers
0.1221
1.1133
Words
Alphabets 0.1103
1.1662
Words 0.0804
1.1457
ONE
FOUR
TWO
TRREE
FIVE
8
SIX
SEVEN
LIGHT
NINE
(a) Number predictions displayed screen
(b) Alphabet predictions displayed sereen
DOC OR
THIS
YOUR
MINE
(c) Word predictions displayed on screen
Figure S: Output

---

Under review as a conference paper at ICLR 2023
Table 7: Benchmarking Convex Hull Optimization with selected ML/DL Models
Model
Classes trained
Dataset Specifications
Model Architecture
Performance Analysis
Support Vector
0. 1.
2. 3.
RGB image dataset of static single hand gestures of shape [400x400x3]]
A pipeline with Randomized- PCA for dimensionality reduc- tion lollowed SVC classi- fier with "rbf" kermel and "bal- anced" class weight. Trained data using GridSeurchCV.
Testing accuracy: 95.86%
Classifier
5, 8,
9 [otal:
Testing 4.14%
loss:
classes]
300 images per class [Totall: 2700 images].
Validation
Training set: 2518 images
racy: 93.68% Validation 6.32%
Validation sct: class
images per
loss:
Testing set: 13 images per class
Artificial Neu-
2.
4.
Numerical dataset of 8 extracted features of hand gestures [nor- malised datal.
A Sequential Model with hid- den layers which uses relu ac- tivation function. The put layer uses softmax activa- tion function, adam optimiser and categorical crossentropy loss function.
Training accuracy: 94.59%
ral Network (only numbers)
6,
7.
8,
9 [Total:
Training 17.97%
loss:
classes]
1000x8 features per class [Total: 9000x8 features].
Validation acCu-
Training set: 6300x8 features class
racy: 97.33% Validation 10.94%
loss:
Testing 1800x8 features per
class
Artificial Neu- ral Network (numbers, al- and
2.
Numerical dataset extracted fea- tures of hand gestures [normalised data].
Sequential Model with hid- den layers which uses relu ac- tivation function. The put layer uses sofImax activa- tion function, adam optimiser and categorical crossentropy loss function.
Training accuracy: 93.70%
5. 6,
7. 8,
B. L.
Training 17.80%
loss:
phabets words)
P. R, W.
1000x8 features per class [Total: 20000x8 features).
Doctor,
Validation accu-
This, Your, Mine [Total: 20 clusses]
Training set: 6300x8 features per
racy: 95.32% Validation 14.01%
class
loss:
Testing 1800x8 features per
class
Convolutional
RGB image datasct of static single hand gestures of shape [400x400x3].
Sequential Model with 2 Convolution2D layers, 2 Max- Pooling2D layers and hid- den layers of neural network with relu activation function. softmax activation function, adam opti- miser and categorical tropy as loss function.
Accuracy: 86.67%
Neural Network
4, 5, 7, 8, [Total:9
Loss: 13.33%
classes]
60 images per class [Total: 540 im- ages].
Precision: 91%
Rocall: 87%
Training 40 images per class Testing set: 20 images per class
The output layer
Fl-score: 87%
Convolutional
RGB image dataset of static single hand gestures of shape [400x400x3).
Sequential Model with Accuracy: 98.89%
Neural Network
4, 5,
7, 8,
Convolution2D layers, 2 Max- Pooling2D layers and den layers of neural network with relu activation function. softmax
[Total:9
Loss: 2.46%
classes]
300 images class [Total: 2700 images].
Precision: 98%
Rocall: 98%
Training 250 images per class Validation set: 30 images per class Testing set: 20 images per class
The output layer activation function, adam opti- miser categorical tropy as loss function.
Fl-score: 98%
Convex Hull Optimization technique
2, 3,
images in video stream: (200x200) pixels for num- ber gestures and [400x200] pixels for alphahet and word gestures.
Real-time
rule based classifier as dis- cussed in subsection 3.6
Refer Table
7, 9, B.
C. L, R,
W. T, Doc-
tor, This, Your, Mine
[Total:20 clusses]
3.6 GESTURE PREDICTION
The gestures of the Indian sign language studied are mentioned in the Table 5 the extracted features along with additional values, predictions made for gestures of numbers, alphabets and words as described in Table
By employing
4 RESULTS AND DISCUSSION
4.1 URVOICE RESULTS
The algorithm relays the prediction both in terms of text and audio. The algorithm with output as text uses cv2 library OpenCV to display the predictions on the screen. Fig. (1b.)) shows the text output of various gestures displayed the screen. Likewise, the algorithm with audio as
8

---

Under review as a conference paper at ICLR 2023
output makes of pyttsæ3 library. Once the predictions are made, the text is instantaneously converted to speech. One advantage of using Python's 's pyttsæ3 library is that it works offline and hence it is convenient to use anywhere. The mid-range of the time taken process the code to obtain the output of prediction is provided in T The data in the table illustrates the fact that the time required for audio output is greater than text output due the text to speech conversion module and also there is small lag in the run-time due to the time spent to play the audio.
4.2 BENCHMARKING CONVEX HULL OPTIMIZATION WITH SELECTED ML/DL MODELS
We trained and deployed a few other Machine learning and Deep learning models for hand gesture recognition, the details of which are listed in Table For this, created two different datasets:
1. Image Dataset: 2700 RGB images of 3 individuals were captured for 9 classes of number gestures with different brightness levels.
2. Numerical Dataset: Extracted real-time hand gesture features of 2 individuals which includes: sum of distances between fingers, sum of lengths of fingers from tip to convex defects on both sides, sum of distances from tip of fingers to the centroid of the palm, sum of angles between the fingers, total area of hand gesture covered, total perimeter covered by the convex points of the hand gesture and number of convex defects. These features were extracted using the same algorithm discussed in this paper.
Although these models performed well, the convex hull optimization technique was preferred over this based on the obtained performance and the type of system we wanted to implement. Using CNN model would result in designing GPU based system for hand-held device which is not feasible solution due to factors like memory usage, size and cost. The experiments performed using convex hull method deduces best results with the features under study and also was able to reliably recognize the gestures in real-time, though there were some limitations imposed by the presence of image noise. Thus, the convex hull optimization technique proves have better static gesture recognition rates.
5 CONCLUSION AND FUTURE WORK
This paper presents the architectural design of the vocalizer, URVoice for Indian sign language inter- pretation. This device comes under the category Augmentative and Alternative Communication (AAC) Devices. We have described and evaluated the forward communication process of converting static gesture input to text/ audio output. The novelty of this approach is the use convex hull as the computational geometry method for recognition of gestures. We have also trained CNN model for predicting gestures from images and ANN model for predicting gestures from numerical data. We preferred convex hull over CNN due its computational efficiency and architecture.
In our future work, we will build an algorithm to recognize the dynamic gestures and implement the feedback channel of converting audio input from collocutor/ computer to text/ gesture output which will be displayed on the screen to complete the software part of our architecture. We will also be developing hardware prototype using accelerator hardware and embedded firmware. This will serve to be a high-performance low cost and portable infrastructure to be used as gadget that serves the interpreter and translator for the communication impaired.
REFERENCES
Akl and G.T. Toussaint, A fast convex hull algorithm. Inform. Process. Lett., 7:219-222, 1978.
A. M. Andrew. Another efficient algorithm for convex hulls in two dimensions. InformationPra cessing Letters, 1-216-219, 1979.
C. B. Barber. D. dobkin, and h. huhdanpaa.the quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software, 22:469-483, 1996.
B. K. Bhattacharya and G. T. Toussaint. Time-and and storage-efficins effiicient implementation of an optimal planar convex hull algorithm. Image Vision Comput., 1:140-143, 1983.

---

Under review as a conference paper at ICLR 2023
BrainFacts. Language: An overview.
http://www..rrinicccccc/hhhnnnnn and- hwwp:/ngg//aguuuggg////////g/gg lotegge-nn-veeeyee oeeovveriwww 2012.
T. M. Chan. Optimal output-sensitive convex hull algorithms in two and three dimensions. Discrete and Computational Geometry, 16:361-368, 1996.
D. R. Chand and S. Kapur. An algorithm for convex polytopes. Journal of the ACM(JACM), 17: 78-86, 1970.
Aboitiz. Francisco. A brain for speech. 2017.
R. L. Graham. An efficient algorithm for determining the convex hull of a finite planar set. Info. Pro. Lett., 1:132-133, 1972.
C. A. Hoare. Algorithm 64: quicksort. Communications of the ACM, 4:321, 1961.
R. A. Jarvis. On the identification the convex hull of finite set of points in the plane. Information processing letters, 2:18-21, 21, 1973.
M. Kallay, The complexity of incremental convex hull algorithms in rd. Information Process-ing Letters, 19:197, 1984.
D. G. Kirkpatrick and R. Seidel. The ultimate planar convex hull algorithm? SIAM journalon computing, 15:287-299, 1986.
Services NII.. The different types of sign language. http://hiierrrreecccmm/miieeeeee language/, 2017.
F. P. Preparata and S. Hong. Convex hulls of finite sets of points in two and three dimensions. Communications of the ACM, 20:87-93, 1977.
P. Preparata and M. Shamos. Computational geometry an introduction. Springer, 1985.
Brooks, Richard. A guide to the different types of sign language around the world. https://k- http:///wwgiiiiittttttsooossgoga
2018.
Sklansky. Finding the convex hull simple polygon, pp. 79-83, 1982.
Kong. Xianshu, Everett Hazel, and Toussaint Godfried. The graham sean triangulates simple poly- gons. 2003.
Notaand. Yukiko and Honda. Kiyoshi. Brain regions involved in motor control of speech. 2004.
A APPENDIX
A. MATUEMATICAL MODEL
A discrete RGB image is defined as X x k hypermatrix I, where the mnk"h entry represents the mh th pixel in colour channel k. Consider (k = 11thh channel as B matrix, (k 2)th channel as G matrix and (k th channel R matrix. Gaussian function ge(2,, y) is applied separately each th matrix where,
224 20
(x
1/0 Standard deviation
(I)
2
Then be, a Gaussian blur of image Tr. is achieved by convolving it with (x( y).
be be(x,y) y) = () ge
ge(x - a, y ()la(a,
(2)
The blur image B(....) from I matrix is obtained by:
Ba(i,) Em EL
(t 1) [a p- 1,j 1,5 -
(3)
Ep D (((p.
10

---

Under review as a conference paper at ICLR 2023
(i, values normalized and transformed into HSV matrix with hue H [0*, saturatii S [0, 1] and value V E [0, 11. The equations for conversion of BGR matrix into HSV matrix is presented in algorithm defined below:
A.2 A new binary matrix with values 0 or 255 is created from HSV as
for (2 & ((i,)) & 20)A
255,
(0 S S(i.)) S 255
M(
(4)
(0 S V(i,)) S 255)
0,
otherwise
The intensity of white pixel is intensified by dilation using 5 5 unit matrix K. Dilation of matrix M by K defined by:
D MOK-
U
Mr fz: R. #0}
(5)
KEK
where,
K = ( K}
(6a)
K. (,: E K}
(6b)
In matrix transformation, dilation implemented as:
D(i.j) (a mar[K((p M(i p 1.j +1 1)]}
where, p (1, and / (1,
The intensity of white pixel is reduced by erosion using 5 x 5 unit matrix K. Erosion of matrix D
by K is defined by:
D e
D-x fo KG D) n
(8)
kEK
In matrix transformation, erosion is implemented as:
5i. (w min[K(p,)) Díi fii+p +1 1))}
(9)
where, p (1, m) and / (1. n)
Again, Gaussian function is applied to matrix E and is defined as:
f(x.8)
g:(2 a, y F(), ())dd
(10a)
m p- 1 1) + "p Ei (p,
F(i.j)
(10b)
threshold value is set to classify the values of matrix F to create binary matrix of value 0 or The threshold value is defined by:
M4 + poo
(11)
where, and Mo are the means of the distribution in the gray level values in matrix F. The matrix obtained after thresholding F is defined by.
T(i - 0. if F(1j))
(12)
otherwisc
A curve joining points with equal intensities in the matrix T would represent contours. The (i. j) coordinates of contours are obtained as presented in algorithm A.A.2 Let C(x,, y) be the contour line joining all contour points, then the area covered by the line is given by the Green's theorem as:
Area 2 !
rdy ydr
(13)
In coordinate system,
(ox41 x4 (1 yk)
Area
(14)
2
km
The convex hull of of points in n dimensions is the intersection of all convex sets containing C. For N points Pı, ... PN, the convex hull H is given by the expression:
H
Aps : xy 2 for all j and
A1 = 11
(15)
11

---

Under review as a conference paper at ICLR 2023
Let the line joining the hull points be H and the perpendicular line from H to the contour line lying within it be d. If H d maximum, then defect is considered and the coordinates of the contour point, hull points and the distance are recorded. The distances, angle and defects using these points calculated in Algorithm 8 A.2 make various predictions.
A.2 ALGORITHMS
Algorithm 1: Gaussian
(3.3), 0)
1: // Purpose: To obtain the Gaussian filter
2: // Input: crop_inee srop_imaee.. kernel size, sigmaX value
Output: Gaussian filter
4: Parameters: x distance from origin horizontal axis
5: = distance from the origin in the vertical axis
6: the standard deviation of the Gaussian distribution
7:
8: if sigmaX then
9: σ = 0.3 ((ksize 1) + 0.5 1) + 0.8
11 ksize kernel size 33
10: end if
G(x,y) v) 2no? 20
Algorithm HbV(bl)))
Purpose: obtain HSV image
2: // Input: blur with BGR values
3: // Output: HSV image
4: // Parameters: b.g.r= Coordinate pixel values of BGR image
5:
6: for each coordinate pixel value (b.g.r) do
7: Divide b, r by 255
8: emaz mar b, rrr)
9:
cminn mbinbb. r)
di ference cmag crnin
11:
if cmax = cmin then
12:
h -
13:
end if
if emax then
h= (60 ((g - b) ference) 3366%6600
16:
end
17:
if cmax then
18:
h = (60 ((b r) rdd 120)%360
20:
if cmax =b then
21:
h= (60 * ((r gdf + 240)%330
22:
end if
23:
if emax then
25:
end
26: if cmax 1= 0 then
27:
/cmaz) 100
28: end
29: V cmax 100
30:
end for
12

---

Under review as a conference paper at ICLR 2023
Algorithm 3: Mask(hsu, lower limit range, upper limit range)
1: // Purpose: To mask hsv
2: // Input: hsv, lower limit range(2, 0, 0), upper limit range(20, 255, 255)
3: Output: Binary mask
4: Parameters: lowerb lower pixel limit range
5: upperb upper pixel limit range
6: hsv(I) hsv value
7: mask destination image
8:
9: for each coordinate hav pixe! value do
10: if (lowerby < hsu(T)o & upperbo) A (lowerd) & hsv(T)2 S upperbi) (lowerb2 s hoo(7)z 5
upperba) then
11:
mask 255 (white)
13: mask = (black)
14: end
15: end for
Algorithm 4: Threshold(fiteree))
1: // Purpose: To obtain threshold of filtered
2: // Imput: filtered
3: Output: Uhres
4: // Parameters: (x, y) Coordinate Pixel Value of f filtered
5: T Threshold Valuc(127)
6: g(x.y) destination pixel value of thresh
7:
8: for coordinate pixel value do
9: if f(x y) < then
10:
[ = 0
11: else
g(x, v)
13: end
14:
end for
13

---

Under review as a conference paper at ICLR 2023
Algorithm 5; Contours
1: // Purpose: To find contour points
2: // Input: thresh
3: // Output: confours
4:
5: Scan the image from left to right till an object pixel is found
6: Check if the pixel is an outer border or hole border.
7: When new row is scanned, reset L, BD to
8: if pizels 0 then
9: fay and faa-1 0 Jouter border] then
10:
NBD+=1
11:
Set (22, 12) as (z, 1)
12:
else [hole border] then
N NBD+= =1
14:
Set (22, 1/2) as (z, y + 1)
15:
if fxy > then
16:
LNBD fxy
17:
end if
19:
Go to line 42
20:
end if
21:
From this starting point, trace the border
22:
Starting from (222 U2) check clockwise around the pixels in the neighbourhood of (z, V). When nonzero pixel found, denote as (æ1, //1 if no nonzero pixels are found then
23:
24:
Set Jay BDBD
25:
Go to line 29
end
27:
Set (222, y2) = (#1, y1) and (Ia, ya) (z, v)
28:
Starting from the next element of the pixel (I2, 82) in the counter-clockwis order, traverse the neigh-
bourhood of the (It,, yts) in the counter-clockwise direction to find the first nonzero pixel and set it
(7., 74).
if pixel at (I3, V/ + 1) is O-pixel belonging to region outside the houndary then
30:
The current pixel (ra, ya) = - BDD
31:
else the pixel at (æa, Ms 1) is not a O-pixel and the current pixel value is then
32:
The current pixel (æ3, ys) BDD
34:
Do not change the current pixel value
35:
end
36:
if In line 28, (#4, (z, and (It, ya) (21, v1) [back to starting point] then
37:
Go to line 42
38:
39:
Set (I2, y2) (rs, ys) and (03, (s) (Ia, y4)
40:
Go back to line 28
41:
end
42:
if fay! then
43:
Set LNBD [savl
44:
end
45:
Start scanning from the next pixel (z, y
46:
Stop the process when the bottom right corner of the image reached.
end if
14

---

Under review as a conference paper at ICLR 2023
Algorithm 6: Area(contour)
1: // Purpose: To find the area enclosed by the contour
2: // Input: contours
3: // Output: Area
4: Parameters: n number vertices
5: (It, yk.) = kth point when labelled counter-clockwis manner (the co-ordinates are stored the
contour list)
6: (In+1, (n++1) (zo, yo the starting vertex is found both at the start and end of the list of vertices.
7:
8:
(Tre 11 Kaka 226)
9: Area 2k=0
Algorithm 7: Max Distance(hull, contour)
Purpose: find the maximum distance between convex hull edge and contour
2: // Input: feull, contour
3: // Output: Maximum distance
4: // Parameters: H i]] convex hull points
5: C'nn) contour points
6:
7: for == 0 n do
8:
dro Hi + 1], H1
9: dyu H1 + 1/ H H
if (dru 0) (dyo 0) then
11:
scale 0
12:
else
13:
scale
Vdzg dyo-
14:
end
15:
for
j=0 to do
16:
da Chi H[ fly
dy = CDib
18: distance =| (-dyo * dx + dao * dy) +scale
19:
end for
20: distance max (distance)
end for
Algorithm 8: Calculate lengths and angles
1: // Purpose: To calculate the lengths and angles
2: // Input: start, end and far
3: // Output: a, b, and angle
4:
5: a (e start 0))2 + (end[1] start 1]
6: V[Jar 0 start 0)2 + [ start[]]
7: V [e [ar + (end[[]] far(1)²
8: angle sin (.2 2)008
2bcx3.14
15

---

Under review as a conference paper at ICLR 2023
Algorithm 9: ; Number prediction
Purpose: To predict the numbers
2: Inpet: ongle, A- sum lengths of a and of lengths
3: // Output:
Prodicted number
5: count, couns_defects 0 A. B 190 then pred= "ONE"
7: else couniideeeeet fe pred "SIX"
B 190 then
else count_ seruttddefectee
then
10: pred= "TWO"
11: count fectss
2 then
12: pred = "SEVEN"
13: else count coauttddfecte
A A S 70 then
14: pred "THRREE"
15: else if count, fects = A A S 80 then
16: pred= "EIGHT"
17: else if count_ fectss A A I 90 then
18: pred= "FOUR"
19: else counto defettdeffcts
A A 100 then
20: pred= "NINE"
21: else if count. coont__defctt. then
22: pred= "FIVE"
23: end if
Algorithm 10: : Alphabet prediction
1: Purpose: predict alphahets
2: // Input: count_de defects and area
Output: Prodicted alphabet
4:
area TOWDOT
6: coount_dee defects = 3 A 1.5 Saras 2.0 then
7: pred= "B"
8: else count count_dee defects A 0.5 Sareas 0.8 then pred
10: else it count_defects
A 0.5 <aras 0.8 then
pred="L"
12: else count deeeetddeeeceeeeee
0.9 Saras then
13: pred="1" "F"
14: count_dee defttddeffctss
Sureu
15: pred = "R"
else coontt_de feinntddeeette
0.8 <aras
17: pred= = "W"
ebse if count_ fe fects A 0.9 Sarea 1.9 then
19:
pred=
20:
end if
Algorithm 11 Word prediction
1: // Purpose: To predict the words
// Input: count, defefer fects, area
3: // //Otppt: Predicted word.
4:
5:
ureu
6:
areen
count de
7: pred "DOCTOR"
else U.5 SareaS 1.9 then
9:
pred "TUIS"
10:
else 1.9 <aren 2.5 then
11:
pred=
12:
else if Sarea S. 3.5 then
13:
pred "MINE"
14:
end
16

---

Under review as a conference paper at ICLR 2023
A.3 TABLES
Table A1: Existing Sign Languages
Sign Language
Region
Influenced by
Number of Significance
signers
American
Sign
America,
French Sign Martha's Vinevard Sign Lan-
Language,
250,000-
one of the easiest languages to learn because the signs developed to mimic the actual word or phrase representing.
Language
Canada,
500,000
(ALL
Southeast
guage and other local sign
2017
Asia
and
languages
West Africa
British
Sign
Evolved at Thomas Braid- wood's schools for the deaf in the late 1700s and early 1800s
150,000
spread Australia Zealand, thus resulting in similarity of New Zealand Sign Language and Auslan (Australian Sign Language)
New
Language
[BSI,
2018
French
Sign
France and Switzerland
Developed from the Parisian 100,000 deaf community, and taught by Charles Michel de
It is one the earliest European sign languages by educators, and it influcneed other sign languages like ASL. ISI., Rus- sian Sign Language (RSL) and many more.
Language
gain acceptance
(LSF Richard
2018)
Bruzilian
Sign (Li-
Bruzil
French Sign Language
3 million
It was given official status by the Brazilian government in 2002.
Language
bras Richard
2018)
Indo-Pakistan
South Asia
Hindi/Urdu, English, and BST.
Between 1.8 million and 7 million
Unlike ASI. and many of the signed languages of Europe. IPSL does not have Classifier handshapes
Sign
Language
(PSLL
Richard
2018
Table A2: Existing assistive technology for the speech impaired and the deaf and mute
Features
Hand Talk
Assistive
Uni Tablet
MotionSavvy
GnoSys
technology
Category
AAC
AAC
AAC
AAC
Physical
Gloves
Tablet
Tablet App
Phone App
Appearance
Attributes
Sign Speech
Sign Speech, Speech
Speech, Speech
Sign Speech
to Text
to Text
Hardware
Flex sensors, accelerom- eters and gyroscope
Tablet, case, Leap tion sensor
Parts
Software
Anduino technology
Leap motion technology
Leap's 3D motion recog- nition technology
Neural networks computer vision, cloud computing
Cost
Upto Rs. 10,000
Upto 800 USD
20 USD month
USD per day. 4 USD per week and per month
subscription basis
Size
Size the hand
screen size
Fits the tablet screen
the phone screen
Limitations
Size of gloves would vary. works only when the gloves
Bigger size, costly
More storage space re-
Only one way commu- nication is possible, ternet connection is quired
quired, only for tablets
17


Reviews:
[{"review_id": "s381DnF1yvS", "reviewer": "Reviewer_aLeh", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new communication device to \"translate\" from Indian sign language gestures to text and/or voice. Both Indian natural language processing and sign language processing are severely under-resourced and societally significant, hence making this contribution even more substantial.", "summary_of_the_review": "Without addressing research ethics and other concerns related to relevant human subjects more broadly (e.g., the use of recommended terminology and discussion of the broad impact of this study), I can only recommend a strong rejection of this study.", "strength_and_weaknesses": "The paper proposes a new communication device to \"translate\" from Indian sign language gestures to text and/or voice. Both Indian natural language processing and sign language processing are severely under-resourced and societally significant, hence making this contribution even more substantial. \n\nThe paper unfortunately suffers from severe issues in addressing human research ethics. Although it created and used two different datasets including data from human subjects, it did not describe human research ethics at all. I would have expected that obtaining ethical approvals, research permissions, and participants' informed consent to be described in general and in particular because this study focused on a particularly vulnerable cohort of people using sign language.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jguarHLFnpW", "reviewer": "Reviewer_jVQ2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for sign language hand gesture recognition. There is no novelty in the proposed method, and the author only evaluated their method on self-collected data.", "summary_of_the_review": "I think this is a clear reject of the ICLR submission. I did not see the novelty of the proposed method and the evaluation was only done on a small self-collected dataset.", "strength_and_weaknesses": "Strength:\n- Sign language recognition is an important task.\n- The author provided a detailed implementation of their method. I believe the proposed method could be easily re-implemented.\n\nWeakness: \n- There is no novelty in the proposed method. The proposed method uses very basic image processing techniques and very simple contour detection. The method is too naive.\n\n- The author only evaluates their method on a small self-collected dataset. The evaluation dataset is collected from five subjects and only a few class images. There is no evidence showing the generally of the proposed method.\n\n- The paper writing could be significantly improved. There are too many parts of the paper that need to be rewritten. I suggest the author find someone who got accepted a paper in ICLR before to give them professional suggestions in writing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "g9l88yVGVK", "reviewer": "Reviewer_SEnB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a vocalizer that achieves real-time translation of gesture to text/voice using convex hull as the computational geometry.", "summary_of_the_review": "In conclusion, I think the novelty of this paper is insufficient, and the current version still has a lot of room for improvement, including the improvement of the algorithm and sufficient experimental support for the proposed technology. ", "strength_and_weaknesses": "*Strength:\nThe key strengths may lie in the proposed framework URVoice. As the author points out, the URVoice can take the visual / audio as input from the collocutor/ computer and converts it into gesture/ text as output for the signer.\n* Weaknesses:\nFrankly speaking, the title of this paper is very confusing. Why is it called URVoice? “convex hull computation” is simply too broad and vague. Although not explicitly being stated, this work focused on gesture recognition. With that being said, the main technical contributions seem to be the proposed framework to achieve translation of gesture to text/voice. However, with insufficient evidence to demonstrate the effectiveness of the proposed approaches than the state-of-the-art methods, the real impact that this work can bring to the community remains unclear to me.\nAfter reading the whole paper, I strongly feel that the proposed algorithms seem to be trivial and found nothing that excited me. For gesture recognition, there have been many excellent literatures [1,2,3] can be reference.  For the transformation of visual/text signals to speech/visual signals, many multimodal literatures [4,5,6,7] have achieved this goal.\n\n[1] H. Wang, P. Wang, Z. Song, and W. Li, “Large-scale multimodal gesture recognition using heterogeneous networks,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.\n[2] J. Wan, G. Guo, and S. Z. Li, “Explore efficient local features from rgb-d data for one-shot learning gesture recognition,” IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 8,\npp. 1626–1639, 2015\n[3] G. Zhu, L. Zhang, L. Yang, L. Mei, S. A. A. Shah, M. Bennamoun,and P. Shen, “Redundancy and attention in convolutional lstm for gesture recognition,” IEEE transactions on neural networks and learning systems, vol. 31, no. 4, pp. 1323–1335, 2019.\n[4] Jeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Seong Joon Oh, and Hwalsuk Lee. What is wrong with scene text recognition model comparisons? dataset and model analysis. In Proc. ICCV, 2019.\n[5] Zhanzhan Cheng, Yangliu Xu, Fan Bai, Yi Niu, Shiliang Pu, and Shuigeng Zhou. Aon: Towards arbitrarily-oriented text recognition. In Proc. CVPR, 2018.\n[6] Chen-Yu Lee and Simon Osindero. Recursive recurrent nets with attention modeling for OCR in the wild. In Proc. CVPR, 2016.\n[7] Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. Aster: An attentional scene text recognizer with flexible rectification. PAMI, 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UalDyzCnKn4", "reviewer": "Reviewer_eit5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThe paper introduces the URVoice, an Augmentative and Alternative Communication (AAC) device which is supposedly a technological innovation. It is based on the Akl-Toussaint/Graham-Sklansky convex hull algorithms for sign language recognition.", "summary_of_the_review": "Unfortunately, I believe the paper is out of the scope of the conference. I suggest the authors submit their work to another conference or journal focused on technological innovation on such a kind of devices. ", "strength_and_weaknesses": "Main Strengths:\n-The paper is fairly well-written and organized.\n\nMain Weaknesses:\nUnfortunately, I believe the paper is out of the scope of the conference. It is focused on the architecture of a device (the URVoice), which is based on fundamental geometrical and computer vision algorithms. The authors review such fundamental methodology, but they do not present any related state-of-the-art literature on sign language recognition. Finally, the paper does not propose any novel contribution. The experiments are also performed on limited datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject