Comparative framework for vision‐based gesturing modes and implementation of robust colour‐marker detector for practical environments

IET Image ProcessingVolume 13, Issue 9 p. 1460-1469 Research ArticleFree Access Comparative framework for vision-based gesturing modes and implementation of robust colour-marker detector for practical environments Songhita Misra, Corresponding Author Songhita Misra msonghita@gmail.com ECE Department, Speech and Image Processing Lab, NIT Silchar, Silchar, 788010 Assam, IndiaSearch for more papers by this authorRabul Hussain Laskar, Rabul Hussain Laskar ECE Department, Speech and Image Processing Lab, NIT Silchar, Silchar, 788010 Assam, IndiaSearch for more papers by this author Songhita Misra, Corresponding Author Songhita Misra msonghita@gmail.com ECE Department, Speech and Image Processing Lab, NIT Silchar, Silchar, 788010 Assam, IndiaSearch for more papers by this authorRabul Hussain Laskar, Rabul Hussain Laskar ECE Department, Speech and Image Processing Lab, NIT Silchar, Silchar, 788010 Assam, IndiaSearch for more papers by this author First published: 20 June 2019 https://doi.org/10.1049/iet-ipr.2018.5978Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In this study, the authors have provided a detailed analysis based on experimental analysis, user-feedback, and the literature survey to choose the appropriate gesturing mode between colour-markers and bare-hand for virtual dynamic gesturing system. Salient factors such as lexicon duration, muscle strain, target users, detection/tracking accuracy and so on, indicated that colour-marker based models can outperform bare-hand systems in real-world scenarios. Colour-markers are more robust to factors such as uneven illumination, affine transformations, and holds a better similarity to the natural way of writing on paper. However, colour-markers can easily be erroneously detected/tracked due to the presence of imposters in the background. Most of the existing colour-marker systems are developed with this limitation that reduces the naturalness and confines its application to a narrowed environment. This study addresses multiple types of frequently occurring imposters which are static/dynamic in nature and occlude the genuine colour-marker often times. The imposters are distinguished based on factors such as speed, start/end position, the area of motion and so on. Some uncertain types of imposters which are partly similar to genuine gestures are distinguished based on the randomness present in the trajectories and classification models. The proposed techniques have achieved an accuracy of 98.26% in distinguishing the given imposter categories in this system. 1 Introduction Gesture-based computer applications can ease or replace many traditional devices such as the mouse, keyboard, and electrical remotes, without the need for any physical contact or sometimes even without moving from one's location. Such kind of system can be highly beneficial for assisted living applications [1], intelligent wheelchair [2, 3], users with motion-impaired conditions [4], elderly people [5, 6] and so on. However, a study by Wachs et al. [7] briefed that some aspects such as lexicon size, gesturing device, ease of gesturing, target user and so on may influence the application of such gesture-based system in practical applications. Since the ultimate goal is the implementation of such systems in real-world, all the influential factors need to be addressed before developing the systems. Very few existing studies have pondered upon these factors, whereas other systems have displayed high accuracy in a very restricted and controlled environmental condition. Parameters of the gesture-based systems are application dependent which further depends on the need and convenience of the target users. For example, target users of a video game are generally young people between the ages of 10 and 30 years approximately, while target users of sign language systems are people with speaking and hearing disabilities with no age bound. The choice of the lexicon, gesturing device, and other parameters will greatly vary in these two gesture systems. As a result, it is necessary to outline all the possible constraints for which a particular system can be used. In the literature, gesturing devices can be broadly categorised into contact based and vision based devices [8]. The accuracy of existing contact based devices is high and yet cannot be universally applied due to their need for a cumbersome wiring setup. Contact-based systems reduce the naturalness of the system and need the user to know how to use it in prior. Nevertheless, they are employed in systems where target users are niche and priority for accuracy is higher than convenience. For example, they can be employed in health centres where a doctor can examine or a surgeon can perform minor operations from a faraway place [9]. In the cases where naturalness is preferred over cumbersome wiring, the vision-based system plays a vital role. Vision-based systems can be broadly classified into 2D and 3D based processing of gestures. 3D systems hold the advantage of extracting the depth information, making it a lot easier to detect the structural variations of hand. For sign language systems, where various hand articulations convey the gesture information, depth features can ease the system complexity to a lot of extents. Conversely, dynamic systems where hand posture is same and gesture are made in spatiotemporal axis, the depth information of hand plays no significant role in the recognition of gestures. In such cases, 2D cameras can be used to conveniently detect and track the dynamic gestures at a much cheaper cost of production. In this study, we have focused on the development of a dynamic virtual keyboard system similar to existing systems [10, 11]. Dynamic systems are lengthy and involve detection and tracking phase prior to gesture recognition. Therefore, multiple factors such as detection, tracking, accuracy, muscle strain, lexicon size, interaction space, influence the choice of gesturing device. Users not only prefer a highly accurate system, but also give more priority to the easiness of the system. The thorough survey, as discussed in later sections, on technical/non-technical factors and users-feedback suggest that colour-markers can be a more appropriate choice for the realisation of such systems in the real world as compared to bare-hand. However, a frequently occurring real-world scenario is the presence of spurious colour-markers in the background of the gesture video which may significantly degrade the accuracy of a colour-marker system. These practical conditions are unavoidable and yet none of the existing colour-marker systems considered these issues in their system. In this study, multiple types of spurious cases are addressed from the real-world scenarios. The main contributions of this work are given as follows: Comprehensive analysis between colour-marker and bare-hand gesturing mode based on technical/non-technical factors, experimental analysis, literature survey, and user feedback. The proposed system focuses on the deployment of the system which will be beneficial to users with different disabilities. Therefore, user-feedback is measured from users such as elderly people, people with partial vision, partial motion-impaired hand, and common users. Multiple types of spurious cases are addressed which may degrade the accuracy of a real-world colour-marker system. The spurious objects may be static or dynamic in nature and may occlude the genuine gesture while the user is gesturing. Along with velocity, position, and other trajectory features, some uncertain and complex spurious cases are distinguished based on pattern analysis and machine learning tools. The remaining part of the paper is organised as follows. Section 2 presents a comprehensive analysis between colour-marker and bare-hand mode of gesturing. Section 3 addresses the possible spurious cases in colour-marker systems and proposed technical approaches to reduce the effect of each such cases. Section 4 describes the database used in this study. Section 5 gives the experimental results and observations. Section 4 concludes the proposed study with the future scope of research it opens up. 2 Colour-marker versus bare-hand: a comprehensive analysis In this study, a red-marker band placed on the index finger is used to develop the colour-markersystem, whereas the fist is used for gesturing in the bare-hand system. For userfeedback, 60 users of four broad classes are asked to perform the gestures. The fourtypes of users are User type-A (UT-A): 20 users with partial vision areasked to provide feedback in this study. The major causes of partialvision are uncorrected refractive errors, cataracts, glaucoma,age-related macular degeneration, and so on. User type-B (UT-B): 20 users between age group 60 and 75years are asked to perform the gestures. User type-C (UT-C): 20 users with partial hand strengthare asked to provide the feedback. The hand grip, finger and wristmovements may be affected by nerve damage and diseases. Due to damage insensory fibres such users experience abnormal sensations such astingling, numbness or pain along with movement disorders. Some majorsyndromes and disorders are Carpal Tunnel Syndrome, Brachial PlexusInjury/Disorder, Radial Neuropathy, Ulnar Neuropathy, Median NervePalsy, and so on. User type-D (UT-D): 20 common users with no physical andage-related disabilities are asked to give the user feedback. The comprehensive analyses based on the different influential factors are asfollows: Responsiveness: It is the measure of how quickly thesystem detects the device and tracks it to produce the gesture. Most ofthe existing bare-hand system relies on skin-based detection underuniform or simple background conditions. Some researchers who developedfeature based bare-hand detection models are Misra etal. [12], Mittleet al. [13],and so on. Detection of red marker mostly uses colour-cue informationand centroid information for tracking. Detection and tracking ofbare-hand is a challenging task and is affected by many factors such asposition variation, affine transformation, uneven illumination, complexbackground, and so on. The effects of these factors in accuracy arediscussed in a later section. By realising the existing systems [10, 11], we evaluated the detection and tracking time of each ofthe 58 gestures for 50 samples/gesture, as shown in Table 1. NITSHGD database used for thismeasure and is available online at http://joyeetasingha26.wixsite.com/nits-database. Experimentsare carried out on a Dell Inc., Precision T1700, Intel(R) Core(TM)i5–4590 CPU@3.30 GHz. Table 1. Detection and tracking time of colour-marker and bare-hand on 2900 videosfrom NITSHGD database Colour-marker system[10] for bothdetection and tracking (in CPU time) Bare-hand system [11] for both detection andtracking (in CPU time) Bare-hand system [14] for detection (in CPUtime) responsiveness 8.845 45.71 40.52 The average gesture duration of 2900 videos in the colour-marker systemis observed to be 5.46 s CPU time as compared to 6.71 s CPU time of thebare-hand system. It can be observed from Table 1 that even if the gesture duration betweencolour-marker and bare-hand is small, the detection and tracking timedifference is high between the two systems. User adaptability: User adaptability of the devicedepends on how easily a person can use the device to make the gesture.For example, moving a fist to make a gesture will be more difficultlyadapted by the user as compared to moving a finger to make the samegesture. Since the range of camera view is limited, it puts an extraconstraint on the user to make the gesture within the given space. Wehave asked the users to perform 50 colour-marker and bare-hand gesturesfrom NITSHGD database. As per UT-A, it is difficult to perform withinthe camera range using bare-hand as compared to colour-marker. As perUT-B due to lesser muscle strain, they find it easier to perform thegestures using colour-markers. Since bare-hand requires full movement ofhand as compared to colour-marker gestures which can be performed by themere movement of the index finger, elderly people can adapt thecolour-markers more easily. UT-C could hardly make any gesture correctlyusing bare-hand, which was possible using colour-markers. UT-D assertedthat even if they have no physical issues, they found it easier to makegestures using colour-markers than bare-hand. The longer the gesture is,the more the user tends to forget it. Learnability: Due to high muscle drive in the fistmovement, users often tend to get tired while performing long-durationdynamic gestures as compared to colour-marker systems. Also, the averagesize of the colour-markers [10,15] is 70.11 × 86.15 pixels(3 × 1.5 cm) in a 780 × 1280 pixel image, as compared to 256.1 × 186.2pixels of bare-hand [11]. In[16], an open palm postureposition of bare-hand is used to make the gestures which are almosttwice the size of a fist as in [11]. In comparison of their approximate size, it has beenobserved that bare-hand gestures require almost 7.89 times more in-airspace area than colour-marker gestures. More the in-air space requiredhigher is the hand movement and time required to complete the gesture.UT-A, UT-B, and UT-C asserted that due to the longer time required, theyoften tend to forget the remaining section of the gesture to beperformed. Accuracy (detection, tracking, and recognition): Whilethe other factors are not technically challenging, the choice of theinput device can severely affect the accuracy of the gesture systems andis a decisive factor. In bare-hand detection, the hand needs to bedetected from a complex background in practical environments. This isperformed using skin-filtering methods [16], fused motion detection methods [11], or feature-based methods [14]. Skin-filtering methods areaccurate and fast under simple background conditions. However, anyslight variation in illumination, appearance, or complex background candrastically degrade the accuracy of detection. The study by Singhaet al. [11],both skin-filtering and motion detection method are integrated and yetachieved an accuracy of 47.9% under noisy conditions. Feature-basedmodels have achieved an accuracy of 93.8% at the expense of hightime-complexity [13, 14]. Detection of colour-markers in[10] can achieve an accuracyof 100% under controlled environmental conditions. The authors drew thelimitation that no spurious colour-marker should appear in thebackground or foreground of the gesture video except the genuinecolour-marker. However, colour-markers are least affected by affinetransformations, uneven illumination, and can be modelled into arealisable system under non-ideal conditions. The next challenge indynamic gesture system is tracking of the device in subsequent frames toobtain the final trajectory path. Thus the recognition accuracy ofgestures mainly depends on the accurate tracking of gestures. Trackingof colour-marker in [10, 15], is simply performed bydetecting colour-marker in subsequent frames and connecting thecentroids to obtain the final trajectory. It is faster and easier touse. Other tracking methods for colour-marker such as [17, 18] requires much higher processing time. However, trackingof bare-hand is a complex subject. Since detection of bare-hand is acomplex process in itself, detection in each frame and subsequentconnection of centroids will increase the tracking time enormously.Modified KLT tracker and camShift methods are used in [15] for tracking which is faster,accurate, and less complex as compared to Kalman filter method [19], and some other trackingmethods [18, 20]. Earlier it is observed thatthe average detection and tracking time of bare-hand is 45.71 s CPU timeapproximately, as compared to 8.83 s CPU time colour-marker. The choiceof the gesturing device also affects the recognition accuracy of agesture system. It is noted from studies [10, 11,15, 16], that the accuracy of the bare-hand system islower than the colour-marker system for the same set of features. Theundesirable factors which lead to lower accuracy in bare-hand are asfollows: The lower number of trajectory points is observed inbare-hand as compared to the colour-marker system. Let thegesture ‘A’ is made by a user in 7 s CPU time usingcolour-marker and 10 s CPU time using the bare-hand. Afterremoval of redundant trajectory points, the density oftrajectory points in colour-marker remains higher thanbare-hand gestures. Due to the larger fist size, the distance between fist intwo-consecutive frames is much scarcer as compared tocolour-markers. Therefore, the geometrical features inbare-hand are extracted from the lower number of trajectorypoints, eventually generating indistinguishable values formost of the gestures. Due to a lower number of trajectory points, gesturesmoothening on a trembled bare-hand trajectory may lead torejection of some vital information. Fig. 1 shows that the handtrembling is more prominent in bare-hand as compared to thecolour-marker system. The higher density of trajectorypoints in colour-markers makes the smoothening process moreaccurate. A study by Singha et al. [11] showed that withthe increase of noises in the gesture trajectories, theaccuracy of the system can degrade exponentially. Fig. 1Open in figure viewerPowerPoint comparison between hand trajectories made by color-marker andbare-hand (a) Trajectories of gesture ‘A’ bybare-hand, (b) Trajectories by gesture ‘A’ by colour-marker Low mental load: The mental load on users depends on howtough or tiring it is to use the device to make the gesture.Contact-based devices put a high mental load on the users since thewiring of the device needs to be checked continuously while making thegesture. In case of bare-hand gestures, though no wiring is required,the challenge is to perform the gesture within the camera view and alsoadapting to the pain or discomfort due to higher muscle movements.Colour-markers need small in-air space and does not pertain to highmuscle movement. Intuitiveness: It is the measure of how easily agesturing mode comes naturally to the user. Lexicons of a dynamicgesture system should be chosen based on the easiness of the gesturingmode. In this study, we have been considering the lexicons of existingstudies in [10, 11]. If the making of a gesture iscompared with writing on a piece of paper, colour-markers hold bettersimilarity than the bare-hand system. In colour-marker system, the indexfinger is used in place of a pen for making a gesture. Since users arehabituated with writing on paper, use of colour-marker to make in-airgesture is more intuitive to users as compared to bare-hand. Some usersof UT-A, UT-B and UT-C find it difficult to hold the pen to write onpaper due to partial vision, fatigue due to old age, and lower strengthin hand muscles. They found it easier to make in-air gestures usingtheir index finger. Users asserted that bare-hand is more comfortable tomake short dynamic gestures such as moving up or down, left or right,and so on. Higher lexicon duration of bare-hand gives high muscle strainto a user which is not acceptable in day-to-day applications. Comfort: This factor involves all the above factors suchas intuitiveness, low metal load, user adaptability, and learnability.For all these factors, colour-marker stands as a better choice foruser's ease. Lexicon size and multi-hand systems: The 58 charactersin NITSHGD database are subdivided into categories based on theirduration. Gestures such as ‘1’, ‘-’,’/‘, ‘∼’,‘C’, ‘7’, and so on,require 1–3 s CPU time. Gestures such as ‘2’,‘3’, ‘B’, ‘D’, ‘P’, ‘S’,‘T’, and so on, require 4–7 s CPU time, and gestures such as ‘A’, E',‘H’, ‘I’, ‘K’, ‘@’, require more than 7 s. It has been observed thatuser types A, B, and C were not been able to correctly make thebare-hand gestures with a duration higher than 4 s. However, they wereable to perform longer gestures in colour-markers more easily. User typeD also cribbed about an ache in hand muscles once the gesture durationof bare-hand gets higher than 6 s CPU time. Now consider gesticulating using both hands. Static gestures can beconveniently made using both the hands. Conversely, when two hands areneeded to be moved simultaneously in a dynamic gesture, highconcentration and long-time practice is a prerequisite to performing thegesture. As per user-feedback, once the gesture duration gets longerthan 2–3 s CPU time, the users tend to get jumbled up and chaotic in themid-way and fails to complete the gesture. The same users have been ableto make short dynamic gestures in game applications using the two handssimultaneously. The short dynamic gestures usually include, ‘move up’,‘move down’, ‘speed up’, ‘speed down’, ‘move left’, ‘move right’, etc.With one hand the users made the speed control gesture and with anotherhand, they controlled the direction. In such cases, the use of bare-handis preferred more than colour-markers. However, for long-durationgestures the users have chosen to use colour markers. In the literature,the use of two bare-hands has only been made in sign language systems orfor gaming applications. Therefore, the use of two bare-hands inlong-duration gesture systems is quite imaginary from the applicationpoint-of-view. Come as you are: A virtual gesture system in areal-world application is more preferable if it puts no extra effort onthe user-end. Colour-markers systems require the user to attach thecolour band on its index finger before making the gesture. However,bare-hand is the most natural and convenient way of making a gesturesince humans use hand even for day-to-day interactions or easy conveyingof information. Reconfigurability: The lexicons types, gesturing device,and so on, depend on the target user and its application. Forapplication in medical units [9],sensors or contact based systems with high precision is desirable. Suchsystems are mostly not portable and need the developer to arrange thesetup in advance with all the tunable parameters at suitable standards.However, systems such as home appliances, virtual password basedverification systems, users prefer a contactless gesture system withleast arrangement from the user-end. Colour-marker and bare-hand systemseasily fit in the requirement of such systems. However, the ease ofvision-based systems comes at the expense of multiple factors which needto be reconfigured by the developer. For example, detection of bare-handis affected by affine transformations, uneven illumination, occlusion,and complex background conditions. Detection of colour-marker may beaffected by the presence of spurious colour-markers in the background.Between colour-marker and bare-hand detection, it is additionallychallenging to reconfigure the parameters of the bare-hand system undernoisy conditions owing to the higher volumetric area of the fist ascompared to colour-marker. Interaction space: More the distance between thegesturing device and the camera, the higher is the space required by theuser to make a gesture. Also, the smaller the gesturing device, thelesser the time and space it needs. Due to the smaller size ofcolour-markers, it can explore more in-air space at a much lessertime. Gesture spotting: In the case of small dynamic gesturesin video games, gesture spotting is the detection of genuine strokesamidst the presence of any unintended hand movements or spurious objectsin the background. However, gesture spotting in the context of thisstudy is the detection of natural yet unintended strokes made by theuser within the gesture itself. These strokes are made to move from onepoint to another to complete the gesture. In studies of colour-marker[10], and bare-hand system[11], these strokes areaddressed as self-coarticulation strokes which are discarded based onits velocity as compared to the actual stroke of the gesture. As notedfrom experimental observation, if the gesturing by bare-hand is too fastduring the self-coarticulation stroke, the distance covered by fist intwo consecutive frames is large enough to let the self-coarticulation gounrecognised. Though self-coarticulations are not an actual part of thegesture, they are vital and failure of their recognition may lead toloss of key information about the gesture. Therefore, the user needs tolook after the speed of self-coarticulation stroke in bare-hand, whichis not obligatory in colour-marker systems. Colour-markers are smallerin size and covers smaller distances in two consecutive frames even athigh gesturing speed. As a result, gesture spotting in a colour-markersystem is more convenient and does not put any extra mental load onusers regarding the self-coarticulation stroke. Similarity with writing a note on a piece of paper(easiness): Consider a person writing ‘GESTURE’ on a pieceof paper. If we ask him to write the same word in-air, he will naturallyuse his index finger to make the gesture. A colour-marker attached tohis index finger will generate a gesture that can be recognised bycomputer vision techniques. If we ask him to write a bigger sentencelike ‘I LOVE TO MAKE GESTURES’, he will naturally prefer to write on apaper rather than making the in-air gesture, owing to a lesser musclestrain in hand. However, if he practices the method of making the in-airgesture for a number of days, this will eventually become more intuitiveand stress-free for him. Let's remember the fact that for small childrenit is equally painful to write notes on a paper. It is from years ofpractice that humans finally get habituated with the art of writing. Onthe other hand, if bare-hand is used to make an in-air sentence, thehand muscle strain will not only increase by many folds but may alsotake more time to complete it, accompanied by other difficulties. Hence,colour-markers can be more easily realised to replicate the genuinenessof writing on a piece of paper. Indoor versus outdoor: Unlike indoor environments, auser has almost no control over the occurrences of the outdoorenvironment. Both bare-hand and colour-marker detection and trackingprocess can be severely affected by various non-ideal conditions. Sincethe tracking process of colour-markers is not appearance-based, they canbe more easily handled in non-ideal conditions, which is not true forbare-hand systems. Any change in the appearance due to unevenillumination, affine transformations, and complex background can easilyaffect the detection and tracking accuracy of bare-hand. Consequently,deployment of a robust bare-hand detector for the outdoor surroundingsis a computationally expensive task. 3 Practical issues faced by colour-marker systems As mentioned above, detection and tracking of colour-marker can achieve an accuracy of 100% accuracy under controlled environmental conditions. Existing colour-markers systems in [10, 15, 21-23] have clearly stated that the restriction that no imposters should appear in the gesture video. This limitation makes the system unusable in most applications. The imposter object may be static or dynamic in nature. They may also be complex in nature and occlude the genuine colour-marker. Since tracking is performed by detection in each frame, any imposter of area higher than a given threshold will be automatically discarded by the system. The different types of static and dynamic imposters addressed in this study are: Non-occluded static (case-I): This is a frequently occurring non-ideal condition where a person may choose to make a gesture with static spurious objects in the background. For example, a person may make a gesture with a red-colour marker while sitting on a red chair, or with a red vase in the background. Consider a situation where a person is making a gesture while the traffic control light is appearing red in the background of the video. In all these cases, the person consciously makes the gesture without overlapping the spurious object in the background. Since the spurious red-object will have no velocity and no change of position as compared to actual gesture, they can be easily detected and discarded based on a threshold value. This is applicable for any number of static and non-overlapping spurious object in the background. Occluded static (case-II): In a similar case as above, it may happen that user unconsciously overlaps the marker with the spurious red-colour object in the background. In this case, multiple imposter trajectories will generate from the point of intersection as shown in Fig. 2b. The number of spurious trajectories generated depends on the number of occlusions made. The imposter trajectory, in this case, can be easily detected based on its length of duration. It should be noted that till the occurrence of occlusion the imposter trajectory is static in the video and will bear the same coordinate values until that point. These coordinate values can be merged into one value till that point and the spurious trajectory can be discarded based on its final length. However, consider a case where the overlapping happened at the start of the gesture. The spurious trajectory will follow the same path as genuine trajectory after the point of occlusion. In these cases, the length of the spurious trajectory may be the same, equal, or slightly higher than genuine trajectory by some trajectory points. Such trajectories can be averaged into one based on the propos