# Machine Learning-Based Smartphone Grip Posture Image Recognition and Classification
###### Abstract  
Uncomfortable smartphone grip postures resulting from inappropriate user interface design can degrade smartphone usability. This study aims to develop a classification model for smartphone grip postures by detecting the positions of the hand and fingers on smartphones using machine learning techniques. Seventy participants (35 males and 35 females with an average of 38.5 \(\pm\) 12.2 years) with varying hand sizes participated in the smartphone grip posture experiment. The participants performed four tasks (making calls, listening to music, sending text messages, and web browsing) using nine smartphone mock-ups of different sizes, while cameras positioned above and below their hands recorded their usage. A total of 3278 grip posture images were extracted from the recorded videos and were preprocessed using a skin color and hand contour detection model. The grip postures were categorized into seven types, and three models (MobileNetV2, Inception V3, and ResNet-50), along with an ensemble model, were used for classification. The ensemble-based classification model achieved an accuracy of 95.9%, demonstrating higher accuracy than the individual models: MobileNetV2 (90.6%), ResNet-50 (94.2%), and Inception V3 (85.9%). The classification model developed in this study can efficiently analyze grip postures, thereby improving usability in the development of smartphones and other electronic devices. grip posture; smartphone user interface; machine learning; image detection; image classification +
Footnote †: journal: Applied sciences  
0000-0001-5000-0000-0000-0000-0000-0000-00000-0000-00000-0000-00000-00000-00000-00000-00000-00000-000000-000000-000000-000000-0000000-high error rates and low task performance [4; 5]. Although it is acknowledged that using a smartphone with both hands may enhance stability [6], users prefer a one-handed grip for convenience [7]. Therefore, it is necessary to understand one-handed grip posture. For stable and convenient smartphone use, it is necessary to systematically identify and analyze grip postures and, based on these categories, develop a grip posture classification model needed for designing a smartphone UI with proper usability. Different grip postures may be identified by three different methods: (1) manual identification by researchers, (2) use of hardware equipment such as 2D/3D cameras, depth cameras, and gloves, and (3) application of computer vision techniques such as image detection and modeling. First, in manual identification, researchers define grip posture classification criteria and manually categorize the grip postures. [PERSON] et al. (2020) identified smartphone grip postures by placing web cameras above and below participants' hands to record usage tasks [1]. Grip postures were manually classified from the recorded videos based on the number of fingers positioned on each part of the smartphone (L: left, R: right, T: top, B: bottom) and were labeled accordingly. For instance, 3L-1R-1B refers to a grip posture where three fingers are positioned on the left side of the device, one on the right side, and one on the back. Next, the method for identifying grip posture using equipment includes using cameras, such as depth cameras or 3D cameras, and equipping the user's hand with devices like gloves or markers. [PERSON] et al. (2015) used a depth camera to capture images of the user's hand and applied a self-organizing maps (SOM) technique to construct a finger link model for the five fingers (thumb, index, middle, ring, and little) [8]. Using the constructed link model, five different hand postures were identified. [PERSON] et al. (2023) conducted an experiment using the Manus Prime X data glove worn by participants to recognize 56 different hand gestures [9]. The collected 3480 hand gesture data samples were classified using a voting meta-classifier (VL2) model, which ensembles three models: support vector machine (SVM), random forest (RF), and logistic regression (LR). The classification accuracy was reported to be 95.5%.