# FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart Glasses
## 3 FlickPose System  
We employ a method that uses virtual objects. As for the key layout, this study implements FlickPose based on a flicking input layout with a small number of keys, as shown in Figure 1. The lower left key represents the current mode, with representative characters for Japanese, English, and numbers.  
The characters to be input by the character input system created in this study are hiragana, katakana, kanji, English characters, numbers, and symbols. To enable the input of these characters with a key layout based on flick input, the following six major functions are required.  
1. Character input: inputting characters.
2. Delete: deleting the last character in the string being entered.
3. Character conversion: converting lowercase and muddled hiragana, and upper and lowercase alphabetical characters.
4. Word conversion: converting a string of hiragana characters into a string containing katakana or kanji characters.
5. Decision: fixing hiragana without word conversion.
6. Mode switch: switching between Japanese mode and alphanumeric mode.  
Figure 1: Key layout for flick input.  
To implement these functions, existing flick input uses the key layout shown in Figure 1.  
In the key layout for flick input, up to five different characters are assigned to each key for character input. After touching the key assigned to the character to be input, the user selects the character by either lifting the finger from the screen as it is or moving the finger up, down, left, or right, and then releasing it with a flick motion. However, it is difficult for users to perform an accurate flick operation in the air unless they are able to feel the finger. Therefore, instead of flick input, this system uses left-hand poses to select the character on the key; the user inputs the character corresponding to the hand pose. The hand poses used in this system are shown in Figure 2. The hand pose determination method is explained in detail in Section 3.  
When virtual keyboards are used, they are typically placed either near the input field or in the user's hand. Placing the keyboard near the input field raises the user's arm, increasing physical strain. [PERSON] et al. (2019) note that extended pointing at mid-air targets can lead to the \"gorilla arm\" effect, where the arm becomes fatigued. Alternatively, placing the keyboard in the user's hand allows input using only the hand, regardless of the input field's position, but makes it difficult to view the input field while typing. To address this, FlickPose displays a keyboard panel below the input field and an additional UI panel in the user's hand, allowing key selection through the UI panel.  
The UI panel consists of a panel with the same arrangement as the keyboard panel and a panel for all deletions, and keys can be selected by touching the panel corresponding to the key to be selected. The system also highlights the selected UI panel and its corresponding key to allow selection while checking the highlighted text input key without directly checking the UI panel.  
Based on the above, this study implements FlickPose using virtual objects as shown in Figure 3. Figure 3 shows some Japanese characters: The character on the lower right of the Word conversion keys/panel means \"Next\". The character on the Mode switch key/panel represents a typical Japanese character. The character written on the panel means \"All detection\". The keyboard panel and UI panel are displayed directly below the input field and near the user's hand, respectively. The position of virtual objects is shown in Figure 4 when viewed from the side. In this study, each object on the keyboard panel is referred to as a key and each object that makes up the UI panel is referred to as a panel. The hand pose list is placed on the left side of the input column for confirmation of the user's hand pose, and when the user changes the hand pose, the corresponding panel changes to red.  
Figure 2: Left-hand poses used in the text input system.