Continuous-Motion Text Input in Virtual Reality

Text input in virtual reality is a problem that does not currently have a widely accepted standard method. As VR headsets have become more commonplace, text input has also become more important. Using a physical keyboard is not possible with a head-mounted display that blocks the users visual field. The two most popular solutions for text input in VR today are a virtual keyboard interfacing with VR controllers and voice recognition. However, they either require a handheld controller or a quiet environment. 3D-tracked controllers with a virtual keyboard can simulate a real keyboard to an extent, but they suffer from a lack of tactile feedback that makes typing slow and unintuitive. A more intuitive solution is a Swype or SwiftKey-like algorithm, where the path the users finger travels is used as input, as opposed to individually pressing each key. We implemented a prototype for the Oculus Rift with a Leap Motion controller on it that combines a novel continuous-motion text input method with hand gestures to demonstrate an all-purpose, intuitive method of text input. We compare it to state-of-the-art VR keyboard input with a virtual keyboard, as well as a head-directed input method. Introduction The main barrier to finding a comfortable and reliable way of taking in text input is the fact that virtual reality headsets block the users vision. Some headsets are add-ons to existing hardware, so they have limited to no options for text input when they are used in VR mode. This means they might not even come with a controller, choosing instead to rely on gaze selection (looking at a specific spot long enough to select it) along with any buttons that are already on the headset itself. Examples of this are the Samsung Gear VR and the Google Cardboard. Those who do have a pre-existing controller follow the Playstation VRs example. Their solution is to use either the controllers that are already being used for the Playstation 4 or their Playstation Move motion controllers. Their system for typing messages is therefore the same as it is has been for most consoles: with a controller, the user uses the thumbstick (or touchpad) to slide over to the letter they want. With motion controls, the controller itself becomes a cursor in 2D space that the user moves over to the desired letter. Each letter must be individually confirmed with the push of a button. Since these headsets are mainly for playing video games, there is no pressing demand to accommodate for text input, especially when the user has ways to input text on the existing system with their headset off. However, there is a growing demand when it comes to staying connected to ones phone while in VR. Some headsets allow the user to connect their phone through Bluetooth (or the headset display is their phone). The HTC Vive, for example, allows the user to see text messages sent by their contacts and call them with the built-in microphone. If its Vive for Android, the user can also send Quick Replies, which are pre-made messages [21]. Generally, the user is expected to take off their VR headset to type a custom message. For headsets hooked up to a computer, the straightforward solution is to use the keyboard. But again, these headsets completely block the users vision. The Oculus Rift, HTC Vive, and Playstation VR all prefer to use their own controllers because they will always be in the users hand. This mostly removes the issue of having to search around in the dark for the right button once the user becomes familiar with the controllers button layout, and allows them to move around instead of having to sit at their desk. However, there are some independent projects that accepted this constraint and have made prototypes involving a physical keyboard. Previous Work Lachlan Sleight created a 3D replica of their keyboard and used the Leap Motion controller to track where their hands are, so the user can see both their hands and the keyboard through their VR headset [24]. It is an ideal solution when sitting in ones office and it is easy to learn, which will result in a fairly high word-perminute performance compared to the one-letter-at-a-time input of most controllers, but not for general use. In lieu of having to use a physical keyboard, some have opted to use a digital one. A big hurdle that this option has to overcome is the lack of tactile feedback, because there is no tangible controller to assure the user that their input was received. If there is no physical surface to press against, the user can’t intuitively know when they are pressing something. For some, the solution is to create hardware that can simulate that feedback. A group of researchers from Rice University created special gloves that use bladders that expand and contract to simulate the sensation of touch in VR [30]. However, it is still in the prototype phase, so how well it performs with a 3D keyboard remains to be seen. Some have made prototypes that provide ample visual and audio feedback to try to make up for the lack of tactile feedback. Researchers and VR enthusiasts have created a variety of virtual keyboard designs that can interact with hand-tracking sensors (such as the Leap Motion Controller or the Microsoft Kinect): One has the appearance of a regular QWERTY keyboard on a mostly flat surface, with keys that move down as the users fingers press them [5]. Another has the keyboard directly facing the user, with keys lighting up and making a clicking sound when the users finger is near enough to them [31]. Some may also add in a hover check, where the user keeps their finger pressed on a key for a short amount of time to confirm that it is the desired key [12]. These projects do free up the users hands by not requiring a controller, but the lack of tactile feedback and the imperfect tracking mechanisms of the sensors make typing slow and error-prone. There are other ways to type on a virtual keyboard, however. The difficulty in selecting keys partially stems from a depthsensing issue, so a good way to circumvent this is to not rely on depth at all. Some keyboards rely exclusively on the aforementioned hover check. This is how the Myo armband, a motion control device, allows the user to search the web [18]. But there is still a more efficient way that does not force the user to wait for every letter they want to type. The Swype method of typing also does not need depth to work [25]. On a phone, typing with Swype works as follows: the user places their finger on the first letter of the word, then drags their finger to each required letter without stopping. They only lift up their finger once the word is finished. Swype will then guess what word they were typing. If it guesses wrong, multiple autocomplete options would be available to choose from. A phone knows when to start and stop tracking input by sensing when the users finger is touching the screen, but in VR, it could be controlled by a button on the headset or a hand gesture. There have been tests using the Leap Motion with a Swype keyboard [33], and there have been rumors of Swype looking into virtual reality [22], but to our knowledge nobody has implemented this yet. For our prototype, we chose to create a VR application with hand tracking and a 3D keyboard like the examples mentioned above, but we combined it with a Swype-like method of text input to demonstrate how this method works in virtual reality. Hardware and Software Our prototype was created in Unity version 5.5. 1f3 and written in C#. When deciding on the developer environment, we chose Unity over Unreal Engine or Lumberyard for a number of reasons. A large majority of developers use Unity because of how quickly they can create a prototype [10]. Many of the projects mentioned in the Related Work section used Unity to develop their prototypes. While there are many commercially available headsets, we focused on choosing a headset that is widely used and compatible with game-developing programs. The Oculus Rift and HTC Vive are two of the most popular PC-based headsets, and are compatible with a variety of devices. Unity is mainly compatible with the Oculus Rift, Gear VR, the HTC Vive, and the HoloLens, though it continues to add support for more types of headsets [26]. In this project we use an Oculus DK2 head-mounted display. As for a hand-tracking device, the two main competitors in use are the Leap Motion Controller and the second-generation Kinect (because the first-generation Kinect cannot keep track of individual fingers). Both are compatible with the Oculus Rift, but the library the Leap Motion Controller comes with is much easier to work with when it comes to tracking individual fingers, while the Kinect V2 requires finding a third-party library or algorithm [Pte16]. The camera placement is also more convenient since the Leap can be placed directly onto the headset, so we ultimately chose the Leap Motion Controller, along with Leap Motion SDK version 2.3. 1. We developed on a PC with a Windows 10 operating system. It is the recommended OS by Oculus because they have stopped development on OS X and Linux for the time being [3]. The Leap Motion Controller SDK is also only available for Windows and Android [16]. Backend For the Swype-like algorithm, Swype, Swiftkey, and Google Keyboard have no publicly available source code online, though Android phones automatically have a Swype keyboard. However, the Android keyboard has a limited API to work with that is targeted towards touchscreens, and it is quite difficult to interact with through scripting, so we decided against directly using this keyboard and instead made our own keyboard with Unity. There are many alternative open source projects available online, but most if not all lack the complexity that the commercial apps have under the hood, and our own backend would be easier to debug and build upon. For our prototype, we focused on creating an intuitive, responsive keyboard and determining effective criteria for filtering u