3D interaction input for text in augmented reality

A method, computer system, and computer program product for implementing three-dimensional text input in an augmented reality system are provided. The embodiment may include capturing a first user hand position along a first axis of a three-dimensional virtual space. The embodiment may also include identifying a first character along the first axis corresponding to the first user hand position. The embodiment may further include capturing a second user hand position along a second axis of the three-dimensional virtual space. The embodiment may also include identifying a second character along the second axis corresponding to the second user hand position. The embodiment may further include identifying one or more proposed words beginning, consecutively, with the identified first character and the identified second character using a dictionary database. The embodiment may also include displaying the one or more identified proposed words on a third axis of the three-dimensional virtual space.

BACKGROUND

The present invention relates, generally, to the field of computing, and more particularly to augmented reality.

Augmented reality relates to technology that modifies a direct or indirect user view of a real-world environment with computer-generated elements using various inputs, such as sound data, image data, or location data. Various technologies may be implemented when utilizing augmented reality, such as eyeglasses, head-mounted displays, head-up displays, contact lenses, virtual reality displays, and handheld displays. Augmented reality may have numerous applications within society including uses in literature, architecture, visual art, education, emergency management, video gaming, medicine, military, navigation, tourism, language translation, and music production.

SUMMARY

According to one embodiment, a method, computer system, and computer program product for implementing three-dimensional text input in an augmented reality system are provided. The embodiment may include capturing a first user hand position along a first axis of a three-dimensional virtual space. The embodiment may also include identifying a first character along the first axis corresponding to the first user hand position. The embodiment may further include capturing a second user hand position along a second axis of the three-dimensional virtual space. The embodiment may also include identifying a second character along the second axis corresponding to the second user hand position. The embodiment may further include identifying one or more proposed words beginning, consecutively, with the identified first character and the identified second character using a dictionary database. The embodiment may also include displaying the one or more identified proposed words on a third axis of the three-dimensional virtual space.

DETAILED DESCRIPTION

Embodiments of the present invention relate to the field of computing, and more particularly to augmented reality. The following described exemplary embodiments provide a system, method, and program product to, among other things, implement a three-dimensional space for the input of text using an augmented reality system. Therefore, the present embodiment has the capacity to improve the technical field of augmented reality by improving character and word selection accuracy when a user is interacting in a three-dimensional, augmented reality space.

As previously described, augmented reality relates to technology that modifies a direct or indirect user view of a real-world environment with computer-generated elements using various inputs, such as sound data, image data, or location data. Various technologies may be implemented when utilizing augmented reality, such as eyeglasses, head-mounted displays, head-up displays, contact lenses, virtual reality displays, and handheld displays. Augmented reality may have numerous applications within society including uses in literature, architecture, visual art, education, emergency management, video gaming, medicine, military, navigation, tourism, language translation, and music production.

Augmented reality and virtual reality (AR/VR) systems are becoming ever present in the current technological landscape. These technologies allow for new and more dynamic interactions between humans and computers. Utilizing an interaction space, AR/VR systems are able to identify user locations via sensors placed around the user's body. For example, the user may wear gloves with embedded sensors to allow an AR/VR system to identify the location of the user's hands in a virtual space. However, many AR/VR systems have difficulty in identifying the exact location within the three-dimensional space since many computers may have difficulty in recognizing three-dimensional movement of a user's hands and fingers versus identifying user movements in a two-dimensional space. This difficulty is ever present for the process of text input where the common method of using AR/VR devices is more likely to result in mishandling failures since may characters exist on a virtual QWERTY keyboard and each character on the keyboard occupies a small amount of space. As such, it may be advantageous to, among other things, implement a system that recognizes user movements in a three-dimensional augmented reality space as various movements in one or two dimensions and predicting subsequent user movements based on the current and previous user selected characters or words.

According to one embodiment, a three-dimensional virtual keyboard may be utilized to capture user movements when selecting characters. Since augmented reality systems may be more accurate when capturing movements in a one or two dimensional space, user movements along one axis of the three-dimensional virtual space may be initially monitored to capture the first character of a word. For example, the user may select the first character from a list of character displayed along the x-axis of a virtual space. The second character of a word may be captured by following subsequent user movements in the three-dimensional virtual space along a perpendicular axis. For example, the second character may be selected by the user from a list of characters along the z-axis of a virtual space. Using a dictionary database, common words beginning with the first and second characters may be displayed to the user along the third axis perpendicular to the x-z plane (e.g., y-axis). The user may then be capable of selecting the desired word from the displayed proposed words.

The following described exemplary embodiments provide a system, method, and program product to utilize a virtual keyword to capture user movements in one or two dimensions at a time when determining user input text. Furthermore, proposed words may be displayed to and selected by the user on the virtual axis perpendicular to the plane created by the axes used to select the characters of the user-desired words.

Referring toFIG.1, an exemplary networked computer environment100is depicted, according to at least one embodiment. The networked computer environment100may include client computing device102and a server112interconnected via a communication network114. According to at least one implementation, the networked computer environment100may include a plurality of client computing devices102and servers112of which only one of each is shown for illustrative brevity.

Client computing device102may include a processor104and a data storage device106that is enabled to host and run a software program108and an augmented reality text input program110A and communicate with the server112via the communication network114, in accordance with one embodiment of the invention. Client computing device102may be, for example, an augmented reality device, a virtual reality device, a mobile device, a telephone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running a program and accessing a network. As will be discussed with reference toFIG.3, the client computing device102may include internal components302aand external components304a, respectively.

The server computer112may be a laptop computer, netbook computer, personal computer (PC), a desktop computer, or any programmable electronic device or any network of programmable electronic devices capable of hosting and running an augmented reality text input program110B and a database116, such as a dictionary database, and communicating with the client computing device102via the communication network114, in accordance with embodiments of the invention. As will be discussed with reference toFIG.3, the server computer112may include internal components302band external components304b, respectively. The server112may also operate in a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server112may also be located in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud.

According to the present embodiment, the augmented reality text input program110A,110B may be a program capable of capturing user movements while interacting with a three-dimensional keyboard spread across an axis. Upon receiving user designations of characters, the augmented reality text input program110A,110B may display proposed words across another axis displayed within the three-dimensional virtual space to allow the user to select the desired word. The augmented reality text input program110A,110B may be utilized in various languages, including Chinese, English, Spanish, and German. The augmented reality text input method is explained in further detail below with respect toFIG.2.

FIG.2is an operational flowchart illustrating an augmented reality text input process200according to at least one embodiment. At202, the augmented reality text input program110A,110B captures a first user hand position along a first axis of a three-dimensional virtual space. The augmented reality text input program110A,110B may display a three-dimension keyboard to a user utilizing an augmented reality or virtual reality headset. The three dimension keyboard may be a multi-axis graph with alphanumeric characters along each axis. For example, the three dimension keyboard may have an x-axis, a y-axis, and a z-axis where the x-axis and z-axis have evenly spaced markers with alphanumeric characters, such as the letter “A” through the letter “Z”, displayed. In at least one embodiment, the three-dimensional keyboard may have three axes but is not specifically limited to only three axes in other embodiments as any number of axes may be utilized.

As a user navigates the three-dimensional keyboard, the augmented reality text input program110A,110B may capture position data from sensors corresponding to the augmented reality or virtual reality system that translate to the user's hand or arm positions. The position data may represent the user engaging with the three-dimensional keyboard to select a character represented along an axis of the keyboard that begins a word the user wishes to input to the client computing device102. For example, if the user wishes to input the word “BANANA” using the three-dimensional keyboard. The user may interact with the virtual space of an augmented reality system to move along the x-axis until the user's hand position is aligned with the letter “B”. The augmented reality text input program110A,110B may capture the coordinates within the virtual space that correspond to the user's hand position with the letter “B”. In at least one embodiment, the augmented reality text input program110A,110B may determine that the user has reached the desired location on the three-dimensional virtual space by either detecting the user's hand has stayed over the position for a preconfigured period of time, the user has performed a preconfigured hand movement (e.g., finger pinch), or the user has interacted with the augmented reality or virtual reality system in some manner (e.g., button press).

Then, at204, the augmented reality text input program110A,110B identifies a first character along the first axis corresponding to the first user hand position. Once the augmented reality text input program110A,110B captures the user hand position data within the virtual space of an augmented reality or a virtual reality system, the augmented reality text input program110A,110B may then relate the captured hand position data to a specific character corresponding to the user hand position on an axis of the three-dimensional keyboard used to select the first letter of a word the user wishes to input. For example, in the previous example where the user wishes to spell the word “BANANA” and the x-axis is designated as the axis on which the user navigates along to select a first letter of a word, the user hand position data may be analyzed to determine that the user hand position corresponds to the position of the letter “B” on the x-axis. Therefore, the augmented reality text input program110A,110B may identify the letter “B” as the first character of the word the user wishes to input.

Next, at206, the augmented reality text input program110A,110B captures a second hand position along a second axis of the three-dimensional virtual space. Once the augmented reality text input program110A,110B identifies the first character, the augmented reality text input program110A,110B may allow the user to navigate along a second axis on the three-dimensional keyboard to a second location that corresponds to a character on the second axis corresponding to the second letter of the word the user wishes to input. For example, in the previous example where the user is spelling the word “BANANA”, the user may change hand position from the letter “B” represented along the x-axis to the letter “A” represented along the z-axis. Once the augmented reality text input program110A,110B determines the user hand position has reached the desired second location (e.g., the user hand movement has ceased), the augmented reality text input program110A,110B may capture the coordinates of the user hand position within the three-dimensional virtual space of the augmented reality or virtual reality system.

Then, at208, the augmented reality text input program110A,110B identifies a second character along the second axis corresponding to the second hand position. Substantially similar to step204, once the augmented reality text input program110A,110B captures the user hand position data within the virtual space of an augmented reality or a virtual reality system, the augmented reality text input program110A,110B may then relate the captured hand position data to a specific character corresponding to the user hand position on an axis of the three-dimensional keyboard used to select the first letter of a word the user wishes to input. For example, in the previous example where the user wishes to spell the word “BANANA” and the z-axis is designated as the axis on which the user navigates along to select a second letter of a word, the user hand position data may be analyzed to determine that the user hand position corresponds to the position of the letter “A” on the z-axis. Therefore, the augmented reality text input program110A,110B may identify the letter “B” as the first character and the letter “A” as the second character of the word the user wishes to input.

Next, at210, the augmented reality text input program110A,110B identifies one or more proposed words based on the first and second characters using a dictionary database. The augmented reality text input program110A,110B may have access to or include a dictionary database, such as database116, that includes words within a user designated language. For example, if the user designated the English language as the user's preferred language, the dictionary database may include all known words within the English language. Once the first two characters of the user-desired word are identified in steps204and208, the augmented reality text input program110A,110B may query the dictionary database to identify words within the user designated language that begin with the characters. For example, in the previous example where the user is spelling the word “BANANA” and has input the letter “B” as the first character and the letter “A” as the second character, the augmented reality text input program110A,110B may query the dictionary database to identify words within the English language that begin with the letter sequence “BA”. Once the dictionary database has returned results to the query, the augmented reality text input program110A,110B may sort the returned results in order of commonality within the preconfigured language. For example, if the dictionary database returned the results of “BATCH”, “BACKPEDAL”, and “BANANA”, the augmented reality text input program110A,110B may rank the words in the order of “BANANA”, “BATCH”, and “BACKPEDAL” since the word “BANANA” is more commonly used in the English language than the word “BATCH” and the word “BATCH” is more commonly used than the word “BACKPEDAL”.

Then, at212, the augmented reality text input program110A,110B displays the one or more proposed words on a third axis of the three-dimensional virtual space. Once the augmented reality text input program110A,110B has identified the proposed words from the dictionary database, the augmented reality text input program110A,110B may display the words along the y-axis to allow the user to select the desired word. For example, the words “BANANA”, “BATCH”, and “BACKPEDAL” may be displayed along the y-axis in the same manner the characters may be displayed along the x-axis and the z-axis.

In at least one embodiment, the augmented reality text input program110A,110B may allow the user to select the desired word depicted on the y-axis in a substantially similar manner to how the user selected the characters in steps202and206. Once selected by the user, the augmented reality text input program110A,110B may transmit the selected word to an active application as a text file.

It may be appreciated thatFIG.2provides only an illustration of one implementation and does not imply any limitations with regard to how different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements. For example, although the described embodiment includes a three axis keyboard, the augmented reality text input program110A,110B may have many axes with which the user interacts to identify words. Therefore, the user may interact with the keyboard to fully spell a word regardless of how long the word. For example, when spelling the word “BANANA”, the augmented reality text input program110A,110B may depict seven axes where the user can select the appropriate letter on each axis to correctly spell the word “BANANA”. In at least one embodiment, the augmented reality text input program110A,110B may originally depict a two dimensional keyboard in a three-dimensional virtual space that includes an x-axis where the user may select the first letter of the desired word and the y-axis may display proposed words based on the most common words that include the user selected letters. As the user selects a letter on the keyboard, the augmented reality text input program110A,110B may display a replica of the alphanumeric sequence along another axis to allow the user to keep spelling the desired word should the desired word not be displayed along the y-axis. For example, if the first two letters selected by the user are the letters “B” and “A” but the desired word “BANANA” is not depicted along the y-axis, the user may wish to keep spelling the word. Therefore, the augmented reality text input program110A,110B may generate a w-axis (i.e., a fourth axis) on which the user may select the letter “N”. The selection process may iterate until the user-desired word is depicted along the y-axis or the user has successfully spelled the desired word.

Additionally, the augmented reality text input program110A,110B may alternate axes in a three-dimensional keyboard when a user is selecting characters to spell a word rather than generating and displaying additional axes. For example, if the first two letters selected by the user are the letters “B” and “A” but the desired word “BANANA” is not depicted along the y-axis, rather than generating a w-axis, the augmented reality text input program110A,110B may allow the user to select a third letter of the word (e.g., the letter “N” for the instant scenario) along the x-axis on which the user originally selected the first letter. Therefore, the augmented reality text input program110A,110B may alternate between the x-axis and the z-axis when allowing the user to select subsequent characters of the desired word.

The data processing system302,304is representative of any electronic device capable of executing machine-readable program instructions. The data processing system302,304may be representative of a smart phone, a computer system, PDA, or other electronic devices. Examples of computing systems, environments, and/or configurations that may represented by the data processing system302,304include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

The client computing device102and the server112may include respective sets of internal components302a,band external components304a,billustrated inFIG.3. Each of the sets of internal components302include one or more processors320, one or more computer-readable RAMs322, and one or more computer-readable ROMs324on one or more buses326, and one or more operating systems328and one or more computer-readable tangible storage devices330. The one or more operating systems328, the software program108and the augmented reality text input program110A in the client computing device102and the augmented reality text input program110B in the server112are stored on one or more of the respective computer-readable tangible storage devices330for execution by one or more of the respective processors320via one or more of the respective RAMs322(which typically include cache memory). In the embodiment illustrated inFIG.3, each of the computer-readable tangible storage devices330is a magnetic disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices330is a semiconductor storage device such as ROM324, EPROM, flash memory or any other computer-readable tangible storage device that can store a computer program and digital information.

Each set of internal components302a,balso includes a R/W drive or interface332to read from and write to one or more portable computer-readable tangible storage devices338such as a CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk or semiconductor storage device. A software program, such as the augmented reality text input program110A,110B, can be stored on one or more of the respective portable computer-readable tangible storage devices338, read via the respective R/W drive or interface332, and loaded into the respective hard drive330.

Each set of internal components302a,balso includes network adapters or interfaces336such as a TCP/IP adapter cards, wireless Wi-Fi interface cards, or 3G or 4G wireless interface cards or other wired or wireless communication links. The software program108and the augmented reality text input program110A in the client computing device102and the augmented reality text input program110B in the server112can be downloaded to the client computing device102and the server112from an external computer via a network (for example, the Internet, a local area network or other, wide area network) and respective network adapters or interfaces336. From the network adapters or interfaces336, the software program108and the augmented reality text input program110A in the client computing device102and the augmented reality text input program110B in the server112are loaded into the respective hard drive330. The network may comprise copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.

Each of the sets of external components304a,bcan include a computer display monitor344, a keyboard342, and a computer mouse334. External components304a,bcan also include touch screens, virtual keyboards, touch pads, pointing devices, and other human interface devices. Each of the sets of internal components302a,balso includes device drivers340to interface to computer display monitor344, keyboard342, and computer mouse334. The device drivers340, R/W drive or interface332, and network adapter or interface336comprise hardware and software (stored in storage device330and/or ROM324).

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Workloads layer90provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation91; software development and lifecycle management92; virtual classroom education delivery93; data analytics processing94; transaction processing95; and augmented reality text input96. Augmented reality text input96may relate capturing user movements across axes of a virtual space to proposed words to a user for selection.