Augmented reality user interface

A technique for interacting with a computing device includes operating an AR (Augmented Reality) headset as a UI (user interface) component of the computing device. The technique includes pairing the AR headset with the computing device to establish a communication pathway between the two. Once pairing is established, the AR headset detects gestures of the user and transmits UI metadata derived from the gestures to the computing device. The computing device is configured to receive the UI metadata over the communication pathway and to map the UI metadata to user operations to be performed on the computing device. The technique thereby enables the user to control the computing device using gestures.

BACKGROUND

Computerized systems commonly employ input devices for receiving user input. For example, users can enter text into their computers using keyboards and can operate user controls using mouse devices, trackballs, touchpads, and/or touchscreens. Such input devices may be integral to the computing devices, as in the case of tablet computers, or they may be separate devices connected to the computing devices using cables, e.g., USB cables, or wirelessly, e.g., using Bluetooth.

Computerized systems commonly also employ visual display devices, such as monitors, projectors, passive screens, touch screens, and the like. These devices render graphical and/or command line output, which the user can see and to which the user can respond.

SUMMARY

Unfortunately, conventional UI devices tend to keep users in a fixed position relative to their computers. Users must generally sit or stand in front of their computers, where they are not free to move around. Many people find sitting or standing in a fixed position for a long time to be uncomfortable, and recent studies have pointed to detrimental health effects of spending large amounts of time sitting.

Also, when giving presentations based on content viewed from a computer, presenters must often stay close to their computers. Presenters are generally not free to venture far from their machines for long, as they must typically return in order to advance their presentations. Such constraints can negatively impact the quality of presentations, as presenters may be less active and demonstrative as they might be otherwise.

In contrast with prior approaches for interacting with computers, which can highly constrain a user's physical position and activities, an improved technique for interacting with a computing device includes operating an AR (Augmented Reality) headset as a UI (user interface) component of the computing device. The AR headset includes a computer, as well as a set of cameras and other equipment. The technique includes pairing the AR headset with the computing device to establish a communication pathway between the two. Once pairing is established, the AR headset detects gestures of the user and transmits UI metadata derived from the gestures to the computing device. The UI metadata describes the gestures and/or translations of gestures into control instructions and/or data. The computing device is configured to receive the UI metadata over the communication pathway and to map the UI metadata to user operations to be performed on the computing device. The improved technique thereby enables the user to control the computing device using gestures, and frees the user from always having to sit or stand directly in front of the computing device in order to use it.

In some examples, the AR headset identifies, within its field of view, a display area of the computing device (e.g., a monitor area, screen, projected image, other graphical rendering, or any portion thereof) and constructs a “portal” that circumscribes the display area. The portal defines a graphical region that the computing device and the AR headset have in common. The portal also provides a shared canvas on which both the computing device and the AR headset can render content. The computing device can render content on the portal directly by outputting pixel data to its display, and the AR headset can render content on the portal by providing pixel data for the portion of its own display where the display area of the computing device appears. In some examples, the computing device provides digital content to the AR headset, which the AR headset renders in the portal. Also, in some examples, the AR headset provides digital content to the computing device, which the computing device renders in the portal. The AR headset and the computing device may thus act as peers, with each able to render content on the display of the other.

According to some examples, the portal includes two layers: a first layer of graphical content to be displayed by the computing device and a second layer of graphical content to be rendered by the AR headset, i.e., in the region of the AR headset's own display identified as the portal. In an example, both the computing device and the AR headset can contribute content to both layers. Content in the first layer is visible to anyone in sight of the computing device's display, and is thus public, whereas content in the second layer is visible only to the user through the AR headset, and is thus private.

In some examples, the AR headset presents user controls in the form of “holograms,” i.e., virtual 3-D objects projected through the AR headset, and the user interacts with a software program running on the computing device by interacting with the holograms. The AR headset may render such holograms in any location, such as within the portal, outside the portal, on a plane parallel with that of the portal, or in any location and/or orientation.

In some examples, at least some of the holograms and their functions are specific to a software program running on the computing device. For example, the AR headset receives UI control input from the computing device specifying AR-aware UI controls that the software program supports. In further examples, the AR headset displays holograms that provide general control options applicable to most software programs, whether they are AR-aware or not.

In some examples, pairing is performed in response to the AR headset detecting an image element shown on the display of the computing device. The image element presents address or other identifying information that identifies the computing device (e.g., a network address). The AR headset acquires an image of the image element and obtains the address or other identifying information therefrom. The AR headset then sends a connection request to the computing device identified by the image element. The computing device receives the connection request and responds to establish the communication pathway.

In some examples, pairing proceeds in an entirely automatic fashion, with a connection between the AR headset and the computing device established without user intervention, such that the user may seamlessly and effortlessly assume gesture-based control over the computing device, as naturally as would occur if the user were to operate a computer's keyboard or mouse. In other examples, pairing is semi-automatic, with the user required to perform some gesture or other act to confirm that pairing should proceed.

In some examples, the AR headset emits a presence signal, which computing devices in the vicinity of the AR headset can detect. Each such computing device responds to the presence signal by displaying an image element, thereby enabling pairing to proceed, e.g., in response to the AR headset acquiring the image element for a particular computing device and, in some cases, by detecting a particular gesture performed by the user.

In some examples, a server broadcasts information about computing devices, such as their identifying information and locations, in a local space. The AR headset may receive this information and initiate pairing with one of the local computing devices, e.g., in response to the user orienting (e.g., establishing gaze) in the direction of a particular computing device and/or in response to detecting some other user gesture.

In some examples, the AR headset stores or otherwise has access to resources, such as documents, slide shows, pictures, and the like. In accordance with some examples, the user can operate the AR headset to transfer content from the AR headset (or a network-connected location available to the AR headset) to the first layer of the portal, such that the computing device displays the content. For example, the AR headset can detect a gesture that the user performs on a system hologram that represents particular content, followed by a user gaze in a direction that intersects with the portal. The AR headset interprets these user actions as an instruction to render the particular content on the display of the computing device, such that the content is made visible to anyone in view of the computing device's display. If the display is rendered using a projector, these actions make the content from the AR headset visible to anyone who can see the projected images.

In some examples, the software program is a web conferencing program and displaying the content in the software program effects sharing of that content with other participants in the web conference.

In some examples, the software program is a remote desktop control program that controls a remote computing device, such that operating the AR headset effects control over the remote computing device.

In some examples, the AR headset displays UI controls but not screen content from the computing device. Rather, such screen content is viewable directly by the user through the transparent display. In other examples, the AR headset does display screen content from the computing device. Examples include screen content of a primary display, screen content of an extended display (e.g., a second monitor), and/or screen content of a mirrored display.

Certain embodiments are directed to a method of controlling a computing device using an AR (Augmented Reality) headset. The method includes pairing the AR headset with the computing device to establish a communication pathway between the AR headset and the computing device. The method further includes detecting, by the AR headset, gestures of a user of the AR headset and transmitting UI (User Interface) metadata derived from the gestures to the computing device over the communication pathway, such that the computing device is enabled to map the UI metadata to user operations to be performed on the computing device in response to the gestures.

Other embodiments are directed to an AR headset constructed and arranged to perform a method of controlling a computing device, such as the method described above. Still other embodiments are directed to a computer program product. The computer program product stores instructions which, when executed on control circuitry of an AR headset, cause the AR headset to perform a method of controlling a computing device, such as the method described above.

The foregoing summary is presented for illustrative purposes to assist the reader in readily grasping example features presented herein; however, the foregoing summary is not intended to set forth required elements or to limit embodiments hereof in any way. One should appreciate that the above-described features can be combined in any manner that makes technological sense, and that all such combinations are intended to be disclosed herein, regardless of whether such combinations are identified explicitly or not.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the invention will now be described. It should be appreciated that such embodiments are provided by way of example to illustrate certain features and principles of the invention but that the invention hereof is not limited to the particular embodiments described.

An improved technique for interacting with a computing device includes operating an AR (Augmented Reality) headset as a UI (user interface) component of the computing device. The AR headset includes its own computer, and the technique includes pairing the AR headset with the computing device to establish a communication pathway between the two. Once pairing is established, the AR headset detects gestures of the user and transmits UI metadata derived from the gestures to the computing device. The computing device is configured to receive the UI metadata over the communication pathway and to map the UI metadata to user operations to be performed on the computing device.

AR is a quickly-developing technology that projects holograms, i.e., computer-generated 3-D images, onto a transparent display, such that the holograms appear to the user as if they are part of the user's natural environment. The display is part of a headset, which typically also includes cameras, speakers, a microphone, and a computer. The cameras scan the local environment, and the computer creates a map of the user's surroundings. The AR headset may project holograms in such a way that they appear to have constant spatial locations and orientations. For example, an AR headset can place a hologram in what appears to the user to be a stable position, such as on a wall, a table, or at a fixed distance from the user. The AR headset detects movement and changes in orientation and adjusts the holograms so that they appear to remain stationary. The headset projects different images for left and right eyes, producing a true 3-D effect. Users can interact with a particular hologram by using one or more “gestures,” e.g., motions of hands, feet, etc., which can be detected by the cameras. The term gestures as used herein also includes “gaze,” i.e., a sustained orientation of the AR headset in a particular direction. A non-limiting example of an AR headset that is suitable for use with embodiments hereof is the Microsoft HoloLens, which is available from Microsoft Corporation of Redmond, Wash.

FIG. 1shows an example environment100in which embodiments of the improved technique hereof can be practiced. Here, a user102wears an AR headset110in a location that includes a computing device120, such as a desktop computer, laptop computer, tablet computer, smart phone, PDA (Personal Data Assistant), or any other computing device capable of running software programs. The computing device120is connected to a display130, e.g., using VGA (Video Graphics Array), HDMI (High Definition Multimedia Interface), USB (Universal Serial Bus), FireWire, Bluetooth, or any other means, which may be wired or wireless. The display130has a viewable area130ain which pixels can be rendered. The computing device140is connected to a network/medium140, such as a LAN (Local Area Network), a Bluetooth network, or any other wired or wireless network or medium. The AR headset110is also connected to the network/medium140, e.g., using a wireless connection, such as Wi-Fi, Bluetooth, a cell phone network, etc. Although shown as a computer monitor, one should appreciate that display130may also be realized using a television display, a projector, a laptop screen, a smart phone screen, a tablet screen, a PDA screen, and/or as any display device capable of rendering visually perceptible images of electronically-generated content.

In example operation, user102wears the AR headset110and can observe the environment100through the transparent display110a. For example, the user102can see the computing device120and the display130. The user may also see system holograms that the AR headset110has projected onto the AR display110a, such that they appear as three-dimensional objects in the environment100. These system holograms enable the user102to control the AR headset110, such as to run programs loaded onto the AR headset110, to view content accessible through the AR headset110, and so forth.

In accordance with improvements hereof, the user102may take action to effect pairing150of the AR headset110with the computing device120, or pairing may proceed automatically. As will be described, pairing150can proceed in numerous ways, with one example being that the computing device150displays an image element132on the display130. The image element132presents, in graphical form, an address or other identifier of the computing device120, such as a network address. One or more cameras in the AR headset110acquire an image of the element132, and a processor within the AR headset110extracts the address information from the element132. The AR headset110may then issue a communication request to the computing device120at the indicated address. The computing device120receives the request and responds. The AR headset110and the computing device120may then establish a communication pathway142.

With pairing150complete, the AR headset110may display UI control holograms for enabling the user102to control the computing device120and/or any software program running on the computing device120. In some examples, the AR headset110communicates with the computing device120to identify a software program that currently has focus and presents UI control holograms that are suitable for use with that software program. In some examples, the software program may be a legacy program, which is not AR-aware, i.e., not designed specifically to support AR control, and the UI control holograms present general controls, which may act, for example, as replacements for keyboard and/or pointer controls (e.g., scroll up, scroll down, point, click, etc.). In other examples, the software program may be AR-aware. In such cases, the software program running on the computing device120may provide UI control input154to the AR headset110, which specifies AR-aware controls that the software program supports. The AR headset110may then render the AR-aware controls as holograms to enable the user to interact with the software program in arbitrarily sophisticated ways.

The user102may interact with UI control holograms using gestures112(e.g., hand motions, gaze, etc.). When the user interacts with a UI control hologram, the AR headset110generates UI metadata152and sends the UI metadata152to the computing device120over the communication pathway142. In some examples, the AR headset110translates the user gestures into conventional computing inputs, such as pointer actions, keyboard entry, and so forth. For instance, the AR headset110may translate gaze into pointer movement and may translate simple hand motions into mouse clicks. The computing device120receiving the UI metadata152can then apply the input as it would apply input from a local pointer or keyboard. In some examples, the AR headset110sends the UI metadata152in some other format besides pointer action, keyboard entry, and so forth. A component running in the computing device120translates the UI metadata152into actions to be performed in connection with the software program.

In a particular example, a software program running on the computing device120is a web conferencing program, and the user102can interact with the web conferencing program using gestures. For instance, the user102can mute a conference participant by performing a particular gesture on a UI control hologram that represents that participant. The user102can thus avoid having to perform a more complex set of actions using the keyboard and/or mouse. As the AR headset renders the entire environment as potential space for UI controls, such controls can be larger, more conspicuous, and easier to access and manipulate the than the analogous controls on a traditional display.

In some examples, the user102can share content available from the AR headset110with other conference participants, such as by performing a gesture112on a system hologram that represents a particular resource (e.g., a document, picture, slide show, etc.), and by performing another gesture112, such as a gaze112a, to drag and drop that resource into a conference window. Again, the user102can perform such actions in a simple and intuitive way, using the entire virtual environment as a potential workspace. The user102can also control whether content is rendered publicly (e.g., on the display130) or privately (e.g., only on the display110aof the AR headset).

In another particular example, a software program performs remote desktop functions that control a remote computing device. In such cases, controlling the software program via the AR headset110effects control over the remote computing device.

According to some embodiments, the AR headset110receives screen content156from the computing device120. For example, the screen content156represents graphical information that would normally be used for rendering content on a primary monitor, an extended monitor, or a mirrored monitor. Rather than using a monitor, however, the AR headset110renders the screen content as a hologram. In an example, the AR headset110presents the screen content156on a surface, such as a wall, table, etc., or floating in empty space. The user102is thus able to view the hologram of the screen content156for use as a primary display, an extended display, or a mirrored display. In some examples, screen content156also flows from the AR headset110to the computing device120, e.g., to render controls and/or content generated by the AR headset110on the display130.

FIGS. 2A and 2Brespectively show example arrangements of the AR headset110and computing device120. As shown inFIG. 2A, the AR headset110includes the display110a, including left (110aL) and right (110aR) portions, cameras210, sensors212(e.g., motion sensors), one or more communications interfaces214(e.g., Wi-Fi, Bluetooth, etc.), a set of processing units216(e.g., one or more processing chips, cores, and/or assemblies), and memory220. The memory220includes both volatile memory (e.g., RAM), and non-volatile memory, such as one or more ROMs, disk drives, solid state drives, and the like. The set of processing units216and the memory220together form control circuitry, which is constructed and arranged to carry out various methods and functions as described herein. Also, the memory220includes a variety of software constructs realized in the form of executable instructions. When the executable instructions are run by the set of processing units216, the set of processing units216are caused to carry out the operations of the software constructs. Although certain software constructs are specifically shown and described, it is understood that the memory220typically includes many other software constructs, which are not shown, such as an operating system, various applications, processes, and daemons.

As further shown inFIG. 2A, the memory220“includes,” i.e., realizes by operation of software instructions, gesture detector222, gaze detector224, and UI agent230. The gesture detector222is configured to detect gestures112of the user102, while the gaze detector224is configured to detect gaze112aof the user102, e.g., by detecting orientation of the AR headset110in a particular direction for greater than some minimum, predetermined interval of time.

The UI agent230includes discovery manager232, portal manager234, UI metadata generator236, and UI content processor238. In an example, the UI agent230is provided for the purpose of controlling computing devices, like the computing device120. Discovery manager232manages discovery of computing devices and pairing150of the AR headset110with computing devices. Portal manager234identifies displays (like display130) of computing devices and projects holograms onto the AR display110a, such that the user102can see the holograms in a stable spatial relationship with respect to such displays. UI metadata generator236generates UI metadata152based on gestures112, and UI content processor238processes UI control input154.

Turning now toFIG. 2B, computing device120is seen to include a set of communication interfaces254(e.g., Ethernet, Wi-Fi, Bluetooth, etc.), a set of processing units (e.g., one or more processing cores, chips, and/or assemblies), and memory260, which may include both volatile and non-volatile memory. The memory260includes, i.e., realizes by operation of software instructions, one or more software programs262(e.g., a web conferencing program, a remote desktop program, or any type of program), a screen driver264(for rendering screen content), and an AR agent270. The AR agent270includes a discovery manager272, which participates in discovery and pairing150, a portal manager274, which combines graphical content of the computing device120with graphical content from the AR headset110, and a UI metadata processor276, which receives, interprets, and applies UI metadata152received from the AR headset110.

FIG. 3shows an example view that the user102may observe while using the AR headset110after pairing150is complete. Rectangle310shows the field of view of the AR display110a. As indicated, the user102can see the computing device120and its display130through the transparent display110aof the AR headset110and/or using peripheral vision outside a frame of the display110a. The display130is seen to have a viewable area130a, which has a perimeter320.

The AR headset110constructs a portal330within the field of view310. The portal330has a rectangular shape, which conforms to the perimeter320. In an example, portal manager234(FIG. 2) generates the portal330by detecting the viewable area130aof the display130, identifying its borders, and constructing the portal330such that it encloses the viewable area130a. In some examples, the AR headset110projects a rectangular hologram that identifies the portal330, e.g., as a glowing outline, box, or in some other manner. There is no requirement for the AR headset110to display any visual indication of the portal330to the user102, however, as the display area130amay already be visible to the user102directly.

The portal330defines a shared canvas on which both the AR headset110and the computing device120can render visual content. The AR headset110renders pixels in the portal330in the portion of its own display110awhere the viewable area130aappears. The computing device120renders pixels in the portal330directly, via output to the display130. The portal330can thus be regarded as having two layers: a first layer330a, which the computing device120renders on display130, and a second layer330b, which the AR headset110renders on AR display110a. In an example, both the computing device120and the AR headset110can contribute content to both layers330aand330b. Content of the first layer330ais visible to anyone in sight of the display130and is thus public, whereas content of the second layer330bis visible only to the user102through the AR headset110and is thus private.

As further shown inFIG. 3, the AR headset110projects holograms of UI controls350, e.g., in response to UI control input154or otherwise, for controlling the computing device120and/or any software program262running thereon. The AR headset110may display different UI controls350for different software programs262. In some examples, the AR headset110renders UI controls350such that they maintain fixed apparent locations in the environment (or relative to the display130). However, the UI controls350may be rendered anywhere in the environment and may follow the user102, rather than staying in place relative to the display130.

One should appreciate that the field of view310is typically mobile relative to the display130, as the AR headset110changes position and/or orientation. The AR headset110compensates for motion by continually updating the position, orientation, and perspective of the second layer330brendered on the display110a, such that the second layer330bappears to the user102to remain registered with the first layer330a. If the display130is itself moved, the AR headset adjusts by updating the second layer330bsuch that it continues to overlay the first layer330aon the display130.

FIG. 4shows an example arrangement in which the computing device120is connected to a projector402. The computing device120employs the projector402as a sole monitor, a mirrored monitor, or an extended monitor, for example. The projector402projects an image420onto a screen, wall, or other surface. The projected image420has a perimeter422. Elements are shown after pairing150has taken place. Here, the AR headset110treats the projected image420as it did the viewable area130ainFIG. 3. For example, the AR headset110identifies the perimeter422of the projected image420and constructs a portal330so as to encompass the projected image420. The AR headset110projects UI controls450, in this case both within the portal330and outside the portal330.

In this example, the AR headset110also projects a local AR control440. The local AR control440is a hologram that represents a resource stored in or otherwise accessible to the AR headset110. In an example, the user102summons the local AR control440, e.g., by performing a particular gesture. The user102can then perform actions on the local AR control440, e.g., to open a file, copy a file, etc.

In an example, the AR local control440represents a document, presentation, slide show, picture, or the like. For instance, the local AR control440may represent an item in a virtual bookshelf supported by the AR headset110. As shown by arrow460, the user102may perform one or more hand gestures and/or may direct gaze in a manner that directs the AR headset110to transfer the document or other item to the computing device120. For example, the user102may perform a hand gesture112to grab the local AR control440and direct gaze112ato an area within the portal330. Upon detecting this gesture and gaze, the AR headset110copies the item represented by local AR control440to the computing device120, e.g., over the communication pathway142(FIG. 1), and sends control information that directs the computing device120to display the copied item. As a result, the item represented by local AR control440appears within the projected image420, where it can be viewed publicly by anyone in view of the projected image420. If the item is a slideshow, the user102may advance through the slideshow using simple gestures, like tapping, swiping, etc., may zoom in or out using pinching, and so forth. Many possibilities are thus made available for presenting, sharing, and collaborating.

Sharing of content in this manner is not limited to circumstances involving a projector. For example, the computing device120may run web conferencing or other collaboration software, and the user102may employ the above-described grab-and-drop feature to share content available from the AR headset110with other participants.

According to some variants, the item represented by local AR control440is not itself copied to the computing device120. Rather, screen data is sent for viewing the item. The computing device120then renders the screen data in the projected image420, i.e., in the first layer330aof the portal330, where the screen content is publicly viewable.

FIG. 5shows another example view from the perspective of the user102. Reference510depicts the user's view510through the AR headset110. In this example, the AR headset110receives screen content156from the computing device120(FIG. 1) and displays the screen content156within a hologram520. The hologram520may be projected to appear on the back wall, on some other surface, or floating in space. The AR headset110may project the hologram520as pinned to a particular location relative to the environment, or it may project the hologram520at some location that follows the user102as the user moves. The screen content may represent a sole display, an extended display, or a mirrored display of the computing device120, for example. The hologram520may be of any apparent size. Therefore, the user102may enjoy the benefit of a very large display without the associated cost, power, and need for physical space.

In some examples, there is no need for an actual display130to be present in order for the AR headset110to render the hologram520. For example, the computing device120may have no connected display130or the display130may be invisible or turned off. In such cases, the AR headset110may display the hologram520and construct the portal330such that it circumscribes the hologram520. The portal330continues to act as a shared canvas, with the computing device120rendering data for the first layer330aand the AR headset110rendering data for the second layer330b, but with the AR headset110displaying content for both layers330aand330b.

In some examples, the AR headset110projects the hologram520in a manner that follows the user102as the user moves from one location to another. For example, the user102can leave a physical space where the computing device120is located, while continuing to see the projected hologram520located close by. Thus, the user102can continue to work or otherwise interact with the computing device120using the AR headset110, even when the user102and the computing device are in different physical spaces. If the communication pathway142extends over network, the AR headset110can move nearly anywhere in space relative to the computing device120, as long as the AR headset110and the computing device120can each connect to the network. The AR headset110and the computing device120can be on different sides of a room or on opposite sides of a planet.

As there is no need for the AR headset110and the computing device120to be local to each other, new opportunities arise for remote desktop control. For example, the AR headset110can remotely control any computing device as long as it can pair with that computing device. The ability to project a hologram520of screen content enables the AR headset110to interact with the remote computing device over any distance.

One should appreciate that the AR headset110can project holograms520for any number of displays of the computing device120. For example, if the computing device120has two monitors, the AR headset110can project two holograms520, one for each monitor. In addition, the AR headset110can project holograms520even for monitors that are not physically present. The user102could be surrounded by any number of holograms520in virtual space, with each hologram520displaying screen content of a display space from the computing device120, even if the computing device120has only a single monitor, or no monitor at all.

FIG. 6shows another example view from the perspective of the user102. Here, the depicted view shows an arrangement prior to pairing150. The display130displays an image element132(shown also inFIG. 1), which presents address or other identifying information of the computing device120, such as its IP (Internet Protocol) address, MAC (Media Access Control) address, or other address. The image element132may include a QR (Quick Response) code, a Vuforia VuMark, a barcode, a series of characters, or any representation or combination thereof that can identify the computing device120.

In some embodiments, a label620is provided in place of an image element132, or in addition thereto. The label620may be an adhesive-backed label or other type of label and may assume any of the same forms as the image element132. In further examples, the label620is provided as an RFID (Radio Frequency IDentification) label, which may be read by an AR headset equipped with an RFID reader.

FIG. 7shows an example arrangement for facilitating pairing150. As shown, the AR headset110is configured to emit a presence signal710, e.g., a signal that broadcasts the fact that it is an AR headset110. In this example, the computing device120receives the presence signal710and responds by displaying the image element132. Thus, for example, the computing device120is configured to display the image element132whenever it detects the presence of an AR headset in its vicinity. Conversely, the computing device120may be configured not to display the image element132when no presence signal710is detected, thus enabling the computing device120to use the screen space occupied by the image element132for other purposes.

One should appreciate that the presence signal710enables the AR headset110to pair seamlessly with computing devices in its vicinity. For example, the user102can approach any computing device120, and the AR headset110can capture an image of the displayed element132, initiating pairing150with that computing device with little or no effort on the part of the user102. The presence signal may assume any suitable form, such as a Wi-Fi signal, a Bluetooth signal, a radio signal, an infrared signal, an RFID signal, or any other signal capable of identifying the presence of the AR headset110to local computing devices.

FIG. 8shows another example arrangement for facilitating pairing150of the AR headset110with a computing device. Here, the user102enters a space802, such as an office, lab, or other room, which includes multiple computing devices120athrough120f. In this example, a server810broadcasts a local map signal820. The local map signal820is generally a short-range signal that provides addresses (e.g., IP addresses, MAC addresses, etc.) and/or other identifying information about computing devices in the space802, as well as their respective locations in the space802. Location information may be relative to the space802itself, e.g., relative to its walls, floor, ceiling, and or other features. The AR headset110receives the local map signal820and compares the direction of gaze112awith the locations of the computing devices. If the AR headset110detects that the gaze112aintersects with one of the computing devices, the AR headset110initiates pairing150with that computing device, i.e., the computing device that the signal820associates with the gazed-to location. In some examples, an additional gesture112may be required for confirmation before pairing is initiated, e.g., to avoid inadvertently pairing as a result of the user102merely looking in a certain direction. With this arrangement, the user102may seamlessly and with little effort move from one computing device to another, controlling each computing device in turn.

The server810may be a stand-alone server, or it may be implemented in any of the computing devices120athrough120for in some other computing device. In some examples, the server810transmits the local map signal820in response to detecting a presence signal710(FIG. 7) emitted by the AR headset110. Upon such detection, the server810may direct the computing devices120athrough120fto display their respective image elements132, or the computing devices120athrough120fmay themselves detect the presence signal710and respond by displaying their image elements132.

One should appreciate that embodiments hereof are not limited to pairing150through the use of image elements132and/or labels620. Rather, image elements and labels are merely convenient options. Other options may include, for example, a list stored in the AR headset110of computing devices that are candidates for pairing and associated addresses or other identifying information. The AR headset110can then pair with any listed computing device, regardless of its location, by using gestures to select that computing device from the list.

FIG. 9shows an example method900that may be carried out in connection with the environment100. The method900is typically performed, for example, by the software constructs described in connection withFIG. 2, which reside in the memory220and/or260of the AR headset110and/or computing device120and are run by the set of processing units216and/or256. The various acts of method900may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in orders different from that illustrated, which may include performing some acts simultaneously.

At910, the AR headset110is paired with the computing device120to establish a communication pathway142between the AR headset110and the computing device120. For example, pairing150is established in response to the AR headset110acquiring an image of an element132and connecting to the computing device identified by the image element132over the network/medium140.

At920, the AR headset110detects gestures112of a user102of the AR headset110. For example, the AR headset110may detect a hand gesture112, a gaze112a, or a combination of hand gestures and gazes.

At930, UI (User Interface) metadata152derived from the gestures112are transmitted to the computing device120over the communication pathway142, such that the computing device120is enabled to map the UI metadata152to user operations to be performed on the computing device120in response to the gestures112.

In the manner described, the user102is able to control the computing device120using gestures. The user102is thus free to move around while continuing to interact with the computing device120. New techniques for controlling computing devices have been presented and are envisioned, which enable users to interact with computing devices in more natural and less constrained ways, and which are expected to improve users' well-being, health, and effectiveness.

Having described certain embodiments, numerous alternative embodiments or variations can be made. For example, the AR headset110may take a variety of different forms. For example, the AR headset110can be provided as an integrated unit, as with the Microsoft HoloLens, or as different components, such as AR glasses and a separate control unit. Thus, embodiments hereof are not limited to any particular implementation of AR headset.

Also, the communication pathway142need not be established through a single network or medium. Rather, the communication pathway142may include multiple networks and/or media having any number of protocols.

Further, although features are shown and described with reference to particular embodiments hereof, such features may be included and hereby are included in any of the disclosed embodiments and their variants. Thus, it is understood that features disclosed in connection with any embodiment are included as variants of any other embodiment.

Further still, the improvement or portions thereof may be embodied as a computer program product including one or more non-transient, computer-readable storage media, such as a magnetic disk, magnetic tape, compact disk, DVD, optical disk, flash drive, solid state drive, SD (Secure Digital) chip or device, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), and/or the like (shown by way of example as medium950inFIG. 9). Any number of computer-readable media may be used. The media may be encoded with instructions which, when executed on one or more computers, processors, or other control circuitry, perform the processes described herein. Such media may be considered articles of manufacture or machines, and may be transportable from one machine to another.

As used throughout this document, the words “comprising,” “including,” “containing,” and “having” are intended to set forth certain items, steps, elements, or aspects of something in an open-ended fashion. Also, as used herein and unless a specific statement is made to the contrary, the word “set” means one or more of something. This is the case regardless of whether the phrase “set of” is followed by a singular or plural object and regardless of whether it is conjugated with a singular or plural verb. Further, although ordinal expressions, such as “first,” “second,” “third,” and so on, may be used as adjectives herein, such ordinal expressions are used for identification purposes and, unless specifically indicated, are not intended to imply any ordering or sequence. Thus, for example, a “second” event may take place before or after a “first event,” or even if no first event ever occurs. In addition, an identification herein of a particular element, feature, or act as being a “first” such element, feature, or act should not be construed as requiring that there must also be a “second” or other such element, feature or act. Rather, the “first” item may be the only one. Although certain embodiments are disclosed herein, it is understood that these are provided by way of example only and that the invention is not limited to these particular embodiments.