Triggering Field Transitions for Artificial Reality Objects

Aspects of the present disclosure are directed to triggering object transitions between near-field and far-field regions in an artificial reality environment. For example, the near-field region can be within arm's length for the user, while the far-field region can be greater than arm's length from the user. Objects can be displayed within these regions such that the user can interact with the objects in the artificial reality environment. A field manager can cause a displayed object to transition from the near-field region to the far-field region and/or from the far-field region to the near-field region. The field manager can compare sensed user movements to one or more trigger criteria (e.g., distance threshold(s), velocity threshold(s), etc.) to trigger transition of an object between field regions.

TECHNICAL FIELD

The present disclosure is directed to triggering object transitions between near-field and far-field regions in an artificial reality environment.

BACKGROUND

Artificial reality devices provide opportunities for users to experience interactions in an artificial reality environment. However, user interactions with virtual objects in these environments have continued to present challenges. Touch based interactions often require virtual objects to be displayed near the user, however at times this can create a cluttered space near the user. For virtual objects displayed further from the user, the distance between the user and the virtual object can create its own set of interaction challenges.

DETAILED DESCRIPTION

Aspects of the present disclosure are directed to triggering object transitions between near-field and far-field regions in an artificial reality environment. Implementations of an artificial reality system can project a user into an artificial reality environment that includes virtual objects. For example, the virtual objects can be two-dimensional panels, three-dimensional volumes, or any other suitable configuration. Within the artificial reality environment, the user can interact with a virtual object using different interaction mechanisms. In some implementations, the artificial reality environment can include regions relative to the user. A near-field region can be within arm's length for the user, while a far-field region can be greater than arm's length from the user. Objects can be displayed in these regions such that the user can interact with the objects in the artificial reality environment in different modalities (e.g., with rays for the far field and direct touch in the near field).

Implementations of a field manager can manage display of objects within the near-field region and far-field region. For example, the field manager can cause a displayed object to transition from the near-field region to the far-field region (or cause transition from the far-field region to the near-field region). In the near-field region, a user can interact with a displayed object using touch interactions, such as with the user's hands. For example, the user can engage the displayed object using an engagement gesture (e.g., pinch gesture), move the display location for the object in the near-field region, and otherwise interact with the object by touching elements of the object. In the far-field region, the user can interact with the displayed object using ray interactions. For example, a ray can extend from a user's body (e.g., from the user's hand or wrist) to target objects in the far-field region. The user can also engage with an object displayed in the far-field using an engagement gesture, however certain interactions may pose challenges.

For example, the ray-based interaction with an object display in the far-field can pose challenges for fine-grain selection. If an object displays text elements to a user, such as a news object, the user may find it difficult to use the ray interactions to select news articles, scroll through the articles, navigate different areas of the news interface displayed at the object, and the like. Moreover, the text displayed by the object may be difficult to read due to the distance between the far-field region and the user. Instead, the field manager can detect user movement that triggers a transition for the object from the far-field region to the near-field region. The benefits of object display in the near-field region can mitigate these challenges, such as a smaller distance between the object and the user and the availability of touch interactions with the object, which can achieve enhanced accuracy for fine-grain selection.

Implementations of the field manager compare sensed user movements to one or more trigger criteria to trigger transition of an object between field regions. For example, detection of an engagement gesture (e.g., pinch gesture) can initiate an engagement with a displayed object (in either the far-field or the near-field regions). When an object is engaged, user movements can move the location of the object within its current field region. While the object is engaged, the field manager can compare sensed user movement to trigger criteria, such as a distance threshold (e.g., radial distance from the user), a velocity threshold (e.g., overall velocity and/or change in velocity), any combination of these, or any other suitable trigger criteria. When the sensed user movement meets the trigger criteria, the field manager can trigger a transition between field regions (e.g., from near-field region to far-field region or from far-field region to near-field region).

In some implementations, while the object is engaged, user movements that do not meet the trigger criteria can move the object within its current field region, and user movement that meets the trigger criteria can cause transition between field regions. For example, the near-field region can be a free-form three-dimensional space, and a user can engage an object in the near-field and move with object anywhere within the free-form three-dimensional space (by way of user movements that do not meet the trigger criteria). To transition the object to the far-field region, for example to declutter the near-field region, the user can perform a movement that meets the trigger criteria while the object is engaged (e.g., movements that exceed a velocity threshold and/or exceed a radial distance threshold can trigger the transition). The field manager can sense the user movement and cause the object to transition to the far-field region. In this example, the user can manipulate the object in the near-field region using a variety of user movements (e.g., any movement that does not meet the trigger criteria), thus permitting the user to manage the near-field region with a high degree of customization. In other words, because field region transitions are limited to user movements that meet the trigger criteria, the user is free to perform other user movements to design and/or curate the display of objects within a given field region.

In some implementations, the far-field region has a radial distance from the user that is greater than 1 meter (e.g., 1.3 meters), meets a convergence criteria for an artificial reality system that implements the artificial reality environment, both of these, or can have any other suitable distance from the user. When an object is displayed in the far-field region, the user can engage the object and move the object, however the object's movements may be limited. For example, the object may be freely moved along a flat or curved two-dimensional surface defined around the user in the artificial reality environment (e.g., the radial distance of the object from the user may be fixed or may have a limited range of movement, such as 0.1 meters). In some implementations, objects displayed in the far-field region can be pinned to a predetermined radial distance for the far-field region.

When an object is transitioned, such as from the near-field region to the far-field region or from the far-field region to the near-field region, the field manager can dynamically alter the object dimensions, dynamically position the object within the new region, and maintain a display at the object. For example, the object can include at least a two-dimensional surface (e.g., a panel), and when transitioning from the near-field region to the far-field region the object can be scaled to a larger size. In some implementations, the field of view of the object relative to the user can be maintained. For example, at a distance closer to the user (e.g., the near-field region) the object may occupy a first portion of the user's field of view. When the object is transitioned to a distance further from the user (e.g., the far-field region) in order to occupy the first portion of the user's field of view the field manager can scale the object to a larger size. On the other hand, when transitioning from the far-field region to the near-field region the object can be scaled to a smaller size. In this example, when the object is transitioned to a distance closer to the user (e.g., the near-field region) in order to occupy the same portion of the user's field of view the field manager can scale the object to a smaller size.

The field manager can also dynamically position the object, such as when transitioning from the near-field region to the far-field region. For example, a ray cast from the user's body can intersect with the far-field region at an intersection point. The field manager can determine the display location for the engaged object according to the intersection point where the ray cast from the user's body meets the far-field region (e.g., at the time the field manager detects that the tracked movement meets the trigger criteria). The field manager can pin the object at the determined post-transition location.

In some implementations, the field manager can also dynamically position the object when transitioning from the far-field region to the near-field region. For example, a ray cast from the user's body that targets the object in the far-field region can intersect the near-field region. In some implementations, the intersection can be represented by a line through the three-dimensional space that defines the near-field region. The field manager can determine the display location for the engaged object based on the intersection line where the ray cast from the user's body passes through the near-field region, such as at a point on the intersection line. For example, the point on the intersection line can be a midpoint, a point on the line that is a predetermined radial distance from the user, or any other suitable point.

Implementations of the field manager can also maintain a display at the object during the transition. For example, the transition can include movement of the object from the pre-transition location to the post-transition location and, in some examples, a dynamic resizing of the object. Content at the object can continue during performance of this transition. For example, the field manager can continue playing a video at the object during the transition. In some implementations, the video can be dynamically resized according to the dynamic resizing of the entire object. In another example, the field manager can continue display of images and/or text during the transition, and these elements can also be dynamically resized according to the dynamic resizing of the entire object.

“Virtual reality” or “VR,” as used herein, refers to an immersive experience where a user's visual input is controlled by a computing system. “Augmented reality” or “AR” refers to systems where a user views images of the real world after they have passed through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display the images on the screen on the opposite side of the tablet from the camera. The tablet can process and adjust or “augment” the images as they pass through the system, such as by adding virtual objects. “Mixed reality” or “MR” refers to systems where light entering a user's eye is partially generated by a computing system and partially composes light reflected off objects in the real world. For example, a MR headset could be shaped as a pair of glasses with a pass-through display, which allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR headset, allowing the MR headset to present virtual objects intermixed with the real objects the user can see. “Artificial reality,” “extra reality,” or “XR,” as used herein, refers to any of VR, AR, MR, or any combination or hybrid thereof.

Conventional virtual object interactions fail to provide experiences that meet user expectations, can cause congested spaces, and fail to support efficient interaction mechanisms for the user. For example, a conventional XR environment can display virtual objects near the user, however this can create congested space around the user. Another example conventional XR environment may permit the user to position virtual objects around the user in a free-form three-dimensional space. However, these XR environment can also become congested, and these systems fail to effectively utilize the space in the XR environment that is further than arm's length from the user.

Implementations of the disclosed technology are expected to overcome these deficiencies in conventional systems by providing a near-field region, a far-field region, and an efficient transition mechanism for transitioning virtual objects between these field regions. For example, the user can interact with virtual objects in the near-field region using touch-based interactions for fine control and interaction capability. Virtual objects can be transitioned to the far-field region to declutter the near-field region. Ray-based interactions permit user interaction with virtual objects in the far-field region, and the user can efficiently transition virtual objects from the far-field to the near-field for fine control and interaction. Implementations further improve over conventional systems through the comparisons of user movements to trigger criteria to trigger virtual object transitions between the near-field region and the far-field region. For example, a user can perform an engagement gesture to engage with a virtual object. Once engaged, the virtual object can be moved within its current field region according to user movements that do not meet the trigger criteria. When user movement that meets the trigger criteria is detected, the engaged virtual object can be transitioned to the new field region. According, the user is able to manipulate a virtual object in the near-field region using a variety of user movements (e.g., any movement that does not meet the trigger criteria), thus permitting the user to manage the near-field region with a high degree of customization. In other words, because field region transitions are limited to user movements that meet the trigger criteria, the user is free to perform other user movements to design and/or curate the display of objects within a given field region.

Several implementations are discussed below in more detail in reference to the figures.FIG.1is a block diagram illustrating an overview of devices on which some implementations of the disclosed technology can operate. The devices can comprise hardware components of a computing system100that trigger object transitions between near-field and far-field regions in an artificial reality environment. In various implementations, computing system100can include a single computing device103or multiple computing devices (e.g., computing device101, computing device102, and computing device103) that communicate over wired or wireless channels to distribute processing and share input data. In some implementations, computing system100can include a stand-alone headset capable of providing a computer created or augmented experience for a user without the need for external processing or sensors. In other implementations, computing system100can include multiple computing devices such as a headset and a core processing component (such as a console, mobile device, or server system) where some processing operations are performed on the headset and others are offloaded to the core processing component. Example headsets are described below in relation toFIGS.2A and2B. In some implementations, position and environment data can be gathered only by sensors incorporated in the headset device, while in other implementations one or more of the non-headset computing devices can include sensor components that can track environment or position data.

Computing system100can include one or more processor(s)110(e.g., central processing units (CPUs), graphical processing units (GPUs), holographic processing units (HPUs), etc.) Processors110can be a single processing unit or multiple processing units in a device or distributed across multiple devices (e.g., distributed across two or more of computing devices101-103).

Computing system100can include one or more input devices120that provide input to the processors110, notifying them of actions. The actions can be mediated by a hardware controller that interprets the signals received from the input device and communicates the information to the processors110using a communication protocol. Each input device120can include, for example, a mouse, a keyboard, a touchscreen, a touchpad, a wearable input device (e.g., a haptics glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, e.g., an infrared sensor), a microphone, or other user input devices.

In some implementations, input from the I/O devices140, such as cameras, depth sensors, IMU sensor, GPS units, LiDAR or other time-of-flights sensors, etc. can be used by the computing system100to identify and map the physical environment of the user while tracking the user's location within that environment. This simultaneous localization and mapping (SLAM) system can generate maps (e.g., topologies, girds, etc.) for an area (which may be a room, building, outdoor space, etc.) and/or obtain maps previously generated by computing system100or another computing system that had mapped the area. The SLAM system can track the user within the area based on factors such as GPS data, matching identified objects and structures to mapped objects and structures, monitoring acceleration and other position changes, etc.

Computing system100can include a communication device capable of communicating wirelessly or wire-based with other local computing devices or a network node. The communication device can communicate with another device or a server through a network using, for example, TCP/IP protocols. Computing system100can utilize the communication device to distribute operations across multiple network devices.

The processors110can have access to a memory150, which can be contained on one of the computing devices of computing system100or can be distributed across of the multiple computing devices of computing system100or other external devices. A memory includes one or more hardware devices for volatile or non-volatile storage, and can include both read-only and writable memory. For example, a memory can include one or more of random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, and so forth. A memory is not a propagating signal divorced from underlying hardware; a memory is thus non-transitory. Memory150can include program memory160that stores programs and software, such as an operating system162, field manager164, and other application programs166. Memory150can also include data memory170that can include, e.g., object data, criteria data, region configuration data, social graph data, configuration data, settings, user options or preferences, etc., which can be provided to the program memory160or any element of the computing system100.

FIG.2Ais a wire diagram of a virtual reality head-mounted display (HMD)200, in accordance with some embodiments. The HMD200includes a front rigid body205and a band210. The front rigid body205includes one or more electronic display elements of an electronic display245, an inertial motion unit (IMU)215, one or more position sensors220, locators225, and one or more compute units230. The position sensors220, the IMU215, and compute units230may be internal to the HMD200and may not be visible to the user. In various implementations, the IMU215, position sensors220, and locators225can track movement and location of the HMD200in the real world and in an artificial reality environment in three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locators225can emit infrared light beams which create light points on real objects around the HMD200. As another example, the IMU215can include e.g., one or more accelerometers, gyroscopes, magnetometers, other non-camera-based position, force, or orientation sensors, or combinations thereof. One or more cameras (not shown) integrated with the HMD200can detect the light points. Compute units230in the HMD200can use the detected light points to extrapolate position and movement of the HMD200as well as to identify the shape and position of the real objects surrounding the HMD200.

The electronic display245can be integrated with the front rigid body205and can provide image light to a user as dictated by the compute units230. In various embodiments, the electronic display245can be a single electronic display or multiple electronic displays (e.g., a display for each user eye). Examples of the electronic display245include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., microLED, LASER, etc.), some other display, or some combination thereof.

In some implementations, the HMD200can be coupled to a core processing component such as a personal computer (PC) (not shown) and/or one or more external sensors (not shown). The external sensors can monitor the HMD200(e.g., via light emitted from the HMD200) which the PC can use, in combination with output from the IMU215and position sensors220, to determine the location and movement of the HMD200.

FIG.2Bis a wire diagram of a mixed reality HMD system250which includes a mixed reality HMD252and a core processing component254. The mixed reality HMD252and the core processing component254can communicate via a wireless connection (e.g., a 60 GHz link) as indicated by link256. In other implementations, the mixed reality system250includes a headset only, without an external compute device or includes other wired or wireless connections between the mixed reality HMD252and the core processing component254. The mixed reality HMD252includes a pass-through display258and a frame260. The frame260can house various electronic components (not shown) such as light projectors (e.g., LASERs, LEDs, etc.), cameras, eye-tracking sensors, MEMS components, networking components, etc.

The projectors can be coupled to the pass-through display258, e.g., via optical elements, to display media to a user. The optical elements can include one or more waveguide assemblies, reflectors, lenses, mirrors, collimators, gratings, etc., for directing light from the projectors to a user's eye. Image data can be transmitted from the core processing component254via link256to HMD252. Controllers in the HMD252can convert the image data into light pulses from the projectors, which can be transmitted via the optical elements as output light to the user's eye. The output light can mix with light that passes through the display258, allowing the output light to present virtual objects that appear as if they exist in the real world.

Similarly to the HMD200, the HMD system250can also include motion and position tracking units, cameras, light sources, etc., which allow the HMD system250to, e.g., track itself in 3DoF or 6DoF, track portions of the user (e.g., hands, feet, head, or other body parts), map virtual objects to appear as stationary as the HMD252moves, and have virtual objects react to gestures and other real-world objects.

FIG.2Cillustrates controllers270(including controller276A and276B), which, in some implementations, a user can hold in one or both hands to interact with an artificial reality environment presented by the HMD200and/or HMD250. The controllers270can be in communication with the HMDs, either directly or via an external device (e.g., core processing component254). The controllers can have their own IMU units, position sensors, and/or can emit further light points. The HMD200or250, external sensors, or sensors in the controllers can track these controller light points to determine the controller positions and/or orientations (e.g., to track the controllers in 3DoF or 6DoF). The compute units230in the HMD200or the core processing component254can use this tracking, in combination with IMU and position output, to monitor hand positions and motions of the user. The controllers can also include various buttons (e.g., buttons272A-F) and/or joysticks (e.g., joysticks274A-B), which a user can actuate to provide input and interact with objects.

In various implementations, the HMD200or250can also include additional subsystems, such as an eye tracking unit, an audio system, various network components, etc., to monitor indications of user interactions and intentions. For example, in some implementations, instead of or in addition to controllers, one or more cameras included in the HMD200or250, or from external cameras, can monitor the positions and poses of the user's hands to determine gestures and other hand and body motions. As another example, one or more light sources can illuminate either or both of the user's eyes and the HMD200or250can use eye-facing cameras to capture a reflection of this light to determine eye position (e.g., based on set of reflections around the user's cornea), modeling the user's eye and determining a gaze direction.

FIG.3is a block diagram illustrating an overview of an environment300in which some implementations of the disclosed technology can operate. Environment300can include one or more client computing devices305A-D, examples of which can include computing system100. In some implementations, some of the client computing devices (e.g., client computing device305B) can be the HMD200or the HMD system250. Client computing devices305can operate in a networked environment using logical connections through network330to one or more remote computers, such as a server computing device.

In some implementations, server310can be an edge server which receives client requests and coordinates fulfillment of those requests through other servers, such as servers320A-C. Server computing devices310and320can comprise computing systems, such as computing system100. Though each server computing device310and320is displayed logically as a single server, server computing devices can each be a distributed computing environment encompassing multiple computing devices located at the same or at geographically disparate physical locations.

Client computing devices305and server computing devices310and320can each act as a server or client to other server/client device(s). Server310can connect to a database315. Servers320A-C can each connect to a corresponding database325A-C. As discussed above, each server310or320can correspond to a group of servers, and each of these servers can share a database or can have their own database. Though databases315and325are displayed logically as single units, databases315and325can each be a distributed computing environment encompassing multiple computing devices, can be located within their corresponding server, or can be located at the same or at geographically disparate physical locations.

Network330can be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless networks. Network330may be the Internet or some other public or private network. Client computing devices305can be connected to network330through a network interface, such as by wired or wireless communication. While the connections between server310and servers320are shown as separate connections, these connections can be any kind of local, wide area, wired, or wireless network, including network330or a separate public or private network.

FIG.4is a block diagram illustrating components400which, in some implementations, can be used in a system employing the disclosed technology. Components400can be included in one device of computing system100or can be distributed across multiple of the devices of computing system100. The components400include hardware410, mediator420, and specialized components430. As discussed above, a system implementing the disclosed technology can use various hardware including processing units412, working memory414, input and output devices416(e.g., cameras, displays, IMU units, network connections, etc.), and storage memory418. In various implementations, storage memory418can be one or more of: local devices, interfaces to remote storage devices, or combinations thereof. For example, storage memory418can be one or more hard drives or flash drives accessible through a system bus or can be a cloud storage provider (such as in storage315or325) or other network storage accessible via one or more communications networks. In various implementations, components400can be implemented in a client computing device such as client computing devices305or on a server computing device, such as server computing device310or320.

Mediator420can include components which mediate resources between hardware410and specialized components430. For example, mediator420can include an operating system, services, drivers, a basic input output system (BIOS), controller circuits, or other hardware or software systems.

Specialized components430can include software or hardware configured to perform operations for triggering object transitions between near-field and far-field regions in an artificial reality environment. Specialized components430can include transition manager434, near-field manager436, far-field manager438, movement tracker440, and components and APIs which can be used for providing user interfaces, transferring data, and controlling the specialized components, such as interfaces432. In some implementations, components400can be in a computing system that is distributed across multiple computing devices or can be an interface to a server-based application executing one or more of specialized components430. Although depicted as separate components, specialized components430may be logical or other nonphysical differentiations of functions and/or may be submodules or code-blocks of one or more applications.

A near-field region can be a three-dimensional space near a user (e.g., within a threshold radial distance of the user) in the XR environment, such as within arm's length of the user. Near-field manager436can manage objects (e.g., two-dimensional panels, three-dimensional volumes, etc.) and user interactions in the near-field region. For example, the user (e.g., user projected in the XR environment) can engage objects in the near-field region, such as by targeting the objects with a hand movement (e.g., touch interaction) and performing an engagement gesture (e.g., pinch gesture). Near-field manager436can initiate and maintain an engagement between the user and the engaged object until the engagement gesture is released by the user.

Near-field manager436can dynamically move the display of the object according to user movements while the user is engaged with object. The near-field region can be a three-dimensional free-form space, and the user can dynamically move the engaged object anywhere within the three-dimensional free-form space according to the user's hand movements. When the user releases the engagement gesture, near-field manager436can position the object (e.g., lock or pin the object) at the location in the three-dimensional free-form space. The user can interact with objects in the near-field region using touch interactions, such as interactions to scroll content displayed by the objects, actuate buttons that perform functions (e.g., play a video, cause an application specific function), select a displayed element at the object, and other suitable interactions.

In some implementations, a far-field region can be at a distance greater than arm's length from the user in the XR environment, such as greater than 1 meter, 1.3 meters, at a convergence distance for the XR system that implements the XR environment, any combination thereof, or at any other suitable distance. Far-field manager438can manage objects and user interactions in the far-field region. For example, the user can engage objects in the far-field region, such as by targeting the objects with a hand movement (e.g., ray interaction) and performing an engagement gesture (e.g., pinch gesture). Far-field manager438can initiate and maintain an engagement between the user and the engaged object until the engagement gesture is released by the user.

Far-field manager438can dynamically move the display of the object according to user movements while the user is engaged with object. The far-field region can be a two-dimensional space at a fixed radial distance from the user or a two-dimensional space with a limited third dimension (e.g., 0.1 meters, etc.). The user can dynamically move the engaged object in the two-dimensions at the fixed radial distance or move the engaged object in the two dimensions and the limited third dimension. When the user releases the engagement gesture, far-field manager438can position the object (e.g., lock or pin the object) at the location in the two-dimensional space (or limited three-dimensional space). The user can interact with objects in the far-field region using ray interactions, such as interactions to target the object, select a displayed element at the object, and other suitable interactions. In some implementations, the ray-based interactions available for objects at the far-field region can be similar to the touch-based interactions available for objects in the near-field region, however the ray-based interaction may be less precise than the touch-based interaction.

A ray-based interaction can include a ray projection (i.e., straight line) from a control point along a casting direction. For example, the control point can be a palm, fingertips, a fist, a wrist, etc., and the casting direction can be along a line that passes through the control point and an origin point, such as a shoulder, eye, or hip. In other implementations, the control point can be based on other tracked body parts such as a user's eye, head, or chest. For example, the control point can be an estimated position of a center of a user's pupil and the origin point can be an estimated position of the center of a user's retina. In some cases, a graphical representation of the ray projection (the whole line or just a point where the ray hits an object) can be displayed in the artificial reality environment, while in other cases the ray projection is tracked by the XR system without displaying the ray projection. In various implementations, the ray projection can extend from the control point until it intersects with a first object or the ray projection can extend through multiple objects. In some implementations, the direction of the ray projection can be adjusted to “snap” to objects that it is close to intersecting or the ray projection can be curved up to a threshold amount to maintain intersection with such objects.

Transition manager434can manage transitions between a near-field region and a far-field region within a XR environment. In some implementations, while an object is engaged by the user (e.g., after the user has targeted an object, performed an engagement gesture, and maintained the engagement gesture), transition manager434can compare tracked movement by the user (e.g., hand movement tracked by movement tracker440) to a trigger criteria. When the user movement meets the trigger criteria, the engaged object can be transitioned from its current field region to a new field region (e.g., from the near-field region to the far-field region or from the far-field region to the near-field region).

The trigger criteria can include one or more of: a) a threshold radial distance from the user; b) a velocity threshold; c) a change in velocity threshold; or d) any combination thereof. For example, when the tracked user movement meets the velocity threshold, transition manager434can perform a field region transition for the engaged object. In another example, when the tracked user movement exceeds the radial distance threshold (or falls below the radial distance threshold) transition manager434can perform a field region transition for the engaged object.

In an example, transition manager434can transition an engaged object in the near-field region to the far-field region. In this example, the object management for the engaged object can be transitioned from near-field manager436to far-field manager438. Transition manager434can dynamically position the engaged object in the far-field region. For example, a ray cast from the user's body can intersect with the far-field region at an intersection point. The display location for the engaged object post-transition can be determined as the intersection point where the ray cast from the user's body meets the far-field region (e.g., at the time when transition manager434detects that the tracked movement meets the trigger criteria). When the user releases the gesture used to engage the object, transition manager434can pin the object to the far-field region at the determined post-transition location.

When the engaged object transitions from the near-field region to the far-field region, transition manager434can dynamically scale the engaged object to a larger size. In some implementations, transition manager434can maintain the field of view for the engaged object during the transition. For example, at a distance closer to the user (e.g., the near-field region) the engaged object may occupy a first portion of the user's field of view. When the engaged object is transitioned to a distance further from the user (e.g., the far-field region) in order to occupy the first portion of the user's field of view transition manager434can scale the engaged object display to a larger size.

In another example, transition manager434can transition an engaged object in the far-field region to the near-field region. In this example, the object management for the engaged object can be transitioned from far-field manager438to near-field manager436. Transition manager434can dynamically position the engaged object in the near-field region. For example, a ray cast from the user's body that targets the engaged object in the far-field can intersect with the near-field region. In some implementations, the intersection can be represented by a line through the three-dimensional space that defines the near-field region. The display location for the engaged object post-transition can be determined as a point on the intersection line where the ray cast from the user's body passes through the near-field region. For example, the point on the intersection line can be a midpoint, a point on the line that is a predetermined radial distance from the user, or any other suitable point.

When the engaged object transitions from the far-field region to the near-field region, transition manager434can dynamically scale the engaged object to a smaller size. In some implementations, transition manager434can maintain the field of view for the engaged object during the transition. For example, at a distance further from the user (e.g., the far-field region) the engaged object may occupy a first portion of the user's field of view. When the engaged object is transitioned to a distance closer to the user (e.g., the near-field region), to occupy the first portion of the user's field of view, transition manager434can scale the engaged object display to a smaller size.

Implementations of transition manager434can also maintain a display at the engaged object during the transition. For example, the transition can include movement of the engaged object from the pre-transition location to the post-transition location. Content displayed by the engaged object can continue to be displayed while transition manager434transitions the object. For example, a video displayed by the engaged object can continue to play, the display of images and/or text can be maintained, and other suitable content can continue to be displayed. In some implementations, the video, images and/or text, and other suitable content can be dynamically resized during the transition, such as according to the dynamic resizing of the engaged object.

Movement tracker440can track real-world user movements sensed by an XR system. For example, an XR system can include one or more cameras, hand-held controllers, wearable devices, or any other suitable sensors. These sensors can sense real-world movements from the user. Movement tracker440can track the sensed real-world movements such that the user movements can be projected in the XR environment. For example, the user projected in the XR environment can interact with objects in the XR environment based on the sensed and tracked movements. Tracked user hand movements can include a distance traveled, velocity (e.g., speed and direction), and any other suitable tracking. In some implementations, transition manager434can compare the tracked movements (e.g., tracked user hand movements) to one or more trigger criteria to trigger field transitions for engaged objects in the XR environment.

Implementations can estimate an individual user's maximum hand/arm extension using any suitable technique. For example, an XR system can track the user's movements over time. The maximum extension (e.g., radial distance from the user) for the user's hand/arm can be tracked and updated over numerous XR environment interactions. Based on these tracked values, an arm's length maximum distance can be estimated for the user.

FIGS.5A,5B, and5Care diagrams illustrating near-field and far-field object displays in artificial reality environment. Diagrams500A includes objects502, user504, and user hand506. Diagrams500B and500C include object502, user504, user hand506, and thresholds508and510.FIGS.6A and6Bare diagrams illustrating field transitions for an object display in artificial reality environment. Diagrams600A and600B include object502, user504, user hand506, and thresholds508and510.

Diagrams500A,500B,500C,600A, and600B show XR environments implemented by an XR system. The XR environments can include a near-field region and a far-field region. User504can be the projection of a user in the XR environment. User hand506in the XR environment can simulate user hand movements sensed in the real-world. Object502can be a virtual object, such as a two-dimensional object (e.g., a panel), a three-dimensional volume, or any other suitable virtual object.

In diagram500A, object502is located in a far-field region of the illustrated XR environment. For example, user504can interact with object504at the far-field region according to ray-based interactions. In some implementations, a ray cast from user hand506can target object502. In diagram500B, user hand506has performed an engagement gesture (e.g., pinch gesture) to engage object502in the far-field region. For example, the ray cast from user hand506targeted object502and the engagement gesture engaged object502. In diagram500C, object502is located in a near-field region of the illustrated XR environment, and user hand506has performed an engagement gesture to engage object502in the near-field region. In this example, user504can interact with object502at the near-field region according to touch-based interactions.

Diagrams500B and500C also illustrate thresholds508and510. Thresholds508and510can be example trigger criteria, such as radial distance thresholds from user504. In some implementations, the radial distances can be measured from a predetermined location on user504(e.g., the shoulder, a point on the chest, or other suitable points). Threshold508can correspond to a radial distance for triggering a transition from the near-field region to the far-field region and threshold510can correspond to a radial distance for triggering a transition from the far-field region to the near-field region. For example, threshold508can be set to a radial distance at or near the user's arm length offset from a maximum extension and threshold510can be set to a radial distance offset from a maximum retraction. Thresholds508and510can be set to any other suitable distances and can represent a radial distance or any other suitable distance type.

Diagrams500B and600A illustrate an example transition for object502from the far-field region to the near-field region based on user hand506meeting threshold510. For example, diagram500B illustrates object502in a pre-transition location and diagram600A illustrates object502in a post-transition location. Diagram500B illustrates that object502is engaged. A transition can be triggered when tracked user hand movement (e.g., represented by movement of user hand506) meets threshold510. Diagram600A illustrates that user hand506has moved past threshold510. For example, threshold510can be a minimal radial distance threshold, and user hand506can move to a radial distance from user504that is less than or equal to threshold510. Based on the tracked user had movement meeting threshold510(e.g., a trigger criteria), object502can be transitioned from the far-field region to the near-field region. Accordingly, diagram600A illustrates object502in a post-transition location in the near-field region.

Diagrams600A and600B illustrate an example transition for object502from the near-field region to the far-field region based on user hand506meeting threshold508. For example, diagram600A illustrates object502in a pre-transition location and diagram600B illustrates object502in a post-transition location. In some implementations, a user can first transition object502from the far-field region to the near-field region as illustrated by the sequence of diagrams500B and600A, and second transition object502from the near-field region to the far-filed region, as illustrated by the sequence of diagrams600A and600B.

Diagram600A illustrates that object502is engaged. A transition can be triggered when tracked user hand movement (e.g., represented by movement from user hand506) meets threshold508. Diagram600B illustrates that user hand506has moved past threshold508. For example, threshold508can be a maximum radial distance threshold, and user hand506can move to a radial distance from user504that is greater than or equal to threshold508. Based on the tracked user had movement meeting threshold508(e.g., a trigger criteria), object502can be transitioned from the near-field region to the far-field region. Accordingly, diagram600B illustrates object502in a post-transition location in the far-field region.

In some implementations, thresholds508and510can be dynamic according to where user hand506performs the engagement gesture and establishes engagement with object502. For example, if user hand506first engages object502near threshold508(e.g., corresponding to a radial distance at or near a maximum extension for the user), a transition can be unintentionally triggered by a slight movement of user hand506. In some implementations, when user hand506engages object502within a proximity threshold of threshold508, a delta distance (e.g., predetermined radial distance) can be added to threshold508. In this example, threshold508may be extended past the original threshold to mitigate against unintentionally triggering transition of object502. In some cases, when a user's hand passes a threshold distance, hysteresis can be applied to prevent the continual triggering of filed transitions as the user's hand is near a trigger distance. In some implementations, the delta distance can be temporarily added to threshold508, and the threshold can return to the original radial distance threshold value after a period of time (e.g., 1 second, 2 seconds, 5 seconds, and the like).

In another example, if user hand506first engages object502near threshold510(e.g., corresponding to a radial distance at or near a maximum retraction for the user), a transition can be unintentionally triggered by a slight movement of user hand506. In some implementations, when user hand506engages object502within a proximity threshold of threshold510, a delta distance (e.g., predetermined radial distance) can be subtracted from threshold510. In this example, threshold510may be moved back from the original threshold to mitigate against unintentionally triggering transition of object502. In some implementations, the delta distance can be temporarily subtracted from threshold510, and the threshold can return to the original radial distance threshold value after a period of time.

FIG.7is a flow diagram illustrating a process used in some implementations of the present technology for triggering field transitions for an object in an artificial reality environment. In some implementations, process700can be performed by an application or operating system of an XR system. Process700can occur while a user interacts with an XR environment.

At block702, process700can display a virtual object to a user in an XR environment. For example, the virtual object can be a virtual object with a two-dimensional surface (e.g., a panel or interface), three-dimensional volume, or any other suitable virtual object. The user can be an XR representation of a user projected into a VR environment by an XR system or can view a live version of themselves in an MR or AR environment.

In some implementations, the XR environment can include a near-field region and a far-field region. For example, the near-field region can be a three-dimensional free-form space that is at a distance from the user within arm's length. In another example, the far-field region can be a two-dimensional space at a fixed distance from the user, or a two-dimensional space with a third dimension (distance the user) that has a limited span (e.g., 0.1 meters, and the like). The far-field region can have: a) a radial distance greater than 1 meter from the user; b) a radial distance from the user that meets a convergence criteria for the XR system that implements the XR environment; or c) any combination thereof. The virtual object can be located in the near-field region or the far-field region.

In some implementations, when displayed in the near-field region the user interacts with the virtual object in the XR environment using touch interactions, and when displayed in the far-field region the user interacts with the virtual object in the XR environment using ray projection interactions. For example, the user interacts with the virtual object using touch interactions by touching the object with a hand of the user in the XR environment. In another example, the user interacts with the virtual object using ray projection interactions using a ray that extends from the user's body in the XR environment.

At block704, process700can detect an engagement gesture by the user. For example, the XR system can track user movements and detect an engagement gestured (e.g., pinch gesture, an air-tap gesture, a grab gesture, etc.) from the user. In some implementations, the engagement gesture can initiate engagement with the virtual object. For example, the user can target the virtual object, such as using a touch-based interaction and/or a ray-based interaction. While targeting the virtual object, the user can perform the engagement gesture to initiate engagement with the virtual object. In some implementations, the engagement between the user and the virtual object can be maintained until the user releases the engagement.

At block706, process700can detect user movements. In some cases (e.g., for some VR embodiments), the XR system can translate real-world user movements (e.g., hand movements) into movements for the user projected in the XR environment.

At block708, process700can move the virtual object according to the user movements. For example, the detected user movements can occur while engagement with the virtual object is maintained, and the virtual object can be moved according to the user movements. For example, user movements can dynamically move an engaged virtual object in the near-field anywhere within the three-dimensional free-form space defined for the near-field (e.g., according to the user's hand movements). In another example, user movements can dynamically move an engaged virtual object in the far-field in the two dimensions at the fixed distance of the far-field (or the two dimensions and the limited third dimension of the far-field).

At block710, process700can determine whether the user movements meet a distance criteria. When a virtual object is located in the near-field region, the user movements can meet the distance criteria when a radial distance for the user's hand location (e.g., in the XR system relative to the user's body) meets or exceeds a radial distance threshold that corresponds to a maximum arm extension for the user (e.g., an estimate of the user's arm reach at full extension). When virtual object is located in the far-field region, the user movements can meet the distance criteria when a radial distance for the user's hand location (e.g., in the XR system relative to the user's body) meets or falls below a radial distance threshold that corresponds to a maximum arm retraction for the user. When the user movements meet the distance criteria, process700can progress to block716. When the user movements do not meet the distance criteria, process700can progress to block712.

At block712, process700can determine whether the user movements meet a velocity criteria. For example, the velocity criteria can include a velocity threshold, change in velocity threshold, or any other suitable velocity or speed-based threshold. In some embodiments, when a determined velocity/speed for the user movements meets or exceed the velocity threshold(s), the velocity criteria can be met. When the user movements meet the velocity criteria, process700can progress to block716. When the user movements do not meet the velocity criteria, process700can progress to block714.

In some implementations, blocks710and712can be combined such that a field transition for an engaged virtual object is triggered by detected user movement when the distance criteria is met and the velocity criteria is met. For example, the field region transition can be triggered when detected user movement meets the distance criteria (e.g., meets or exceeds a first radial distance threshold, meets or falls below a second radial distance threshold, etc.), and meets the velocity criteria (e.g., meets or exceeds a velocity threshold and/or meets or exceeds a changed in velocity threshold).

At block716, process700can transition the field region for the engaged virtual object. For example, the engaged virtual object can be transitioned from the near-field region to the far-field region or from the far-field region to the near field region. Process800ofFIG.8further describes performance of a field region transition.

At block714, process700can determine whether the user has maintained the engagement gesture. For example, the engagement gesture (e.g., pinch gesture) can be maintained until a disengagement gesture (e.g., release of the pinch gesture) is detected. Implementations of the XR system sense real-world user hand movements to detect whether the user performs a disengagement gesture.

In some implementations, the engagement and/or disengagement gesture can be detected by a model configured to detect the gestures (e.g., a trained machine learning model). For example, the model can output as a confidence value (e.g., value between 1 and 100) that the user has performed the disengagement gesture (or that the user has performed or continues to perform the engagement gesture). In some implementations, the disengagement gesture can be detected when the confidence value meets a threshold confidence value (e.g., 60, 70, or any other suitable threshold).

In some implementations, the threshold confidence value can be dynamically increased in relation to a determined velocity/speed for detected user movement. For example, the higher the value of a determined velocity/speed for detected user movement, the higher the threshold confidence value (e.g., the larger the dynamic increase). The dynamic increase in the threshold confidence value can mitigate against the risk of unintended disengagement at high speeds for user movement. In particular, gesture predictions generated by some models may decrease in accuracy at higher speed user movements. Accordingly, some implementations can increase the threshold confidence value to counterbalance the reduced accuracy. In some implementations, the amount of change in a user's gesture to disengage the gesture can also be based on movement velocity. For example, if the engagement gesture is a pinch, the faster the user's hand is moving, the further apart the thumb and index finger tips have to be moved for the system to identify a disengagement of the gesture.

When the user has not maintained the engagement gesture, process700can progress to block718. When the user has maintained the engagement gesture, process700can loop back to block706, where additional user movements can be detected.

At block718, process700can disengage with the virtual object. For example, the virtual object can be positioned at a location in the current field region in which the virtual object is located. The positioned location can correspond to the location for the virtual object at the time disengagement was detected. In some implementations, when disengagement occurs in the near-field region, the virtual object can be positioned (e.g., locked or pinned) at the location in the three-dimensional free-form space. In some implementations, when disengagement occurs in the far-field region, the virtual object can be positioned (e.g., locked or pinned) at the location in the two-dimensional space (e.g., at the fixed distance from the user).

FIG.8is a flow diagram illustrating a process used in some implementations of the present technology for performing a field transition of an object in an artificial reality environment. In some implementations, process800can be performed by an application or operating system of an XR system. Process800can occur while a user interacts with an XR environment. In some implementations, process800can reflect a transition process for virtual object, such as block714ofFIG.7.

At block802, process800can determine a post-transition location can be for the virtual object. In an example where an engaged virtual object is transitioned from the near-field region to the far-field region, a ray cast from the user's body can intersect with the far-field region at an intersection point. At the time of detection that the user's movements meet the trigger criteria, the determined display location at the far-field region for the virtual object post-transition can be the intersection point where the ray cast from the user's body meets the far-field region.

In another example where an engaged virtual object is transitioned from the far-field region to the near-field region, a ray cast from the user's body that targets the virtual object in the far-field region can intersect the near-field region. In some implementations, the intersection can be represented by a line through the three-dimensional space that defines the near-field region. At the time of detection that the user's movements meet the trigger criteria, the determined display location at the near-field region for the virtual object post-transition can be at a point on the intersection line, such as a midpoint, a point on the line that is a predetermined radial distance from the user, or any other suitable point.

While any block can be removed or rearranged in various implementations, block804is shown in dashed lines to indicate there are specific instances where block804is skipped. At block804, process800can adjust dimensions for the display of the virtual object according to the triggered transition. For example, when transitioning from the near-field region to the far-field region the virtual object can be scaled to a larger size. In some implementations, the field of view of the interface display relative to the user can be maintained. For example, at a distance closer to the user (e.g., the near-field region) the virtual object may occupy a first portion of the user's field of view. When the virtual object is transitioned to a distance further from the user (e.g., the far-field region) in order to occupy the first portion of the user's field of view the virtual object can be dynamically scaled to a larger size. On the other hand, when transitioning from the far-field region to the near-field region the virtual object can be scaled to a smaller size. In this example, when the virtual object is transitioned to a distance closer to the user (e.g., the near-field region) in order to occupy the same portion of the user's field of view the virtual object can be dynamically scaled to a smaller size.

At block806, process800can position the display of the virtual object at the determined location for the triggered transition. For example, the virtual object can be pinned or locked to the determined post-transition location. Engagement with the virtual object (e.g., continued engagement or engagement via a new detected engagement gesture) and detected user movement can move the virtual object from the post-transition location.

At block808, process800can maintain, during performance of the triggered transition, a dynamic display at the virtual object. For example, the transition can include movement of the virtual object from the pre-transition location to the post-transition location and, in some examples, a dynamic resizing of the virtual object. Content at the virtual object can continue during performance of this transition. For example, a video played at the virtual object can continue playing during the transition. In some implementations, the video can be dynamically resized according to the dynamic resizing of the entire virtual object. In another example, images and/or text can continue to be displayed during the transition, and these elements can also be dynamically resized according to the dynamic resizing of the entire virtual object.