METHOD OF INTERACTING WITH OBJECTS IN AN ENVIRONMENT

Methods for interacting with objects and user interface elements in a computer-generated environment provide for an efficient and intuitive user experience. In some embodiments, a user can directly or indirectly interact with objects. In some embodiments, while performing an indirect manipulation, manipulations of virtual objects are scaled. In some embodiments, while performing a direct manipulation, manipulations of virtual objects are not scaled. In some embodiments, an object can be reconfigured from an indirect manipulation mode into a direct manipulation mode by moving the object to a respective position in the three-dimensional environment in response to a respective gesture.

FIELD OF THE DISCLOSURE

This relates generally to methods for interacting with objects in a computer-generated environment.

BACKGROUND OF THE DISCLOSURE

Computer-generated environments are environments where at least some objects displayed for a user's viewing are generated using a computer. Users may interact with objects displayed in a computer-generated environment, such as by moving the objects, rotating the objects, etc.

SUMMARY OF THE DISCLOSURE

Some embodiments described in this disclosure are directed to methods of interacting with virtual objects in a computer-generated environment. Some embodiments described in this disclosure are directed to methods of performing direct and indirect manipulation of virtual objects. These interactions provide a more efficient and intuitive user experience. The full descriptions of the embodiments are provided in the Drawings and the Detailed Description, and it is understood that this Summary does not limit the scope of the disclosure in any way.

DETAILED DESCRIPTION

In the following description of embodiments, reference is made to the accompanying drawings which form a part hereof, and in which it is shown by way of illustration specific embodiments that are optionally practiced. It is to be understood that other embodiments are optionally used and structural changes are optionally made without departing from the scope of the disclosed embodiments.

A person can interact with and/or sense a physical environment or physical world without the aid of an electronic device. A physical environment can include physical features, such as a physical object or surface. An example of a physical environment is physical forest that includes physical plants and animals. A person can directly sense and/or interact with a physical environment through various means, such as hearing, sight, taste, touch, and smell. In contrast, a person can use an electronic device to interact with and/or sense an extended reality (XR) environment that is wholly or partially simulated. The XR environment can include mixed reality (MR) content, augmented reality (AR) content, virtual reality (VR) content, and/or the like. An XR environment is often referred to herein as a computer-generated environment. With an XR system, some of a person's physical motions, or representations thereof, can be tracked and, in response, characteristics of virtual objects simulated in the XR environment can be adjusted in a manner that complies with at least one law of physics. For instance, the XR system can detect the movement of a user's head and adjust graphical content and auditory content presented to the user similar to how such views and sounds would change in a physical environment. In another example, the XR system can detect movement of an electronic device that presents the XR environment (e.g., a mobile phone, tablet, laptop, or the like) and adjust graphical content and auditory content presented to the user similar to how such views and sounds would change in a physical environment. In some situations, the XR system can adjust characteristic(s) of graphical content in response to other inputs, such as a representation of a physical motion (e.g., a vocal command).

Many different types of electronic devices can enable a user to interact with and/or sense an XR environment. A non-exclusive list of examples include heads-up displays (HUDs), head mountable devices, projection-based devices, windows or vehicle windshields having integrated display capability, displays formed as lenses to be placed on users' eyes (e.g., contact lenses), headphones/earphones, input devices with or without haptic feedback (e.g., wearable or handheld controllers), speaker arrays, smartphones, tablets, and desktop/laptop computers. A head mountable device can have one or more speaker(s) and an opaque display. Other head mountable devices can be configured to accept an opaque external display (e.g., a smartphone). The head mountable device can include one or more image sensors to capture images/video of the physical environment and/or one or more microphones to capture audio of the physical environment. A head mountable device may have a transparent or translucent display, rather than an opaque display. The transparent or translucent display can have a medium through which light is directed to a user's eyes. The display may utilize various display technologies, such as uLEDs, OLEDs, LEDs, liquid crystal on silicon, laser scanning light source, digital light projection, or combinations thereof. An optical waveguide, an optical reflector, a hologram medium, an optical combiner, combinations thereof, or other similar technologies can be used for the medium. In some implementations, the transparent or translucent display can be selectively controlled to become opaque. Projection-based devices can utilize retinal projection technology that projects images onto users' retinas. Projection devices can also project virtual objects into the physical environment (e.g., as a hologram or onto a physical surface).

FIG.1illustrates an electronic device100configurable to display a computer-generated environment according to some embodiments of the disclosure. In some embodiments, electronic device100is a portable electronic device, such as a tablet computer, laptop computer or a smartphone, among other possibilities. Example architectures of electronic device100are described in further detail with reference toFIGS.2A-2B.FIG.1illustrates electronic device100and table104A located in the physical environment102. In some embodiments, electronic device100is configured to capture and/or display areas of physical environment102including table104A (illustrated in the field of view of electronic device100). In some embodiments, the electronic device100is configured to display one or more virtual objects in the computer-generated environment that are not present in the physical environment102, but are displayed in the computer generated environment (e.g., positioned on or otherwise anchored to the top surface of a computer-generated representation104B of real-world table104A). InFIG.1, for example, an object106not present in the physical environment (e.g., a virtual object) is displayed on the surface of the table104B in the computer-generated environment displayed via device100, optionally in response to detecting the planar surface of table104A in the physical environment102. It should be understood that object106is a representative object and one or more different objects (e.g., of various dimensionality such as two-dimensional or three-dimensional objects) can be included and rendered in a two-dimensional or a three-dimensional computer-generated environment. For example, the virtual objects can include an application or a user interface displayed in the computer-generated environment. Additionally, it should be understood, that the three-dimensional (3D) environment (or 3D object) described herein may be a representation of a 3D environment (or 3D object) displayed in a two dimensional (2D) context (e.g., displayed on a 2D display screen).

FIGS.2A-2Billustrate example block diagrams of architectures for a device or devices in accordance with some embodiments of the disclosure. The blocks inFIG.2Acan represent an information processing apparatus for use in a device. In some embodiments, device200is a portable device, such as a mobile phone, smart phone, a tablet computer, a laptop computer, an auxiliary device in communication with another device, etc. As illustrated inFIG.2A, device200optionally includes various sensors (e.g., one or more hand tracking sensor(s)202, one or more location sensor(s)204, one or more image sensor(s)206, one or more touch-sensitive surface(s)209, one or more motion and/or orientation sensor(s)210, one or more eye tracking sensor(s)212, one or more microphone(s)213or other audio sensors, etc.), one or more display generation component(s)214, one or more speaker(s)216, one or more processor(s)218, one or more memories220, and/or communication circuitry222. One or more communication buses208are optionally used for communication between the above mentioned components of device200.

Communication circuitry222optionally includes circuitry for communicating with electronic devices, networks, such as the Internet, intranets, a wired network and/or a wireless network, cellular networks and wireless local area networks (LANs). Communication circuitry222optionally includes circuitry for communicating using near-field communication (NFC) and/or short-range communication, such as Bluetooth®.

Processor(s)218optionally include one or more general purpose processors, one or more graphics processors, and/or one or more digital signal processors (DSPs). In some embodiments, memory220is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage) that stores computer-readable instructions configured to be executed by processor(s)218to perform the techniques, processes, and/or methods described below. In some embodiments, memories220include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium can be any medium (e.g., excluding a signal) that can tangibly contain or store computer-executable instructions for use by or in connection with the instruction execution system, apparatus, or device. In some embodiments, the storage medium is a transitory computer-readable storage medium. In some embodiments, the storage medium is a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium can include, but is not limited to, magnetic, optical, and/or semiconductor storages. Examples of such storage include magnetic disks, optical discs based on CD, DVD, or Blu-ray technologies, as well as persistent solid-state memory such as flash, solid-state drives, and the like.

Display generation component(s)214optionally include a single display (e.g., a liquid-crystal display (LCD), organic light-emitting diode (OLED), or other types of display). In some embodiments, display generation component(s)214include multiple displays. In some embodiments, display generation component(s)214includes a display with a touch-sensitive surface (e.g., a touch screen), a projector, a holographic projector, a retinal projector, etc.

In some embodiments, device200includes touch-sensitive surface(s)209configured to receive user inputs (touch and/or proximity inputs), such as tap inputs and swipe inputs or other gestures. In some embodiments, display generation component(s)214and touch-sensitive surface(s)209together form touch-sensitive display(s) (e.g., a touch screen integrated with device200or external to device200that is in communication with device200). It should be understood, that device200optionally includes or receives input from one or more other physical user-interface devices than a touch-sensitive surface, such as a physical keyboard, a mouse, a stylus and/or a joystick (or any other suitable input device).

Image sensors(s)206optionally include one or more visible light image sensor, such as charged coupled device (CCD) sensors, and/or complementary metal-oxide-semiconductor (CMOS) sensors operable to obtain images of physical objects from the real-world environment. Image sensor(s)206optionally include one or more infrared (IR) or near infrared (NIR) sensors, such as a passive or an active IR or NIR sensor, for detecting infrared or near infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. Image sensor(s)206optionally include one or more cameras configured to capture movement of physical objects in the real-world environment. Image sensor(s)206optionally include one or more depth sensors configured to detect the distance of physical objects from device200. In some embodiments, information from one or more depth sensors can allow the device to identify and differentiate objects in the real-world environment from other objects in the real-world environment. In some embodiments, one or more depth sensors can allow the device to determine the texture and/or topography of objects in the real-world environment.

In some embodiments, device200uses CCD sensors, event cameras, and depth sensors in combination to detect the physical environment around device200. In some embodiments, image sensor(s)206include a first image sensor and a second image sensor. The first image sensor and the second image sensor work together and are optionally configured to capture different information of physical objects in the real-world environment. In some embodiments, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some embodiments, device200uses image sensor(s)206to detect the position and orientation of device200and/or display generation component(s)214in the real-world environment. For example, device200uses image sensor(s)206to track the position and orientation of display generation component(s)214relative to one or more fixed objects in the real-world environment.

In some embodiments, device200optionally includes hand tracking sensor(s)202and/or eye tracking sensor(s)212. Hand tracking sensor(s)202are configured to track the position/location of a user's hands and/or fingers, and/or motions of the user's hands and/or fingers with respect to the computer-generated environment, relative to the display generation component(s)214, and/or relative to another coordinate system. Eye tracking sensor(s)212are configured to track the position and movement of a user's gaze (eyes, face, or head, more generally) with respect to the real-world or computer-generated environment and/or relative to the display generation component(s)214. The user's gaze can include a direction in which the eyes are directed, and optionally intersection with a particular point or region of space and/or intersection with a particular object. In some embodiments, hand tracking sensor(s)202and/or eye tracking sensor(s)212are implemented together with the display generation component(s)214(e.g., in the same device). In some embodiments, the hand tracking sensor(s)202and/or eye tracking sensor(s)212are implemented separate from the display generation component(s)214(e.g., in a different device).

In some embodiments, the hand tracking sensor(s)202uses image sensor(s)206(e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that capture three-dimensional information from the real-world including one or more hands. In some examples, the hands can be resolved with sufficient resolution to distinguish fingers and their respective positions. In some embodiments, one or more image sensor(s)206are positioned relative to the user to define a field of view of the image sensor(s) and an interaction space in which finger/hand position, orientation and/or movement captured by the image sensors are used as inputs (e.g., to distinguish from a user's resting hand or other hands of other persons in the real-world environment). Tracking the fingers/hands for input (e.g., gestures) can be advantageous in that it provides an input means that does not require the user to touch or hold input device, and using image sensors allows for tracking without requiring the user to wear a beacon or sensor, etc. on the hands/fingers.

In some embodiments, eye tracking sensor(s)212includes one or more eye tracking cameras (e.g., IR cameras) and/or illumination sources (e.g., IR light sources/LEDs) that emit light towards a user's eyes. Eye tracking cameras may be pointed towards a user's eyes to receive reflected light from the light sources directly or indirectly from the eyes. In some embodiments, both eyes are tracked separately by respective eye tracking cameras and illumination sources, and gaze can be determined from tracking both eyes. In some embodiments, one eye (e.g., a dominant eye) is tracked by a respective eye tracking camera/illumination source(s).

Device200optionally includes microphones(s)213or other audio sensors. Device200uses microphone(s)213to detect sound from the user and/or the real-world environment of the user. In some embodiments, microphone(s)213includes an array of microphones that optionally operate together (e.g., to identify ambient noise or to locate the source of sound in space of the real-world environment). In some embodiments, audio and/or voice inputs can be used to interact with the user interface or computer-generated environment captured using one or more audio sensors (e.g., microphones), as permitted by the user of the electronic device.

Device200optionally includes location sensor(s)204configured to detect a location of device200and/or of display generation component(s)214. For example, location sensor(s)204optionally includes a GPS receiver that receives data from one or more satellites and allows device200to determine the device's absolute position in the physical world.

Device200optionally includes motion and/or orientation sensor(s)210configured to detect orientation and/or movement of device200and/or display generation component(s)214. For example, device200uses orientation sensor(s)210to track changes in the position and/or orientation of device200and/or display generation component(s)214(e.g., with respect to physical objects in the real-world environment). Orientation sensor(s)210optionally include one or more gyroscopes, one or more accelerometers, and/or one or more inertial measurement units (IMUs).

It is understood that the architecture ofFIG.2Ais an example architecture, but that device200is not limited to the components and configuration ofFIG.2A. For example, the device can include fewer, additional, or other components in the same or different configurations. In some embodiments, as illustrated inFIG.2B, system250can be divided between multiple devices. For example, a first device260optionally includes processor(s)218A, memory or memories220A, and communication circuitry222A, optionally communicating over communication bus(es)208A. A second device270(e.g., corresponding to device200) optionally includes various sensors (e.g., one or more hand tracking sensor(s)202, one or more location sensor(s)204, one or more image sensor(s)206, one or more touch-sensitive surface(s)209, one or more motion and/or orientation sensor(s)210, one or more eye tracking sensor(s)212, one or more microphone(s)213or other audio sensors, etc.), one or more display generation component(s)214, one or more speaker(s)216, one or more processor(s)218B, one or more memories220B, and/or communication circuitry222B. One or more communication buses208B are optionally used for communication between the above mentioned components of device270. The details of the components for devices260and270are similar to the corresponding components discussed above with respect to device200and are not repeated here for brevity. First device260and second device270optionally communicate via a wired or wireless connection (e.g., via communication circuitry222A-222B) between the two devices.

A computer-generated environment may be displayed using an electronic device (e.g., electronic device100, device200, device270), including using one or more display generation components. The computer-generated environment can optionally include various graphical user interfaces (“GUIs”) and/or user interface objects.

In some embodiments, the electronic device can detect or estimate a lighting characteristic of the real world. The estimate of the lighting characteristic can provide some understanding of lighting in the environment. For example, the estimate of the lighting characteristic may provide an indication of which regions of the real-world environment are light or dark. The estimate of the lighting characteristic may provide an indication of the position of light sources (e.g., parametric light sources, directional light sources, point light sources, area light sources, etc.) and/or orientation of light sources. In some embodiments, the lighting characteristic is estimated as a per-voxel incident light field indicating brightness, color and/or direction. For example, the lighting characteristic can be parameterized as an image-based lighting (IBL) environment map. It should be understood that other parameterizations of the lighting characteristic are possible. In some examples, the lighting characteristic is estimated on a per pixel basis of using a triangle mesh with the lighting characteristic defining lighting for each vertex or for each face. Additionally, it should be understood that the estimate of the lighting characteristic is optionally derived from an intermediate representation (e.g., environment map).

In some embodiments, sensors such as cameras (e.g., image sensor(s)206) are used to capture images of the real-world environment. The images can be processed by processing circuitry (one or more of processor(s)218) to localize and measure light sources. In some embodiments, light can be determined from the reflections and or shadows cast by light sources in the environment. In some embodiments, deep learning (e.g., supervised) or other artificial intelligence or machine learning is used to estimate the lighting characteristic based on input image(s).

As described herein, a computer-generated environment including various graphics user interfaces (“GUIs”) may be displayed using an electronic device, such as electronic device100or device200, including one or more display generation components. The computer-generated environment can include one or more virtual objects. In some embodiments, the one or more virtual objects can be interacted with or manipulated within the three-dimensional environment. For example, a user is able to move or rotate a virtual object. As will be described in further detail below, interactions with a virtual object can be either direct or indirect and the device can automatically interpret user inputs as either a direct or indirect manipulation based on the context, such as the position of the hands of the user and/or the position of the virtual object to be manipulated.

FIG.3illustrates a method of displaying a three-dimensional environment300with one or more virtual objects according to some embodiments of the disclosure. InFIG.3A, an electronic device (e.g., such as devices100or200described above) is displaying a three-dimensional environment300. In some embodiments, three-dimensional environment300includes one or more real-world objects (e.g., representations of objects in the physical environment around the device) and/or one or more virtual objects (e.g., representations of objects generated and displayed by the device that are not necessarily based on real world objects in the physical environment around the device). For example, inFIG.3A, table302and picture frame304can be representations of real world objects in the physical environment around the device. In some embodiments, table302and picture frame304are displayed by the display generation component by capturing one or more images of table302and picture frame304(e.g., in the physical environment around the device) and displaying a representation of the table and picture frame (e.g., a photorealistic representation, a simplified representation, a caricature, etc.), respectively, in three-dimensional environment300. In some embodiments, table302and picture frame304are passively provided by the device via a transparent or translucent display by not obscuring the user's view of table302and picture frame304. InFIG.3A, cube306is a virtual object and is displayed in three-dimensional environment300on top of table302and does not exist in the physical environment around the device. In some embodiments, virtual devices can interact with representations of real-world objects in both the case where the representations of real world objects are actively displayed by the device and passively displayed by the device, such as cube306being displayed as being placed on the top of table302inFIG.3.

In some embodiments, table302and picture frame304are representations of real-world objects in the environment around the device and thus may not be manipulated by the user via the device. For example, because table302exists in the physical environment around the device, to move or otherwise manipulate table302, a user could physically move or manipulate table302in the physical environment around the device to cause table302to be moved or manipulated in three-dimensional environment300. By contrast, because cube306is a virtual object, cube306can be manipulated by a user of the device via the device (e.g., without requiring the user to manipulate objects in the physical world around the device), as will be described in further detail below.

FIGS.4A-4Dillustrate a method of indirectly manipulating a virtual object according to some embodiments of the disclosure. InFIG.4A, the device (e.g., device100or device200) displays three-dimensional environment400(e.g., similar to three-dimensional environment300) via a display generation component, including cube406on table402. In some embodiments, cube406is a virtual object similar to cube306described above with respect toFIG.3.FIG.4Aillustrates cube406twice, but it is understood that the second cube406that is displayed near the bottom of the figure (e.g., near hand410) is not displayed in three-dimensional environment400and is shown inFIG.4Afor the purpose of illustrating the distance of hand410from cube406(e.g., on table402) when performing Gesture A, as will be described in further detail below. In other words, three-dimensional environment400does not include two copies of cube406(e.g., the second cube406near hand410is a duplicate of cube406on table402and is displayed for illustration purposes, and the duplicate is not illustrated inFIGS.4B-4D).

InFIG.4A, hand410is the hand of the user of the device and the device is able to track the position and/or detect gestures performed by hand410(e.g., via one or more hand tracking sensors). In some embodiments, a representation of hand410is displayed in three-dimensional environment400, for example, if hand410is held in front of the device, the device can capture an image of hand410and display a representation of hand410at a corresponding location in the three-dimensional environment (or passively provide visibility of hand410). In other embodiments, hand410may be a real world object in the physical environment that is passively provided by the device via a transparent or translucent display by not obscuring the user's view of the hand. As used herein, reference to a physical object such as hand can refer to either a representation of that physical object presented on a display, or the physical object itself as passively provided by a transparent or translucent display. Thus, as the user moves hand410, a representation of hand410moves in three-dimensional environment400accordingly.

In some embodiments, a user is able to use hand410to interact with virtual objects in three-dimensional environment400as if the user were interacting with real world objects in the physical environment around the device. In some embodiments, a user's interaction with a virtual object can be referred to as either a direct manipulation interaction or an indirect manipulation interaction. In some embodiments, direct manipulation interactions include interactions in which the user uses one or more hands to intersect with a virtual object (or come within a threshold distance of a virtual object) to directly manipulate the virtual object. In some embodiments, indirect manipulation interactions include interactions in which a user uses one or more hands to manipulate a virtual object without the one or more hands intersecting the virtual object (or coming within the threshold distance of the virtual object).

Returning toFIG.4A, the device detects that hand410is performing a first gesture (e.g., “Gesture A”) corresponding to a selection input (e.g., via one or more hand tracking sensors) while gaze408is directed at virtual object (e.g., cube406). In some embodiments, gaze408is detected via one or more eye tracking sensors and is able to determine the location or objects that the user's eyes are looking at or toward. InFIG.4A, when hand410performs the first gesture, hand410is farther than threshold distance412from cube406.

In some embodiments, the distance between hand410and cube406is determined based on the distance between the location of hand410in the physical world and the corresponding location of cube406on table402in the physical world. For example, cube406is displayed at a location in three-dimensional environment400that has a corresponding location in the physical world, and the distance between the corresponding location of cube406in the physical world and the location of hand410of the user in the physical world is used to determine whether hand410is farther than threshold distance412from cube406. In some embodiments, the distance can be determined based on the distance between the location of hand410in three-dimensional environment and cube406in three-dimensional environment400. For example, a representation of hand410is displayed at a respective location in three-dimensional environment400and the distance between the respective position of hand410in three-dimensional environment400and the position of cube406in three-dimensional environment400is used to determine whether hand410is farther than threshold distance412from cube406. For example, if hand410is held one foot in front of the user (e.g., has not reached towards cube406), and the cube406is 6 feet away from the user, then hand410is determined to be five feet away from hand410. In some embodiments, threshold distance412can be 1 inch, 3 inches, 6 inches, 1 foot, 3 feet, etc.

In some embodiments, the first gesture corresponding to a selection input can be a pinch gesture by two or more fingers or one or more hands of the user (e.g., a pinch between the thumb and forefinger of hand410). In some embodiments, the first gesture corresponding to a selection input can be a pointing gesture or a tapping gesture by a finger of hand410(e.g., the forefinger of hand410). In some embodiments, any other gesture that is predetermined to correspond to a selection input is possible.

In some embodiments, in accordance with a determination that the selection gesture (e.g., pinch gesture, “Gesture A”) is performed by hand410while hand410is more than threshold distance412from cube406(e.g., optionally more than threshold distance412from any virtual object), the device is configured in an indirect manipulation mode in which user inputs are directed to the virtual object that the user's gaze is directed to when inputs are received. For example, inFIG.4A, gaze408is directed at cube406(e.g., looking at cube406, focused on cube406, etc.) when hand410performed the selection input. Thus, the selection input is performed on cube406(e.g., cube406is selected for manipulation). In some embodiments, cube406remains selected while hand410maintains the selection gesture. While cube406remains selected, manipulation gestures by hand410cause manipulation operations to be performed on cube406(e.g., optionally even if gaze408moves away from cube406).

FIG.4Billustrates a method of moving a virtual object in three-dimensional environment400. InFIG.4B, while maintaining the selection gesture, the device detects hand410moving rightwards (e.g., in the “x” axis) by a respective amount414. In some embodiments, moving hand410rightwards by respective amount414corresponds to an angular movement of hand410by a respective angle416. For example, to move hand410by respective amount414, the user pivots the user's respective arm by respective angle416. In some embodiments, respective angle416is the angle formed between a first ray that extends from the position of the device outwards to the previous position of the hand and a second ray that extends from the position of the device outwards to the new position of the hand.

InFIG.4B, in response to detecting the rightward movement by hand410by respective amount414while maintaining the selection gesture, cube406is similarly moved rightwards (e.g., in the “x” axis) in three-dimensional environment400by a second respective amount418. In some embodiments, second respective amount418is different from respective amount414. In some embodiments, second respective amount418is the respective amount414scaled by a scaling factor. In some embodiments, the scaling factor is based on the distance of cube406from the user (e.g., the distance of cube406from the “camera” of three-dimensional environment400, the distance of cube406from the location in three-dimensional environment400associated with the user, and/or the location from which the user is viewing three-dimensional environment400). In some embodiments, the second respective amount418is calculated such that the angular change by cube406is the same as the angular change by hand410. For example, the second respective angle420(e.g., the angle formed between a first ray that extend from the position of the device outwards to the previous position of cube406and a second ray that extends from the position of the device outwards to the new position of cube406) is equal to respective angle416. Thus, in some embodiments, the scaling factor for the second respective amount414is calculated based on the distance of cube406from the user and the distance of hand410from the user (e.g., a ratio of the two distances).

In some embodiments, as will be described in further detail below, the movement of cube406can move in any direction based on the movement of hand410(e.g., cube406exhibits six degrees of freedom). In some embodiments, the movement of cube406can lock into one dimension based on the movement of hand410. For example, if the initial movement of hand410is in the x direction (e.g., the horizontal component of the movement of hand410is larger than the other movement components of the movement of hand410, for the first 0.1 seconds, 0.3 seconds, 0.5 seconds, 1 second, or first 1 cm, 3 cm, 10 cm, or movement, etc.), then the movement of cube406locks into only horizontal movements (e.g., cube406only moves horizontally based on the horizontally component of the movement of cube406, and will not move vertically or change depth, even if hand410includes vertical and/or depth movement components, and/or moves vertically and/or changes depth), until the selection input is terminated.

FIG.4Cillustrates a method of rotating a virtual object in three-dimensional environment400. InFIG.4C, while maintaining the selection gesture, the device detects the rotation of hand410by a respective amount422. In some embodiments, the rotation of hand410is in the yaw orientation (e.g., clockwise such that the fingers are rotated rightwards with respect to the wrist and the wrist is rotated leftwards with respect to the fingers). In some embodiments, the rotation of hand410is in the roll orientation (e.g., the fingers and wrist maintain their respective positions with respect to each other but hand410is rotated to reveal portions of hand410that were previously facing the other direction (e.g., portions that were previously obscured and/or facing away from the device). In some embodiments, the rotation of hand410(e.g., in any orientation) that does not include a lateral movement (e.g., horizontal movement, vertical movement, or a change in depth) or includes a lateral movement of less than a threshold amount (e.g., less than 1 inch, less than 3 inches, less than 6 inches, less than 1 foot, etc.) is interpreted as a request to rotate cube406.

InFIG.4C, in response to detecting the rotation by hand410by respective amount422while maintaining the selection gesture, cube406is rotated in accordance with the rotation of hand410by a second respective amount424. In some embodiments, cube406rotates in the same orientation as the rotation of hand410. For example, if hand410rotated in the yaw orientation, then cube406rotates in the yaw orientation and if hand410rotates in a roll orientation, then cube406rotates in the roll orientation, etc. In some embodiments, the second respective amount424that cube406is rotated is the same as the respective amount422that hand410is rotated. For example, if hand410performed a 90 degree rotation, then cube406is rotated in the same direction by 90 degrees.

In some embodiments, the second respective amount424that cube406is rotated is different than the respective amount422of rotation by hand410(e.g., the rotation is dampened or amplified). For example, if cube406can only be rotated by 180 degrees (e.g., a property of cube406is that cube406cannot be upside down, for example), then the rotation of cube406may be scaled by half (e.g., a 90 degree rotation of hand410causes a 45 degree rotation of cube406). In another example, if cube406can only be rotated by 180 degrees, then cube406rotates 180 degrees in response to a 180 degree rotation of hand410, but then cube406either does not rotate in response to further rotation by hand410(e.g., beyond 180 degrees) or exhibits a rubberbanding effect or resistance to further rotation by hand410(e.g., cube406temporarily rotates beyond its maximum amount while hand410is continuing to rotate, but returns to its maximum rotation value when the rotation and/or input terminates).

FIG.4Dillustrates a method of moving a virtual object toward or away from the user in three-dimensional environment400. InFIG.4D, while maintaining the selection gesture, the device detects a movement of hand410by a respective amount426toward the user (e.g., pulling hand410back from an extended position towards the body of the user and/or towards the device). Thus, the distance between hand410and the device is reduced (e.g., a z-direction movement).

InFIG.4D, in response to detecting the movement by hand410moving towards the user and/or device by respective amount426while maintaining the selection gesture, cube406is moved towards the user (e.g., closer to the “camera” of the three-dimensional environment400) by a second respective amount428. In some embodiments, the amount that cube406moves (e.g., second respective amount428) is the same as the amount of movement by hand410(e.g., respective amount426), optionally in the same direction as hand410. In some embodiments, the amount that cube406moves (e.g., second respective amount428) is different from the amount of movement by hand410(e.g., respective amount426), optionally in the same direction as hand410. In some embodiments, the amount that cube406moves is based on the distance of cube406from the user and/or the distance of hand410from the user. For example, if cube406is farther away from the user, then cube406moves by more than if cube406is closer to the user, in response to the same amount of movement by hand410. For example, if hand410moved towards the user (e.g., towards the device, towards the cameras of the device) by 6 inches, then if cube406is far away from the user, cube406can move closer by 2 feet but if cube406is close to the user, cube406can move closer by 6 inches.

In some embodiments, the amount of movement by cube406is scaled based on a ratio between the distance that cube406is from the user and/or device and the distance that hand410is from the user and/or device when the selection input (e.g., pinch gesture) was initially received. For example, if hand410is two feet away from the user (e.g., two feet away from the eyes of the user, two feet away from the device, two feet away from the camera of the device), and cube406is ten feet away from the user (e.g., ten feet away from the eyes of the user ten two feet away from the device, ten feet away from the camera of the device) when the selection input was received, then the scaling factor is five (e.g., the distance of cube406divided by the distance of hand410). Thus, a 1 inch movement of hand410in the z axis (e.g., towards the user or away from the user) causes a 5 inch movement of cube406in the same direction (e.g., towards the user or away from the user). Thus, cube406moves closer to the user as the user brings hand410closer to the user such that when hand410reaches the user, cube406also reaches the user. In this way, a user is able to use hand410to bring cube406from its initial position to the user, without requiring the user to perform the input multiple times. In some embodiments, cube406is brought to the location of the user. In some embodiments, cube406is brought to the location of hand410, such that cube406is in contact with hand410or within a threshold distance of hand410(e.g., 1 inch, 3 inches, 6 inches, etc.). In some embodiments, when cube406is brought to the location of hand410, the user is able to perform direct manipulations of cube406using hand410, as will be described in further detail below with reference toFIGS.5A-5D and6A-6B.

In some embodiments, instead of scaling the movement based on the distance (e.g., of cube406and/or of hand410) from the user, the movement is based on the distance (e.g., of cube406and/or of hand410) from a location that is a predetermined distance in front of the user (e.g., a predetermined reference location that is optionally the location of the user or a location in front of the user). For example, the reference location can be the location of the user, the location of the face of the user, the location of the device (e.g., as described above), or 3 inches in front of the user (or face of the user, or device), 6 inches in front of the user (or face of the user, or device), 1 foot, 3 feet, etc. Thus, using a reference location that is not exactly the location of the user allows a user to bring cube406from the distant location to the user and/or to hand410by bringing hand410to the reference location that is slightly in front of the user (e.g., without requiring the user to bring hand410all the way to the location to the user, which is potentially an awkward gesture).

In some embodiments, the above scaling of the movement of cube406is applied to movements both towards and farther away from the user. In some embodiments, the above scaling is applied only to movements towards the user and movements away from the user (e.g., in the z axis) are scaled different (e.g., scaled 1-to-1 with the movement of hand410). In some embodiments, the above-described scaling is applied to movements in particular directions based on the context and/or type of element being manipulated. For example, if the user is moving a virtual object in a direction that is not intended by the designer of the three-dimensional environment, then the movement of the virtual object can be dampened (e.g., scaled less), but if the user is moving the virtual object in a direction that is intended by the designer, then the movement of the virtual object can be amplified (e.g., scaled more). Thus, the scaling factor can be different based on the direction of movement to provide the user with feedback on whether certain directions of movement are compatible or intended.

It is understood that the movement of virtual objects described above is not limited to only one type of manipulation at a time or movement in one axis at a time. For example, a user is able to move a virtual object (e.g., such as cube406) in both the x, y directions (e.g., as inFIG.4B) and the z direction (e.g., changing depth, as inFIG.4D) while simultaneously rotating the virtual object (e.g., as inFIG.4C). Thus, the device is able to determine different movement and/or rotation components of hand410and perform the appropriate manipulations to a virtual object. For example, if hand410moved leftwards while simultaneously moving closer to the user (e.g., while maintaining selection of cube406), the device can move cube406leftwards in a manner described above with respect toFIG.4Bwhile simultaneously moving cube406closer to the user in a manner described with respect toFIG.4D. Similarly, if hand410moved leftwards while simultaneously rotating, the device can move cube406leftwards in a manner described above with respect toFIG.4Bwhile simultaneously rotating cube406in a manner described above with respect toFIG.4C.

Thus, as described above, while performing indirect manipulations, the direction, magnitude, and/or speed of a manipulation can depend on the direction, magnitude, and/or speed of the movement of the hand of the user. For example, while performing a movement manipulation, if the hand of the user moves rightwards, the virtual object being manipulated moves rightwards, if the hand of the user moves leftwards, the virtual object moves leftwards, if the hand of the user moves forwards (e.g., away from the user), the virtual object moves forward (e.g., away from the user), etc. Similarly, if the hand moves quickly, the virtual object optionally moves quickly, and if the hand moves slowly, the virtual object optionally moves slowly. And as described above, the amount of movement depends on the amount of movement of the hand (e.g., optionally scaled based on distance from the user, as described above). In some embodiments, while performing a rotational manipulation, the direction of rotation, magnitude, and/or speed depends on the direction of rotation, magnitude of rotation, and/or speed of the rotation of the hand of the user, in a manner similar to described above for movement manipulations.

FIGS.5A-5Dillustrate a method of directly manipulating a virtual object according to some embodiments of the disclosure. InFIG.5A, the device is displaying three-dimensional environment500(e.g., similar to three-dimensional environments300and400) via a display generation component, including cube506on table502. In some embodiments, cube506is a virtual object similar to cube306and406described above with respect toFIGS.3and4A-4D. Similarly to described above with respect toFIG.4A,FIG.5Aillustrates cube506twice, but it is understood that the second cube506that is displayed near the bottom of the figure (e.g., near hand510) is not displayed in three-dimensional environment500and is shown inFIG.5Afor the purpose of illustrating the distance of hand510from cube506(e.g., on table502) when performing Gesture A, as will be described in further detail below. In other words, three-dimensional environment500does not include two copies of cube506(e.g., the second cube506near hand510is a duplicate of cube506on table502and is displayed for illustration purposes, and the duplicate is not illustrated inFIGS.5B-5D).

As discussed above, a direct manipulation is an interaction with a virtual object in which a user uses one or more hands to intersect with a virtual object when manipulating the virtual object. For example, grabbing a virtual object in a manner similar to grabbing a physical object and moving the hand that is grabbing the virtual object is an example of moving the virtual object via direct manipulation. In some embodiments, whether a user is performing a direct manipulation or indirect manipulation operation on a virtual object depends on whether the hand of the user is within a threshold distance from the virtual object being manipulated. For example, if the hand of the user is in contact with the virtual object (e.g., at least a portion of the hand of the user is at a location in physical space such that it appears as if the portion of the hand is contacting or intersecting with the virtual object in the three-dimensional environment), then the user is directly interacting with the virtual object. In some embodiments, a device can interpret a user's interactions as a direct manipulation if the hand of the user is within a threshold distance512from a virtual object to be manipulated (e.g., within 1 inch, within 6 inches, within 1 foot, within 3 feet, etc.). In some embodiments, user inputs when hand510is within a threshold distance512of a virtual object are directed to the virtual object. For example, if hand510is within threshold distance512of one virtual object, the user's inputs are directed to that virtual object (optionally without regard to whether the gaze of the user is directed to that virtual object). If hand510is within threshold distance512of two virtual objects, the user's inputs can be directed to the virtual object that is closer or closer to the portion of hand510that is performing the inputs (e.g., closer to the pinch location if the selection input is a pinch gesture) or the virtual object that the user's gaze is directed to. If hand510is not within threshold distance512of any virtual objects, then the device can determine whether the user is performing an indirect manipulation of a virtual object as described above with respect toFIGS.4A-4D(e.g., if the user's gaze is directed to a particular virtual object).

InFIG.5A, the device detects that hand510is performing a gesture corresponding to a selection input (e.g., “Gesture A”, a pinch gesture, a tap gesture, a poke gesture, etc.) when hand510is within threshold distance512of cube506. In some embodiments, in response to hand510performing the selection input when within threshold distance512of cube506(and optionally hand510is not within threshold distance512of any other virtual object), cube506is selected for input such that further user inputs (e.g., object manipulation inputs, etc.) are performed on cube506. InFIG.5A, cube506is selected for input despite gaze514of the user being directed to table502when the selection input was performed. Thus, in some embodiments, the user is able to interact with a virtual object without requiring that the user look at the virtual object via direct manipulation of the virtual object.

InFIG.5B, in response to cube506being selected for input, in some embodiments, cube506is automatically rotated by a respective amount516such that cube506is aligned with one or more axes and/or one or more surfaces of an object. For example, the orientation of cube506is snapped to the nearest axis such that at least one border of cube506is aligned with the x-axis (e.g., perfectly horizontal), the y-axis (e.g., perfectly vertical), or the z-axis (e.g., perfectly flat). In some embodiments, cube506is automatically snapped to an upward orientation (e.g., aligned with gravity and/or other objects in the environment). In some embodiments, in response to cube506being selected for input, cube506snaps to the same orientation as hand510. For example, if hand510is oriented diagonally at a 30 degree angle (e.g., such as shown inFIG.5B), then cube506can be snapped to a 30 degree rotated orientation. In some embodiments, cube506does not change orientations in response to being selected for input and maintains the orientation that it had when the selection input was received (e.g., such as inFIG.5A). In some embodiments, cube506is automatically snapped to the orientation of the surface of table502(e.g., such that the bottom surface of cube506is flush with the top surface of table502).

FIG.5Cillustrates a method of moving a virtual object in three-dimensional environment500. InFIG.5C, while maintaining the selection gesture (e.g., maintaining the pinch gesture, pointing gesture, tapping gesture, etc.), the device detects hand510moving rightwards (e.g., in the “x” axis) by a respective amount518. In response to detecting hand510moving rightwards, the device optionally moves cube506rightwards by a second respective amount520. In some embodiments, cube506moves by the same amount as hand510such that the relative distance and/or relative position between cube506and hand510is maintained. For example, if cube506was 3 inches in front of hand510when the selection input was received, then in response to the user input (and optionally while receiving the user input), cube506moves with the movement of hand510and remains 3 inches in front of hand510. In some embodiments, the movement of cube506in the x and y directions scale 1-to-1 with the movement of hand510. Thus, in some embodiments, the movement of cube506simulates hand510physically holding onto and moving cube506, in which cube506moves in the same direction, by the same amount, and at the same speed as the hand510(e.g., whereas during indirect manipulation, cube506optionally moves more or less than the movement of hand510, as described above with reference toFIG.4B). In some embodiments, the movement of cube506during direct manipulation does not lock into a respective movement orientation and is able to move in any direction (e.g., 6 degrees of freedom) based on the movement of the hand (e.g., whereas during some embodiments of indirect manipulation, the movement of the virtual object locks into one movement orientation, such as the x, y, or z axis, and movements of the hand in other directions are filtered, ignored, or otherwise do not cause the virtual object to move in those other directions).

FIG.5Dillustrates a method of moving a virtual object toward or away from the user in three-dimensional environment500. InFIG.5D, while maintaining the selection gesture (e.g., maintaining the pinch gesture, pointing gesture, tapping gesture, etc.), the device detects hand510moving forward (e.g., away from the user and/or device, in the z direction) by a respective amount522. In response to detecting hand510moving farther away, the device optionally moves cube506farther away by a second respective amount524. In some embodiments, cube506moves by the same amount as hand510such that the relative distance and/or position between cube506and hand510is maintained. Thus, changes in the distance of cube506from the user and/or device (e.g., away from and toward the user) optionally scale 1-to-1 with the movement of hand510(e.g., whereas during indirect manipulation, movements towards and/or away from the user optionally do not scale 1-to-1 with the movement of hand510).

In some embodiments, while performing a direct manipulation of cube506, a rotation of hand510while maintaining the selection gesture causes cube506to also rotate in the same way (optionally exhibiting the same or similar behavior as described above with respect toFIG.4C).

Thus, as illustrated above, when a user is performing a direct manipulation of a virtual object, movements of the virtual object optionally scale 1-to-1 with the movement of the hand that is performing the selection input, but when performing an indirect manipulation of a virtual object, movements of the virtual object do not always scale 1-to-1 with the movement of the hand that is performing the selection input. In some embodiments, rotation inputs scale by the same amount regardless of whether the manipulation is a direct or indirect manipulation. In some embodiments, whether a user is performing a direct manipulation input or an indirect manipulation input is based on whether the hand of the user is within a threshold distance of a virtual object when the selection input (e.g., selection gesture) is received.

Thus, as described above, while performing direct manipulations, the direction, magnitude, and/or speed of a manipulation can depend on the direction, magnitude, and/or speed of the movement of the hand of the user. For example, while performing a movement manipulation, if the hand of the user moves rightwards, the virtual object being manipulated moves rightwards, if the hand of the user moves leftwards, the virtual object moves leftwards, if the hand of the user moves forwards (e.g., away from the user), the virtual object moves forward (e.g., away from the user), etc. Similarly, if the hand moves quickly, the virtual object optionally moves quickly, and if the hand moves slowly, the virtual object optionally moves slowly. And as described above, the amount of movement is scaled 1-to-1 with the amount of movement of the hand (e.g., as opposed to being scaled by distance, as described above inFIGS.4A-4D). In some embodiments, while performing a rotational manipulation, the direction of rotation, magnitude, and/or speed depends on the direction of rotation, magnitude of rotation, and/or speed of the rotation of the hand of the user, in a manner similar to described above for movement manipulations.

FIGS.6A-6Billustrate a method of moving a virtual object according to some embodiments of the disclosure. InFIG.6A, the device is displaying three-dimensional environment600(e.g., similar to three-dimensional environments300,400, and500) via a display generation component, including cube606on table602. In some embodiments, cube606is a virtual object similar to cube306,406, and506described above with respect toFIGS.3,4A-4D, and5A-5D. Similarly to described above with respect toFIG.4AandFIG.5A,FIG.6Aillustrates cube606twice, but it is understood that the second cube606that is displayed near the bottom of the figure (e.g., near hand610) is not displayed in three-dimensional environment600and is shown inFIG.6Afor the purpose of illustrating the distance of hand610from cube606(e.g., on table602) when performing Gesture B, as will be described in further detail below. In other words, three-dimensional environment600does not include two copies of cube606(e.g., the second cube606near hand610is a duplicate of cube606on table602and is displayed for illustration purposes, and the duplicate is not illustrated inFIG.6B).

InFIG.6A, while hand610is more than threshold distance612from cube606, the device detects that hand610performed a respective gesture (e.g., “Gesture B”). In some embodiments, the respective gesture includes a pinch gesture (e.g., between the thumb and forefinger of the hand, or any two or more fingers of one or more hands, such as described above with respect to “Gesture A”). In some embodiments, the respective gesture includes a pinch gesture followed by a predetermined movement and/or rotation of hand610while maintaining the pinch gesture (e.g., Gesture B includes Gesture A, followed by a respective movement by hand610). For example, a tugging gesture by hand610(e.g., an upward rotation of hand610such that the fingers and/or pinch location is moved closer and/or rotated towards to the user while the wrist optionally maintains its position). In some embodiments, the respective gesture includes a pinch gesture followed by a movement of hand610bringing hand610to the location of the user or to a predetermined reference location in front of the user, thus bringing cube606from a distant location to the location of hand610(e.g., such as described above with reference toFIG.4D). In some embodiments, the respective gesture corresponds to request to move cube606to a location for direct manipulation (e.g., to a location associated with hand610). In some embodiments, because the respective gesture is an indirect manipulation input (e.g., hand610is more than threshold distance612from cube606), the device uses gaze614to determine that the user's input is directed to cube606. It is understood that the respective gesture can be any gesture predetermined to correspond to a request to move cube606to a location for direct manipulation (e.g., including, but not limited to, selection of a selectable option to snap cube606to the location of hand610).

In some embodiments, in response to detecting the respective gesture by hand610(e.g., Gesture B) while gaze615is directed to cube606, the device moves cube606to a location associated with hand610, as shown inFIG.6B. In some embodiments, the respective gesture includes a pinch gesture and cube606is moved to the location of the pinch (e.g., a portion of cube606is located at the location of the pinch such that it appears as if hand610is pinching the portion of cube606) or to a location that is within a predetermined distance from the pinch (e.g., 1 inch, 3 inches, 6 inches, etc.). Thus, after moving cube606to the location of the pinch, the user is able to perform direct manipulations to cube606by maintaining the pinch gesture (e.g., maintaining the selection input) and performing direct manipulation gestures, similar to those described above with respect toFIGS.5A-5D(e.g., lateral movements, forward and back movements, rotations, etc.). In some embodiments, moving cube606to the location of the pinch allows a user to use direct manipulation inputs to manipulate an object that is at a location in three-dimensional environment600that otherwise would be too far away to reach using the hand of the user.

It is understood that while the figures and description above describes movements in particular directions or rotations in particular directions, this is merely exemplary and virtual objects can exhibit the same or similar behavior for movements or rotations in any direction. For example, virtual objects can be moved leftwards and exhibit the same response to the user input as the examples illustrated above for moving the virtual object rightwards. Similarly, virtual objects can be rotated in a counter-clockwise fashion and exhibit the same response to the user input as the example illustrated above for rotating the virtual object in a clockwise fashion.

It is also understood that while the figures and description above describe manipulations of virtual objects, the above-described methods can be applied to any type of user interface element or control elements. For example, a button, a slider, dials, knobs, etc. can be moved or rotated according to the direct or indirect manipulation methods described above.

FIG.7is a flow diagram illustrating a method700of manipulating a virtual object according to some embodiments of the disclosure. The method700is optionally performed at an electronic device such as device100, and device200when displaying selectable options on a surface described above with reference toFIGS.3A-3C,4A-4B,5A-5B, and6A-6B. Some operations in method700are, optionally combined (e.g., with each other or with operations in method800) and/or order of some operations is, optionally, changed. As described below, the method700provides methods of manipulating a virtual object in accordance with embodiments of the disclosure (e.g., as discussed above with respect toFIGS.3-6B).

In some embodiments, an electronic device (e.g., a mobile device (e.g., a tablet, a smartphone, a media player, or a wearable device), a computer, etc. such as device100and/or device200) in communication with a display generation component (e.g., a display integrated with the electronic device (optionally a touch screen display) and/or an external display such as a monitor, projector, television, etc.) and one or more input devices (e.g., a touch screen, mouse (e.g., external), trackpad (optionally integrated or external), touchpad (optionally integrated or external), remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera (e.g., visible light camera), a depth sensor and/or a motion sensor (e.g., a hand tracking sensor, a hand motion sensor), etc.) presents (702), via the display generation component, a computer-generated environment including a first user interface element, such as three-dimensional environment300including cube306inFIG.3.

In some embodiments, while presenting the computer-generated environment, the electronic device receives (704) a plurality (e.g., a sequence) of user inputs including a selection input and a manipulation input, such as hand410performing a gesture (e.g., Gesture A) corresponding to a selection input inFIG.4Aand moving hand410while maintaining the gesture inFIGS.4B-4D.

In some embodiments, in accordance with a determination that a representation of a hand of a user of the electronic device is within a threshold distance from the first user interface element when the selection input was received, such as hand510within threshold distance512from cube506inFIG.5A, the electronic device manipulates (706) the first user interface element in accordance with the manipulation input, such as the movement of cube506in accordance with the movement of hand510inFIGS.5C-5D. In some embodiments, manipulating the first user interface includes a movement operation, a rotation operation, a resizing operation, or any other suitable manipulation operation. In some embodiments, the threshold distance is 1 inch, 3 inches, 6 inches, 1 foot, 3 feet, etc.

In some embodiments, in accordance with a determination that the representation of the hand of the user of the electronic device is not within the threshold distance from the first user interface element when the selection input was received (708), such as hand410being further than threshold distance412from cube406inFIG.4A: in accordance with a determination that a gaze of the user of the electronic device is directed at the first user interface element, the electronic device manipulates (710) the first user interface element in accordance with the manipulation input, such as gaze408being directed to cube406when hand410performed the selection input (e.g., “Gesture A”) inFIG.4A, and cube406being manipulated in accordance with the movement of hand410inFIGS.4B-4D, and in accordance with a determination that the gaze of the user of the electronic device is not directed at the first user interface element, the electronic device forgoes (712) manipulating the first user interface element in accordance with the manipulation input, such as if gaze408were not directed to cube406when hand410performed the selection input, then cube406is optionally not manipulated in accordance with the movement of hand410. In some embodiments, if the gaze were directed to another object when the selection input was received, then the other object is manipulated in accordance with the movement of hand410. In some embodiments, a non-virtual object is not manipulable, such that if the gaze is directed to an object that is not a virtual object (e.g., a representation or depiction of a real world object), then the non-virtual object is not manipulated in accordance with the movement of hand410(e.g., the user input is optionally discarded or ignored, and/or a notification is displayed indicating to the user that the object is not manipulable).

In some embodiments, in accordance with a determination that the representation of the hand of the user of the electronic device is within the threshold distance from a second user interface element when the selection input was received, the electronic device manipulates the second user interface element in accordance with the manipulation input. For example, if the hand of the user is within a threshold distance of any virtual object, the respective virtual object that is closest to the hand and/or closest to the pinch point of the hand is selected for input (e.g., such that subsequent movements of the hand cause a manipulation of the respective virtual object).

In some embodiments, in accordance with a determination that the representation of the hand of the user of the electronic device is not within the threshold distance from the second user interface element when the selection input was received, in accordance with a determination that the gaze of the user of the electronic device is directed at the second user interface element, the electronic device manipulates the second user interface element in accordance with the manipulation input, and in accordance with a determination that the gaze of the user of the electronic device is not directed at the second user interface element, the electronic device forgoes manipulating the second user interface element in accordance with the manipulation input. For example, if the hand of the user is not within the threshold distance of any virtual object, then the object that the gaze of the user is directed to is the object that is selected for input in response to detecting the selection input. In some embodiments, if the gaze is directed to a first virtual object, the first virtual object is selected for manipulation, but if the gaze is directed to a second virtual object, the second virtual object is selected for manipulation. As described herein, the determination of whether the gaze of the user is directed to a particular object or location is based on one or more gaze tracking sensors. In some embodiments, if the gaze of the user is directed to a particular location in the physical world that maps to (e.g., corresponds to) a particular location in the three-dimensional environment, then the gaze of the user is considered to be directed to the corresponding location in the three-dimensional environment (e.g., if a virtual object is at that corresponding location in the three-dimensional environment, then the gaze of the user is interpreted to be directed to that virtual object).

In some embodiments, the manipulation input includes a movement of the hand of the user, such as the horizontal movement of hand410inFIG.4B, and the movement towards the user inFIG.4D. In some embodiments, in accordance with the determination that the representation of a hand of the user of the electronic device is within the threshold distance from the first user interface element when the selection input was received, manipulating the first user interface element in accordance with the manipulation input includes moving the first user interface element by an amount equal to an amount of the movement of the hand of the user, such as cube506moving rightwards by the same amount as the rightward movement of hand510inFIG.5C. In some embodiments, in accordance with the determination that the representation of the hand of the user of the electronic device is not within the threshold distance from the first user interface element when the selection input was received, manipulating the first user interface element in accordance with the manipulation input includes moving the first user interface element by an amount not equal to the amount of the movement of the hand of the user, such as cube406moving rightwards by more than the amount of rightward movement of hand410inFIG.4B.

In some embodiments, in response to receiving the selection input and before manipulating the first user interface element in accordance with the manipulation input, the electronic device changes an orientation of the first user interface element based on an orientation of the hand of the user, such as cube516snapping to a particular orientation that is optionally based on the orientation of hand510inFIG.5B. In some embodiments, cube516is snapped to its “upward” orientation. In some embodiments, cube516is snapped to the closest axis. In some embodiments, cube516is snapped to the same orientation as hand510(e.g., if hand510is held diagonally, then cube516is snapped to the same diagonal angle).

In some embodiments, the manipulation input includes a rotation of the hand of the user and manipulating the first user interface element in accordance with the manipulation input includes rotating the first user interface element, such as the rotation of cube406in accordance with the rotation of hand410inFIG.4C. In some embodiments, the virtual object is rotated in the same direction and the same amount as the rotation of the hand. For example, if the hand rotated in the yaw orientation, then the virtual object rotates in the yaw orientation, and if the hand rotated in the pitch orientation, then the virtual object rotates in the pitch orientation, etc. Similarly, if the hand rotated by 30 degrees, then the virtual object optionally rotates by 30 degrees. In some embodiments, a user is able to perform both rotation and movement manipulations simultaneously by both rotating and moving the user's hand while maintaining the selection input.

In some embodiments, the first user interface element includes a control element, such as a button, a slider, a dial, or any other suitable control element. In some embodiments, in response to manipulating the first user interface element in accordance with the manipulation input, the electronic device performs an operation associated with the control element. For example, a user is able to manipulate a control element in a manner similar to that described above with respect to virtual objects, and manipulating the control element optionally causes one or more functions associated with the control element to be performed. For example, sliding a volume slider can cause the volume to change accordingly, etc.

In some embodiments, in accordance with the determination that the representation of the hand of the user is not within the threshold distance from the first user interface element when the selection input was received, and in accordance with a determination that the plurality of user inputs includes a predetermined gesture by the hand of the user, moving the first user interface element to a location in the computer-generated environment associated with the representation of the hand of the user, such as detecting a predetermined gesture (e.g., “Gesture B”) that corresponds to a request to move cube606to a position for direct manipulation (e.g., a remote request to directly manipulate) inFIG.6A, moving cube606towards the user, optionally to or near the location of a pinch by hand610inFIG.6B. Thus, by performing a particular gesture, a user is able to cause an object to move to (e.g., fly towards) the location of the hand (or within a threshold distance of the location of the hand) such that the user is able to perform direct manipulation operations on the object. In this way, a user can directly manipulate the object without resorting to indirect manipulation operations and without requiring the user to walk towards the object. In some embodiments, after completing the manipulation operations, such as after detecting the termination of the selection input (e.g., the termination of the pinch gesture, the termination of Gesture B, and/or detection of another gesture that corresponds to a request to return the virtual object back to its original position), cube606is moved back to its original position before the user input (optionally maintaining the manipulations that were performed while being held by the user, such as rotations, etc.). In some embodiments, after completing the manipulation operations, such as after detecting the termination of the selection input, cube606remains at the location that it was at when the selection input was terminated (e.g., cube606does not move back to its original position, but stays at the position that the user placed it).

FIG.8is a flow diagram illustrating a method800of moving a virtual object by an amount based on the distance of the virtual object to the user according to some embodiments of the disclosure. The method800is optionally performed at an electronic device such as device100, and device200when displaying selectable options on a surface described above with reference toFIGS.3A-3C,4A-4B,5A-5B, and6A-6B. Some operations in method800are, optionally combined (e.g., with each other or with operations in method700) and/or order of some operations is, optionally, changed. As described below, the method800provides methods of moving a virtual object by an amount based on the distance of the virtual object to the user in accordance with embodiments of the disclosure (e.g., as discussed above with respect toFIGS.3-6B).

In some embodiments, an electronic device (e.g., a mobile device (e.g., a tablet, a smartphone, a media player, or a wearable device), a computer, etc. such as device100and/or device200) in communication with a display generation component (e.g., a display integrated with the electronic device (optionally a touch screen display) and/or an external display such as a monitor, projector, television, etc.) and one or more input devices (e.g., a touch screen, mouse (e.g., external), trackpad (optionally integrated or external), touchpad (optionally integrated or external), remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera (e.g., visible light camera), a depth sensor and/or a motion sensor (e.g., a hand tracking sensor, a hand motion sensor), etc.) presents (802), via the display generation component, a computer-generated environment including a first user interface element, such as three-dimensional environment300including cube306inFIG.3.

In some embodiments, while presenting the computer-generated environment, the electronic device receives (804) a user input including a movement component directed at the first user interface element, such as the rightward movement of hand410inFIG.4B. In some embodiments, in accordance with a determination that the electronic device is in a first manipulation mode, the electronic device moves (806) the first user interface element by a first amount in accordance with the movement component, such as moving cube506by an amount520while in direct manipulation mode inFIG.5C. In some embodiments, in accordance with a determination that the electronic device is in a second manipulation mode, different from the first manipulation mode, the electronic device moves (808) the first user interface element by a second amount, greater than the first amount, in accordance with the movement component, such as moving cube406by an amount418while in indirect manipulation mode inFIG.4B.

In some embodiments, the first manipulation mode is a direct manipulation mode, wherein a representation of a hand of a user of the electronic device is within a threshold distance of the first user interface element when the user input was received, such as hand510being within threshold distance512of cube506inFIG.5A, and the second manipulation mode is an indirect manipulation mode, wherein the representation of the hand of the user is not within the threshold distance of the first user interface element when the user input was received, such as hand410being farther than threshold distance412of cube406inFIG.4A.

In some embodiments, the first amount is a same amount as a movement of the movement component of the user input, such as inFIG.5Cand the second amount is a different amount than the movement of the movement component of the user input, such as inFIG.4B.

In some embodiments, the second amount is an amount of a movement of the movement component of the user input scaled by a scaling factor, such as the movement of cube406being scaled by a scaling factor that is based on the distance of cube406from the user and/or the distance of hand410from the user inFIG.4B.

In some embodiments, in accordance with a determination that the movement of the movement component is in a first direction relative to a user of the electronic device, the scaling factor is a first scaling factor, and in accordance with a determination that the movement of the movement component is in a second direction relative to the user, different from the first direction, the scaling factor is a second scaling factor, different from the first scaling factor. For example, if the object is being moved away from the user, then the scaling factor is optionally not based on the distance of object from the user and/or the distance of hand from the user (e.g., optionally the scaling factor is 1), but if the object is being moved towards the user, then the scaling factor is optionally based on the distance of object from the user and/or the distance of hand from the user (e.g., optionally the scaling factor is greater than 1), such as inFIG.4D.

In some embodiments, the second scaling factor is based on at least a distance of the first user interface element from a predetermined reference location in the computer-generated environment (e.g., the location in the three-dimensional environment corresponding to the location of the head of a user of the electronic device, the location of the user of the electronic device, the location of the electronic device, 1 inch, 3 inches, 6 inches, 1 foot, 3 feet, in front of any of the foregoing) and a distance of the representation of the hand of the user from the predetermined reference location (e.g., the distance from the location in the three-dimensional environment corresponding to the hand of the user to the corresponding location of the location of the head of a user of the electronic device, the location of the user of the electronic device, the location of the electronic device, 1 inch, 3 inches, 6 inches, 1 foot, 3 feet, in front of any of the foregoing), such as described inFIG.4B.

In some embodiments, the movement component of the user input includes a lateral movement component parallel to a user of the electronic device (e.g., a horizontal movement and/or a vertical movement while maintaining the same distance from the user), such as inFIG.4B. In some embodiments, an angle of movement of the second amount with respect to a user of the electronic device is a same as an angle of movement of the lateral movement component of the user input with respect to the user of the electronic device, such as cube406being moved rightwards by an amount such that the angle of change420is the same as the angle of change416in the movement of hand410due to the rightward movement of hand410by the respective amount414. Thus, in some embodiments, the scaling factor for lateral movements is proportional to the ratio of the distance of the object from the user to the distance of the hand from the user.