Virtual avatar animation based on facial feature movement

The present disclosure generally relates to displaying visual effects such as virtual avatars. An electronic device having a camera and a display apparatus displays a virtual avatar that changes appearance in response to changes in a face in a field of view of the camera. In response to detecting changes in one or more physical features of the face in the field of view of the camera, the electronic device modifies one or more features of the virtual avatar.

FIELD

The present disclosure relates generally to computer user interfaces, and more specifically to techniques for displaying visual effects.

BACKGROUND

Visual effects are used to enhance a user's experience when capturing and viewing media using electronic devices. Visual effects can alter the appearance of image data or can represent an idealized or completely fictional representation of an environment captured in an image.

BRIEF SUMMARY

Some techniques for displaying visual effects using electronic devices, however, are generally cumbersome and inefficient. For example, some existing techniques use a complex and time-consuming user interface, which may include multiple key presses or keystrokes. Existing techniques require more time than necessary, wasting user time and device energy. This latter consideration is particularly important in battery-operated devices.

Accordingly, the present technique provides electronic devices with faster, more efficient methods and interfaces for displaying visual effects. Such methods and interfaces optionally complement or replace other methods for displaying visual effects. Such methods and interfaces reduce the cognitive burden on a user and produce a more efficient human-machine interface. For battery-operated computing devices, such methods and interfaces conserve power and increase the time between battery charges.

A method is described. The method is performed at an electronic device having one or more cameras and a display apparatus. The method comprises: displaying, via the display apparatus, a virtual avatar having a plurality of avatar features that change appearance in response to detected changes in pose of a face in a field of view of the one or more cameras, the plurality of avatar features including: a first avatar feature; a second avatar feature different from the first avatar feature; and a third avatar feature different from the first and second avatar features; while the face is detected in the field of view of the one or more cameras, the face including a plurality of detected facial features including a first facial feature and a second facial feature different from the first facial feature, detecting movement of one or more facial features of the face; and in response to detecting the movement of the one or more facial features: in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in a first direction: moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction; and moving the second avatar feature and the third avatar feature in a second direction different from the first direction and based on the detected movement of the first facial feature in the first direction; and in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature: moving the second avatar feature based on at least one of a direction and a magnitude of movement of the second facial feature.

A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with one or more cameras and a display apparatus, the one or more programs including instructions for: displaying, via the display apparatus, a virtual avatar having a plurality of avatar features that change appearance in response to detected changes in pose of a face in a field of view of the one or more cameras, the plurality of avatar features including: a first avatar feature; a second avatar feature different from the first avatar feature; and a third avatar feature different from the first and second avatar features; while the face is detected in the field of view of the one or more cameras, the face including a plurality of detected facial features including a first facial feature and a second facial feature different from the first facial feature, detecting movement of one or more facial features of the face; and in response to detecting the movement of the one or more facial features: in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in a first direction: moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction; and moving the second avatar feature and the third avatar feature in a second direction different from the first direction and based on the detected movement of the first facial feature in the first direction; and in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature: moving the second avatar feature based on at least one of a direction and a magnitude of movement of the second facial feature.

A transitory computer-readable storage medium is described. The transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with one or more cameras and a display apparatus, the one or more programs including instructions for: displaying, via the display apparatus, a virtual avatar having a plurality of avatar features that change appearance in response to detected changes in pose of a face in a field of view of the one or more cameras, the plurality of avatar features including: a first avatar feature; a second avatar feature different from the first avatar feature; and a third avatar feature different from the first and second avatar features; while the face is detected in the field of view of the one or more cameras, the face including a plurality of detected facial features including a first facial feature and a second facial feature different from the first facial feature, detecting movement of one or more facial features of the face; and in response to detecting the movement of the one or more facial features: in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in a first direction: moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction; and moving the second avatar feature and the third avatar feature in a second direction different from the first direction and based on the detected movement of the first facial feature in the first direction; and in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature: moving the second avatar feature based on at least one of a direction and a magnitude of movement of the second facial feature.

An electronic device is described. The electronic device comprises: a display apparatus; one or more cameras; one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, via the display apparatus, a virtual avatar having a plurality of avatar features that change appearance in response to detected changes in pose of a face in a field of view of the one or more cameras, the plurality of avatar features including: a first avatar feature; a second avatar feature different from the first avatar feature; and a third avatar feature different from the first and second avatar features; while the face is detected in the field of view of the one or more cameras, the face including a plurality of detected facial features including a first facial feature and a second facial feature different from the first facial feature, detecting movement of one or more facial features of the face; and in response to detecting the movement of the one or more facial features: in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in a first direction: moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction; and moving the second avatar feature and the third avatar feature in a second direction different from the first direction and based on the detected movement of the first facial feature in the first direction; and in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature: moving the second avatar feature based on at least one of a direction and a magnitude of movement of the second facial feature.

An electronic device is described. The electronic device comprises: a display apparatus; one or more cameras; means for displaying, via the display apparatus, a virtual avatar having a plurality of avatar features that change appearance in response to detected changes in pose of a face in a field of view of the one or more cameras, the plurality of avatar features including: a first avatar feature; a second avatar feature different from the first avatar feature; and a third avatar feature different from the first and second avatar features; means for, while the face is detected in the field of view of the one or more cameras, the face including a plurality of detected facial features including a first facial feature and a second facial feature different from the first facial feature, detecting movement of one or more facial features of the face; and means for, in response to detecting the movement of the one or more facial features: in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in a first direction: moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction; and moving the second avatar feature and the third avatar feature in a second direction different from the first direction and based on the detected movement of the first facial feature in the first direction; and means for, in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature: moving the second avatar feature based on at least one of a direction and a magnitude of movement of the second facial feature.

Thus, devices are provided with faster, more efficient methods and interfaces for displaying visual effects, thereby increasing the effectiveness, efficiency, and user satisfaction with such devices. Such methods and interfaces may complement or replace other methods for displaying visual effects.

DESCRIPTION OF EMBODIMENTS

There is a need for electronic devices that provide efficient methods and interfaces for displaying visual effects. For example, while programs already exist for displaying visual effects, these programs are inefficient and difficult to use compared to the techniques below, which allow a user to displaying visual effects in various applications. Such techniques can reduce the cognitive burden on a user who displays visual effects in an application, thereby enhancing productivity. Further, such techniques can reduce processor and battery power otherwise wasted on redundant user inputs.

Below,FIGS. 1A-1B, 2, 3, 4A-4B, and 5A-5Bprovide a description of exemplary devices for performing the techniques for displaying visual effects.FIGS. 6A-6Fillustrate exemplary user interfaces for displaying visual effects in accordance with some embodiments.FIG. 7is a flow diagram illustrating methods for displaying visual effects in accordance with some embodiments. The user interfaces inFIGS. 6A-6Fare used to illustrate the processes described below, including the processes inFIG. 7.

Device100optionally also includes one or more depth camera sensors175.FIG. 1Ashows a depth camera sensor coupled to depth camera controller169in I/O subsystem106. Depth camera sensor175receives data from the environment to create a three dimensional model of an object (e.g., a face) within a scene from a viewpoint (e.g., a depth camera sensor). In some embodiments, in conjunction with imaging module143(also called a camera module), depth camera sensor175is optionally used to determine a depth map of different portions of an image captured by the imaging module143. In some embodiments, a depth camera sensor is located on the front of device100so that the user's image with depth information is, optionally, obtained for video conferencing while the user views the other video conference participants on the touch screen display and to capture selfies with depth map data. In some embodiments, the depth camera sensor175is located on the back of device, or on the back and the front of the device100. In some embodiments, the position of depth camera sensor175can be changed by the user (e.g., by rotating the lens and the sensor in the device housing) so that a depth camera sensor175is used along with the touch screen display for both video conferencing and still and/or video image acquisition.

In some embodiments, a depth map (e.g., depth map image) contains information (e.g., values) that relates to the distance of objects in a scene from a viewpoint (e.g., a camera, an optical sensor, a depth camera sensor). In one embodiment of a depth map, each depth pixel defines the position in the viewpoint's Z-axis where its corresponding two-dimensional pixel is located. In some embodiments, a depth map is composed of pixels wherein each pixel is defined by a value (e.g., 0-255). For example, the “0” value represents pixels that are located at the most distant place in a “three dimensional” scene and the “255” value represents pixels that are located closest to a viewpoint (e.g., a camera, an optical sensor, a depth camera sensor) in the “three dimensional” scene. In other embodiments, a depth map represents the distance between an object in a scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depth of various features of an object of interest in view of the depth camera (e.g., the relative depth of eyes, nose, mouth, ears of a user's face). In some embodiments, the depth map includes information that enables the device to determine contours of the object of interest in a z direction.

As used here, the term “affordance” refers to a user-interactive graphical user interface object that is, optionally, displayed on the display screen of devices100,300, and/or500(FIGS. 1A, 3, and 5A-5B). For example, an image (e.g., icon), a button, and text (e.g., hyperlink) each optionally constitute an affordance.

FIGS. 6A-6Fillustrate exemplary user interfaces for displaying visual effects, in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes inFIG. 7.

FIGS. 6A-6Fillustrate exemplary user inputs and corresponding changes to an exemplary virtual avatar (e.g., shark avatar605), which may be displayed on an electronic device, such as electronic device600shown inFIGS. 6E and 6F, and having display601, which in some cases is a touch-sensitive display, and camera602, which, at a minimum, includes an image sensor that is capable of capturing data representing a portion of the light spectrum (e.g., visible light, infrared light, or ultraviolet light). In some embodiments, camera602includes multiple image sensors and/or other types of sensors. In addition to capturing data representing sensed light, in some embodiments, camera602is capable of capturing other types of data, such as depth data. For example, in some embodiments, camera602also captures depth data using techniques based on speckle, time-of-flight, parallax, or focus. Image data that device600captures using camera602includes data corresponding to a portion of the light spectrum for a scene within the field of view of the camera. Additionally, in some embodiments, the captured image data also includes depth data for the light data. In some other embodiments, the captured image data contains data sufficient to determine or generate depth data for the data for the portion of the light spectrum. In some embodiments, electronic device600includes one or more elements and/or features of devices100,300, and500.

In some examples, electronic device600includes a depth camera, such as an infrared camera, a thermographic camera, or a combination thereof. In some examples, the device further includes a light-emitting device (e.g., light projector), such an IR flood light, a structured light projector, or a combination thereof. The light-emitting device is, optionally, used to illuminate the subject during capture of the image by a visible light camera and a depth camera (e.g., an IR camera) and the information from the depth camera and the visible light camera are used to determine a depth map of different portions of subject captured by the visible light camera. In some embodiments, a depth map (e.g., depth map image) contains information (e.g., values) that relates to the distance of objects in a scene from a viewpoint (e.g., a camera). In one embodiment of a depth map, each depth pixel defines the position in the viewpoint's Z-axis where its corresponding two-dimensional pixel is located. In some examples, a depth map is composed of pixels wherein each pixel is defined by a value (e.g., 0-255). For example, the “0” value represents pixels that are located at the most distant place in a “three dimensional” scene, and the “255” value represents pixels that are located closest to a viewpoint (e.g., camera) in the “three dimensional” scene. In other examples, a depth map represents the distance between an object in a scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depth of various features of an object of interest in view of the depth camera (e.g., the relative depth of eyes, nose, mouth, ears of a user's face). In some embodiments, the depth map includes information that enables the device to determine contours of the object of interest in a z direction. In some embodiments, the lighting effects described herein are displayed using disparity information from two cameras (e.g., two visual light cameras) for rear facing images and using depth information from a depth camera combined with image data from a visual light camera for front facing images (e.g., selfie images). In some embodiments, the same user interface is used when the two visual light cameras are used to determine the depth information and when the depth camera is used to determine the depth information, providing the user with a consistent experience, even when using dramatically different technologies to determine the information that is used when generating the lighting effects. In some embodiments, while displaying the camera user interface with one of the lighting effects applied, the device detects selection of a camera switching affordance and switches from the front facing cameras (e.g., a depth camera and a visible light camera) to the rear-facing cameras (e.g., two visible light cameras that are spaced apart from each other) (or vice versa) while maintaining display of the user interface controls for applying the lighting effect and replacing display of the field of view of the front facing cameras to the field of view of the rear facing cameras (or vice versa).

In some embodiments, a virtual avatar is a representation of the user that can be graphically depicted. In some embodiments, the virtual avatar is non-photorealistic (e.g., cartoonish). In some embodiments, the avatar is an anthropomorphic construct such as a stylized animal (e.g., shark) or a stylization of a normally inanimate object. In some embodiments, the virtual avatar includes an avatar face having one or more avatar features (e.g., avatar facial features). In some embodiments, the avatar features correspond (e.g., are mapped) to one or more physical features of a user's face such that detected movement of the user's physical feature(s) affects the avatar feature (e.g., affects the graphical representation of the features).

In some embodiments, a user is able to manipulate characteristics or features of a virtual avatar using a camera sensor (e.g., camera602) (e.g., camera module143, optical sensor164, depth camera sensor175). As a user's physical features (such as facial features) and position (such as head position, head rotation, or head tilt) changes, the electronic device detects the changes and, in response, modifies the displayed image of the virtual avatar (e.g., to reflect the changes in the user's physical features and position). In some embodiments, the changes to the user's physical features and position are indicative of various expressions, emotions, context, tone, or other non-verbal communication. In some embodiments, the electronic device modifies the displayed image of the virtual avatar to represent these expressions, emotions, context, tone, or other non-verbal communication.

In some embodiments, a virtual avatar can be displayed in the context of various applications such as, for example, a messaging application, a camera application, a media viewer application (e.g., a photo application or other application for viewing multimedia content such as pictures and videos), and a video communication application. For example, in the context of a messaging application, a virtual avatar can be used to generate visual effects (e.g., multimedia content) including stickers, static virtual avatars, and animated virtual avatars, which can be communicated to users of the messaging application. As another example, in the context of a messaging application, camera application, media viewer application, or video communication application, a virtual avatar can be used to display various visual effects when displaying image data (e.g., image data captured by a camera (e.g.,602) of an electronic device (e.g., device100,300,500,600)). Details for generating and sending visual effects (e.g., including virtual avatars) in a messaging application, and displaying visual effects in a messaging application, camera application, media viewer application, and video communication application, are provided in US Patent Publication No. US 2018/0335927 and U.S. patent application Ser. No. 16/110,514, which are hereby incorporated by reference for all purposes.

The representations (e.g., user states611-1through611-12) on the left side ofFIGS. 6A-6Drepresent a user as detected by the electronic device when the user is within the field of view of one or more cameras (e.g., camera602) (e.g., camera module143, optical sensor164, depth camera sensor175) and/or other sensors (e.g., infrared sensors). In other words, the representations of the user are from the perspective of the camera (e.g., camera602) (e.g., camera module143, optical sensor164, depth camera sensor175), which may be positioned on the electronic device (e.g., device100,300,500,600) in some embodiments and, in other embodiments, may be positioned separate from the electronic device (e.g., an external camera or sensor passing data to the electronic device). In some embodiments, the borders of the representations on the left side ofFIGS. 6A-6Drepresent the boundaries of the field of view of the one or more cameras (e.g.,602) (e.g., camera module143, optical sensor164, depth camera sensor175) and/or other sensors (e.g., infrared sensors). In some embodiments, the representations of the user are displayed on a display (e.g., touch screen112, display340, display450, display504, display601) of the electronic device as image data. In some embodiments, the image data is transmitted to an external electronic device for display. In some embodiments, the external electronic device includes one or more elements and/or features of devices100,300,500, and600. In some embodiments, the image data is collected and processed by the device (e.g.,100,300,500,600), but is not immediately displayed on the device or transmitted to an external electronic device.

Each of the representations (e.g., avatar states612-1through612-12) on the right side ofFIGS. 6A-6Dillustrate a virtual avatar (e.g., shark avatar605) in a state that is presented (e.g., displayed after being modified) based on the corresponding detected state of the user (e.g., user states611-1through611-12) located on the left side of the figure. In some embodiments, the virtual avatar is shown from the perspective of a user viewing the virtual avatar. In some embodiments, the virtual avatar is displayed on the display of the electronic device (e.g., touch screen112, display340, display450, display504, display601). In some embodiments, the virtual avatar is transmitted to the external electronic device for display (e.g., with or without the image data of the user). In some embodiments, the representations on the right side ofFIGS. 6A-6Drepresent a position of the virtual avatar within a display region (e.g.,620) of the display of the electronic device (e.g., touch screen112, display340, display450, display504, display601), and the borders of the representations on the right side ofFIGS. 6A-6Drepresent the boundaries of the display region that includes the virtual avatar. In some embodiments, the display region represented on the right side corresponds to an avatar display region (e.g.,620) of an application user interface (e.g.,615), such as a virtual avatar interface, message composition area, or message area (or a portion thereof) that may be presented in the context of, for example, a messaging application.

In some embodiments, the magnitude of a reaction of an avatar feature (e.g., a discrete element of the avatar that can be moved or modified discretely with respect to other avatar features) corresponds to a magnitude of a change in a physical feature of the user (e.g., a detected or tracked feature such as a user muscle, muscle group, or anatomical feature such as an eye or jaw). For example, the magnitude of the change in the physical feature is, in some embodiments, determined in accordance with a potential range of motion of the physical feature, wherein the magnitude is representative of a relative position of the physical feature within the range of motion (e.g., the predicted or modeled range of motion) of that physical feature. In such embodiments, the magnitude of the reaction of the avatar feature is similarly a relative position of the avatar feature within a range of motion of the avatar feature. In some embodiments, the magnitude of change is determined based on a comparison or measurement (e.g., a distance) of the starting position and ending position of the physical feature, through the change. In such embodiments, the change in the physical feature may be translated to a modification of the first avatar feature by applying the measured change in the physical feature to the avatar feature (e.g., directly or as a scaled or adjusted value).

In some embodiments, modifications to an avatar feature have both a magnitude component and a directional component, with the directional component of the modification in the avatar feature being based on a directional component of a change in one or more of the physical features (e.g., facial features of the user's face) to which the avatar feature reacts. In some embodiments, the direction of a reaction of an avatar feature corresponds (e.g., directly or inversely) to a relative direction of a change in a physical feature of the user, wherein the relative direction of the change in the physical feature is determined based on a direction of movement of the physical feature from an initial position (e.g., a neutral, resting position of the physical feature or, in some embodiments, a position of the physical feature that is initially detected by the device). In some embodiments, the direction of the reaction of the avatar feature corresponds directly (e.g., the physical feature moves up, the avatar feature also moves up) to the relative direction of the change in the physical feature. In other embodiments, the direction of the reaction of the avatar feature corresponds inversely (e.g., the physical feature moves up, the avatar feature moves down) to the relative direction of the change in the physical feature.

In some embodiments, the directional component of the change in the avatar feature is mirrored with respect to the directional component of the change in the physical feature. For example, when the physical feature (e.g., user's mouth) moves left, the avatar feature (e.g., avatar mouth) moves right. In some embodiments, the directional component of the change in the avatar feature is the same as the directional component of the change in the physical feature for movement along a vertical axis and mirrored for movement along a horizontal axis, similar to the effect seen when looking in a mirror. In some embodiments, a change in relative position of a physical feature (e.g., the user's iris or eyebrow) is in a direction determined from a neutral, resting position of the physical feature. For example, in some embodiments, the neutral, resting position of a user's iris is determined to be a particular position (e.g., centered) relative to the perimeter of the user's eyeball.

FIGS. 6A-6Dillustrate user610in user states611-1through611-12and shark avatar605in avatar states612-1through612-12. User610includes various detected physical features (e.g., facial features) including, for example, jaw610-1, eyes610-2, nose610-3, cheeks610-4, eyebrows610-5, and head610-6. In some embodiments, tracked physical features may include other facial features such as eyelids, lips, muscles, muscle groups, and so forth. Shark avatar605includes various shark avatar features including shark jaw605-1, shark eyes605-2, shark nose605-3(e.g., nostrils), shark tail605-4, gills605-5, pectoral fins605-6, dorsal fin605-7, torso605-8, and gums605-9. In some embodiments, various features of shark avatar605comprise a macro-feature of the shark, such as upper head portion605-10(seeFIG. 6C). For example, in some embodiments, upper head portion605-10includes shark eyes605-2, shark nose605-3, and, optionally, upper jaw605-11.

In the embodiments illustrated inFIGS. 6A-6D, the electronic device (e.g.,600) modifies shark avatar605based on one or more of: (1) behavioral characteristics that are used to define the shark avatar, and (2) changes detected in one or more physical features of user610. Behavioral characteristics include modeled behavior of the virtual avatar that is determined for a specific avatar character. This modeled behavior can be dependent on detected changes in the physical features of the user (e.g., if detecting “X” change in user feature “Y,” modify the avatar to do “Z”) or independent of detected changes in the user (e.g., modify avatar to do “A” regardless of what changes are (or are not) detected in the user's physical features). In some embodiments, a behavioral characteristic that is independent of detected changes in the user is referred to herein as an “independent behavioral characteristic.”

For example, in some embodiments, shark tail605-4is an avatar feature that moves (e.g., modified by the electronic device) based on an independent behavioral characteristic of shark avatar605. Specifically, the independent behavioral characteristic is one in which shark tail605-4has a continuous side-to-side motion that simulates a shark's tail movement when swimming. The independence of this behavioral characteristic is demonstrated inFIGS. 6A-6Dby illustrating embodiments in which shark tail605-4moves when no changes are detected in user610(e.g.,FIG. 6A) and embodiments in which shark tail605-4continues to move when various changes are detected in user610(e.g.,FIGS. 6B-6D). Thus, the electronic device modifies shark tail605-4to have a continuous side-to-side motion regardless of what changes are detected in user610, when there are no detected changes in user610, and even when user610is not detected at all (e.g., shark tail605-4continues moving even if face tracking fails).

InFIG. 6A, shark avatar605is shown having three displayed states (612-1,612-2, and612-3), with each of the three displayed states of shark avatar605corresponding, respectively, to three detected states of user610(611-1,611-2, and611-3). For example, the electronic device detects user610in a neutral pose in which user head610-6is facing forward, user jaw610-1is in a closed position, user eyes610-2and nose610-3are in a neutral, front-facing position (e.g., position of eyes and nose is not raised or turned to the side; eyes are looking forward and not up, down, or to the side), and user cheeks610-4and eyebrows610-5are in a neutral, resting position (e.g., cheeks are not puffed out and eyebrows are not raised or lowered). Based on the positions of these detected features of user610, the electronic device displays shark avatar605having a neutral pose in which shark jaw605-1is closed; shark eyes605-2and nose605-3are in a neutral, front-facing position (e.g., position of eyes and nose is not raised or turned to the side; eyes are looking forward and not up, down, or to the side); gills605-5, pectoral fins605-6, dorsal fin605-7, and torso605-8are in a neutral position (e.g., gills are not puffed out, all fins are relaxed and not rotated, lifted, curled, etc., and torso is not flexed to the side); and shark gums605-9are not displayed. This neutral pose of user610is maintained in user states611-1through611-3, and the neutral pose of shark avatar605is maintained in avatar states612-1through612-3. However, despite the neutral, constant pose of user610in user states611-1through611-3, shark tail605-4is modified to have continuous movement as explained in greater detail below.

As discussed above,FIG. 6Aillustrates an embodiment in which the electronic device (e.g.,600) detects no changes in user610(e.g., the electronic device does not detect any changes in user610in user states611-1through611-3) and, therefore, maintains a constant appearance of the shark features that are mapped to the user's features (e.g., the electronic device forgoes modifying all shark avatar features except the tail, because there are no detected changes to user610). The electronic device continues, however, to modify shark avatar605based on the independent behavioral characteristic in which shark tail605-4moves continuously from side-to-side (e.g., at a consistent frequency), as shown in avatar states612-1through612-3. In this specific example, the electronic device displays shark avatar605having shark tail feature605-4that sways back and forth in avatar states612-1through612-3, and no other changes are made to shark avatar605(shark avatar605maintains the neutral pose discussed above). The position of shark tail605-4in avatar states612-1through612-3(as well as avatar states612-4through612-12) is an instant position of the tail for each respective avatar state. However, it should be understood that movement of shark tail605-4is continuous (e.g., a smooth animation of the tail moving side-to-side with an animated curving of the tail that simulates a shark's tail movement).

FIG. 6Billustrates an exemplary embodiment illustrating the electronic device (e.g.,600) modifying shark avatar605in response to detecting changes in facial features of user610. Shark avatar605is shown having three displayed states (612-4,612-5, and612-6), with each of the three displayed states of shark avatar605corresponding, respectively, to three detected states of user610(611-4,611-5, and611-6). In each displayed state inFIG. 6B, the electronic device positions or modifies features of shark avatar605in response to detecting a position, or change in position, of one or more of the physical features, such as facial features, of user610detected in the respective states of the user shown in user states611-4through611-6. In the embodiment shown inFIG. 6B, the electronic device detects user610in the same neutral position as shown inFIG. 6A, but with user eyes610-2shifted to the side. In response to detecting the changed position of user eyes610-2, the electronic device modifies shark eyes605-2based on the changed position of user eyes610-2.

For example, in user state611-4, the electronic device detects user eyes610-2shifted to the right (e.g., from the perspective of the electronic device as shown inFIG. 6B) and modifies shark avatar605in avatar state612-4by shifting shark eyes605-2to the left (e.g., from the perspective of a user viewing shark avatar605as shown inFIG. 6B), effectively mirroring the movement of user eyes610-2. In user state611-5, the electronic device detects no changes from user state611-4and, therefore, maintains the position of shark avatar605in avatar state612-5(with the exception of shark tail605-4, which continues to move independent of the detected changes to user610). In user state611-6, the electronic device detects user eyes610-2shifted to the left and modifies shark avatar605in avatar state612-6by shifting shark eyes605-2to the right to mirror movement of user eyes610-2. In avatar states612-4through612-6, the electronic device does not modify any other features of shark avatar605based on the detected change in user eyes610-2. For example, shark jaw605-1remains in a closed position, and shark nose605-3remains in the neutral, front-facing position. Again, shark tail605-4continues moving, but the movement of the tail is not based on detected changes in user610. In some embodiments, such as that shown inFIG. 6B, shark eyes605-2may be represented as an eyeball having an iris portion. In such embodiments, the shifting of shark eyes605-2, in response to detected movement of user eyes610-2, is a shift of the iris portion of the shark eye. In some embodiments, the shark eyes include only an eyeball that does not include a distinguished iris portion (e.g., the eyeball is solid black). In such embodiments, the shifting of the shark eyes is a shift of the entire eyeball.

FIG. 6Cillustrates an exemplary embodiment illustrating the electronic device (e.g.,600) modifying shark avatar605in response to detecting changes in facial features of user610. Shark avatar605is shown having three displayed states (612-7,612-8, and612-9), with each of the three displayed states of shark avatar605corresponding, respectively, to three detected states of user610(611-7,611-8, and611-9). In each displayed state inFIG. 6C, the electronic device positions or modifies features of shark avatar605in response to detecting a position, or change in position, of one or more of the physical features, such as facial features, of user610detected in the respective states of the user shown in user states611-7through611-9. In the embodiment shown inFIG. 6C, the electronic device detects user610in the same neutral position as shown inFIG. 6A, but with user jaw610-1opening to various positions, while other features of user610(e.g., user eyes610-2, user nose610-3, and user head610-6) remain stationary. In response to detecting the changed position of user jaw610-1, the electronic device modifies shark jaw605-1, shark eyes605-2, and shark nose605-3based on the changed position of user jaw610-1. As illustrated in avatar states612-7through612-9, shark tail605-4continues moving independent of the detected changes to user610.

In user state611-7, the electronic device detects user610having a neutral position (e.g., the same neutral position shown in user state611-1ofFIG. 6A). In response to detecting this neutral position of user610, the electronic device displays shark avatar605(in avatar state612-7) having the neutral position (e.g., the same neutral position shown in avatar state612-1ofFIG. 6A, but with continued movement of shark tail605-4). In the embodiments shown inFIG. 6C, vertical midpoint604and vertical midpoint606are shown as a vertical midpoints of user states611-7through611-9and avatar states612-7through612-9, respectively. In user state611-7, user jaw610-1, user eyes610-2, and user nose610-3have a vertical offset from vertical midpoint604that is represented by J0, E0, and N0, respectively. In other words, J0represents a vertical distance between vertical midpoint604and an initial position of user jaw610-1, E0represents a vertical distance between vertical midpoint604and the position of user eyes610-2, and N0represents a vertical distance between vertical midpoint604and the position of user nose610-3. In avatar state612-7, shark eyes605-2and shark nose605-3have a vertical offset from vertical midpoint606that is represented by e0and n0, respectively. In other words, e0represents a vertical distance between vertical midpoint606and an initial position of shark eyes605-2, and n0represents a vertical distance between vertical midpoint606and an initial position of shark nose605-3.

In user state611-8, the electronic device detects movement of user jaw610-1to an opened position and, in response, modifies shark avatar605in avatar state612-8by lowering shark jaw605-1(e.g., to match the lowered position of user jaw610-1) and shifting the position of shark eyes605-2and shark nose605-3in an upward direction based on the movement of user jaw610-1. The vertical offset of the downward shifted user jaw610-1is represented by J1, which is increased (the absolute value of the vertical offset is increased) with respect to J0. The positions of user eyes610-2and user nose610-3do not change and, therefore, maintain a vertical offset of E0and N0, respectively. The vertical offset of the upward shifted shark eyes605-2and shark nose605-3is represented by e1and n1, respectively, which are increased with respect to e0and n0, respectively. The vertical offset of the lowered shark jaw605-1is represented by j1, which is a vertical offset from vertical midpoint606that is in an opposite direction (e.g., downward) from vertical (e.g., upward) offsets n0, n1, e0, and e1. In some embodiments, vertical midpoint606also represents the original vertical position of shark jaw605-1(e.g., in avatar state612-7, the vertical position of shark jaw605-1is equal to vertical midpoint606). In some embodiments, the difference between n0and n1is equal to the difference between e0and e1. In some embodiments, the difference between n0and n1is not equal to the difference between e0and e1. In some embodiments, the difference between n0and n1is equal to j1. In some embodiments, the difference between e0and e1is equal to j1. In some embodiments, J1is equal to j1. In some embodiments, J1is not equal to j1(e.g., j1is a scaled value of J1). In some embodiments, the difference between J0and J1is equal to j1. In some embodiments, the electronic device also modifies the shape of an upper head portion of shark avatar605to accommodate the shifted movement of shark eyes605-2and shark nose605-3, as shown in the changes to upper head portion605-10of shark avatar605in avatar states612-7and612-8. In some instances, modifying upper head portion605-10results in hiding varying amounts of dorsal fin605-7, depending on the magnitude of the modification to upper head portion605-10, as shown in avatar states612-8and612-9. In some embodiments, the electronic device also modifies shark avatar605by moving upper jaw605-11(e.g., moving upper jaw in an upward direction based on the detected downward movement of user jaw610-1).

In user state611-9, the electronic device detects movement of user jaw610-1to an opened position that is greater than that shown in611-8and, in response, modifies shark avatar605in avatar state612-9by further lowering shark jaw605-1(e.g., to match the farther lowered position of user jaw610-1in user state611-9) and further shifting the position of shark eyes605-2and shark nose605-3in an upward direction based on the additional movement of user jaw610-1. The vertical offset of the lowered user jaw610-1is represented by J2, which is increased with respect to J1. The positions of user eyes610-2and user nose610-3do not change and, therefore, maintain a vertical offset of E0and N0, respectively. The vertical offset of the shifted shark eyes605-2and shark nose605-3is represented by e2and n2, respectively, which are increased with respect to e1and n1, respectively. The vertical offset of the lowered shark jaw605-1is represented by j2, which is increased (e.g., in a downward direction) with respect to j1. In some embodiments, the difference between n1and n2is equal to the difference between e1and e2. In some embodiments, the difference between n1and n2is not equal to the difference between e1and e2. In some embodiments, the difference between n1and n2is equal to the difference between j1and j2. In some embodiments, the difference between e1and e2is equal to the difference between j1and j2. In some embodiments, the difference between J1and J2is equal to the difference between j1and j2. In some embodiments, the electronic device also modifies the shape of an upper head portion of shark avatar605to accommodate the shifted movement of shark eyes605-2and shark nose605-3, as shown in the changes to upper head portion605-10of shark avatar605in avatar states612-7and612-8. In some embodiments, the electronic device also modifies shark avatar605by moving upper jaw605-11in an upward direction (e.g., compared to the position of upper jaw605-11in avatar state612-8and based on the detected downward movement of user jaw610-1). In some embodiments, the electronic device modifies shark avatar605to display gums605-9as shown in avatar state612-9(e.g., in response to detecting movement of user jaw610-1beyond a threshold amount of downward movement (e.g., openness)). In some embodiments, in response to detecting user jaw610-1moving in a closing direction, the electronic device modifies shark avatar605by reversing the above-described movements of the avatar features (e.g., transitioning from the appearance in avatar state612-9to the appearance in avatar state612-8; transitioning from the appearance in avatar state612-8to the appearance in avatar state612-7).

In some embodiments, such as that shown inFIG. 6C, shark eyes605-2may be represented as an eyeball having an iris portion. In such embodiments, the shifting of shark eyes605-2, in response to detected movement of user jaw610-1, is a shift of the entire eyeball (including the iris portion), while maintaining a relative position of the iris portion in the eyeball. In some embodiments, the shark eyes include only an eyeball that does not include a distinguished iris portion (e.g., the eyeball is solid black). In such embodiments, the shifting of the shark eyes is a shift of the entire eyeball.

FIG. 6Dillustrates an exemplary embodiment illustrating the electronic device (e.g.,600) modifying shark avatar605in response to detecting changes in facial features of user610. Shark avatar605is shown having three displayed states (612-10,612-11, and612-12), with each of the three displayed states of shark avatar605corresponding, respectively, to three detected states of user610(611-10,611-11, and611-12). In each displayed state inFIG. 6D, the electronic device positions or modifies features of shark avatar605in response to detecting a position, or change in position, of one or more of the physical features, such as facial features, of user610detected in the respective states of the user shown in user states611-10through611-12. In the embodiment shown inFIG. 6D, the electronic device detects user610in the same neutral position as shown inFIG. 6A, but with user head610-6turning to various positions. In response to detecting the changed position of user head610-6, the electronic device modifies various features of shark avatar605based on the changed positions of user head610-6. As illustrated in avatar states612-10through612-12, shark tail605-4continues moving independent of the detected changes to user610.

In user state611-10, the electronic device detects user610having a neutral position (e.g., the same neutral position shown in user state611-1ofFIG. 6A). In response to detecting this neutral position of user610, the electronic device displays shark avatar605(in avatar state612-10) having the neutral position (e.g., the same neutral position shown in avatar state612-1ofFIG. 6A, but with continued movement of shark tail605-4).

In user state611-11, the electronic device detects a rightward rotation (e.g., from the perspective of the electronic device as shown inFIG. 6D) of user head610-6and, optionally, face (or various physical features comprising the face such as user jaw610-1, user eyes610-2, user nose610-3, user cheeks610-4, and user eyebrows610-5). In response, the electronic device modifies shark avatar605by flexing shark torso605-8to the left (e.g., from the perspective of a user viewing shark avatar605as shown inFIG. 6D) and modifying the appearance of various avatar features based on the flexed position of shark avatar605, as shown in avatar state612-11. For example, the electronic device modifies shark torso605-8by flexing it to the left, effectively mirroring the rotation of user head610-6. As shown in avatar state612-11, the flexing of shark torso605-8includes rotating the position of the torso and attached features such as shark jaw605-1, shark eyes605-2, shark nose605-3, gills605-5, pectoral fins605-6, and dorsal fin605-7. Because shark torso605-8is flexed (e.g., rotated without affecting the displayed shark tail605-4), various shark avatar features are hidden from view and, thus, the electronic device omits these features from shark avatar605. For example, a portion of shark jaw605-1that was previously displayed in avatar state612-10is not displayed in avatar state612-11. Similarly, only one shark eye605-2, a portion of shark nose605-3(e.g., one nostril), a portion of pectoral fin605-6B, and one set of gills605-5are displayed in avatar state612-11. Conversely, portions of various shark features that were previously hidden from view are now displayed due to the rotation of shark torso605-8. For example, a greater portion of shark torso605-8, dorsal fin605-7, and pectoral fin605-6A is displayed due to the rotated position of shark torso605-8.

In user state611-12, the electronic device detects a leftward rotation (e.g., from the perspective of the electronic device as shown inFIG. 6D) of user head610-6and, optionally, face (or various physical features comprising the face such as user jaw610-1, user eyes610-2, user nose610-3, user cheeks610-4, and user eyebrows610-5). In response, the electronic device modifies shark avatar605by flexing shark torso605-8to the right (e.g., from the perspective of a user viewing shark avatar605as shown inFIG. 6D) and modifying the appearance of various avatar features based on the flexed position of shark avatar605, as shown in avatar state612-12. It should be appreciated that the appearance of user610in user state611-12is a mirror image of the appearance of user610in user state611-11. Therefore, the appearance of shark avatar605is modified in avatar state612-12to be a mirror image of the appearance of shark avatar605in avatar state612-11. As illustrated in avatar states612-10through612-12, shark tail605-4continues moving independent of the detected changes to user610. That is, the flexing of shark torso605-8does not affect shark tail605-4.

In the examples provided inFIGS. 6A-6D, the electronic device detects the user transitioning between various positions of the user's physical features, and then updates or modifies the virtual avatar (e.g., shark avatar) in accordance with the detected changes in the user's physical features. In these examples, the electronic device transitions display of the virtual avatar between various expressions and positions by maintaining characteristics of certain features of the virtual avatar and animating changes to other virtual avatar features. For example, as shown inFIG. 6B, the device maintains the position of shark jaw605-1while modifying other avatar features, such as shark eyes605-2. It should be appreciated that while states of the avatar correspond to the respective detected states of the user (e.g., displayed avatar states612-4through612-6correspond to user states611-4through611-6, respectively), the order of the detected user states (and resulting avatar states) is not limited to that shown inFIGS. 6A-6D. Accordingly, the user can change any physical features or assume any desired positions, and the electronic device modifies the virtual avatar in response to the detected changes, regardless of the order in which they occur.

For example, in addition to modifying the various avatar features as shown inFIG. 6C, the electronic device can further modify one or more of these features based on additional changes detected in user610. For example, in response to detecting user610having the appearance shown in user state611-8, the electronic device modifies shark avatar605to have the appearance shown in avatar state612-8. If the electronic device then detects a rightward shift of user eyes610-2(e.g., similar to the eye shift shown in user state611-4) while continuing to maintain the opened jaw position in user state611-8, the electronic device modifies shark eyes605-2in avatar state612-8to have a leftward shift (e.g., the iris of shark avatar has a shift similar to that shown in avatar state612-4), while maintaining the upward shifted position of shark eyes605-2and shark nose605-3and maintaining the position of the other avatar features (e.g., shark jaw605-1, gills605-5, fins605-6and605-7, torso605-8, upper head portion605-10, and upper jaw605-11) shown in avatar state612-8.

In some embodiments, the electronic device can also modify shark avatar605in various manners other than those illustrated inFIGS. 6A-6D. For example, in response to detecting upward or downward movement of user eyebrows610-5, the electronic device can modify pectoral fins605-5to move (e.g., curl) in an upward or downward direction based on the detected upward or downward movement of user eyebrows610-5. As another example, the electronic device can modify gills605-5to expand and compress (similar to movement of a shark's gills when breathing) in response to detecting user cheeks610-4expanding (e.g., puffing out when filled with air) and returning to a neutral position (e.g., when the user lets the air out of their cheeks). As yet another example, in some embodiments, in response to detecting a translational movement of the position of user610in the field of view of the camera (e.g.,602) (e.g., the position of user610moves up, down, left, or right in the field of view of the camera), the electronic device modifies the displayed position of shark avatar605in the display region to mirror the movement of the position of user610in the field of view of the camera. For example, if the electronic device detects the user shifted downward in the field of view, the electronic device shifts the position of shark avatar605downward in the display region. In some embodiments, the electronic device fixes or anchors the position of shark avatar605such that the position of shark avatar605in the display region does not change in response to detected translational movement of the position of user610.

FIGS. 6E and 6Fdepict an example embodiment in which device600, having display601and camera602, enables display of shark avatar605in a camera application user interface615. Camera application interface615includes image display region620, which displays a representation of image data such as, for example, streamed image data (e.g., a live camera preview, live camera recording, or live video communications session) representing objects positioned within a field-of-view of a camera (e.g., a rear-facing camera or camera602), or a media item such as, for example, a photograph or a video recording. It should be appreciated that device600may display shark avatar605similarly in the context of various other applications such as, for example, a messaging application, a media viewer application, or a video communication application. Therefore, the display of shark avatar605, as discussed with respect toFIGS. 6A-6F, should be understood to apply similarly to a virtual avatar displayed in the context of these other applications. For the sake of brevity, details of the display of the virtual avatar in these various other application is not discussed herein.

In the embodiment illustrated inFIG. 6E, image display region620shows live camera preview620-1from camera602, showing a representation of subject632positioned in the field-of-view of camera602and background636displayed behind subject632. In the embodiments discussed herein, subject632corresponds to user610discussed inFIGS. 6A-6D. As discussed herein, image data captured using camera602includes, in some embodiments, depth data that can be used to determine a depth of objects in the field-of-view of camera602. In some embodiments, device600parses objects (e.g., in image data) based on a detected depth of those objects, and uses this determination to apply the visual effects (e.g., virtual avatars) discussed herein. For example, device600can categorize subject632as being in the foreground of the live camera preview620-1and objects positioned behind the user as being in the background of the live camera preview620-1. These background objects are referred to generally herein as background636.

InFIG. 6E, device600displays live camera preview620-1without shark avatar605(e.g., device600is operating in a mode in which visual effects have not been enabled, or the virtual avatar has not been selected for applying to the representation of subject632). InFIG. 6F, however, device600displays live camera preview620-1with shark avatar605displayed over a portion (e.g., head) of the representation of subject632(e.g., visual effects have been enabled, or the virtual avatar has been selected for applying to the representation of subject632). Specifically, shark avatar605is transposed onto the face of the subject in the image display region, while other portions of the image in image display region620(such as a background or other portions of the user, such as their body) remain displayed. A user (e.g., subject632) positioned in the field-of-view of camera602can control visual aspects of shark avatar605by changing the pose (e.g., rotation or orientation) of their face, including moving various facial features (e.g., winking, sticking out their tongue, smiling, etc.), as discussed in greater detail with respect toFIGS. 6A-6D.

In some embodiments, portions of shark avatar605are conditionally displayed based on a simulated position (e.g., depth) of the avatar portions relative to a depth and/or position of the subject in the field of view of camera602. For example, inFIG. 6F, device600omits shark tail605-4from shark avatar605, because shark tail605-4is positioned behind the representation of subject632, and therefore, is not visible to a user viewing display601. In some embodiments, portions of shark tail605-4can be displayed if the simulated relative position of the tail changes such that the tail becomes visible based on an updated relative position of the tail with respect to the representation of the subject632. For example, if the user turns their head to the side and the resulting flexing of shark torso605-8results in a position change that reveals a portion of shark tail605-4, then shark tail605-4may be displayed. In such instances, the continual motion of shark tail605-4is maintained, even if it results in intermittent visibility of the tail (e.g., caused by the continuous tail motion).

FIG. 7is a flow diagram illustrating a method for displaying visual effects using an electronic device in accordance with some embodiments. Method700is performed at a device (e.g.,100,300,500,600) with one or more cameras (e.g.,602) and a display (e.g.,601). Some operations in method700are, optionally, combined, the orders of some operations are, optionally, changed, and some operations are, optionally, omitted.

As described below, method700provides an intuitive way for displaying visual effects such as virtual avatars. The method reduces the cognitive burden on a user for generating, modifying, and controlling virtual avatars, thereby creating a more efficient human-machine interface. For battery-operated computing devices, enabling a user to control the virtual avatar faster and more efficiently conserves power and increases the time between battery charges.

The electronic device (e.g.,600) displays (702), via the display apparatus (e.g.,601), a virtual avatar (e.g., shark avatar605) (e.g., a representation of the user that can be graphically depicted) having a plurality of avatar features (e.g., shark jaw605-1, shark eyes605-2, shark nose605-3, shark tail605-4, gills605-5, pectoral fins605-6, dorsal fin605-7, shark torso605-8, shark gums605-9, upper head portion605-10, upper jaw605-11) (e.g., facial features (e.g., eyes, mouth, part of mouth (gums, tongue), teeth); e.g., a body part (e.g., fins, tail, gills); e.g., a macro feature (e.g., head, neck, body)) that change appearance in response to detected changes in pose (e.g., orientation, translation) (e.g., a change in a facial expression) of a face in a field of view of the one or more cameras (e.g.,602). In some embodiments, the virtual avatar includes an avatar face having one or more avatar features (e.g., avatar facial features). In some embodiments, the avatar features correspond (e.g., are mapped) to one or more physical features of a user's face such that detected movement of the user's physical feature(s) affects the avatar feature (e.g., affects the graphical representation of the features). In some embodiments, an avatar feature anatomically corresponds to a physical feature (e.g., the avatar feature is modelled based on one or more of the location, movement characteristics, size, color, and/or shape of the physical feature) (e.g., the avatar feature and physical feature are both eyebrows). In some embodiments, an avatar feature does not anatomically correspond (e.g., is anatomically distinct or anatomically corresponds to a different physical feature) to a physical feature (e.g., the avatar feature corresponds to a shark fin (e.g.,605-6) and the physical feature corresponds to a user's eyebrow (e.g.,610-5)). In some embodiments, the virtual avatar is non-photorealistic. In some embodiments, the avatar is an anthropomorphic construct such as stylized animal (e.g., shark).

In method700, electronic device (e.g.,600) displays (704) the plurality of avatar features including: a first avatar feature (e.g., shark jaw605-1) (e.g., a jaw; e.g., a lower jaw and, optionally, an upper jaw portion such as upper teeth and/or upper gums); a second avatar feature (e.g., shark eyes605-2) (e.g., avatar eyes) different from the first avatar feature; and a third avatar feature (e.g., an avatar nose) (e.g., shark nose605-3) different from the first and second avatar features. In some embodiments, the second and third avatar features form a first portion of the avatar such as a head portion (e.g., upper head portion605-10) that does not include the first avatar feature (e.g., a lower jaw)).

In method700, while the face is detected in the field of view of the one or more cameras (e.g.,602), the face including a plurality of detected facial features including a first facial feature (e.g., jaw610-1of user610; e.g., a lower jaw that does not include the upper jaw, upper lip, upper teeth, upper gums, or nose) (e.g., the first facial feature corresponds (e.g., anatomically) to the first avatar feature) (e.g., the first avatar feature is a graphical representation of the first facial feature) (e.g., the first facial feature is a jaw of the user and the first avatar feature is jaw of the avatar) and a second facial feature (e.g., user eyes610-2) (e.g., one or more of the user's eyes) (e.g., the second facial feature corresponds (e.g., anatomically) to the second avatar feature) (e.g., the second avatar feature is a graphical representation of the second facial feature) (e.g., the second facial feature is one or more of the user's eyes and the second avatar feature is one or more avatar eyes) different from the first facial feature, the electronic device (e.g.,600) detects (706) movement of one or more facial features of the face.

In some embodiments, in response to detecting the movement of the one or more facial features, and in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature (e.g., user jaw610-1) in a third direction (e.g., the second direction; e.g., a direction different from (e.g., opposite from) the first direction; e.g., movement of the first facial feature in an upward direction such as upward movement of a jaw based on closing of a mouth), the electronic device (e.g.,600) moves the second avatar feature (e.g., shark eyes605-2) and the third avatar feature (e.g., shark nose605-3) in a fourth direction different (e.g., opposite) from the third direction and based on (e.g., based on a magnitude of) the detected movement of the first facial feature in the third direction. Moving the second avatar feature and the third avatar feature in the fourth direction different from the third direction and based on the detected movement of the first facial feature in the third direction provides the user with feedback indicating that movement of the first facial feature in various directions will cause the device to modify the respective second and third avatar features in different various directions than those in which the first facial feature moves. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the first facial feature that does not correspond to the second and third avatar features, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, further in accordance with the determination that the detected movement of the one or more facial features includes movement of the first facial feature (e.g., user jaw610-1) in the third direction, the electronic device (e.g.,600) moves the first avatar feature (e.g., shark jaw605-1) (e.g., avatar lower jaw) in the third direction based on (e.g., based on a magnitude of) the detected movement of the first facial feature in the third direction. Moving the first avatar feature in the third direction based on the detected movement of the first facial feature in the third direction provides the user with feedback indicating that further movement of the same first facial feature in various directions will cause the device to further modify the respective first avatar feature in the same various directions. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the first facial feature, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In method700, the electronic device (e.g.,600) performs (708) one or more of the items described below in response to detecting the movement of the one or more facial features.

In accordance with a determination (710) that the detected movement of the one or more facial features includes movement of the first facial feature (e.g., user jaw610-1) in a first direction (e.g., movement of the first facial feature in a downward direction such as downward movement of a jaw based on opening of a mouth), the electronic device (e.g.,600) performs one or more of the steps discussed below with respect to elements712and714of method700. In some embodiments, the detected movement of the first facial feature has both a directional component and a magnitude component. In some embodiments, modifications to an avatar feature have both a magnitude component and a directional component. In some embodiments, the directional component of the modification in the avatar feature is based on a directional component of a change in the one or more physical features (e.g., facial features of the user's face) that the avatar feature is reactive to. In some embodiments, the directional component of the change in the avatar feature is the same as the directional component of the change in the physical feature. For example, when the physical feature (e.g., jaw) moves down, the corresponding (e.g., reactive) avatar feature (e.g., avatar jaw) moves down. In some embodiments, the directional component of the change in the avatar feature is mirrored with respect to the directional component of the change in a corresponding physical feature (e.g., the physical feature for which the avatar feature is reactive to detected changes). For example, when the physical feature (e.g., a user's eye (e.g., iris)) moves left, the reactive avatar feature (e.g., avatar eye (e.g., iris)) moves right. In some embodiments, the directional component of the change in the avatar feature is the same as the directional component of the change in the corresponding physical feature for movement along a vertical axis and mirrored for movement along a horizontal axis, similar to the effect seen when looking in a mirror. In some embodiments, a change in relative position of a physical feature (e.g., the user's iris or eyebrow) is in a direction determined from a neutral, resting position of the physical feature. In some embodiments, the neutral, resting position of a user's iris is determined to be a particular position (e.g., centered) relative to the perimeter of the user's eyeball. In some embodiments, the direction of a reaction of an avatar feature corresponds (e.g., directly or inversely) to a relative direction of a change in a physical feature of the user. In some embodiments, the relative direction of the change in the physical feature is determined based on a direction of movement of the physical feature from a neutral, resting position of the physical feature. In some embodiments, the direction of the reaction of the avatar feature corresponds directly (e.g., the physical feature moves up, the avatar feature moves up) to the relative direction of the change in the physical feature. In some embodiments, the direction of the reaction of the avatar feature corresponds inversely (e.g., the physical feature moves up, the avatar feature moves down) to the relative direction of the change in the physical feature.

In some embodiments, the first facial feature is a jaw (e.g., user jaw610-1) (e.g., a lower jaw) of the user (e.g., user610); the first avatar feature is an avatar jaw feature (e.g., shark jaw605-1) (e.g., a lower avatar jaw); the second avatar feature is one or more avatar eyes (e.g., shark eyes605-2); and the third avatar feature is an avatar nose (e.g., shark nose605-3).

In response to detecting the movement of the one or more facial features, and in accordance with the determination that the detected movement of the one or more facial features includes movement of the first facial feature (e.g., user jaw610-1) in the first direction, the electronic device (e.g.,600) moves (712) the first avatar feature (e.g., shark jaw605-1) in the first direction based on (e.g., based on a magnitude) the detected movement of the first facial feature in the first direction (e.g., a position of the first avatar feature (e.g., avatar jaw) is determined based on the detected movement of the first facial feature (e.g., the user's jaw). Moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction provides the user with feedback indicating that further movement of the same first facial feature will cause the device to further modify the respective first avatar features in the first direction. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the first facial feature, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, the direction of movement of the first avatar feature (e.g., shark jaw605-1) is in a same direction (e.g., downward) as the detected movement of the first facial feature (e.g., user jaw610-1). In some embodiments, the movement of the first avatar feature has a magnitude (e.g., an amount of movement) of movement in the same direction that is determined based on (e.g., based on a proportional amount of movement of the feature with respect to a range of motion available for the respective feature) the magnitude of movement of the first facial feature as the first facial feature moves in the first direction. In some embodiments, the magnitude of a change in an avatar feature corresponds to a magnitude of a change in a physical feature of the user. In some embodiments, the magnitude of the change in the physical feature is determined in accordance with a potential range of motion of the physical feature, wherein the magnitude is representative of a relative position of the physical feature within the range of motion (e.g., the predicted or modeled range of motion) of that physical feature. In such embodiments, the magnitude of the reaction (e.g., change in position) of the avatar feature is similarly a relative position of the avatar feature within a range of motion of the avatar feature. In some embodiments, the magnitude of change is determined based on a comparison or measurement (e.g., a distance) of the starting position and ending position of the physical feature, through the change. In such embodiments, the change in the physical feature (e.g., first facial feature) is translated to a modification of the first avatar feature by applying the measured change in the physical feature to the avatar feature (e.g., directly or as a scaled or adjusted value).

In some embodiments, moving the first avatar feature (e.g., shark jaw605-1) in the first direction based on the detected movement of the first facial feature (e.g., user jaw610-1) in the first direction includes: in accordance with a determination that the detected movement of the first facial feature is less than or equal to a second threshold amount of movement of the first facial feature (e.g., the detected movement (e.g., a degree of change in the position of the first facial feature) of the user's jaw is less than a threshold amount of movement (e.g., less than 75% of a range of openness (e.g., a maximum open position) of the user's mouth) (e.g., user state611-8), moving the first avatar feature in the first direction (e.g., in a downward direction) by a first amount (e.g., a first magnitude of movement) determined based on the detected movement of the first facial feature (e.g., moving the lower avatar jaw portion downward in response to downward movement of the user's jaw) (e.g., avatar state612-8). In some embodiments, when movement of the first facial feature (e.g., the user's jaw) is within a first range of movement (e.g., less than 75% of a maximum open position of the user's mouth), the first avatar feature is reactive to movement of the first facial feature (e.g., mirroring movement of the first facial feature). For example, when a user opens their mouth (e.g., downward movement of the user's lower jaw) by less than 75% of a maximum open position of the user's mouth (e.g., less than 75% of a maximum downward position of the user's lower jaw) (e.g., user state611-8), an avatar lower jaw portion moves in the same downward direction as the user's lower jaw when opening (e.g., avatar state612-8).

In some embodiments, moving the first avatar feature (e.g., shark jaw605-1) in the first direction based on the detected movement of the first facial feature (e.g., user jaw610-1) in the first direction further includes: in accordance with a determination that the detected movement of the first facial feature is greater than the second threshold amount of movement of the first facial feature: moving the first avatar feature in the first direction by a second amount of movement greater than the first amount of movement, and displaying a first subfeature (e.g., shark gums605-9) (e.g., avatar gums) of the first avatar feature that was not displayed when the movement of the first facial feature was less than the second threshold amount of movement (e.g., inFIG. 6C, shark gums605-9are not displayed in avatar state612-8, but are displayed in avatar state612-9) (e.g., avatar gums are displayed on the avatar jaws when the detected movement of the first facial feature is greater than the threshold amount of movement). In some embodiments, when movement of the first facial feature (e.g., the user's jaw) is within a second range of movement (e.g., greater than 75% of a maximum open position of the user's mouth) (e.g., user state611-9), the first avatar feature is reactive to movement of the first facial feature (e.g., mirroring movement of the first facial feature), and the avatar (e.g., shark avatar605) is modified to introduce gums or additional teeth on the jaws of the avatar. For example, when a user opens their mouth by more than 75% of a maximum open position of the user's mouth (e.g., user state611-9), the avatar jaw moves in the same downward direction as the user's lower jaw when opening, but the avatar jaws are modified to display the avatar's gums and/or additional teeth (e.g., avatar state612-9). In some embodiments, the avatar's mouth has a first range of openness when the user's mouth is opened less than or equal to the threshold position, and has a second range of openness (greater than the first range of openness) when the user's mouth is opened greater than the threshold position. In some embodiments, the avatar is a shark, and this introduction of the gums on the avatar shark's jaws mimic a behavior in which a shark's jaws open wide (e.g., to attack prey) revealing additional parts of the shark's mouth.

Moving the first avatar feature in the first direction by the first or second amount determined based on the detected movement of the first facial feature compared to a threshold amount of movement for the first facial feature, and further displaying the first subfeature of the first avatar feature when the detected movement of the first facial feature is greater than the threshold amount of movement, provides the user with feedback indicating that different amounts of movement of the first facial feature will cause the device to modify the first avatar feature differently depending on whether the amount of movement of the first facial feature is above or below the threshold amount of movement. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the first facial feature, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In response to detecting the movement of the one or more facial features, and in accordance with the determination that the detected movement of the one or more facial features includes movement of the first facial feature (e.g., user jaw610-1) in the first direction, the electronic device (e.g.,600) moves (714) the second avatar feature (e.g., shark eyes605-2) and the third avatar feature (e.g., shark nose605-3) in a second direction different (e.g., opposite) from the first direction and based on (e.g., based on a magnitude of) the detected movement of the first facial feature in the first direction (e.g., a position of the second avatar feature (e.g., the avatar eyes) is determined based on the detected movement of the first facial feature (e.g., the user's jaw)). Moving the second avatar feature and the third avatar feature in the second direction different from the first direction and based on the detected movement of the first facial feature in the first direction provides the user with feedback indicating that further movement of the same first facial feature will cause the device to further modify the respective second and third avatar features in the second direction different from the first direction. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the first facial feature that does not correspond to the second and third avatar features, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, the direction of movement of the second avatar feature (e.g., avatar eyes) (e.g., shark eyes605-2) is in a substantially opposite direction from the detected movement of the first avatar feature (e.g., avatar jaw) (e.g., shark jaw605-1), but has a magnitude (e.g., an amount) of movement in the opposite direction that is determined based on the magnitude of movement of the first facial feature as the first facial feature moves in the first direction (e.g., a position of the third avatar feature (e.g., the avatar nose) is determined based on the detected movement of the first facial feature (e.g., the user's jaw)). In some embodiments, the direction of movement of the third avatar feature (e.g., avatar nose) (e.g., shark nose605-3) is in a substantially opposite direction from the detected movement of the first avatar feature (e.g., avatar jaw), but has a magnitude (e.g., an amount) of movement in the opposite direction that is determined based on the magnitude of movement of the first facial feature as the first facial feature moves in the first direction. In some embodiments, the positions of both the second and third avatar features move simultaneously in the second direction in response to detecting movement of the first facial feature in the first direction (e.g., as the user's jaw moves in a downward direction, the avatar eyes and avatar nose both move in an upward direction).

In response to detecting the movement of the one or more facial features, and in accordance with a determination (716) that the detected movement of the one or more facial features includes movement of the second facial feature (e.g., user eyes610-2) (e.g., without detected movement of the first facial feature (e.g., user jaw610-1)), the electronic device (e.g.,600) moves (718) the second avatar feature (e.g., shark eyes605-2) (e.g., avatar eyes) based on at least one of a direction and a magnitude of movement of the second facial feature. Moving the second avatar feature based on at least one of the direction and the magnitude of movement of the second facial feature provides the user with feedback indicating that further movement of the same second facial feature will cause the device to further modify the respective second avatar feature based on at least one of the direction and magnitude of the further movement of the second facial feature. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the second facial feature, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, method700further includes: in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature (e.g., user eyes610-2) without more than a first threshold amount of movement of the first facial feature (e.g., user jaw610-1) (e.g., without detected movement of the first facial feature; e.g., with a nominal amount of detected movement of the first facial feature) (e.g., seeFIG. 6B), the electronic device (e.g.,600) moves the second avatar feature (e.g., avatar eyes) based on at least one of the direction and the magnitude of movement of the second facial feature; and foregoes moving the first and third avatar features (e.g., as shown inFIG. 6B). Moving the second avatar feature based on at least one of the direction and the magnitude of the movement of the second facial feature without more than the first threshold amount of movement of the first facial feature and foregoing movement of the first and third avatar features provides the user with feedback indicating that isolated movement of the second facial feature will cause the device to modify the second avatar feature without also modifying the first and third avatar features. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the second facial feature (without a threshold amount of movement of the first facial feature), and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, method700further includes: in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature (e.g., user jaw610-1) (e.g., in the first direction) without more than a second threshold amount of movement of the second facial feature (e.g., user eyes610-2) and a third facial feature (e.g., user nose610-3) (e.g., the user's nose) (e.g., an affirmative determination that the second and third facial features do not move), the electronic device (e.g.,600) moves the second avatar feature (e.g., shark eyes605-2) and the third avatar feature (e.g., shark nose605-3) based on the detected movement of the first facial feature (e.g., moving the second avatar feature and the third avatar feature in a direction different (e.g., opposite) from the direction of movement of the first facial feature and based on (e.g., based on a magnitude of) the detected movement of the first facial feature). Moving the second and third avatar features based on the detected movement of the first facial feature without more than the second threshold amount of movement of the second facial feature and the third facial feature provides the user with feedback indicating that isolated movement of the first facial feature will cause the device to modify the second and third avatar features based on the movement of the first facial feature. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the first facial feature (without a threshold amount of movement of the second and third facial features), and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, method700further includes: in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of both the first and second facial features (e.g., user jaw610-1, user eyes610-2) (e.g., movement of both the user's jaw and one or more of the user's eyes), the electronic device (e.g.,600) moves the first avatar feature (e.g., shark jaw605-1) (e.g., the avatar jaw) based on the detected movement of the first facial feature (e.g., based on a direction and/or magnitude of movement of the user's jaw) (e.g., without being based on movement of the second facial feature); and moves the second avatar feature (e.g., shark eyes605-2) (e.g., one or more of the avatar's eyes) based on the detected movement of the first and second facial features (e.g., based on a direction and/or magnitude of movement of the user's eye(s) and based on a direction and/or magnitude of movement of the user's jaw). Moving the first avatar feature based on the detected movement of the first facial feature and moving the second avatar feature based on the detected movement of the first and second facial features provides the user with feedback indicating that further movement of the same first and second facial features will cause the device to further modify the respective first avatar feature based on the further movement of the first facial feature and further modify the respective second avatar feature based on the further movement of the first and second facial features. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the first and second facial features, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, method700further includes: in response to detecting the movement of the one or more facial features, performing one or more of the following items. In accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial (e.g., user jaw610-1) feature by a first magnitude (e.g., a first amount of movement; e.g., an amount of movement of the first facial feature in a downward direction such as downward movement of a jaw based on opening of a mouth from a closed position to a 25% opened position), moving the second avatar feature (e.g., shark eyes605-2) and the third avatar feature (e.g., shark nose605-3) by a first amount of movement (e.g., a same amount of movement for both the second avatar feature and the third avatar feature) (e.g., a first amount of movement that is proportional (e.g., scaled or directly) to the magnitude of the first magnitude of movement of the first facial feature). In accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature by a second magnitude different from the first magnitude (e.g., a second amount of movement; e.g., an amount of movement of the first facial feature in a downward direction such as downward movement of a jaw based on opening of a mouth from a closed position to a half-opened position), moving the second avatar feature and the third avatar feature by a second amount of movement that is different from the first amount of movement (e.g., a same amount of movement for both the second avatar feature and the third avatar feature) (e.g., a second amount of movement that is proportional (e.g., scaled or directly) to the magnitude of the second magnitude of movement of the first facial feature) (e.g., magnitude of movement of second and third avatar features is proportional to the magnitude of movement of the user's jaw. In some embodiments, the magnitude of movement of the second and third avatar features are the same when jaw movement occurs without independent movement of facial features that are separately mapped to the second and/or third avatar features.

Moving the second and third avatar features by the first and second amounts of movement depending on whether detected movement of the first facial feature is by a first magnitude or second magnitude provides the user with feedback indicating that movement of the first facial feature by different magnitudes will cause the device to modify the respective second and third avatar features by different amounts determined based on the different magnitudes of movement of the first facial feature. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the first facial feature that does not correspond to the second and third avatar features, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, method700further includes: further in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature (e.g., user jaw610-1) by the first magnitude, moving the first avatar feature (e.g., shark jaw605-1) by a third amount of movement (e.g., an amount of movement that corresponds to (e.g., is equal to) the amount of movement of the first facial feature) (e.g., an amount of movement that is proportional (e.g., scaled or directly) to the first magnitude of the movement of the first facial feature); and further in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature by the second magnitude, moving the first avatar feature by a fourth amount of movement different from the third amount of movement (e.g., an amount of movement that corresponds to (e.g., is equal to) the amount of movement of the first facial feature) (e.g., an amount of movement that is proportional (e.g., scaled or directly) to the second magnitude of the movement of the first facial feature). Moving the first avatar feature by the third or fourth amounts of movement depending on whether detected movement of the first facial feature is by the first or second magnitude provides the user with feedback indicating that movement of the first facial feature by different magnitudes will cause the device to modify the respective first avatar feature by different amounts determined based on the different magnitudes of movement of the first facial feature. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the first facial feature, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, the electronic device (e.g.,600) detects a change in pose (e.g., position and/or orientation) of the face within the field of view of the one or more cameras (e.g.,602). In response to detecting the change in pose of the face, the electronic device changes an appearance of the virtual avatar (e.g., shark avatar605). In some embodiments, changing the appearance of the virtual avatar includes: in accordance with a determination that the change in pose of the face includes a first type of change in pose of the face (e.g., a change in orientation of the face; e.g., the user shaking or turning their head to the side or nodding or tilting their head up or down) (e.g., user states611-11or611-12), changing the appearance of the virtual avatar includes moving a first portion (e.g., a shark head portion of a shark avatar (e.g., upper head portion605-10)) (in some embodiments, the first portion optionally includes one or more fins (e.g., pectoral fins605-6; dorsal fin605-7)) of the virtual avatar relative to a second portion (e.g., a shark torso portion of a shark avatar (e.g.,605-8)) (e.g., avatar states612-11or612-12). Moving the first portion of the virtual avatar relative to the second portion when the change in pose of the face is a first type of change in pose provides the user with feedback indicating that different types of changes in pose of the face will cause the device to modify various portions of the virtual avatar differently depending on the types of changes in pose of the face. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of types of changes to facial features of a user (and the magnitude and/or direction of those changes), and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, the second portion optionally includes one or more fins (e.g., dorsal fin605-7, pelvic fins605-6)) of the virtual avatar (e.g., shark avatar605). In some embodiments, the movement of the first portion relative to the second portion occurs without moving the second portion relative to the displayed frame of view (e.g., the first portion is upper head portion605-10and the second portion is shark torso605-8). That is, the first portion moves within the displayed frame of view whereas the second portion remains fixed or substantially fixed within the displayed frame of view. In some embodiments, the movement of the first portion is a rotational movement around an axis, such as a rotation around a y-axis (e.g., vertical axis, as displayed). In some embodiments, the virtual avatar is a shark avatar and the position of the avatar is fixed at the location of the shark's dorsal fin (e.g., dorsal fin605-7), and movement of the first portion relative to the second portion includes flexing a head portion of the shark (e.g. upper head portion605-10) (e.g., including the shark's eyes (605-2), nose (605-3), jaws (605-1)) to the side (e.g., to the left or right), in an upward direction, or in a downward direction (e.g., mirroring left/right turning movement of the user's head or up/down nodding movement of the user's head). In some embodiments, the position of the avatar is not fixed. That is, the first portion of the avatar (e.g., the shark's head) is still configured to flex relative to the second portion (e.g., torso portion of the shark), but the position of the shark is not fixed in the displayed frame of view. In some embodiments, the avatar is not fixed and the avatar body does not flex with movement of the user's head.

In some embodiments, the virtual avatar (e.g., shark avatar605) further includes a fourth avatar feature (e.g., shark tail605-4) (e.g., an avatar tail) that moves independently of the detected changes in pose of the face in the field of view of the one or more cameras (e.g.,602) (e.g., the fourth avatar feature does not change in response (e.g., is non-responsive) to any detected changes in pose of the face in the field of view of the one or more cameras). In some embodiments, the electronic device (e.g.,600) displays the fourth avatar feature having a relative position (e.g., positioned having a simulated depth that places the fourth avatar feature behind the other avatar features) with respect to the first, second, and third avatar features (e.g., shark jaw605-1, shark eyes605-2, shark nose605-3). In some embodiments, the electronic device displays movement (e.g., continuous movement) of the fourth avatar feature relative to the first, second, and third avatar features (e.g., the avatar tail feature is displayed having side-to-side movement). In some embodiments, while continuing to display the movement of the fourth avatar feature (e.g., the side-to-side movement of the avatar tail feature is continuously displayed) (e.g., the avatar tail feature is displayed moving side-to-side at a consistent pace), the electronic device changes an appearance of the fourth avatar feature based on the relative position of the fourth avatar feature with respect to the first, second, and third avatar features (e.g., the appearance of the avatar tail feature changes based on its relative position to the first, second, and third avatar features as it continues side-to-side movement). Changing the appearance of the fourth avatar feature based on the relative position of the fourth avatar feature with respect to the first, second, and third, avatar features provides the user with feedback indicating that relative positions of the fourth avatar feature with respect to other avatar features will cause the device to modify the appearance of the fourth avatar feature depending on whether the fourth avatar feature is visible based on its relative position to other avatar features. This provides improved visual feedback to the user by providing a more realistic appearance of the virtual avatar and various portions of the virtual avatar. Providing improved visual feedback enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. This also conserves computational resources (thereby reducing power usage and improving battery life) by foregoing the generation and display of features that should not be displayed based on their relative position on the display with respect to other rendered image data. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, changing the appearance of the fourth avatar feature (e.g., shark tail605-4) based on the relative position of the fourth avatar feature with respect to the first, second, and third avatar features (e.g., shark jaw605-1, shark eyes605-2, shark nose605-3) includes one or more of the following determinations. In accordance with a determination, based on the relative position of the fourth avatar feature, that a first portion of the fourth avatar feature (e.g., an end portion of an avatar tail feature) is positioned behind one or more of the first, second, and third avatar features, the electronic device (e.g.,600) foregoes displaying the first portion of the fourth avatar feature (e.g., while continuing to display a second portion of the fourth avatar feature (e.g., a base portion of the avatar tail feature); e.g., when an end portion of the avatar tail feature is positioned behind another avatar feature, and a base portion of the avatar tail feature is not positioned behind another avatar feature, the end portion of the avatar tail is not displayed (e.g., because it is hidden behind the other avatar feature(s)), but the base portion of the avatar tail is displayed). In accordance with a determination, based on the relative position of the fourth avatar feature, that the first portion of the fourth avatar feature is not positioned behind one or more of the first, second, and third avatar features, the electronic device displays the first portion of the fourth avatar (e.g., while continuing to display a second portion of the fourth avatar feature; e.g., when both the end portion of the avatar tail feature and the base portion of the avatar tail feature are not positioned behind another avatar feature, both the end portion and the base portion of the avatar tail are displayed).

In some embodiments, the electronic device (e.g.,600) determines that the face is no longer detected within the field of view of the one or more cameras (e.g.,602) (e.g., face tracking has failed) (e.g., because the face is no longer within the field of view of the one or more cameras, the features of the face that the device was using to track the face are obscured, or the device has otherwise failed to be able to track the face), and in response to determining that the face is no longer detected within the field of view of the one or more cameras, continues to display the movement of the fourth avatar feature, and changes the appearance of the fourth avatar feature based on the relative position of the fourth avatar feature with respect to the first, second, and third avatar features. Continuing to display the movement of the fourth avatar feature and changing the appearance of the fourth avatar feature based on the relative position of the fourth avatar feature with respect to the first, second, and third avatar features after the face is no longer detected within the field of view of the one or more cameras provides the user with more control over the device by allowing the user to continue the process for modifying a virtual avatar, even when face tracking has failed. Providing additional control of the device without cluttering the UI with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), and through programmed behavioral characteristics of the virtual avatar provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display) even when detection of the input (e.g., changes to facial features) fails. This provides improved visual feedback to the user by providing a more realistic appearance of the virtual avatar and various portions of the virtual avatar that continue to move in accordance with the behavioral characteristics of the virtual avatar, even when face tracking has failed. The improved visual feedback enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, the electronic device (e.g.,600) displays, via the display apparatus (e.g.,601), a representation of image data captured via the one or more cameras (e.g.,602), wherein the representation of image data includes a representation of a subject (e.g.,632) (e.g., a representation of at least a portion of a subject), and displays via the display apparatus, one or more avatar features (e.g.,605-1,605-2,605-3,605-5,605-6,605-7,605-8,605-9,605-10,605-11) of the virtual avatar (e.g., shark avatar605) in place of (e.g., occludes or is displayed on top of) at least a portion of (e.g., with at least a portion of the virtual avatar partially or completely overlaying (e.g., obscuring) at least a portion of the subject) the representation of the subject (e.g.,FIG. 6F), wherein the one or more avatar features of the virtual avatar does not include the fourth avatar feature (e.g., shark tail605-4) (e.g., the shark's tail is not displayed). Displaying one or more avatar features of the virtual avatar in place of at least a portion of the representation of the subject, wherein the one or more avatar features of the virtual avatar does not include the fourth avatar feature, makes the user-device interface more efficient by only displaying portions of the virtual avatar that should be visible to the user based on a simulated depth and relative position of the virtual avatar and representation of the subject. This conserves computational resources (thereby reducing power usage and improving battery life) by foregoing the generation and display of features that should not be displayed based on their simulated depth and relative position on the display with respect to other rendered image data. This also provides a more realistic display of the virtual avatar, which improves the visual feedback of the virtual avatar displayed with respect to a displayed representation of the subject. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, the image data corresponds to depth data (e.g., the image data includes data captured by a visible light camera and a depth camera) (e.g., image data that includes a depth aspect (e.g., depth data independent of RGB data) of a captured image or video) that includes depth data for the subject (e.g., information about the relative depth positioning of one or more portions of the subject with respect to other portions of the subject and/or to other objects within the field of view of the one or more cameras). In some embodiments, the image data includes at least two components: an RGB component that encodes the visual characteristics of a captured image, and depth data that encodes information about the relative spacing relationship of elements within the captured image (e.g., the depth data encodes that a user is in the foreground, and background elements, such as a tree positioned behind the user, are in the background). In some embodiments, the depth data is a depth map. In some embodiments, a depth map (e.g., depth map image) contains information (e.g., values) that relates to the distance of objects in a scene from a viewpoint (e.g., a camera). In one embodiment of a depth map, each depth pixel defines the position in the viewpoint's Z-axis where its corresponding two-dimensional pixel is located. In some examples, a depth map is composed of pixels wherein each pixel is defined by a value (e.g., 0-255). For example, the “0” value represents pixels that are located at the most distant place in a “three dimensional” scene and the “255” value represents pixels that are located closest to a viewpoint (e.g., camera) in the “three dimensional” scene. In other examples, a depth map represents the distance between an object in a scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depth of various features of an object of interest in view of the depth camera (e.g., the relative depth of eyes, nose, mouth, ears of a user's face). In some embodiments, the depth map includes information that enables the device to determine contours of the object of interest in a z direction. In some embodiments, the depth data has a first depth component (e.g., a first portion of depth data that encodes a spatial position of the subject in the camera display region; a plurality of depth pixels that form a discrete portion of the depth map, such as a foreground or a specific object) that includes the representation of the subject (e.g.,632) in the camera display region (e.g.,620). In some embodiments, the depth data has a second depth component (e.g., a second portion of depth data that encodes a spatial position of the background in the camera display region; a plurality of depth pixels that form a discrete portion of the depth map, such as a background), separate from the first depth component, the second depth aspect including the representation of the background (e.g.,636) in the camera display region (e.g.,620). In some embodiments, the first depth aspect and second depth aspect are used to determine a spatial relationship between the subject in the camera display region and the background in the camera display region. This spatial relationship can be used to distinguish the subject from the background. This distinction can be exploited to, for example, apply different visual effects (e.g., visual effects (e.g., virtual avatars) having a depth component) to the subject and background. In some embodiments, all areas of the image data that do not correspond to the first depth component (e.g., areas of the image data that are out of range of the depth camera) are segmented out (e.g., excluded) from the depth map. In some embodiments, the depth data is in the form of a depth map or depth mask.

In some embodiments, the virtual avatar (e.g.,605) is placed at simulated depth (e.g., at a location selected so that the virtual avatar is displayed slightly in front of the representation of the subject (e.g.,632) in a depth dimension of the user interface) relative to the representation of the subject as determined based on depth data for the subject (e.g., a displayed head or face portion of the user is replaced with (or overlaid by (e.g., opaquely, transparently, translucently)) a head of a virtual avatar that corresponds to the selected avatar option) (e.g., shark avatar605). In some embodiments, displaying a representation of the selected avatar option on the representation of the subject includes using depth information obtained using one or more depth cameras (e.g.,602) of the electronic device (e.g.,600).

In some embodiments, in accordance with a determination, based on the depth data, that a first portion of the virtual avatar (e.g., a tail feature of a shark avatar;605-4) does not satisfy a set of depth-based display criteria for a first portion of the subject (e.g., the subject's head) (e.g., because the depth data for the subject indicate that the first portion of the virtual avatar has a simulated depth that is behind the corresponding first portion of the subject), the device (e.g.,600) excludes, from the representation of the virtual avatar (e.g., shark avatar605), the representation of the first portion of the virtual avatar (e.g., the shark's tail is not displayed because it is positioned behind the subject's head) (e.g., shark tail605-4is not shown inFIG. 6Fbecause it is positioned behind the shark605that is displayed in place of the subject's head). In some embodiments, the electronic device also displays a portion of the representation of the subject (e.g., the representation of the subject's shoulder or neck) in a region that would have been occupied by the first portion of the virtual avatar (e.g., forgo including as part of the representation of the virtual avatar, a representation of the first portion of the virtual avatar because that portion should be obscured by the subject). When the first portion of the virtual avatar does not satisfy depth-based display criteria because it is obscured by a portion of the subject, the first portion of the virtual avatar is excluded from the displayed representation of the virtual avatar. For example, all or a portion of the shark avatar's tail is not displayed when it is positioned behind the representation of the subject's shoulder or neck.

In some embodiments, the virtual avatar is a shark (e.g., shark avatar605) that includes a tail feature (e.g., shark tail605-4). In some embodiments, the electronic device (e.g.,600) displays the representation of the virtual avatar (e.g., shark avatar605) by displaying a first portion of the avatar (e.g., a shark head) (e.g., upper head portion605-10) and conditionally displaying a second portion of the avatar (e.g., the shark tail feature) (e.g., shark tail605-4) based on whether or not a simulated depth of the second portion of the avatar is in front of or behind a portion of the representation of the subject (e.g., the subject's head, neck, or shoulder) based on the depth data for the subject. For example, when the tail feature of the shark avatar has a simulated depth that is behind the depth of the representation of the subject (e.g., the subject's head, neck, or shoulder), and when the avatar tail feature is positioned behind the representation of the subject (e.g., the subject's head, neck, or shoulder (e.g., from the perspective of a user viewing the representation of the subject and virtual avatar) (or behind the avatar (e.g., shark head) displayed in place of the representation of the subject), the electronic device excludes all, or a portion, of the avatar tail feature from the displayed shark avatar (e.g., depending upon whether the respective portion of the avatar tail feature is positioned behind the representation of the subject (or the avatar shark displayed in place of the representation of the subject), such that the representation of the subject (or, for example, shark head) is positioned between the portion of the avatar tail and a user viewing the representation of the subject). By determining the relative depth and position of the avatar tail feature with respect to the representation of the subject (or other displayed portions of the avatar, such as the shark avatar's head), the electronic device can selectively display portions of the avatar (e.g., such as portions of the shark tail) that should be visible to a user viewing the avatar on the representation of the subject, while selectively excluding portions of the avatar that should be hidden from view based on the position and depth of the tail feature with respect to the representation of the subject or avatar (e.g., such as when the shark's tail is positioned behind the shark's head that is displayed over, or in place of, the representation of the subject's head).

In some embodiments, in accordance with a determination that the detected movement of the one or more facial features includes movement of a fourth facial feature (e.g., user cheeks610-4) (e.g., one or more of a user's cheeks) from a first position (e.g., a non-expanded or relaxed position) to a second position (e.g., an expanded or “puffed” position), the electronic device (e.g.,600) moves a fifth avatar feature (e.g., shark gills605-5) (e.g., gills when the virtual avatar is a fish or shark) from an initial position (e.g., a non-expanded or relaxed position (in some embodiments, the initial position of the avatar feature is a position that is undistinguished)) to a modified position (e.g., an expanded position in which the gills of the shark are extended from the sides of the shark) based on a magnitude of displacement of the fourth facial feature when moving from the first position to the second position. Moving the fifth avatar feature from the initial position to the modified position based on the magnitude of displacement of the fourth facial feature when moving from the first position to the second position provides the user with feedback indicating that the displacement of fourth facial feature by different magnitudes will cause the device to modify the respective fifth avatar feature to different positions determined based on the magnitude of displacement of the fourth facial feature. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the fourth facial feature, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.

In some embodiments, in accordance with a determination that the detected movement of the one or more facial features includes movement of a fifth facial feature (e.g., one or more of a user's eyebrows610-5), the electronic device (e.g.,600) moves a sixth avatar feature (e.g., pectoral fins605-6) (e.g., fins when the virtual avatar is a fish or shark) based on at least one of a direction and magnitude of movement of the fifth facial feature. In some embodiments, the sixth avatar feature does not anatomically correspond to the fifth facial feature (e.g., it is anatomically distinct or anatomically corresponds to a different physical feature) (e.g., the fifth facial feature is one or more user eyebrows and the sixth avatar feature is one or more fins of an aquatic avatar animal such as a shark or fish). In some embodiments, the movement of the fifth facial feature includes a magnitude and/or direction of movement of the facial feature(s) (e.g., in an upward or downward direction such as when the user raises or lowers their eyebrows). In some embodiments, the avatar fins move (e.g., slide, curl, tilt) in an upward/downward direction on the body of the avatar and by an amount that is determined based on the amount of movement of the user's eyebrows in an upward/downward direction. Moving the sixth avatar feature based on at least one of the direction and magnitude of movement of the fifth facial feature provides the user with feedback indicating that further movement of the fifth facial feature by different magnitudes and/or directions will cause the device to further modify the respective sixth avatar feature based on the different magnitudes and/or directions of movement of the fifth facial feature. This provides a control scheme for operating a virtual avatar on a display of an electronic device, wherein the system detects and processes input in the form of changes to facial features of a user (and the magnitude and/or direction of those changes), including the fifth facial feature, and through an iterative feedback loop provides the desired output in the form of the appearance of the virtual avatar, while eliminating the need for manual handling of the user interface (e.g., providing touch inputs on the display). This provides improved visual feedback to the user regarding how to manipulate the display to control the virtual avatar using facial movements. This enhances operability of the device and makes the user-device interface more efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting with the device) which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, this control scheme can require fewer inputs to generate or control animations of the virtual avatar than would be necessary if a different animation control scheme were used (e.g., a control scheme requiring manipulation of individual control points for each frame of an animation sequence). Moreover, this type of animation can be done in real time during, for example, a conversation such as a text conversation or a video conversation, whereas manual animation of an avatar would have to be done before the conversation started or after it had ended.