Systems and methods for generating viewpoints for visual content based on gaze

Visual information defining visual content may be accessed. The visual content may include one or more views of one or more scenes. The scene(s) may include one or more human subjects. Gaze directions of the human subject(s) may be determined. The gaze directions may indicate one or more viewing directions in which the human subject(s) are looking. One or more directions of view for the visual content may be determined based on the gaze directions of the human subject(s). The direction(s) of view may include one or more of the viewing directions looked at by the human subject(s). The visual content may be presented on a display based on the direction(s) of view for the visual content.

FIELD

This disclosure relates to systems and methods that generate viewpoints for visual content based on gaze.

BACKGROUND

For visual content (e.g., images, videos) including a greater content field of view than can normally be viewed within a single viewpoint, it may be difficult and time consuming to manually set viewpoints for the visual content. For example, different spherical images may include a thing/event of interest within different portions of the spherical images. Direction of a thing/event of interest captured within spherical videos may change during playback (due to movement of the thing/event and/or movement of the viewpoint). It may be difficult to manually set viewpoints for the visual content to be directed to/follow the thing/event of interest during view/playback.

SUMMARY

This disclosure relates to generating viewpoints for visual content based on gaze. Visual information defining visual content may be accessed. The visual content may include one or more views of one or more scenes. The scene(s) may include one or more human subjects. Gaze directions of the human subject(s) may be determined. The gaze directions may indicate one or more viewing directions in which the human subject(s) are looking. One or more directions of view for the visual content may be determined based on the gaze directions of the human subject(s). The direction(s) of view may include one or more of the viewing directions looked at by the human subject(s). The visual content may be presented on a display based on the direction(s) of view for the visual content.

A system that generates viewpoints for visual content may include one or more processors, and/or other components. The processor(s) may be configured by machine-readable instructions. Executing the machine-readable instructions may cause the processor(s) to facilitate generating viewpoints for visual content. The machine-readable instructions may include one or more computer program components. The computer program components may include one or more of a visual information component, a gaze direction component, a direction of view component, a presentation component, and/or other computer program components. In some implementations, the computer program components may include an extent of view component.

The visual information component may be configured to access visual information. The visual information may define visual content. Visual content may refer to media content that may be consumed visually. Visual content may include one or more of an image, a sequence of images, a frame of a video, a video having a progress length, and/or other visual content. Visual content may have been captured by one or more image sensors. Visual content may have been captured at a time or at different times. Visual content may have been captured at one or more locations. Visual content may include spherical visual content. Spherical visual content may include visual content obtained by a spherical capture. Spherical visual content may include 360 degrees or less than 360 degrees capture of visual content at one or more locations. Visual content may include one or more views of one or more scenes. One or more scenes may include one or more human subjects.

The gaze direction component may be configured to determine gaze directions of the one or more human subjects. The gaze directions may indicate one or more viewing directions in which the one or more human subjects within the scene(s) are looking. The gaze directions may be characterized by a yaw parameter, a pitch parameter, and/or other parameters. In some implementations, the gaze directions may be further characterized by a roll parameter.

The gaze directions may be determined based on visual analysis of the visual content and/or other information. In some implementations, determining the gaze directions may include determining one or more portions of the scene(s) viewed by the one or more human subjects. In some implementations, the gaze directions of the one or more human subjects may be determined as a function of progress through the video. For example, the gaze directions may include a first direction at a first point in the progress length of the video, a second direction at a second point in the progress length of the video, and/or other directions at other points in the progress length of the video.

The direction of view component may be configured to determine one or more directions of view for the visual content. In some implementations, the one or more directions of view for the visual content may be determined as a function of progress through the video. The one or more directions of view for the visual content may be determined based on the gaze directions of the one or more human subjects and/or other information. The one or more directions of view may include one or more of the viewing directions looked at by the one or more human subjects. In some implementations, two or more of the gaze directions may be weighed differently for determining the one or more directions of view for the visual content.

In some implementations, the one or more directions of view may be further determined based on the one or more portions of the scene(s) viewed by the one or more human subjects. In some implementations, determining the one or more directions of view may include determining one or more points of convergence based on the gaze directions. The one or more points of convergence may include one or more points within the scene(s) in which two or more of the gaze directions converge.

In some implementations, the extent of view component may be configured to determine one or more extents of view for the visual content. In some implementations, the one or more extents of view for the visual content may be determined as a function of progress through the video. The one or more extents of view for the visual content may be determined based on the gaze directions and/or other information.

The presentation component may be configured to present the visual content on a display. The visual content may be presented on the display based on the one or more directions of view for the visual content and/or other information. In some implementations, the visual content may be presented on the display further based on the one or more extents of view for the visual content. In some implementations, presenting the visual content based on the one or more directions of view for the visual content may include generating one or more versions of the visual content based on the one or more directions of view for the visual content and/or other information.

DETAILED DESCRIPTION

FIG. 1illustrates system10for generating viewpoints for visual content based on gaze. System10may include one or more of processor11, electronic storage12, interface13(e.g., bus, wireless interface), and/or other components. Visual information defining visual content may be accessed by processor11. The visual content may include one or more views of one or more scenes. The scene(s) may include one or more human subjects. Gaze directions of the human subject(s) may be determined. The gaze directions may indicate one or more viewing directions in which the human subject(s) are looking. One or more directions of view for the visual content may be determined based on the gaze directions of the human subject(s). The direction(s) of view may include one or more of the viewing directions looked at by the human subject(s). The visual content may be presented on a display based on the direction(s) of view for the visual content.

Electronic storage12may be configured to include electronic storage medium that electronically stores information. Electronic storage12may store software algorithms, information determined by processor11, information received remotely, and/or other information that enables system10to function properly. For example, electronic storage12may store information relating to visual information, visual content, scenes included in the visual content, human subjects included in the visual content, gaze directions of human subjects, directions of view for the visual content, and/or other information.

Processor11may be configured to provide information processing capabilities in system10. As such, processor11may comprise one or more of a digital processor, an analog processor, a digital circuit designed to process information, a central processing unit, a graphics processing unit, a microcontroller, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information. Processor11may be configured to execute one or more machine readable instructions100to facilitate generating viewpoints for visual content based on gaze. Machine readable instructions100may include one or more computer program components. Machine readable instructions100may include one or more of visual information component102, gaze direction component104, direction of view component106, presentation component110and/or other computer program components. In some implementations, machine readable instructions100may include extent of view component108.

The visual information component102may be configured to access visual information. The visual information may define one or more visual content. Visual content may include one or more views of one or more scenes. One or more scenes may include one or more human subjects. Visual information component102may access one or more visual information from one or more storage locations. A storage location may include electronic storage12, electronic storage of one or more image sensors (not shown inFIG. 1), and/or other locations. Visual information component102may be configured to access visual information defining one or more visual content during acquisition of the visual content and/or after acquisition of the visual content by one or more image sensors. For example, visual information component102may access visual information defining a video while the video is being captured by one or more image sensors. Visual information component102may access visual information defining a video after the video has been captured and stored in memory (e.g., electronic storage12).

Visual content may refer to media content that may be consumed visually. Visual content may include one or more of an image, a sequence of images, a frame of a video, a video having a progress length, and/or other visual content. Visual content may be stored in one or more formats/containers. Visual content may have been captured by one or more image sensors. Visual content may have been captured at a time or at different times. Visual content may have been captured at one or more locations. Visual content may include spherical visual content. Spherical visual content may include visual content obtained by a spherical capture. Spherical visual content may include 360 degrees or less than 360 degrees capture of visual content at one or more locations. Spherical visual content may be captured through the use of one or more image sensors to capture images/videos from one or more locations. The captured images/videos may be stitched together to form the spherical visual content.

Gaze direction component104may be configured to determine gaze directions of one or more human subjects. Gaze directions may indicate where things/events of interest are located within the visual content. Gaze directions may indicate one or more viewing directions in which one or more human subjects within the scene(s) are looking. Viewing directions may indicate which points/parts of the visual content are looked at by one or more human subjects within the scene(s).

Gaze directions may be characterized by a yaw parameter, a pitch parameter, and/or other parameters. In some implementations, the gaze directions may be further characterized by a roll parameter. A yaw parameter may define an amount of yaw rotation of gaze direction for visual content. A pitch parameter may define an amount of pitch rotation of gaze direction for visual content. A roll parameter may define an amount of roll rotation of gaze direction for visual content.

FIG. 3illustrates examples of rotational axes for spherical visual content300. Rotational axes for spherical visual content300may include z-axis310, y-axis320, x-axis330, and/or other axes. A yaw parameter may define an amount of rotation of gaze direction for spherical visual content300around z-axis310. A pitch parameter may define an amount of rotation of gaze direction for spherical visual content300around y-axis320. A roll parameter may define an amount of rotation of gaze direction for spherical visual content300around a viewing direction.

For example, a 0-degree rotation around z-axis310and a 0-degree rotation around y-axis320of a gaze direction for spherical visual content300may correspond to front viewing direction350. Front viewing direction350may indicate that a point/part of spherical visual content300located at 0-degree rotation of spherical visual content300around z-axis310and 0-degree rotation of spherical visual content300around y-axis320was looked at by one or more human subjects within a scene.

A −90-degree rotation of around z-axis310and a 0-degree rotation around y-axis320of a gaze direction for spherical visual content300may correspond to left viewing direction360. Left viewing direction360may indicate that a point/part of spherical visual content300located at −90-degree rotation of spherical visual content300around z-axis310and 0-degree rotation of spherical visual content300around y-axis320was looked at by one or more human subjects within a scene.

A 0-degree rotation of around z-axis310and a −90-degree rotation around y-axis320of a gaze direction for spherical visual content300may correspond to bottom viewing direction370. Bottom viewing direction370may indicate that a point/part of spherical visual content300located at 0-degree rotation of spherical visual content300around z-axis310and −90-degree rotation of spherical visual content300around y-axis320was looked at by one or more human subjects within a scene. Other viewing directions are contemplated.

While front viewing direction350, left viewing direction360, and bottom viewing direction370are shown as extending from center of spherical visual content300, this is merely for ease of reference. Human subjects looking at one or more points/parts of spherical visual content300may or may not be located at the center of spherical capture of spherical visual content300.

One or more viewing directions may be characterized by a roll parameter. For example, front viewing direction350may be characterized by a 45-degree of rotation about front viewing direction350based on one or more human subjects within a scene looking at the corresponding point/part of spherical visual content300with their heads tilted with respect to ground by 45-degrees. Other rotations of viewing directions are contemplated.

Gaze directions may be determined based on visual analysis of the visual content and/or other information. Gaze directions determination may be an extension of face detection. For example, gaze component104may determine gaze directions based on detecting faces/heads of humans subjects within a scene, determining orientations of faces/heads within the scene, and/or determining eye gazes within the scene. Gaze direction component104may use geometry of one or more image sensors capturing a scene to determine where one or more human subjects are looking within the visual content (e.g., spherical visual content300). In some implementations, gaze directions may be determined based on a combination of iris eccentricity and head orientation. In some implementations, gaze direction component104may use information about human subjects to determine gaze directions. For example, gaze direction component104may access physical characteristic information (e.g., height, head shape, features of faces/head contour) of the human subjects in the scene to determine gaze directions of the human subjects.

In some implementations, determining the gaze directions may include determining one or more portions of the scene(s) viewed by one or more human subjects. For example, determining a gaze direction in front viewing direction350may include determining that a point/part of spherical visual content300located at 0-degree rotation of spherical visual content300around z-axis310and 0-degree rotation of spherical visual content300around y-axis320was looked at by one or more human subjects within a scene. Determining a gaze direction in left viewing direction360may include determining that a point/part of spherical visual content located at −90-degree rotation of spherical visual content300around z-axis310and 0-degree rotation of spherical visual content300around y-axis320was looked at by one or more human subjects within a scene. Determining a gaze direction in bottom viewing direction370may include determining that a point/part of spherical visual content located at 0-degree rotation of spherical visual content300around z-axis310and −90-degree rotation of spherical visual content300around y-axis320was looked at by one or more human subjects within a scene.

FIG. 4illustrates exemplary gaze directions400of multiple human subjects for visual content. Gaze directions400may represent gaze directions of multiple human subjects an image or a frame of a video. Gaze directions400may include gaze directions of ten human subjects within a scene. As shown inFIG. 4, two human subjects may be looking at a point/part of the visual content located at 0-degree yaw angle and 0-degree pitch angle, two human subjects may be looking at a point/part of the visual content located at 180-degree yaw angle and −30-degree pitch angle, and six human subjects may be looking at a point/part of the visual content located at 90-degree yaw angle and 45-degree pitch angle. Other gaze directions are contemplated.

In some implementations, gaze directions of one or more human subjects may be determined as a function of progress through a video. Gaze directions may include a first direction at a first point in the progress length of the video, a second direction at a second point in the progress length of the video, and/or other directions at other points in the progress length of the video. For example, a video may include one or more views of a scene including a horse race and a human subject. The human subject's gaze may be following a particular horse in the horse race and the gaze direction of the human subject may change as a function of progress through the video based on the human subject's gaze following the horse.

Gaze direction component104may be configured to determine gaze directions of one or more human subjects during acquisition of the visual content and/or after acquisition of the visual content by one or more image sensors. For example, gaze direction component104may determine gaze directions for a video (e.g., for different frames of the video) while the video is being captured by one or more image sensors. Gaze direction component104may determine gaze directions for an image after the image has been captured and stored in memory (e.g., electronic storage12). Gaze direction determined during and/or after capture may be stored as part of visual information corresponding to the visual content (e.g., metadata) and/or in other locations.

Direction of view component106may be configured to determine one or more directions of view for the visual content. A direction of view for the visual content may correspond to a direction of a viewing field of view within which the visual content may be viewed. A viewing field of view for the visual content may define one or more portions of the visual content to be presented on a display. Directions of view for the visual content may be characterized by a yaw parameter, a pitch parameter, and/or other parameters. In some implementations, directions of view for the visual content may be further characterized by a roll parameter.

Direction of view component106may determine one or more directions of view for the visual content based on the gaze directions of one or more human subjects and/or other information. One or more directions of view may include one or more of the viewing directions looked at by one or more human subjects. In some implementations, one or more directions of view may include one or more of the viewing directions most looked at by multiple human subjects. For example, based on gaze directions400, direction of view component106may determine a direction of view for the visual content characterized by a 90-degree yaw angle and a 45-degree pitch angle. Determining directions of view based on the viewing directions most looked at by human subjects may allow the directions of view of visual content to include/follow the points/parts of the visual content most looked at by the human subjects.

In some implementations, one or more directions of view for the visual content may be determined as a function of progress through a video.FIG. 5illustrates exemplary directions of view500for a video as a function of progress through the video. Directions of view500may correspond to and/or may be determined based on gaze directions of one or more human subjects and/or other information. Directions of view500may change as a function of progress through the video. For example, at 0% progress mark, directions of view500may be characterized by a zero-degree yaw angle and a zero-degree pitch angle. At 25% progress mark, directions of view500may be characterized by a positive yaw angle and a negative pitch angle. At 50% progress mark, directions of view500may be characterized by a zero-degree yaw angle and a zero-degree pitch angle. At 75% progress mark, directions of view500may be characterized by a negative yaw angle and a positive pitch angle. At 87.5% progress mark, directions of view500may be characterized by a zero-degree yaw angle and a zero-degree pitch angle.

In some implementations, one or more directions of view may be further determined based on one or more portions of the scene(s) viewed by one or more human subjects. One or more directions of view may be determined to include within the viewing field of view one or more portions of the scene(s) viewed by one or more human subjects. For example, based on gaze directions400, one or more directions of view may be determined to include a portion of the scene located at a 90-degree yaw angle and a 45-degree pitch angle.

In some implementations, determining one or more directions of view may include determining one or more points of convergence based on the gaze directions. One or more points of convergence may include one or more points within the scene(s) in which two or more of the gaze directions converge. For example, determining one or more directions of view based on gaze direction400may include determining a point of convergence located at 0-degree yaw angle and 0-degree pitch angle (for two human subjects), a point of convergence located at 180-degree yaw angle and −30-degree pitch angle (for two human subjects), and a point of convergence located at 90-degree yaw angle and 45-degree pitch angle (for six human subjects). One or more directions of view may be determined to include one or more points of convergence.

In some implementations, two or more of the gaze directions may be weighed differently for determining one or more directions of view for the visual content. For example, gaze directions of human subjects may be given different weight based on the identity of the human subjects. Different weighing of gaze directions of the visual content based on identities may allow for the direction(s) of view to be determined based on/depend more/less on views of particular persons. For example, spherical visual content300may include a spherical capture (image or video) of a birthday party. The birthday person's gaze direction may be weighed more than other persons at the party (e.g., 2×, 5×, 100×). As another example, other persons' gaze directions may be given no weight and the birthday person's gaze direction may determine the direction(s) of view.

Gaze directions of human subjects may be given different weights based on the actions of the human subjects. Gaze directions of human subjects performing particular action(s) (e.g., shouting, jumping, pointing) may be weighed more than gaze directions of human subjects performing other actions (e.g., sitting/standing still). Particular actions may convey emotion/excitement/surprise of the human subjects. Different weighing of gaze directions of the visual content based on actions of human subjects may allow for the direction(s) of view to be determined based on/depend more on context of the scene. For example, spherical visual content300may include a spherical capture (image or video) of a football event. A touchdown by a team may cause one or more human subjects in the scene to jump up, raise their arms, shout, and/or provide other visual/audio clues as to what is happening within the scene. The gaze directions of humans subjects whose actions display more emotion/excitement/surprise may be weighed more so that the gaze direction is focused on the portion of spherical visual content300that caused the emotion/excitement/surprise.

In some implementations, one or more directions of view may precede one or more gaze directions. For example, at a particular point in the progress length of a video, a group of human subjects may quickly change their gaze directions to look at a particular point/part of the visual content. The rapid and/or collective changes of gaze directions may indicate that an event of interest just occurred at the particular point/part of the visual content. Direction of view component106may determine one or more directions of view for the visual content such that the direction of view is directed toward the particular point/part of the visual content before the particular point in the progress length, i.e., the direction of view is directed to where the humans subjects will look. Determining one or more directions of view that precede the gaze direction(s) may allow for the directions of view to be directed at a particular point/part of the visual content before occurrence(s) of event(s) of interest and allow for the viewing of more/entire event(s) of interest (e.g., from beginning to the end).

In some implementations, extent of view component108may be configured to determine one or more extents of view for the visual content. In some implementations, one or more extents of view for the visual content may be determined as a function of progress through the video. The extent of view for the visual content may correspond to sizes of viewing fields of view within which visual content may be viewed. Extents of view for the visual content may be characterized by one or more angles (e.g., horizontal angle, vertical angle, diagonal angle) and/or zooms (e.g., default zoom, 0.5× zoom, 3× zoom).FIG. 6illustrates exemplary viewing field of view A610and viewing field of view B620with different extents of view. The extent of view for viewing field of view B620may be larger than the extent of view for viewing field of view A610.

One or more extents of view for the visual content may be determined based on the gaze directions and/or other information. One or more extents of view may be determined to cover multiple gaze directions. For example, spherical visual content300may include a spherical capture of a music concert. Gaze directions of human subjects within the scene may be directed towards different members of a band on a stage. One or more extents of view may be determined to cover multiple/all member of the band such that a view of spherical visual content300using the extent(s) of view includes multiple/all member of the band.

In some implementations, one or more extents of view may be determined based on object/action detection within the visual content. For example, spherical visual content300may include a person performing a trick (e.g., jump) using a skateboard. The direction(s) of view following the person may be determined based on the gaze directions of human subjects observing the person. One or more extents of view for spherical visual content300may be determined such that a view of spherical visual content300using the extent(s) of view includes the entire person/entire trick.

Presentation component may110be configured to present the visual content on a display. The visual content may be presented on the display based on one or more directions of view for the visual content and/or other information. In some implementations, the visual content may be presented on the display further based on one or more extents of view for the visual content. The portions of the visual content presented on the display may be defined by the directions of view, the extents of view, and/or other information.

In some implementations, presenting the visual content based on one or more directions of view for the visual content may include generating one or more versions of the visual content based on one or more directions of view for the visual content and/or other information. For example, the directions of view for an image/a video content may be encoded into the image/the video and/or may be stored separately. Generating versions of the visual content based on one or more directions of view may allow a user to view the visual content using direction(s) of view. In some implementations, a user may deviate from the directions of view. For example, one or more directions of view may be used as a “default” view of the visual content and a user may change the directions of view during presentation of the visual content (e.g., via manual adjustment of the direction of view). In some implementations, the direction(s) of view may return to the “default” view when a user stops changing the direction(s) of view.

Although processor11and electronic storage12are shown to be connected to interface13inFIG. 1, any communication medium may be used to facilitate interaction between any components of system10. One or more components of system10may communicate with each other through hard-wired communication, wireless communication, or both. For example, one or more components of system10may communicate with each other through a network. For example, processor11may wirelessly communicate with electronic storage12. By way of non-limiting example, wireless communication may include one or more of radio communication, Bluetooth communication, Wi-Fi communication, cellular communication, infrared communication, or other wireless communication. Other types of communications are contemplated by the present disclosure.

Although processor11is shown inFIG. 1as a single entity, this is for illustrative purposes only. In some implementations, processor11may comprise a plurality of processing units. These processing units may be physically located within the same device, or processor11may represent processing functionality of a plurality of devices operating in coordination. Processor11may be configured to execute one or more components by software; hardware; firmware; some combination of software, hardware, and/or firmware; and/or other mechanisms for configuring processing capabilities on processor11.

It should be appreciated that although computer components are illustrated inFIG. 1as being co-located within a single processing unit, in implementations in which processor11comprises multiple processing units, one or more of computer program components may be located remotely from the other computer program components.

The electronic storage media of electronic storage12may be provided integrally (i.e., substantially non-removable) with one or more components of system10and/or removable storage that is connectable to one or more components of system10via, for example, a port (e.g., a USB port, a Firewire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage12may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EPROM, EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media. Electronic storage12may be a separate component within system10, or electronic storage12may be provided integrally with one or more other components of system10(e.g., processor11). Although electronic storage12is shown inFIG. 1as a single entity, this is for illustrative purposes only. In some implementations, electronic storage12may comprise a plurality of storage units. These storage units may be physically located within the same device, or electronic storage12may represent storage functionality of a plurality of devices operating in coordination.

FIG. 2illustrates method200for generating viewpoints for visual content. The operations of method200presented below are intended to be illustrative. In some implementations, method200may be accomplished with one or more additional operations not described, and/or without one or more of the operations discussed. In some implementations, two or more of the operations may occur substantially simultaneously.

Referring toFIG. 2and method200, at operation201, visual information defining visual content may be accessed. The visual content may include a view of a scene including one or more human subjects. In some implementation, operation201may be performed by a processor component the same as or similar to visual information component102(Shown inFIG. 1and described herein).

At operation202, gaze directions of the one or more human subjects may be determined. The gaze directions may indicate one or more viewing directions in which one or more human subjects within the scene are looking. In some implementations, operation202may be performed by a processor component the same as or similar to gaze direction component104(Shown inFIG. 1and described herein).

At operation203, one or more directions of view for the visual content may be determined based on the gaze directions of the one or more human subjects. The one or more directions of view may include one or more of the viewing directions looked at by the one or more human subjects. In some implementations, operation203may be performed by a processor component the same as or similar to direction of view component106(Shown inFIG. 1and described herein).

At operation204, the visual content may be presented on a display based on the one or more directions of view for the visual content. In some implementations, operation204may be performed by a processor component the same as or similar to presentation component110(Shown inFIG. 1and described herein).