Patent Description:
A cinemagram or cinemagraph is an image with one or more moving regions, for example, an image that has a portion in which a minor and repeated movement or animation occurs. A cinemagram includes both a motion component where the movement occurs in the image and a static component in which no movement occurs. Cinemagrams have the effect of making an image appear like a video or animation through the use of the subtle movements in a portion of the image while providing the ability to maintain the overall clarity and sharpness typically associated with images. Cinemagrams are commonly created from image frame sequences or short videos. Cinemagrams are typically created manually using image editing software. <CIT> discloses creating a cinemagram. <CIT> discloses triggering animation of a cinemagram.

A method, electronic device, and computer-readable medium for interactive cinemagrams is provided.

Embodiments of the present disclosure provide for generation and interactivity of cinemagrams.

In one embodiment, a method for interactive cinemagrams is provided. The method includes displaying a still frame of a cinemagram on a display of an electronic device. The cinemagram has an animated portion. The method also includes after displaying the still frame, identifying occurrence of a triggering event based on an input from one or more sensors of the electronic device. Additionally, the method includes initiating animation of the animated portion of the cinemagram in response to identifying the occurrence of the triggering event.

In another embodiment, an electronic device is provided. The electronic device includes a display, one or more sensors, and a processor operably connected to the display and the one or more sensors. The display is configured to display a still frame of a cinemagram. The cinemagram has an animated portion. The processor is configured to identify, after display of the still frame, occurrence of a triggering event based on an input from the one or more sensors; and initiate, in response to identification of the occurrence of the triggering event, animation of the animated portion of the cinemagram by the display.

In yet another embodiment, a non-transitory, computer-readable medium is provided. The non-transitory, computer-readable medium comprises computer code that, when executed by at least one processor of an electronic device, causes the electronic device to cause a display of the electronic device to display a still frame of a cinemagram, where the cinemagram has an animated portion; identify, after display of the still frame, occurrence of a triggering event based on an input from one or more sensors of the electronic device; and initiate, in response to identification of the occurrence of the triggering event, animation of the animated portion of the cinemagram by the display.

Embodiments of the present disclosure recognize that applications for creating cinemagrams work well when the motion region is relatively consistent across most frames. For example, the motion region is limited to a predefined area of the image where a repetitive motion is repeated or looped without the motion impacting other areas of the image. Techniques for creating cinemagrams involve segmenting motion regions from still background regions and then blending the motion regions from different frames into the still background frame. Embodiments of the present disclosure recognize and take into consideration that such techniques may produce a poor quality cinemagram (e.g., heavily pixilated from over blending, contain significant artifacts, with holes in background frames, and/or missing objects that should be in motion) when the motion regions have significant displacement from frame to frame or within the motion region the user wants to keep some objects still and other objects in motion.

Embodiments of the present disclosure further recognize and take into consideration that it may be desirable to have interactive cinemagram including cinemagram with a semantic connection between triggering the cinemagram and motion of the cinemagram. Accordingly, embodiments of the present disclosure provide improved techniques for the generation of cinemagrams and provide interactive cinemagrams.

<FIG> illustrates an example networked system <NUM> in which various embodiments of the present disclosure may be implemented. The embodiment of the networked system <NUM> shown in <FIG> is for illustration only. Other embodiments of the networked system <NUM> could be used without departing from the scope of this disclosure.

As shown in <FIG>, the system <NUM> includes a network <NUM>, which facilitates communication between various components in the system <NUM>. For example, the network <NUM> may communicate Internet Protocol (IP) packets or other information between network addresses. The network <NUM> may include one or more local area networks (LANs); metropolitan area networks (MANs); wide area networks (WANs); all or a portion of a global network, such as the Internet; or any other communication system or systems at one or more locations.

The network <NUM> facilitates communications between at least one server <NUM> and various client devices <NUM>-<NUM>. Each server <NUM> includes any suitable electronic computing or processing device that can provide computing services for one or more client devices. Each server <NUM> could, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication over the network <NUM>. For example, server <NUM> may operate one or more applications to generate cinemagrams in accordance with one or more embodiments of the present disclosure. In another example, server <NUM> may facilitate transfer of cinemagrams and/or images or videos for generating cinemagrams among the client devices <NUM>-<NUM>.

Each client device <NUM>-<NUM> represents any suitable electronic computing or processing device that interacts with at least one server or other computing device(s) over the network <NUM>. In this example, the client devices <NUM>-<NUM> include a desktop computer <NUM>, a mobile telephone or smartphone <NUM>, a personal digital assistant (PDA) <NUM>, a laptop computer <NUM>, a tablet computer <NUM>; a set-top box and/or television <NUM>, etc. However, any other or additional client devices could be used in the networked system <NUM>. In various embodiments, client devices <NUM>-<NUM> implement techniques for the generation and interactivity of cinemagrams as discussed in greater detail below.

In this example, some client devices <NUM>-<NUM> communicate indirectly with the network <NUM>. For example, the client devices <NUM>-<NUM> communicate via one or more base stations <NUM>, such as cellular base stations or eNodeBs. Also, the client devices <NUM>-<NUM> communicate via one or more wireless access points <NUM>, such as IEEE <NUM> wireless access points. Note that these are for illustration only and that each client device could communicate directly with the network <NUM> or indirectly with the network <NUM> via any suitable intermediate device(s) or network(s).

Although <FIG> illustrates one example of a networked system <NUM>, various changes may be made to <FIG>. For example, the system <NUM> could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular configuration. While <FIG> illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

<FIG> illustrates an example electronic device <NUM> according to embodiments of the present disclosure. The embodiment of the electronic device <NUM> illustrated in <FIG> is for illustration only, and the client devices <NUM>-<NUM> of <FIG> could have the same or similar configuration. However, electronic devices come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular implementation of an electronic device.

As shown in <FIG>, the electronic device <NUM> includes a communication interface <NUM>, TX processing circuitry <NUM>, a microphone <NUM>, and receive (RX) processing circuitry <NUM>. The communication interface <NUM> may include, for example, an RF transceiver, a Bluetooth transceiver, or a Wi-Fi transceiver. In another example, the communication interface <NUM> may support wired communications, for example, via a network interface card. The electronic device <NUM> also includes a speaker <NUM>, a processor <NUM>, an input/output (I/O) interface (IF) <NUM>, an input <NUM>, a display <NUM>, a memory <NUM>, and sensor(s) <NUM>.

For embodiments utilizing wireless communication, the communication interface <NUM> may receive an incoming RF signal such as a Bluetooth signal or a Wi-Fi signal. The communication interface <NUM> may down-convert the incoming RF signal to generate an intermediate frequency (IF) or baseband signal. The communication interface <NUM> receives the outgoing processed baseband or IF signal from the TX processing circuitry <NUM> and up-converts the baseband or IF signal to an RF signal that is transmitted via an antenna.

The processor <NUM> can include one or more processors or other processing devices and execute the OS <NUM> stored in the memory <NUM> in order to control the overall operation of the electronic device <NUM>. The processor <NUM> is also capable of executing other applications <NUM> resident in the memory <NUM>, such as, one or more applications for the generation and interactivity of cinemagrams as discussed in greater detail below.

The processor <NUM> can move data into or out of the memory <NUM> as required by an executing process. The processor <NUM> is also coupled to the I/O interface <NUM>, which provides the electronic device <NUM> with the ability to connect to other devices, such as laptop computers and handheld computers. The I/O interface <NUM> is the communication path between these accessories and the processor <NUM>.

The processor <NUM> is also coupled to the input <NUM> and the display <NUM>. The operator of the electronic device <NUM> can use the input <NUM> to enter data and inputs into the electronic device <NUM>. For example, the input <NUM> may be a touchscreen, button, keyboard, track ball, mouse, stylus, electronic pen, etc. The display <NUM> may be a liquid crystal display, light emitting diode display, or other display capable of rendering text and/or at least limited graphics, such as from web sites.

Electronic device <NUM> further includes one or more sensor(s) <NUM> that are operably connected to the processor <NUM>. For example, the sensor(s) <NUM> detect some measureable effect in proximity to the electronic device <NUM>. The sensor(s) <NUM> may include inertial sensors (e.g., accelerometers, gyroscope, magnetometer), optical sensors, motion sensors, cameras, pressure sensors, heart rate sensors, altimeter, breath sensors (e.g., microphone <NUM>), etc. As discussed in greater detail below, in various embodiments, the sensor(s) <NUM> may be used to identify occurrence of one or more semantic triggers for triggering the motion component of a cinemagram to provide interactivity.

Although <FIG> illustrates one example of electronic device <NUM>, various changes may be made to <FIG>. In another example, the electronic device may include an antenna or a set of multiple antennas. Also, while <FIG> illustrates the electronic device <NUM> configured as a mobile telephone or smartphone, electronic devices could be configured to operate as other types of mobile, stationary devices, or electronic devices for generating or interacting with cinemagrams.

As will be discussed in greater detail below, embodiments of the present disclosure provide for interactivity of cinemagrams. Embodiments of the present disclosure provide improved cinemagram generation quality with object segmentation and tracking of moving object across frames such that motion regions across multiple frames with significant displacement can still be blended to form a cinemagram. In various embodiments, the interactive cinemagram generation may be performed using an automatic mode (e.g., as illustrated in <FIG> below) that does not require an input from a user to generate the cinemagram or using a manual mode (e.g., as illustrated in <FIG> below) that requests one or more inputs from a user at one or more points during the generation of the cinemagram.

<FIG> illustrates a flowchart of a process for an automatic mode for generation of interactive cinemagrams in accordance with various embodiments of the present disclosure. For example, the process depicted in <FIG> is described as implemented by the electronic device <NUM> in <FIG>. The process may also be implemented by any of the devices <NUM>-<NUM> in <FIG>.

For the automatic mode, the process begins with the electronic device <NUM> receiving a video sequence (step <NUM>). In step <NUM>, any type of video or image data may be used. For example, the video sequence may be a series of sequential frames stored as a video file or a graphics interchange format (GIF) file. The frames may be individual images or video frames with interdependencies. The electronic device <NUM> then performs reference frame selection (also may be called a key frame or background frame) (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may select the reference frame selection based on image quality and location of certain motion components. For instance, for a cinemagram with a looping component that has a starting point, the reference frame would be picked as close to the starting point of the looping component as possible. In another example, an image of high quality or low amount of area in which movement occurs over the course of the sequence may be selected.

Thereafter, the electronic device <NUM> performs static component identification (step <NUM>). For example, in step <NUM>, the electronic device <NUM> identifies the portions of the reference frame that are relatively constant or do not move over the course of the video sequence. The electronic device <NUM> then performs affine matrix calculation for frame alignment (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may compute the affine matrix to align frames other than the reference frame with the reference frame. The electronic device <NUM> computes the affine matrix based on tracking of the location of the static components of the reference frame over the course of the video sequence. Frame alignment can compensate for movement of the camera that occurred during generation of the video sequence. For example, frame alignment can improve tracking of objects that move throughout the video sequence relative to their position in the reference frame.

Thereafter, the electronic device <NUM> identifies motion components (step <NUM>). For example, in step <NUM>, the electronic device <NUM> identifies which regions or areas of the video sequence are in motion over the duration of the sequence after the frame alignment. The electronic device <NUM> performs object segmentation and tracking (step <NUM>). For example, in step <NUM>, the electronic device <NUM> selects from the motion components one or more objects to track over the course of the video sequence to form the motion component of the cinemagram. The object selection may be performed automatically based on amount of movement or, as will be discussed in greater detail below, a deep learning process to identify which object animation will produce a quality cinemagram. Object selection may also be performed manually based on a user's selection as will be discussed in greater detail below. As part of this step, the electronic device <NUM> segments or removes the moving object from the frames to form a series of frame segments that show motion or animation of the object over the sequence.

In some embodiments, the cinemagram creator can indicate an alternate trajectory or motion pattern for one or more of the motion components and based on the motion patterns learned from the video sequence, the electronic device <NUM> can apply perspective transformations and occlusion effects to render the desired cinemagram. These embodiments differ from the usual definition of cinemagrams since the motion patterns are not solely based on the recorded video sequence but allow for greater creativity and freedom for the user to create intriguing cinemagrams. In other embodiments, the processes disclosed herein can be used to create three dimensional (3D) cinemagrams, for example, where the motion component has the effect of moving into or out of the screen. Additionally, cinemagrams can be made for augmented reality (AR) or virtual reality (VR) environments to make cinemagrams that can be consumed in these formats.

Thereafter, the electronic device <NUM> identifies semantic trigger options (step <NUM>). As used herein, semantic trigger options are options for events that trigger animation of a cinemagram that are logically related to the cinemagram animation. In various embodiments, the present disclosure provides for the interactivity of cinemagrams through the use of semantic triggering of the motion component of the cinemagram. In this manner, the cinemagram provides a high level of interactivity through the semantic link between the action triggering the motion and the motion of the cinemagram.

<FIG> illustrate examples of cinemagram interactivity in accordance with one or more embodiments of the present disclosure. In the example illustrated by <FIG> and <FIG>, tilting of the phone <NUM> triggers the flow of liquid <NUM> in the cinemagram. For example, the phone <NUM> may be implemented by electronic device <NUM> that includes sensors <NUM>, such as inertial sensors, to detect changes in orientation. When the orientation change is detected or meets or exceeds some threshold values, the electronic device <NUM> triggers the motion component of the cinemagram?the flow of liquid <NUM> from the can <NUM> in this illustrative example. Additionally, the rate or amount of orientation change may be proportional to the speed at which the motion component is played. For example, when the phone is tipped well past vertical or rapidly, the electronic device <NUM> may speed up the rate at which the motion component is played to give the effect that the liquid is being poured more quickly and vice versa for a slower effect.

In the example illustrated by <FIG> and <FIG>, blowing on the screen <NUM> triggers the movement of the candles <NUM> simulating blowing out of candles <NUM> on a cake. For example, the electronic device <NUM> may include sensors <NUM>, such as a breath sensor or microphone, to detect a sound for the user blowing to trigger the motion component. Similarly, a harder detected blowing may cause motion component to move quicker and vice versa. While the can tipping and candle blowing examples are illustrated, these are just examples and not a limitation on the number and types of additional embodiments are provided by the present disclosure. For example, tapping or hovering on an object detected by a proximity or touch sensor (e.g., such as a capacitive or inductive touch screen included in input <NUM>, a discrete proximity or touch sensor included in sensors <NUM>, a camera, etc.) can trigger motion of that object. In other embodiments, the electronic device <NUM> may use a forward facing camera and image processing of images of a user of the electronic device <NUM> viewing and interacting with the displayed cinemagram to detect more complicated user actions as triggers. For example, a wink, smile, wave, or blowing of a kiss by a user can be detected by the electronic device <NUM> using image processing and pattern matching to trigger a displayed cinemagram of a loved one performing a corresponding action.

The electronic device <NUM> performs searches for semantic audio (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may use deep learning processes to automatically tag a type of motion component and use the tag to search an audio database and provide users the option to add meaningful audio to the motion component of the cinemagram. For instance, the electronic device <NUM> may tag the motion component as a waterfall and then search an audio tagged database for the sound of waterfall to create a cinemagram with an animated waterfall. In another example, the electronic device <NUM> may use deep learning processes to identify characteristics of the overall cinemagram including static components, e.g., based on the reference frame, and identify appropriate background audio to add.

As part of step <NUM>, the electronic device <NUM> identifies the type of cinemagram and motion therein to identify the semantic trigger options. For example, as discussed above, the trigger is semantically linked to the type and motion of the cinemagram. The electronic device <NUM> may automatically recognize the type and motion of the cinemagram and the various different types of triggering actions. In one example, the electronic device <NUM> may use a table correlating semantic linking between cinemagram motion types and trigger options to identify which trigger or triggers should be used. In another example, the electronic device <NUM> may provide a user with component options for user selection (step <NUM>). The component options may include both triggering options as well as audio options. For example, the identified audio options could be presented to the user as a selection option in the creation process. In another example, identified audio may be automatically added if certain inclusion parameters are met. The electronic device <NUM> receives a selection of a component option (step <NUM>), which may include a selection of one or more semantic triggers and/or semantic audio for cinemagram interactivity.

Thereafter, the electronic device <NUM> computes blend maps (step <NUM>) and performs hole filling for the reference frame (step <NUM>). In these steps, the electronic device <NUM> performs blending in of the motion component to produce a high quality image with little or no artifacts as a result of the segmentation and blending. When the motion component is confined to a particular area of the reference frame, the blending may be a straight forward blending of the motion area into the remainder of the reference frame, for example, via alignment (spatial and temporal) and smoothing of boundaries. However, when objects have substantial motion including a large displacement across frames, the blending is more complex. Embodiments of the present disclosure provide semantic object segmentation and tracking across frames so that objects with large displacement across frames can still be blended into the cinemagram. For example, as part of step <NUM>, the electronic device <NUM> determines whether movement of objects will uncover portions of the reference frame that will not include pixel information at one or more points during the animation. If so, as part of step <NUM>, the electronic device <NUM> will fill these uncovered portions from portions of other frames or by interpolation based on nearby pixels, for example, to perform hole filling.

The desire for hole filling arises when multi-frame blending alone will not provide the information needed. Such an example of pixel information missing from a reference frame during playing of a cinemagram is illustrated in <FIG> and <FIG>. In this illustrative example where the video sequence is of a person is throwing a ball <NUM> up in the air, both the hand <NUM> of the subject and the ball <NUM> are moving. If the creator of the cinemagram decides to only have the ball <NUM> move but keep the hand <NUM> stationary then in frames other than the reference frame where the ball <NUM> is in the subjects hand <NUM>, the palm of the user has a hole <NUM> because the ball <NUM> is segmented and removed for the static component used for blending in frames other than the reference frame. As depicted, if the reference frame <NUM> is chosen as shown in <FIG> and the motion component is the ball <NUM> being thrown up in the air, when the ball <NUM> has been segmented from the reference frame <NUM>, a hole <NUM> exists for points during the animation of the ball movement other than the reference frame as illustrated in <FIG>. In this example, blending the motion component across frames may not fill in this hole <NUM> since the hand <NUM> is also moving across the other frames and is not in the same location. Accordingly, the electronic device <NUM> can track the motion of the hand <NUM> across several frames, which would otherwise be considered a static component for this cinemagram, to identify the pixel information to fill the hole <NUM> in the static portion from the reference frame used for blending. In particular, the electronic device <NUM> may identify pixels corresponding to the inner palm of the tracked hand <NUM> in other frames and use this pixel information to fill the hole <NUM> in the static portion of the reference frame as part of blending.

The electronic device <NUM> then performs multi-frame blending (step <NUM>). For example, the electronic device <NUM> may perform blending in at least one of two ways. In a first example, the motion component from each frame is blended into a copy of the static component creating the frames of the cinemagram. In a second example, a static part (this can be a single instance of a moving component) can be blended into subsequent frames. This second example implementation may be useful for cinemagrams with a large quantity of small objects that are moving and are not localized and there is a centerpiece that the user wants to keep static across frames. Based on the type of video sequence, the electronic device <NUM> may present the user with a choice between the two blending options. In the automatic mode, the electronic device <NUM> may select the blending option without a user selection. In this example, the electronic device <NUM> uses a deep learning process to identify components that may make an aesthetically pleasing cinemagram. The motion components would also be analyzed to determine which type of blending would yield a better quality cinemagram. In the automatic mode, the electronic device <NUM> could also determine what blend types are available and provide as an option to the user.

As used herein, deep learning is a type of machine learning that utilizes a series of examples along with feedback to produce an objectively better output. For example, without limitation, when referring to cinemagrams a better looking or better quality cinemagram may refer the image quality resulting from the cinemagrams, such as, for example, reduction in pixilation or artifacts, and/or refer to the quality of the motion effect in the cinemagram, such as, for example, being aesthetically pleasing. While parts of this analysis may be considered subjective, certain types of cinemagrams can be rated and based on comparison of a type of cinemagram or motion therein being created to rate cinemagrams, an objectively better quality cinemagram can be produced.

Embodiments of the present disclosure further provide deep learning techniques to automatically identify regions that are better candidates for the different components of the cinemagram, which may be utilized as part of steps <NUM>-<NUM> to identify or provide options for component selection. For example, embodiments of the present disclosure utilize such deep learning techniques to automatically determine different components of the cinemagram such as the static and motion components while incorporating physical constraints and artistic or aesthetic considerations.

Embodiments of the present disclosure further utilize deep learning techniques for semantically segmenting objects. Embodiments of the present disclosure recognize that motion might cause components to change shape, color, pattern etc. For example, clothes blowing in the wind show deformation in shape and a moving ball that's also rotating might show different patterns or colors. Accordingly, embodiments of the present disclosure use deep learning techniques and networks to semantically segment objects in the image. For example, if the region around the segmented object remains the same between two frames then blending the two frames is straight forward but if the movement of the object reveals holes in the reference frame, the electronic device <NUM> uses the hole filling techniques described above. Once objects are segmented (especially objects in motion) in each frame, tracking across frames is easier.

Thereafter, the electronic device <NUM> performs trigger and audio inclusion (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may include within metadata for the cinemagram an identifier of the trigger and audio if included. The metadata may include properties of the trigger, such as, for example and without limitation, type of trigger, what sensor outputs to identify, threshold values for triggering, whether the speed of the cinemagram is proportionate to the sensed outputs, and associated values therefore. Additionally, the metadata may include the audio or an identifier or tag for the audio to be triggered. For example, the identifier or tag may be a reference to a database where the audio may be retrieved from such as a uniform resource identifier (URI). The metadata associated with the created cinemagram may also include informational assistance on how to trigger the cinemagram. For example, the electronic device <NUM> displaying the cinemagram may display an identifier or suggested action for a user to perform to trigger the cinemagram if, for example, the user is having difficulty identifying the trigger.

Thereafter, the electronic device <NUM> stores the cinemagram (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may store the cinemagram in any file format for media files. The created cinemagram can later be transmitted to any device (e.g., any of client devices <NUM>-<NUM>) for display and triggering or may be displayed and triggered on the electronic device <NUM> that created the cinemagram.

<FIG> illustrates a flowchart of a process for a manual mode for generation of interactive cinemagrams in accordance with various embodiments of the present disclosure. For example, the process depicted in <FIG> is described as implemented by the electronic device <NUM> in <FIG>. The process may also be implemented by any of the devices <NUM>-<NUM> in <FIG>. In <FIG>, several steps of the process may be performed similarly to or the same as steps discussed above with regard to <FIG>. In the interests of brevity, the descriptions of corresponding steps from <FIG> are not repeated but rather incorporated by reference here into the discussion of <FIG>.

The process begins with the electronic device <NUM> receiving a video sequence (step <NUM>) similarly to step <NUM>. The electronic device <NUM> performs reference frame selection (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may receive a user input including selection of the reference frame or the reference frame may be selected by the electronic device similarly to step <NUM>.

Thereafter, the electronic device <NUM> receives a component selection (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may receive a user input including a selection of the cinemagram components including the motion and/or static components. For example, the user may select either or both of the motion and static components via object section inputs. <FIG> illustrates an example of assisted segmentation for cinemagram generation. For example, the user may select one or more objects <NUM> for the motion component and the electronic device may identify that any non-selected areas of the reference frame <NUM> to be the static component. In the manual mode, the electronic device <NUM> may receive an input from a user to provide more accurate input to the segmentation component of the process. For example, further improvement object segmentation for defining the motion and static components can be achieved using a tool <NUM>, such as an electronic pen or stylus, which can better localize and follow the contour of the object <NUM>. The electronic device <NUM> then performs affine matrix calculation for frame alignment (step <NUM>) and object segmentation and tracking based on user input (step <NUM>) similarly to steps <NUM> and <NUM>, but using the user selected components and objects as discussed with regard to step <NUM>. Thereafter, the electronic device <NUM> computes blend maps (step <NUM>); performs hole-filling for the reference frame (step <NUM>); and performs multi-frame blending (step <NUM>) similarly to steps <NUM>, <NUM>, and <NUM>, respectively, based on the user selected component(s) as discussed with regard to step <NUM>.

The electronic device <NUM> performs trigger and audio inclusion (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may identify semantic triggers and audio to include similarly to step <NUM> discussed above. However, additionally or instead, the electronic device <NUM> may receive one or more user selections including audio cues and triggers to use for the cinemagram. For example, the selection may be an input of the type of trigger to use from a list of presented trigger options or a manual input defining trigger properties such as a user action to detect, sensors, and/or associated sensor values. Also, for example, the audio cues may be an identifier of an audio file, a search input for a type of audio, and/or an indication of a specific area in the image that the audio should be based on (e.g., a tap on a waterfall depicted in an image coupled with feature extraction by the electronic device <NUM> to determine that a waterfall sound is desired). Thereafter, the electronic device <NUM> stores the cinemagram (step <NUM>) similarly to step <NUM>.

Although <FIG> and <FIG> illustrate examples of processes for an automatic mode and a manual mode for generation of interactive cinemagrams, respectively, various changes could be made to <FIG> and <FIG>. For example, while shown as a series of steps, various steps in each figure could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps.

<FIG> illustrates a process for interactive cinemagrams in accordance with various embodiments of the present disclosure. For example, the process depicted in <FIG> is described as implemented by the electronic device <NUM> in <FIG> on which a user is viewing and interacting with a created cinemagram. The process may also be implemented by any of the devices <NUM>-<NUM> in <FIG>.

The process begins with the electronic device <NUM> displaying a still frame of a cinemagram (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may display an image having a portion that can be animated but is not presently animated. In other words, the cinemagram has not yet been triggered to play. The animated portion can be a repeated video sequence or GIF. In some instances, the still frame may be the reference frame. In other instances, the still frame could be a frame of the cinemagram frozen at any point during the animation of the cinemagram. For example, if the triggering event is no longer sensed, the electronic device <NUM> may pause or freeze the animation of the cinemagram and await another occurrence of the triggering event as discussed below with regard to step <NUM>. The electronic device <NUM> then identifies a triggering event for the cinemagram (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may determine what actions of a user should occur in order to trigger playing of the cinemagram. For example, the electronic device <NUM> may identify the type of triggering event for the cinemagram from metadata associated with the cinemagram. The metadata may indicate a type of event and/or sensor values as the triggering event. As part of this step, the electronic device <NUM> may also identify audio for the cinemagram, for example, based on metadata associated with the cinemagram. In one example, the metadata may contain a tag or identifier for audio to be retrieved from a database. In this example, the electronic device <NUM> may retrieve this audio in anticipation of occurrence of the triggering event. In another example, the audio may be included or embedded with the cinemagram (e.g., part of the file for the cinemagram) similar to audio included with videos.

Thereafter, the electronic device <NUM> determines whether the triggering event has occurred (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may identify the occurrence based on an input from one or more sensors. In one example, the sensor is a motion sensor and the triggering event is based on sensed movement or orientation change of the electronic device. In another example, the sensor is a camera and the triggering event is based on detection of an action performed by the user via image/ vision processing and recognition. If no occurrence of the triggering event was identified, the electronic device <NUM> may continue to monitor for occurrence of the triggering event while the un-triggered cinemagram is displayed.

If occurrence of the triggering event was identified, the electronic device <NUM> initiates animation of the cinemagram (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may play the motion component of the cinemagram in response to identifying the occurrence of the triggering event. If audio is part of the cinemagram, as part of this step, the electronic device <NUM> may initiate play of the audio for the image in response to identifying the occurrence of the triggering event. In some embodiments, different triggers may be defined, identified, and used to trigger the animation and audio.

<FIG> illustrates a process for generating a cinemagram in accordance with various embodiments of the present disclosure. For example, the process depicted in <FIG> is described as implemented by the electronic device <NUM> in <FIG>. The process may also be implemented by any of the devices <NUM>-<NUM> in <FIG>.

The process begins with the electronic device <NUM> identifying a reference frame from a plurality of frames (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may identify a starting frame from a short video clip of GIF file as the reference frame. The electronic device <NUM> then identifies at least one object in the reference frame (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may automatically identify the object based on analysis of moving objects or may receive a selection from a user, for example, via a stylus. The object is selected to be at least a part of the motion component to be animated in the cinemagram. Before generation of the cinemagram, the electronic device <NUM> may also use deep learning techniques and analyze multiple objects movement across the plurality of frames to generate a recommendation of one or more objects to be animated for the cinemagram.

Thereafter, the electronic device <NUM> generates a static component of the cinemagram (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may generate the static component by segmenting the identified object from the reference frame. The electronic device <NUM> then generates the motion component of the cinemagram (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may track the segmented object across multiple frames throughout the video sequence.

Thereafter, the electronic device <NUM> determines whether a portion of the reference frame lacks pixel information during motion of the object (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may determine whether the motion component of the cinemagram moves to other areas of the reference frame resulting in lack of pixel information due to the segmentation of the object from the reference frame. If 'no' at step <NUM> (e.g., no hole was determined to have been created during the cinemagram), the process proceeds to step <NUM> where the electronic device generates the cinemagram via blending of the motion and static components.

If 'yes' at step <NUM> (e.g., a hole was determined to have been created during the cinemagram that needs to be), the electronic device <NUM> identifies pixel information to add to the reference frame during motion of the object (step <NUM>). For example, in step <NUM>, the electronic device <NUM> may identify the portion of the reference frame (e.g., the hole) in multiple of the frames other than the reference frame and identify the pixel information for the portion of the reference frame (e.g., the pixel information to use for hole filling) based on pixel information for the portion in the other frames. In another example, the electronic device <NUM> may fill the pixel information into the portion of the reference frame using nearby pixel interpolation. Thereafter, the electronic device <NUM> blends of the motion and static components to create the cinemagram (step <NUM>).

Although <FIG> illustrate examples of processes for interactive images and generating a cinemagram, respectively, various changes could be made to <FIG>. For example, while shown as a series of steps, various steps in each figure could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps.

Embodiments of the present disclosure provide for interactivity and use of audio to add additional dimensions to cinemagrams. The automatic detection, segmentation and tracking of various cinemagram components provided by the present disclosure allow for increase in ease of creating cinemagrams. In the manual mode, the use of stylus as disclosed herein enables more precise segmentation.

Although the present disclosure has been described with an exemplary embodiment, various changes and modifications may be suggested to one skilled in the art. It is intended that the present disclosure encompass such changes and modifications as fall within the scope of the appended claims.

Claim 1:
A method performed by an electronic device (<NUM>) for displaying an image having a motion component and a static component, the method comprising:
receiving from a camera a plurality of frames in a form of a video sequence;
selecting a reference frame of the plurality of frames;
displaying, on a display (<NUM>) of the electronic device (<NUM>), the reference frame of the plurality of frames;
while the reference frame is being displayed on the display (<NUM>), receiving, through the display (<NUM>) of the electronic device (<NUM>), a user input for selecting a one or more objects for the motion component within the displayed reference frame;
identifying the motion component within the displayed reference frame, based on the user input for selecting the motion component corresponding to the motion component within the displayed first frame;
based on the identified motion component within the displayed reference frame, identifying a static component based on any non-selected areas of the reference frame;
performing affine matrix calculation for frame alignment based on the plurality of frames to compensate for movement of the camera by tracking the location of static components of the reference frame over the course of the video sequence;
obtaining the image having the motion component and the static component, wherein the motion component includes an animation provided based on the tracking; and
displaying, on the display (<NUM>) of the electronic device (<NUM>), the image having the motion component and the static component, wherein the displaying the image includes displaying a still frame, initiating the animation in response to identifying the occurrence of a triggering event based on an input from one or more sensors of the electronic device.