Adjusting the image of an object to search for during video encoding due to changes in appearance caused by camera movement

Systems and methods for utilizing on-device sensor information to improve video encoding quality are discussed herein. Specifically, the systems and methods may utilize on-device sensor information to efficiently determine whether to encode a particular frame within a set of frames as an intra frame. When captured on video, a particular arrangement of a group of pixels within a frame may comprise a visual representation of an object within the frame. When encoding video footage, motion information characterizing motion of an image capturing device over time may be used to predict the modification of an arrangement of a group of pixels between frames. These predictions may be used to efficiently determine whether to encode a particular frame as an intra frame.

FIELD OF THE INVENTION

The field of the invention relates generally to video encoding systems and methods, and more particularly, to systems and methods for utilizing on-device sensor information to improve video encoding quality.

BACKGROUND

Image capturing devices, such as cameras or smartphones, are used to capture photos and videos of individuals, buildings, landscapes, objects, and/or other images within the surroundings of the image capturing device. Oftentimes, image capturing devices may be used to capture videos of an activity or event in which the image capturing device may be moving as the activity or event is being captured. This movement of the image capturing device may present various problems for encoders attempting to generate an encoded video file using the video footage captured by the image capturing device.

When captured on video, a particular arrangement of a group of pixels within a frame may comprise a visual representation of an object within the frame. When encoding video footage, movement of the arrangement of the group of pixels from frame-to-frame may be tracked based on the position and/or particular arrangement of the group of pixels in a previous frame. For example, in a given frame, an arrangement of a group of pixels may be searched for in a fixed area based on the last known position of the arrangement within a previous frame. If located, the knowledge of the arrangement's movement may be used to reduce and/or eliminate temporal redundancy by encoding the frame in terms of the transformation of a previous, reference frame to the frame. However, if movement of the image capturing device results in a failure to locate the arrangement within the fixed search area due to a modification of the particular arrangement of the group of pixels in the previous frame, the arrangement may not be located within the fixed search area and the entire frame may be searched to locate the arrangement. This process is both power and time consuming.

Furthermore, when movement of an image capturing device results in a failure to locate a particular arrangement of a group of pixels from a previous frame within a given frame, the given frame must be encoded as an intra frame as opposed to an inter frame. An inter frame is encoded by reference to one or more related frames in a set of grouped frames, whereas an intra frame is encoded without reference to one or more related frames and thus contains independently decodable information. While intra frames require less processing power than inter frames, intra frames utilize a greater number of bits than inter frames. As such, when movement of an image capturing device causes a greater number of frames to be encoded as intra frames, an encoder may reach its bit limit sooner, requiring the encoder to enter a bit rate preservation mode. This often results in degraded video quality.

These and other problems exist when attempting to generate an encoded video file using video footage captured by a moving image capturing device.

SUMMARY

This disclosure relates to systems and methods for utilizing on-device sensor information to improve video encoding quality. The systems and methods may utilize on-device sensor information to efficiently determine whether to encode a particular frame within a set of frames as an intra frame. As an image capturing device moves while capturing video, an object within the field of view may appear differently within subsequent captured frames (e.g., due to changes in size and/or orientation). As a result, the arrangement of a group of pixels representing the object may change within subsequent captured frames. This may make the arrangement of the group of pixels representing a given object difficult to locate from one frame to the next. Systems and methods described herein may utilize motion information characterizing the motion of the image capturing device when encoding video footage to determine a particular arrangement of a group of pixels to search for within a given frame. As such, a determination as to whether an arrangement of a group of pixels representing the object may be located within a given frame, and in turn whether the frame is to be encoded as an intra frame, may be made without having to sacrifice time and power searching the entire frame.

In various implementations, the system described herein may be configured to utilize on-device sensor information to improve video encoding quality, in accordance with one or more implementations. The system may include one or more of an interface, one or more physical processors, an electronic storage, an image capturing device, and/or other components. The one or more physical processors may be configured by computer-readable instructions. Executing the computer-readable instructions may cause the one or more physical processors to utilize on-device sensor information to improve video encoding quality. The computer-readable instructions may include one or more computer program components. The computer program components may include one or more of an object identification component, a motion component, a search identification component, a frame search component, a video compression component, and/or other computer program components.

The object identification component may be configured to identify an object within a frame captured by an image capturing device by identifying a particular arrangement of a group of pixels representing the object within the frame and determining a position of the particular arrangement within the frame.

The motion component may be configured to obtain motion information characterizing motion of the image capturing device over time. For example, the motion component may be configured to obtain motion information characterizing motion of the image capturing device between a first time at which a first frame was captured and a second time at which a second frame was captured. The second time may come after the first time. As such, the second frame may come after the first frame within a sequence of frames. The first frame and the second frame may be adjacent within the sequence of frames. In some implementations, other frames may separate the first frame and the second frame within the sequence of frames.

The motion information may be generated by one or more motion sensors associated with the image capturing device. As a non-limiting example, the one or more motion sensors may include one or more of an inertial measurement unit, a GPS component, an accelerometer, a gyroscope, an altimeter, a distance measurement sensor, a magnetometer, a magnetic position sensor, a radio-frequency position sensor, and/or other sensors. The motion information may characterize one or more motions. The one or more motions may include one or more of movement of the image capturing device and/or the image sensor(s), change in position of the image capturing device and/or the image sensor(s), and/or other motion of the image capturing device and/or the image sensor(s). The motion information characterizing motion of the image capturing device may include one or more of translational motion, rotational motion, vertical motion, horizontal motion, acceleration motion, deceleration motion, and/or other types of motion.

The search identification component may be configured to determine a modified arrangement of a group of pixels representing an object to search for within a frame based on the motion information and the arrangement of the group of pixels representing the same object in a related frame. The related frame may comprise a frame captured prior to or after the frame to be searched within a sequence of frames that includes the frame to be searched. As the image capturing device moves while capturing a video, the arrangement of the group of pixels representing the object may change within subsequent captured frames. The search identification component may be configured to determine a modified arrangement of the group of pixels representing the object predicted to appear within a frame based upon movement of the image capturing device and the arrangement of the group of pixels within a related frame. Determining a modified arrangement of the group of pixels may include determining a size of the modified arrangement of the group of pixels and/or boundaries of the modified arrangement of the group of pixels within the frame.

The frame search component may be configured to search within a frame and determine whether an arrangement of a group of pixels representing an object is located within a particular frame. For example, the frame search component may be configured to search within a frame and determine whether a modified arrangement of a group of pixels representing an object determined by the search identification component is located within the frame. In some implementations, the frame search component may be configured to search all and/or one or more portions of the frame. For example, the frame search component may be configured to search for a modified arrangement of a group of pixels in only a portion of the frame. In some implementations, if the arrangement of the group of pixels is not found within a first portion of a frame, the frame search component may be configured to search a second portion and/or the remainder of the frame.

The video compression component may be configured to receive a sequence of frames and generate compressed video information as an output. For example, the video compression component may be configured to encode a frame as an intra frame based on the determination of whether a particular arrangement of a group of pixels representing an object was located by the frame search component.

In various implementations, the video compression component may be configured to encode a frame as an intra frame if a modified arrangement of a group of pixels determined by the identification component is not located within at least a portion of a frame. If the modified arrangement of the group of pixels is located within at least a portion of a frame, the video compression component may be configured to encode the frame as an inter frame. When encoding a frame as an inter frame, the video compression component may be configured to encode the frame by referencing one or more previous frames. The video compression component may be configured to use motion compensation to encode the video data, such that the second frame may be described in terms of the transformation (e.g., differences) from a first frame to a second frame.

DETAILED DESCRIPTION

This disclosure relates to systems and methods for utilizing on-device sensor information to improve video encoding quality. Specifically, the systems and methods may utilize on-device sensor information to efficiently determine whether to encode a particular frame within a set of frames as an intra frame, thereby adaptively adjusting an intra frame insertion rate associated with a sequence of frames.

An intra frame insertion rate identifies the frequency of intra frames within a predefined number of frames. An inter frame is encoded by reference to one or more related frames in a set of grouped frames, whereas an intra frame is encoded without reference to one or more related frames and thus contains independently decodable information. While intra frames require less processing power than inter frames, intra frames utilize a greater number of bits than inter frames. The intra frame insertion rate for a group of frames may affect various parameters associated with the encoded video file. For example, the intra frame insertion rate may affect the available memory and/or power, the ability to randomly access individual frames, compression efficiency, and/or system latency.

FIG. 5AandFIG. 5Bdepict exemplary GOP structures that may be utilized in video encoding. A GOP (or “group of pictures”) is a group of successive pictures within a coded video stream. A GOP structure may specify the order in which types of frames are arranged within a group of successive pictures.FIG. 5Adepicts an example of an IPPP3 GOP structure, wherein three inter frames (or P frames) are located between any two intra frames (or I frames).FIG. 5Bdepicts an example of an IBBP8 GOP structure, wherein eight frames (i.e., a combination of eight bi-predictive (or B frames) and inter frames) are located between any two intra frames.

Typically, an intra frame insertion rate is predefined and does not change based on the motion of an image capturing device. However, different intra frame insertion rates may better suit different scenes based on the complexity of the scene and/or the movement of the image capturing device while capturing the scene. In various implementations, the systems and methods described herein may utilize motion information characterizing the motion of an image capturing device to efficiently determine whether to encode a particular frame within a set of frames as an intra frame, thereby adaptively adjusting an intra frame insertion rate associated with a sequence of frames to better suit the scene depicted by the sequence of frames.

It will be appreciated by those having skill in the art that the implementations described herein may be practiced without these specific details or with an equivalent arrangement. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the implementations of the invention.

Exemplary System Architecture

FIG. 1illustrates a system100for utilizing on-device sensor information to improve video encoding quality, in accordance with one or more implementations. The system may include one or more of interface102, one or more physical processors110, electronic storage130, image capturing device140, and/or other components.

The one or more physical processors110(also interchangeably referred to herein as processor(s)110, processor110, or processors110for convenience) may be configured to provide information processing capabilities in system100. As such, the processor(s)110may comprise one or more of a digital processor, an analog processor, a digital circuit designed to process information, a central processing unit, a graphics processing unit, a microcontroller, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information. Processor(s)110may be configured to execute one or more computer readable instructions112. Computer readable instructions112may include one or more computer program components. Computer readable instructions112may include one or more of object identification component114, motion component116, search identification component118, frame search component120, video compression component122, and/or other computer program components. As used herein, for convenience, the various computer readable instructions112will be described as performing an operation, when, in fact, the various instructions program the processor(s)110(and therefore system100) to perform the operation.

Object identification component114may be configured to identify an object within a frame captured by image capturing device140. Image capturing device140may capture visual content including a first frame captured at a first point in time. Visual content may include one or more of an image, a sequence of images, a frame of a video, a video, audio content, and/or other visual content. The video may be composed of many still images, also known as frames. The frames may compose a moving picture, such as the visual content.

The visual content (e.g., the visual content including the first frame captured at the first point in time) may be obtained through use of one or more image sensors144. In some implementations, the image sensor(s)144may be carried (e.g., attached to, supported, held, disposed on, and/or otherwise carried) by an object (e.g., a gimbal). In some implementations, the image sensor(s)144may be carried by a vehicle (e.g., a car, a bike, a boat, an airplane, etc.). In some implementations, the image sensor(s)144may be carried by a remote controlled vehicle (e.g., remote controlled airplane, remote controlled car, remoted controlled submarine, etc.). In some implementations, the image sensor(s)144may be carried by an unmanned aerial vehicle (e.g., drones, etc.). In some implementations, the image sensor(s)144may be carried by a person. In some implementations, the image sensor(s)144may be carried by an animal. Other carryings of the image sensor(s)144may be contemplated.

Image capturing device140, via the one or more image sensors144, may capture an object within a first frame at a first point in time. The object may include a stationary object or a moving object. Non-limiting examples of an object may include a person, a child, an adult, one or more parts of a person (e.g., arm, hand, head, body, leg, feet, face, etc.), an animal, one or more parts of an animal, a vehicle, one or more parts of a vehicle (e.g., wheel, door, engine, trunk, window, wing, propeller, rotor, etc.), a tree, a building, a lamppost, a picture, and/or other objects.

Object identification component114may be configured to identify an object within a frame captured by image capturing device140and determine a position of the object within the frame. When captured on video, a particular arrangement of a group of pixels within a frame may comprise a visual representation of an object within the frame. In various implementations, object identification component114may be configured to identify an object within a frame captured by image capturing device140by identifying a particular arrangement of a group of pixels representing the object within the frame. For example, object identification component114may be configured to identify a particular arrangement of a group of pixels representing an object by determining dimensions and/or boundaries of the arrangement of the group of pixels. In some implementations, the dimensions and/or boundaries of an arrangement of a group of pixels may be defined by reference to coordinates of a coordinate system, such as a Cartesian coordinate system.

For example, and referring toFIG. 2, first frame200is depicted. An object represented by arrangement of pixels202and an object represented by arrangement of pixels204are depicted within first frame200captured by an image capturing device (e.g., image capturing device140) at a first point in time. Object identification component114may be configured to identify objects within first frame200captured by image capturing device140. For example, object identification component114may be configured to identify arrangement of pixels202and arrangement of pixels204within frame200. In various implementations, object identification component114may be configured to determine that arrangement of pixels202comprises an arrangement of a group of pixels representing a first object and that arrangement of pixels204comprises an arrangement of a group of pixels representing a second object.

Returning toFIG. 1, motion component116may be configured to obtain motion information characterizing motion of an image capturing device over time. For example, motion component116may be configured to obtain motion information characterizing motion of image capturing device140between a first time at which a first frame was captured and a second time at which a second frame was captured. The second point in time may come after the first point in time. As such, the second frame may come after the first frame within a sequence of frames. The first frame and the second frame may be adjacent within the sequence of frames. In some implementations, other frames may separate the first frame and the second frame within the sequence of frames.

The motion information may be generated by one or more motion sensors146associated with image capturing device140. As a non-limiting example, one or more motion sensors146may include one or more of an inertial measurement unit, a GPS component, an accelerometer, a gyroscope, an altimeter, a distance measurement sensor, a magnetometer, a magnetic position sensor, a radio-frequency position sensor, and/or other sensors. In some implementations, motion component116may be configured to obtain motion information from one or more motion sensors located external to image capturing device140and/or system100and may provide information obtained via the one or more motion sensors external to image capturing device140and/or system100.

The motion information may characterize one or more motions. The one or more motions may include one or more of movement of image capturing device140and/or image sensor(s)144, change in position of image capturing device140and/or image sensor(s)144, and/or other motion of image capturing device140and/or image sensor(s)144. The motion information characterizing motion of image capturing device140may include one or more of translational motion, rotational motion, vertical motion, horizontal motion, acceleration motion, deceleration motion, and/or other types of motion.

Image capturing device140and/or image sensor(s)144associated with image capturing device140may move for various reasons. The videographer may move image capturing device140and/or image sensor(s)144to capture one or more objects within a field of view in a slightly different field of view by moving image capturing device140in a different direction, by increasing or decreasing the altitude of image capturing device140, by rotating image capturing device140, and/or by moving image capturing device140in another manner. In various implementations, motion component116may be configured to analyze the motion of image capturing device140based on the motion information. Image capturing device140may be carried by the videographer while the videographer performs an act, such as a skateboard trick, snowboarding while capturing a snowboarder ahead of the videographer, and/or other acts. In various implementations, motion component116may be configured to identify an activity being performed during the capture of a sequence of frames based on the motion information.

In an exemplary implementation, motion component116may be configured to obtain motion information from one or more motion sensors146characterizing a rotational motion of image capturing device140between the first point in time at which the first frame may have been captured and a second point in time at which a second frame may be captured. Image capturing device140may have moved (e.g., rotated) to the right, left, up, down, or any other direction by a number of degrees represented by a coordinate system, such as a Cartesian coordinate system, a cylindrical and/or polar coordinate system, a spherical and/or polar coordinate system, and/or any other coordinate system. For example, image capturing device140may have rotated clockwise by 45-degrees between the first point in time and the second point in time, thus capturing a slightly different field of view within the second frame at the second point in time than within the first frame at the first point in time. Due to movement of image capturing device140, the object may appear differently within the field of view of image capturing device140between the first frame and the second frame. For example, movement of image capturing device140may result in the object appearing larger or smaller within the field of view and/or appearing from a different angle or orientation. As such, the particular arrangement of the group of pixels comprising a visual representation of the object within the first frame, captured at a first point in time, may have been modified and appear as a different arrangement of a group of pixels within a second frame, captured at a second point in time.

Referring toFIG. 3A, second frame300is depicted inFIG. 3A. Motion component116may be configured to obtain motion information characterizing motion of image capturing device140between the first point in time at which first frame200was captured and a second point in time at which second frame300was captured. Based upon the motion information, motion component116may be configured to determine that image capturing device140rotated clockwise by 45-degrees, while remaining at the same level or altitude. As such,FIG. 3Ais a depiction of second frame300captured at the 45-degree clockwise rotation. Based upon the 45-degree clockwise rotation, arrangement of pixels202appears to have moved from the top left corner of first frame200(as depicted in dashed lines withinFIG. 3A) to the bottom left corner of second frame300(now depicted by arrangement of pixels302inFIG. 3A). Similarly based upon the 45-degree clockwise rotation, arrangement of pixels204appears to have moved from the bottom right corner of first frame200(as depicted in dashed lines withinFIG. 3A) to the top right corner of second frame300(now depicted by arrangement of pixels304inFIG. 3A). Additionally, arrangement of pixels202in first frame200and arrangement of pixels204in first frame200each appear to have rotated clockwise by 45-degrees in second frame300(now depicted by arrangement of pixels302and arrangement of pixels304inFIG. 3A, respectively).

Referring back toFIG. 1, search identification component118may be configured to determine a modified arrangement of a group of pixels representing an object to search for within a frame based on motion information and the arrangement of the group of pixels representing the same object in a related frame. The related frame (e.g., the first frame) may comprise a frame captured prior to or after the frame to be searched within a sequence of frames that includes the frame to be searched. For example, if system100knows the arrangement of a group of pixels representing the object within a first frame, and if system100knows that image capturing device140has rotated and/or moved in a clockwise direction by a certain number of degrees from the first point in time when it captured the first frame, then the arrangement of pixels, if still within the field of view, should be modified and appear as a different arrangement of a group of pixels within the second frame captured at a second point in time.

Determining a modified arrangement of a group of pixels to search for may include determining a size of the modified arrangement of the group of pixels within the second frame, determining dimensions of the modified arrangement of the group of pixels within the second frame, and/or determining boundaries of the modified arrangement of the group of pixels within the second frame. For example, the modified arrangement of the group of pixels may comprise a geometric shape, such as a circle, a rectangle, a square, and/or other geometric shape. In some implementations, determining the modified arrangement of the group of pixels may include determining dimensions and/or boundaries of the geometric shape. Alternatively or additionally, determining the modified arrangement of the group of pixels may include determining dimensions and/or boundaries of a geometric shape that encompasses the modified arrangement of the group of pixels.

In an exemplary implementation, image capturing device140may move toward or away from an object while capturing video. As a result, the object may appear larger within the field of view of image capturing device140as image capturing device140moves towards the object and may appear smaller within the field of view of image capturing device140as image capturing device140moves away from the object. In turn, the arrangement of the group of pixels representing the object within a frame may appear larger or smaller in consecutive frames based upon the movement of image capturing device140towards or away from the object while capturing the consecutive frames. Search identification component118may be configured to determine a modified arrangement of the group of pixels to search for that accounts for the change in appearance of the arrangement of the group of pixels in a related frame based on the motion information characterizing this motion of image capturing device140.

As another example, and referring toFIG. 3B, second frame300is depicted inFIG. 3B.FIG. 3Bdepicts second frame300without the 45-degree clockwise rotation fromFIG. 3A(e.g., similar to first frame200fromFIG. 2). Search identification component118may be configured to determine a modified arrangement of a group of pixels to search for in second frame300based upon the motion information and the arrangement of pixels202and/or the arrangement of pixels204within first frame200. For example, based upon the motion information (e.g., that image capturing device140rotated clockwise by 45 degrees) and the arrangement of pixels202within first frame200, search identification component118may be configured to determine that the object represented by arrangement of pixels202in first frame200should appear as a modified arrangement of pixels in second frame300, wherein the modified arrangement of pixels is the arrangement of pixels202rotated by 45-degrees (as seen by arrangement of pixels302in second frame300). Similarly, based upon the motion information (e.g., that image capturing device140rotated clockwise by 45 degrees) and the arrangement of pixels204within first frame200, search identification component118may be configured to determine that the object represented by arrangement of pixels204in first frame200should appear as a modified arrangement of pixels in second frame300, wherein the modified arrangement of pixels is the arrangement of pixels204rotated by 45-degrees (as seen by arrangement of pixels304in second frame300).

Referring back toFIG. 1, search identification component118may be configured to determine a modified arrangement of a group of pixels representing an object to search for within a frame based on motion information and the arrangement of the group of pixels representing the same object in a frame captured prior to or after the frame to be searched within a sequence of frames that includes the frame to be searched. For example, and referring toFIG. 2andFIG. 3B, first frame200fromFIG. 2may be captured by an image capturing device (e.g., image capturing device140) at a first point in time and second frame300fromFIG. 3Bmay be captured by an image capturing device at a second point in time. In various implementations, the first point in time at which first frame200was captured may occur prior to the second point in time at which second frame300was captured. Alternatively, the first point in time at which first frame200was captured may occur after the second point in time at which second frame300was captured. As such, and referring back toFIG. 1, search identification component118may be configured to determine a modified arrangement of a group of pixels to search for within a frame based on motion information characterizing the motion of the image capturing device between the time at which the frame was captured and a time prior to or after the time at which the frame was captured, at which a related frame was captured.

Frame search component120may be configured to search within a frame and determine whether an arrangement of a group of pixels representing an object is located within a particular frame. In various implementations, frame search component120may be configured to search for the modified arrangement of pixels identified by search identification component118within a second frame. In some implementations, only a portion of the second frame may be searched. In some implementations, if the arrangement of pixels is found within the portion of the second frame, other portions, such as a second portion, of the second frame may not be searched. In some implementations, if the arrangement of pixels is not found within the portion of the second frame, frame search component120may be configured to search a second portion of the second frame. The second portion of the second frame may include a portion adjacent to the portion. The second portion of the second frame may include the rest of the second frame (e.g., the entirety of the second frame minus the portion). The second portion of the second frame may include a portion of the second frame that is less than the rest of the second frame.

Video compression component122may be configured to receive a sequence of frames and generate compressed video information as output. Video compression component122may be configured to generate compressed video information resulting in higher overall video quality than with existing motion estimation processes. For example, video compression component122may be configured to encode a frame as an intra frame based on the determination of whether a particular arrangement of a group of pixels representing an object was located by frame search component120.

In various implementations, video compression component122may be configured to encode a frame as an intra frame if a modified arrangement of a group of pixels identified by search identification component118was not located within the frame. If the modified arrangement of the group of pixels identified by search identification component118is located within the frame, video compression component122may be configured to encode the frame as an inter frame. When encoding a frame as an inter frame, video compression component122may be configured to encode the frame by referencing one or more previous frames. Video compression component122may be configured to use motion compensation to encode the video data, such that the second frame may be described in terms of the transformation (e.g., differences) from a first frame to a second frame.

Video compression component122may be configured to generate compressed video information resulting in higher overall video quality than with existing motion estimation processes. If the entire second frame is searched for the arrangement of a group of pixels representing an object because the arrangement of the group of pixels was not located, encoders may be forced to spend more computational power (e.g., encoders may spend more bits) replacing blocks within the second frame with Intra-blocks. If there is a lot of movement within frames, encoders may be forced to go into bit preservation mode because limits may exist for constant or near-constant bit rate videos. This may result in an overall degraded video quality. With the improved search process disclosed herein, video compression component122may be configured to find a matching block faster than traditional block-matching because system100may determine a modified arrangement of pixels to search for within the second frame with a higher likelihood of being found by accounting for device movement. As such, video compression component122may be configured to generate higher overall video quality because searching for the modified arrangement of pixels within the second frame may consume less power.

In various implementations, video compression component122may be configured to receive the sequence of frames and generate compressed video information as output that includes the motion information. For example, video compression component122may be configured to output compressed video information related to a sequence of frames in association with motion information characterizing the motion of the image capturing device while capturing the sequence of frames. In various implementations, video compression component122may be configured to encode and/or embed the motion information in a bitstream associated with the sequence of frames. For example, the motion information may be embedded as metadata within the compressed video information, such as described in U.S. patent application Ser. No. 14/920,427, entitled, “APPARATUS AND METHODS FOR EMBEDDING METADATA INTO VIDEO STREAM,” filed on Oct. 22, 2015, the contents of which are hereby incorporated by reference herein in their entirety. When decoding the compressed video information, a decoding device may be configured to utilize the motion information to perform one or more of the techniques described herein. Additionally or alternatively, a decoding device may be configured to utilize the motion information to perform image stabilization, 3D scene reconstruction, and/or other techniques related to video decoding.

Electronic storage130may include electronic storage media that electronically stores information. The electronic storage media of electronic storage130may be provided integrally (i.e., substantially non-removable) with one or more components of system100and/or removable storage that is connectable to one or more components of system100via, for example, a port (e.g., a USB port, a Firewire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage130may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EPROM, EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media. Electronic storage130may be a separate component within system100, or electronic storage130may be provided integrally with one or more other components of system100(e.g., processor110). Although electronic storage130is shown inFIG. 1as a single entity, this is for illustrative purposes only. In some implementations, electronic storage130may comprise a plurality of storage units. These storage units may be physically located within the same device, or electronic storage130may represent storage functionality of a plurality of devices operating in coordination

Electronic storage130may store software algorithms, information determined by processor110, information received remotely, and/or other information that enables system100to function properly. For example, electronic storage130may store information relating to an object, the particular arrangement of a group of pixels representing an object within a frame, the position of the particular arrangement of the group of pixels within a frame, motion information characterizing motion of an image capturing device, a modified arrangement of the group of pixels to search for in a given frame, and/or other information.

Image capturing device140may comprise one or more of a computing platform, a mobile device (e.g., a smart phone, a tablet, and/or other mobile device), a camera (e.g., an action camera, a sports camera, and/or other type of camera), a video recorder, and/or other device configured to capture images and/or video segments. Image capturing device140may capture visual content including the first frame captured at the first point in time. Users may capture visual content using image capturing device140. Image capturing device140may include one or more of optical element142, one or more image sensor(s)144, one or more motion sensors146, and/or other components. In various implementations, processor(s)110may be located within image capturing device140.

Optical element142may be configured to guide light to an image sensor (e.g., one or more image sensor(s)144). Optical element142may include one or more of standard lens, macro lens, zoom lens, special-purpose lens, telephoto lens, prime lens, achromatic lens, apochromatic lens, process lens, wide-angle lens, ultra-wide-angle lens, fisheye lens, infrared lens, ultraviolet lens, perspective control lens, other lens, and/or other optical elements. Optical element142may guide light received from an object within the field of view of an image sensor (e.g., one or more image sensor(s)144) directly, or indirectly through use of one or more light manipulating components. For example, a light manipulating component may include one or more of a mirror, a prism, lenses, and/or other light manipulating components.

One or more image sensor(s)144(also interchangeably referred to herein as image sensor(s)144, image sensor144, or image sensors144for convenience) may be configured to generate a first output signal conveying visual information present in the light guided thereto by optical element142within the field of view of one or more image sensor(s)144. One or more image sensor(s)144may include one or more of a charge-coupled device sensor, an active pixel sensor, a complementary metal-oxide semiconductor sensor, an N-type metal-oxide-semiconductor sensor, and/or other image sensors. Visual information may include content within the field of view of one or more image sensor(s)144, such as one or more objects within the field of view of one or more image sensor(s)144, a landscape within the field of view of one or more image sensor(s)144, and/or other content within the field of view of one or more image sensor(s)144. An image and/or video segment captured by the image capturing device may include visual information and other information, including audio information.

One or more motion sensors146(also interchangeably referred to herein as motion sensor(s)110, motion sensor146, or motion sensors146for convenience) may be configured to generate motion information characterizing motion of the image capturing device over time. As described further above, one or more motion sensors146may generate motion information characterizing motion of the image capturing device between a first time at which a first frame was captured and a second time at which a second frame was captured.

Although processor110, electronic storage130, and image capturing device140are shown to be connected to interface102inFIG. 1, any communication medium may be used to facilitate interaction between any components of system100. One or more components of system100may communicate with each other through hard-wired communication, wireless communication, or both. For example, one or more components of system100may communicate with each other through a network. For example, processor110may wirelessly communicate with electronic storage130. By way of non-limiting example, wireless communication may include one or more of radio communication, Bluetooth communication, Wi-Fi communication, cellular communication, infrared communication, or other wireless communication. Other types of communications are contemplated by the present disclosure.

Although processor110is illustrated inFIG. 1as a single component, this is for illustrative purposes only. In some implementations, processor110may comprise a plurality of processing units. These processing units may be physically located within the same device, or processor110may represent processing functionality of a plurality of devices operating in coordination. Processor110may be configured to execute one or more components by software; hardware; firmware; some combination of software, hardware, and/or firmware; and/or other mechanisms for configuring processing capabilities on processor110.

Furthermore, it should be appreciated that although the various instructions are illustrated inFIG. 1as being co-located within a single processing unit, in implementations in processor(s)110include multiple processing units, one or more instructions may be executed remotely from the other instructions.

The description of the functionality provided by the different computer-readable instructions described herein is for illustrative purposes, and is not intended to be limiting, as any of instructions may provide more or less functionality than is described. For example, one or more of the instructions may be eliminated, and some or all of its functionality may be provided by other ones of the instructions. As another example, processor(s)110may be programmed by one or more additional instructions that may perform some or all of the functionality attributed herein to one of the computer-readable instructions.

Exemplary Flowchart of Process

FIG. 4illustrates a method400for utilizing sensor information to improve motion estimation in video encoding, in accordance with one or more implementations. The operations of method400presented below are intended to be illustrative and, as such, should not be viewed as limiting. In some implementations, method400may be accomplished with one or more additional operations not described, and/or without one or more of the operations discussed. In some implementations, two or more of the operations may occur substantially simultaneously. The described operations may be accomplished using some or all of the system components described in detail above.

In an operation402, method400may include identifying a first arrangement of a group of pixels within a first frame captured at a first time by an image capturing device. The first frame may have been captured at a first time by an image capturing device. The arrangement of the group of pixels may comprise a visual representation of an object captured by the video capturing device. In some implementations, operation402may be performed by a processor component the same as or similar to object identification component114(shown inFIG. 1and described herein).

In an operation404, method400may include obtaining motion information characterizing motion of the image capturing device. The motion information may characterize motion of the image capturing device between the first time at which the first frame was captured and a second time at which a second frame was captured. The motion information may be generated and/or obtained from one or more motion sensors associated with the image capturing device. In some implementations, operation404may be performed by a processor component the same as or similar to motion component116(shown inFIG. 1and described herein).

In an operation406, method400may include determining a second arrangement of the group of pixels to search for within the second frame based on the motion information and the first arrangement of the group of pixels. The second arrangement of the group of pixels may be determined based on the motion information and the first arrangement of the group of pixels within the first frame. In some implementations, operation406may be performed by a processor component the same as or similar to search identification component118(shown inFIG. 1and described herein).

In an operation408, method400may include searching for the second arrangement of the group of pixels within the second frame. In some implementations, operation408may be performed by a processor component the same as or similar to frame search component120(shown inFIG. 1and described herein).

In an operation410, method400may include determining whether the second arrangement of the group of pixels is located within the second frame. In some implementations, operation410may be performed by a processor component the same as or similar to frame search component120(shown inFIG. 1and described herein).

In an operation412, method400may include determining whether to encode the second frame as an intra frame based on whether the second arrangement of the group of pixels is located within the second frame. Responsive to a determination that the second arrangement of the group of pixels is not located within the second frame, the second frame may be encoded as an intra frame. Responsive to a determination that the second arrangement of the group of pixels is located within the second frame, the second frame may be encoded as an inter frame. In some implementations, operation412may be performed by a processor component the same as or similar to video compression component122(shown inFIG. 1and described herein).

Reference in this specification to “one implementation”, “an implementation”, “some implementations”, “various implementations”, “certain implementations”, “other implementations”, “one series of implementations”, or the like means that a particular feature, design, structure, or characteristic described in connection with the implementation is included in at least one implementation of the disclosure. The appearances of, for example, the phrase “in one implementation” or “in an implementation” in various places in the specification are not necessarily all referring to the same implementation, nor are separate or alternative implementations mutually exclusive of other implementations. Moreover, whether or not there is express reference to an “implementation” or the like, various features are described, which may be variously combined and included in some implementations, but also variously omitted in other implementations. Similarly, various features are described that may be preferences or requirements for some implementations, but not other implementations.

The language used herein has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. Other implementations, uses and advantages of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. The specification should be considered exemplary only, and the scope of the invention is accordingly intended to be limited only by the following claims.