Patent Description:
The specification relates to a video application that generates compressed videos that are consistent across different device platforms.

People all around the world capture millions of videos every day. By encoding videos every nth fraction of a second, video files are large. For example, a one-minute video with encoding at <NUM> megabits (Mbits) per second is <NUM> megabytes (MB). Videos are often captured in markets or conditions where storing or transferring large files is impractical or cost prohibitive. For example, where the devices use <NUM> cellular connections or where the device storage is very limited. Possible solutions to the problem include video compression to smaller compressed output bitrates.

However, video compression is problematic because, when software decoders and encoders are used, the device video compression can be slow, contain errors, or have inconsistent bitrates. The error may be, for example, a green line in the video. The bitrate may be different for devices manufactured by different companies producing different bitrates for the same input. In addition to variability, it is disadvantageous to produce compressed videos of smaller sizes because a size of the video is correlated with its quality.

Work of the presently named inventor, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

<CIT> discloses a method of making a coding for images with a two-pass encode or a pre-analysis, wherein the control is taken so that code quantity is suitably given to a buffer for coding, and an image coding device that makes a compression and coding for moving images, comprising means for: pre-analyzing images, which exist in a constant interval, prior to making a coding for the images that were input to observe characteristics of each image; based on said observed characteristics, estimating complexity degrees of the images; based on said estimated complexity degrees, allocating code quantity to the images in a constant interval, of which the head is the image that was not coded yet, to compute target code quantity of each image for all images in the above interval; calculating a transition of occupancy in a buffer for coding within said interval of said code in assigning said computed target code quantity to each of said images to regulate the target code quantity so that the buffer for coding does not give rise to an overflow or an underflow; and making a compression and coding for the images, which were not coded yet, according to said regulated target code quantity.

Enabling disclosure for the protected invention is provided with the embodiments described in relation to <FIG>. The other figures, aspects, and embodiments are provided for illustrative purposes and do not represent embodiments of the invention unless when combined with all of the features respectively defined in the independent claims.

The disclosure is illustrated by way of example, and not by way of limitation in the FIGs of the accompanying drawings in which like reference numerals are used to refer to similar elements.

A video application may receive a request to compress an input video to generate an actual video. The request includes a request bitrate, which is used to obtain an actual bitrate of the actual video. The request bitrate may be different from the actual bitrate of the actual video, but is provided to the encoder because it results in the actual video having the actual bitrate. As a result of the process described below, the video application generates compressed videos that have a similar resolution and a similar bitrate independent of the device that receives the video.

In some embodiments, the video application parses parameters of an input video, the parameters including an input width, an input height, and an input format. The video application generates a blank video with a fixed duration based on the parameters of the input video. The video application generates a representative video based on providing the blank video as input to a decoder. The video application compresses the input video with a specified output width and output height.

<FIG> illustrates a block diagram of an example system <NUM> that generates compressed videos. The illustrated system <NUM> includes a video server <NUM>, a user device 115a, a user device 115n, a second server <NUM>, and a network <NUM>. The user 125a may be associated with the user device 115a and the user 125n may be associated with the user device 115n. In some embodiments, the system <NUM> may include other servers or devices not shown in <FIG>. In <FIG> and the remaining FIGs, a letter after a reference number, e.g., "103a," represents a reference to the element having that particular reference number. A reference number in the text without a following letter, e.g., "<NUM>," represents a general reference to embodiments of the element bearing that reference number. Although only one video server <NUM>, one user device 115a, one user device 115n, one second server <NUM>, and one network <NUM> are illustrated in <FIG>, persons of ordinary skill in the art will recognize that <FIG> may include one or more video servers <NUM>, one or more user devices 115a, one or more user devices 115n, one or more second servers <NUM>, and one or more networks <NUM>.

The video server <NUM> may include a processor, a memory, and network communication capabilities. In some embodiments, the video server <NUM> is a hardware server. The video server <NUM> is communicatively coupled to the network <NUM> via signal line <NUM>. Signal line <NUM> may be a wired connection, such as Ethernet, coaxial cable, fiber-optic cable, etc., or a wireless connection, such as Wi-Fi®, Bluetooth®, or other wireless technology. In some embodiments, the video server <NUM> sends and receives data to and from one or more of the user device 115a, the user device 115n, and the second server <NUM> via the network <NUM>. The video server <NUM> may include a video application 103a and a database <NUM>.

The video application 103a may be code and routines operable to provide compressed videos to the user devices 115a, 115n. In some embodiments, the video application 103a receives an input video, for example, from the database <NUM> on the video server <NUM> or from the second server <NUM>. The video application 103a may calibrate the input video, compress the input video into a compressed video, and verify that the compressed video has expected attributes. For example, the video application 103a may confirm that the compressed video has a resolution and a bitrate that conform to a resolution and bitrate provided as default values or by a user. Compressing the input video may include copying metadata from the input video and add it to the compressed video.

In some embodiments, the video application 103a may be implemented using hardware including a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). In some embodiments, the video application 103a may be implemented using a combination of hardware and software.

The database <NUM> may store input videos, blank videos, representative videos, and actual video. The database <NUM> may also store social network data associated with the users 125a, 125n, user preferences for the user 125a and/or the user 125n, etc..

The user device 115a may be a computing device that includes a memory and a hardware processor. For example, the user device 115a may include a desktop computer, a mobile device, a tablet computer, a mobile telephone, a wearable device, a head-mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device capable of accessing a network <NUM>.

In the illustrated implementation, the user device 115a is coupled to the network <NUM> via signal line <NUM> and the user device 115n is coupled to the network <NUM> via signal line <NUM>. Signal lines <NUM> and <NUM> may be a wired connection, such as Ethernet, coaxial cable, fiber-optic cable, etc., or a wireless connection, such as Wi-Fi®, Bluetooth®, or other wireless technology. The user device 115a is accessed by a user 125a and the user device 115n is accessed by a user 125n.

In some embodiments, the user device 115a is included in a wearable device worn by the user 125a. For example, the user device 115a is included as part of a clip (e.g., a wristband), part of jewelry, or part of a pair of glasses. In another example, the user device 115a can be a smart watch. The user 125a may view data associated with the video application <NUM> on a display of the device worn by the user 125a. For example, the video application 103a may display actual videos on a display of a smart watch or a smart wristband.

In some embodiments, video application 103b may be stored on a user device 115a. The video application 103b may be operable to receive the actual video from the video server <NUM> and display the actual video on a display of the user device 115a.

The user device 115n may be a computing device that includes a memory and a hardware processor. For example, the user device 115n may include a desktop computer, a mobile device, a tablet computer, a mobile telephone, a wearable device, a head-mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device capable of accessing a network <NUM>.

In some embodiments, the user device 115n includes a video application 103c. The video application 103c may be operable to receive the actual video and display the actual video on a display of the user device 115n. The actual video displayed on the user device 115n may have the same resolution and size as the actual video displayed on the user device 115a.

The second server <NUM> may include a processor, a memory, and network communication capabilities. The second server <NUM> may access the network <NUM> via signal line <NUM>. The second server <NUM> may provide services to the video server <NUM>, the user device 115a, and/or the user device 115n. For example, the second server <NUM> may generate input videos and provide the input videos to the video server <NUM> for compression.

In the illustrated implementation, the entities of the system <NUM> are communicatively coupled via a network <NUM>. The network <NUM> may be a conventional type, wired or wireless, and may have numerous different configurations including a star configuration, token ring configuration or other configurations. Furthermore, the network <NUM> may include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), and/or other interconnected data paths across which multiple devices may communicate. In some embodiments, the network <NUM> may be a peer-to-peer network. The network <NUM> may also be coupled to or include portions of a telecommunications network for sending data in a variety of different communication protocols. In some embodiments, the network <NUM> includes Bluetooth® communication networks, WiFi®, wireless local area network (WLAN) computer communication specified by IEEE <NUM>, or a cellular communications network for sending and receiving data including via short messaging service (SMS), multimedia messaging service (MMS), hypertext transfer protocol (HTTP), direct data connection, email, etc. Although <FIG> illustrates one network <NUM> coupled to the user devices <NUM> and the video server <NUM>, in practice one or more networks <NUM> may be coupled to these entities.

<FIG> illustrates a block diagram of an example video server <NUM> that generates compressed videos according to some embodiments. Although <FIG> is illustrated as being a video server <NUM>, some or all of the functions may be performed by the user devices 115a, 115n in whole or in part. The video server <NUM> may include a processor <NUM>, a memory <NUM>, a communication unit <NUM>, and a database <NUM>. Additional components may be present or some of the previous components may be omitted depending on whether the steps are all performed by the video server <NUM> or the user device <NUM>.

The video server <NUM> may store the video application 103a in the memory <NUM>. In embodiments where the video server <NUM> is a wearable device, the video server <NUM> may not include database <NUM>. In some embodiments, the video server <NUM> may include other components not listed here, such as a battery, etc. The components of the video server <NUM> may be communicatively coupled by a bus <NUM>.

The processor <NUM> includes an arithmetic logic unit, a microprocessor, a general purpose controller, or some other processor array to perform computations and provide instructions to a display device. Processor <NUM> processes data and may include various computing architectures including a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or an architecture implementing a combination of instruction sets. Although <FIG> includes a single processor <NUM>, multiple processors <NUM> may be included. Other processors, operating systems, sensors, displays and physical configurations may be part of the user device 115a. The processor <NUM> is coupled to the bus <NUM> for communication with the other components via signal line <NUM>.

The memory <NUM> stores instructions that may be executed by the processor <NUM> and/or data. The instructions may include code for performing the techniques described herein. The memory <NUM> may be a dynamic random access memory (DRAM) device, a static RAM, or some other memory device. In some embodiments, the memory <NUM> also includes a non-volatile memory, such as a static random access memory (SRAM) device or flash memory, or similar permanent storage device and media including a hard disk drive, a compact disc read only memory (CD-ROM) device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, or some other mass storage device for storing information on a more permanent basis. The memory <NUM> includes code and routines operable to execute the video application <NUM>, which is described in greater detail below. The memory <NUM> is coupled to the bus <NUM> for communication with the other components via signal line <NUM>.

The communication unit <NUM> transmits and receives data to and from at least one of the user device 115a and the video server <NUM> depending upon where the video application <NUM> may be stored. In some embodiments, the communication unit <NUM> includes a port for direct physical connection to the network <NUM> or to another communication channel. For example, the communication unit <NUM> includes a universal serial bus (USB), secure digital (SD), category <NUM> cable (CAT-<NUM>) or similar port for wired communication with the user device 115a or the video server <NUM>, depending on where the video application <NUM> may be stored. In some embodiments, the communication unit <NUM> includes a wireless transceiver for exchanging data with the user device 115a, video server <NUM>, or other communication channels using one or more wireless communication methods, including IEEE <NUM>, IEEE <NUM>, Bluetooth® or another suitable wireless communication method. The communication unit <NUM> is coupled to the bus <NUM> for communication with the other components via signal line <NUM>.

In some embodiments, the communication unit <NUM> includes a cellular communications transceiver for sending and receiving data over a cellular communications network including via short messaging service (SMS), multimedia messaging service (MMS), hypertext transfer protocol (HTTP), direct data connection, e-mail or another suitable type of electronic communication. In some embodiments, the communication unit <NUM> includes a wired port and a wireless transceiver. The communication unit <NUM> also provides other conventional connections to the network <NUM> for distribution of files and/or media objects using standard network protocols including, but not limited to, user datagram protocol (UDP), TCP/IP, HTTP, HTTP secure (HTTPS), simple mail transfer protocol (SMTP), SPDY, quick UDP internet connections (QUIC), etc..

The database <NUM> may be a non-transitory computer-readable storage medium that stores data that provides the functionality described above with reference to <FIG>. The database <NUM> may be a DRAM device, a SRAM device, flash memory or some other memory device. In some embodiments, the database <NUM> also includes a non-volatile memory or similar permanent storage device and media including a hard disk drive, a CD-ROM device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, or some other mass storage device for storing information on a permanent basis. The database <NUM> is coupled to the bus <NUM> for communication with the other components via signal line <NUM>.

The video application 103a may include an encoder <NUM> and a decoder <NUM>.

The encoder <NUM> may include code and routines for encoding a video through compression. In some embodiments, the encoder <NUM> includes a set of instructions executable by the processor <NUM> to encode the video. In some embodiments, the encoder is stored in the memory <NUM> of the video server <NUM> and is accessible and executable by the processor <NUM>.

In some embodiments, the encoder <NUM> calibrates an input video, compresses the input video into a compressed video, adds metadata from the input video to the compressed video, and verifies that the compressed video has proper attributes.

The encoder <NUM> may receive an input video V(I) of resolution W(I) x H(I) and bitrate B(I). The goal is to compress the input video V(I) to generate an actual video based on resolution W(O) x H(O) and output bitrate B(O). In some embodiments, resolution W(O) x H(O) are specified by a user or a default resolution based on the goal of having an output video V(O) with a reduced file size. To reduce the file size of the actual video as compared to the input video, W(O) x H(O) have to be smaller than W(I) x H(I) and/or B(O) has to be smaller than B(I).

The encoder <NUM> performs calibration by solving the following problem. Compression of the input video requires asking the encoder <NUM> to perform video compression and generate output bitrate B(O). However, the encoder <NUM> generates bitrate B(X), which can be substantially different than the output bitrate B(O). As a result, the encoder performs calibration to determine what request bitrate B(Y) should be requested in order to achieve a bitrate as close as possible to the output bitrate B(O).

As a result, the encoder <NUM> parses the parameters of the input video V(I). Because the request bitrate B(Y) drastically changes based on the input width W(I), the input height H(I), the input format, and the input frame rate F(I) of the input video V(I), these parameters are useful for calibration.

The encoder <NUM> generates a representative blank video V(B) with a fixed duration D(B) using the W(I), H(I), frame rate F(I), and the format of the input video. The fixed duration D(B) is chosen such that this step can be performed quickly, but is not too short. For example, the fixed duration D(B) may be between <NUM>-<NUM> seconds. Encoders tend to optimize shorter videos leading to incorrect calibrations. Instead, the encoder <NUM> generates a blank video by rendering F(I) x D(B) black frames because the blank video can be generated very quickly and at a tiny file size.

The encoder <NUM> generates a representative video V(P) using the blank video V(B) as input to the decoder <NUM>. Decoders and encoders are often paired such that a valid input video is required to generate a representative video. A graphics framework, such as OpenGL is used to generate a geometric texture on top of V(P). The texture changes colors frame by frame, mimicking the contents of a real video. After generation, the representative video's V(P)'s bitrate can be compared to the output bitrate B(O) to determine the request bitrate B(Y).

The encoder <NUM> requests the input video V(I) to be compressed to W(O) x H(O) with request bitrate B(Y), which was determined in the previous step, to get actual video V(A) with resolution width W(A) x height H(A) and actual bitrate B(A). The encoder <NUM> may also copy audio streams in this step.

The encoder <NUM> performs verification by comparing the input video I(V) with the actual video V(A). In some embodiments, the encoder <NUM> determines whether the actual width and the actual height are within a threshold width value and a threshold height value of the output width and the output height. If the actual width and the actual height are not within the threshold width value, the encoder <NUM> may determine that the compression process failed. In some embodiments, the encoder <NUM> determines whether a number of frames of the actual video are a same number as the number of frames of the input video. If the number of frames of the actual video are not the same number as the number of frames of the input video, the encoder <NUM> may determine that the compression process failed.

In some embodiments, the encoder <NUM> verifies the actual video by confirming that the actual bitrate of the actual video is within a threshold bitrate value of the output bitrate. If the actual bitrate is not within the threshold bitrate value, the encoder <NUM> may modify the parameters and requesting that the input video be compressed with the output width and the output height based on modified parameters. For example, the encoder <NUM> may use a longer fixed duration for the representative video.

In instances where the encoder <NUM> added or attempted to add one or more audio streams to the actual video, the encoder <NUM> may compare the input video to the actual video to confirm that the actual video includes the one or more audio streams. If the actual video fails to include the one or more audio streams, the encoder <NUM> may determine that the actual video failed to have the resolution and bitrate that were specified at the beginning of the process. For example, the resolution and the bitrate may be default values or specified by a user.

In some embodiments, the encoder <NUM> may compare the input video to the actual video to confirm that the output width multiplied by the output height is equal to a common resolution. The encoder <NUM> may perform this comparison on all frames or a subset of frames. If the output width multiplied by the output height fails to equal the common resolution, the encoder <NUM> may use a higher bitrate to compress the input video. The encoder <NUM> may generate a new actual video and confirm whether the new actual video satisfies the verification step.

Many encoders encode video formats in a way where the video consists of frames of two or more different types: key frames and delta frames. A key frame is a full representation of the image in the frame and does not depend on previous frames. Conversely, delta frames encode the different between a current frame and a previous frame.

In situations where the encoder compares only a subset of frames, to save time the encoder <NUM> skips rendering of certain frames. However, the key frames for the input video and the actual video may be located in different positions. As a result, in some embodiments, the encoder <NUM> determines, for each frame to be compared, a closest key frame before that in both the input video and the actual video and ensures that the decoder <NUM> decodes the key frame and all delta frames after it, before reaching the frame that is needed for the comparison.

The encoder <NUM> may use different techniques to compare an actual image in the frames. Because there are common errors associated with the comparison, and one in particular is that a line in the video, such as a left edge of the video is discolored, the encoder <NUM> may perform the following comparison:.

Given two images, S and T, both of size (x, y) containing c channels, where each channel value is represented as a floating point number between <NUM> and <NUM>, the encoder <NUM> computes:.

We determine if the images are similar enough by comparing the error with a threshold value.

The encoder <NUM> may perform metadata copying. When the input video is compressed it can lose some extra metadata, e.g., creation time. The encoder <NUM> may correct the actual video by merging it into all the required metadata from the input video. For example, for creation time, if the input file is an MP4 file, it may consist of copying creation time from MVHD, TKHD, and MDHD atoms from the input video to create the actual video.

The decoder <NUM> may include code and routines for decoding a video. In some embodiments, the decoder <NUM> includes a set of instructions executable by the processor <NUM> to decode the video. In some embodiments, the decoder is stored in the memory <NUM> of the video server <NUM> and is accessible and executable by the processor <NUM>.

In some embodiments, the decoder <NUM> receives a blank video V(B) from the encoder <NUM>, decodes the blank video, and provides parameters associated with the decoded blank video to the encoder.

<FIG> illustrates a flowchart of an example method <NUM> to generate compressed videos according to some embodiments. The method <NUM> is performed by any combination of a video application 103a stored on a video server <NUM> and a video application 103b stored on a user device 115a of <FIG>.

At step <NUM>, an input video is calibrated. For example, the encoder <NUM> performs calibration to determine what request bitrate B(Y) should be provided to the encoder <NUM> with a request to compress the input video to generate the actual video in order to achieve a bitrate that is as close as possible to an output bitrate B(O).

At step <NUM>, the input video is compressed into an actual video using an output width and an output height. In some embodiments, requesting that the input video be compressed to the actual video results in an actual video with an actual width, an actual height, and an actual bitrate. If the calibration process worked well, the actual video should have a similar resolution (e.g., actual width and actual height) to the input video.

At step <NUM>, it is verified that the actual video has attributes that satisfy predetermined criteria. For example, verification includes confirming that the actual width and the actual height are within a threshold width value and a threshold height value of the output width and the output height. If the actual width and the actual height are not within the threshold width value and the threshold height value, it is determined that the compression failed.

<FIG> illustrates a flowchart of another example method <NUM> to generate a compressed video according to some embodiments. The method <NUM> is performed by any combination of a video application 103a stored on a video server <NUM> and a video application 103b stored on a user device 115a of <FIG>.

At step <NUM>, an output bitrate is identified. The output bitrate may be specified by a user or a default value. An output width and an output height may also be identified. At step <NUM>, parameters of an input video are parsed. The parameters include an input width, an input height, and an input format. At step <NUM>, a blank video is generated with a fixed duration based on the parameters of the input video. At step <NUM>, a representative video is generated based on providing the blank video as input to a decoder <NUM>. At step <NUM>, a request bitrate is determined based on the representative video and the output bitrate. According to the present invention, the request bitrate is determined by comparing a representative bitrate of the representative video to the output bitrate. At step <NUM>, the input video is compressed using the request bitrate to generate an actual video. In some embodiments, the input video may be further compressed using an output width and an output height.

In the above description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the specification. It will be apparent, however, to one skilled in the art that the disclosure can be practiced without these specific details. In some instances, structures and devices are shown in block diagram form in order to avoid obscuring the description. For example, the embodiments can be described above primarily with reference to user interfaces and particular hardware. However, the embodiments can apply to any type of computing device that can receive data and commands, and any peripheral devices providing services.

Reference in the specification to "some embodiments" or "some instances" means that a particular feature, structure, or characteristic described in connection with the embodiments or instances can be included in at least one implementation of the description. The appearances of the phrase "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiments.

Some portions of the detailed descriptions above are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. An algorithm is here, and generally, conceived to be a selfconsistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, or the like.

Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms including "processing" or "computing" or "calculating" or "determining" or "displaying" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices.

The embodiments of the specification can also relate to a processor for performing one or more steps of the methods described above. The processor may be a special-purpose processor selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including, but not limited to, any type of disk including optical disks, ROMs, CD-ROMs, magnetic disks, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memories including USB keys with non-volatile memory, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

The specification can take the form of some entirely hardware embodiments, some entirely software embodiments or some embodiments containing both hardware and software elements. In some embodiments, the specification is implemented in software, which includes, but is not limited to, firmware, resident software, microcode, etc..

Furthermore, the description can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.

Claim 1:
A method that generates, from an input video, a compressed video that is consistent across different devices, the method comprising:
identifying (<NUM>) a desired output bitrate;
parsing (<NUM>) parameters of the input video, the parameters including width, height, frame rate, and format of the input video;
generating (<NUM>), by an encoder (<NUM>) rendering black frames, a blank video with a fixed duration using the parameters of the input video;
generating (<NUM>) a representative video based on providing the blank video as input to a decoder (<NUM>);
wherein the decoder (<NUM>) receives the blank video from the encoder (<NUM>), decodes the blank video, and provides parameters associated with the decoded blank video to the encoder,
wherein the representative video is the decoded blank video;
determining (<NUM>) a request bitrate based on comparing a bitrate of the representative video to the desired output bitrate; and
compressing (<NUM>) the input video using the request bitrate to generate the compressed video.