Enhanced live multibitrate video encoding

Multibitrate (MBR) live video broadcasting is disclosed in which live video input is copied into a plurality of streams each designated for encoding into a different bitrate. The MBR broadcasting operation is made efficient by performing pre-quantization calculations only the first of the plurality of streams. The results of those calculations are then merely copied to the other streams. Quantization and encoding processes may then be applied to each of the streams to process the streams into their respective, predetermined bitrates.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to co-pending, and commonly assigned U.S. patent application Ser. No. 12/201,952 entitled DYNAMICALLY ALTERING PLAYLISTS, the disclosure of which is hereby incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates, in general, to streaming media, and, more particularly, to enhanced live multibitrate (MBR) video encoding.

BACKGROUND

As networking technology and bandwidth capabilities have increased, the delivery of richer multimedia resources has also increased and improved in quality and accessibility. Internet services such as Google Inc.'s YOUTUBE™ allow individuals to access video content that has been stored to remote servers. Services are also available that provide web seminars, called “webinars”, in which live or recorded multimedia or video content is broadcast to participants who register or request to receive the streaming data. Live video broadcasting, in particular, provides numerous challenges to ensure that users receive a reliable and cognizable representation of the live content.

In general, streaming live video content entails a number of steps to process the raw, image data into the compressed/encoded format to be transmitted to the recipient. Raw video data captured by a video recorder is typically input into a computer-based signal processing system for encoding into the resulting format and bitrate. The raw video data generally comes into the system divided into a series of frames. Each frame represents a snap shot of the live content based on the recording or sampling rate of the recording equipment, which is typically given in frames per second (fps). The encoding system may determine various frames to drop from the stream in order to meet a certain bitrate or quality requirement. Furthermore, a variety of different filtering or computational processes may be applied to the frames to reduce noise or change the resulting size of the frames.

In the encoding process, each frame is broken down into multiple macroblocks, which are blocks of pixels measuring 8 pixels-by-8 pixels. The macroblocks are then analyzed and assigned a particular mode based on the relative content between the macroblock and a previous macroblock. In order to conserve bandwidth, macroblocks can be encoded either as intra-mode blocks, in which all of the video information in the block is preserved and encoded, or as inter-mode blocks, in which only the video information representing the difference or delta from another macroblock is encoded. This process is similar to the animation process in which key frames include all of the information for the scene and subsequent frames until the next key frame only include the stepped changes from the key frame. Assigning macroblock modes generally entails comparing the current macroblock with one or more previous macroblocks and analyzing any changes that occur in any of the elements in the image data. Based on the level of movement or change in such elements, the encoding system will determine whether the current macroblock should be an intramode block or an intermode block.

Depending on the particular encoding scheme, there are various different types of macroblock modes. However, in general, those various types can still be broken into mode that do not depend on any other macroblock, i.e., intramode blocks, and nodes that depend on other blocks in order to calculate a difference or other such relationship, i.e., intermode blocks.

Once the appropriate mode is assigned to a macroblock, the macroblock image data is processed or transformed into the frequency domain by applying a Fourier-related transform to it. Typically, a discrete cosine transform (dct), which is a type of Fourier transform, is used in signal compression. The image processing is performed on all of the image data in intramode macroblocks and on the delta information in intermode blocks. After processing the image data using the transform, the result is typically divided by the quantization value. The quantization value is a measure of the detail that is desired to represent the sampled continuous signal data in the digital signal. The quantization value will determine how many bits will be used to represent that signal, and, thus, is related to the quality and the bitrate of the desired data stream. Therefore, the specific quantization value used will be determined based on the bitrate and quality intended. The result of quantizing the transformed signal data generally results in 64 coefficients. These coefficients, of which all or a subset may be used depending on the bitrate, are then used to encode or compress the macroblocks. Each macroblock of each frame of the live streaming video goes through this computationally intensive process.

Presentation of video or multimedia content is not generally limited to only those users having a particular bandwidth or bitrate availability. Video may be delivered over various bitrates according to the bitrate that the user has access to or desires to use. Thus, it is common practice to provide live video broadcasts in multiple bitrates (MBRs). When the user requests access to the live broadcast, he or she will select a desired bitrate to use. In general services that are providing the live video broadcast generate the encoded streams in the various bitrates that are offered to users so that the user simply selects the desired bitrate after which the broadcasting system directs the video stream associated with that bitrate to the user.

These services offering MBR live video broadcasting perform each of the computationally intensive processes on each stream intended for the different bitrates. Therefore, there is a large computational and processing requirement for providing such MBR live video broadcasting services. These computational and processing requirements may limit the number of services that may be capable of providing such MBR broadcasts.

BRIEF SUMMARY

The embodiments presented in this disclosure are directed to systems, methods, and computer program products that process live video streams for MBR live video broadcasting. At least a first and second of streams are generated from the live video stream, each such stream designated for encoding into a different bitrate. The teachings herein provide efficient operation of MBR broadcasting by performing pre-quantization calculations only the first of the streams. The results of those calculations are then applied to the other streams. Quantization and encoding processes may then be applied to each of the streams to process the streams into their respective predetermined bitrates.

Representative embodiments of the present teaching are directed to methods that include receiving a live video feed to be broadcast at a plurality of bitrates, wherein the live video feed comprises a plurality of frames and wherein each of the plurality of frames comprises a plurality of macroblocks. The methods further include generating at least first and second streams from the live video feed, wherein each of the streams is identified for transmission at a different bitrate from each other, calculating an encoding mode for each macroblock making up the first stream, applying the calculated encoding mode from the first stream to the second stream, image processing image data on each macroblock of the first stream using a Fourier-related transform, applying results of the image processing to the second stream, quantizing each of the streams according to the different bitrates identified for the each stream, and compressing each of the streams for broadcast according to the identified different bitrates.

Additional representative embodiments of the present disclosure are directed to computer implemented systems that include a processor, memory coupled to the processor, a multibitrate (MBR) live video broadcasting application stored in the memory, and a network interface configured to broadcast the encoded video streams at the predetermined bitrates. When executed by the processor, the MBR live video broadcasting application includes an input interface configured to receive raw video input, a stream separation component configured to copy a plurality of frames of the raw video input into a plurality of video streams, wherein the plurality of frames is made up of a plurality of macroblocks. Each of the video streams is designated for a different predetermined bitrate from the other video streams. The computer implemented systems also include a macroblock mode component configured to calculate a mode of a current macroblock from a first video stream, a mode copy component configured to copy the calculated mode to the remaining video streams, and an encoding component configured to encode each of the video steams according to their associated predetermined bitrates.

Still further embodiments of the present teaching are directed to computer program products having a computer readable medium with computer program logic recorded thereon. The computer program product includes code for receiving a live video feed to be broadcast at a plurality of bitrates, wherein the live video feed comprises a plurality of frames and wherein each of the plurality of frames comprises a plurality of macroblocks. The computer program products also include code for generating at least first and second streams from the live video feed, wherein each of the first and second streams is identified for transmission at a different bitrate from each other, code for calculating an encoding mode for each macroblock of the plurality of macroblocks making up the first stream, code for applying the calculated encoding mode from the first stream to the second stream, code for image processing image data on each macroblock of the first stream using a Fourier-related transform, code for applying results of the image processing to the second stream, code for quantizing each of the first and second streams according to the identified different bitrates for the each stream, and code for compressing each of the first and second streams for broadcast according to the identified different bitrate.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are set forth to provide a thorough understanding of claimed subject matter. However, it will be understood by those skilled in the art that claimed subject matter may be practiced without these specific details. In other instances, methods, apparatuses or systems that would be known by one of ordinary skill have not been described in detail so as not to obscure claimed subject matter. Some portions of the detailed description which follow are presented in terms of algorithms or symbolic representations of operations on data bits or binary digital signals stored within a computing system memory, such as a computer memory. These algorithmic descriptions or representations are examples of techniques used by those of ordinary skill in the data processing arts to convey the substance of their work to others skilled in the art.

An algorithm is here, and generally, considered to be a self-consistent sequence of operations or similar processing leading to a desired result. In this context, operations or processing involve physical manipulation of physical quantities. Typically, although not necessarily, such quantities may take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared or otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to such signals as bits, data, values, elements, symbols, characters, terms, numbers, numerals or the like. It should be understood, however, that all of these and similar terms are to be associated with appropriate physical quantities and are merely convenient labels. Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout this specification discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining” or the like, refer to actions or processes of a computing platform, such as a computer or a similar electronic computing device, that manipulates or transforms data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.

FIG. 1is a diagram illustrating MBR live video broadcasting system10. Live event100is being recorded by video camera101. Video camera101is coupled to broadcast server102, which is configured according to one embodiment of the teachings herein. Broadcast server102is coupled to network103, which may be any various network, such as a wide area network (WAN), local area network (LAN), or network such as the Internet. Video system10is configured to offer MBR broadcasting of live event100. Video system10offers broadcast streams in 300 kilobits per second (Kbps), 600 Kbps, 1000 Kbps, and 1500 Kbps.

Users wishing to view the live broadcast connect, in some fashion, to network103. Once connected to network103the users access broadcast server102, request access to the live broadcast of live event100, select the appropriate bitrate, and then begin receiving the broadcast material. For the sake of clarity of the described embodiment, the user devices connecting to network103will be referred to as users104-107. In operation, users104-107comprise users at their respective devices using those devices to access network103.

It should be noted that various users desiring to view the live broadcast may connect to network113in a number of different ways. For example, mobile phone users may connect to network103using a wireless telecommunication network. Other users may connect to network103using a short range wireless technology, such as Institute of Electrical and Electronics Engineers (IEEE) 802.11, or the like. Still other users may connect through wired networks, LANs, WANs, and the like.

User's104and107, through desktop computers, connect to network103via LAN and select the 1500 Kbps bitrate to receive the live video feed of live event100. User105, through a notebook computer, connects to network103via a IEEE 802.11g connection and selects the 1000 Kbps bitrate to receive the live video feed. User106, through a mobile phone, connects to network103via a mobile telecommunications network and selects the 300 Kbps bitrate to receive the live video feed.

Broadcast server102provides its MBR video streams without performing all computations and processes on each of the available video streams. Instead, prior to quantization, the calculations and computations are performed of one of the streams and then the results of the computations are merely applied to the other streams. In this manner, the amount of computations are drastically reduced while still providing the same number of streams.

FIG. 2is an operational chart illustrating operations performed by broadcast server20as configured according to one embodiment of the present teachings. Input frame200enters broadcast server20for preparation of each bitrate stream. Broadcast server20maintains separate processes for managing operations on the various streams that will be produced. Various code components are stored on broadcast server20for implementing operations of broadcasting live video content. Stream separator operation201creates stream structures202-205by copying input frame200into separate stream structures202-205. The copies of input frame200maintain the size of input frame200for each of streams202-205. Stream structures202-205examine their respective frames and make a determination whether or not to drop the frame from the stream in frame drop operation206. The decision of whether to drop is made by a majority of separate processes managing streams202-205. The decision may be based on bitrate limitations, frame delay, or the like.

Noise reduction operation207performs noise reducing filter calculations on stream202. Block208indicates that while noise reduction operation207operates on stream202, no operations are being performed on streams203-205. As the results of the filter calculations are obtained from noise reduction operation207, those results are simply copied from stream202onto streams203-205in copy operation209. While copy operation209uses processor time in performing the copying operations, the copying operations performed are not as computationally intensive as the filtering calculations of noise reduction operation207.

The frame in stream202is divided into macroblocks at macroblock operation210. Block211indicates that while macroblock operation210operates on stream202, no operations are being performed on streams203-205. The determined macroblocks are then copied over to streams203-205in copy operation212. Macroblock mode operation213next analyzes the macroblocks in stream202comparing the current macroblocks against previous macroblocks in stream202in order to determine whether the macroblocks should be intermode macroblocks or intramode macroblocks. Block214indicates that while macroblock mode operation213operates on stream202, no operations are being performed on streams203-205. After each of the macroblock modes are determined for stream202, they are copied over to streams203-205in copy operation215. Again, the computational intensity of copying the results from stream202onto streams203-205is far less than performing the comparisons and analysis for determining the macroblock mode for each of streams203-205.

Dct operation216performs a discrete cosine transformation (dct) on the image data of the macroblocks of stream202. For each intramode block of stream202, the dct is performed on all of the image data making up the macroblock of the frame. When intermode blocks are encountered, the dct processes only the image data representing the difference or delta between the represented macroblock and one or more of the chronologically previous macroblocks in stream202. Block217indicates that while dct operation216operates on the macroblocks of stream202, no operations are being performed on streams203-205. The results of the dct are then copied from stream202onto streams203-205at copy operation218.

Quantization operation219performs quantization on each of streams202-205. A quantization value is selected based on the desired bitrate of the stream. Thus, at quantization operation219, the quantizing calculations are performed on each of streams202-205. The selected quantization value divides the dct of the macroblocks resulting in a number of coefficients. The resulting coefficients are then used in compression operation220to arithmetically compress each of streams202-205according to their associated bitrates. Once compression of streams202-205is complete, streams202-205are broadcast to a network through network interface221. Because far fewer computations and calculations were performed in broadcast server20, the preparation of MBR live video streams occurs much more efficiently than if those calculations were performed on all of the bitrate streams offered by broadcast server20.

It should be noted that, prior to quantization, various other types of operation calculations may be performed on the macroblocks of one of the streams

FIG. 3is a flowchart illustrating example steps executed to implement one embodiment of the present teaching. In step300, a live video feed is received for broadcasting at a plurality of bitrates, wherein the live video feed comprises a plurality of frames and wherein each of the plurality of frames comprises a plurality of macroblocks. At least first and second streams are generated, in step301, from the live video feed, wherein each of the first and second streams is identified for transmission at a different bitrate from each other. An encoding mode is calculated, in step302, for each macroblock of the plurality of macroblocks making up the first stream. The calculated encoding mode is applied, in step303, from the first stream to the second stream. Image data is processed, in step304, on each macroblock of the first stream using a Fourier-related transform. The image processing results are then applied to the second stream in step305. Each of the streams is quantized, in step306, according to the different bitrates identified for the each stream. Each of the first and second streams is compressed for broadcast, in step307, according to the identified different bitrates.

It should be noted that in some encoding and compression systems, the standard of the system prevents the quality level of the encoded/compressed stream to fall below a certain, designated minimum level. In such systems, the system would prevent the quality level of a stream encoded for a higher level to drop to a quality that is standard for another lower bitrate stream. Embodiments of the present teachings provide specific allowances for this quality degradation to occur in order to gain in the efficiency of performing the majority of the computationally intensive calculations on only one of a number of live video streams in an MBR system.

FIG. 4is a flowchart illustrating example steps executed to implement another embodiment of the present teachings. In step400, a live video feed is received from a video capture device. The live video feed is copied, in step401, into a plurality of streams, wherein each of the plurality of streams is identified for transmission at a different bitrate. An encoding mode is calculated, in step402, for each macroblock of a plurality of frames making up the highest bitrate stream of the plurality of streams. The determined encoding mode is copied, in step403, from the first of the plurality of streams to a remainder of the plurality of streams. In step404, a frame modification function, such as a noise reduction operation, a frame size modification, or an image processing operation is performed on the first of the plurality of streams, wherein the frame modification function operates to modify a current frame. The modified current frame is copied from the first of the plurality of streams, in step405, to the remainder of the plurality of streams. Image processing is performed using a Fourier-related transform on the image data of each macroblock of the first of the plurality of streams in step406. The results of the image processing are copied to the remainder of the plurality of streams in step407. Each stream of the plurality of streams is quantized, in step408, according to the different bitrate associated with the each stream. In step409, the quality of one or more of the plurality of streams identified for a lower bitrate is allowed to degrade below a minimum quality associated with the encoding mode for the highest bitrate stream. Each stream is them compressed for broadcast, in step410, according to the different bitrates.

Embodiments, or portions thereof, may be embodied in program or code segments operable upon a processor-based system (e.g., computer system) for performing functions and operations as described herein. The program or code segments making up the various embodiments may be stored in a computer-readable medium, which may comprise any suitable medium for temporarily or permanently storing such code. Examples of the computer-readable medium include such tangible computer-readable media as an electronic memory circuit, a semiconductor memory device, random access memory (RAM), read only memory (ROM), erasable ROM (EROM), flash memory, a magnetic storage device (e.g., floppy diskette), optical storage device (e.g., compact disk (CD), digital versatile disk (DVD), etc.), a hard disk, and the like.

Embodiments, or portions thereof, may be embodied in a computer data signal, which may be in any suitable form for communication over a transmission medium such that it is readable for execution by a functional device (e.g., processor) for performing the operations described herein. The computer data signal may include any signal that can propagate over a transmission medium such as electronic network channels, optical fibers, air, electromagnetic media, radio frequency (RF) links, and the like, and thus the data signal may be in the form of an electrical signal, optical signal, radio frequency or other wireless communication signal, etc. The code segments may, in certain embodiments, be downloaded via computer networks such as the Internet, an intranet, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), the public switched telephone network (PSTN), a satellite communication system, a cable transmission system, and/or the like.

FIG. 5illustrates an exemplary computer system500which may be employed to implement the broadcast servers and operations therein according to certain embodiments. Central processing unit (CPU)501is coupled to system bus502. CPU501may be any general-purpose CPU. The present disclosure is not restricted by the architecture of CPU501(or other components of exemplary system500) as long as CPU501(and other components of system500) supports the inventive operations as described herein. CPU501may execute the various logical instructions described herein. For example, CPU501may execute machine-level instructions according to the exemplary operational flow described above in conjunction withFIGS. 3 and 4. When executing instructions representative of the operational steps illustrated inFIGS. 3 and 4, CPU501becomes a special-purpose processor of a special purpose computing platform configured specifically to operate according to the various embodiments of the teachings described herein.

Computer system500also includes random access memory (RAM)503, which may be SRAM, DRAM, SDRAM, or the like. Computer system500includes read-only memory (ROM)504which may be PROM, EPROM, EEPROM, or the like. RAM503and ROM504hold user and system data and programs, as is well known in the art.

Computer system500also includes input/output (I/O) adapter505, communications adapter511, user interface adapter508, and display adapter509. I/O adapter505, user interface adapter508, and/or communications adapter511may, in certain embodiments, enable a user to interact with computer system500in order to input information.

I/O adapter505connects to storage device(s)506, such as one or more of hard drive, compact disc (CD) drive, floppy disk drive, tape drive, etc., to computer system500. The storage devices are utilized in addition to RAM503for the memory requirements associated performing the operations, copying and compressing the frame macroblocks. Communications adapter511is adapted to couple computer system500to network512, which may enable information to be input to and/or output from system500via such network512(e.g., the Internet or other wide-area network, a local-area network, a public or private switched telephony network, a wireless network, any combination of the foregoing). User interface adapter508couples user input devices, such as keyboard513, pointing device507, and microphone514and/or output devices, such as speaker(s)515to computer system500. Display adapter509is driven by CPU501to control the display on display device510to, for example, for setting up the various MBR to offer for broadcast. Display adapter509transmits instructions for transforming or manipulating the state of the various numbers of pixels used by display device510to visually present the desired information to a user. Such instructions include instructions for changing state from on to off, setting a particular color, intensity, duration, or the like. Each such instruction makes up the rendering instructions that control how and what is displayed on display device510.

It shall be appreciated that the present disclosure is not limited to the architecture of system500. For example, any suitable processor-based device may be utilized for implementing the MBR live video broadcast system, including without limitation personal computers, laptop computers, computer workstations, multi-processor servers, and even mobile telephones. Moreover, certain embodiments may be implemented on application specific integrated circuits (ASICs) or very large scale integrated (VLSI) circuits. In fact, persons of ordinary skill in the art may utilize any number of suitable structures capable of executing logical operations according to the embodiments.