Streamlined transcoder architecture

Systems and methods for a streamlined transcoder architecture. A transcoder system includes an encoder and a decoder. The encoder compares a decoded frame and a encoder reference frame to produce an output stream. The decoder produces the decoded frame including decoder reference frame and the encoder reference frame. The decoded frame is produced from an input stream, and the encoder reference frame is produced from the output stream of the encoder. In one embodiment, the encoder refines motion vectors, quantization, and macroblock type/mode from the input stream for reuse in the output stream. Furthermore, the decoded frames from the input stream can be modified in various ways including changing picture resolution and performing image enhancement on them before encoding.

FIELD OF THE INVENTION

The present invention relates generally to media processing, and more specifically, to transcoding of media streams.

BACKGROUND

Conventionally, multimedia such as video and audio has been delivered using analog delivery mediums such as NTSC (National Television System Committee) signals, and has been stored using analog storage mediums such as video cassette recorders. The analog signals typically contain uncompressed frames of video. Thus, a significant part of the electronic components in a display device are dedicated to analog receiving hardware, and if the display device has a digital output, electronic components are needed to convert the analog signal to a digital signal. With the advent of digital delivery mediums, such as ATSC (Advanced Television Systems Committee) signals, and of digital storage mediums and DVDs, multimedia can be delivered and stored using pure digital signals. Digital signals typically contain compressed frames of video.

Meanwhile, consumers and business have an increasing number of digital playback devices such as high-definition televisions, digital video recorders, MP3 players and the like. However, the digital playback devices are typically incompatible with each other in ways such as compression format, resolution, and encryption. Furthermore, the digital playback devices are likely to use a digital format that is optimized for particular storage and playback capabilities. For example, a high-definition television can display a conventional high-definition signal, but a standard-definition television or a portable video player typically can only display a standard-definition digital signal with different characteristics. Differences in digital formats can include encoding, bit rate, resolution, and the like.

Due to differences in conventional playback devices, there are limits in the types of digital formats that can be read or written by the devices. In order to handle more digital formats, the complexity of related hardware increases dramatically. One reason for this is that the digital formats are typically decompressed in order to perform operations in the spatial domain to make use of legacy analog techniques which operate on decompressed video. Decompressed multimedia, especially video, requires high-performance processing hardware to handle the high bandwidth for data transfers. Decompressed video also requires significant amounts of storage.

A particular need in digital media applications involves changing media from a first compression format into a second compression format. Such a need may arise, for example, when a digital media broadcast feed is in a format that is not compatible with a certain playback system. The need to change digital media formats is becoming increasingly pervasive as more digital broadcast, distribution, storage, processing, and playback systems are brought into use.

Traditional approaches to transcoding have involved the implementation of a complete decoder that is separate from a complete encoder. Because decoders and encoders are sophisticated components that are difficult to design, the encoder and the decoder are typically designed separately, with interaction between the two limited to the uncompressed video frames. Referring toFIG. 1, a decoder design101is responsible for decoding one or more motion vectors106, an error term108, and a variety of the compression parameters including quantization, macroblock type/mode, etc. to produce a decoded frame110. A separate and independent encoder design103is responsible for encoding the decoded frame110to produce an error term114, one or more motion vectors116and a variety of the compression parameters including quantization, macroblock type/mode, etc.

From time-to-time, a frame will be received without interframe compression—such frames are used to directly establish or refresh the reference frame112. For frames having interframe compression, one or more motion vectors106, an error term108, and a variety of the compression parameters including quantization, macroblock type/mode, etc. describe the currently decoded frame with reference to a previously decoded or received frame, the reference frame112. The decoder102applies the motion vectors106to the reference frame112, adds the error term108, and applies a variety of other compression parameters including quantization, macroblock type/mode, etc., to the resulting macroblock to produce a decoded frame110. The decoded frame110is stored for future use in the decoder as a reference frame112, and is the output of the decoder design101.

The decoded frame110is the input to the encoder design103. In the encoder design103, an encoder104compares the decoded frame110to a reference frame120to produce an error term114, one or more motion vectors116, and a variety of the compression parameters including quantization, macroblock type/mode, etc. The error term114, the motion vectors116, and a variety of the compression parameters including quantization, macroblock type/mode, etc. are the outputs of the encoder design103. From time-to-time, a decoded frame110will pass through the encoder design103without interframe compression, for example, to establish reference frames at the remote receiver's decoder. Such a frame will typically also be stored locally as a reference frame120in reference frame storage122.

The reference frame120represents a copy of the expected recently decoded frame at the remote receiver's decoder. The reference frame120is used in the encoder design103to determine an error term114, one or more motion vectors116, and a variety of the compression parameters including quantization, macroblock type/mode, etc. that will produce a frame similar to the decoded frame110in the remote receiver's decoder. Typically, the encoder design103will include a complete decoder105, which applies the motion vectors116to the reference frame120and adds the error term114and applies a variety of other compression parameters including quantization, macroblock type/mode, etc., to produce a new reference frame120. The new reference frame120is used for encoding by the encoder104, and is also used as a reference frame120for the decoder105to use for decoding of subsequent frames.

Because the decoder design101and the encoder design103are separate and independent, conventional transcoders are inefficient and costly. Therefore, what is needed is a streamlined transcoder architecture.

SUMMARY

The present invention includes systems and methods for a streamlined transcoder architecture. A unified decoder provides both decoded frames, which includes decoder reference frames, and encoder reference frames to an encoder. Because the same decoder that produces decoded frames also produces encoder reference frames, the power consumption, size, and cost of the transcoder is improved in comparison to architectures using separate decoders for producing decoded frames including decoder reference frames and encoder reference frames.

Advantageously, because the transcoder architecture is streamlined, data present in the decode step is also available in the encode step. In one embodiment, for example, frame information including compression parameters such as motion vectors, quantization, macroblock type/mode selection, etc. received by the transcoder for the purpose of decoding can be reused for the purpose of encoding.

DETAILED DESCRIPTION

Systems and methods for a streamlined transcoder architecture are described. In one embodiment, an encoder compares a decoded frame and a reference frame to produce an output stream. A decoder produces the decoded frame, which includes decoder reference frames, and the encoder reference frame. The decoded frame is produced from an input stream, and the encoder reference frame is produced from the output stream of the encoder. Because the decoder produces the decoded frame and the encoder reference frame, resource consumption of the transcoder architecture can be advantageously reduced.

As will be apparent to one of ordinary skill in the art, the systems and methods described may also be applied to image or video fields instead of frames and the fields or frames may be interlaced or deinterlaced. Thus, although various embodiments are described within in terms of video or image frames, the techniques may also be applied to video or image fields without departing from the scope of the invention.

FIG. 1is a block diagram of a conventional transcoder architecture. As described previously, the conventional transcoder architecture ofFIG. 1includes a decoder design101separate from the encoder design103.

FIG. 2is a block diagram illustrating a streamlined transcoder architecture, according to one embodiment of the present invention. For the purposes of illustration, the streamlined transcoder architecture is described in terms of a system. According to various embodiments of the present invention, a streamlined transcoder architecture can be implemented as hardware, software, an integrated circuit, and so on. Other methods for electronically implementing computational steps will be apparent to one of skill in the art without departing from the scope of the present invention.

The system is adapted to convert a compressed input stream201to a compressed output stream220. The compressed input201and output streams220can be encoded formatted under various audio/video protocols such as, MPEG-2, MPEG-4, MP3, H.263, H.264, AVS, a RealVideo format, a Windows Media Player format such as VC-1, other video formats, other audio formats, and the like. The formats can vary in characteristics such as bit rate and resolution. The transcoding may involve changes in picture timing, picture dimensions, image enhancement, and the like. As will be apparent to one of skill in the art, media formats discussed herein are intended to be exemplary, and other forms of compressed media and/or data may be used without departing from the scope of the present invention.

From time-to-time, the input stream201will include a frame without interframe compression. Such a frame is used as the first decoded frame210, and provides a boot-strap or refresh for subsequent or prior interframe compression. While the following discussion of transcoding is primarily directed towards the decoding of frames with interframe compression, it will be apparent to one of skill in the art that the input stream of201will from time-to-time include frames without interframe compression. Similarly, from time-to-time, the output stream220will include a frame without interframe compression. Such a frame is used as the first encoder reference frame218, and provides a boot-strap or refresh for subsequent interframe compression at a remote receiver. While the following discussion of transcoding is primarily directed towards the encoding of frames with interframe compression, it will be apparent to one of skill in the art that the output stream of220will from time-to-time include frames without interframe compression. Advantageously, because the decoder202and the encoder204are included in the streamlined transcoder architecture, information about frames without interframe compression in the input stream201can be usefully employed to produce frames without interframe compression in the output stream220.

As shown in the figure, a unified decoder202produces both decoded frames and encoder reference frames. In one embodiment, the decoder202can be usefully understood as operating in one of at least two modes. In a first mode, the decoder202functions to produce a decoded frame, which can be a decoder reference frame,210from the input stream201. In a second mode, the decoder202functions to produce a encoder reference frame218from previous output of the encoder204. While in one embodiment the decoder202transitions between its two modes based on time, other methods for multiplexing a unified decoder202will be apparent to one of skill in the art without departing from the scope of the present invention.

In one embodiment, if the encoder reference frame uses motion compensation, the motion compensated reference frame pixel data is passed to the decoder202instead of the motion vectors from the encode204. This saves the bandwidth of another motion compensation fetch.

In another embodiment, the output of the decoder is passed one macroblock at a time directly to the input of the encoder without storing the results to memory for non-reference frames to dramatically boost the transcoder performance.

In the first mode, the decoder202receives frame information. In one embodiment the frame information comprises one or more motion vectors206, and an error or residual term208from the input stream201. In another embodiment, the frame information further comprises compression parameters, such as, for example, a quantization parameter, a macroblock type parameter, a macroblock mode parameter, and a variable number of other parameters based on the compression format. The decoder202further uses one or more previous or future decoded frames210as a reference frame212. While not shown in the figure, according to one embodiment of the present invention, the decoder202receives reference frames212from a repository of previous decoded frames210. In addition, decoder202can perform intraprediction based on input stream201.

In the first mode, the decoder202uses the frame information and one or more reference frames212to produce a decoded frame210. The frame information may include, for example, one or more motion vectors206, the error term208, and a variety of the compression parameters including quantization, macroblock type/mode, etc. A method used by the decoder202is described herein with reference toFIG. 3. The decoded frame210is a frame without interframe compression. According to various embodiments, the decoded frame210can be described in either a spatial or a compressed domain.

In the second mode, the decoder receives frame information including, for example, one or more motion vectors206, an error term208from the output of the encoder204. In one embodiment, frame information from the encoder204may further comprise a variety of the compression parameters including quantization, macroblock type/mode, etc. The decoder202further uses a previous encoder reference frame218as a reference frame212. According to one embodiment of the present invention, the decoder202receives a reference frame212from a repository of previous and/or future encoder reference frames218that may be stored in a reference frame storage222.

In the second mode, the decoder202uses the motion vectors206, the error term208, and the reference frame212to produce a encoder reference frame218. In one embodiment, the decoder further uses a variety of the compression parameters such as quantization, macroblock type/mode, etc. to produce the encoder reference frame218. A method used by the decoder202is described herein with reference toFIG. 3. The encoder reference frame218is a frame with no interframe compression. According to various embodiments, the decoded frame210can be described in either a spatial or a compressed domain.

In one embodiment, the system further includes an image processor213. The image processor213performs further transformation on the decoded frame210to produce a decoded frame211. For example, the image processor213can be configured to change the size, characteristics, or sampling rate of the decoded frame210, or to perform image enhancement or other modifications on the decoded frame210. The image processor213can operate on the decoded frame210in a spatial domain, a compressed domain, or both. Other examples of transformation that can be performed by the image processor213will be apparent to one of skill in the art without departing from the scope of the present invention. In embodiments in which the image processor213is not included, the decoded frame211can be equivalent to the decoded frame210.

The image processor213(if present) uses the decoded frame210to produce the decoded frame211. The decoded frame211typically represents the desired output of a decoded frame at a remote receiver's decoder. (A remote receiver could be, for example, a recipient of the output stream220.) The decoder202also processes components of the output stream to produce the encoder reference frame218. The encoder reference frame218typically represents the expected output of a previously decoded frame at a remote receiver's decoder. In one embodiment, the encoder204uses at least the encoder reference frame218and the decoded frame211to produce the output stream220. The output stream220describes how a decoded reference frame at the remote receiver's decoder should be modified to produce a frame similar to the decoded frame211.

In one embodiment, the encoder204compares the encoder reference frame218to the decoded frame211to produce error or residual term214, macroblock type/mode, quantization factor, and one or more motion vectors216. A method used by the encoder204is described herein with reference toFIG. 4. The output frame information including an error term214and the motion vectors216are included in the output stream220. In one embodiment, the output frame information further comprises a variety of the compression parameters including quantization, macroblock type, and macroblock mode included in output stream220. The output frame information including error term214, motion vectors216, and compression parameters are also fed back to the inputs of the decoder202for production of future encoder reference frames218.

The system is configured so that the format and compression method of the input stream201can be different from the format and compression method of the output stream220. The input frame information including the error terms, motion vectors, and compression parameters (such as quantization, macroblock type/mode, etc.) of the input stream201may be described differently from the output frame information including the error terms and motion vectors, and compression parameters of the output stream220. Furthermore, the encoder reference frame218and the decoded frame210can be of different size, compression ratio, and so on. Because the decoder202receives error terms and motion vectors, and a variety of the compression parameters including quantization, macroblock type/mode, etc. of the input stream201to produce decoded frames210, as well as error terms and motion vectors, and a variety of the compression parameters including quantization, macroblock type/mode, etc. of the output stream220to produce encoder reference frames218, the decoder202is typically configured to operate on error terms, motion vectors, compression parameters used in a variety of formats. In the first mode, for example, the decoder202receives an error term, motion vectors, and a variety of the compression parameters including quantization, macroblock type/mode, etc. of a first format to produce a decoded frame210, and in the second mode, the decoder202receives an error term, motion vectors, and a variety of the compression parameters including quantization, macroblock type/mode, etc. of a second format to produce an encoder reference frame218. In one embodiment, the decoder202is configured to alternate between processing frames of a first format and processing frames of a second format. For example, for some first amount of time, the decoder202produces decoded frames of a first size, and for some second amount of time, the decoder202produces encoder reference frames of a second size.

Because the same decoder202is used to produce the decoded frame210and the encoder reference frame218, the total cost, size and power consumption of the streamlined transcoder architecture is improved compared to conventional transcoders. Advantageously, the software, hardware, and/or integrated circuitry comprising the decoder202can be reused, providing a more efficient transcoder architecture.

FIG. 3illustrates an interprediction method for decoding a frame of video, according to one embodiment of the present invention. In one embodiment, the method is performed by the decoder202. Note the decoder202can also perform an intraprediction method which is not shown.

The decoder202receives a reference frame212. The reference frame212can be decoded from the input stream201, or it can be decoded from the output of the encoder204. For example, the reference frame212can be a decoded frame210or a encoder reference frame218. Reference frame212is a frame of video without interframe compression.

The decoder202also receives motion vectors206. The motion vectors can be received from the input stream201or the output of the encoder204(for example, from the output stream220). The motion vectors206describe, generally in a spatial domain, how macroblocks from one frame are related to macroblocks of a previous or subsequent frame. As an optimization, the encoder204can, instead of sending the motion vectors206directly, send the motion compensated pixels from the encoder reference frame to save memory bandwidth and calculation.

In one embodiment, the decoder202applies306the motion vectors206and macroblock type/mode to the reference frame212for interframe predicted macroblocks. The motion vectors206can be applied306to the reference frame in a spatial or a compressed domain. The application306of the motion vectors206to the reference frame212produces a macroblock308.

The decoder202receives a transformed and quantized residual or error term208and dequantization term. The error term304describes how the macroblock308should be modified to improve the fidelity of the resulting frame and the dequantization term describes how the error term304is reconstructed from208. For example, the error term208may include information related to transients not encoded in the motion vector206. The error term208and dequantization term can be described in a spatial or compressed domain.

In one embodiment, the decoder202decompresses302the error term208to produce an error term304. For example, according to various standards, the error term can be encoded using various lossy and/or lossless compression techniques. In one embodiment, decompressing302the error term208can include transforming the error term208from a compressed to a spatial domain, for example, by applying a transformation derived from an Inverse Discrete Cosine Transform. In one embodiment, the decoder202dequantizes302the error term to produce the error term304. The decompression and/or dequantization302performed by the decoded202can depend on the format of the input/output stream processed by the decoder202.

As described herein with reference toFIG. 2, the same decoder202is used to produce the decoded frame210and the encoder reference frame218. In one embodiment, the method of the decoder can be understood as operating in one of at least two modes. In a first mode, frame information including motion vectors206and an error term208are received from an input stream201, and a reference frame212is received from a previous or future decoded frame210. In one embodiment the frame information further comprises a variety of the compression parameters such as quantization, macroblock type/mode, etc. In the first mode, the output of the decoder202is a decoded frame210. The inputs, output, and steps of the first mode are typically consistent with the format of the input stream201. For example, the error term208may be given in a specific range, the motion vectors may be described in a particular format, and/or the decoded frame210may be of a certain size. Furthermore, decompression/dequantization302, motion vector application306, and addition310steps may be performed according to a format associated with the input stream201.

In the second mode, the frame information including a motion vector206, and an error term208is received from the output of the encoder204, and a reference frame212is received from a previous or future encoder reference frame218. The frame information received from the encoder204may also include and a variety of the compression parameters such as quantization, macroblock type/mode, etc. In the second mode, the output of the decoder202is a encoder reference frame218. The inputs, output, and steps of the second mode are typically consistent with the format of the output stream220. For example, the error term208may be given in a specific range, the motion vectors may be described in a particular format, and/or the decoded frame210may be of a certain size. The range, format, and/or size of parameters such as the error term208, the motion vectors206, and the reference frame212can be different when the decoder202is operating in the first mode versus when the decoder202is operating in the second mode. Furthermore, decompression/dequantization302, motion vector application306, and addition310steps may be performed according to a format associated with the output stream220. Therefore, decoding can be performed differently when the decoder is in the first mode versus the second mode.

An efficient implementation of steps such as those illustrated inFIG. 2, such as decompression/dequantization302, motion vector application306, and addition310, can involve dedicated hardware, software, and/or integrated circuitry specialized for the performance of the various steps. For example, when the system is implemented in hardware, it is common for the decompression302step to make use of a dedicated device such as an Inverse discrete cosine Transform module. As the decoder202is used for dual purposes (i.e. the production of both decoded210and encoder reference218frames), a system according to an embodiment of the present invention advantageously provides efficient use of any such dedicated hardware, software and/or integrated circuitry.

FIG. 4illustrates a method for encoding a frame of video, according to one embodiment of the present invention. In one embodiment of the present invention, the method is performed by the encoder204.

The encoder receives a decoded frame211and an encoder reference frame218. In one embodiment, the decoded frame211is the output of an image processor213. In another embodiment, the decoded frame211is the output of the decoder202operating in a first mode. The encoder reference frame218is typically the output of the decoder202operating in a second mode.

In one embodiment, the encoder204generates402using motion vectors and macroblock type/mode and other parameters224passed from decoder202, one or more motion vectors216and the macroblock type/mode. The encoder204can generate402motion vectors216, for example, by comparing the decoded frame211to the encoder reference frame218. The encoder204attempts to generate402motion vectors216that describe the changes between the encoder reference frame218and the decoded frame211. In another embodiment, the encoder204refines402motion vectors received from the input stream201. Because the decoded frame211will often be similar to the decoded frame210, the motion vectors from the input stream201can increase the efficiency and effectiveness of the generation of motion vectors for the output stream220. Reusing the motion vectors from the input stream201beneficially reduces the computation involved in transcoding.

In one embodiment, the encoder204applies404the motion vectors216to the encoder reference frame218to produce a macroblock406. In another embodiment, the encoder generates/refines402and applies404the motion vector in a unified step to produce a macroblock406. By combining the generation and application steps of motion vectors, the efficiency of the encoder204can be advantageously improved.

The encoder204subtracts408the macroblock406from the decoded frame211to produce an error term410. The encoder204compresses/quantizes412the error term410to produce an error term214. In one embodiment, parameters224passed from the decoder are used in compressing/quantizing412the error term410. The motion vectors216and the error term214are components of the output stream220. Furthermore, as illustrated inFIG. 2, the motion vectors216and the error term214are fed back to the decoder202, advantageously facilitating reuse of the decoder implementation.

If the reference frame used motion compensation, one optimization is to pass the motion-compensated reference frame pixel data or Macroblock406instead of the motion vectors216from the encode204to the decoder202. This saves the bandwidth of another motion compensation fetch.

Another optimization is to pass the output of the encoder one macroblock at a time directly to the input of the decoder without storing the results to memory for non-reference frames to dramatically boost the transcoder performance.

For the purposes of illustration, both the input stream201and the output stream220are discussed as being of generalized forms common among a variety of compression formats. The methods described herein are useful for a variety of compression formats, some of which may differ from the generalized format described herein for the purposes of illustration. It will be apparent to one of skill in the art that the techniques may be applied to various compression formats without departing from the scope of the present invention.

Further, for the purposes of illustration, the methods and systems are described in terms of video or image frames. It will be apparent to one of skill in the art that the techniques may also be applied to video or image fields without departing from the scope of the present invention. Further, according to various embodiments, the video or image frames or fields may be interlaced or deinterlaced.

The order in which the steps of the methods of the present invention are performed is purely illustrative in nature. The steps can be performed in any order or in parallel, unless otherwise indicated by the present disclosure. The methods of the present invention may be performed in hardware, firmware, software, or any combination thereof operating on a single computer or multiple computers of any type. Software embodying the present invention may comprise computer instructions in any form (e.g., source code, object code, interpreted code, etc.) stored in any computer-readable storage medium (e.g., a ROM, a RAM, a magnetic media, a compact disc, a DVD, etc.). Such software may also be in the form of an electrical data signal embodied in a carrier wave propagating on a conductive medium or in the form of light pulses that propagate through an optical fiber.

While particular embodiments of the present invention have been shown and described, it will be apparent to those skilled in the art that changes and modifications may be made without departing from this invention in its broader aspect and, therefore, the appended claims are to encompass within their scope all such changes and modifications, as fall within the true spirit of this invention. For example, the systems and methods of the present invention can be used to establish a connection between a client computer and a server computer using any type of stateless protocol.

The present invention also relates to an apparatus for performing the operations herein. This apparatus can be specially constructed for the required purposes, or it can comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.

The algorithms and modules presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems can be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatuses to perform the method steps. The required structure for a variety of these systems will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the invention as described herein. Furthermore, as will be apparent to one of ordinary skill in the relevant art, the modules, features, attributes, methodologies, and other aspects of the invention can be implemented as software, hardware, firmware or any combination of the three. Of course, wherever a component of the present invention is implemented as software, the component can be implemented as a standalone program, as part of a larger program, as a plurality of separate programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/or in every and any other way known now or in the future to those of skill in the art of computer programming. Additionally, the present invention is in no way limited to implementation in any specific operating system or environment.

It will be understood by those skilled in the relevant art that the above-described implementations are merely exemplary, and many changes can be made without departing from the true spirit and scope of the present invention. Therefore, it is intended by the appended claims to cover all such changes and modifications that come within the true spirit and scope of this invention.