Patent Document

FIELD OF THE INVENTION 
       [0001]    This invention relates to video and/or audio streaming and more particularly to a system allowing adaptive rate shifting. 
       BACKGROUND OF THE INVENTION 
       [0002]    Many open research problems deal with in video and/or audio streaming and are related to compression, network design, network transport, error correction, error concealment, and caching. 
         [0003]    Current video and/or audio streaming systems suffer from occasional short-lived faults such as temporary loss of video and/or audio signals and/or artifacts arising due to network congestions and/or transmission errors. These problems generate video and/or audio streams which cannot be readable and/or viewable and/or listenable. Nevertheless, Streaming allows live access to video or/and audio resources having a lower quality as compared with the same content obtained from a classically downloaded file. 
         [0004]    The so-called “Progressive Download” technology allows to download a media file and to visualize and/or listen to it with a better quality than streaming. However “Progressive Download” is not as powerful and as flexible as streaming, because it suffers from limitations, such as it cannot be used for casting live events, it cannot be automatically adjusted to the available bandwidth of the end user&#39;s connection, and it is less secure because video and/or audio files are saved on the end-user&#39;s computer. Technically, when a Progressive Download process is initiated, media file download begins, and the media player waits to begin playing the complete download of said file. Waiting times before playing the downloaded file can be extremely variable (a few minutes to a few days) depending on networks conditions. Therefore, Progressive Download is not a fully acceptable solution to overcome the problem of media distribution and its presentation over a network. 
         [0005]    WO/2005/109224 describes Simulcoding techniques and Adaptative Streaming. Simulcoding is a protocol dividing large video files into many small files called “streamlets”, in WO/2005/109224. Each “streamlet” is a video segment of a predefined short time. Servers process each “streamlet” and apply the publisher-determined parameters (bit rate, frame size, frame rate, codec type, constant or variable bit rate, 1-pass or 2-pass encoding, etc.), bit rate by bit rate. There are many versions of each “streamlet”, each version with a different bit rate. Encoded “streamlets” are stored on standard HTTP Web servers, in contrast to what most streaming providers do, which store video files on media servers). 
         [0006]    The Simulcoding approach can be used in “Adaptive Streaming”. As an example, when an Internet user requests a video, by using a standard HTTP “GET” request, “streamlets” are transferred over a network from a server to a client browser or client application where they are reassembled in the correct initial order. The delivery protocol uses multiple TCP sessions to improve the reliability of the transmission and increase the total carrying capacity during each unit of time. 
         [0007]    WO/2005/109224 discloses a characteristic of the Adaptive Streaming, i.e., the ability to adapt to the available bandwidth of each client connection anytime during streaming. “Adaptive Streaming” can avoid buffering by adjusting image quality to fit with the available bandwidth of a client connection. This is achieved according to a set of “streamlets” for each bit rate specified in the profile of the publisher. Since the client protocol needs to upshift or downshift the bit rate, the correct time-indexed “streamlet” from the appropriate bit rate set is retrieved from the server. Therefore, the media player can easily interchange bit rates by retrieving the appropriate time-indexed streamlet from the desired bit rate pool. Thus, bit rate can change quickly and seamlessly as network conditions fluctuate and because each “streamlet” is a small segment of video, seeking and starting can happen quickly (within the time length of one individual streamlet). 
         [0008]    It is an object of the present invention to overcome the limitations of Simulcoding. 
         [0009]    It is another object of the present invention to overcome the limitations of Adaptive Streaming. 
         [0010]    It is a further object of the present invention to provide a method allowing to split the available bandwidth to sub-bandwidth. 
         [0011]    Further purposes and advantages of this invention will appear as the description proceeds. 
       SUMMARY OF THE INVENTION 
       [0012]    The method for carrying out video and/or audio adaptive-rate streaming according to the invention comprises providing two or more encoders, wherein each encoder is tuned to and responsible for a specific range of bandwidth, and a media bridge forwarding data packets from an encoder to one or more clients, wherein the encoder is selected according to statistics representing one or more communication quality parameter. According to one embodiment the statistics are a combination of blockiness level (as hereinafter defined) and packet loss level. 
         [0013]    According to another embodiment of the invention the statistics are a combination of a value of a parameter relating to visual quality, and a value relating to the level of quality of a channel. 
         [0014]    In an embodiment of the invention a media bridge switches users continuously to an encoder according to client statistics computed and send by client to the media bridge. Said media bridge controller module may decide from which encoder to send packets and to which client to forward said packets. The packet may comprise, for instance, a group of pictures. In another embodiment of the invention the media bridge controller is configured to generate an average performance factor for each encoder according to statistics received from users connected to said encoder. 
         [0015]    The invention is also directed to a system for the video and audio adaptive-rate streaming, comprising a plurality of encoders, each of which is tuned to and responsible for a specific range of bandwidth, each encoder being suitable to adapt the bit rate in its bandwidth range by averaging feedback of clients and to stream continuously to a media bridge as a Group of Pictures resolution. 
         [0016]    In one embodiment of the invention the lowest encoder group has the lowest quality and the highest encoder group has the highest quality. In another embodiment of the invention the media bridge module is configured to take Group of Pictures from each encoder and to forward said Group of Pictures to users. 
         [0017]    In yet another embodiment of the invention the media bridge controller is configured to connect any new user to a specific encoder depending on feedback statistics sent by the client to the media bridge controller using the Group of Pictures resolution. The media bridge can be configured to check that the statistics sent by the client match its encoder and are suitable to update it. 
         [0018]    The media bridge, among other things, may switch the client to another group corresponding to the new statistics received from client as Group of Pictures resolution. The statistics may be, without limitation, a combination of blockiness level and packet loss level. The media bridge controller module can be configured to up-shift to a higher quality of the encoder group when the statistics factors are into higher ranges and the media controller defines the higher quality sustained according to a combination of factors. Furthermore, the media bridge controller can be configured to change a client from an encoder group to another according to the statistics received from said client to downshift to a lower quality encoder group. 
         [0019]    In one embodiment of the invention the media bridge controller is configured to change a client from an encoder to another according to the statistics received from said client to up-shifting to a higher quality dynamically at Group Of Pictures resolution. 
         [0020]    In one embodiment of the invention circuitry is provided in the system to carry out one or more of the following:
       A) a video or an audio frame is split into new sub-frames replacing the initial one, using a wavelet 2D approach;   B) each sub-frame of a video or of an audio is encoded separately;   C) a new compressed raw data is created by joining each of the sub-frames encoded;   D) new compressed data is split into four compressed data;   E) each one of the four compressed data is decoded;   F) after decoding the video or the audio frame, the process is reversible; and   G) a filter provides the filter length correspond to a level of packet loss.       
 
         [0028]    All the above and other characteristics and advantages of the invention will be further understood through the following illustrative and non-limitative description of preferred embodiments thereof, with reference to the appended drawings; wherein similar components are designated by the same reference numerals. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0029]      FIG. 1  schematically shows a global view of the casting of the audio and/or video stream; 
           [0030]      FIG. 2  schematically shows a module including a video encoder and a streamer adaptive using real time adaptive reconfiguration; 
           [0031]      FIG. 3  schematically shows the internal process of the statistical decision block; 
           [0032]      FIG. 4  schematically shows the internal process of the module dealing with the size adaptation per group of pictures; 
           [0033]      FIG. 5  schematically shows the internal process of the module dealing with the statistical analysis and splitting to the corresponding group; and 
           [0034]      FIG. 6  schematically shows the internal process of the decoder player module. 
       
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0035]    According to an embodiment of the present invention, Simulcoding is performed by splitting the available bandwidth into sub-bandwidths; each encoder is responsible for a sub-bandwidth. As an example, an available bandwidth of 1 Mbit/s is divided into four bandwidths; the first encoder is responsible for the first sub-bandwidth {0 to 150 Kbit/s}, the second encoder is responsible for second sub-bandwidth {151 Kbit/s to 300 Kbit/s}, the third encoder is responsible for third sub-bandwidth {301 Kbit/s to 600 Kbit/s}, and the fourth encoder is responsible for the fourth sub-bandwidth {601 Kbit/s to 1000 Kbit/s}. 
         [0036]    According to the previous example, each encoder makes an adaptive bit rate into the selected sub-bandwidth. For each selected sub-bandwidth the system chooses the adequate video codec and/or audio codec. 
         [0037]    Adaptive Streaming is performed in two steps. In the first step the sub-band adapted to a client is selected and attached to said client. The second step is a continuous adaptive bit rate into the selected bandwidth. 
         [0038]    According to an embodiment of the present invention a media bridge is constituted by a “statistic analyze and split to the corresponding group” block  108  and a “Multi Mux” block  110  which is a set of multiplexers. The number of multiplexers included in said set of multiplexers is equal to the number of available groups of encoders ( 104 ,  106 ). 
         [0039]    Each multiplexer of a “Multi Mux” block  110  splits the video and/or the audio stream to all the clients attached to a same encoder group. 
         [0040]    Each encoder continuously streams the video and/or the audio stream to the media bridge as a reflection of the produced stream. Said streams are not stored on standard HTTP Web servers, but the video is continuously streamed to standard servers running a media bridge ( 108 ,  110 ). 
         [0041]    Each encoder can adaptively change the bit rate for a bandwidth range, because each encoder configuration is able to respond to a range of bandwidths. 
         [0042]    For each bandwidth range, for example, the frame rate of the encoder can be configured, and the motion estimation parameter can be set. Any other configuration is possible. 
         [0043]    Each encoder can be finely tuned to the best working case using the average feedback received by all clients attached to the same bandwidth range requirement. Accordingly, the encoder is set to a new bit rate in the specific bandwidth range. 
         [0044]    According to an embodiment of the present invention, Adaptive Streaming works as described hereinbelow. When an Internet user requests a video and/or an audio stream, a request is sent to the media bridge, which creates a link between the user and one of the encoders in the group, attaching the user to the specific multiplexer splitting the corresponding group, the decision on which to select being made from statistics received at “statistical analyze” block  108 . Media bridges can switch dynamically the link from an encoder to another, according to said statistics. The dynamical switch is done on the “Group Of Pictures” (GOP) resolution. It is possible to switch to one or multiple GOP frame. A second step of the process is the computation of the average of all the statistics attached to said group. Said average is sent to said group of encoders in order to update the encoder setting for the specific group. 
         [0045]      FIG. 1  describes a system according to another embodiment of the present invention, which allows a new way to optimize unicast and/or multicast transmissions over all types of digital channels. Said system is used as a statistical multiplexing and as broadcast transmission. It allows transmitting video and/or audio signals adapted to the transmission channel capacities. 
         [0046]    According to still another embodiment of the present invention, the system is based on two main new approaches, namely “No More Buffering” (NMB) and “Dynamic Client/Server Reconfiguration” (DCSR). The NMB approach of the present invention prevents the need for buffering on the client side. A buffer use is hereby optional. The DCSR approach of the present invention allows adapting the streaming flow to the user bandwidth capacities in real time. 
         [0047]      FIG. 1  schematically shows a global view of the system according to an embodiment of the present invention. Said system receives a video and/or an audio stream ( 100 ) from one source which can be compressed or uncompressed. If the input signal is compressed, the block  100  decompresses said input video and/or audio signal. 
         [0048]    The output result of block  100  is sent to multiplexer  102 . Multiplexer  102  sends the uncompressed signal simultaneously to encoders  104  and  106 . 
         [0049]    Each encoder is responsible for a number of clients having the same requirements. Each encoder deals with clients having the same properties, and needing the same bit rates range. 
         [0050]    According to yet another embodiment of the present invention, using block  110 , the Statistic Decision block  108  receives statistics from each user  112 ,  114 , and  116  and decides with which encoder  104  or  106  said clients  112 ,  114 , and  116  are associated. 
         [0051]    Statistic Decision block  108  decides dynamically to switch the connection of a client managed by a first encoder to another encoder, which is more adequate to the requirements of said client. 
         [0052]    According to still a further embodiment of the present invention, the video and/or audio stream packets sent to a client are destreamed and reordered, according to their initial arrangement (before network transmission), using a destreamer ( 112 ,  116 ) which is a device working like a streamer but inversely. 
         [0053]    According to another embodiment of the present invention, when the video and/or the audio destreamed flow arrive to client, a client decoder ( 118 , 120 ) plays said video and/or said audio frame. 
         [0054]    According to one embodiment of the present invention, a multiple frame is group into a number of frame, called Groups of Pictures (GOP). Said GOP are further subdivided in sequences of a pre-defined number of frames. 
         [0055]    Typically, a Group of Pictures (GOP) comprises an “I frame” (which is an intra-coding frame) and a few number of “P frames” (which is a motion-based predictive coding frame) and potentially “B frames” (which is a motion-based bidirectional predictive coding frame). As an example, a GOP may comprise a set of frame defines such as “I, B, B, P, B, B, P, B, B, P, B, B, P, B, B” and sent with a frequency of 30 frames per second. The “I frame” is independently compressed. The number of said “I frame” generated packets is higher than the “P frames” or “B frames” which only encode changes from the previous frame. 
         [0056]    Common parameters defining a GOP are the GOP length (the distance in frames from one I-frame to the next one) and the GOP structure (the arrangement of frames in said GOP). 
         [0057]    According to still an embodiment of the present invention, Blockiness level is a perceptual measure of the block structure that is common to all discrete cosine transformation (DCT) based image compression techniques. The DCT is typically performed on N×M blocks. Blocks in the frame, and the coefficients in each block are quantized separately, leading to artificial horizontal and vertical borders between these blocks. Blockiness can also be generated by transmission errors, which often affect entire blocks in the video. 
         [0058]    According to still another embodiment of the present invention the decoder ( 118 ,  120 ) send to the statistical analysis block  108  the “blockiness level” (as defined below) in the decode frame and the “degradation level” existing in the “Group Of Pictures”. The “degradation level” is the ratio between the number of packets emitted and the number of packets received; it corresponds to the loss of quality of the transmitted frame. This information allows the Statistic Decision block  108  to update the “packet size” and “RTP filter length” (which is the length of said packet) in order to optimize the encoder allocation to a client and more particularly the statistical multiplexing approach and the adaptive bandwidth decision. As a first advantage, said optimization overcomes the limitation of the transmission channel capacity for all the clients and the congestion problem for each client. As a second advantage, said optimization allows to use low bit rates and provides a better streaming quality. 
         [0059]    “Blockiness” of the decoded frame is a scale from 1 to 5. Value 1 corresponds to a good quality of the video and/or the audio signals; value 3 defines a video and/or an audio quality with high blockiness but without deformation into the frame; value 5 defines that a part of the video and/or the audio frame is lost. 
         [0060]      FIG. 2  shows in details an encoder block ( 104 , 106 ) according to yet another embodiment of the present invention. In said encoder, the input frame  200  is filtered in the horizontal direction by low pass filter  202  and high pass filter  204 . The output block  202  is low pass filter  206  and high pass filter  208 ; the output block  204  is low pass filter  201  and high pass filter  212 . The outputs of frames  206 ,  208 ,  210 , and  212  are then sent to the encoders  216 ,  220 ,  224 , and  228  respectively. Each encoder has its own rate control (respectively  214 ,  218 ,  222 , and  226 ). As an example, the output of encoder Q 1   216  generates a signal as shown on  300 . In the same way, the encoder Q 2   220  generates a signal as shown on  302 , the encoder Q 3   224  generates a signal as shown on  304 , and the encoder Q 4   228  generates a signal as shown on  306 . Splitting into four-streams allows to perform the reduction of the size of each packet. The blocks  308  and  312  assemble the four streams  300 ,  302 ,  304 , and  306  to create a new one. According to information received from the Statistic Decision of block  108  by the Streamer Statistical Decision block  310 , the packet size unit is increased or decreased by changing the GOP resolution in  314 . 
         [0061]    The “Packet into lower size” block  312  is explained using  FIG. 4 . According to yet a further embodiment of the present invention, Codec  400 , the compressed media frame  402  and Fragment  404  summarize the previous steps of the process. A first packet is forwarded successively to  414 , and to  416  by the way of  406 ; a second packet is forwarded too to  414 , and to  416  by the way of  408 ; a next one is forwarded too to  414 , and to  416  by the way of  410 . Packets forwarded by the way of blocks  406 ,  408 , and  410  are summed in order to generate an average of three consecutive packets. From the three initial packets the system generates four output packets which are respectively sent on a network  422  using the input queries  418  and  420 . 
         [0062]    From the “Div by 3” block  412  it is possible to modify the length of the filter. According to the quality of the transmission channel  110  defined by the “statistic decision” block  108 , it is possible to adapt the size of the packets generated by the encoders. “Statistical decision” block  108  works using the level of blockiness defined between level  1  to level  5 . When said blockiness is high, the size of the packets is reduced and vice versa. 
         [0063]    Streaming (RTP) packets are received ( 108 ) and treated by the channel coder block  428  in order to evaluate the payload buffer  430 . The payload buffer is used to repair the media  432  before decoding it ( 434 ). 
         [0064]    According to an embodiment of the present invention, the system uses two parameters: the “Blockiness” and the “packets loss per frame and per filter with length N”, also previously called “degradation level”. These parameters are both used on the system  500 . 
         [0065]    The “packets loss per frame and per filter with length N”, previously called degradation level, is a ratio between the number of packets emitted and the number of packets received. If the “degradation level” is low it is necessary to decrease the size of the filter; if the “degradation level” is high it is necessary to increase the filter length. These parameters allow to define the level of Blockiness  502  and the level of packets lost  504 . In view of these values, block  506  performs the decision of the new packet size and block  508  performs the decision about the filter length. Block  510  takes these two decisions and decides to which group to assign a client. Block  110  takes the stream from block  104  (or  106 ). In the other side block  110  forwards to the statistical decision block  108  all the information received from a client. 
         [0066]      FIG. 6  schematically shows how the decoders  118 ,  120 , and  122  work according to an embodiment of the present invention. After taking out redundant data and reordering the compressed stream packets ( 300 ,  302 ,  304  and  306 ) those are sent respectively to decoder Q 1   600 , decoder Q 2   602 , decoder Q 3   604 , and decoder Q 4   606 . Each decoder sends received data respectively to low pass filter  608 , to high pass filter  610 , to low pass filter  612 , and to high pass filter  614 . The results of low pass filter  608  and high pass filter  610  go to low pass filter  616  and the results of low pass filter  612  and high pass filter  614  go to low pass filter  618 . The summation of low pass filter  616  and high pass filter  618  generates the decoded frame  620 . 
         [0067]    According to a further embodiment of the present invention uses a standard RTP protocol is used in order to avoid the need for a proprietary Streamer and De-streamer. 
         [0068]    Although embodiments of the invention have been described by way of illustration, it will be understood that the invention may be carried out with many variations, modifications, and adaptations, without exceeding the scope of the claims.

Technology Category: 5