Source: http://www.google.com/patents/US7327791?dq=5,950,200
Timestamp: 2015-06-03 15:52:26
Document Index: 217722802

Matched Legal Cases: ['art; 2', 'art; 3', 'art; 4', 'art; 5', 'art; 6', 'art; 7', 'art; 8', 'art; 9', 'art 20', 'art 21', 'arts 22', 'art 40', 'art 40', 'art 44', 'art 48', 'art 1', 'art 6', 'art 105', 'art 106', 'art 105', 'art 105', 'art 105', 'art 105', 'art 104', 'art 105', 'art 106', 'art 106', 'art 105', 'art 111', 'art 111', 'art 113', 'art 114', 'art 113', 'art 111', 'art 113', 'art 114', 'art 114', 'art 113', 'art 111', 'art 111', 'art 111', 'art 113', 'art 114', 'art 113', 'art 111', 'art 105', 'art 104', 'art 106', 'art 105', 'art 106', 'art 111', 'art 111', 'art 111', 'art 114', 'art 113', 'art 114', 'art 113', 'art 111', 'art 105', 'art 104', 'art 106', 'art 105', 'art 106', 'art 106', 'art 120', 'art 120', 'art 114', 'art 120', 'art 114', 'art 120', 'art 114', 'art 120', 'art 111', 'art 114', 'art 111', 'art 120']

Patent US7327791 - Video decoding method performing selective error concealment and ... - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inAdvanced Patent SearchPatentsA moving image coded string is mapped with a set of information areas as one packet, and each packet is added with an error code and control information in a multiplexing part so that it is decided at the receiving side, from the result of error detection by decoding the error detection code, whether...http://www.google.com/patents/US7327791?utm_source=gb-gplus-sharePatent US7327791 - Video decoding method performing selective error concealment and resynchronizationAdvanced Patent SearchPublication numberUS7327791 B1Publication typeGrantApplication numberUS 09/692,720Publication dateFeb 5, 2008Filing dateOct 20, 2000Priority dateFeb 22, 1999Fee statusLapsedAlso published asUS7630442, US20060013321Publication number09692720, 692720, US 7327791 B1, US 7327791B1, US-B1-7327791, US7327791 B1, US7327791B1InventorsShunichi Sekiguchi, Fuminobu Ogawa, Kohtaro Asai, Yoshimi Isu, Shinichi Kuroda, Yuri HasegawaOriginal AssigneeMitsubishi Denki Kabushiki KaishaExport CitationBiBTeX, EndNote, RefManPatent Citations (15), Referenced by (14), Classifications (14), Legal Events (4) External Links: USPTO, USPTO Assignment, EspacenetVideo decoding method performing selective error concealment and resynchronization
US 7327791 B1Abstract
1. A video decoding method which receives a coded video stream, together with an error detection result indicating whether an error is contained in a coded stream in each packet, and decodes said coded video stream, wherein:
said coded video stream is composed of plural pieces of compressed block coded data, said plural pieces of compressed block coded data are composed of plural kinds of data elements, said data elements of the same kind are arranged in succession over plural blocks, and said coded video stream is divided, at the point of change in the kind of said data elements arranged in succession, into said each packet, said each packet being added, for each of said divided video coded streams, with an error detecting code for obtaining said error detection result;
and upon detecting a decoding error at the time of receiving and decoding said coded video stream for said each packet, the position of said decoding error in said coded video stream is decided based on an error detection result received and error concealment is selectively performed based on said decided position of said decoding error, wherein
said plural kinds of data elements contain a data stream composed of motion vectors contained in plural blocks and a data stream composed of pieces of texture information contained in plural blocks such that motion vector data is provided in separate packets than texture information; and, based on said error detection result received together with each data stream and the position of said decoding error detected in the decoding of said each data stream, it is decided whether to perform error concealment using decoded motion vectors or abandon said motion vectors and said texture information data and perform error concealment.
2. The video decoding method of claim 1, wherein said texture information is coded macro block DCT coefficient data.
3. The video decoding method of claim 1, wherein, based on the error detection result received for a packet containing motion vector data, said method abandons corresponding texture information data and performs error concealment.
4. The video decoding method of claim 3, wherein said plural kinds of data elements further include a resynchronization marker, which is detected during decoding to indicate the beginning of the next block coded data.
5. The video decoding method of claim 1, wherein said plural kinds of data elements further include coded video packet header data.
6. The video decoding method of claim 5, wherein, based on the error detection result received for a packet containing video packet header data, said method abandons corresponding texture information data and performs error concealment.
7. The video decoding method of claim 5, wherein, said method performs error concealment for a packet containing coded texture information data using motion vector data when a decoding error did not occur for the motion vector data.
8. A video decoding method which receives a coded video stream, together with an error detection result indicating whether an error is contained in a coded stream in each packet, and decodes said coded video stream, wherein: said coded video stream is composed of plural pieces of compressed block coded data, and for each of said compressed block coded data of plural blocks, header information is coded which contains a unique code indicating the head of said each block coded data and its block number, and said coded video stream is divided into packets at the point of change between said header information and said block coded data, said packets being added, for each of said plural video segments, with an error detecting code for obtaining said error detection result; and
upon detecting a decoding error during decoding of said coded video stream received for each packet, the position of resynchronization is decided based on said unique code and said error detection result received together with coded data of said header information and resynchronization is performed from the bit position of error detection to a unique code indicating the beginning of the next block coded data.
To facilitate a better understanding of the present invention, a description will be given first of the case where a coded video stream encoded by a conventional video coding system, for example, the ITU-T Recommendation H.263 (hereinafter referred to as H.263), is sent after being multiplexed by the ITU-T Recommendation H.223 (hereinafter referred to as H.223). H.223 defines a multiplexing system by which compress-coded audio, video and data streams are multiplexed into one bit stream for transmission.
FIG. 1 depicts the configuration of an H.233-recommended multiplexing part. As shown, the multiplexing part comprises two hierarchies of an adaptation layer and a multiplexing layer. The adaptation layer input thereto, as packets called AL-SDU (Adaptation Layer Service Data Unit), coded streams from an application layer that encodes speech and video, and adds each packet (AL-SDU) with an error correcting code (CRC) and other necessary control information (AL-PDU (Adaptation Layer Protocol Data Unit). The multiplexing layer inputs thereto, as packets called MUX-SDU (Multiplex Service Data Unit), the packets (AL-PDU) of various media from the adaptation layer, and multiplexes them into one bit stream for transmission. The bit stream multiplexed in the multiplexing layer is called MUX-PDU (Multiplex Protocol Data Unit), and comprises a sync flag, a header and information fields for storing the packets (MUX-SDU). Incidentally, the method of dividing the coded streams from the application layer into packets (AL-SDU) for processing in the multiplexing part is outside of the scope defined by H.223.
The above-mentioned multiplexing part is a function at the transmitting side. The receiving side has a demultiplexing part whose function is the inverse of the multiplexing at the transmitting side. In the demultiplexing part the bit stream (MUX-PDU) received in a demultiplexing layer is demultiplexed into packets of various media (MUX-SDU), which are output to the adaptation layer (AL-PDU). In the adaptation layer the error detecting code (CRC) contained in the packet (AL-PDU) is decoded and a check is made for an error in the received packet (AL-PDU). The result of error detection is output to the application layer together with the coded media information stream (AL-SDU) in the packet. It is outside the scope defined by H.223 how to use the result of error detection in the application layer.
By the way, the video coding system by H.263 encodes the input image into blocks of a predetermined size (each of which is comprised of a luminance signal of a 16 by 16 pixel size and a color difference signal of an 8 by 8 pixel size and is called a macroblock); it is considered that the coded video stream by H.263 may be rendered into the packet AL-SDU, for example, by a method of combining piece of coded data of plural macroblocks into one packet.
As referred to above, however, it is recommended by H.223 that the error detecting code be added for each packet AL-SDU and that the application layer at the receiving side receive the result of error detection together with the packet AL-SDU. Hence, when pieces of coded data of plural macroblocks are combined into one packet AL-SDU, it can be known from the result of error detection that there is an error in any one of the plural macroblocks contained in the packet AL-SDU, but it is impossible to specify the macroblock in which the error has occurred and the information in which the error has occurred. Accordingly, the prior art fails to practice error concealment by making effective use of the result of error detection received in the application layer.
With the recent broadening the band of a digital transmission line and development of multimedia communication technology, it has become feasible to transmit moving image signals over various transmission lines. In the field of communication there is in widespread use a teleconference/videophone system that uses the H.261 moving picture coding system on the ITU-TH.320 terminal intended for connection to ISDN. Further, at the H.324 terminal assumed to be connected to an analog public network, there are supported not only the H.261 moving image coding system but also the H.263 coding system which is higher in coding efficiency than the former coding system.
Moreover, there is recommended, as a terminal assumed to be connected to an IP network such the Internet, H.323 that supports H.261/H263 as is the case with H.324. Accordingly, since at such wired system terminals there is supported a set of limited moving image coding systems based on ITU Recommendations, the interconnection of different protocols between terminals for limited use is guaranteed to some extent.
FIG. 1 is a diagram showing the configuration of an H.223-recommended multiplexing part;
FIG. 2 is a block diagram illustrating a transmitting apparatus of a multimedia communication system according to a first embodiment (Embodiment 1) of the present invention in which a coded video stream, encoded by the MPEG-4 coding system (ISO/IEC 14496-2), is multiplexed by H.223 together with coded speech and data streams at the transmitting side and transmitted therefrom and at the receiving side the multiplexed stream is demultiplexed into the original speech, video and data. Reference numeral 1 denotes an MPEG-4 video coding part; 2 denotes a speech coding part; 3 denotes a data coding part; 4 denotes a packetizing part; 5 denotes a multiplexing part; 6 denotes a coded video stream packetizing part; 7 denotes a coded speech stream packetizing part; 8 denotes a coded data stream packetizing part; 9 denotes an input image signal; 10 denotes an input speech signal; 11 denotes an input data signal; 12 denotes a coded video stream; 13 denotes a coded speech stream; 14 denotes a coded data stream; 15 denotes a packet of the coded video stream; 16 denotes a packet of the coded speech stream; 17 denotes a packet of the coded data stream; and 18 a multiplexed stream.
The video packet is a set of plural macro blocks in the image space. On the coded video stream, as shown in FIG. 3( a), the video packet is composed of coded data in units of macro blocks (macro block data), a unique word for resynchronization (a resynchronization marker) and header information necessary for resynchronization (a VP header). In the first video packet of each frame, a unique code (a VOP start code) indicating the front end of the frame and header information (VOP header) necessary for decoding coded data in each frame are coded in place of the resynchronization marker and the VP header, respectively, as depicted in FIG. 3( a).
What is intended to mean by “data partitioning” is to provide such a structure as shown in FIG. 3( b) in which when the macro block data in the video packet contains information about n macro blocks, motion vectors and pieces of header information on the n macro blocks and pieces of texture information on the n macro blocks are arranged at different locations. The boundary between the motion vector and the texture information is defined by a unique word referred to as a motion marker.
As depicted in FIG. 4, the input coded video stream 15 is divided into the video packets shown in FIG. 3( a), and a set of motion information and macro block header information, a set of motion marker and texture information and a set of resynchronization marker and VP header that are contained in the respective video packets are each mapped as one packet (AL-SDU). Incidentally, since the front-end video packet has added thereto the VOP start code and the VOP header in place of the resynchronization marker and the VP header as referred to above, the VOP start code and the VOP header are mapped as one packet (AL-SDU) as a substitute for the set of resynchronization marker and VP header.
The length of each packet (AL-SDU) needs to be an integral multiple of 8 bits, but the information (the motion information and the macro block header information, the motion marker and the texture information, and the resynchronization marker and the VP header) to be mapped as one packet is not always an integral multiple of 8 bits. Accordingly, there are cases where each of the sets of motion information and macro block header information, motion marker and texture information, and resynchronization marker and VP header are not accurately mapped as one packet but the motion information is partly contained in the packet in which the resynchronization marker and the VP header.
A demultiplexing part 20 demultiplexes the multiplexed stream 25 into speech-video- and data media packets (MUX-SDU) through utilization of a synchronization flag and header information contained in the stream 25. The media packets 26 to 28 are fed into an error detecting/packet separating part 21, in which error detecting codes contained in the media packets 26 to 28 are decoded, and the packets (AL-SDU) 29, 31 and 33 from which the error detecting codes and the control information have been removed, and the decoded error detection results 30, 32 and 34 are provided to information source decoding parts 22 to 24 respectively corresponding to them.
The variable-length decoding part 40 is equipped with mechanism for detecting an error in the decoding of the coded video stream 29; when the decoded value is incorrect, or when the continuation of decoding is impossible due to loss of synchronization, it is detected as a decoding error. Upon detecting such an error, the variable-length decoding part 40 provides the error detection flag 56 to the compensated image generating part 44. In the case of decoding the packet of the coded video stream encoded by the data partitioning function as depicted in FIG. 3( c), when an error is detected not in the motion information area but in the texture information area (see FIG. 4), the error detection flag 56 is provided to the switching part 48.
In this embodiment no particular limitations are imposed on the structure of the macro block data of the coded video stream that is output from the MPEG-4 video coding part 1. That is, the coded video stream shown in FIG. 3( a) may be the macro block data encoded using the data partitioning scheme defined by MPEG-4 as depicted in FIG. 3( b), or macro block data of a structure in which the macro block header, the motion information and the texture information are arranged for each of n macro blocks as depicted in FIG. 9. This embodiment does not impose any particular limitations on the macro block data structure as mentioned above, but the following description will be given on the assumption that the macro block data has the structure shown in FIG. 9.
The coded video stream packetizing part 6 inputs therein the coded video stream of the structure depicted in FIG. 3( a) and divides it into packets (AL-SDU) for processing in the multiplexing part.
When the macro block data is encoded using the data partitioning scheme as depicted in FIG. 3( b), it may be mapped as described previously with reference to FIG. 4. The macro block data may also be mapped into one packet, but in this embodiment the macro block data is mapped into two packets as shown in FIG. 10. Since the length of each packet (AL-SDU) is a multiple of 8 bits as is the case with Embodiment 1, information about the macro block data may sometimes be contained, for example, in the packet that contains the VOP start code and the VOP header. Though not shown in FIG. 10, the packet containing the resynchronization marker and the VP header is followed by a macro block header for the resynchronization maker and the VP header.
Now, consider the case where an error is contained in the VOP start code. In order to detect the error earlier than the VOP start code and establish resynchronization, a search is made for the next VOP start code. Since the VOP start code to be detected is lost by the error as depicted in FIG. 12, no VOP start code is detected and the next resynchronization marker is detected as the position of resynchronization; however, the VOP header and the macro block data between them is not decoded.
Next, consider the case where an error is detected in the macro block address contained in the VP header. This situation arises when there is no continuity between the last macro block address of the immediately previously decoded video packet and the macro block address contained in its VP header (the macro block address at the head of the video packet). It is considered that such discontinuity occurs in the two cases mentioned below.
(1) When the macro block data decoded is wrong in number because of the occurrence of an error during decoding of the macro block data in the immediately preceding video packet (FIG. 13( a)).
(2) When an error occurs in the macro block address of the VP header (FIG. 13( b)).
FIG. 17 is a diagram depicting the configuration of a system to which a bit stream converter according to a fifth embodiment (Embodiment 5) of the present invention. A description will be given of a system that carries out moving image transmissions between a terminal A connected to an ISDN circuit A1 and a multimedia terminal B connected to a radio channel B1. Reference numeral 101 denotes a system converter for converting the video coding scheme mutually between the terminal A and the multimedia terminal B in the case where the former is the transmitting side and the latter the receiving side. Reference numeral 102 denotes a bit stream (a second bit stream) coded by H.263 as a multimedia multiplexed stream packetized by an ISDN media multiplexing system such as ITU-TH.221. Reference numeral 103 denotes a bit stream (a first bit stream) coded by MPEG-4 as a multimedia multiplexed stream packetized by a media multiplexing system for radio channel use such as ITU-TH.223. As depicted in FIG. 18, the coded bit stream 103 is composed of an up-link MPEG-4 coded bit stream 103 and a down-link MPEG-4 coded bit stream 103 b. Incidentally, there are some pieces of other media information such as those of audio information accompanying moving images, respectively, these pieces of multimedia information are sent as one multiplexed stream to the channel. The following description will be given, for brevity sake, on the assumption that only pieces of coded video data are multiplexed by a media multiplexing system predetermined for each channel.
Reference numeral 106 denotes a syntax converting part, which, when supplied with the error resistance syntax selection result 109 from the error resistance syntax determining part 105, converts the syntax of the down-link H.263 coded bit stream 103 b to the syntax of the up-link MPEG-4 coded bit stream 103 a. The conversion of syntax in the syntax converting part 106 corresponds to processing by which: a bit stream encoded by the H.263 coding scheme is once analyzed in a coded data area; the bits are converted into a bit string based on the definition of the MPEG-4 syntax; and, based on the selected error resistance syntax, the bit stream is used to reconstruct the syntax of the MPEG-4 video coded bit stream. This syntax conversion involves various processes, but they falls outside the scope of the present invention.
Next, the error resistance syntax determining part 105 inserts video packets as the minimum error resistance syntax required to cope with the condition of the radio channel B1 (step ST103). When the radio channel B1 is in good condition, there are cases where the error resistance syntax determining part 105 does not insert any video packets or reduces the number of video packets to be inserted. The coded moving image bit stream 102 (H.236 coded bit stream) can be added with one resynchronization marker for each unit GOB (Group OF Block). Since GOB is always fixedly positioned on the video plane, it is impossible to insert the resynchronization marker in accordance with the property or error characteristic of the image. The video packet insertion may also be done by converting the resynchronization marker of GOB to the resynchronization marker of the video packet without changing its position.
Next, at the instant of the syntax change timing based on the error resistance syntax changing period 107, the error resistance syntax determining part 105 makes a check to determine whether the channel error rate is lower than a predetermined threshold value EL1 (step ST104). In the case of “YES,” that is, if the error rate is lower than the threshold value EL1, the H.263 MPEG-4 coded bit stream 102 is converted to the MPEG-4 coded bit stream 103 without using any error resistance syntaxes (step ST108). In the case of “NO,” that is, when the channel error rate is higher than the predetermined threshold value EL1, the data partitioning syntax is used (step ST105) and the H.263 coded bit stream 102 is converted to the MPEG-4 coded bit stream 103.
Next, at the instant the syntax change timing based on the error resistance syntax changing period 107, a check is made to see if the channel error rate is lower than a predetermined threshold value EL2 (which is smaller than EL1) (step ST106). In the case of “YES,” that is, when the error rate is lower than the threshold value EL2, the H.263 coded bit stream 102 is converted to the MPEG-4 coded bit stream 103 without using other error resistance syntaxes (step ST108). On the other hand, in the case of “NO,” that is, when the channel error rate is higher than the threshold value EL2, the error resistance needs to be increased; then, the reversible VLC is used (step ST107) and H.263 coded bit steam 102 is converted to the MPEG-4 coded bit stream 103 (step ST108).
FIG. 23 is a flowchart showing another operation procedure of the bit stream converter according to Embodiment 5. Steps ST101 to ST104 are the same as those in FIG. 21, and hence no description will be given of them. In the case of “NO” in step ST104, that is, when the channel error rate is higher than the predetermined threshold value EL1, the syntax of the HEC field is adopted (step ST109) and the H.263 coded bit stream 102 is converted to the PMEG-4 coded bit stream 103.
Next, at the instant the syntax change timing based on the error resistance syntax changing period 107, a check is made to determine whether the channel error rate is lower than the predetermined threshold value EL2 (step ST110). In the case of “YES,” that is, when the channel error rate is lower than the threshold value EL2, the H.263 coded bit stream 102 is converted to the MPEG-4 coded bit stream 103 without using any other error resistance syntaxes (step ST108). On the other hand, in the case of “NO” in step ST 110, that is the channel error rate is higher than the threshold value EL2, the error resistance needs to be increased; then, the data partitioning syntax is adopted (step ST111) and the H.263 coded bit stream 102 is converted to the MPEG-4 coded bit stream 103.
Following this, at the instant of the syntax change timing based on the error resistance syntax changing period 107, a check is made again to determine whether the channel error rate is lower than a predetermined threshold value EL3 (which is larger than EL2) (step ST112). In the case of “YES,” that is, when the channel error rate is lower than the threshold value EL3, the H.263 coded bit stream 102 is converted to the MPEG-4 coded bit stream 103 without using any other error resistance syntaxes (step ST108). On the other hand, in the case of “NO” in step ST112, that is, when the channel error rate is higher than the threshold value EL3, the error resistance needs to be increased; then, the reversible VLC is adopted (step ST113) and the H.263 coded bit stream 102 is converted to the MPEG-4 coded bit stream 103 (step ST108).
FIG. 24 is a flowchart showing another operation procedure of the bit stream converter according to Embodiment 5. The steps ST101, 102 and 108 are the same as those in FIG. 22, and hence no description will be given of them. At the instant of the syntax change timing based on the error resistance syntax changing period 107, the error resistance syntax determining part 105 makes a check to see if the channel error rate is higher than a predetermined threshold value EL (step ST120; in the case of “YES,” that is, when the error rate is higher than the threshold value, all the error resistance syntaxes (step ST121) are selected and the H.263 coded bit string 102 is converted to the MPEG-4 coded bit stream 103 (step ST108). On the other hand, in the case of “NO” in step ST120, that is, when the channel error rate is lower than the predetermined threshold value EL, the H.263 coded bit stream 102 is converted to the MPEG-4 coded bit stream 103 (step ST108) without using any error resistance syntaxes.
As described above, this embodiment is provided with the channel quality monitoring part 104 which implements channel quality monitoring means and the error resistance syntax determining part 105 which implements syntax rule decision means; hence it is possible to provide an efficient bit stream according to the channel condition while achieving a balance between the degree of error resistance of the moving image coded bit stream to be fed over a channel of high error rate and the transmission efficiency as a whole. This embodiment is particularly effective in implementing moving picture communications between the communication terminal connected to the radio channel which supports the MPEG-4 video and the communication terminal connected to the ISDN or existing public network which supports the ITU-TH.263 video.
Incidentally, the system converter 101 according to this embodiment performs conversion for the moving image coded bit stream, and hence it produces the same effects as described above even when the terminal A is an H.324 terminal assumed to be connected to an ordinary analog public network or ISDN circuit based on H.263, or an H.323 terminal assumed to be connected to the Internet.
FIG. 26 is a block diagram illustrating a bit stream converter according to a seventh embodiment (Embodiment 7) of the present invention. This embodiment will be described in connection with a system converter which is contained in a media server which stores multimedia contents and delivers the contents in response to a request. Reference numeral 125 denotes contents stored as a video coded bit stream in a storage 126. Assume that the contents include bit streams encoded offline by an extremely high-quality processing system, or bit streams created on the premise that they are transmitted over a high-quality channel virtually insusceptible to errors. That is, assume that the contents 125 contains no particular syntax for error resistance. Reference numeral 127 denotes a media server provided with a system converter 124 which converts the contents 125 to those 128 for transmission over the radio channel B1.
FIG. 27 illustrates in block form the internal configuration of the system converter 124 that constitutes the bit stream converter 127 according to this embodiment. The parts corresponding to those in Embodiment 5 are identified by the same reference numerals and no description will be repeated. In the case where the video encoding schemes for the contents 125 and 128 are H.263 and MPEG-4, respectively, the same functional block can be used for the syntax converting part 106 and the error resistance syntax determining part 106. The error resistance syntax determining part 105 selects the error resistance syntax of the video coded bit stream of the contents 128 in accordance with the error rate 123 provided from the outside. The system converter may also be configured to select the error resistance syntax as described previously with reference to FIGS. 22 and 23. Further, it may also be configured to adopt the alternative of selecting or not selecting the error resistance syntax.
FIG. 28 illustrates in block form the internal configuration of a system converter according to an eighth embodiment (Embodiment 8). The parts corresponding to those in Embodiment 5 are identified by the same reference numerals and no description will be repeated. This embodiment is directed to a system converter 110 which converts MPEG-4 on the radio channel B1 side to H.263 on the ISDN side in the same system as shown in FIG. 17. In this embodiment the terminal B is the transmitting terminal and the terminal A is the receiving terminal. Reference numeral 111 denotes an MPEG-4 syntax analysis part (syntax analysis means) which: analyzes the input MPEG-4 bit stream 103 according to the MPEG-4 standard; makes a check for a decoding error during analysis; if a decoding error is detected, outputs an error detection signal 115; and separates the MPEG-4 coded bit stream into individual pieces of coded data 116 and outputs them.
Reference numeral 112 denotes a switch which: upon receiving the error detection signal 115 from the MPEG-4 syntax analysis part 111, outputs the input coded data 116 from the analysis part 111 to an error data converting part 113; and when not supplied with the error detection signal 115, provides the coded data 116 to an MPEG-4 syntax constituting part 114. Reference numeral 113 denotes an error data converting part in which a normally unanalyzable bit stream portion of the input coded data 116 is converted, with a minimum deterioration of picture quality, to a substitute value which does not cause an analysis error on the H.263 syntax after being converted. Reference numeral 114 denotes an H.263 syntax constituting part (coded data converting means) in which the coded data 117 provided from the error data converting part 113 or the coded data 116 provided via the switch 112 from the MPEG-4 syntax analysis part 111 is reconstructed as the H.263 coded bit stream 102.
Then, the error data converting part 113 converts a normally unanalyzable bit stream portion of the input coded data 116, with a minimum deterioration of picture quality, to an alternate value (data for concealment) which does not cause an analysis error on the H.263 syntax after being converted (step ST114). The alternate value thus obtained is provided to the H.263 syntax constituting part 114. In the H.263 syntax constituting part 114, the coded data 117 from the error data converting part 113, or the coded data 116 provided via the switch 112 from the MPEG-4 syntax analysis part 111 is reconstructed as the H.263 coded bit stream (step ST115).
Now, a description will be given of a method of converting the normally unanalyzable bit stream portion, while keeping the deterioration of the picture quality to a minimum, to an alternate value that does not cause an analysis error on the H.263 syntax.
Accordingly, it is unknown what value should be used for conversion to the H.263 syntax; for the MPEG-4 coded bit stream area impossible of normal decoding, the system converter 110 of this embodiment sets an alternate value which will minimize the deterioration of picture quality, and uses it as analysis data for conversion to the H.263 syntax.
This is a situation in which the reference image to be used by the VOP for prediction is stored in a frame memory. In the case where no motion is made between VOPs by setting the motion vector to zero, a sufficiently reliable predictive image can be obtained, and by setting the DCT coefficients to zeroes, it is possible to reconstruct the predictive image intact as the H.263 coded bit stream without containing therein extra prediction residual components. Further, in the analysis of the syntax using the MPEG-4 data partitioning scheme, when an error is detected after the unique word shown in FIG. 21, it is possible to choose a procedure by which the motion vector data analyzed before the unique word is used intact and only the DCT coefficients are set to zeroes.
In this case, if the motion vector analyzed before the unique word is reliable data, a highly reliable predictive image can be obtained with an extremely high degree of accuracy and it can be reconstructed as the H.263 coded bit stream. Incidentally, by setting the DCT coefficients to zeroes, the prediction residual components are ignored. In a coded bit stream encoded at a low bit rate, however, since the dynamic rage of the DCT coefficients is inherently so narrow that they tend to be distributed in the neighborhood of zero, the decoded image can sufficiently be approximated only with the predictive image in many cases.
As described above, according to this embodiment, even if a bit error gets mixed in the MPEG-4 coded bit stream that is sent over the radio channel, it is possible to achieve the conversion to the H.263 syntax while keeping the influence of the error to a minimum. Accordingly, stable decoding can be done even in the case where the channel quality is high and an H.263 decoder of low resistance to errors is used.
Incidentally, in this embodiment the 320 terminal is used as the terminal A, but since the system converter according to this embodiment performs conversion of the moving image coded bit stream, it produces the same effects as described above even when the terminal A is an H.324 terminal assumed to be connected to an ordinary analog public network or ISDN circuit based on H.263, or an H.323 terminal assumed to be connected to the Internet. This embodiment is particularly effective in implementing moving picture communications between the communication terminal connected to the radio channel which supports the MPEG-4 video and the communication terminal connected to the ISDN or existing public network which supports the ITU-TH.263 video.
Reference numeral 112 denotes a switch which: upon receiving the error detection signal 115 from the MPEG-4 syntax analysis part 111, outputs the input coded data 116 from the analysis part 111 to an error data converting part 113; and when not supplied with the error detection signal 115, provides the coded data 116 to an MPEG-4 syntax constituting part 114. Reference numeral 113 denotes an error data converting part in which a normally unanalyzable bit stream portion of the input coded data 116 is converted, with a minimum deterioration of picture quality, to a substitute value which does not cause an analysis error on the H.263 syntax after being converted. Reference numeral 114 denotes an H.263 syntax constituting part in which the coded data 117 provided from the error data converting part 113 or the coded data 116 provided via the switch 112 from the MPEG-4 syntax analysis part 111 is reconstructed to the up-link H.263 coded bit stream 102 a. Reference numeral 104 denotes a channel quality monitoring part which: receives and demultiplexes the down-link MPEG-4 coded bit stream 103 b into media multiplexing packets (AL-PDU) to be converted; makes a check to see if bit errors are contained in a CRC field added to each packet; counts the number of bit errors; calculates an average error rate at predetermine time intervals based on the error count value; and outputs it as an internal signal 108 to an error resistance syntax determining part 105.
Reference numeral 105 denotes an error resistance syntax determining part, which: receives from the channel quality monitoring part 104 the internal signal 108 indicating the error rate; selects, in accordance with the error rate, MPEG-4 error resistance syntaxes one by one or in combination of them; reads therein from an external device an error resistance syntax changing period 107 which determines the timing for changing the error resistance syntax according to the channel condition; and, based on the error resistance syntax changing period 107, outputs the error resistance syntax selection result 109 to a syntax converting part 106 b. Reference numeral 106 a denotes an H.263 syntax analysis part (syntax analysis means) which analyzes the input down-link H.263 coded bit stream 102 b according to the H.263 standard and separates the down-link coded bit stream 102 b into individual pieces of coded data 116. Reference numeral 106 b denotes an MPEG-4 syntax constituting part (coded data converting mean), which, when supplied with the error resistance selection result 109 from the error resistance syntax determining part 105, converts the coded data 116 from the H.263 syntax analysis part 106 a to the up-link MPEG-4 coded bit stream 103 a. Next, the operation of this embodiment will be described below.
A description will be given first of the procedure for converting the MPEG-4 coded bit stream to the H.263 coded bit stream.
Upon receiving the error detection signal 115 from the MPEG-4 syntax analysis part 111, the switch 112 provides the coded data 116 also fed from the MPEG-4 syntax analysis part 111 to the error data converting part 111. In the absence of the error detection signal 115, the switch 115 provides the coded data 116 to the H.263 syntax constituting part 114. Then, the error data converting part 113 converts a normally unanalyzable bit stream portion of the input coded data 116, with a minimum deterioration of picture quality, to an alternate value which does not cause an analysis error on the H.263 syntax after being converted. And, in the H.263 syntax constituting part 114, the coded data 117 from the error data converting part 113, or the coded data 116 provided via the switch 112 from the MPEG-4 syntax analysis part 111 is reconstructed as the up-link H.263 coded bit stream 102 a. Next, a description will be given of the procedure for converting the H.263 coded bit stream to the MPEG-4 coded bit stream.
Then, error resistance syntax determining part 105: receives from the channel quality monitoring part 104 the internal signal 108 indicating the error rate; selects, in accordance with the error rate, MPEG-4 error resistance syntaxes one by one or in combination of them; reads therein from an external device the error resistance syntax changing period 107 which determines the timing for changing the error resistance syntax according to the channel condition; and, based on the error resistance syntax changing period 107, outputs the error resistance syntax selection result 109 to the syntax converting part 106 b. And, the H.263 syntax analysis part (syntax analysis means) 106 a analyzes the input down-link H.263 coded bit stream 102 b according to the H.263 standard and separates the down-link coded bit stream 102 b into individual pieces of coded data 116. Next, when supplied with the error resistance selection result 109 from the error resistance syntax determining part 105, the MPEG-4 syntax constituting part 106 b converts the coded data 116 from the H.263 syntax analysis part 106 a to the up-link MPEG-4 coded bit stream 103 a. As described above, according to this embodiment, conversions concerning the error resistance syntax can efficiently made, in accordance with the error rate of the channel connected, between terminals of a two-way moving picture communication which support MPEG-4 or H.263, such as visual telephone or teleconference. This embodiment is particularly effective in implementing moving picture communications between the communication terminal connected to the radio channel which supports the MPEG-4 video and the communication terminal connected to the ISDN or existing public network which supports the ITU-TH.263 video.
FIG. 31 illustrates in block form a system converter of a bit stream converting device according to tenth embodiment (Embodiment 10) of the present invention. The parts corresponding to those in Embodiments 5 and 6 are identified by the same references and no description will be repeated. This embodiment is directed to a system converter 119 which converts MPEG-4 at the radio channel side to H.283 at the ISDN side. In this case, the terminal B is the transmitting terminal and the terminal A is the receiving terminal in FIG. 17.
Reference numeral 120 denotes an error data generating part. For a normally unanalyzable bit stream area of the coded data 116 input thereto via the switch 112, the error data generating part 120 intentionally generates H.263 data 121 containing an error so that data at the position of that area or in its vicinity is similarly detected as a decoding error at the terminal which receives the converted H.263 bit stream.
For example, consider the case where an error occurs in the DSCT coefficient area of a seventh one of 10 macro blocks contained in a certain MPEG-4 video packet. Let it be assumed, however, that decoding is restored normal in the next video packet. In this instance, all the pieces of data from the DCT coefficient area of the seventh macro block to eighth, ninth and tenth macro blocks of the said certain video packet cannot be decoded normally. Accordingly, it is unknown what value should be used for conversion to the H.263 syntax.
For such an MPEG-4 coded bit stream area impossible of normal decoding, the system converting device 119 of this embodiment intentionally generates H.263 data containing an error so that data at the position of that area or in its vicinity is similarly detected as a decoding error at the terminal which receives the converted H.263 bit stream.
In the case where the DCT coefficient area is unanalyzable, the error data generating part 120 intentionally generates a code word that is not contained in a VLC table of DCT coefficients for H.263, or a code word which contains 64 or more DCT coefficients (Since the discrete cosine transform is usually performed for each block composed of 8 by 8 pixels, decoding of 64 or more pieces of coefficient data is actually impossible under normal decoding condition.) and such a code word is provided to an H.263 syntax constituting part 114.
Alternatively, when an error is detected in the motion vector area, the error data generating part 120 intentionally generates a motion vector that lies off screen, or a code word that is not contained in the VLC table of DCT coefficients for H.263, and provides it to the H.263 syntax constituting part 114.
Then, for the normally unanalyzable bit stream are of the coded data 116 input via the switch 112, the error data generating part 120 intentionally generates and outputs the H.263 data 121 containing an error so that data in the unanalyzable bit stream area or in its vicinity is similarly detected as a decoding error at the terminal receiving the converted H.263 bit stream (step ST118). Then, the H.263 syntax constituting part 114 reconstructs and output, as the H.263 coded bit stream 102, the data 121 from the error data generating part 120 or the coded data 116 provided via the switch 112 from the MPEG-4 syntax analysis part (step ST119).
As described above, according to this embodiment, even if a bit error gets mixed in the MPEG-4 coded bit stream that is sent over the radio channel, it is possible to perform the syntax conversion to H.263 while keeping the error condition unchanged. Accordingly, this embodiment eliminates the necessity of taking measures for the system converter 119 itself against errors, permits simplification of the device configuration and ensures maintaining the picture quality according to measures taken at the receiving side against errors.
The system converter 119 of this embodiment is also applicable to the conversion of the H.283 bit stream to the MPEG-4 one. In this instance, the system converter 119 employs an H.263 syntax analysis part and an MPEG-4 syntax constituting part in place of the MPEG-4 syntax analysis part 111 and the H.263 syntax constituting part 114, respectively. As regards an error detected in the H.263 syntax analysis part 111, the error data generating part 120 intentionally generates data so that the MPEG-4 receiving terminal detects an error at the same position as the above error detection or in its vicinity. The operation procedure is similar to that depicted in FIG. 32.
This embodiment uses the H.320 terminal as the terminal A, but since the system converter 119 performs conversion for the moving image coded bit stream, it produces the same effects as described above even when the terminal A is an H.324 terminal assumed to be connected to an ordinary analog public network or ISDN circuit using H.263, or an H.323 terminal assumed to be connected to the Internet. This embodiment is particularly effective in implementing moving picture communications between the communication terminal connected to the radio channel which supports the MPEG-4 video and the communication terminal connected to the ISDN or existing public network which supports the ITU-TH.263 video.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS4651206 *Jan 10, 1985Mar 17, 1987Nec CorporationInter-frame coding apparatus for video signalUS5801781Jun 19, 1996Sep 1, 1998Fujitsu LimitedApparatus for converting moving picture stream of MPEG1 to transport stream of MPEG2US5835144 *Sep 28, 1995Nov 10, 1998Oki Electric Industry Co., Ltd.Methods of coding and decoding moving-picture signals, using self-resynchronizing variable-length codesUS6304607 *Mar 18, 1998Oct 16, 2001Texas Instruments IncorporatedError resilient video coding using reversible variable length codes (RVLCS)US6643729 *Oct 29, 1999Nov 4, 2003Sony CorporationData processing apparatus and data recording apparatusUS6654544 *Nov 8, 1999Nov 25, 2003Sony CorporationVideo data recording apparatus, video data recording method, video data reproducing apparatus, video data reproducing method, video data recording and reproducing apparatus, and video data recording and reproduction methodUS6741793 *Feb 24, 2000May 25, 2004Sony CorporationData transmitting apparatus and method thereof, recording apparatus, and recording and reproducing apparatusUS6807366 *May 12, 2000Oct 19, 2004Sony CorporationData recording apparatus, data recording/reproducing apparatus, data recording method, and data recording/reproducing methodUS20020009232 *Aug 31, 1998Jan 24, 2002Iraj SodagarApparatus and method for packetizing significance-based informationEP0861001A2Feb 4, 1998Aug 26, 1998Texas Instruments IncorporatedError resilient video encodingJPH1056480A Title not availableJPH06205384A Title not availableJPH06276509A Title not availableJPH07143480A Title not availableJPH09191457A Title not available* Cited by examinerReferenced byCiting PatentFiling datePublication dateApplicantTitleUS7792159 *Jan 10, 2006Sep 7, 2010Nec CorporationMultiplexing device and data processing method thereofUS7965736 *Aug 24, 2005Jun 21, 2011Qualcomm IncorporatedTransmission of multiplex protocol data units in physical layer packetsUS8270484 *Jun 24, 2008Sep 18, 2012Kabushiki Kaisha ToshibaInformation processing apparatus, conversion circuit, and programUS8411760 *Jan 28, 2010Apr 2, 2013Broadcom CorporationSystem, method, and apparatus for displaying picturesUS8432936May 16, 2011Apr 30, 2013Qualcomm IncorporatedTransmission of multiplex protocol data units in physical layer packetsUS8566690 *Dec 13, 2010Oct 22, 2013Electronics And Telecommunications Research InstituteApparatus and method for assessing image quality in real-timeUS8605786 *Sep 3, 2008Dec 10, 2013The Regents Of The University Of CaliforniaHierarchical motion vector processing method, software and devicesUS20090003442 *Jun 24, 2008Jan 1, 2009Kabushiki Kaisha ToshibaInformatoin processing apparatus, conversion circuit, and programUS20100080305 *Sep 23, 2009Apr 1, 2010Shaori GuoDevices and Methods of Digital Video and/or Audio Reception and/or Output having Error Detection and/or Concealment Circuitry and TechniquesUS20100118980 *Jan 28, 2010May 13, 2010Qin-Fan ZhuSystem, method, and apparatus for displaying picturesUS20110154172 *Dec 13, 2010Jun 23, 2011Electronics And Telecommunications Research InstituteApparatus and method for assessing image quality in real-timeUS20120219062 *Feb 28, 2011Aug 30, 2012Cisco Technology, Inc.System and method for managing video processing in a network environmentUS20120230388 *Feb 8, 2012Sep 13, 2012Broadcom CorporationMethod and system for protecting image data in frame buffers of video compression systemsUS20140098898 *Oct 5, 2012Apr 10, 2014Nvidia CorporationVideo decoding error concealment techniques* Cited by examinerClassifications U.S. Classification375/240.27, 375/E07.277International ClassificationH04N11/04, H04N7/12, H04N11/02, H04B1/66Cooperative ClassificationH04N21/4341, H04N21/2402, H04N21/234309, H04N21/2368European ClassificationH04N21/24D, H04N21/434A, H04N21/2368, H04N21/2343FLegal EventsDateCodeEventDescriptionMar 27, 2012FPExpired due to failure to pay maintenance feeEffective date: 20120205Feb 5, 2012LAPSLapse for failure to pay maintenance feesSep 12, 2011REMIMaintenance fee reminder mailedOct 20, 2000ASAssignmentOwner name: MITSUBISHI DENKI KABUSHIKI KAISHA, JAPANFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SEKIGUCHI, SHUNICHI;OGAWA, FUMINOBU;ASAI, KOHTARO;AND OTHERS;REEL/FRAME:011262/0616Effective date: 20001018RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services