Patent Publication Number: US-11399052-B2

Title: Timestamp processing methods for streaming media

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention generally relates to streaming media, and, more particularly, to the timestamp processing method for streaming media. 
     2. Description of Related Art 
     An index file (e.g., a .m3u8 file) or a playlist of streaming media (including but not exclusively HTTP Live Streaming (HLS)) usually contains multiple media segment files. A media segment file can also be referred to as a “chip” (e.g., a .ts file) and contains presentation timestamps (PTSs) (hereinafter referred to as timestamps), which are associated with the playback time or display time and used for providing the user with the time information of the streaming media. However, a timestamp jump (PTS jump, which means the timestamps do not show in the expected order) may sometimes occur in media segment files, causing errors in the display time of the streaming media, or even leading to stuttering playback (i.e., not playing smoothly). Therefore, a good timestamp processing method is needed to address the above-mentioned problems. 
     SUMMARY OF THE INVENTION 
     In view of the issues of the prior art, an object of the present invention is to provide a timestamp processing method for streaming media, so as to make an improvement to the prior art. 
     According to one aspect of the present invention, a timestamp processing method of generating a current output video timestamp and a current output audio timestamp for streaming media is provided. The method includes the steps of: extracting a plurality of video data and a plurality of audio data from a multimedia data; processing the video data to obtain a plurality of video timestamps; processing the audio data to obtain a plurality of audio timestamps; initializing a start timestamp; calculating a first characteristic value and a second characteristic value based on a plurality of target timestamps, wherein the target timestamps are the video timestamps or the audio timestamps; and updating the start timestamp and using the start timestamp as the current output video timestamp or the current output audio timestamp when an absolute difference between the first characteristic value and the second characteristic value is greater than a threshold value. 
     According to another aspect of the present invention, a timestamp processing method of generating a current output video timestamp and a current output audio timestamp for streaming media is provided. The method includes the steps of: extracting a plurality of video data and a plurality of audio data from a multimedia data; processing the video data to obtain a plurality of video timestamps; processing the audio data to obtain a plurality of audio timestamps; initializing a start timestamp; calculating a first characteristic value and a second characteristic value based on a plurality of target timestamps, wherein the target timestamps are the video timestamps or the audio timestamps; calculating a standard deviation based on the target timestamps and the second characteristic value when an absolute difference between the first characteristic value and the second characteristic value is not greater than a first threshold value; and updating the start timestamp and using the start timestamp as the current output video timestamp or the current output audio timestamp when the absolute difference between the first characteristic value and the second characteristic value is greater than the first threshold value, or when the standard deviation is greater than or equal to a second threshold value. 
     These and other objectives of the present invention no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiments with reference to the various figures and drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a functional block diagram of a multimedia playback device. 
         FIG. 2  illustrates a flowchart of a timestamp processing method for streaming media according to an embodiment of the present invention. 
         FIG. 3  illustrates a detailed flowchart of the initialization step S 225  of  FIG. 2 . 
         FIG. 4  illustrates a detailed flowchart of the video timestamp processing procedures and the audio timestamp processing procedures of  FIG. 2 . 
         FIG. 5  illustrates a detailed flowchart of step S 450  in  FIG. 4 . 
         FIG. 6  illustrates a flowchart of step S 440  in  FIG. 4  according to one embodiment. 
         FIG. 7  illustrates a flowchart of step S 440  in  FIG. 4  according to another embodiment. 
     
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
     The following description is written by referring to terms of this technical field. If any term is defined in this specification, such term should be interpreted accordingly. In addition, the connection between objects or events in the below-described embodiments can be direct or indirect provided that these embodiments are practicable under such connection. Said “indirect” means that an intermediate object or a physical space exists between the objects, or an intermediate event or a time interval exists between the events. 
     Some or all of the processes of the timestamp processing methods for streaming media may be implemented by software and/or firmware and can be performed by the multimedia playback device of the present invention or its equivalent. A person having ordinary skill in the art can choose components or steps equivalent to those described in this specification to carry out the present invention, which means that the scope of this invention is not limited to the embodiments in the specification. 
       FIG. 1  is a functional block diagram of a multimedia playback device according to one embodiment. The multimedia playback device  100  is capable of processing streaming media and includes a calculation circuit  110  and a memory  120 . The calculation circuit  110  may be a circuit or an electronic component with program execution capability, such as a central processing unit, a microprocessor, a micro-processing unit, or an equivalent thereof. The calculation circuit  110  executes program codes or program instructions stored in the memory  120  to process the timestamps of the streaming media. The memory  120  can also store media segment files and index files. 
       FIG. 2  is a flowchart of a timestamp processing method for streaming media according to an embodiment of the present invention. After obtaining the multimedia data  130  which includes multiple media segment files, the calculation circuit  110  extracts the video data  132  and the audio data  134  from the multimedia data  130  (i.e., from the media segment files) (step S 205 ), and processes the video data  132  to obtain the currently processed video timestamp Vi (step S 212 ) as well as processes the audio data  134  to obtain the currently processed audio timestamp Ai (step S 214 ). People having ordinary skill in the art can perform step S 205 , step S 212 , and step S 214  according to the specifications, so the details of these three steps shall be omitted herein. 
     Next, the calculation circuit  110  determines whether the latest video timestamp obtained in step S 212  (i.e., the currently processed video timestamp Vi) is the first video timestamp (step S 222 ), and determines whether the latest audio timestamp obtained in step S 214  (i.e., the currently processed audio timestamp Ai) is the first audio timestamp (step S 224 ). The first video timestamp and the first audio timestamp respectively refer to the earliest generated video timestamp and the earliest generated audio timestamp after the multimedia playback device  100  starts to play the multimedia data  130  (maybe from the beginning or the middle of the multimedia data  130 ). When the results of step S 222  and step S 224  are YES, the calculation circuit  110  initializes some parameters used in the timestamp processing method (step S 225 ). When the results of step S 222  and step S 224  are NO, the calculation circuit  110  generates an output video timestamp based on some or all of the obtained video timestamps (step S 232 ) and generates an output audio timestamp based on some or all of the obtained audio timestamps (step S 234 ). In some embodiments, the calculation circuit  110  performs step S 232  and step S 234  synchronously. The details of step S 225 , step S 232 , and step S 234  are discussed below. 
       FIG. 3  is a detailed flowchart of the initialization step S 225  of  FIG. 2 . In the initialization step, the calculation circuit  110  sets the threshold value K 1  and the threshold value K 2  (step S 310 ), resets the first characteristic value GapAvePre (step S 320 ), resets the start timestamp startPts (step S 330 ), and sets the base timestamp basePts (step S 340 ). In some embodiments, the calculation circuit  110  sets the first characteristic value GapAvePre to zero (step S 320 ), sets the start timestamp startPts to zero (step S 330 ), and sets the base timestamp basePts to the smaller one of the first video timestamp and the first audio timestamp (step S 340 ). The threshold value K 1  is a positive integer. 
       FIG. 4  is a detailed flowchart of the video timestamp processing procedures (step S 232 ) and the audio timestamp processing procedures (step S 234 ) of  FIG. 2 . Because the contents of step S 232  are substantially identical to those of step S 234  (i.e.,  FIG. 4  is the detailed flow for both step S 232  and step S 234 ), in the following discussion, the target timestamps (PTS i , i=0, 1, 2, . . . , n, n is an integer) are used to represent the video or audio timestamps obtained, the currently processed target timestamp Pt to represent the currently processed video timestamp Vi or the currently processed audio timestamp Ai, the current output target timestamp outputPts to represent the current output video timestamp or the current output audio timestamp, and the preceding output target timestamp lastOutputPts to represent the preceding output video timestamp or the preceding output audio timestamp. The current output target timestamp outputPts immediately follows in time the preceding output target timestamp lastOutputPts; namely, the current output video timestamp immediately follows in time the preceding output video timestamp, and the current output audio timestamp immediately follows in time the preceding output audio timestamp. 
     Reference is made to  FIG. 4 . At first, the calculation circuit  110  stores the currently processed target timestamp Pt in the memory  120  (step S 410 ), and then sorts the stored target timestamps (step S 415 ) (e.g., descending or ascending). The reason for sorting is because in steps S 212  and S 214  the currently processed video timestamp Vi and the currently processed audio timestamp Ai are obtained in the decoding order of the video frames. Then the calculation circuit  110  determines whether the number (n) of the stored target timestamps is greater than the threshold value K 1  (step S 420 ). When the result of step S 420  is YES, the calculation circuit  110  performs step S 425 ; when the result of step S 420  is NO, the calculation circuit  110  performs step S 445 . 
     In step S 425 , the calculation circuit  110  calculates the second characteristic value GapAve based on some or all of the stored target timestamps. In some embodiments, the calculation circuit  110  generate the second characteristic value GapAve by calculating the average value of the differences between adjacent (or contiguous) target timestamps based on equation (1). 
     
       
         
           
             
               
                 
                   GapAve 
                   = 
                   
                     
                       1 
                       
                         K 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         1 
                       
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           
                             n 
                             - 
                             
                               K 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               1 
                             
                           
                         
                         
                           n 
                           - 
                           1 
                         
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             PTS 
                             i 
                           
                           - 
                           
                             PTS 
                             
                               i 
                               - 
                               1 
                             
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     In some embodiments, the threshold value K 1  is set to three in step S 310 ; thus, when the number (n) of stored target timestamps is greater than or equal to four, the calculation circuit  110  performs step S 425 . It should be noted that the threshold value K 1  is not limited to three. Please also note that equation (1) corresponds to the stored target timestamps that are sorted in the ascending order (i.e., PTS i &gt;=PTS i-1 ); when the stored target timestamps are sorted in the descending order (i.e., PTS i-1 &gt;=PTS i ), Equation (1) should be modified to equation (1a). 
     
       
         
           
             
               
                 
                   GapAve 
                   = 
                   
                     
                       1 
                       
                         K 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         1 
                       
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           
                             n 
                             - 
                             
                               K 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               1 
                             
                           
                         
                         
                           n 
                           - 
                           1 
                         
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             PTS 
                             
                               i 
                               - 
                               1 
                             
                           
                           - 
                           
                             PTS 
                             i 
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   
                     1 
                     ⁢ 
                     a 
                   
                   ) 
                 
               
             
           
         
       
     
     Next, the calculation circuit  110  determines whether the first characteristic value GapAvePre is greater than zero (step S 430 ). When the result of step S 430  is YES, the calculation circuit  110  performs step S 440 ; when the result of step S 430  is NO, the calculation circuit  110  performs step S 435 . 
     In step S 440 , the calculation circuit  110  determines, based on the first characteristic value and the second characteristic value, whether a PTS jump occurs (the details of step S 440  will be discussed below). When the PTS jump occurs (YES branch of step S 440 ), the calculation circuit  110  resets the first characteristic value GapAvePre, updates the start timestamp startPts, updates the base timestamp basePts, and deletes the old timestamp(s) in the memory  120  (step S 450 ), and then uses the start timestamp startPts as the current output target timestamp outputPts (step S 460 ) and updates the preceding output target timestamp lastOutputPts by using the current output target timestamp outputPts as the preceding output target timestamp lastOutputPts (step S 465 ). 
     When the PTS jump does not occur (NO branch of step S 440 ), the calculation circuit  110  updates the first characteristic value GapAvePre by using the second characteristic value GapAve as the first characteristic value GapAvePre (step S 435 ) and then determines whether the currently processed target timestamp Pt is greater than the base timestamp basePts (step S 445 ). 
     When the result of step S 445  is YES, the calculation circuit  110  calculates the current output target timestamp outputPts based on the start timestamp startPts, the currently processed target timestamp Pt, and the base timestamp basePts (i.e., outputPts=startPts+Pt-basePts) (step S 455 ); when the result of step S 445  is NO, the calculation circuit  110  performs step S 460 . After step S 465  is finished, the calculation circuit  110  processes the next media segment file (step S 470 ) before returning to step S 212  and step S 214 . 
     It should be noted that for step S 232 , the target timestamp mentioned in the flow of  FIG. 4  refers to the video timestamp, and for step S 234 , the target timestamp mentioned in the flow of  FIG. 4  refers to the audio timestamp. The start timestamp startPts and the target timestamp outputPts are shared by step S 232  and step S 234 . 
       FIG. 5  is a detailed flowchart of step S 450  in  FIG. 4 . In step S 450 , the calculation circuit  110  sets the first characteristic value GapAvePre to zero (step S 510 ), uses the greater one of the preceding output video timestamp and the preceding output audio timestamp as the start timestamp startPts (step S 520 ), and uses the currently processed target timestamp Pt as the base timestamp basePts (step S 530 ). After the calculation circuit  110  deletes the old timestamp(s) in the memory  120 , the only timestamp left in the memory  120  is the currently processed target timestamp Pt. More specifically, after the calculation circuit  110  performs step S 450  of step S 232 , the only timestamp remaining in the memory  120  is the currently processed video timestamp Vi, and after the calculation circuit  110  performs step S 450  of step S 234 , the only timestamp remaining in the memory  120  is the currently processed audio timestamp Ai. 
       FIG. 6  is a flowchart of step S 440  in  FIG. 4 . Step S 440  includes step S 441 : the calculation circuit  110  determines whether the absolute difference between the first characteristic value GapAvePre and the second characteristic value GapAve is greater than the threshold value K 2 . When the result of step S 441  is YES, the calculation circuit  110  determines that the PTS jump has occurred; when the result of step S 441  is NO, the calculation circuit  110  determines that no PTS jump has occurred. As for the video timestamp processing procedures (i.e., step S 232 ), in some embodiments, the threshold value K 2  can be set to ten times the duration of the video frame, and the duration of the video frame is the reciprocal of the frame rate. As for the audio timestamp processing procedures (i.e., step S 234 ), in some embodiments, the threshold value K 2  can be set to 2*2048 times the sample duration, and the sample duration is the reciprocal of the sample rate. It should be noted that the above-mentioned multipliers are not limited to ten and 2*2048. People having ordinary skill in the art can choose various multipliers based on actual operations and/or empirical rules. 
       FIG. 7  is a flowchart of step S 440  in  FIG. 4 . Step S 440  includes step S 441 , step S 442 , and step S 443 . When the result of step S 441  is YES, the calculation circuit  110  determines that the PTS jump has occurred; when the result of step S 441  is NO, the calculation circuit  110  calculates the standard deviation Std of the differences between adjacent (or contiguous) target timestamps based on equation (2) (which corresponds to equation (1)) or (2a) (which corresponds to equation (1a)) (step S 442 ). 
     
       
         
           
             
               
                 
                   Std 
                   = 
                   
                     
                       
                         1 
                         
                           K 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           1 
                         
                       
                       ⁢ 
                       
                         
                           ∑ 
                           
                             i 
                             = 
                             
                               n 
                               - 
                               
                                 K 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 1 
                               
                             
                           
                           
                             n 
                             - 
                             1 
                           
                         
                         ⁢ 
                         
                           
                             ( 
                             
                               
                                 PTS 
                                 i 
                               
                               - 
                               
                                 PTS 
                                 
                                   i 
                                   - 
                                   1 
                                 
                               
                               - 
                               GapAve 
                             
                             ) 
                           
                           2 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
             
               
                 
                   Std 
                   = 
                   
                     
                       
                         1 
                         
                           K 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           1 
                         
                       
                       ⁢ 
                       
                         
                           ∑ 
                           
                             i 
                             = 
                             
                               n 
                               - 
                               
                                 K 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 1 
                               
                             
                           
                           
                             n 
                             - 
                             1 
                           
                         
                         ⁢ 
                         
                           
                             ( 
                             
                               
                                 PTS 
                                 
                                   i 
                                   - 
                                   1 
                                 
                               
                               - 
                               
                                 PTS 
                                 i 
                               
                               - 
                               GapAve 
                             
                             ) 
                           
                           2 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   
                     2 
                     ⁢ 
                     a 
                   
                   ) 
                 
               
             
           
         
       
     
     Next, the calculation circuit  110  determines whether the standard deviation Std is greater than or equal to the threshold value K 3  (step S 443 ). When the result of step S 443  is YES, the calculation circuit  110  determines that the PTS jump has occurred; when the result of step S 443  is NO, the calculation circuit  110  determines that no PTS jump has occurred. In some embodiments, the threshold value K 3  can be set to one. People having ordinary skill in the art can set the threshold value K 3  to other values based on actual operations and/or empirical rules. 
     In comparison with the method in  FIG. 6 , the method in  FIG. 7  can more accurately determine whether the PTS jump has occurred. 
     In summary, the present invention analyzes the average and standard deviation of the differences between adjacent timestamps based on the certainty of the video frame rate and audio sample rate to find out abnormal timestamp(s) (i.e., to determine whether this is any PTS jump) and perform exception handling (i.e., steps S 450  and S 460 ). After the exception handling, the calculation circuit  110  generates a reasonable current output video timestamp and a reasonable current output audio timestamp. As a result, smooth playback and the display of correct time can be achieved without complex fault-tolerant processing in the subsequent audio and video synchronization processes. 
     Please note that the shape, size, and ratio of any element and the step sequence of any flowchart in the disclosed figures are exemplary for understanding, not for limiting the scope of this invention. 
     The aforementioned descriptions represent merely the preferred embodiments of the present invention, without any intention to limit the scope of the present invention thereto. Various equivalent changes, alterations, or modifications based on the claims of the present invention are all consequently viewed as being embraced by the scope of the present invention.