Abstract:
Audiovisual data storage is enhanced using an expanded physical object table utilizing an ordered list of unique identifiers for a particular object for every object instance of an object contained in segments of a data file. Two object instances of the same object in the same segment have different object identifiers. Therefore, different instances of the same object use different identification and the different object instances may be differentiated from one another for access, editing and transmission. The necessary memory required for randomly accessing data contained in files using the expanded physical object table may be reduced by distributing necessary information within a header of a file to simplify the structure of the physical object table. In this way, a given object may be randomly accessed by means of an improved physical object table/segment object table mechanism.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS  
       [0001]    This application is related to U.S. Provisional Application Ser. No. 60/062,120 filed Oct. 15, 1997, from which priority is claimed, and is also related to, a continuation-in-part of, and commonly assigned with U.S. application Ser. No 09/055,933, entitled “System and Method for Processing Object-Based Audiovisual Information” filed Apr. 7, 1998. 
     
    
     
       BACKGROUND OF THE INVENTION  
         [0002]    1. Field of Invention  
           [0003]    The invention relates to information processing, and more particularly to advanced storage and retrieval of audiovisual data objects according to the MPEG-4 standard, including utilization of an expanded physical object table including a list of local object identifiers.  
           [0004]    2. Description of Related Art  
           [0005]    In the wake of rapidly increasing demand for network, multimedia, database and other digital capacity, many multimedia coding and storage schemes have evolved. Graphics files have long been encoded and stored in commonly available file formats such as TIF, GIF, JPG and others, as has motion video in Cinepak, Indeo, MPEG-1 and MPEG-2, and other file formats. Audio files have been encoded and stored in RealAudio, WAV, MIDI and other file formats. These standard technologies have advantages for certain applications, but with the advent of large networks including the Internet the requirements for efficient coding, storage and transmission of audiovisual (AV) information have only increased.  
           [0006]    Motion video in particular often taxes available Internet and other system bandwidth when running under conventional coding techniques, yielding choppy video output having frame drops and other artifacts. This is in part because those techniques rely upon the frame-by-frame encoding of entire monolithic scenes, which results in many megabits-per-second data streams representing those frames. This makes it harder to reach the goal of delivering video or audio content in real-time or streaming form, and to allow editing of the resulting audiovisual scenes.  
           [0007]    In contrast with data streams communicated across a network, content made available in random access mass storage facilities (such as AV files stored on local hard drives) provide additional functionality and sometimes increased speed, but still face increasing needs for capacity. In particular, taking advantage of the random access characteristics of the physical storage medium, it is possible to allow direct access to, and editing of, arbitrary points within a graphical scene description or other audiovisual object information. Besides random access for direct playback purposes, such functionality is useful in editing operations in which one wishes to extract, modify, reinsert or otherwise process a particular elementary stream from a file.  
           [0008]    In conjunction with the development of MPEG-4 coding and storage techniques, it is desirable to provide an improved ability to perform random access of audiovisual objects within video sequences. The opportunity to streamline random access would highlight and strengthen the potential of advanced capabilities provided by MPEG-4, and relieve the demands that those capabilities may impose on resources.  
           [0009]    Part of the approach underlying MPEG-4 formatting is that a video sequence consists of a sequence of related scenes separated in time. Each picture is comprised of a set of audiovisual objects that may undergo a series of changes such as translations, rotations, scaling, brightness in color variations, etc., from one scene to the next. New objects can enter a scene and existing objects can depart, leaving certain objects present only in certain pictures. When scene changes occur, the entire scene and all the objects comprising the picture may be reorganized or initialized.  
           [0010]    One of the identified functionalities of MPEG-4 is improved temporal random access, with the ability to efficiently perform random access of data within an audiovisual sequence in a limited time, and with fine resolution parts (e.g., frames or objects). Improved temporal random access techniques compatible with MPEG-4 involve content based interactivity requiring not only the ability to perform conventional random access, accessing individual pictures, but also the ability to access regions or objects within a scene.  
           [0011]    While the MPEG-4 file format described in U.S. application Ser. No. 09/055,933, entitled “System and Method for Processing Object-Based Audiovisual Information” realizes such advantages, that approach includes at least two disadvantages prompted in part on that file format&#39;s reliance on a standard physical object table (POT) and segment object table (SOT) structure.  
           [0012]    The first problem occurs when multiple instances of the same object exist in the same data segment. In the SOT, different instances of the same object use the same object identification (OBID). Therefore, there is no way using mainstream. MPEG-4 to access the different object instances from the POT because the data field used as an access key, i.e., the OBID, is identical.  
           [0013]    A second problem is that the POT/SOT structure does not recognize the possibility that object identifiers, OBIDs, can be reused. The POT does not include a list of temporal changes that the OBID assumes. Therefore, while MPEG-4 represents a powerful and flexible object-based standard for audiovisual processing, enhancements are desirable.  
         SUMMARY OF THE INVENTION  
         [0014]    The invention overcomes these and other problems in the art and relates to an enhanced audiovisual coding and storage technique, related to MPEG-4, by introducing enhanced formatting including an expanded physical object table which utilizes an “ordered” list of unique identifiers for a particular object for every object instance. Therefore, using the invention, two object instances of the same object in the same segment can be separately identified. Thus, among other advantages, different instances of the identical object may be differentiated from one another.  
           [0015]    The term “ordered” herein denotes that all adaptation layer protocol data (AL PDUs) of the same object instance are placed in the file in their natural order of occurrence, or coding order.  
           [0016]    An additional benefit of the invention is that a given object instance can change its local identifier in time and still be randomly accessed by means of an improved POT/SOT mechanism.  
           [0017]    The invention in one aspect relates to a method of composing data in a file, and a medium for storing that file, the file including a file header containing physical object information and logical object information, and generating a sequence of audiovisual segments, each including a plurality of audiovisual objects. The physical object information and the physical object information contains pointers to access the audiovisual segments.  
           [0018]    In another aspect the invention provides a corresponding method of extracting data from a file, including by accessing a file having a header which contains physical object information and logical object information, and accessing audiovisual segments contained therein.  
           [0019]    In another aspect the invention provides a system for processing a data file including a processor unit and a storage unit connected to the processor unit, the storage unit storing a file including a file header and a sequence of audiovisual segments. The file header contains physical object information and logical object information, and the physical object information contains pointers to access the audiovisual segments. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0020]    The invention will be described with reference to the accompanying drawings, in which like elements are designated by like numbers and in which:  
         [0021]    [0021]FIG. 1 illustrates a file format structure for stored files (with segments containing AL PDUs) according to a first illustrative embodiment of the invention;  
         [0022]    [0022]FIG. 2 illustrates a file format structure for streaming files (with segments containing FlexMux PDUs) according to a second illustrative embodiment of the invention;  
         [0023]    [0023]FIG. 3 illustrates an apparatus for storing audiovisual objects to audiovisual terminals according to the invention;  
         [0024]    [0024]FIG. 4 illustrates an apparatus for extracting audiovisual data stored and accessed according to the invention;  
         [0025]    [0025]FIG. 5 illustrates the format of the EPOT utilized in the first illustrative embodiment of the invention;  
         [0026]    [0026]FIG. 6 illustrates a data access algorithm performed in connection with the first illustrative embodiment of the invention;  
         [0027]    [0027]FIG. 7 illustrates the format of the FPOT utilized in the second illustrative embodiment of the invention;  
         [0028]    [0028]FIG. 8 illustrates a data access algorithm performed in connection with the second illustrative embodiment of the invention;  
         [0029]    [0029]FIG. 9 illustrates the memory format utilized in conjunction with the FPOT according to the second illustrative embodiment of the invention;  
         [0030]    [0030]FIG. 10 illustrates the file format of a local POT (LPOT) utilized in the third illustrative embodiment of the invention;  
         [0031]    [0031]FIG. 11 illustrates the file structure based on the LPOT illustrated in FIG. 10 according to the third illustrative embodiment of the invention; and  
         [0032]    [0032]FIG. 12 illustrates data access algorithm performed in connection with the third illustrative embodiment of the invention. 
     
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS  
       [0033]    The invention will be described in terms of illustrative embodiments in which audiovisual data is accessed from, and output to, file structures for use in data streams configured according to the MPEG-4 format. Further description of that format is made in the aforementioned copending U.S. application Ser. No. 09/055,933, the disclosure of which is incorporated by reference.  
         [0034]    [0034]FIG. 1 illustrates the stored format utilized in relation to a first illustrative embodiment of the invention for MPEG-4 files. Although the present invention is illustratively described in accordance with the stored format, the invention is not limited to utilization with stored files. The present invention may be for instance utilized directly with streamed files.  
         [0035]    The stored format supports random accessing of AV objects. Accessing an AV object at random by object number involves looking up the AL PDU table  190  of a file segment  30  for the OBID. If the OBID is found, the corresponding AL PDU  60  is retrieved. Since an access unit can span more than one AL PDU  60 , it is possible that the requested object is encapsulated in more than one AL PDU  60 . In order to retrieve all the AL PDUs  60  that constitute the requested object, all the AL PDUs  60  with the requested OBID are examined and retrieved until an AL PDU  60  with the first bit set is found.  
         [0036]    The first bit of an AL PDU  60  indicates the beginning of an access unit. If the ID is not found, the AL PDU table  190  in the next segment is examined. All AL PDU  60  segments are listed in the AL PDU table  190 . This format allows more than one object (instance) with the same ID to be present in the same stream segment. It is assumed that AL PDUs  60  of the same OBID are placed in the file in their natural time (or playout) order.  
         [0037]    The invention involves altering the POT structure to provide an expanded physical object table (EPOT). As illustrated in FIG. 5, the format of the EPOT  500  includes a counter (COUNT)  510  of the objects in the EPOT. For each object contained in the POT, the EPOT also contains a count of the different object instances inside the file (ICOUNT)  520 , a list of the local OBID (LLOBID)  530 , an object profile/level (OPL)  540  and a list of positions in the file of the first segment of logical object instance (FSLOI)  550 . The LLOBID  530  is substituted for the OBID in the MPEG-4 standard and the FSLOI  550  is substituted for the first segment of object instance FSOI in the MPEG-4 standard.  
         [0038]    The data access algorithm utilizing the operation of the EPOT  500  will now be described in relation to FIG. 6. The data access algorithm looks up the physical object table EPOT  500  corresponding to the first element of the list of local object identifiers (LLOBID)  530  in step  600 . The list of positions in the file for the first segment of object instance (FSLOI)  550  associated with the first element of the list of local object identifiers (LLOBID)  530  is then accessed in step  605 . The next segment offset (NSOFF) is set equal to the FSLOI  550  position for the first object in step  610 . A pointer position is then incremented to the next segment offset position (NSOFF) in step  615 .  
         [0039]    The current list of object identifiers (CURRLOBID) is set equal to the list of local object identifiers (LLOBID)  530  in step  620 . The algorithm then looks up the segment object table (SOT) corresponding to the current list of object identifiers (CURRLOBID) in step  625 . The local segment offset (LSOFF) and the local AL PDU size (LUS)  195  are located in step  630  and the local segment offset (LSOFF) and the local AL PDU size (LUS)  195  data are accessed in step  635 . Subsequently, the AL PDUs  60  in the segment  30  are loaded and processed in step  640 .  
         [0040]    In step  645 , the continuity flags (CF) are parsed in order to determine if the object is fully contained in an AL PDU  60  or if the AL PDU  60  is the first, the last, or a middle section of an object in step  650 . If the continuity flags denote that the end of the object has been reached, the current list of object identifiers (CURRLOBID) increments to the next element contained within the EPOT LOBID  530  in step  655  and the algorithm is terminated in step  660 . Alternatively, the algorithm accesses the next segment offset (NSOFF) in step  665  and returns to step  615  to increment the pointer position to NSOFF.  
         [0041]    With this operation utilizing the expanded physical object table (EPOT)  500 , random access of the AV object data can be streamlined by removing the lookup mechanism of the segment object table (SOT). The EPOT  500  can be further extended to include the offsets directly to the data objects instead of the beginning of the segment containing the objects by means of a next object offset (NOFF) variable and a local AL PDU size (LUS)  195  variable. The AL PDU LUS  195  has not been used before as a controlling variable during data transmission; however, by using the AL PDU LUS as a variable during data transmission, a unit receiving data is capable of recognizing whether it has sufficient memory available to store the received data and whether the total data has been received during the receiving process.  
         [0042]    The processing flow illustrated in FIG. 6 may be controlled by a file format interface  200  such as that illustrated in FIG. 3. FIG. 3 illustrates an apparatus for processing an MPEG-4 file  100  for playback according to the invention. In the apparatus illustrated in FIG. 3, MPEG-4 files  100  are stored on a storage media, such as a hard disk or CD ROM, which is connected to a file format interface  200  capable of programmed control of audiovisual information, including the processing flow illustrated in FIG. 6.  
         [0043]    In a second illustrative embodiment of the invention, there is provided a further expanded EPOT, denoted FPOT  700  for “fat” POT. As shown in FIG. 7, the format of the FPOT  700  includes a counter (COUNT)  710  of the objects in the FPOT. The FPOT  700  also contains a count of the different object instances inside the file (ICOUNT)  720  and a list of local object identifiers (LLOBID)  730 . The FPOT  700  also contains, for each object entry, an object profile/level (OPL)  740 , a list of positions in the file of the first object instance (FLOI)  750 , a table of next object offsets (NOFFs)  745  and local AL PDU sizes (LUSs)  760  relative to each segment.  
         [0044]    The data access algorithm utilizing the operation of the FPOT  700  will now be described in relation to FIG. 8. The data access algorithm looks up the physical object table FPOT  700  corresponding to the first element of the local object ID (LLOBID)  730  in step  800 . The list of positions in the file for the first object instance (FLOI)  750  associated with the first element of the LLOBID  730  and associated LUS  760  are accessed in step  805 . A pointer position is incremented to the location of the first object instance (FLOI)  750  in step  810  and the LUS data  760  is accessed in step  815 . Next, the AL PDUs  60  in the segment are loaded and processed in step  820 .  
         [0045]    In step  825 , the continuity flags are parsed to determine if the object is fully contained in the AL PDU  60  or if the AL PDU  60  is the first, the last, or a middle section of an object during step  830 . If the continuity flags denote that the end of the object has been reached, the algorithm is terminated in step  835 . Alternatively, if the continuity flags have not reached the end of the object, the algorithm relocates to the next object offset (NOFF)  745  and the size of the adaptation layer process definition unit (AL PDU LUS)  760  is determined in step  840 . Subsequently, the algorithm returns to step  810  to increment the pointer position to the next location of the first object instance (FLOI)  750  and subsequently access the LUS  760 . The processing flow illustrated in FIG. 8 may be controlled by a file format interface  200  such as that illustrated in FIG. 3.  
         [0046]    Throughput for MPEG-4 data access is thus faster according to the invention, because all the information necessary for accessing the objects is contained in the FPOT. Such an approach also simplifies a backward search (reverse traversal) because all the information necessary to access the objects is contained in the FPOT. Thus, implementation using the FPOT structure is the preferred mode for file editing. Further, the FPOT simplifies file conversion into a basic streaming file with or without data access via sequential data scanning based on segment start codes (SSC).  
         [0047]    In terms of data structure, the data following the FPOT  700  is a concatenation of AL PDUs  60 . The format illustrated in FIG. 9 is memory oriented and requires large memory for the FPOT. However, the format allows easy on-the-fly separation of the data access information (i.e., the FPOT entries) and object data (i.e., the AL PDUs). Therefore, the data access information and the object data can be sent over a network with different priorities. When indexing information is not required at the receiver (which is usually the case for most applications), the data access information does not need to be transmitted at all.  
         [0048]    In a third illustrative embodiment of the present invention, a further structure is utilized to more efficiently manage the FPOT  700  of the second illustrative embodiment. In some cases a large FPOT requires extensive memory resources and creates problems with a CPU. For example, in mobile units containing scarce CPU/memory resources, utilization of the FPOT structure may be difficult. Thus, simplifying the FPOT structure by distributing the next object offset (NOFF)  745  and LUS  760  along with the AL PDU data  60  is beneficial.  
         [0049]    Distributed next object chunk offset (DNOFF) information contains the offset value required for positioning to the first AL PDU  60  in the next segment. In the file structure according to the third illustrative embodiment, a further structure, denoted LPOT (local POT)  1000 , is employed. In this structure, illustrated in FIG. 11, the DNOFF  1110  field is the first field before the first AL PDU  60  of the object to which the DNOFF  1110  refers. The distributed LUS (DLUS)  1160  field follows the DNOFF  1110 .  
         [0050]    More detail of the LPOT  1000  structure is shown in FIG. 10, with corresponding file structure shown in FIG. 11. Data access via the LPOT  1000 , DNOFF  1110  and DLUS  1160  may be performed, for example, by a data access algorithm manipulating the loading and processing the AL PDUs  60  based on the distributed next object chunk offset (DNOFF)  1110 .  
         [0051]    The data access operation utilizing the LPOT  1000 , DNOFF  1110  and DLUS  1160  structures of the third illustrative embodiment will now be described in relation to FIG. 12.  
         [0052]    The physical object table LPOT  1000  corresponding to the first element of the LOBID is looked up in step  1200 . Subsequently, the value for DNOFF  1110  is set equal to FLOI  1050  in step  1205 . The pointer position is incremented to the location for DNOFF  1110  in step  1210  and the DLUS  1160  data is accessed in step  1215 . The AL PDUs  60  in the segment are loaded and processed in step  1220 .  
         [0053]    The continuity flags (CF) are parsed in step  1225  in order to determine if the object is fully contained in the AL PDU or if the AL PDU is the first, last or a middle section of an object in step  1230 . If the continuity flags denote that the end of the object has been reached, the algorithm is terminated in step  1235 . Alternatively, the algorithm accesses DNOFF at step  1240 , returns to step  1205  and sets the value of DNOFF to be equal to FLOI. The processing flow illustrated in FIG. 12 may be controlled by a file format interface  200  such as that illustrated in FIG. 3.  
         [0054]    The foregoing description of the system, method and medium for processing audiovisual-information of the invention is illustrative, and variations in construction and implementation will occur to persons skilled in the art. For instance, data access may be similarly performed via sequential data scanning (SSCA) based on segment start codes (SSC), segment size (SS) and the distributed next object chunk offset (DNOFF) and the distributed LUS (DLUS) of the third illustrative embodiment. Accessing the data using segments would be faster in locating the object chunks but slower in locating the LOBID which requires parsing of the AL PDU. The scope of the invention is therefore intended to be limited only by the following claims.