System and method for processing object-based audiovisual information

Audiovisual data storage is enhanced using an expanded physical object table utilizing an ordered list of unique identifiers for a particular object for every object instance of an object contained in segments of a data file. Two object instances of the same object in the same segment have different object identifiers. Therefore, different instances of the same object use different identification and the different object instances may be differentiated from one another for access, editing and transmission. The necessary memory required for randomly accessing data contained in files using the expanded physical object table may be reduced by distributing necessary information within a header of a file to simplify the structure of the physical object table. In this way, a given object may be randomly accessed by means of an improved physical object table/segment object table mechanism.

BACKGROUND OF THE INVENTION
 1. Field of Invention
 The invention relates to information processing, and more particularly to
 advanced storage and retrieval of audiovisual data objects according to
 the MPEG-4 standard, including utilization of an expanded physical object
 table including a list of local object identifiers.
 2. Description of Related Art
 In the wake of rapidly increasing demand for network, multimedia, database
 and other digital capacity, many multimedia coding and storage schemes
 have evolved. Graphics files have long been encoded and stored in commonly
 available file formats such as TIF, GIF, JPG and others, as has motion
 video in Cinepak, Indeo, MPEG-1 and MPEG-2, and other file formats. Audio
 files have been encoded and stored in RealAudio, WAV, MIDI and other file
 formats. These standard technologies have advantages for certain
 applications, but with the advent of large networks including the Internet
 the requirements for efficient coding, storage and transmission of
 audiovisual (AV) information have only increased.
 Motion video in particular often taxes available Internet and other system
 bandwidth when running under conventional coding techniques, yielding
 choppy video output having frame drops and other artifacts. This is in
 part because those techniques rely upon the frame-by-frame encoding of
 entire monolithic scenes, which results in many megabits-per-second data
 streams representing those frames. This makes it harder to reach the goal
 of delivering video or audio content in real-time or streaming form, and
 to allow editing of the resulting audiovisual scenes.
 In contrast with data streams communicated across a network, content made
 available in random access mass storage facilities (such as AV files
 stored on local hard drives) provide additional functionality and
 sometimes increased speed, but still face increasing needs for capacity.
 In particular, taking advantage of the random access characteristics of
 the physical storage medium, it is possible to allow direct access to, and
 editing of, arbitrary points within a graphical scene description or other
 audiovisual object information. Besides random access for direct playback
 purposes, such functionality is useful in editing operations in which one
 wishes to extract, modify, reinsert or otherwise process a particular
 elementary stream from a file.
 In conjunction with the development of MPEG-4 coding and storage
 techniques, it is desirable to provide an improved ability to perform
 random access of audiovisual objects within video sequences. The
 opportunity to streamline random access would highlight and strengthen the
 potential of advanced capabilities provided by MPEG-4, and relieve the
 demands that those capabilities may impose on resources.
 Part of the approach underlying MPEG-4 formatting is that a video sequence
 consists of a sequence of related scenes separated in time. Each picture
 is comprised of a set of audiovisual objects that may undergo a series of
 changes such as translations, rotations, scaling, brightness in color
 variations, etc., from one scene to the next. New objects can enter a
 scene and existing objects can depart, leaving certain objects present
 only in certain pictures. When scene changes occur, the entire scene and
 all the objects comprising the picture may be reorganized or initialized.
 One of the identified functionalities of MPEG-4 is improved temporal random
 access, with the ability to efficiently perform random access of data
 within an audiovisual sequence in a limited time, and with fine resolution
 parts (e.g., frames or objects). Improved temporal random access
 techniques compatible with MPEG-4 involve content based interactivity
 requiring not only the ability to perform conventional random access,
 accessing individual pictures, but also the ability to access regions or
 objects within a scene.
 While the MPEG-4 file format described in U.S. application Ser. No.
 09/055,933 now U.S. Pat. No. 6,079,566, entitled "System and Method for
 Processing Object-Based Audiovisual Information" realizes such advantages,
 that approach includes at least two disadvantages prompted in part on that
 file format's reliance on a standard physical object table (POT) and
 segment object table (SOT) structure.
 The first problem occurs when multiple instances of the same object exist
 in the same data segment. In the SOT, different instances of the same
 object use the same object identification (OBID). Therefore, there is no
 way using mainstream MPEG-4 to access the different object instances from
 the POT because the data field used as an access key, i.e., the OBID, is
 identical.
 A second problem is that the POT/SOT structure does not recognize the
 possibility that object identifiers, OBIDs, can be reused. The POT does
 not include a list of temporal changes that the OBID assumes. Therefore,
 while MPEG-4 represents a powerful and flexible object-based standard for
 audiovisual processing, enhancements are desirable.
 SUMMARY OF THE INVENTION
 The invention overcomes these and other problems in the art and relates to
 an enhanced audiovisual coding and storage technique, related to MPEG-4,
 by introducing enhanced formatting including an expanded physical object
 table which utilizes an "ordered" list of unique identifiers for a
 particular object for every object instance. Therefore, using the
 invention, two object instances of the same object in the same segment can
 be separately identified. Thus, among other advantages, different
 instances of the identical object may be differentiated from one another.
 The term "ordered" herein denotes that all adaptation layer protocol data
 (AL PDUs) of the same object instance are placed in the file in their
 natural order of occurrence, or coding order.
 An additional benefit of the invention is that a given object instance can
 change its local identifier in time and still be randomly accessed by
 means of an improved POT/SOT mechanism.
 The invention in one aspect relates to a method of composing data in a
 file, and a medium for storing that file, the file including a file header
 containing physical object information and logical object information, and
 generating a sequence of audiovisual segments, each including a plurality
 of audiovisual objects. The physical object information and the physical
 object information contains pointers to access the audiovisual segments.
 In another aspect the invention provides a corresponding method of
 extracting data from a file, including by accessing a file having a header
 which contains physical object information and logical object information,
 and accessing audiovisual segments contained therein.
 In another aspect the invention provides a system for processing a data
 file including a processor unit and a storage unit connected to the
 processor unit, the storage unit storing a file including a file header
 and a sequence of audiovisual segments. The file header contains physical
 object information and logical object information, and the physical object
 information contains pointers to access the audiovisual segments.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
 The invention will be described in terms of illustrative embodiments in
 which audiovisual data is accessed from, and output to, file structures
 for use in data streams configured according to the MPEG-4 format. Further
 description of that format is made in the aforementioned copending U.S.
 application Ser. No. 09/055,933, now U.S. Pat. No. 6,079,566, the
 disclosure of which is incorporated by reference.
 FIG. 1 illustrates the stored format utilized in relation to a first
 illustrative embodiment of the invention for MPEG-4 files. Although the
 present invention is illustratively described in accordance with the
 stored format, the invention is not limited to utilization with stored
 files. The present invention may be for instance utilized directly with
 streamed files.
 The stored format supports random accessing of AV objects. Accessing an AV
 object at random by object number involves looking up the AL PDU table 190
 of a file segment 30 for the OBID. If the OBID is found, the corresponding
 AL PDU 60 is retrieved. Since an access unit can span more than one AL PDU
 60, it is possible that the requested object is encapsulated in more than
 one AL PDU 60. In order to retrieve all the AL PDUs 60 that constitute the
 requested object, all the AL PDUs 60 with the requested OBID are examined
 and retrieved until an AL PDU 60 with the first bit set is found.
 The first bit of an AL PDU 60 indicates the beginning of an access unit. If
 the ID is not found, the AL PDU table 190 in the next segment is examined.
 All AL PDU 60 segments are listed in the AL PDU table 190. This format
 allows more than one object (instance) with the same ID to be present in
 the same stream segment. It is assumed that AL PDUs 60 of the same OBID
 are placed in the file in their natural time (or playout) order.
 The invention involves altering the POT structure to provide an expanded
 physical object table (EPOT). As illustrated in FIG. 5, the format of the
 EPOT 500 includes a counter (COUNT) 510 of the objects in the EPOT. For
 each object contained in the POT, the EPOT also contains a count of the
 different object instances inside the file (ICOUNT) 520, a list of the
 local OBID (LLOBID) 530, an object profile/level (OPL) 540 and a list of
 positions in the file of the first segment of logical object instance
 (FSLOI) 550. The LLOBID 530 is substituted for the OBID in the MPEG-4
 standard and the FSLOI 550 is substituted for the first segment of object
 instance FSOI in the MPEG-4 standard.
 The data access algorithm utilizing the operation of the EPOT 500 will now
 be described in relation to FIG. 6. The data access algorithm looks up the
 physical object table EPOT 500 corresponding to the first element of the
 list of local object identifiers (LLOBID) 530 in step 600. The list of
 positions in the file for the first segment of object instance (FSLOI) 550
 associated with the first element of the list of local object identifiers
 (LLOBID) 530 is then accessed in step 605. The next segment offset (NSOFF)
 is set equal to the FSLOI 550 position for the first object in step 610. A
 pointer position is then incremented to the next segment offset position
 (NSOFF) in step 615.
 The current list of object identifiers (CURRLOBID) is set equal to the list
 of local object identifiers (LLOBID) 530 in step 620. The algorithm then
 looks up the segment object table (SOT) corresponding to the current list
 of object identifiers (CURRLOBID) in step 625. The local segment offset
 (LSOFF) and the local AL PDU size (LUS) 195 are located in step 630 and
 the local segment offset (LSOFF) and the local AL PDU size (LUS) 195 data
 are accessed in step 635. Subsequently, the AL PDUs 60 in the segment 30
 are loaded and processed in step 640.
 In step 645, the continuity flags (CF) are parsed in order to determine if
 the object is fully contained in an AL PDU 60 or if the AL PDU 60 is the
 first, the last, or a middle section of an object in step 650. If the
 continuity flags denote that the end of the object has been reached, the
 current list of object identifiers (CURRLOBID) increments to the next
 element contained within the EPOT LOBID 530 in step 655 and the algorithm
 is terminated in step 660. Alternatively, the algorithm accesses the next
 segment offset (NSOFF) in step 665 and returns to step 615 to increment
 the pointer position to NSOFF.
 With this operation utilizing the expanded physical object table (EPOT)
 500, random access of the AV object data can be streamlined by removing
 the lookup mechanism of the segment object table (SOT). The EPOT 500 can
 be further extended to include the offsets directly to the data objects
 instead of the beginning of the segment containing the objects by means of
 a next object offset (NOFF) variable and a local AL PDU size (LUS) 195
 variable. The AL PDU LUS 195 has not been used before as a controlling
 variable during data transmission; however, by using the AL PDU LUS as a
 variable during data transmission, a unit receiving data is capable of
 recognizing whether it has sufficient memory available to store the
 received data and whether the total data has been received during the
 receiving process.
 The processing flow illustrated in FIG. 6 may be controlled by a file
 format interface 200 such as that illustrated in FIG. 3. FIG. 3
 illustrates an apparatus for processing an MPEG-4 file 100 for playback
 according to the invention. In the apparatus illustrated in FIG. 3, MPEG-4
 files 100 are stored on a storage media, such as a hard disk or CD ROM,
 which is connected to a file format interface 200 capable of programmed
 control of audiovisual information, including the processing flow
 illustrated in FIG. 6.
 In a second illustrative embodiment of the invention, there is provided a
 further expanded EPOT, denoted FPOT 700 for "fat" POT. As shown in FIG. 7,
 the format of the FPOT 700 includes a counter (COUNT) 710 of the objects
 in the FPOT. The FPOT 700 also contains a count of the different object
 instances inside the file (ICOUNT) 720 and a list of local object
 identifiers (LLOBID) 730. The FPOT 700 also contains, for each object
 entry, an object profile/level (OPL) 740, a list of positions in the file
 of the first object instance (FLOI) 750, a table of next object offsets
 (NOFFs) 745 and local AL PDU sizes (LUSs) 760 relative to each segment.
 The data access algorithm utilizing the operation of the FPOT 700 will now
 be described in relation to FIG. 8. The data access algorithm looks up the
 physical object table FPOT 700 corresponding to the first element of the
 local object ID (LLOBID) 730 in step 800. The list of positions in the
 file for the first object instance (FLOI) 750 associated with the first
 element of the LLOBID 730 and associated LUS 760 are accessed in step 805.
 A pointer position is incremented to the location of the first object
 instance (FLOI) 750 in step 810 and the LUS data 760 is accessed in step
 815. Next, the AL PDUs 60 in the segment are loaded and processed in step
 820.
 In step 825, the continuity flags are parsed to determine if the object is
 fully contained in the AL PDU 60 or if the AL PDU 60 is the first, the
 last, or a middle section of an object during step 830. If the continuity
 flags denote that the end of the object has been reached, the algorithm is
 terminated in step 835. Alternatively, if the continuity flags have not
 reached the end of the object, the algorithm relocates to the next object
 offset (NOFF) 745 and the size of the adaptation layer process definition
 unit (AL PDU LUS) 760 is determined in step 840. Subsequently, the
 algorithm returns to step 810 to increment the pointer position to the
 next location of the first object instance (FLOI) 750 and subsequently
 access the LUS 760. The processing flow illustrated in FIG. 8 may be
 controlled by a file format interface 200 such as that illustrated in FIG.
 3.
 Throughput for MPEG-4 data access is thus faster according to the
 invention, because all the information necessary for accessing the objects
 is contained in the FPOT. Such an approach also simplifies a backward
 search (reverse traversal) because all the information necessary to access
 the objects is contained in the FPOT. Thus, implementation using the FPOT
 structure is the preferred mode for file editing. Further, the FPOT
 simplifies file conversion into a basic streaming file with or without
 data access via sequential data scanning based on segment start codes
 (SSC).
 In terms of data structure, the data following the FPOT 700 is a
 concatenation of AL PDUs 60. The format illustrated in FIG. 9 is memory
 oriented and requires large memory for the FPOT. However, the format
 allows easy on-the-fly separation of the data access information (i.e.,
 the FPOT entries) and object data (i.e., the AL PDUs). Therefore, the data
 access information and the object data can be sent over a network with
 different priorities. When indexing information is not required at the
 receiver (which is usually the case for most applications), the data
 access information does not need to be transmitted at all.
 In a third illustrative embodiment of the present invention, a further
 structure is utilized to more efficiently manage the FPOT 700 of the
 second illustrative embodiment. In some cases a large FPOT requires
 extensive memory resources and creates problems with a CPU. For example,
 in mobile units containing scarce CPU/memory resources, utilization of the
 FPOT structure may be difficult. Thus, simplifying the FPOT structure by
 distributing the next object offset (NOFF) 745 and LUS 760 along with the
 AL PDU data 60 is beneficial.
 Distributed next object chunk offset (DNOFF) information contains the
 offset value required for positioning to the first AL PDU 60 in the next
 segment. In the file structure according to the third illustrative
 embodiment, a further structure, denoted LPOT (local POT) 1000, is
 employed. In this structure, illustrated in FIG. 11, the DNOFF 1110 field
 is the first field before the first AL PDU 60 of the object to which the
 DNOFF 1110 refers. The distributed LUS (DLUS) 1160 field follows the DNOFF
 1110.
 More detail of the LPOT 1000 structure is shown in FIG. 10, with
 corresponding file structure shown in FIG. 11. Data access via the LPOT
 1000, DNOFF 1110 and DLUS 1160 may be performed, for example, by a data
 access algorithm manipulating the loading and processing the AL PDUs 60
 based on the distributed next object chunk offset (DNOFF) 1110.
 The data access operation utilizing the LPOT 1000, DNOFF 1110 and DLUS 1160
 structures of the third illustrative embodiment will now be described in
 relation to FIG. 12.
 The physical object table LPOT 1000 corresponding to the first element of
 the LOBID is looked up in step 1200. Subsequently, the value for DNOFF
 1110 is set equal to FLOI 1050 in step 1205. The pointer position is
 incremented to the location for DNOFF 1110 in step 1210 and the DLUS 1160
 data is accessed in step 1215. The AL PDUs 60 in the segment are loaded
 and processed in step 1220.
 The continuity flags (CF) are parsed in step 1225 in order to determine if
 the object is filly contained in the AL PDU or if the AL PDU is the first,
 last or a middle section of an object in step 1230. If the continuity
 flags denote that the end of the object has been reached, the algorithm is
 terminated in step 1235. Alternatively, the algorithm accesses DNOFF at
 step 1240, returns to step 1205 and sets the value of DNOFF to be equal to
 FLOI. The processing flow illustrated in FIG. 12 may be controlled by a
 file format interface 200 such as that illustrated in FIG. 3.
 The foregoing description of the system, method and medium for processing
 audiovisual information of the invention is illustrative, and variations
 in construction and implementation will occur to persons skilled in the
 art. For instance, data access may be similarly performed via sequential
 data scanning (SSCA) based on segment start codes (SSC), segment size (SS)
 and the distributed next object chunk offset (DNOFF) and the distributed
 LUS (DLUS) of the third illustrative embodiment. Accessing the data using
 segments would be faster in locating the object chunks but slower in
 locating the LOBID which requires parsing of the AL PDU. The scope of the
 invention is therefore intended to be limited only by the following
 claims.