# Presentation: 144506

## Motion JPEG 2000 and Video Archiving

- - Metadata from a Tool Implementer’s Viewpoint
- Glenn Pearson
***Lister Hill National Center for Biomedical Communications***
***National Library of Medicine***

## JPEG 2000 for Stills – A Success for Image Archives

- Excellent lossy image quality
- Integrated lossless mode
- Lossy can be added to lossless with negligible increase in filesize
- Metadata!
- Basic metadata support in JP2 (ISO 15444 Part 1)
- Easy to annotate whole image
- Flexible and open XML lets communities develop annotation standards
- More in extended-format JPX (Part 2)
- Specific XML schemas defined
- Easy to annotate regions within image

**Notes:**
Acronym: JPEG 2000 eXtensions

## JPEG 2000 and Video

- At least 2 ways to use JP2 as encoder of video frames
- Motion JPEG 2000 (MJ2) *** This talk*
- DCI’s DCDM, combines MXF and JP2
- Lossy 2K/4K
- Could MJ2 be Successful as a Lossless Format for Non-Studio Video Archives?
- For MJ2 availability, performance
- Pearson and Gill, Archiving 2005
- http://archive.nlm.nih.gov/pubs/pearson/ MJ2_video_archiving.pdf

**Notes:**
Acronym: JPEG 2000 eXtensions

## MJ2 Metadata

- My Recent Tech Report
- “Methods to Store Metadata within Motion JPEG 2000 Files”, June 6, 2005
- http://archive.nlm.nih.gov/pubs/pearson/ MJ2_Metadata2005.pdf
- “Core MJ2” has basic metadata support
- Core = JP2 (Part 1) + MJ2 (Part 3)
- Hybridization with other parts of standard encouraged

**Notes:**
Acronym: JPEG 2000 eXtensions

## Metadata in this Talk

- Structural
- Required by core MJ2 standards
- Descriptive That’s Easy to Implement
- Optional
- Broad: Whole presentation or whole track
- Narrow: Per frame
- How and Where to Store It
- Guidelines of What to Store
- More Difficult Metadata To Embed

## The Structure of Core MJ2

- MJ2 File:  Nested “Boxes” (aka “Atoms”)
- Includes one or more video tracks
- Each frame is a JP2 image:
- Header (most metadata) – stored separately
- Codestream (compressed image data)
- Focus on “Simple Profile” MJ2 = MJ2S
- Single video track, optional audio
- Single self-contained file with track media within

- ‘type’

- length

- Structured Contents (often variable)

- Structural  Metadata

## Interesting Structural Metadata in a “Simple Profile” MJ2S File

- Structural  Metadata

- Presentation as a Whole
- Brands (e.g., “mj2s”)
- Create/Change Timestamp
- Time unit, Duration
- Video Transform (Scaling)
- Audio Volume
- Optional Copyrights (multiple languages)
- Video Track
- Most of above, plus...
- Resolution, color space, color vs. gray, subsampling
- Field interleaving, frame rate
- Structure hierarchy can be mapped to XML
- For historic items, reflects digitalization, not original

## My Software to Show Metadata (Structural So Far)

- Two programs
- Both show MJ2 (and high-level JP2) data as home-brew XML derived from standards
- “Mj2_to_metadata” command-line program
- Included with version 0.97, Open JPEG 2000 software
- www.tele.ucl.ac.be/projects/OpenJPEG/
- In process:  Allow descriptive metadata
- Early prototype GUI viewer
- Views structural metadata with tree control
- To do:
- View descriptive metadata; edit metadata; view video

- Structural  Metadata

**Notes:**
Acronyms: National Information Standards Organization, Metadata for Images in XML Standard, Tagged Image File Format.  TIFF 6: Adobe.  TIFF/EP “Electronic Photography”: ISO/DIS 12234-2 (ISO Tech Comm. 42, WG18)
DCF/EXIF: Design rule for Camera File system (ISO/DIS 12234-3) / EXchangable Image File (JEITA: Japan Electronics and Information Technology Industries Assoc.);

## Formats for Storing Descriptive Text Metadata in MJ2

- Simple Text Strings
- Example: ‘cprt’ box
- Structured as XML, in generic ‘xml ’ box
- can appear “anywhere” in file hierarchy
- No contents suggested in core MJ2
- XML Example:  Digital Still Camera Metadata
- Hybrid with JPX’s “Digital Still Camera (DSC) Profile”?
- Or use independent NISO Technical Metadata
- Drawbacks for our needs
- Applies best to given frame, not video as whole
- For historic items, most tech info (e.g., camera model, lens stop, lighting, etc.) isn’t known or of interest to the archivist

- Descriptive Metadata

- Copyright [‘cprt’]  {...
- String notice = “Copyright 2005 Disney” }

**Notes:**
Acronyms: American National Standards Institute, Inc.; Digital Imaging Group  I3A; International Imaging Industry Association, Inc.

## JPX’s Digital Still Camera (DSC) Profile

- Defined:  Part 2 Annex N
- based on I3A’s DIG35 scheme
- Extended by DSC Profile
- ANSI/I3A IT10.2000-2004
- Five XML boxes
- Image Creation
- Content Description
- History
- Intellectual Property Rights (‘jp2i” box, but still xml)
- Image Identifier

- Descriptive Metadata

**Notes:**
Acronym: JPEG 2000 eXtensions

## Another Format - Binary XML

- Example: MPEG-7’s “BiM” binarization
- Not in core MJ2, but as hybrid
- Still image metadata
- ‘mp7b’ box in JPX
- Movie metadata
- ISO Base File - Part 12 draft Amendment 1
- ‘bxml’ within ‘meta’
- ‘hdlr’ box specifies ‘mp7b’ handler for BiM
- Example Content - Possible MPEG-7 Subsets?
- “Core Description Profile”
- MIC Database

- Descriptive Metadata

**Notes:**
Acronyms: International Standards Organization; World Wide Web Consortium; 

## Yet Another Format - KLV

- Key-Length-Value [SMPTE 336M]
- Basis for MXF; also for user data in AAF
- Allowed in JP2 header (& thus MJ2)
- In ‘uuid’ (Universal Unique ID) box

- Descriptive Metadata

**Notes:**
Acronyms: Society of Motion Picture and Television Engineers, Material eXchange Format; Advanced Authoring Format; 

## Store Within the Video File?

- ... instead of a separate file or database?
- Never
- Main purpose for metadata is for search and retrieval, and indexing must be at collection level
- Separate metadata facilitates update
- Sometimes  *** assume for this talk*
- Ties metadata closer to media (e.g., for rights)
- Aids long-term mobility and preservation (independence from database; redundancy)
- Particularly apt for narrow-scope metadata

- Descriptive Metadata – Guidelines for Embedment

## Storing Within – 

- Metadata describing a collection or composite digital object of which the media in question is just a part

- Descriptive Metadata – Guidelines for Embedment

**Notes:**
Acronyms: Encoded Archival Description, Making of America 2, Metadata Encoding and Transmission Standard, Research Libraries Group

## Storing Within – 

- Domain-specific notes in XML form
- Library or museum metadata describing media file alone
- Video production, distribution, or digital archiving metadata describing media file alone
- MJ2 (beyond the Simple Profile) can support simple edits or edit decision lists

- Descriptive Metadata – Guidelines for Embedment

## Storing Within MJ2 – 

- Complex, video-editing-session metadata can exceed MJ2’s descriptive power
- Stick with editor project file formats, e.g.
- Avid’s OMF
- AAF

- Descriptive Metadata – Guidelines for Embedment

**Notes:**
Acronyms: Open Media Framework

## Implementing Scene Level or Shot Descriptions

- Core MJ2
- Add comment to given JP2 frame
- But no standard for describing frame ranges
- Or make each scene a separate track
- Put metadata in track headers, offset start times
- But now it’s not “Simple Profile” MJ2
- Hybridize with JPX GIF-style animation?
- “Association” box links ‘xml’ annotation with “number list” box of image numbers [Part 2 Annex M]

- More Difficult Metadata To Embed

## Scene Descriptions with  MPEG-7?

- Supports scene synch with MPEG4 files
- MPEG-7 is not just a header in MPEG4... also one or more streams [tracks]
- Please - add new track type to MJ2 standard

- More Difficult Metadata To Embed

## Scene Descriptions with DMS-1?

- MXF’s Descriptive Metadata Scheme 1
- SMPTE S380M
- Has 3 “Frameworks”
- Production (as a whole)
- Clip (scenes from a given source – like “track”)
**Scene** (with frame synchronization)
- Embed DMS-1 in MJ2?
- Use ‘uuid’ box with KLV
- But how to synchronize scenes?
- Might be easier to embed MJ2 in MXF

- More Difficult Metadata To Embed

## Regions and Moving Objects

- Regions within a Frame
- Hybridize with JPX “Label” boxes
- for rectangular or elliptical regions only
- Moving Objects across Frames
- a sequence of Labels with same keyword or object ID
- Somewhat weak, non-standardized
- Would need “smart annotator” to make easy

- More Difficult Metadata To Embed

## Timed Text – Captions & Subtitles

- May come from video source
- “line21” EIA 608; 708B
- No core MJ2 support for this synchronization
- Could use scene-based approaches
- Most practical now – handle from without
- AVI wrapper – separate text stream
- Techniques for web-based streaming playback
- Synchronized Accessible Media Interchange (SAMI)
- Synchronized Multimedia Integration Language (SMIL)
- RealText
- QuickTime Text

- More Difficult Metadata To Embed

## Don’t bother trying
Typically index is external, not embedded

- Typically index is external, not embedded

- Content-based Indexing & Retrieval

- More Difficult Metadata To Embed

## Conclusion

- Low-hanging fruit for builders of MJ2 metadata viewers and editors
- Structural metadata
- Descriptive metadata - scope
- Whole presentation
- Secondarily, individual tracks, frames
- Descriptive metadata – formats & schemas
- Plug-ins for ‘xml ’ box
- Subset of JPX’s DSC Profile
- Variants of Dublin Core (not much work)
- Subset of BiM MPEG-7