Infrastructure for managing HLT data and annotations