Before revision: We present HERO, a Hierarchical EncodeR for Omni-representation learning, for large-scale video+language pre-training . HERO encodes multimodal inputs in a hierarchical fashion , where local textual context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer.
After revision: We present HERO, a Hierarchical EncodeR for Omni-representation learning, for large-scale video+language pre-training . HERO encodes multimodal inputs in a hierarchical fashion , where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer.