Query: Video-Text

title: ActBERT: Learning Global Local Video Text Representations
abstract: In this paper, we introduce ActBERT for self supervised learning of joint video text representations from unlabeled data. First, we leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce an ENtangled Transformer block (ENT) to encode three sources of information, i.e. global actions, local regional objects, and linguistic descriptions. Global local correspondences are discovered via judicious clues extraction from contextual information. It enforces the joint videotext representation to be aware of fine grained objects as well as global human intention. We validate the generalization capability of ActBERT on downstream video and language tasks, i.e. text video clip retrieval, video captioning, video question answering, action segmentation, and action step localization. ActBERT significantly outperform the state of the arts, demonstrating its superiority in video text representation learning.

title: Fine Grained Video Text Retrieval With Hierarchical Graph Reasoning
abstract: Cross modal retrieval between videos and texts has attracted growing attentions due to the rapid emergence of videos on the web. The current dominant approach is to learn a joint embedding space to measure cross modal similarities. However, simple embeddings are insufficient to represent complicated visual and textual details, such as scenes, objects, actions and their compositions. To improve fine grained video text retrieval, we propose a Hierarchical Graph Reasoning (HGR) model, which decomposes video text matching into global to local levels. The model disentangles text into a hierarchical semantic graph including three levels of events, actions, entities, and generates hierarchical textual embeddings via attention based graph reasoning. Different levels of texts can guide the learning of diverse and hierarchical video representations for cross modal matching to capture both global and local details. Experimental results on three video text datasets demonstrate the advantages of our model. Such hierarchical decomposition also enables better generalization across datasets and improves the ability to distinguish fine grained semantic differences. Code will be released at \url{https:/github.com/cshizhe/hgr_v2t}

title: Local Global Video Text Interactions for Temporal Grounding
abstract: This paper addresses the problem of text to video temporal grounding, which aims to identify the time interval in a video semantically relevant to a text query. We tackle this problem using a novel regression based model that learns to extract a collection of mid level features for semantic phrases in a text query, which corresponds to important semantic entities described in the query (e.g. actors, objects, and actions) and reflect bi modal interactions between the linguistic features of the query and the visual features of the video in multiple levels. The proposed method effectively predicts the target time interval by exploiting contextual information from local to global during bi modal interactions. Through in depth ablation studies, we find out that incorporating both local and global context in video and text interactions is crucial to the accurate grounding. Our experiment shows that the proposed method outperforms the state of the arts on Charades STA and ActivityNet Captions datasets by large margins, 7.44\ and 4.61\ points at Recall@tIoU=0.5 metric, respectively.

title: COOT: Cooperative Hierarchical Transformer for Video Text Representation Learning
abstract: Many real world video text tasks involve different levels of granularity, such as frames and words, clip and sentences or videos and paragraphs, each with distinct semantics. In this paper, we propose a Cooperative hierarchical Transformer (COOT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities. The method consists of three major components: an attention aware feature aggregation layer, which leverages the local temporal context (intra level, e.g. within a clip) a contextual transformer to learn the interactions between low level and high level semantics (inter level, e.g. clip video, sentence paragraph) and a cross modal cycle consistency loss to connect video and text. The resulting method compares favorably to the state of the art on several benchmarks while having few parameters. All code is available open source at this https URL

title: Exploiting Visual Semantic Reasoning for Video Text Retrieval
abstract: Video retrieval is a challenging research topic bridging the vision and language areas and has attracted broad attention in recent years. Previous works have been devoted to representing videos by directly encoding from frame level features. In fact, videos consist of various and abundant semantic relations to which existing methods pay less attention. To address this issue, we propose a Visual Semantic Enhanced Reasoning Network (ViSERN) to exploit reasoning between frame regions. Specifically, we consider frame regions as vertices and construct a fully connected semantic correlation graph. Then, we perform reasoning by novel random walk rule based graph convolutional networks to generate region features involved with semantic relations. With the benefit of reasoning, semantic interactions between regions are considered, while the impact of redundancy is suppressed. Finally, the region features are aggregated to form frame level features for further encoding to measure video text similarity. Extensive experiments on two public benchmark datasets validate the effectiveness of our method by achieving state of the art performance due to the powerful semantic reasoning.

title: Learning Coarse to Fine Graph Neural Networks for Video Text Retrieval
abstract: We address the problem of video text retrieval that searches videos via natural language description or vice versa. Most state of the art methods only consider cross modal learning for two or three data points in isolation, ignoring to get benefit from the structural information of other data points from a global view. In this paper, we propose to exploit the comprehensive relationships among cross modal samples via Graph Neural Networks (GNN) To improve the discriminative ability for accurately finding the positive sample, a Coarse to Fine GNN is constructed, which can progressively optimize the retrieval results via multi step reasoning. Specifically, we first adopt heuristic edge features to represent relationships. Then we design a scoring module in each layer to rank the edges connected to the query node and drop the edges with lower scores. Finally, to alleviate the class imbalance issue, we propose a random drop focal loss to optimize the whole framework. Extensive experimental results show that our method consistently outperforms the state of the arts on four benchmarks.

title: A Novel Convolutional Architecture For Video Text Retrieval
abstract: The prevalent video text retrieval methods usually use recurrent neural networks to encode sequences of frames in videos and sequences of words in text. In this paper, we introduce an encoding architecture based entirely on convolutional neural networks. Compared to recurrent models, the complexity is smaller, and computations over all elements can be fully parallelized during training to better exploit the GPU. We use the stacking of convolution kernels of different scales to realize the encoding of local and long term features of video and text. Experiments validate that our method achieves a new state of the art for the video text retrieval on MSR VTT and MSVD datasets with less training time.

title: Stacked Convolutional Deep Encoding Network For Video Text Retrieval
abstract: Existing dominant approaches for cross modal video text retrieval task are to learn ajoint embedding space to measure the cross modal similarity. However, these methods rarely explore long range dependency inside video frames or textual words leading to insufficient textual and visual details. In this paper, we propose a stacked convolutional deep encoding network for video text retrieval task, which considers to simultaneously encode long range and short range dependency in the videos and texts. Specifically, a multi scale dilated convolutional (MSDC) block within our approach is able to encode short range temporal cues between video frames or text words by adopting different scales of kernel size and dilation size of convolutional layer. A stacked structure is designed to expand the receptive fields by repeatedly adopting the MSD C block, which further captures the long range relations between these cues. Moreover, to obtain more robust textual representations, we fully utilize the powerful language model named Transformer in two stages: pretraining phrase and fine tuning phrase. Extensive experiments on two different benchmark datasets (MSR VTT, MSVD) show that our proposed method outperforms other state of the art approaches.

title: Deep Semantic Multimodal Hashing Network for Scalable Image Text and Video Text Retrievals.
abstract: Hashing has been widely applied to multimodal retrieval on large scale multimedia data due to its efficiency in computation and storage. In this article, we propose a novel deep semantic multimodal hashing network (DSMHN) for scalable image text and video text retrieval. The proposed deep hashing framework leverages 2 D convolutional neural networks (CNN) as the backbone network to capture the spatial information for image text retrieval, while the 3 D CNN as the backbone network to capture the spatial and temporal information for video text retrieval. In the DSMHN, two sets of modality specific hash functions are jointly learned by explicitly preserving both intermodality similarities and intramodality semantic labels. Specifically, with the assumption that the learned hash codes should be optimal for the classification task, two stream networks are jointly trained to learn the hash functions by embedding the semantic labels on the resultant hash codes. Moreover, a unified deep multimodal hashing framework is proposed to learn compact and high quality hash codes by exploiting the feature representation learning, intermodality similarity preserving learning, semantic label preserving learning, and hash function learning with different types of loss functions simultaneously. The proposed DSMHN method is a generic and scalable deep hashing framework for both image text and video text retrievals, which can be flexibly integrated with different types of loss functions. We conduct extensive experiments for both single modal and cross modal retrieval tasks on four widely used multimodal retrieval data sets. Experimental results on both image text and video text retrieval tasks demonstrate that the DSMHN significantly outperforms the state of the art methods.

title: Learning a Video Text Joint Embedding using Korean Tagged Movie Clips
abstract: For intelligent multimedia services, video contents understanding is a major challenge. In the existing video retrieval approaches, manual descriptive sentence data is necessary for retrieving desired videos against user's search intent. To overcome these limitations, modeling visual concepts included in video and sentence is necessary to learn a mapping of video and text into a common vector space, where relevant videos and texts are close to each other. In this study, we construct a new dataset containing 250 Korean movies with manual text description in Korean. Also, video text joint embedding model and its quantitative and qualitative search results are introduced. With our proposed model, video manual tagging is no longer necessary for video retrieval services.