Towards Efficient Temporal Activity Detection from Videos
Creators: name: Nag, Sauradip
Publisher: name: University of Surrey
Year: 2024

description: With the tremendous growth in the consumption of videos due to the proliferation
description: of social-media-based applications, video understanding has become
description: a pressing and challenging concern in computer vision. Temporal action
description: detection (TAD) emerges as one of the most crucial and challenging problems
description: within video understanding in computer vision, garnering significant
description: attention in recent years due to its widespread applications in daily life. Notably,
description: TAD has witnessed substantial progress, particularly with the recent
description: advancements in deep learning. There is a growing demand for temporal
description: action detection in untrimmed videos, considering the practicality of naturally
description: occurring untrimmed videos. For lengthy untrimmed videos, temporal
description: action detection primarily addresses two tasks: a) determining when the
description: action occurs, including the start and end times, and b) identifying the
description: category to which each proposal belongs (such as Waving, Climbing, or
description: Basketball-Dunk). Given that a video may contain one or more action clips,
description: temporal action detection aims to develop models and techniques that provide
description: fundamental information for computer vision applications: What are
description: the actions, and when do these actions occur? This task is commonly called
description: action localization, temporal action localization, or action detection. While
description: both action recognition and action detection are crucial aspects of video
description: understanding, temporal action detection poses greater challenges than action
description: recognition. The relationship between action recognition and action
description: detection mirrors that of object recognition and object detection. However,
description: due to the inclusion of temporal information, temporal action detection is
description: a significantly more complex problem than object detection. The difficulties are
description: as follows: a) temporal information: Because of the 1-dimension
description: temporal series information, temporal action detection can’t use static image
description: information. It must combine the information of temporal series. b)
description: Unclear boundaries: Different from object detection, the boundaries of the
description: object are usually obvious, so that we can mark out a more explicit bounding
description: box for the object. However, there exists no sensible definition of the
description: exact temporal extent of an action. So it’s impossible to assign an accurate
description: boundary when the action starts and ends. c) Large temporal spans: The
description: span of temporal action fragments can be huge. For example, waving hands
description: can only take a few seconds, while climbing or cycling can last for tens of
description: minutes. Their spans differ in length which makes them extremely difficult
description: to extract proposals. d) Costly Annotation: To annotate a single video, the
description: annotator has to watch the entire video once, before annotating the start
description: and end point of the action, hence obtaining such annotations is very cost
description: ineffective, thus generalizing it to the real-world data is a key aspect of this
description: problem. In addition, in the open environment, many problems exist such
description: as multiscale, multi-target, and camera movement. Temporal action detection
description: is very close to our lives: it has extensive application prospects and
description: social value in the fields of video summarization, public video surveillance,
description: skill assessment and daily life security. So it has received a lot of attention
description: in recent years. In the following few sections we will discuss the potential
description: drawbacks of the existing TAD approaches and how we have attempted to
description: resolve the shortcomings.<br>
description: <br>
description: Our first chapter (Chapter 3) starts by figuring out how efficiently we
description: can detect actions without using the notorious proposal-based methods. We
description: thus reformulate the conventional proposal-based TAD into a proposal-free
description: design. We start by converting the problem of detection as a regression
description: problem into a classification problem. We realize this using a 1-D segmentation
description: mask instead of discrete start/end points. As a result, the training is
description: faster and predictions are accurate.<br>
description: <br>
description: The second chapter (Chapter 4) discovers that the lack of labelled samples
description: bottleneck TAD performance. This is largely due to an omni-present
description: localization error propagation issue that persists within such networks. This
description: is more prevalent in the semi-supervised setting where we have only partial
description: labels. This problem stems from the model design wherein all the existing
description: approaches are based on a localize-then-classify paradigm for TAD. We
description: therefore introduce a single-stage design to mitigate the localization error
description: and separate out the subtasks. We re-used the unlabelled samples to
description: pretrain the overall network using novel pre-text tasks for a better model prior,
description: while fine-tuning using labelled samples.<br>
description: <br>
description: Thirdly, we extend the second chapter to a more challenging few-shot
description: TAD setting in Chapter 5, where only a few annotated videos per class
description: are available. In such scenarios, one challenging aspect is intra-class
description: variance, where the same class has diverse foreground frame distribution within
description: the support set. Traditional few-shot TAD fails to handle this intra-class
description: variance, which is far inferior to even semi-supervised TAD. We propose a
description: transformer based solution that solves this intra-class variation with the help
description: of self-attention. In addition to this, we investigated the role of multi-modal
description: input (e.g video and text) in the support set, and observed that additional
description: text modality helps to compensate for the data scarcity issue and also address
description: the intra-class variance further thanks to large-scale pretraining. We
description: realize this by meta-learning a novel video-to-text inversion module that
description: helps bridge the vanilla and multi-modal few-shot setup.<br>
description: <br>
description: Finally, in our last chapter (Chapter 6), we extend the TAD problem in
description: the zero-shot open vocabulary setup. We notice that the same localization
description: error propagation problem in chapter 2 also extends to the zero-shot setting,
description: where the error is significant due to the lack of any labeled samples. This
description: lack of labelled samples can be compensated with the usage of large-scale
description: pretrained vision-language models like CLIP. The base design of our network
description: is a single-stage TAD model. However, CLIP was never designed for dense
description: detection tasks in videos, hence we propose a representation masking concept
description: which masks out the foreground and aligns it with the CLIP embeddings.
description: The class-agnostic representation masks additionally makes the model better
description: generalizable to unseen classes. As a result, our proposed design achieves
description: SOTA for both open-set and closed-set TAD settings.<br>