# A Self-Paced Multiple Instance Learning Framework for Weakly Supervised Video Anomaly Detection
## 3 Proposed Method
### SP-MIL Framework  
#### 3.2.1 Baseline Model Initialization  
Let \(\{\mathbf{X}_{k}\}_{k=1}^{K}\) denote the given \(K\) training videos which include \(K_{+}\) abnormal videos and \(K_{-}\) normal videos, and the index set of all abnormal videos is \(I_{+}=\{1,2,\cdots,K_{+}\}\) while the negative ones \(I_{-}=\{K_{+}+1,K_{+}+2,\cdots,K\}\). Given a video \(\mathbf{X}_{k}=\{\mathbf{x}_{ki}\}_{i=1}^{N}\) with \(N\) continuous and non-overlapped segments, the corresponding weak label \(y_{k}\in\{0,1\}\) indicates whether \(\mathbf{X}_{k}\) contains abnormal events. The baseline model consists of two primary components: a feature extractor \(g\) and a classifier \(f\). For each video \(\mathbf{X}_{k}\), the feature extractor \(g\) processes video segments \(\{\mathbf{x}_{ki}\}_{i=1}^{N}\) as input, generating segment features \(\{\mathbf{z}_{ki}\}_{i=1}^{N}\), which are subsequently fed into the classifier \(f\) to determine the anomaly scores \(\{s_{ki}\}_{i=1}^{N}\). The top-1 or top-\(k\) segments with the highest anomaly scores are selected and used to train the classifier using video-level labels. 1. Feature extractor \(g\). Feature extractors used in the existing WS-VAD task can be generally categorized into two types. The first type extractors rely solely on a pre-trained network \(h_{0}\) for feature extraction, such as convolutional 3D neural networks (C3D) [20], Inflated 3D ConvNets (I3D) [21], and contrastive language-image pre-training (CLIP) [22]. The output of the feature extractor is generally represented as \(\mathbf{z}_{ki}=g(\mathbf{x}_{ki})=h_{0}(\mathbf{x}_{ki})\). The second type further refines features using a temporal modeling network \(h_{1}\) built upon the pre-trained networks \(h_{0}\), such as graph convo  
Figure 2: Overview of the proposed general SP-MIL framework. lutional network (GCN) [1], multi-scale temporal network (MTN) [5], and temporal self-attention (TSA) [23], etc. In this case, the output of the feature extractor can be represented as \(\mathbf{z}_{ki}=g(\mathbf{x}_{ki})=h_{1}(h_{0}(\mathbf{x}_{ki}))\). Despite the variations in feature extraction methods and output dimensions among different pre-trained networks, each video segment can be consistently represented as a corresponding feature vector, ensuring the compatibility between \(h_{1}\) to \(h_{0}\) modeling. To demonstrate the generality of our SP-MIL framework, we evaluate the effectiveness of two different pre-trained feature extractors (IBD and CLIP) along with DeepMIL [7], which relies solely on the pre-trained network \(h_{0}\) for feature extraction, RTFM [5], and UR-DMU [24], which additionally incorporate the temporal modeling network \(h_{1}\). 2. Classifier \(f\). The standard classifier \(f\) used in the WS-VAD task typically comprises a fully connected network. Each video segment feature \(\mathbf{z}_{ki}\) is fed into \(f\), producing its corresponding anomaly score \(s_{ki}=f(\mathbf{z}_{ki})=f(g(\mathbf{x}_{ki}))\). The widely used classification objective function is as follows: \[L_{\alpha c}=-y_{k}\text{log}p_{k}-(1-y_{k})\text{log}(1-p_{k}),\] (1) where \(p_{k}=\max\limits_{i\in\{1,2,\cdots,N\}}s_{ki}\) or \(p_{k}=\frac{1}{k}\sum_{i\in\Omega_{k}(\mathbf{x}_{k})}s_{ki}\), with \(\Omega_{k}(\mathbf{X}_{k})\) denotes the segment index set of video \(\mathbf{X}_{k}\) with the top-\(k\) largest anomaly scores. This indicates that one or more of the most representative video segments are typically selected using the video-level label to supervise the training of the classifier \(f\).