###### Abstract  
Given the scarcity of annotated datasets, learning the context-dependency of anomalous events as well as mitigating false alarms represent challenges in the task of anomalous activity detection. We propose a framework, Deep-network with Multiple Ranking Measures (DMRMs), which addresses context-dependency using a joint learning technique for motion and appearance features. In DMRMs, the spatial-time-dependent features are extracted from a video using a 3D residual network (ResNet), and deep motion features are extracted by integrating the motion flow maps' information with the 3D ResNet. Afterward, the extracted features are fused for joint learning. This data fusion is then passed through a deep neural network for deep multiple instance learning (DMIL) to learn the context-dependency in a weakly-supervised manner using the proposed multiple ranking measures (MRMs). These MRMs consider multiple measures of false alarms, and the network is trained with both normal and anomalous events, thus lowering the false alarm rate. Meanwhile, in the inference phase, the network predicts each frame's abnormality score along with the localization of moving objects using motion flow maps. A higher abnormality score indicates the presence of an anomalous event. Experimental results on two recent and challenging datasets demonstrate that our proposed framework improves the area under the curve (AUC) score by 6.5% compared to the state-of-the-art method on the UCF-Crime dataset and shows AUC of 68.5% on the ShanghaiTech dataset. anomalous event; deep multiple instance learning; deep motion flow; multiple ranking measures; data fusion +
Footnote †: journal: Medical  
all possible types of anomalous events is unfeasible due to their subjectivity, complexity, and ambiguity. Third, anomalous events are context-dependent; for example, jumping while playing basketball is a normal event, though jumping into someone's house through a window is not. Fourth, the temporal annotation of all data is unfeasible. Finally, proper feature extraction is key to an effective anomalous event detection method. Recent studies have exploited the latest advancements in deep learning techniques [1; 3; 4; 10; 12; 13; 16; 17; 18]. However, one shortcoming of these methods is the lack of relation between appearance and motion features, which affects these methods' performance and outcomes. One study on deep representations of appearance and motion [12] achieved the goal of learning the relationship between these features, but it was done by training three separate networks. Motivated by this work [12], we accomplished the goal of learning the relationship between appearance and motion features by training only a single network. Moreover, our framework has the advantage of preserving the individuality of the motion flow maps and spatial-time-dependent features until they are passed together for joint learning after a fusion technique. This method allows for the deep feature extraction of vital information related to motion and visual data individually, which helps our framework to maximize the utilization of both features, while the fusion technique allows for learning the relationship between the features. In another recent study, [PERSON] et al. [14] has proposed a new approach to detect anomalous events. The authors used video-level annotations to train their model in a weakly supervised manner using multiple instance learning (MIL) [19; 20]. Their work motivates our method. However, our method is quite different from their work in several aspects. First, our method exploits the motion flow maps along with spatial-time-dependent features. Second, we use a different feature extraction technique. Third, we propose a new objective function for DMIL using MRMs. Fourth, we include anomalous event recognition as well as localization of all moving objects. Our feature extraction technique uses a 3D ResNet [21] inspired by related studies [21; 22] and the recent advancements in the action recognition task [21; 22; 23].