Reviewerly/exHarmony huggingface dataset:
Enhancing Anomaly Detection in Surveillance Videos with Transfer Learning from Action Recognition

Anomaly detection in surveillance videos, as a special case of video-based action recognition, has been of increasing interest in multimedia community and public security. Action recognition in videos faces some challenges, such as cluttered background, illumination conditions. Besides these above difficulties, detecting anomaly in surveillance videos has several unique problems to be solved. For example, the lack of sufficient training samples is one of the main challenges for detecting anomalies in surveillance videos. In this paper, we propose to utilize transfer learning to leverage the good results from action recognition for anomaly detection in surveillance videos. More specially, we explore some techniques based on action recognition models from the following aspects: training samples, temporal modules for action recognition, network backbones. We draw some conclusions. First, more training samples from surveillance videos lead to higher classification accuracy. Second, stronger temporal modules designed for recognizing action and deeper networks do not achieve better results. This conclusion is reasonable since deeper networks tend to over-fitting, especially for the small-scale training set. Besides, to distinguish the hard examples from normal activities, we separately train a neural network to classify the hard category and normal events. Then we fuse the binary network and previous network to generate the final prediction for general anomaly detection. On the benchmarks of CitySCENE, our framework achieves promising performance and obtains the first prize for general anomaly detection and the second prize for specific anomaly detection.
Phase unwrapping method for point diffraction interferometer based on residual auto encoder neural network

In this paper, a phase unwrapping method based on Residual Auto Encoder Network is proposed. Phase unwrapping is regarded as a multiple classification problem and it will be solved by the trained network model. Through training and validation stages, optimal network model can be served as a predictor to predict wrap count distribution map of wrapped phase. Then merge wrapped phase and count together to complete unwrapping. Software simulation and hardware acquisition are the sources for the generation of training dataset. To further improve the accuracy of unwrapping, an image analysis based optimization method is designed that can remove misclassification and noise points in initial result. In addition, phase data stitching by Iterative Closest Point (ICP) is adopted to realize dynamic resolution and enhance the flexibility of method. Point diffraction interferometer (PDI) and multi-step phase extraction technique are introduced, which is the foundation of proposed method. It can be concluded from the experiments that the proposed method is superior to state-of-art ones in terms of unwrapping performance, time efficiency, anti-noise ability and flexibility.
Multi-Features Fusion and Decomposition for Age-Invariant Face Recognition

Although the General Face Recognition (GFR) research achieves great success, Age-Invariant Face Recognition (AIFR) is still a challenging problem since facial appearance changing over time brings significant intra-class variations. The existing discriminative methods for the AIFR task mostly focus on decomposing the facial feature from a sigle image into age-related feature and age-independent feature for recognition, which suffer from the loss of facial identity information. To address this issue, in this work we propose a novel Multi-Features Fusion and Decomposition (MFFD) framework to learn more discriminative feature representations and alleviate the intra-class variations for AIFR. Specifically, we first sample multiple face images of different ages with the same identity as a face time series. Next, we combine feature decomposition with fusion based on the face time series to ensure that the final age-independent features effectively represent the identity information of the face and have stronger robustness against aging. Moreover, we also present two feature fusion methods and several different training strategies to explore the impact on the model. Extensive experiments on several cross-age datasets (CACD, CACD-VS) demonstrate the effectiveness of our proposed method. Besides, our method also shows comparable generalization performance on the well-known LFW dataset.
Privacy-Preserving Visual Content Tagging using Graph Transformer Networks

With the rapid growth of Internet media, content tagging has become an important topic with many multimedia understanding applications, including efficient organisation and search. Nevertheless, existing visual tagging approaches are susceptible to inherent privacy risks in which private information may be exposed unintentionally. The use of anonymisation and privacy-protection methods is desirable, but with the expense of task performance. Therefore, this paper proposes an end-to-end framework (SGTN) using Graph Transformer and Convolutional Networks to significantly improve classification and privacy preservation of visual data. Especially, we employ several mechanisms such as differential privacy based graph construction and noise-induced graph transformation to protect the privacy of knowledge graphs. Our approach unveils new state-of-the-art on MS-COCO dataset in various semi-supervised settings. In addition, we showcase a real experiment in the education domain to address the automation of sensitive document tagging. Experimental results show that our approach achieves an excellent balance of model accuracy and privacy preservation on both public and private datasets.
Low-latency FoV-adaptive Coding and Streaming for Interactive 360° Video Streaming

Virtual Reality (VR) and Augmented Reality (AR) technologies have become popular in recent years. Encoding and transmitting the omni-directional or $360^\circ $ video is critical and challenging for those applications. The $360^\circ $ video requires much higher bandwidth than the traditional planar video. A premium quality $360^\circ $ video with 120 frames per second (fps) and 24K resolution can easily consume bandwidth in the range of Gigabits-per-second~\cite1. On the other hand, at any given time, a user only watches a small portion of the $360^\circ$ scope within her Field-of-View (FoV). An effective way to reduce the bandwidth requirement of $360^\circ $ video is through FoV-adaptive streaming, which codes and delivers the predicted FoV region at higher quality, and discards or codes at lower quality the remaining regions. Such strategy has been quite extensively studied for video-on-demand \citefov_adapt_2,fov_adapt_3,1,tile_based_3,qian2016optimizing and live video streaming applications\citelive_1,live_2,live_3, sun2020flocking. Interactive applications, such as conferencing, gaming, and remote collaboration, can also benefit from $360^\circ $ video by creating an immersive environment for participants to interact with each other citeinteractive_gamming \citevr_conferencing \citelee2015outatime. However, realtime coding and streaming of $360^\circ $ video with extremely low latency, required for interactive applications, has not been sufficiently addressed. This work focuses on developing low-latency and FoV-adaptive coding and streaming strategies for interactive $360^\circ$ video streaming. We assume the sender and the receiver are connected by a network path with dynamically varying throughput without short-latency guarantee. The sender is either the video source, or a proxy server relaying the source video. The receiver is either the end user device that directly renders the video, or a local edge server that renders the video and transmit to the end user \citeHou2017.
Salient Object Detection Combining a Self-Attention Module and a Feature Pyramid Network

Salient object detection has achieved great improvements by using the Fully Convolutional Networks (FCNs). However, the FCN-based U-shape architecture may cause dilution problems in the high-level semantic information during the up-sample operations in the top-down pathway. Thus, it can weaken the ability of salient object localization and produce degraded boundaries. To this end, in order to overcome this limitation, we propose a novel pyramid self-attention module (PSAM) and the adoption of an independent feature-complementing strategy. In PSAM, self-attention layers are equipped after multi-scale pyramid features to capture richer high-level features and bring larger receptive fields to the model. In addition, a channel-wise attention module is also employed to reduce the redundant features of the FPN and provide refined results. Experimental analysis demonstrates that the proposed PSAM effectively contributes to the whole model so that it outperforms state-of-the-art results over five challenging datasets. Finally, quantitative results show that PSAM generates accurate predictions and integral salient maps, which can provide further help to other computer vision tasks, such as object detection and semantic segmentation.
Jointly optimizing global and local color consistency for multiple image mosaicking

In multiple image mosaicking, color inconsistency is a common and severe problem that degrades the quality of composite images. To avoid the appearance of visible seams, we need to optimize the color consistency before mosaicking multiple images. To facilitate the global optimization framework, existing approaches mainly use less flexible global models, e.g., the linear or gamma model, to eliminate the color differences between multiple images. These models can effectively ensure that the global tones in multiple images are consistent. However, their ability to correct local color discrepancies is usually poor. In this paper, we propose a novel image color correction approach that can correct global and local color discrepancies simultaneously and preserve image gradient as much as possible. First, we design an effective and flexible color correction model to represent the color mapping function for each image. Instead of using the same global model to correct color discrepancies for all pixels in one image, we apply a series of local linear models for color correction. For different superpixel regions, different linear models are applied to model the mapping functions. Second, based on this model, a specific global cost function that considers both gradient preservation and color consistency is designed and solved. In addition, a global color constraint is fused into this cost function to ensure that the corrected images have the similar global tone. Thus, we can jointly optimize the global and local color consistency by minimizing this cost function. The experimental results on several challenging datasets captured by different sensors demonstrate that the proposed approach outperforms the state-of-the-art color correction approaches in both visual quality and quantitative metrics.
Finding rising stars through hot topics detection

Topic modeling methods have usually been applied in the past to identify the research interests of researchers. Observing the scientific growth, the trending topics can be identified as Stable, Hot, or Cold. Finding rising stars (junior researchers, who are at the start of their career) from a bibliometric network is a challenging task, specifically if the researchers have an interest in multiple sub-domains or are working on diverse topics. Existing methods for finding rising stars explore the co-author networks or citation networks, and ignore the textual content, which may help in finding rising stars through hot topics detection over time. A publication contributing to a hot topic can be an indication that the author of that publication may be a rising star and can become an expert in that domain in the future. This study proposes the Hot Topics Rising Star Rank (HTRS-Rank) method for finding rising stars by detecting hot topics. HTRS-Rank finds the junior scholars, who contribute to hot topics at the start of their career and ranks them based on the presence of hot topics in their publications. AMiner five years dataset ranging from 2005–2009 is selected for experimentation. Top 10 researchers are considered to measure the association strength using rank correlation among HTRS-Rank and baseline methods. Experimental results show the efficiency of HTRS-Rank in comparison to the baseline methods. The proposed HTRS Rank (TF–IDF) provides low standard deviation for productivity, citations and sociality as compared to baseline methods for more social and highly cited authors. It is identified that HTRS-Rank (WordNet) emphasizes the semantic similarity of two sentences, whereas HTRS-Rank (TF–IDF) scheme emphasizes the uniqueness or importance of each term, therefore TF–IDF approach performs better than WordNet approach due to having higher correlation with StarRank and WMIRank.
Context-aware Attention Network for Predicting Image Aesthetic Subjectivity

Image aesthetic assessment involves both fine-grained details and the holistic layout of images. However, most of current approaches learn the local and the holistic information separately, which has a potential loss of contextual information. Additionally, learning-based methods mainly cast image aesthetic assessment as a binary classification or a regression problem, which cannot sufficiently delineate the potential diversity of human aesthetic experience. To address these limitations, we attempt to render the contextual information and model the varieties of aesthetic experience. Specifically, we explore a context-aware attention module in two dimensions: hierarchical and spatial. The hierarchical context is introduced to present the concern of multi-level aesthetic details while the spatial context is served to yield the long-range perception of images. Based on the attention model, we predict the distribution of human aesthetic ratings of images, which reflects the diversity and similarity of human subjective opinions. We conduct extensive experiments on the prevailing AVA dataset to validate the effectiveness of our approach. Experimental results demonstrate that our approach achieves state-of-the-art results.
Pupil size as a soft biometrics for age and gender classification

The pupil size is a soft biometric trait but in-depth study to analyze it for biometric purposes is lacking in the literature, as well as datasets focused on this field of research are missing. On the basis of these observations, we present an extensive study with the objective of demonstrating that pupil size and dilation over time can be potentially used to classify people by age and by gender. To do this, 14 supervised classifiers were applied on a dataset meant for gaze analysis. Measuring the right and left pupil individually and also simultaneously, the performances of the classifiers have been compared and the worst and best performing selected to support potential fusion strategies. If good results have been obtained for age classification, ranging in 79%–82% of accuracy, it cannot be said the same for gender. This is due to the dual nature of this biometric trait. Pupil size can be considered as a physical trait for the age classification and behavioral trait for the gender. The results achieved in this study suggest the need for more balanced and reliable datasets.
Design and application of a personal credit information sharing platform based on consortium blockchain

The technical features of blockchain, including decentralization, data transparency, tamper-proofing, traceability, privacy protection and open-sourcing, make it a suitable technology for solving the information asymmetry problem in personal credit reporting transactions. Applying blockchain technology to credit reporting meets the needs of social credit system construction and may become an important technical direction in the future. This paper analyzed the problems faced by China’s personal credit reporting market, designed the framework of personal credit information sharing platform based on blockchain 3.0 architecture, studied the technical details of the platform and the technical advantages, and finally, applied the platform to the credit blacklist sharing transaction and explored the possible implementation approach. The in-depth integration of blockchain technology and personal credit reporting helps to realize the safe sharing of credit data and reduce the cost of credit data collection, thereby helping the technological and efficiency transformation of the personal credit reporting industry and promoting the overall development of the social credit system.
The Recognition of Teacher Behavior Based on Multimodal Information Fusion

Teaching reflection based on videos is the main method in teacher education and professional development. However, it takes a long time to analyse videos, and teachers are easy to fall into the state of information overload. With the development of “AI + education,” automatic recognition of teacher behavior to support teaching reflection has become an important research topic. In this paper, taking online open classroom teaching video as the data source, we collected and constructed a teacher behavior dataset. Using this dataset, we explored the behavior recognition methods based on RGB video and skeleton information, and the information fusion between them is carried out to improve the recognition accuracy. The experimental results show that the fusion of RGB information and skeleton information can improve the recognition accuracy, and the early-fusion effect is better than the late-fusion effect. This study helps to solve the problems of time-consumption and information overload in teaching reflection and then helps teachers to optimize the teaching strategies and improve the teaching efficiency.
Region of interest selection for functional features

Feature selection is a critical component in supervised learning to improve model performance. Searching for the optimal feature candidates can be NP-hard. With limited data, cross-validation is widely used to alleviate overfitting, which unfortunately suffers from high computational cost. We propose a highly innovative strategy in feature selection to reduce the overfitting risk but without cross-validation. Our method selects the optimal sub-interval, i.e., region of interest (ROI), of a functional feature for functional linear regression where the response is a scalar and the predictor is a function. For each candidate sub-interval, we evaluate the overfitting risk by calculating a necessary sample size to achieve a pre-specified statistical power. Combining with a model accuracy measure, we rank these sub-intervals and select the ROI. The proposed method has been compared with other state-of-the-art feature selection methods on several reference datasets. The results show that our proposed method achieves an excellent performance in prediction accuracy and reduces computational cost substantially.
Few-Shot Ensemble Learning for Video Classification with SlowFast Memory Networks

In the era of big data, few-shot learning has recently received much attention in multimedia analysis and computer vision due to its appealing ability of learning from scarce labeled data. However, it has been largely underdeveloped in the video domain, which is even more challenging due to the huge spatial-temporal variability of video data. In this paper, we address few-shot video classification by learning an ensemble of SlowFast networks augmented with memory units. Specifically, we introduce a family of few-shot learners based on SlowFast networks which are used to extract informative features at multiple rates, and we incorporate a memory unit into each network to enable encoding and retrieving crucial information instantly. Furthermore, we propose a choice controller network to leverage the diversity of few-shot learners by learning to adaptively assign a confidence score to each SlowFast memory network, leading to a strong classifier for enhanced prediction. Experimental results on two widely-adopted video datasets demonstrate the effectiveness of the proposed method, as well as its superior performance over the state-of-the-art approaches.
Root-cause analysis for time-series anomalies via spatiotemporal graphical modeling in distributed complex systems

Performance monitoring, anomaly detection, and root-cause analysis in complex cyber–physical systems (CPSs) are often highly intractable due to widely diverse operational modes, disparate data types, and complex fault propagation mechanisms. This paper presents a new data-driven framework for root-cause analysis, based on a spatiotemporal graphical modeling approach built on the concept of symbolic dynamics for discovering and representing causal interactions among sub-systems of complex CPSs. We formulate the root-cause analysis problem as a minimization problem via the proposed inference based metric and present two approximate approaches for root-cause analysis, namely the sequential state switching (S3, based on free energy concept of a restricted Boltzmann machine, RBM) and artificial anomaly association (A3, a classification framework using deep neural networks, DNN). Synthetic data from cases with failed pattern(s) and anomalous node(s) are simulated to validate the proposed approaches. Real dataset based on Tennessee Eastman process (TEP) is also used for comparison with other approaches. The results show that: (1) S3 and A3 approaches can obtain high accuracy in root-cause analysis under both pattern-based and node-based fault scenarios, in addition to successfully handling multiple nominal operating modes, (2) the proposed tool-chain is shown to be scalable while maintaining high accuracy, and (3) the proposed framework is robust and adaptive in different fault conditions and performs better in comparison with the state-of-the-art methods.
A Survey on the Progression and Performance of Generative Adversarial Networks

Generative Adversarial Networks (GANs) are a class of deep neural networks that provide a unique way of modeling and generating data in an unsupervised manner. The literature shows that GANs are currently an important research area, being used in a variety of applications. Our survey paper gives an overview of GANs and how they have progressed to become more efficient and powerful. We discuss the efficiency of GAN-variant models, demonstrate how GANs are used in different applications, elaborate on the shortcomings of GANs, review the various ways that researchers have addressed these issues and list several important metrics that are most used in this field. The objective of this paper is to provide a summary of the progression and performance of GANs and the current research that is being conducted to improve them.