Reviewerly/exHarmony huggingface dataset:
LIQA: Lifelong Blind Image Quality Assessment

The image distortions are complex and dynamically changing in the real-world scenario, due to the fast development of the image processing system. The blind image quality assessment (BIQA) models may encounter the challenge of processing images with distortion types never seen before deployment. However, existing BIQA models generally cannot evolve with unseen distortion types adaptively, which greatly limits the deployment and application of BIQA models in real-world scenarios. To address this problem, we propose a novel Lifelong blind Image Quality Assessment (LIQA) approach, targeting to achieve the lifelong learning of BIQA. Without accessing to previous training data, our proposed LIQA can not only learn new knowledge, but also mitigate the catastrophic forgetting of learned knowledge. Specifically, we adopt the Split-and-Merge distillation strategy to train a single-head network that makes task-agnostic predictions. In the split stage, we first employ a distortion-specific generator to generate pseudo features of each previously seen distortion. Then, we utilize an auxiliary multi-head regression network to keep the response of each distortion. In the merge stage, we replay the pseudo features and use the pseudo labels generated by the auxiliary multi-head network to distill the knowledge of the multiple heads, which can build the final regression single head. Extensive experiments demonstrate that LIQA can perform well in handling both inner-dataset distortion shift and cross-dataset distortion shift. More importantly, our model can achieve stable performance even if the task sequences are long.
Detecting Insider Threat via a Cyber-Security Culture Framework

Insider threat has been recognized by both scientific community and security professionals as one of the gravest security hazards for private companies, institutions, and governmental organizations. Extended research on the types, associated internal and external factors, detection approaches and mitigation strategies has been conducted over the last decades. Various frameworks have been introduced in an attempt to understand and reflect the danger posed by this threat, whereas multiple identified cases have been classified in private or public databases. This paper aims to present how a cyber-security culture framework with a clear focus on the human factor can assist in detecting possible threats of both malicious and unintentional insiders. We link current insider threat categories with specific security domains of the framework and introduce an assessment methodology of the core contributing parameters. Specific approach takes into consideration technical, behavioral, cultural, and personal indicators and assists in identifying possible security perils deriving from privileged individuals.
Vid2Player: Controllable Video Sprites That Behave and Appear Like Professional Tennis Players

We present a system that converts annotated broadcast video of tennis matches into interactively controllable video sprites that behave and appear like professional tennis players. Our approach is based on controllable video textures and utilizes domain knowledge of the cyclic structure of tennis rallies to place clip transitions and accept control inputs at key decision-making moments of point play. Most importantly, we use points from the video collection to model a player’s court positioning and shot selection decisions during points. We use these behavioral models to select video clips that reflect actions the real-life player is likely to take in a given match-play situation, yielding sprites that behave realistically at the macro level of full points, not just individual tennis motions. Our system can generate novel points between professional tennis players that resemble Wimbledon broadcasts, enabling new experiences, such as the creation of matchups between players that have not competed in real life or interactive control of players in the Wimbledon final. According to expert tennis players, the rallies generated using our approach are significantly more realistic in terms of player behavior than video sprite methods that only consider the quality of motion transitions during video synthesis. The supplementary material/video are available at our https://cs.stanford.edu/~haotianz/research/vid2player/ project website.
Learned Spatial Representations for Few-shot Talking-Head Synthesis

We propose a novel approach for few-shot talking-head synthesis. While recent works in neural talking heads have produced promising results, they can still produce images that do not preserve the identity of the subject in source images. We posit this is a result of the entangled representation of each subject in a single latent code that models 3D shape information, identity cues, colors, lighting and even background details. In contrast, we propose to factorize the representation of a subject into its spatial and style components. Our method generates a target frame in two steps. First, it predicts a discrete and dense spatial layout for the target image. Second, an image generator utilizes the predicted layout for spatial denormalization and synthesizes the target frame. We experimentally show that this disentangled representation leads to a significant improvement over previous methods, both quantitatively and qualitatively.
Deep Learning Based Just Noticeable Difference and Perceptual Quality Prediction Models for Compressed Video

Human visual system has a limitation of sensitivity in detecting small distortion in an image/video and the minimum perceptual threshold is so called Just Noticeable Difference (JND). JND modelling is challenging since it highly depends on visual contents and perceptual factors are not fully understood. In this paper, we propose deep learning based JND and perceptual quality prediction models, which are able to predict the Satisfied User Ratio (SUR) and Video Wise JND (VWJND) of compressed videos with different resolutions and coding parameters. Firstly, the SUR prediction is modeled as a regression problem that fits deep learning tools. Then, Video Wise Spatial SUR method (VW-SSUR) is proposed to predict the SUR value for compressed video, which mainly considers the spatial distortion. Thirdly, we further propose Video Wise Spatial-Temporal SUR (VW-STSUR) method to improve the SUR prediction accuracy by considering the spatial and temporal information. Two fusion schemes that fuse the spatial and temporal information in quality score level and in feature level, respectively, are investigated. Finally, key factors including key frame and patch selections, cross resolution prediction and complexity are analyzed. Experimental results demonstrate the proposed VW-SSUR method outperforms in both SUR and VWJND prediction as compared with the state-of-the-art schemes. Moreover, the proposed VW-STSUR further improves the accuracy as compared with the VW-SSUR and the conventional JND models, where the mean SUR prediction error is 0.049, and mean VWJND prediction error is 1.69 in quantization parameter and 0.84 dB in peak signal-to-noise ratio.
Granular Fuzzy Rule-Based Modeling With Incomplete Data Representation

Incomplete data are frequently encountered and bring difficulties when it comes to further processing. The concepts of granular computing (GrC) help deliver a higher level of abstraction to address this problem. Most of the existing data imputation and related modeling methods are of numeric nature and require prior numeric models to be provided. The underlying objective of this study is to introduce a novel and straightforward approach that uses information granules as a vehicle to effectively represent missing data and build granular fuzzy models directly from resulting hybrid granular and numeric data. The evaluation and optimization of this method are guided by the principle of justifiable granularity engaging the coverage and specificity criteria and carried out with the help of particle swarm optimization. We provide a collection of experimental studies using a synthetic dataset and several publicly available real-world datasets to demonstrate the feasibility and analyze the main features of this method.
SPB-YOLO: An Efficient Real-Time Detector For Unmanned Aerial Vehicle Images

Recently, using unmanned Aerial Vehicle(UAV) to capture images has become a popular application. However, the large scale variation and dense object distribution characteristic of UAV images brings challenges to object detection. Hence, we propose an efficient end-to-end detector named SPB-YOLO for UAV images. In this paper, firstly we design a Strip Bottleneck (SPB) module to better understand the width-height dependency by using an attention mechanism for improving the detection sensitivity of different scales' objects in the UAV image. Secondly, we propose an upsample strategy based on Path Aggregation Network(PANet) for the feature map and add another one detection head compared to YOLOv5, which specially deal with the detection task of dense objects distribution. Finally, we execute some experiments on two public datasets, and the results show that the proposed SPBYOLO outperforms other latest UAV image detectors and makes a good trade-off between detection accuracy and speed.
A comprehensive review of blockchain applications in industrial Internet of Things and supply chain systems

Abstract The industrial Internet of Things (IIoT) and supply chain systems have become one of the most important technologies to positively impact our daily life activities. Blockchain technology has recently gained popularity and it is increasingly being employed in a large number of applications and industries. The use cases for blockchain are growing rapidly in different fields and domains. Combining blockchain into IIoT and in our supply chain logistics could provide many solutions to the critical issues that current systems face today. Blockchain features such as immutability, decentralization, and data encryption are useful in solving the shortcomings of IIoT and the current supply chain systems security and integrity. This article presents a comprehensive review of blockchain applications in IIoT and supply chain systems. This article also focuses on the security aspects of blockchain integration within industry applications with the support of the emerging edge computing technology.
Explicit and implicit measures of body ownership and agency: affected by the same manipulations and yet independent

People are assumed to represent themselves in terms of body ownership and agency. Studies using the rubber- or virtual-hand illusion have assessed ownership and agency by means of explicit ownership and agency ratings and implicit measures, like proprioceptive drift in the case of ownership. These measures often show similar effects but also some discrepancies, suggesting that they rely on data sources that overlap, but not completely. To systematically assess commonalities and discrepancies, we adopted an immersed virtual hand illusion (VHI) design, in which three independent factors were manipulated: the synchrony between the movement of real and virtual effector, the type of effector, which was a virtual hand or triangle, and the spatial congruency between the real and virtual effector. Commonalities and discrepancies in the effects of these factors were assessed by crossing explicit and implicit measures for ownership and agency. While standard ratings were used as explicit measures, implicit ownership was assessed by means of proprioceptive drift and implicit agency by means of intentional binding. Results showed similar effect patterns for the two agency measures, which, however, were not correlated, different effect patterns for the two ownership measures, and a strong correlation between the two explicit measures. Taken altogether, our findings suggest that explicit and implicit measures of ownership and agency partly rely on shared informational sources, but seem to differ with respect to other sources that are integrated or with respect to the processed dimension (shape vs. time). The findings also suggest that some findings obtained with RHI designs might reflect more the unnatural situation that that design puts individuals into rather than generalizable mechanisms of computing perceived ownership and agency.
Integrated Grad-Cam: Sensitivity-Aware Visual Explanation of Deep Convolutional Networks Via Integrated Gradient-Based Scoring

Visualizing the features captured by Convolutional Neural Networks (CNNs) is one of the conventional approaches to interpret the predictions made by these models in numerous image recognition applications. Grad-CAM is a popular solution that provides such a visualization by combining the activation maps obtained from the model. However, the average gradient-based terms deployed in this method underestimates the contribution of the representations discovered by the model to its predictions. Addressing this problem, we introduce a solution to tackle this issue by computing the path integral of the gradient-based terms in Grad-CAM. We conduct a thorough analysis to demonstrate the improvement achieved by our method in measuring the importance of the extracted representations for the CNN’s predictions, which yields to our method’s administration in object localization and model interpretation.
Bangla Document Classification Using Deep Recurrent Neural Network with BiLSTM

Classification of large unlabelled document collection is a difficult task. Natural language processing can be used as a powerful tool for the classification of an unlabelled large document collection. Document classification is the task of assigning some predefined class to unlabelled large bodies of documents. In this paper, a new way of classifying Bangla news documents was proposed using a Deep Recurrent Neural Network. At first the collected news data was preprocessed. Designing the model architecture training data was fitted out into the model. Finally, model performance was evaluated by calculating the accuracy and F1-score on testing dataset. The Deep Recurrent Neural Network with BiLSTM achieved 98.33% accuracy which is higher than other well-known classification algorithms in Bangla text classification.
The Role of Activation Function in CNN

We all know that the purpose of introducing activation function is to give neural network nonlinear expression ability, so that it can better fit the results, so as to improve the accuracy. However, different activation functions have different performance in different neural networks. In this paper, several activation functions commonly used by researchers are compared one by one, and qualitative comparison results are given by combining with specific neural network models. For example, when using the MNIST dataset in LeNet, PReLU achieved the highest accuracy of 98.724%, followed by Swish at 98.708%. When cifar-10 data set was used, the highest accuracy rate of ELU was 64.580%, followed by Mish at 64.455%. When Using VGG16, ReLU reached the highest accuracy of 90.226%, followed by PReLU at 90.197%. When using ResNet50, ELU achieved the highest accuracy of 89.943%, followed by Mish at 89.780%.
Multi-level transfer learning for improving the performance of deep neural networks: Theory and practice from the tasks of facial emotion recognition and named entity recognition

Transfer learning has become a promising field in machine learning owing to its wide application prospects. Its effectiveness has spawned various methodologies and practices. Transfer learning refers to improving the performance of target learners in the target domain by transferring the knowledge contained in different yet related source domains. In other words, we can use data from additional domains or tasks to train a model with superior generalization. Using transfer learning, the dependence on considerable target-domain data can be reduced, thereby constructing target learners. Recently, the fields of computer vision (CV) and natural language processing (NLP) have witnessed the emergence of transfer learning, which has significantly improved the most advanced technology on a wide range of CV and NLP tasks. A typical approach of applying transfer learning to deep neural networks is to fine-tune a pretrained model of the source domain with data obtained from the target domain. This paper proposes a novel framework, based on the fine-tuning approach, called multilevel transfer learning (mLTL). Under this framework, we concluded the crucial findings and principles regarding the training sequence of related domain datasets and demonstrated its effectiveness by performing facial emotion and named entity recognition tasks. According to the experimental results, the deep neural network models using mLTL outperformed the original models on the target tasks.
D2D-Assisted Federated Learning in Mobile Edge Computing Networks

With the proliferation of edge intelligence and the breakthroughs in machine learning, Federated Learning (FL) is capable of learning a shared model across several edge devices by preserving their private data from being exposed to external adversaries. However, the distributed architecture of FL naturally introduces communication between the central parameter server and the distributed learning nodes. The huge communication cost poses a challenge to practical FL, especially for FL in mobile edge computing (MEC) networks. Existing communication-efficient FL systems predominantly optimize their intrinsic learning process and are not concerned with the implications on the network. In this paper we propose a FL scheme that leverages Device-to-Device (D2D) communication (hence called D2D-FedAvg) and is suitable for mobile edge networks. D2D-FedAvg creates a two-tier learning model where D2D learning groups communicate their results as a single entity to the MEC server leading to traffic reduction. We propose the schemes for D2D grouping, master UE selection, and also D2D exit in the learning process and then form a complete D2D-assisted federated averaging algorithm. Via extensive simulations on the Federated Extended MNIST dataset, the feasibility and convergence of D2D-FedAvg scheme are evaluated. Our results show that D2D-FedAvg lowers the communication cost relative to the typical Federated Averaging (FedAvg) in cellular networks as the number of users is increased (for 100 cellular users 37% traffic reduction), while keeping the same learning accuracy with FedAvg across the board.
Neural variational sparse topic model for sparse explainable text representation

Texts are the major information carrier for internet users, from which learning the latent representations has important research and practical value. Neural topic models have been proposed and have great performance in extracting interpretable latent topics and representations of texts. However, there remain two major limitations: (1) these methods generally ignore the contextual information of texts and have limited feature representation ability due to the shallow feed-forward network architecture, (2) Sparsity of the representations in topic semantic space is ignored. To address these issues, in this paper, we propose a semantic reinforcement neural variational sparse topic model (SR-NSTM) towards explainable and sparse latent text representation learning. Compared with existing neural topic models, SR-NSTM models the generative process of texts with probabilistic distributions parameterized with neural networks and incorporates Bi-directional LSTM to embed contextual information at the document level. It achieves sparse posterior representations over documents and words with zero-mean Laplace distribution and topics with sparsemax. Moreover, we propose a supervised extension of SR-NSTM via adding the max-margin posterior regularization to tackle the supervised tasks. The neural variational inference method is utilized to learn our models efficiently. Experimental results on Web Snippets, 20Newsgroups, BBC, and Biomedical datasets demonstrate that the contextual information and revisiting generative process can improve the performance, leading to the competitive performance of our models in learning coherent topics and explainable sparse representations for texts. • We propose a SR-NSTM for sparse and explainable text representation. • We extend SR-NSTM to supervised learning tasks with the max-margin posterior constraints. • Experimental results demonstrate the superiority of our models in perplexity, topic coherence and text classification accuracy.
Towards Mitigating Device Heterogeneity in Federated Learning via Adaptive Model Quantization

Federated learning (FL) is increasingly becoming the norm for training models over distributed and private datasets. Major service providers rely on FL to improve services such as text auto-completion, virtual keyboards, and item recommendations. Nonetheless, training models with FL in practice requires significant amount of time (days or even weeks) because FL tasks execute in highly heterogeneous environments where devices only have widespread yet limited computing capabilities and network connectivity conditions.
Cloud Resource Demand Prediction using Machine Learning in the Context of QoS Parameters

Abstract Predicting demand for computing resources in any system is a vital task since it allows the optimized management of resources. To some degree, cloud computing reduces the urgency of accurate prediction as resources can be scaled on demand, which may, however, result in excessive costs. Numerous methods of optimizing cloud computing resources have been proposed, but such optimization commonly degrades system responsiveness which results in quality of service deterioration. This paper presents a novel approach, using anomaly detection and machine learning to achieve cost-optimized and QoS-constrained cloud resource configuration. The utilization of these techniques enables our solution to adapt to different system characteristics and different QoS constraints. Our solution was evaluated using a system located in Microsoft’s Azure cloud environment, and its efficiency in other providers’ computing clouds was estimated as well. Experiment results demonstrate a cost reduction ranging from 51% to 85% (for PaaS/IaaS) over the tested period.
Shared Gaze Visualizations in Collaborative Interactions: Past, Present and Future

Abstract The past decade has witnessed a growing interest for using dual eye tracking to understand and support remote collaboration, especially with studies that have established the benefits of displaying gaze information for small groups. While this line of work is promising, we lack a consistent framework that researchers can use to organize and categorize studies on the effect of shared gaze on social interactions. There exists a wide variety of terminology and methods for describing attentional alignment; researchers have used diverse techniques for designing gaze visualizations. The settings studied range from real-time peer collaboration to asynchronous viewing of eye-tracking video of an expert providing explanations. There has not been a conscious effort to synthesize and understand how these different approaches, techniques and applications impact the effectiveness of shared gaze visualizations (SGVs). In this paper, we summarize the related literature and the benefits of SGVs for collaboration, describe important terminology as well as appropriate measures for the dual eye-tracking space and discuss promising directions for future research. As eye-tracking technology becomes more ubiquitous, there is pressing need to develop a consistent approach to evaluation and design of SGVs. The present paper makes a first and significant step in this direction.