Reviewerly/exHarmony huggingface dataset:
Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization

Localizing persons and recognizing their actions from videos is a challenging task towards high-level video understanding. Recent advances have been achieved by modeling direct pairwise relations between entities. In this paper, we take one step further, not only model direct relations between pairs but also take into account indirect higher-order relations established upon multiple elements. We propose to explicitly model the Actor-Context-Actor Relation, which is the relation between two actors based on their interactions with the context. To this end, we design an Actor-Context-Actor Relation Network (ACAR-Net) which builds upon a novel High-order Relation Reasoning Operator and an Actor-Context Feature Bank to enable indirect relation reasoning for spatio-temporal action localization. Experiments on AVA and UCF101-24 datasets show the advantages of modeling actor-context-actor relations, and visualization of attention maps further verifies that our model is capable of finding relevant higher-order relations to support action detection. Notably, our method ranks first in the AVA-Kinetics action localization task of ActivityNet Challenge 2020, outperforming other entries by a significant margin (+6.71 mAP). The code is available online. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Backdoor Attacks Against Deep Learning Systems in the Physical World

Backdoor attacks embed hidden malicious behaviors into deep learning models, which only activate and cause misclassifications on model inputs containing a specific "trigger." Existing works on backdoor attacks and defenses, however, mostly focus on digital attacks that apply digitally generated patterns as triggers. A critical question remains unanswered: "can backdoor attacks succeed using physical objects as triggers, thus making them a credible threat against deep learning systems in the real world?"We conduct a detailed empirical study to explore this question for facial recognition, a critical deep learning task. Using 7 physical objects as triggers, we collect a custom dataset of 3205 images of 10 volunteers and use it to study the feasibility of "physical" backdoor attacks under a variety of real-world conditions. Our study reveals two key findings. First, physical backdoor attacks can be highly successful if they are carefully configured to overcome the constraints imposed by physical objects. In particular, the placement of successful triggers is largely constrained by the target model’s dependence on key facial features. Second, four of today’s state-of-the-art defenses against (digital) backdoors are ineffective against physical backdoors, because the use of physical objects breaks core assumptions used to construct these defenses.Our study confirms that (physical) backdoor attacks are not a hypothetical phenomenon but rather pose a serious real-world threat to critical classification tasks. We need new and more robust defenses against backdoors in the physical world.
Embedded Discriminative Attention Mechanism for Weakly Supervised Semantic Segmentation

Weakly Supervised Semantic Segmentation (WSSS) with image-level annotation uses class activation maps from the classifier as pseudo-labels for semantic segmentation. However, such activation maps usually highlight the local discriminative regions rather than the whole object, which deviates from the requirement of semantic segmentation. To explore more comprehensive class-specific activation maps, we propose an Embedded Discriminative Attention Mechanism (EDAM) by integrating the activation map generation into the classification network directly for WSSS. Specifically, a Discriminative Activation (DA) layer is designed to explicitly produce a series of normalized class-specific masks, which are then used to generate class-specific pixel-level pseudo-labels demanded in segmentation. For learning the pseudo-labels, the masks are multiplied with the feature maps after the backbone to generate the discriminative activation maps, each of which encodes the specific information of the corresponding category in the input images. Given such class-specific activation maps, a Collaborative Multi-Attention (CMA) module is proposed to extract the collaborative information of each given category from images in a batch. In inference, we directly use the activation masks from the DA layer as pseudo-labels for segmentation. Based on the generated pseudo-labels, we achieve the mIoU of 70.60% on PASCAL VOC 2012 segmentation testset, which is the new state-of-the-art, to our best knowledge. Code and pre-trained models are available online soon.
YOLOv4_Drone: UAV image target detection based on an improved YOLOv4 algorithm

Advanced communications and networks have greatly improved the user experience, and unmanned aerial vehicle (UAV) are an important technology that supports people's daily life and military activities. Since target detection in UAV images is complicated by a complex background, small targets, and target occlusion, the detection accuracy of the You Only Look Once(YOLO) v4 algorithm is relatively low. Therefore, hollow convolution is used to resample the feature image to improve the feature extraction and target detection performance. In addition, the ultra-lightweight subspace attention mechanism (ULSAM) is used to derive different attention feature maps for each subspace of the feature map for multi-scale feature representation. Finally, soft non-maximum suppression (Soft-NMS) is introduced to minimize the occurrence of missed targets due to occlusion. The experimental results prove that the proposed UAV image target detection model (YOLOv4_Drone) has 5% improved to the YOLOv4 algorithm, demonstrating the effectiveness of the method.
Machine Learning/Artificial Intelligence for Sensor Data Fusion–Opportunities and Challenges

During Fusion 2019 Conference (https://www.fusion2019.org/program.html), leading experts presented ideas on the historical, contemporary, and future coordination of artificial intelligence/machine learning (AI/ML) with sensor data fusion (SDF). While AI/ML and SDF concepts have had a rich history since the early 1900s—emerging from philosophy and psychology—it was not until the dawn of computers that both AI/ML and SDF researchers initiated discussions on how mathematical techniques could be implemented for real-time analysis. ML, and in particular deep learning, has demonstrated tremendous success in computer vision, natural language understanding, and data analytics. As a result, ML has been proposed as the solution to many problems that inherently include multi-modal data. For example, success in autonomous vehicles has validated the promise of ML with SDF, but additional research is needed to explain, understand, and coordinate heterogeneous data analytics for situation awareness. The panel identified opportunities for merging AI/ML and SDF such as computational efficiency, improved decision making, expanding knowledge, and providing security; while highlighting challenges for multi-domain operations, human-machine teaming, and ethical deployment strategies.
Network Pruning via Performance Maximization

Channel pruning is a class of powerful methods for model compression. When pruning a neural network, it's ideal to obtain a sub-network with higher accuracy. However, a sub-network does not necessarily have high accuracy with low classification loss (loss-metric mismatch). In the paper, we first consider the loss-metric mismatch problem for pruning and propose a novel channel pruning method for Convolutional Neural Networks (CNNs) by directly maximizing the performance (i.e., accuracy) of sub-networks. Specifically, we train a stand-alone neural network to predict sub-networks' performance and then maximize the output of the network as a proxy of accuracy to guide pruning. Training such a performance prediction network efficiently is not an easy task, and it may potentially suffer from the problem of catastrophic forgetting and the imbalance distribution of sub-networks. To deal with this challenge, we introduce a corresponding episodic memory to update and collect sub-networks during the pruning process. In the experiment section, we further demonstrate that the gradients from the performance prediction network and the classification loss have different directions. Extensive experimental results show that the proposed method can achieve state-of-the-art performance with ResNet, MobileNetV2, and ShuffleNetV2+ on ImageNet and CIFAR-10.
Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical World

Deep learning models are vulnerable to adversarial examples. As a more threatening type for practical deep learning systems, physical adversarial examples have received extensive research attention in recent years. However, without exploiting the intrinsic characteristics such as model-agnostic and human-specific patterns, existing works generate weak adversarial perturbations in the physical world, which fall short of attacking across different models and show visually suspicious appearance. Motivated by the viewpoint that attention reflects the intrinsic characteristics of the recognition process, this paper proposes the Dual Attention Suppression (DAS) attack to generate visually-natural physical adversarial camouflages with strong transferability by suppressing both model and human attention. As for attacking, we generate transferable adversarial camouflages by distracting the model-shared similar attention patterns from the target to non-target regions. Meanwhile, based on the fact that human visual attention always focuses on salient items (e.g., suspicious distortions), we evade the human-specific bottom-up attention to generate visually-natural camouflages which are correlated to the scenario context. We conduct extensive experiments in both the digital and physical world for classification and detection tasks on up-to-date models (e.g., Yolo-V5) and demonstrate that our method outperforms state-of-the-art methods. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Fine-Grained Shape-Appearance Mutual Learning for Cloth-Changing Person Re-Identification

Recently, person re-identification (Re-ID) has achieved great progress. However, current methods largely depend on color appearance, which is not reliable when a person changes the clothes. Cloth-changing Re-ID is challenging since pedestrian images with clothes change exhibit large intra-class variation and small inter-class variation. Some significant features for identification are embedded in unobvious body shape differences across pedestrians. To explore such body shape cues for cloth-changing Re-ID, we propose a Fine-grained Shape-Appearance Mutual learning framework (FSAM), a two-stream framework that learns fine-grained discriminative body shape knowledge in a shape stream and transfers it to an appearance stream to complement the cloth-unrelated knowledge in the appearance features. Specifically, in the shape stream, FSAM learns fine-grained discriminative mask with the guidance of identities and extracts fine-grained body shape features by a pose-specific multi-branch network. To complement cloth-unrelated shape knowledge in the appearance stream, dense interactive mutual learning is performed across low-level and high-level features to transfer knowledge from shape stream to appearance stream, which enables the appearance stream to be deployed independently without extra computation for mask estimation. We evaluated our method on benchmark cloth-changing Re-ID datasets and achieved the start-of-the-art performance.
A Hybrid CNN-LSTM: A Deep Learning Approach for Consumer Sentiment Analysis Using Qualitative User-Generated Contents

With the fastest growth of information and communication technology (ICT), the availability of web content on social media platforms is increasing day by day. Sentiment analysis from online reviews drawing researchers’ attention from various organizations such as academics, government, and private industries. Sentiment analysis has been a hot research topic in Machine Learning (ML) and Natural Language Processing (NLP). Currently, Deep Learning (DL) techniques are implemented in sentiment analysis to get excellent results. This study proposed a hybrid convolutional neural network-long short-term memory (CNN-LSTM) model for sentiment analysis. Our proposed model is being applied with dropout, max pooling, and batch normalization to get results. Experimental analysis carried out on Airlinequality and Twitter airline sentiment datasets. We employed the Keras word embedding approach, which converts texts into vectors of numeric values, where similar words have small vector distances between them. We calculated various parameters, such as accuracy, precision, recall, and F1-measure, to measure the model’s performance. These parameters for the proposed model are better than the classical ML models in sentiment analysis. Our results analysis demonstrates that the proposed model outperforms with 91.3% accuracy in sentiment analysis.
Sustainability of Healthcare Data Analysis IoT-Based Systems Using Deep Federated Learning

Due to recent privacy trends and the increase in data breaches in various industries, it has become imperative to adopt new technologies that support data privacy, maintain accuracy, and ensure sustainability at the same time. The healthcare industry is one of the most vulnerable sectors to cyberattacks and data breaches as health data are highly sensitive and distributed in nature. The use of IoT devices with machine learning models to monitor the health status has made the challenge more acute, as it increases the distribution of health data and adds a decentralized structure to healthcare systems. A new privacy-preserving technology, namely, federated learning (FL), is promising for such a challenge as implementing solutions that integrate FL with deep learning, for healthcare applications that rely on IoT, provides several benefits by mainly preserving data privacy, building robust and high accuracy models, and dealing with the decentralized structure, thus achieving sustainability. This article proposes a deep FL (DFL) framework for healthcare data monitoring and analysis using IoT devices. Moreover, it proposes an FL algorithm that addresses the local training data acquisition process. Furthermore, it presents an experiment to detect skin diseases using the proposed framework. The extensive results collected show that the DFL models can preserve data privacy without sharing it, maintain the decentralized structure of the system made by IoT devices, improve the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">area under the curve</i> (AUC) of the model to reach 97%, and reduce the operational costs (OC) for service providers.
DOLG: Single-Stage Image Retrieval with Deep Orthogonal Fusion of Local and Global Features

Image Retrieval is a fundamental task of obtaining images similar to the query one from a database. A common image retrieval practice is to firstly retrieve candidate images via similarity search using global image features and then re-rank the candidates by leveraging their local features. Previous learning-based studies mainly focus on either global or local image representation learning to tackle the retrieval task. In this paper, we abandon the two-stage paradigm and seek to design an effective single-stage solution by integrating local and global information inside images into compact image representations. Specifically, we propose a Deep Orthogonal Local and Global (DOLG) information fusion framework for end-to-end image retrieval. It attentively extracts representative local information with multi-atrous convolutions and self-attention at first. Components orthogonal to the global image representation are then extracted from the local information. At last, the orthogonal components are concatenated with the global representation as a complementary, and then aggregation is performed to generate the final representation. The whole framework is end-to-end differentiable and can be trained with image-level labels. Extensive experimental results validate the effectiveness of our solution and show that our model achieves state-of-the-art image retrieval performances on Revisited Oxford and Paris datasets. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Improving the Efficiency and Robustness of Deepfakes Detection through Precise Geometric Features

Deepfakes is a branch of malicious techniques that transplant a target face to the original one in videos, resulting in serious problems such as infringement of copyright, confusion of information, or even public panic. Previous efforts for Deepfakes videos detection mainly focused on appearance features, which have a risk of being bypassed by sophisticated manipulation, also resulting high model complexity and sensitiveness to noise. Besides, how to mine the temporal features of manipulated videos and exploit them is still an open question. We propose an efficient and robust framework named LRNet for detecting Deepfakes videos through temporal modeling on precise geometric features. A novel calibration module is devised to enhance the precision of geometric features, making it more discriminative, and a two-stream Recurrent Neural Network (RNN) is constructed for sufficient exploitation of temporal features. Compared to previous methods, our proposed method is lighter-weighted and easier to train. Moreover, our method has shown robustness in detecting highly compressed or noise corrupted videos. Our model achieved 0.999 AUC on FaceForensics+ + dataset. Meanwhile, it has a graceful decline in performance (-0.042 AUC) when faced with highly compressed videos. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Blockchain Based Crop Insurance: A Decentralized Insurance System for Modernization of Indian Farmers

Conventional crop insurance systems are complex and often not economically feasible. Farmers are often reluctant to be covered for their crops due to lack of trust in insurance firms and the fear of delayed or non-payment of claims. In this paper, a blockchain based crop insurance solution is suggested. The solution suggested in this paper is an affordable, efficient, low cost crop insurance solution which will ensure many farmers are insured and benefiting from timely crop insurance. Currently the cost of administering insurance is an essential barrier to accessing this facility. With the proper use of blockchain based on ethereum this expense can be reduced dramatically. We have conducted various tests on platforms such as Google Cloud and found that the least throughput is 165 transactions. Upon analysis we have found that the time taken by the block formation is directly proportional to the timing of processing. The end-to-end average latency of the system was achieved as 31.2 s, which was quite effective for the infrastructure what we are using. Upon conducting acceptance testing, we found that the system suggested in the paper is effective and we are planning to release the application on open source platforms for future improvements.
Virtual reality marketing and customer advocacy: Transforming experiences from story-telling to story-doing

Marketing managers strive to build branded experiences that both excite and engage their customers in novel ways in order to enhance attitudes and encourage positive behaviors towards their brands. As it offers immersive and interactive encounters, Virtual Reality (VR) technology is a promising tool for managers to create these experiences, evidenced by increasing and successful VR marketing applications. Yet, the literature offers little guidance on how VR experiences can be strategically designed to create favorable customer perceptions, attitudes, and behaviors. Based on five semi-structured focus groups of 27 VR consumers, this article constructs a framework that deconstructs VR branded experiences into both narrative and social interactions to optimize strategic customer outcomes. An experimental study validates the findings after which practical recommendations to maximize the success of VR branding campaigns and a future research agenda for VR marketing is provided.
Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer

Unsupervised domain adaptation (UDA) aims to transfer knowledge from a related but different well-labeled source domain to a new unlabeled target domain. Most existing UDA methods require access to the source data, and thus are not applicable when the data are confidential and not shareable due to privacy concerns. This paper aims to tackle a realistic setting with only a classification model available trained over, instead of accessing to, the source data. To effectively utilize the source model for adaptation, we propose a novel approach called Source HypOthesis Transfer (SHOT), which learns the feature extraction module for the target domain by fitting the target data features to the frozen source classification module (representing classification hypothesis). Specifically, SHOT exploits both information maximization and self-supervised learning for the feature extraction module learning to ensure the target features are implicitly aligned with the features of unseen source data via the same hypothesis. Furthermore, we propose a new labeling transfer strategy, which separates the target data into two splits based on the confidence of predictions (labeling information), and then employ semi-supervised learning to improve the accuracy of less-confident predictions in the target domain. We denote labeling transfer as SHOT++ if the predictions are obtained by SHOT. Extensive experiments on both digit classification and object recognition tasks show that SHOT and SHOT++ achieve results surpassing or comparable to the state-of-the-arts, demonstrating the effectiveness of our approaches for various visual domain adaptation problems. Code will be available at https://github.com/tim-learn/SHOT-plus.
Ownership, uses and perceptions of cryptocurrency: Results from a population survey

A decade after the launch of Bitcoin, cryptocurrencies are maturing from a niche phenomenon and appealing to a broader audience. However, the actual prevalence of cryptocurrency ownership and usage, the users’ socio-demographics, the motives to buy, and the popularity of and knowledge about cryptocurrencies have not been sufficiently researched. Based on a representative online survey among 3864 Germans, we find that 83% of the respondents are aware of the phenomenon, yet the respondents’ self-assessed knowledge about cryptocurrencies and the underlying blockchain technology is limited. 9.2% of the respondents owned cryptocurrencies at the time of the survey; another 9.1% have owned cryptocurrencies in the past. Cryptocurrency users tend to be young, male, well-educated and well-off. Ownership is often associated with long-term investments, and 62% of the respondents state that their ownership is ideologically motivated. The empirical analysis discloses that a major driver of ownership is knowledge about cryptocurrencies, mediated by trust. There is some discrepancy between the actual and perceived usage domains of cryptocurrencies, which reflects the polarization of the phenomenon. The findings have implications for regulators and businesses which are potentially affected by the increasing societal relevance of cryptocurrency.