Reviewerly/exHarmony huggingface dataset:
Multi-task convolutional neural network with coarse-to-fine knowledge transfer for long-tailed classification

Long-tailed classifications make it very challenging to deal with class-imbalanced problems using deep convolutional neural networks (CNNs). Existing solutions based on re-balancing methods perform well and use single-task CNNs to train each fine-grained class independently. However, classification tasks are multiplex and involve a coarse-to-fine hierarchical relation. In this paper, we propose a coarse-to-fine knowledge transfer based multi-task CNN, which utilizes the coarse-to-fine structure to promote long-tailed learning. First, we construct a tail hierarchical structure in a coarse-to-fine way to pay greater attention to tail classes than head classes. Second, a multi-task CNN is adopted to simultaneously train coarse- and fine-grained tasks to extract a more generalized knowledge representation than the single-task CNN. Third, we design a coarse-to-fine knowledge transfer strategy to adaptively adjust the task weights to improve fine-grained performance. Extensive experiments on benchmark datasets show that our model achieves better gains than the re-balancing methods. In particular, the proposed model is 3.25% more accurate than the second-best method on the long-tailed tieredImageNet dataset.
Network Support Data Analysis for Fault Identification Using Machine Learning

Machine learning has gained immense popularity in a variety of fields as it has the ability to change the conventional workflow of a process. The abundance of data available serves as the motivation for this. This data can be exploited for a good deal of knowledge. In this article, we focus on operational data of networking devices that are deployed in different locations. This data can be used to predict faults in the devices. Usually, after the deployment of networking devices in customer site, troubleshooting these devices is difficult. Operational data of these devices is needed for this process. Manually analysing the machined produced operational data is tedious and complex due to enormity of data. Using machine learning techniques will be of greater help here as this will help automate the troubleshooting process, avoid human errors and save time for the technical solutions engineers.
Language Use in Conversational Agent–Based Health Communication: Systematic Review

Given the growing significance of conversational agents (CAs), researchers have conducted a plethora of relevant studies on various technology- and usability-oriented issues. However, few investigations focus on language use in CA-based health communication to examine its influence on the user perception of CAs and their role in delivering health care services.This review aims to present the language use of CAs in health care to identify the achievements made and breakthroughs to be realized to inform researchers and more specifically CA designers.This review was conducted by following the protocols of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 statement. We first designed the search strategy according to the research aim and then performed the keyword searches in PubMed and ProQuest databases for retrieving relevant publications (n=179). Subsequently, 3 researchers screened and reviewed the publications independently to select studies meeting the predefined selection criteria. Finally, we synthesized and analyzed the eligible articles (N=11) through thematic synthesis.Among the 11 included publications, 6 deal exclusively with the language use of the CAs studied, and the remaining 5 are only partly related to this topic. The language use of the CAs in these studies can be roughly classified into six themes: (1) personal pronouns, (2) responses to health and lifestyle prompts, (3) strategic wording and rich linguistic resources, (4) a 3-staged conversation framework, (5) human-like well-manipulated conversations, and (6) symbols and images coupled with phrases. These derived themes effectively engaged users in health communication. Meanwhile, we identified substantial room for improvement based on the inconsistent responses of some CAs and their inability to present large volumes of information on safety-critical health and lifestyle prompts.This is the first systematic review of language use in CA-based health communication. The results and limitations identified in the 11 included papers can give fresh insights into the design and development, popularization, and research of CA applications. This review can provide practical implications for incorporating positive language use into the design of health CAs and improving their effective language output in health communication. In this way, upgraded CAs will be more capable of handling various health problems particularly in the context of nationwide and even worldwide public health crises.
Quantifying safety risks of deep neural networks

Abstract Safety concerns on the deep neural networks (DNNs) have been raised when they are applied to critical sectors. In this paper, we define safety risks by requesting the alignment of network’s decision with human perception. To enable a general methodology for quantifying safety risks, we define a generic safety property and instantiate it to express various safety risks. For the quantification of risks, we take the maximum radius of safe norm balls, in which no safety risk exists. The computation of the maximum safe radius is reduced to the computation of their respective Lipschitz metrics—the quantities to be computed. In addition to the known adversarial example, reachability example, and invariant example, in this paper, we identify a new class of risk—uncertainty example—on which humans can tell easily, but the network is unsure. We develop an algorithm, inspired by derivative-free optimization techniques and accelerated by tensor-based parallelization on GPUs, to support an efficient computation of the metrics. We perform evaluations on several benchmark neural networks, including ACSC-Xu, MNIST, CIFAR-10, and ImageNet networks. The experiments show that our method can achieve competitive performance on safety quantification in terms of the tightness and the efficiency of computation. Importantly, as a generic approach, our method can work with a broad class of safety risks and without restrictions on the structure of neural networks.
Decentralized Consensus Blockchain and IPFS-Based Data Aggregation for Efficient Data Storage Scheme

By the development and advancement of blockchain technique, Internet of Things (IoT) proliferation driven devices and the application of blockchain-enabled IoT alter the view and operating infrastructure of the smart networks. The blockchain is responsible for supporting decentralized systems and offers secured means of authentication, management, and access to IoT system thereby deploying smart contracts offered by Ethereum. The increasing demand and the blockchain expansion generate huge volume of sensitive data. The growing demand and expansion of blockchain-IoT systems is generating large volume of sensitive data. Furthermore, distributed denial-of-service (DDoS) attacks are regarded as the most promising threats for smart contracts in the blockchain-based systems. Therefore, there is a need to detect and classify the attack type and the data should be stored in server more securely with the use of blockchain and data aggregation method. For this purpose, this presented technique aims at introducing decentralized consensus blockchain and Interplanetary file system (IPFS) based data aggregation for effective classification and data storage. The attack is detected using meta-hyperparameter random forest (MHP-RF) classifier. Once the attack is detected, the transaction information is stored in server securely by means of smart contract-based blockchain system. The transaction handling stage classifies the transaction type as normal or abnormal one which then followed by execution of business logic by smart contract thereby appending the transaction of blockchain in the network cloud. The consensus blockchain technique is employed with the use of PoW-enabled scheme integrated with Elgamal-based data aggregation. Therefore, the system security is improved and the intrusion is prevented greatly. The performance analysis of the system is analyzed in terms of accuracy, precision, recall, F-score, Encryption time, decryption time, execution time, and space complexity. The attained outcomes are compared with traditional approaches to prove the effectiveness of proposed strategy. The proposed system is said to be effective in time consumption, classifier performance, and in overcoming space complexity issues.
Device-cloud Collaborative Recommendation via Meta Controller

On-device machine learning enables the lightweight deployment of recommendation models in local clients, which reduces the burden of the cloud-based recommenders and simultaneously incorporates more real-time user features. Nevertheless, the cloud-based recommendation in the industry is still very important considering its powerful model capacity and the efficient candidate generation from the billion-scale item pool. Previous attempts to integrate the merits of both paradigms mainly resort to a sequential mechanism, which builds the on-device recommender on top of the cloud-based recommendation. However, such a design is inflexible when user interests dramatically change: the on-device model is stuck by the limited item cache while the cloud-based recommendation based on the large item pool do not respond without the new re-fresh feedback. To overcome this issue, we propose a meta controller to dynamically manage the collaboration between the on-device recommender and the cloud-based recommender, and introduce a novel efficient sample construction from the causal perspective to solve the dataset absence issue of meta controller. On the basis of the counterfactual samples and the extended training, extensive experiments in the industrial recommendation scenarios show the promise of meta controller in the device-cloud collaboration.
Map-Aided Navigation Algorithms Taking into Account the Variability of Position Errors of the Corrected Navigation System

Within the Bayesian approach, correcting the navigation system with the map data is considered with account for the variable system error presented as a sum of constant and varying components. Two algorithms are described based on Rao-Blackwellized particle filter. In the first algorithm, no simplifications are made in Rao-Blackwellization, and therefore the potential accuracy of the algorithm can be estimated, and the effect of the error varying component on the accuracy can be analyzed. In the second algorithm, linearization with respect to the error varying component is performed for simplification. An example illustrating the performance of the algorithms and the lowered computational burden of the second algorithm is presented.
Online deep hashing for both uni-modal and cross-modal retrieval

The batch-based hash learning paradigm for uni-modal or cross-modal retrieval has made great progress in recent decades. However, methods that are based on this paradigm cannot adapt to the scenario of incoming new data streams continuously; in addition, they are inefficient in terms of training time and memory cost in big data searches because they have to accumulate all the database data and new incoming data before training. Although a few online hash retrieval methods have been proposed to address these issues in recent years, they are based on shallow models, and none of them can perform uni-modal and cross-modal retrieval in one framework. To this end, we propose a novel method, namely, Online Deep Hashing for both Uni-modal and Cross-modal retrieval (ODHUC). For online deep hashing, ODHUC first trains image and text neural networks with image and text fundamental databases, and their hash codes are learned. When new data arrive, ODHUC samples images and texts from the newly arriving data and fundamental databases to update the image and text networks and learn the hash codes of the new data, the aim of which is to keep the similarity of the hash codes of the incremental and old data consistent with the supervised semantic similarity. ODHUC is also trained with sampled old data by knowledge distillation to learn new knowledge and avoid catastrophic forgetting of old knowledge. In this way, ODHUC achieves online deep hash learning continually without performance decline. For two categories of retrieval tasks, ODHUC learns image and text deep hash functions in two stages to align image and text features; it generates image hash codes first and then utilizes them to supervise the text hash code learning. Thus, ODHUC can perform both uni-modal and cross-modal retrieval tasks in one framework. Extensive experiments on three real-world datasets demonstrate that ODHUC outperforms several state-of-the-art methods.
Forensic investigation of instant messaging services on linux OS: Discord and Slack as case studies

Instant messaging applications have gained considerable market share over the past decade and have become one of the most used applications for users worldwide. However, due to the low-barrier to entry and ease of use, these applications (apps) have also garnered the attention of criminals wanting to use the apps to facilitate criminal activities. The memory forensic examination of Discord and Slack on Linux operating system (specifically, Ubuntu 20.04.3 LTS), two popular instant messaging apps, has gone largely unexplored. In this paper, we examined both apps and found data remnants of users' activities that are of forensic interest. We detected a variety of information including: Slack specific data, Discord specific data, username, emails, passwords, messages, conversations, and uploaded attachments, all of which could be utilized in a forensic investigation.
A probe-feature for specific emitter identification using axiom-based grad-CAM

Specific emitter identification (SEI) is the process of discriminating different emitters based on the radio fingerprints extracted from the received signals. Without complex parameter adjustment, radio fingerprints can be efficiently extracted by end-to-end deep learning models. However, deep learning models usually rely on a large number of training samples, which can not be provided in the SEI tasks due to data confidentiality. Besides, the mechanism of deep learning models still lacks clear interpretation, reducing the user's trust in the extracted radio fingerprints for the SEI. In this paper, we propose an interpretable SEI feature, based on the convolutional neural network (CNN), Probe-Feature, resorting to axiom-based Class Activation Mapping (XGrad-CAM). In general, the XGrad-CAM is firstly used as a probe into the CNN to discern important regions of the original features contributive to the current classification. Subsequently, those regions are fused into a discriminative feature with vivid extra-class differences. Experimental results demonstrate that the Probe-Feature can significantly improve the CNN's classification accuracy compared to the original features as inputs, especially when only scanty training samples are provided. In addition, Probe-Feature also sheds some light on interpreting CNN's classification with the SEI signals by two axioms, sensitivity, and conservation, used in XGrad-CAM.
Exploring natural language processing techniques to extract semantics from unstructured dataset which will aid in effective semantic interlinking

Designing efficacious semantics for the dynamic interaction and searches has proven to be concretely challenging because of the dynamically of the semantic searches, method of browsing and visualization interfaces for high volume information. This has a direct impact on enhancing the capabilities of the web. To surmount the challenges of providing meaning to high volume unstructured datasets, Natural language processing techniques and implements have been proven to be propitious, however, the reactivity of these techniques should be studied and predicated on the objective of providing meaning to the unstructured data. This paper demonstrates the working of five NLP techniques namely, bag-of-words, TF-IDF, NER, LSA, and LDA. The experiment provides the kindred attribute accomplishment or the identification of the meaning of this unstructured data varies from one technique to another. However, NLP techniques can be efficient as they provide insights into the data and make it human-readable. This will in turn avail in building better human–machine intractable browsing and applications.
GAN-based Face Reconstruction for Masked-Face

Facial recognition and identification which play an important role in human-computer interaction, secure authentication and criminal face recognition, are impeded by the advent of face masks due to COVID-19 pandemic. This is a challenging problem due to the following reasons: (i) masks cover quite a large part of the face even below the chin, (ii) it is not possible to collect and prepare a real paired-face images with and without mask object, (iii) face alterations and the presence of different masks is even more challenging. In this work, we propose a general framework that can be used to reconstruct the hidden part of face concealed by mask. We have employed GAN-based unpaired domain translation technique to translate masked face images from the source to the unmasked images in the destination domain. To this end, we also create a paired datasets of real face images and synthesized correspondence’s with face-masks and use it towards training of our proposed GAN-based facial reconstruction system which can be used for facial identification and secure authentication in human-computer interaction. The obtained results demonstrate that our model outperforms other representative state-of-the-art face completion approaches both qualitatively and quantitatively.
Museum Guidance in Sign Language: the SignGuide project

We present an overview of the SignGuide project. Its main goal is to develop a prototype interactive museum guide system for deaf visitors using mobile devices that will be able to receive visitors' questions in their native (sign language) with regard to the exhibits and to provide additional content also in sign language using an avatar or video, utilizing techniques from the field of computer vision and machine learning. The paper presents the basic ideas and technologies involved as well as some preliminary results.
Dual Attention-Based Industrial Surface Defect Detection with Consistency Loss

In industrial production, flaws and defects inevitably appear on surfaces, resulting in unqualified products. Therefore, surface defect detection plays a key role in ensuring industrial product quality and maintaining industrial production lines. However, surface defects on different products have different manifestations, so it is difficult to regard all defective products as being within one category that has common characteristics. Defective products are also often rare in industrial production, making it difficult to collect enough samples. Therefore, it is appropriate to view the surface defect detection problem as a semi-supervised anomaly detection problem. In this paper, we propose an anomaly detection method that is based on dual attention and consistency loss to accomplish the task of surface defect detection. At the reconstruction stage, we employed both channel attention and pixel attention so that the network could learn more robust normal image reconstruction, which could in turn help to separate images of defects from defect-free images. Moreover, we proposed a consistency loss function that could exploit the differences between the multiple modalities of the images to improve the performance of the anomaly detection. Our experimental results showed that the proposed method could achieve a superior performance compared to the existing anomaly detection-based methods using the Magnetic Tile and MVTec AD datasets.
A novel text analytic methodology for classification of product and service reviews

Classifying the sentiments of online reviews of products or services is important in that it provides the analysts with the ability to extract critical information which can be used to improve the corresponding product or service. The objective of this study is to classify the customer reviews (on a five-star and binary scale) that were collected for four different types of products/services. To achieve this goal, a novel classification framework is built by devising a unique classifier (composite variable), which includes rich information gathered by using all of the extracted features. The proposed framework is compared to commonly used Singular Value Decomposition (SVD) and chi-square-based feature selection (selected features, SF). These approaches are separately deployed in tree-based machine learning algorithms and Logistic Regression using a five-fold cross validation strategy. The results indicate that the proposed methodology outperforms the alternatives for each dataset employed.
A Generic Federated Recommendation Framework via Fake Marks and Secret Sharing

With the implementation of privacy protection laws such as GDPR, it is increasingly difficult for organizations to legally collect users’ data. However, a typical machine learning-based recommendation algorithm requires the data to learn users’ preferences. Some recent works thus turn to develop federated learning-based recommendation algorithms, but most of them either cannot protect the users’ privacy well, or sacrifice the model accuracy. In this article, we propose a lossless and generic federated recommendation framework via fake marks and secret sharing (FMSS). Our FMSS can not only protect the two types of users’ privacy, i.e., rating values and rating behaviors, without sacrificing the recommendation performance, but can also be applied to most recommendation algorithms for rating prediction, item ranking, and sequential recommendation. Specifically, we extend existing fake items to fake marks, and combine it with secret sharing to perturb the data uploaded by the clients to a server. We then apply our FMSS to six representative recommendation algorithms, i.e., MF-MPC and NeuMF for rating prediction, eALS and VAE-CF for item ranking, and Fossil and GRU4Rec for sequential recommendation. The experimental results demonstrate that our FMSS is a lossless and generic framework, which is able to federate a series of different recommendation algorithms in a lossless and privacy-aware manner.
A dual-branch neural network for DeepFake video detection by detecting spatial and temporal inconsistencies

It has become a research hotspot to detect whether a video is natural or DeepFake. However, almost all the existing works focus on detecting the inconsistency in either spatial or temporal. In this paper, a dual-branch (spatial branch and temporal branch) neural network is proposed to detect the inconsistency in both spatial and temporal for DeepFake video detection. The spatial branch aims at detecting spatial inconsistency by the effective EfficientNet model. The temporal branch focuses on temporal inconsistency detection by a new network model. The new temporal model considers optical flow as input, uses the EfficientNet to extract optical flow features, utilize the Bidirectional Long-Short Term Memory (Bi-LSTM) network to capture the temporal inconsistency of optical flow. Moreover, the optical flow frames are stacked before inputting into the EfficientNet. Finally, the softmax scores of two branches are combined with a binary-class linear SVM classifier. Experimental results on the compressed FaceForensics++ dataset and Celeb-DF dataset show that: (a) the proposed dual-branch network model performs better than some recent spatial and temporal models for the Celeb-DF dataset and all the four manipulation methods in FaceForensics++ dataset since these two branches can complement each other; (b) the use of optical flow inputs, Bi-LSTM and dual-branches can greatly improve the detection performance by the ablation experiments.