Reviewerly/exHarmony huggingface dataset:
Detecting Adversarial Attacks on Audiovisual Speech Recognition

Adversarial attacks pose a threat to deep learning models. However, research on adversarial detection methods, especially in the multi-modal domain, is very limited. In this work, we propose an efficient and straightforward detection method based on the temporal correlation between audio and video streams. The main idea is that the correlation between audio and video in adversarial examples will be lower than benign examples due to added adversarial noise. We use the synchronisation confidence score as a proxy for audiovisual correlation and based on it we can detect adversarial attacks. To the best of our knowledge, this is the first work on detection of adversarial attacks on audiovisual speech recognition models. We apply recent adversarial attacks on two audiovisual speech recognition models trained on the GRID and LRW datasets. The experimental results demonstrate that the proposed approach is an effective way for detecting such attacks.
Capturing Multi-Resolution Context by Dilated Self-Attention

Self-attention has become an important and widely used neural network component that helped to establish new state-of-the-art results for various applications, such as machine translation and automatic speech recognition (ASR). However, the computational complexity of self-attention grows quadratically with the input sequence length. This can be particularly problematic for applications such as ASR, where an input sequence generated from an utterance can be relatively long. In this work, we propose a combination of restricted self-attention and a dilation mechanism, which we refer to as dilated self-attention. The restricted self-attention allows attention to neighboring frames of the query at a high resolution, and the dilation mechanism summarizes distant information to allow attending to it with a lower resolution. Different methods for summarizing distant frames are studied, such as subsampling, mean-pooling, and attention-based pooling. ASR results demonstrate substantial improvements compared to restricted self-attention alone, achieving similar results compared to full-sequence based self-attention with a fraction of the computational costs.
Generative Latent Implicit Conditional Optimization when Learning from Small Sample

We revisit the long-standing problem of learning from small sample, to which end we propose a novel method called GLICO (Generative Latent Implicit Conditional Optimization). GLICO learns a mapping from the training examples to a latent space, and a generator that generates images from vectors in the latent space. Unlike most recent works, which rely on access to large amounts of unlabeled data, GLICO does not require access to any additional data other than the small set of labeled points. In fact, GLICO learns to synthesize completely new samples for every class using as little as 5 or 10 examples per class, with as few as 10 such classes without imposing any prior. GLICO is then used to augment the small training set while training a classifier on the small sample. To this end our proposed method samples the learned latent space using spherical interpolation, and generates new examples using the trained generator. Empirical results show that the new sampled set is diverse enough, leading to improvement in image classification in comparison with the state of the art, when trained on small samples obtained from CIFAR-10, CIFAR-100, and CUB-200.
Towards Robust Training of Multi-Sensor Data Fusion Network Against Adversarial Examples in Semantic Segmentation

The success of multi-sensor data fusions in deep learning appears to be attributed to the use of complementary information among multiple sensor datasets. Compared to their predictive performance, relatively less attention has been devoted to the adversarial robustness of multi-sensor data fusion models. To achieve adversarial robust multi-sensor data fusion networks, we propose here a novel robust training scheme called Multi-Sensor Cumulative Learning (MSCL). The motivation behind the MSCL method is based on the way human beings learn new skills. The MSCL allows the multi-sensor fusion network to learn robust features from individual sensors, and then learn complex joint features from multiple sensors just as people learn to walk before they run. The step wise framework of MSCL enables the network to incorporate pre-trained knowledge of robustness with new joint information from multiple sensors. Extensive experimental evidence validated that the MSCL outperforms other multi-sensor fusion training in defending against adversarial examples.
Ancient Document Layout Analysis: Autoencoders meet Sparse Coding

Layout analysis of historical handwritten documents is a key pre-processing step in document image analysis that, by segmenting the image into its homogeneous regions, facilitates subsequent procedures such as optical character recognition and automatic transcription. Learning-based approaches have shown promising performances in layout analysis, however, the majority of them requires tedious pixel-wise labelled training data to achieve generalisation capabilities, this limitation preventing their application due to the lack of large labelled datasets. This paper proposes a novel unsupervised representation learning method for documents' layout analysis that reduces the need for labelled data: a sparse autoencoder is first trained in an unsupervised manner on a historical text document's image; representation of image patches, computed by the sparse encoder, is then used to classify pixels into various region categories of the document using a feed-forward neural network. A new training method, inspired by the ISTA algorithm, is also introduced here to train the sparse encoder. Experimental results on DIVA - HisDB dataset demonstrate that the proposed method outperforms previous approaches based on unsupervised representation learning while achieving performances comparable to the state-of-the-art fully supervised methods.
An efficient GPU-accelerated inference engine for binary neural network on mobile phones

Over the last years, deep neural networks (DNNs) are becoming more powerful and have risen in popularity, especially in mobile computing. Applications running on edge AI devices such as smartphones would potentially benefit from the new opportunities enabled by deep learning techniques. However, DNNs are by nature computationally and memory intensive, making them challenging to deploy on mobile devices. Binary neural networks (BNNs) have been considered as a promising solution that can significantly reduce the memory and computational requirements of DNNs while still offering similar capabilities of full precision DNN models. Currently, existing GPU-accelerated implementations of BNNs are only tailored for desktop platforms. Due to architecture differences, mere porting of such implementations to mobile devices yields suboptimal performance or is impossible in some cases. Therefore, there has still been a missing piece in the literature for GPU-accelerated implementations of BNNs on mobile devices. In this paper, we propose PhoneBit, a GPU-accelerated BNN inference engine for mobile devices that fully exploits the computing power of BNNs on mobile GPUs. PhoneBit provides a set of operator-level optimizations including locality-friendly data layout, bit packing with vectorization and layers integration for efficient binary convolution. We also provide a detailed implementation and parallelization optimization for PhoneBit to optimally utilize the memory bandwidth and computing power of mobile GPUs. Our experiment results show that PhoneBit can achieve significant speedup and energy efficiency compared with state-of-the-art frameworks for mobile devices. The PhoneBit open source library is available for download at https://code.ihub.org.cn/projects/915/repository/PhoneBit.
Image-Assisted Transformer in Zero-Resource Multi-Modal Translation

Humans learn language speaking and translation with the help of common knowledge of the external world, while standard machine translation depends only on parallel corpora. There-fore, zero-resource translation has been proposed to explore a way to make models learn to translate without any parallel corpora but with external knowledge. Current models in the field usually combine an image encoder with a textual decoder, which leads to extra efforts to make use of the original model in machine translation. On the other hand, Transformers have achieved great progress in neural language processing while they are rarely utilized in the field of zero-resource translation with image pivot. In this paper, we investigate how to use visual information as an auxiliary hint for a Transformer-based system in a zero-resource translation scenario. Our model achieves state-of-the-art BLEU scores in the field of zero-resource machine translation with image pivot.
Risk propagation on electric power supply chain networks based on Blockchain technology

With increasingly fierce competition among electric power enterprises, the complexity of the supply chain has increased significantly. The risk of the electric power supply chain has brought a huge impact on of our modern economic and social operation. The emergence of Blockchain technology is helpful to inhibit the spread of the risk. Based on SIR model, combined with Blockchain technology, this paper proposes a new paradigm of risk propagation in electric power supply chain network. The scale of risk propagation in electric power supply chain network is obtained by numerical simulation. The simulation results show that the ratio of trust enterprises will affect the risk diffusion of electric power supply chain.
Steganalysis of H.264/AVC Videos Exploiting Subtractive Prediction Error Blocks

To cope with the abuse of steganography using H.264 videos, i.e., the dominant video format, as the carrier, this paper presents a steganalytic method which works well even in the scenario where both the training data and the prior knowledge of the test data are limited. As a key feature of H.264, intra prediction is incorporated to remove redundancies within one single frame by predicting the current block using previously coded blocks. Unlike in JPEG domain, the quantized discrete cosine transform (QDCT) coefficients in H.264 videos come from the prediction error (residual) blocks (PEBs) instead of the original pixel block, hence we suggest shifting the focal point from the spatial domain to the prediction error domain, i.e., the PEB domain. According to the traits of video coding, 3 types of subtractive PEB (SPEB) are defined to capture the inconsistency between correlated PEBs, and the differences between correlated SPEBs are modeled by first-order Markov chain. Then the so-called SUPERB (SUbtractive Prediction ERror Block) features are engineered by subsets of sample transition probability matrices for a steganalyzer. What's more, the features derived from IPM (Intra Prediction Mode) transition probabilities are also merged into SUPERB to improve detection ability. Extensive experiments are carried out from different aspects. Performance results demonstrate the effectiveness of SUPERB, particularly its essence of general applicability when the training and test data are of quite different attributes, which is more favorable for real-world applications.
Mimicking Complexity of Structured Data Matrix’s Information Content: Categorical Exploratory Data Analysis

We develop Categorical Exploratory Data Analysis (CEDA) with mimicking to explore and exhibit the complexity of information content that is contained within any data matrix: categorical, discrete, or continuous. Such complexity is shown through visible and explainable serial multiscale structural dependency with heterogeneity. CEDA is developed upon all features’ categorical nature via histogram and it is guided by all features’ associative patterns (order-2 dependence) in a mutual conditional entropy matrix. Higher-order structural dependency of k(≥3) features is exhibited through block patterns within heatmaps that are constructed by permuting contingency-kD-lattices of counts. By growing k, the resultant heatmap series contains global and large scales of structural dependency that constitute the data matrix’s information content. When involving continuous features, the principal component analysis (PCA) extracts fine-scale information content from each block in the final heatmap. Our mimicking protocol coherently simulates this heatmap series by preserving global-to-fine scales structural dependency. Upon every step of mimicking process, each accepted simulated heatmap is subject to constraints with respect to all of the reliable observed categorical patterns. For reliability and robustness in sciences, CEDA with mimicking enhances data visualization by revealing deterministic and stochastic structures within each scale-specific structural dependency. For inferences in Machine Learning (ML) and Statistics, it clarifies, upon which scales, which covariate feature-groups have major-vs.-minor predictive powers on response features. For the social justice of Artificial Intelligence (AI) products, it checks whether a data matrix incompletely prescribes the targeted system.
Edge-Aware Multi-Scale Progressive Colorization

Image colorization recovers a colorful image from a grayscale one. Trained by large-scale datasets, recent deep neural networks based methods can produce impressive colorful images. However, they usually directly train a single network using training images of fixed resolution. It is hard for such a single network to learn the features of different scales for colorization. Moreover, they are prone to generate color bleedings and blurry details around objects boundaries. To address these problems, we propose a novel edge-aware multi-scale progressive network (EMSPN). The key idea is to train a series of multi-scale networks in a progressive manner, so that the network in finer scales can leverage the outputs of its previous scale. In addition, we also propose an edge-map loss to effectively prevent bleedings and blurs around the image edges. Experimental results show that our work outperforms existing methods and achieves state-of-the-art results.
TSDLPP: A Novel Two-Stage Deep Learning Framework For Prognosis Prediction Based on Whole Slide Histopathological Images

Recently, digital pathology image-based prognosis prediction has become a hot topic in healthcare research to make early decisions on therapy and improve the treatment quality of patients. Therefore, there has been a recent surge of interest in designing deep learning method solving the problem of prognosis prediction with digital pathology images. However, whole slide histopathological images (WSIs) based prognosis prediction is still a challenge due to the large size of pathological images, the heterogeneity of tumors and the high cost of region of interests (ROIs) labeling. In this study, we design a novel two-stage deep learning framework for prognosis prediction (TSDLPP) based on WSIs. Our proposed framework consists of two-stage paradigms: 1) training tissue decomposition network (TDNet) to divide WSIs into cancerous and non-cancerous regions, 2) integrating general prognosis-related densely connected CNN (GPR-DCCNN) and morphology-specific prognosis-related densely connected CNNs (MSPR-DCCNNs) to extract different level features of pathological images. In the end, we apply TSDLPP to the prognosis prediction of breast cancer using The Cancer Genome Atlas (TCGA) datasets. Experiment results demonstrate that TSDLPP obtains superior performance of prognosis prediction compared with the existing state-of-arts methods.
Neural Network based Real Time Sign Language Interpreter for Virtual Meet

Due to the Covid-19 pandemic, people were forced to stay home, and most professions switched to Work From Home mode. The world switched to Video Conferencing to stay connected from remote environment. This made Translators unavailable to the hearing impaired. This makes it challenging for deaf-mute people to communicate with other people since there are no credible tools present to translate it real-time in these applications. So, a Translator built within these video conferencing applications would be helpful in communication for these people. Neural Network algorithm is used in our model to predict the signs and translate them. This model is implemented in a video conferencing application which will make the use of the Sign Gesture Translation feature and the other person using the application will receive the translation in text on a real time basis.
Evaluating Deep Neural Network Ensembles by Majority Voting Cum Meta-Learning Scheme

Deep neural networks (DNNs) are prone to overfitting and hence have high variance. Overfitted networks do not perform well for a new data instance. So instead of using a single DNN as classifier, we propose an ensemble of seven independent DNN learners by varying only the input to these DNNs keeping their architecture and intrinsic properties same. To induce variety in the training input, for each of the seven DNNs, one-seventh of the data is deleted and replenished by bootstrap sampling from the remaining samples. We have proposed a novel technique for combining the prediction of the DNN learners in the ensemble. Our method is called pre-filtering by majority voting coupled with stacked meta-learner which performs a two-step confidence check for the predictions before assigning the final class labels. All the algorithms in this paper have been tested on five benchmark datasets, namely human activity recognition (HAR), gas sensor array drift, Isolet, Spambase, and Internet advertisements. Our ensemble approach achieves higher accuracy than a single DNN and the average individual accuracies of DNNs in the ensemble, as well as the baseline approaches of plurality voting and meta-learning.
Profiled Power-Analysis Attacks by an Efficient Architectural Extension of a CNN Implementation

In a recent line of works, several masking and unmasking AES design have been proposed to secure hardware implementations against power-analysis techniques. Although Machine-learning profiling techniques have been successful in security testing during the last years, evaluation of hardware security still requires improvement because of the growing complexity of leakage models against profiled side-channel attacks. In this paper, we propose an improved profiling method to exploit the power consumption of complex cryptographic functions based on Deep-Learning. In order to learn the 256-class Deep neural network of an AES-128, we build successful Convolutional Neural Networks to break its implementation. It has been shown by our experiments that our model achieved a success rate of ≥ 99% even with a single trace using Keras library with Tensorflow. For the sake of completeness, we investigate the correct ”key rank” according to the number of traces and as a further performance measure, we use ”recall” metric when attacking the third AES SBox. Our model reaches the key rank ≤ 10 with the recall metric ≥ 0.99.
Algorithms, Data, and Platforms: The Diverse Challenges of Governing AI

Artificial Intelligence (AI) poses interwoven challenges. Defined as technology that uses advanced computation to perform at human cognitive capacity in some task area, AI must be regulated in the context of its broader toolbox - algorithms, data and platforms - and its regulation must be sector-specific. Establishing national and community priorities on how to reap AIâ€™s benefits, while managing its social and economic risks, is an evolving debate. Digital Platform Firms are a fundamental driver of AI tools: they dominate the playing field and often pursue priorities outside the frames of the public sector and of civil society. While its governance is critical to national success, AI pries open a Pandora's box of questions that sweep across the economy and society, engaging diverse communities. Rather than a single, global ethical framework, one must consider how to pursue objectives of interoperability amongst nations with quite different political economies.
Automatic Recognition of Parathyroid Nodules in Ultrasound Images Based on Fused Prior Pathological Knowledge Features

Automation diagnosis of parathyroid nodules is of crucial importance to recognize parathyroid nodules in ultrasound images. Aiming at the different nodule shapes of diverse patients, blurred boundaries, complex backgrounds and inhomogeneous intensity of ultrasound images, we propose a novel hybrid level set model to accurately segment nodules. The adaptive global term weight is determined based on the image local entropy of the region around the evolution contour and two scales are proposed for the local term to drive the evolution contour fast approaching to the boundary in order to avoid large amount of calculation and over-segmentation. We also propose membrane features and relative position features based on prior pathological knowledge to describe the inherent characteristics of parathyroid nodules different from thyroid and other nodules. We fused prior pathological knowledge features, morphology features and texture features of the segmented nodules to recognize parathyroid nodules by the support vector data description(SVDD). The experiment result indicates that the incorporation of the proposed hybrid level set segmentation method and the fused prior pathological knowledge features, morphology features and texture features improve the recognition accuracy and efficiency of parathyroid nodules, which is much higher than that only with morphology and texture features.
CopyCat: Using Sign Language Recognition to Help Deaf Children Acquire Language Skills

Deaf children born to hearing parents lack continuous access to language, leading to weaker working memory compared to hearing children and deaf children born to Deaf parents. CopyCat is a game where children communicate with the computer via American Sign Language (ASL), and it has been shown to improve language skills and working memory. Previously, CopyCat depended on unscalable hardware such as custom gloves for sign verification, but modern 4K cameras and pose estimators present new opportunities. Before re-creating the CopyCat game for deaf children using off-the-shelf hardware, we evaluate whether current ASL recognition is sufficient. Using Hidden Markov Models (HMMs), user independent word accuracies were 90.6%, 90.5%, and 90.4% for AlphaPose, Kinect, and MediaPipe, respectively. Transformers, a state-of-the-art model in natural language processing, performed 17.0% worse on average. Given these results, we believe our current HMM-based recognizer can be successfully adapted to verify children’s signing while playing CopyCat.
GAN-based Thermal Infrared Image Colorization for Enhancing Object Identification

Colorizing a thermal infrared image is considered as a challenging task in computer vision and particularly in object recognition. The solution for thermal infrared colorization is quite problematic that not only transforming a grayscale to RGB image where the chrominance is predicted from the given luminance, but also the luminance should be estimated as well. In this paper, we propose a GAN-based method to encourage the similarity and well preserve details of images. Also, we deploy a multi-term objective function including content, perceptual, total variation, relativistic GAN losses and DSSIM to enhance the generated RGB images. Quantitative and qualitative experiments on a large available dataset show that our method obtains impressive results and outperforms compared to others existing colorized methods.