Reviewerly/exHarmony huggingface dataset:
A Deep Multi-Modal Explanation Model for Zero-Shot Learning

Zero-shot learning (ZSL) has attracted significant attention due to its capabilities of classifying new images from unseen classes. To perform the classification task for ZSL, learning visual and semantic embeddings has been the main research approach in existing literature. At the same time, generating complementary explanations to justify the classification decision has remained largely unexplored. In this paper, we propose to address a new and challenging task, namely explainable zero-shot learning (XZSL), which aims to generate visual and textual explanations to support the classification decision. To accomplish this task, we build a novel Deep Multi-modal Explanation (DME) model that incorporates a joint visual-attribute embedding module and a multi-channel explanation module in an end-to-end fashion. In contrast to existing ZSL approaches, our visual-attribute embedding is associated not only with the decision, but also with new visual and textual explanations. For visual explanations, we first capture several attribute activation maps (AAM) and then merge them into a class activation map (CAM) that visually infers which region of an image is relevant to the class. Textual explanations are generated from the multi-channel explanation module, jointly integrating three long short-term memory models (LSTMs) each of which is conditioned on a different feature representation. Additionally, we suggest that the DME model can retain explanatory consistency for similar instances and explanatory diversity for diverse instances. We conduct qualitative and quantitative experiments to assess the model for ZSL classification and explanation. Specifically, the ablation studies verify the effectiveness of the components in our model. Our results on three well-known datasets are competitive with prior approaches. More importantly, the joint training of our embedding and explanation modules demonstrates mutual performance improvements between ZSL classification and explanation. We shed more light on DME to analyze and diagnose its advantages and limitations.
Multimedia processing using deep learning technologies, high‐performance computing cloud resources, and Big Data volumes

Summary The last few years have been marked by the presence of very large sets of images and videos in our everyday lives. These multimedia objects have a very fast frequency of creation and sharing since images and videos can come from different devices such as smartphones, satellites, cameras, or drones. They are generally used to illustrate objects in different situations (public areas, train stations, hospitals, political and sport events and competitions, etc). As consequence, image and video processing algorithms have got increasing importance for several computer vision applications that should be adapted for managing large‐scale volumes and exploiting high performance computing resources (local or cloud). In this work, we propose a cloud‐based toolbox (platform) for computer vision applications. This platform integrates a toolbox of image and video processing algorithms that can (i) exploit high performance computing cloud resources, (ii) execute applications in real time, and (iii) manage large‐scale database using Big Data technologies. The related libraries and hardware drivers are automatically integrated and configured in order to offer to users an access to the different applications without the need to download, install, and configure software or hardware. Experiments were conducted using three kinds of applications: (i) image and video processing applications, (ii) deep learning techniques for images classification and multiobject localization, and (iii) images indexation and retrieval. These experiments demonstrated the interest of our platform for sharing, in an efficient way, our scientific contributions and annotated databases in order to improve the quality and performance of computer vision applications.
User and item profile expansion for dealing with cold start problem

Recommender Systems (RS) are expected to suggest the accurate goods to the consumers. Cold start is the most important challenge for RSs. Recent hybrid RSs combine ConF and ColF. We introduce an ontological hybrid RS where the ontology has been employed in its ConF part while improving the ontology structure by its ColF part. In this paper, a new hybrid approach is proposed based on the combination of demographic similarity and cosine similarity between users in order to solve the cold start problem of new user type. Also, a new approach is proposed based on the combination of ontological similarity and cosine similarity between items in order to solve the cold start problem of new item type. The main idea of the proposed method is to expand user/item profiles based on different strategies to build higher-performing profiles for users/items. The proposed method has been evaluated on a real dataset and the experimentations indicate the proposed method has the better performance comparing with the state of the art RS methods, especially in the case of the cold start.
Visual question answering with attention transfer and a cross-modal gating mechanism

Visual question answering (VQA) is challenging since it requires to understand both language information and corresponding visual contents. A lot of efforts have been made to capture single-step language and visual interactions. However, answering complex questions requires multiple steps of reasoning which gradually adjusts the region of interest to the most relevant part of the given image, which has not been well investigated. To integrate question related object relations into attention mechanism, we propose a multi-step attention architecture to facilitate the modeling of multi-modal correlations. Firstly, an attention transfer mechanism is integrated to gradually adjust the region of interest considering reasoning representation of questions. Secondly, we propose a cross-modal gating strategy to filter out irrelevant information based on multi-modal correlations. Finally, we achieve the state-of-the-art performance on the VQA 1.0 dataset and favorable results on the VQA 2.0 dataset, which verifies the effectiveness of our proposed method.
Learning Deep Spatiotemporal Feature for Engagement Recognition of Online Courses

This paper focuses on the study of engagement recognition of online courses from students' appearance and behavioral information using deep learning methods. Automatic engagement recognition can be applied to developing effective online instructional and assessment strategies for promoting learning. In this paper, we make two contributions. First, we propose a Convolutional 3D (C3D) neural networks-based approach to automatic engagement recognition, which models both the appearance and motion information in videos and recognize student engagement automatically. Second, we introduce the Focal Loss to address the class-imbalanced data distribution problem in engagement recognition by adaptively decreasing the weight of high engagement samples while increasing the weight of low engagement samples in deep spatiotemporal feature learning. Experiments on the DAiSEE dataset show the effectiveness of our method in comparison with the state-of-the-art automatic engagement recognition methods.
Home-Based Physical Therapy with an Interactive Computer Vision System

In this paper, we present ExerciseCheck. ExerciseCheck is an interactive computer vision system that is sufficiently modular to work with different sources of human pose estimates, i.e., estimates from deep or traditional models that interpret RGB or RGB-D camera input. In a pilot study, we first compare the pose estimates produced by four deep models based on RGB input with those of the MS Kinect based on RGB-D data. The results indicate a performance gap that required us to choose the MS Kinect when we tested ExerciseCheck with Parkinson's disease patients in their homes. ExerciseCheck is capable of customizing exercises, capturing exercise information, evaluating patient performance, providing therapeutic feedback to the patient and the therapist, checking the progress of the user over the course of the physical therapy, and supporting the patient throughout this period. We conclude that ExerciseCheck is a user-friendly computer vision application that can assist patients by providing motivation and guidance to en-sure correct execution of the required exercises. Our re-sults also suggest that while there has been considerable progress in the field of pose estimation using deep learning, current deep learning models are not fully ready to replace RGB-D sensors, especially when the exercises involved are complex, and the patient population being accounted for has to be carefully tracked for its "active range of motion".
Neural Network Compression Framework for Fast Model Inference

We present a new PyTorch-based framework for neural network compression with fine-tuning named Neural Network Compression Framework (NNCF) (https://github.com/openvinotoolkit/nncf) . It leverages recent advances of various network compression methods and implements some of them, namely quantization, sparsity, filter pruning and binarization. These methods allow producing more hardware-friendly models that can be efficiently run on general-purpose hardware computation units (CPU, GPU) or specialized deep learning accelerators. We show that the implemented methods and their combinations can be successfully applied to a wide range of architectures and tasks to accelerate inference while preserving the original model's accuracy. The framework can be used in conjunction with the supplied training samples or as a standalone package that can be seamlessly integrated into the existing training code with minimal adaptations.
Are the optimum angles of photovoltaic systems so important?

Abstract The traditional point of view suggests photovoltaic systems to be installed at the optimum angles. However, various reasons require more flexible photovoltaic installations, especially for building-integrated photovoltaics. Some site-specific cases based on local irradiance data show that small deviations from the optimum will not cause big energy losses, while more theoretical insights on the flexible installation problem are required to achieve higher photovoltaic penetration. This paper tries to demonstrate that quite a large range of photovoltaic installation angles causes ignorable energy losses annually. Furthermore, this paper developed and validated a universal and convenient Loss Bound Equation to determine the 5% Loss Bound worldwide with high accuracy. The 5% Loss Bound is defined as a group of tilt and azimuth angle combinations where the plane-of-array energy losses are limited within 5% of the annual maximum plane-of-array irradiation. The universal Loss Bound Equation is independent of local irradiance data, which is totally different from conventional irradiance-based approaches and it is significantly faster and more convenient to use. The 5% Loss Bound will provide great flexibility for planning array spacing, decreasing dust deposition, sustainable retrofits to existing buildings, design for building-integrated photovoltaics, and so on.
Multi-task Information Bottleneck Co-clustering for Unsupervised Cross-view Human Action Categorization

The widespread adoption of low-cost cameras generates massive amounts of videos recorded from different viewpoints every day. To cope with this vast amount of unlabeled and heterogeneous data, a new multi-task information bottleneck co-clustering (MIBC) approach is proposed to automatically categorize human actions in collections of unlabeled cross-view videos. Our motivation is that, if a learning action category from each view is seen as a single task, it is reasonable to assume that the tasks of learning action patterns from the videos recorded by multiple cameras are dependent and inter-related, since the actions of the same subjects synchronously recorded from different camera viewpoints are complementary to each other. MIBC aims to transfer the shared view knowledge across multiple tasks (i.e., camera viewpoints) to boost the performance of each task. Specifically, MIBC involves the following two parts: (1) extracting action categories for each task by independently maintaining its own relevant information, and (2) allowing the feature representations of all tasks to be compressed into a common feature space, which is utilized to capture the relatedness of multiple tasks and transfer the shared knowledge across different camera viewpoints. These two parts of MIBC work simultaneously and can be solved in a novel co-clustering mechanism. Our experimental evaluation on several cross-view action collections shows that the MIBC algorithm outperforms the existing state-of-the-art baselines.
Deep Neural Models and Retrofitting for Arabic Text Categorization

Arabic text categorization is an important task in text mining particularly with the fast-increasing quantity of the Arabic online data. Deep neural network models have shown promising performance and indicated great data modeling capacities in managing large and substantial datasets. This article investigates convolution neural networks (CNNs), long short-term memory (LSTM) and their combination for Arabic text categorization. This work additionally handles the morphological variety of Arabic words by exploring the word embeddings model using position weights and subword information. To guarantee the nearest vector representations for connected words, this article adopts a strategy for refining Arabic vector space representations using semantic information embedded in lexical resources. Several experiments utilizing different architectures have been conducted on the OSAC dataset. The obtained results show the effectiveness of CNN-LSTM without and with retrofitting for Arabic text categorization in comparison with major competing methods.
Multi-Modal Human Action Recognition With Sub-Action Exploiting and Class-Privacy Preserved Collaborative Representation Learning

Multimodal human action recognition with depth sensors has drawn wide attention, due to its potential applications such as health-care monitoring, smart buildings/home, intelligent transportation, and security surveillance. As one of the obstacles of robust action recognition, sub-actions sharing, especially among similar action categories, makes human action recognition more challenging. This paper proposes a segmental architecture to exploit the relations of sub-actions, jointly with heterogeneous information fusion and Class-privacy Preserved Collaborative Representation (CPPCR) for multi-modal human action recognition. Specifically, a segmental architecture is proposed based on the normalized action motion energy. It models long-range temporal structure over video sequences to better distinguish the similar actions bearing sub-action sharing phenomenon. The sub-action based depth motion and skeleton features are then extracted and fused. Moreover, by introducing within-class local consistency into Collaborative Representation (CR) coding, CPPCR is proposed to address the challenging sub-action sharing phenomenon, learning the high-level discriminative representation. Experiments on four datasets demonstrate the effectiveness of the proposed method.
Privacy-preserving human action recognition as a remote cloud service using RGB-D sensors and deep CNN

Cloud-based expert systems are highly emerging nowadays. However, the data owners and cloud service providers are not in the same trusted domain in practice. For the sake of data privacy, sensitive data usually has to be encrypted before outsourcing which makes effective cloud utilization a challenging task. Taking this concern into account, we propose a novel cloud-based approach to securely recognize human activities. A few schemes exist in the literature for secure recognition. However, they suffer from the problem of constrained data and are vulnerable to re-identification attack, where advanced deep learning models are used to predict an object’s identity. We address these problems by considering color and depth data, and securing them using position based superpixel transformation. The proposed transformation is designed by actively involving additional noise while resizing the underlying image. Due to this, a higher degree of obfuscation is achieved. Further, in spite of securing the complete video, we secure only four images, that is, one motion history image and three depth motion maps which are highly saving the data overhead. The recognition is performed using a four stream deep Convolutional Neural Network (CNN), where each stream is based on pre-trained MobileNet architecture. Experimental results show that the proposed approach is the best suitable candidate in “security-recognition accuracy (%)” trade-off relation among other image obfuscation as well as state-of-the-art schemes. Moreover, a number of security tests and analyses demonstrate robustness of the proposed approach.
Deep Learning and Thresholding with Class-Imbalanced Big Data

Class imbalance is a regularly occurring problem in machine learning that has been studied extensively over the last two decades. Various methods for addressing class imbalance have been introduced, including algorithm-level methods, datalevel methods, and hybrid methods. While these methods are well studied using traditional machine learning algorithms, there are relatively few studies that explore their application to deep neural networks. Thresholding, in particular, is rarely discussed in the deep learning with class imbalance literature. This paper addresses this gap by conducting a systematic study on the application of thresholding with deep neural networks using a Big Data Medicare fraud data set. We use random oversampling (ROS), random under-sampling (RUS), and a hybrid ROS-RUS to create 15 training distributions with varying levels of class imbalance. With the fraudulent class size ranging from 0.03%-60%, we identify optimal classification thresholds for each distribution on random validation sets and then score the thresholds on a 20% holdout test set. Through repetition and statistical analysis, confidence intervals show that the default threshold is never optimal when training data is imbalanced. Results also show that the optimal threshold outperforms the default threshold in nearly all cases, and linear models indicate a strong linear relationship between the minority class size and the optimal decision threshold. To the best of our knowledge, this is the first study to provide statistical results that describe optimal classification thresholds for deep neural networks over a range of class distributions.
An Empirical Study on the Use and Misuse of Java 8 Streams

Streaming APIs allow for big data processing of native data structures by providing MapReduce-like operations over these structures. However, unlike traditional big data systems, these data structures typically reside in shared memory accessed by multiple cores. Although popular, this emerging hybrid paradigm opens the door to possibly detrimental behavior, such as thread contention and bugs related to non-execution and non-determinism. This study explores the use and misuse of a popular streaming API, namely, Java 8 Streams. The focus is on how developers decide whether or not to run these operations sequentially or in parallel and bugs both specific and tangential to this paradigm. Our study involved analyzing 34 Java projects and 5:53 million lines of code, along with 719 manually examined code patches. Various automated, including interprocedural static analysis, and manual methodologies were employed. The results indicate that streams are pervasive, parallelization is not widely used, and performance is a crosscutting concern that accounted for the majority of fixes. We also present coincidences that both confirm and contradict the results of related studies. The study advances our understanding of streams, as well as benefits practitioners, programming language and API designers, tool developers, and educators alike.
Kannada Handwritten Script Recognition using Machine Learning Techniques

Many researchers have contributed to automate the optical character recognition. But handwritten character recognition is still an uncompleted task. In this paper we are proposing two techniques to recognize handwritten Kannada script, which yields high accuracy compared to previous works. There are lot of challenges in recognizing handwritten Kannada scripts. Few of the challenges include: each person have their own handwriting, there is no uniform spacing between alphabets, words and lines. Another main problem when it comes to Kannada language is that there is no large dataset available to train the recognition system, and it is challenging to write all combinations of each alphabet in Kannada script. In the proposed work, we have gathered the handwritten training set from the Web and from the students of our campus and segmented each letter. We have proposed two methods to recognize the handwritten Kannada characters. The first techniques is by Tesseract tool, and second is by using Convolution Neural Network (CNN). With Tesseract tool we have achieved 86% accuracy and through Convolution Neural Network we achieved87% accuracy although it might improve with the data set chosen and further enhanced image processing. The main idea behind this work is to extract text from the scanned images, identify the Kannada letters in it accurately and display or store it for further usage.
Anomaly Monitoring Framework in Lane Detection With a Generative Adversarial Network

The safety of an automated vehicle requires accurate information of surrounding conditions, because a false sensor output can lead to a fatal accident during driving. Thus, monitoring of abnormalities in every sensor is important for robust perception of the environment. Since it is difficult to obtain anomalous data, it is hard to develop a robust detection algorithm using only a relatively small number of anomalies. In this paper, we propose a data augmentation method for oversampling minority anomalies in lane detection. Using a generative adversarial network that makes the generator learn to estimate the distribution of anomalous data, it generates synthesized minority anomalies. The generated anomalies are used to train an anomaly detection network while minimizing latency for use in real situations. During training, the generated anomalies, with various mixed quality, are sampled differently according to their quality. This helps the detection network to be optimized with better quality data. Experimental result shows that when using the proposed anomaly detection framework for monitoring lane abnormality, it improves the performance by 12% when compared to the vanilla recurrent neural network.