Reviewerly/exHarmony huggingface dataset:
VQA-LOL: Visual Question Answering Under the Lens of Logic

Logical connectives and their implications on the meaning of a natural language sentence are a fundamental aspect of understanding. In this paper, we investigate whether visual question answering (VQA) systems trained to answer a question about an image, are able to answer the logical composition of multiple such questions. When put under this Lens of Logic, state-of-the-art VQA models have difficulty in correctly answering these logically composed questions. We construct an augmentation of the VQA dataset as a benchmark, with questions containing logical compositions and linguistic transformations (negation, disjunction, conjunction, and antonyms). We propose our Lens of Logic (LOL) model which uses question-attention and logic-attention to understand logical connectives in the question, and a novel Fréchet-Compatibility Loss, which ensures that the answers of the component questions and the composed question are consistent with the inferred logical operation. Our model shows substantial improvement in learning logical compositions while retaining performance on VQA. We suggest this work as a move towards robustness by embedding logical connectives in visual understanding.
Digital Watermarking: State of The Art and Research Challenges in Health Care &amp; Multimedia Applications

Abstract Nowadays the multimedia data easily available to most people. It is the main cause of illegal access to multimedia content, theft of the intellectual property, easily copying and manipulate the data over the internet, and spreading fake news. With the increase in the availability of the internet to a common man, it is observed that most of the multimedia data misused. Nowadays telemedicine, tele diagnosis, tele consultation, teleradiology, telematic services are necessary. Electronic Patient Record is essential to provide these services. It is necessary to secure the multimedia data by using a suitable watermarking technique. Due to the huge availability of high-speed internet, digital movie piracy was increased rapidly and it causes revenue loss and affects the employment of the people. The high demand for protecting digital videos or movies from unauthorized access. So, it is necessary to protect the intellectual property of the owner, stop unauthorized access of data, also protect and stop the spreading of fake news. In this article, we presented a survey of digital watermarking, bolding its key concepts, embedded and extracted features, state-of-the-art implementation, and research challenges. The primary goal of this article is to provide an improved understanding of the embedding-extracting of watermarking challenges of watermarking and identify important research paths in health care and multimedia applications.
Blockchain-Enabled Federated Learning With Mechanism Design

Federated learning (FL) is a promising decentralized deep learning technique that allows users to collaboratively update models without sharing their own data. However, due to its decentralized nature, no one can monitor workers' behavior, and they may thus deviate protocols (e.g., participating without updating any models). To solve this problem, many researchers have proposed blockchain-enabled FL to reward workers (or users) with cryptocurrencies to encourage workers to follow the protocols. However, there is a lack of theoretical discussions concerning how such rewards impact workers' behavior and how much should be given to workers. In this article, we propose a mechanism-design-oriented FL protocol on a public blockchain network. Mechanism design (MD) is often used to make a rule intended to achieve a specific goal. With MD in mind, we introduce the concept of competition into blockchain-based FL so that only workers who have contributed well can obtain rewards, which naturally prevents workers from deviating from the protocol. We then mathematically answer the following questions with contest theory, a novel field of study in economics: i) What behavior will workers take?; ii) how much effort should workers exert to maximize their profits?; iii) how many workers should be rewarded?; and iv) what is the best proportion for reward distribution?
Technology acceptance model and multi-user virtual reality learning environments for Chinese language education

Virtual reality is a rapidly developing technology and its applications are being explored in a variety of educational fields. While an increasing amount of attention is being paid to VR as a language learning tool, there remains less research on using VR with languages other than English. Adapting the Technology Acceptance Model, this study investigated learner attitudes towards Hubs by Mozilla, a multi-user VR learning environment, for the purpose of learning Chinese as an additional language. Data were collected by means of a post-participation questionnaire that measured seven constructs. It was hypothesized that the unique features of virtual reality technology would have a statistically significant positive association with the constructs of technology acceptance. Structural Equation Modeling was used to explore the relationships between the constructs. Results indicate varying degrees in significance between the impacts of different constructs, but an overall agreeable attitude towards using the virtual environment for Chinese language learning. Furthermore, difficulties in interaction within the virtual environment seemed to affect the learners’ perceived ease of use of the technology, suggesting that increased consideration for the design of VR learning environments and for the virtual literacies of learners may help this technology be more successful in the classroom.
deSpeckNet: Generalizing Deep Learning-Based SAR Image Despeckling

Deep learning (DL) has proven to be a suitable approach for despeckling synthetic aperture radar (SAR) images. So far, most DL models are trained to reduce speckle that follows a particular distribution, either using simulated noise or a specific set of real SAR images, limiting the applicability of these methods for real SAR images with unknown noise statistics. In this paper, we present a DL method, deSpeckNet1, that estimates the speckle noise distribution and the despeckled image simultaneously. Since it does not depend on a specific noise model, deSpeckNet generalizes well across SAR acquisitions in a variety of landcover conditions. We evaluated the performance of deSpeckNet on single polarized Sentinel-1 images acquired in Indonesia, The Democratic Republic of Congo and The Netherlands, a single polarized ALOS-2/PALSAR-2 image acquired in Japan and an Iceye X2 image acquired in Germany. In all cases, deSpeckNet was able to effectively reduce speckle and restore
Blockchain for public health care in smart society

Blockchain is a new technology that demands more efficient and scalable techniques to incorporate it with business models. Therefore, in this paper, we propose a blockchain-based smart healthcare business model, which keeps customers at the center of business. Our proposed smart healthcare business model can predict customer's status and is able to give rewards according to the business rules set by participating organizations. However, businesses also demand something in return, and in our scenario, we are concerned about data in the wild. Our model fetches "data in the wild" from the Internet of Medical Things. Nevertheless, this model can be applied to any business scenario where a customer reward system exists. Our proposed model focuses more on customers and business while utilizing technology to ease the customer and other parties involved in the business. This fusion makes business more effective as the organization can determine the path of business and make decisions accordingly. Incorporating technologies with existing business models such as the "consumer centric model" makes it easy for businesses to modernize.
RNN-Transducer based Chinese Sign Language Recognition

Sign Language Recognition (SLR) targets on interpreting sign language video into natural language, which largely facilitates mutual communication between the deaf and general public. SLR is usually formulated as a sequence alignment problem, wherein connectionist temporal classification (CTC) plays an important role in building effective alignment between video sequence and sentence-level labels. However, CTC-based SLR methods tend to fail if the output label sequence is longer than the input video sequence. Besides, they ignore the interdependencies between output predictions. This paper addresses these two issues and proposes a new RNN-Transducer based SLR framework, i.e., visual hierarchy to lexical sequence alignment network (H2SNet). In the framework, we design a visual hierarchy transcription network to capture the spatial appearance and temporal motion cues of sign video on multiple levels. Meanwhile, we utilize a lexical prediction network to extract effective contextual information from output predictions. RNN-Transducer is applied to learn the mapping between sequential video features and sentence-level labels. Extensive experiments validate the effectiveness and superiority of our approach over state-of-the-art methods.
An interpretable sequential three-way recommendation based on collaborative topic regression

Owing to the imbalance of observed data and user’s preference, it is necessary and meaningful to think over the multilevel characteristics of recommendation information (RI) during the recommendation process. At the same time, to better summarize user’s preference and enhance user’s beliefs, the interpretability of recommendation results also become more and more important in recommender system (RS). In view of the multilevel characteristics of RI and the interpretability of recommendation results, this paper proposes a novel interpretable sequential three-way recommendation strategy, namely, CTR-based cost-sensitive sequential three-way recommendation (CTR-CS3WR). First, in order to construct the interpretable granular features and multilevel information, we introduce collaborative topic regression (CTR) and design three novel granulation methods: PMF-based, LDA-based and CTR-based granulation method. Then, with the consideration of decision cost and time cost, a sequential three-way recommendation strategy is proposed to realize the multilevel recommendation. Finally, extensive experiments on two CiteUlike datasets verify the effectiveness of our proposed granulation methods and recommendation strategy.
Deep learning for pattern recognition of photovoltaic energy generation

With the rapid growth in computational complexities of statistical pattern recognition of photovoltaic (PV) energy measurements, the need for new data-driven models has emerged. Among machine learning frameworks, deep neural networks yield promising solutions due to their high generalization capacity, low estimation bias, and ease of implementation. In this study, first, the discriminative deep models including autoencoders, Long Short-Term Memory networks, and Rectified Linear Units are introduced as a class of mathematical models that directly estimate the future solar energy given historical measurements. In addition, Convolutional Neural Networks are explored to show the merit of spatiotemporal pattern recognition to increase PV generation prediction accuracy. Then, we explore deep sparse coding algorithms for behind-the-meter (BTM) energy disaggregation (ED) with PV generation in residential and commercial customers. We show how the combination of deep discriminative models with dictionary learning leads to a promising performance in decomposing the net electricity demand into the solar generation and demand. Finally, deep generative modeling for BTM ED with PV generation is proposed which learns powerful probability distribution functions from PV time series rather than crisp features captured by the classic discriminative deep learning. Our numerical results on real-world ED datasets of residential units show significant improvements in ED accuracy compared to the state-of-the-art sparse coding approaches.
Salient Object Detection Based on Progressively Supervised Learning for Remote Sensing Images

Salient object detection (SOD) is a crucial task in the field of remote sensing image (RSI) processing. Weakly supervised SOD methods, which generate saliency maps by classification convolutional neural networks (CNNs), considerably reduce labor costs. However, due to the complexity of remote sensing scenes, concerns remain about weakly supervised SOD for RSIs: 1) since the pooling operations are applied in the classification CNNs, the boundary maintenance of weakly supervised methods is unsatisfactory and 2) several sophisticated postprocessing procedures are used in previous weakly supervised methods, which are inevitably time-consuming. To solve these problems, we combine the benefits of weakly and fully supervised learning and propose a new SOD method named progressively supervised learning (PSL) for RSIs. The proposed method realizes end-to-end SOD with a lightweight model under imagewise annotations. First, to reduce the demands on large-scale pixelwise annotations, we propose a pseudo-label generation method based on a classification network and gradient-weighted class activation mapping (Grad-CAM) to compute pseudo saliency maps (PSMs) for training samples and auxiliary images in a weakly supervised manner. Then, to improve the computational efficiency, we construct a feedback saliency analysis network (FSAN), where the generated PSMs are regarded as pixelwise labels. Finally, inspired by curriculum learning, we design a new denoising loss function to further reduce the effect brought by missing judgment in PSMs and enhance the detection accuracy. Comprehensive evaluations with two remote sensing data sets and a comparison with 11 methods validate the superiority of the proposed PSL model.
Revisiting Adaptive Convolutions for Video Frame Interpolation

Video frame interpolation, the synthesis of novel views in time, is an increasingly popular research direction with many new papers further advancing the state of the art. But as each new method comes with a host of variables that affect the interpolation quality, it can be hard to tell what is actually important for this task. In this work, we show, somewhat surprisingly, that it is possible to achieve near state-of-the-art results with an older, simpler approach, namely adaptive separable convolutions, by a subtle set of low level improvements. In doing so, we propose a number of intuitive but effective techniques to improve the frame interpolation quality, which also have the potential to other related applications of adaptive convolutions such as burst image denoising, joint image filtering, or video prediction.
MPRNet: Multi-Path Residual Network for Lightweight Image Super Resolution

Lightweight super resolution networks have extremely importance for real-world applications. In recent years several SR deep learning approaches with outstanding achievement have been introduced by sacrificing memory and computational cost. To overcome this problem, a novel lightweight super resolution network is proposed, which improves the SOTA performance in lightweight SR and performs roughly similar to computationally expensive networks. Multi-Path Residual Network designs with a set of Residual concatenation Blocks stacked with Adaptive Residual Blocks: (i) to adaptively extract informative features and learn more expressive spatial context information; (ii) to better leverage multi-level representations before up-sampling stage; and (iii) to allow an efficient information and gradient flow within the network. The proposed architecture also contains a new attention mechanism, Two-Fold Attention Module, to maximize the representation ability of the model. Extensive experiments show the superiority of our model against other SOTA SR approaches.
WeGleNet: A weakly-supervised convolutional neural network for the semantic segmentation of Gleason grades in prostate histology images

Prostate cancer is one of the main diseases affecting men worldwide. The Gleason scoring system is the primary diagnostic tool for prostate cancer. This is obtained via the visual analysis of cancerous patterns in prostate biopsies performed by expert pathologists, and the aggregation of the main Gleason grades in a combined score. Computer-aided diagnosis systems allow to reduce the workload of pathologists and increase the objectivity. Nevertheless, those require a large number of labeled samples, with pixel-level annotations performed by expert pathologists, to be developed. Recently, efforts have been made in the literature to develop algorithms aiming the direct estimation of the global Gleason score at biopsy/core level with global labels. However, these algorithms do not cover the accurate localization of the Gleason patterns into the tissue. These location maps are the basis to provide a reliable computer-aided diagnosis system to the experts to be used in clinical practice by pathologists. In this work, we propose a deep-learning-based system able to detect local cancerous patterns in the prostate tissue using only the global-level Gleason score obtained from clinical records during training.The methodological core of this work is the proposed weakly-supervised-trained convolutional neural network, WeGleNet, based on a multi-class segmentation layer after the feature extraction module, a global-aggregation, and the slicing of the background class for the model loss estimation during training.Using a public dataset of prostate tissue-micro arrays, we obtained a Cohen's quadratic kappa (κ) of 0.67 for the pixel-level prediction of cancerous patterns in the validation cohort. We compared the model performance for semantic segmentation of Gleason grades with supervised state-of-the-art architectures in the test cohort. We obtained a pixel-level κ of 0.61 and a macro-averaged f1-score of 0.58, at the same level as fully-supervised methods. Regarding the estimation of the core-level Gleason score, we obtained a κ of 0.76 and 0.67 between the model and two different pathologists.WeGleNet is capable of performing the semantic segmentation of Gleason grades similarly to fully-supervised methods without requiring pixel-level annotations. Moreover, the model reached a performance at the same level as inter-pathologist agreement for the global Gleason scoring of the cores.
Revisiting Batch Normalization for Improving Corruption Robustness

The performance of DNNs trained on clean images has been shown to decrease when the test images have common corruptions. In this work, we interpret corruption robustness as a domain shift and propose to rectify batch normalization (BN) statistics for improving model robustness. This is motivated by perceiving the shift from the clean domain to the corruption domain as a style shift that is represented by the BN statistics. We find that simply estimating and adapting the BN statistics on a few (32 for instance) representation samples, without retraining the model, improves the corruption robustness by a large margin on several benchmark datasets with a wide range of model architectures. For example, on ImageNet-C, statistics adaptation improves the top1 accuracy of ResNet50 from 39.2% to 48.7%. Moreover, we find that this technique can further improve state-of-the-art robust models from 58.1% to 63.3%.
Multilingual Review-aware Deep Recommender System via Aspect-based Sentiment Analysis

With the dramatic expansion of international markets, consumers write reviews in different languages, which poses a new challenge for Recommender Systems (RSs) dealing with this increasing amount of multilingual information. Recent studies that leverage deep-learning techniques for review-aware RSs have demonstrated their effectiveness in modelling fine-grained user-item interactions through the aspects of reviews. However, most of these models can neither take full advantage of the contextual information from multilingual reviews nor discriminate the inherent ambiguity of words originated from the user’s different tendency in writing. To this end, we propose a novel Multilingual Review-aware Deep Recommendation Model (MrRec) for rating prediction tasks. MrRec mainly consists of two parts: (1) Multilingual aspect-based sentiment analysis module (MABSA), which aims to jointly extract aligned aspects and their associated sentiments in different languages simultaneously with only requiring overall review ratings. (2) Multilingual recommendation module that learns aspect importances of both the user and item with considering different contributions of multiple languages and estimates aspect utility via a dual interactive attention mechanism integrated with aspect-specific sentiments from MABSA. Finally, overall ratings can be inferred by a prediction layer adopting the aspect utility value and aspect importance as inputs. Extensive experimental results on nine real-world datasets demonstrate the superior performance and interpretability of our model.
Error-robust low-rank tensor approximation for multi-view clustering

Multi-view clustering has been widely developed to improve the clustering performance over that of single-view clustering. However, the types of errors vary and behave inconsistently in each view, which results in performance degradation in real applications. To address this limitation, in this paper, we propose a novel Markov chain-based spectral clustering method for multi-view clustering to handle different types of errors. Unlike most of the existing self-representation-based subspace clustering methods, which process each view separately, ignoring complementary information among views, our method first computes the transition probability matrices of all views, then forms each transition probability matrix as a frontal slice of a third-order tensor to capture the multi-view information, and finally decomposes the tensor into an ideal tensor with the tensor nuclear norm constraint and an error term. Furthermore, the proposed method imposes the group l1 and l2,1 norms on the error matrix for error learning such that various errors can be clearly characterized and processed to improve performance. To solve the challenging optimization model, we propose an efficient algorithm using the augmented Lagrangian multiplier method. Experimental results on three real-world datasets show that the proposed method is superior to the state-of-the art methods in various evaluation metrics.
A Framework for Enhancing Deep Neural Networks Against Adversarial Malware

Machine learning-based malware detection is known to be vulnerable to adversarial evasion attacks. The state-of-the-art is that there are no effective defenses against these attacks. As a response to the adversarial malware classification challenge organized by the MIT Lincoln Lab and associated with the AAAI-19 Workshop on Artificial Intelligence for Cyber Security (AICS'2019), we propose six guiding principles to enhance the robustness of deep neural networks. Some of these principles have been scattered in the literature, but the others are introduced in this paper for the first time. Under the guidance of these six principles, we propose a defense framework to enhance the robustness of deep neural networks against adversarial malware evasion attacks. By conducting experiments with the Drebin Android malware dataset, we show that the framework can achieve a 98.49\% accuracy (on average) against grey-box attacks, where the attacker knows some information about the defense and the defender knows some information about the attack, and an 89.14% accuracy (on average) against the more capable white-box attacks, where the attacker knows everything about the defense and the defender knows some information about the attack. The framework wins the AICS'2019 challenge by achieving a 76.02% accuracy, where neither the attacker (i.e., the challenge organizer) knows the framework or defense nor we (the defender) know the attacks. This gap highlights the importance of knowing about the attack.