Reviewerly/exHarmony huggingface dataset:
Enlarging smaller images before inputting into convolutional neural network: zero-padding vs. interpolation

Abstract The input to a machine learning model is a one-dimensional feature vector. However, in recent learning models, such as convolutional and recurrent neural networks, two- and three-dimensional feature tensors can also be inputted to the model. During training, the machine adjusts its internal parameters to project each feature tensor close to its target. After training, the machine can be used to predict the target for previously unseen feature tensors. What this study focuses on is the requirement that feature tensors must be of the same size. In other words, the same number of features must be present for each sample. This creates a barrier in processing images and texts, as they usually have different sizes, and thus different numbers of features. In classifying an image using a convolutional neural network (CNN), the input is a three-dimensional tensor, where the value of each pixel in each channel is one feature. The three-dimensional feature tensor must be the same size for all images. However, images are not usually of the same size and so are not their corresponding feature tensors. Resizing images to the same size without deforming patterns contained therein is a major challenge. This study proposes zero-padding for resizing images to the same size and compares it with the conventional approach of scaling images up (zooming in) using interpolation. Our study showed that zero-padding had no effect on the classification accuracy but considerably reduced the training time. The reason is that neighboring zero input units (pixels) will not activate their corresponding convolutional unit in the next layer. Therefore, the synaptic weights on outgoing links from input units do not need to be updated if they contain a zero value. Theoretical justification along with experimental endorsements are provided in this paper.
Hetero-Center loss for cross-modality person Re-identification

Cross-modality person re-identification is a challenging problem which retrieves a given pedestrian image in RGB modality among all the gallery images in infrared modality. The task can address the limitation of RGB-based person Re-ID in dark environments. Existing researches mainly focus on enlarging inter-class differences of feature to solve the problem. However, few studies investigate improving intra-class cross-modality similarity, which is important for this issue. In this paper, we propose a novel loss function, called Hetero-Center loss (HC loss) to reduce the intra-class cross-modality variations. Specifically, HC loss can supervise the network learning the cross-modality invariant information by constraining the intra-class center distance between two heterogenous modalities. With the joint supervision of Cross-Entropy (CE) loss and HC loss, the network is trained to achieve two vital objectives, inter-class discrepancy and intra-class cross-modality similarity as much as possible. Besides, we propose a simple and high-performance network architecture to learn local feature representations for cross-modality person re-identification, which can be a baseline for future research. Extensive experiments indicate the effectiveness of the proposed methods, which outperform state-of-the-art methods by a wide margin.
Data Sharing System Integrating Access Control Mechanism using Blockchain-Based Smart Contracts for IoT Devices

In this paper, a blockchain-based data sharing and access control system is proposed, for communication between the Internet of Things (IoT) devices. The proposed system is intended to overcome the issues related to trust and authentication for access control in IoT networks. Moreover, the objectives of the system are to achieve trustfulness, authorization, and authentication for data sharing in IoT networks. Multiple smart contracts such as Access Control Contract (ACC), Register Contract (RC), and Judge Contract (JC) are used to provide efficient access control management. Where ACC manages overall access control of the system, and RC is used to authenticate users in the system, JC implements the behavior judging method for detecting misbehavior of a subject (i.e., user). After the misbehavior detection, a penalty is defined for that subject. Several permission levels are set for IoT devices’ users to share services with others. In the end, performance of the proposed system is analyzed by calculating cost consumption rate of smart contracts and their functions. A comparison is made between existing and proposed systems. Results show that the proposed system is efficient in terms of cost. The overall execution cost of the system is 6,900,000 gas units and the transaction cost is 5,200,000 gas units.
Zero-VAE-GAN: Generating Unseen Features for Generalized and Transductive Zero-Shot Learning

Zero-shot learning (ZSL) is a challenging task due to the lack of unseen class data during training. Existing works attempt to establish a mapping between the visual and class spaces through a common intermediate semantic space. The main limitation of existing methods is the strong bias towards seen class, known as the domain shift problem, which leads to unsatisfactory performance in both conventional and generalized ZSL tasks. To tackle this challenge, we propose to convert ZSL to the conventional supervised learning by generating features for unseen classes. To this end, a joint generative model that couples variational autoencoder (VAE) and generative adversarial network (GAN), called Zero-VAE-GAN, is proposed to generate high-quality unseen features. To enhance the class-level discriminability, an adversarial categorization network is incorporated into the joint framework. Besides, we propose two self-training strategies to augment unlabeled unseen features for the transductive extension of our model, addressing the domain shift problem to a large extent. Experimental results on five standard benchmarks and a large-scale dataset demonstrate the superiority of our generative model over the state-of-the-art methods for conventional, especially generalized ZSL tasks. Moreover, the further improvement of the transductive setting demonstrates the effectiveness of the proposed self-training strategies.
Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit

This paper introduces a new end-to-end text-to-speech (E2E-TTS) toolkit named ESPnet-TTS, which is an extension of the open-source speech processing toolkit ESPnet. The toolkit supports state-of- the-art E2E-TTS models, including Tacotron 2, Transformer TTS, and FastSpeech, and also provides recipes inspired by the Kaldi automatic speech recognition (ASR) toolkit. The recipes are based on the design unified with the ESPnet ASR recipe, providing high reproducibility. The toolkit also provides pre-trained models and samples of all of the recipes so that users can use it as a baseline. Furthermore, the unified design enables the integration of ASR functions with TTS, e.g., ASR-based objective evaluation and semi- supervised learning with both ASR and TTS models. This paper describes the design of the toolkit and experimental evaluation in comparison with other toolkits. The experimental results show that our models can achieve state-of-the-art performance comparable to the other latest toolkits, resulting in a mean opinion score (MOS) of 4.25 on the LJSpeech dataset. The toolkit is publicly available at https://github.com/espnet/espnet.
Digital Twin and Web-Based Virtual Gaming Technologies for Online Education: A Case of Construction Management and Engineering

Mixed reality is advancing exponentially in some innovative industries, including manufacturing and aerospace. However, advanced applications of these technologies in architecture, engineering, and construction (AEC) businesses remain nascent. While it is in demand, the use of these technologies in developing the AEC digital pedagogy and for improving professional competence have received little attention. This paper presents a set of five novel digital technologies utilising virtual and augmented reality and digital twin, which adds value to the literature by showing their usefulness in the delivery of construction courses. The project involved designing, developing, and implementing a construction augmented reality (AR), including Piling AR (PAR) and a virtual tunnel boring machine (VTBM) module. The PAR is a smartphone module that presents different elements of a building structure, the footing system, and required equipment for footing construction. VTBM is developed as a multiplayer and avatar-included module for experiencing mechanisms of a tunnel boring machine. The novelty of this project is that it developed innovative immersive construction modules, practices of implementing digital pedagogy, and presenting the capacity of virtual technologies for education. This paper is also highly valuable to educators since it shows how a set of simple to complex technologies can be used for teaching various courses from a distance, either in emergencies such as corona virus disease (COVID-19) or as a part of regular teaching. This paper is a step forward to designing future practices full of virtual education appropriate to the new generation of digitally savvy students.
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts

Natural-language prompts have recently been used to coax pretrained language models into performing other AI tasks, using a fill-in-theblank paradigm (Petroni et al., 2019) or a few-shot extrapolation paradigm (Brown et al., 2020).For example, language models retain factual knowledge from their training corpora that can be extracted by asking them to "fill in the blank" in a sentential prompt.However, where does this prompt come from?We explore the idea of learning prompts by gradient descent-either fine-tuning prompts taken from previous work, or starting from random initialization.Our prompts consist of "soft words," i.e., continuous vectors that are not necessarily word type embeddings from the language model.Furthermore, for each task, we optimize a mixture of prompts, learning which prompts are most effective and how to ensemble them.Across multiple English LMs and tasks, our approach hugely outperforms previous methods, showing that the implicit factual knowledge in language models was previously underestimated.Moreover, this knowledge is cheap to elicit: random initialization is nearly as good as informed initialization.x performed until his death in y .
Image Encryption Using Josephus Problem and Filtering Diffusion

Image encryption is an efficient visual technology to protect private images. This paper develops an image encryption algorithm utilizing the principles of the Josephus problem and the filtering technology. The encryption algorithm follows the classical diffusion and confusion structure. The principle of Josephus problem is used to shuffle the image pixels to different positions to achieve the confusion property. Using a randomly generated filter, the filtering technology can spread slight changes of the original image to all pixels of the cipher image to obtain diffusion property. The simulation results show that the developed image encryption algorithm is able to encrypt different kinds of images into cipher images with uniform distribution. The security analysis demonstrates that it has an extremely sensitive secret key, can resist various security attacks, and has a better performance than several advanced image encryption algorithms.
Multi-modal deep learning model for auxiliary diagnosis of Alzheimer’s disease

Alzheimer’s disease (AD) is one of the most difficult to cure diseases. Alzheimer’s disease seriously affects the normal lives of the elderly and their families. The mild cognitive impairment (MCI) is a transitional state between the normal aging and Alzheimer’s disease, and MCI is most likely to converted to AD later. MCI is often misdiagnosed as the symptoms of normal aging, which results to miss the best opportunity of treatment. Therefore, the accurate diagnosis of MCI is essential for the early diagnosis and treatment of AD. This paper presents a deep learning model for the auxiliary diagnosis of AD, which simulates the clinician’s diagnostic process. During the diagnosis of AD, clinician usually refers to the results of various neuroimaging, as well as the results of neuropsychological diagnosis. In this paper, the multi-modal medical images are trained by two independent convolutional neural networks. Then the consistency of the output of two convolutional neural networks is judged by the correlation analysis. Finally, the results of multi-modal neuroimaging diagnosis are combined with the results of clinical neuropsychological diagnosis. The proposed model provides a comprehensive analysis about patient’s pathology and psychology at the same time, therefore, it improves the accuracy of auxiliary diagnosis. The diagnosis process is closer to the process of clinician’s diagnosis and easy to implement. Experiments on the public database of ADNI (Alzheimer’s disease neuroimaging initiative) show that the proposed method has better performance, and can achieve an excellent diagnostic efficiency in the auxiliary diagnosis of AD.
Exploit the Connectivity

Multi-object tracking (MOT) is an important topic and critical task related to both static and moving camera applications, such as traffic flow analysis, autonomous driving and robotic vision. However, due to unreliable detection, occlusion and fast camera motion, tracked targets can be easily lost, which makes MOT very challenging. Most recent works exploit spatial and temporal information for MOT, but how to combine appearance and temporal features is still not well addressed. In this paper, we propose an innovative and effective tracking method called TrackletNet Tracker (TNT) that combines temporal and appearance information together as a unified framework. First, we define a graph model which treats each tracklet as a vertex. The tracklets are generated by associating detection results frame by frame with the help of the appearance similarity and the spatial consistency. To compensate camera movement, epipolar constraints are taken into consideration in the association. Then, for every pair of two tracklets, the similarity, called the connectivity in the paper, is measured by our designed multi-scale TrackletNet. Afterwards, the tracklets are clustered into groups and each group represents a unique object ID. Our proposed TNT has the ability to handle most of the challenges in MOT, and achieves promising results on MOT16 and MOT17 benchmark datasets compared with other state-of-the-art methods.
Efficient and Effective Regularized Incomplete Multi-view Clustering

Incomplete multi-view clustering (IMVC) optimally combines multiple pre-specified incomplete views to improve clustering performance. Among various excellent solutions, the recently proposed multiple kernel k-means with incomplete kernels (MKKM-IK) forms a benchmark, which redefines IMVC as a joint optimization problem where the clustering and kernel matrix imputation tasks are alternately performed until convergence. Though demonstrating promising performance in various applications, we observe that the manner of kernel matrix imputation in MKKM-IK would incur intensive computational and storage complexities, over-complicated optimization and limitedly improved clustering performance. In this paper, we first propose an Efficient and Effective Incomplete Multi-view Clustering (EE-IMVC) algorithm to address these issues. Instead of completing the incomplete kernel matrices, EE-IMVC proposes to impute each incomplete base matrix generated by incomplete views with a learned consensus clustering matrix. Moreover, we further improve this algorithm by incorporating prior knowledge to regularize the learned consensus clustering matrix. Two three-step iterative algorithms are carefully developed to solve the resultant optimization problems with linear computational complexity, and their convergence is theoretically proven. After that, we theoretically study the generalization bound of the proposed algorithms. Furthermore, we conduct comprehensive experiments to study the proposed algorithms in terms of clustering accuracy, evolution of the learned consensus clustering matrix and the convergence. As indicated, our algorithms deliver their effectiveness by significantly and consistently outperforming some state-of-the-art ones.
DABNet: Deformable Contextual and Boundary-Weighted Network for Cloud Detection in Remote Sensing Images

In recent years, deep convolutional neural networks (DCNNs) have made significant progress in cloud detection tasks, and the detection accuracy has been greatly improved. However, most existing CNN-based models have high computational complexity, which limits their practical application, especially for spaceborne optical remote sensing. In addition, most of the methods cannot make adaptive adjustments based on the structural information of the clouds, and blurred boundaries often occur in the detection results. In order to address these problems, this article proposes a lightweight network (DABNet) to achieve high-accuracy detection of complex clouds, not only a clearer boundary but also lower false-alarm rate. Specifically, a deformable context feature pyramid module is proposed to improve the adaptive modeling capability of multiscale features. Besides, a boundary-weighted loss function is designed to direct the network to focus on cloud boundary information and optimize the relevant detection results. The proposed method has been validated on two data sets: the public GF-1 WFV benchmark and our self-built GF-2 cloud detection data set with higher spatial resolution. The experimental results exhibit that DABNet achieves state-of-the-art performance while only using 4.12M parameters and 8.29G multiadds.
GLTR: Statistical Detection and Visualization of Generated Text

The rapid improvement of language models has raised the specter of abuse of text generation systems.This progress motivates the development of simple methods for detecting generated text that can be used by and explained to non-experts.We develop GLTR, a tool to support humans in detecting whether a text was generated by a model.GLTR applies a suite of baseline statistical methods that can detect generation artifacts across common sampling schemes.In a human-subjects study, we show that the annotation scheme provided by GLTR improves the human detection-rate of fake text from 54% to 72% without any prior training.GLTR is open-source and publicly deployed, and has already been widely used to detect generated outputs.
Unsupervised Domain Adaptation via Structured Prediction Based Selective Pseudo-Labeling

Unsupervised domain adaptation aims to address the problem of classifying unlabeled samples from the target domain whilst labeled samples are only available from the source domain and the data distributions are different in these two domains. As a result, classifiers trained from labeled samples in the source domain suffer from significant performance drop when directly applied to the samples from the target domain. To address this issue, different approaches have been proposed to learn domain-invariant features or domain-specific classifiers. In either case, the lack of labeled samples in the target domain can be an issue which is usually overcome by pseudo-labeling. Inaccurate pseudo-labeling, however, could result in catastrophic error accumulation during learning. In this paper, we propose a novel selective pseudo-labeling strategy based on structured prediction. The idea of structured prediction is inspired by the fact that samples in the target domain are well clustered within the deep feature space so that unsupervised clustering analysis can be used to facilitate accurate pseudo-labeling. Experimental results on four datasets (i.e. Office-Caltech, Office31, ImageCLEF-DA and Office-Home) validate our approach outperforms contemporary state-of-the-art methods.
Discriminative Adversarial Domain Adaptation

Given labeled instances on a source domain and unlabeled ones on a target domain, unsupervised domain adaptation aims to learn a task classifier that can well classify target instances. Recent advances rely on domain-adversarial training of deep networks to learn domain-invariant features. However, due to an issue of mode collapse induced by the separate design of task and domain classifiers, these methods are limited in aligning the joint distributions of feature and category across domains. To overcome it, we propose a novel adversarial learning method termed Discriminative Adversarial Domain Adaptation (DADA). Based on an integrated category and domain classifier, DADA has a novel adversarial objective that encourages a mutually inhibitory relation between category and domain predictions for any input instance. We show that under practical conditions, it defines a minimax game that can promote the joint distribution alignment. Except for the traditional closed set domain adaptation, we also extend DADA for extremely challenging problem settings of partial and open set domain adaptation. Experiments show the efficacy of our proposed methods and we achieve the new state of the art for all the three settings on benchmark datasets.
Causal Interpretability for Machine Learning - Problems, Methods and Evaluation

Machine learning models have had discernible achievements in a myriad of applications. However, most of these models are black-boxes, and it is obscure how the decisions are made by them. This makes the models unreliable and untrustworthy. To provide insights into the decision making processes of these models, a variety of traditional interpretable models have been proposed. Moreover, to generate more humanfriendly explanations, recent work on interpretability tries to answer questions related to causality such as "Why does this model makes such decisions?" or "Was it a specific feature that caused the decision made by the model?". In this work, models that aim to answer causal questions are referred to as causal interpretable models. The existing surveys have covered concepts and methodologies of traditional interpretability. In this work, we present a comprehensive survey on causal interpretable models from the aspects of the problems and methods. In addition, this survey provides in-depth insights into the existing evaluation metrics for measuring interpretability, which can help practitioners understand for what scenarios each evaluation metric is suitable.
GECToR – Grammatical Error Correction: Tag, Not Rewrite

In this paper, we present a simple and efficient GEC sequence tagger using a Transformer encoder. Our system is pre-trained on synthetic data and then fine-tuned in two stages: first on errorful corpora, and second on a combination of errorful and error-free parallel corpora. We design custom token-level transformations to map input tokens to target corrections. Our best single-model/ensemble GEC tagger achieves an F_0.5 of 65.3/66.5 on CONLL-2014 (test) and F_0.5 of 72.4/73.6 on BEA-2019 (test). Its inference speed is up to 10 times as fast as a Transformer-based seq2seq GEC system.