Reviewerly/exHarmony huggingface dataset:
Methods for Pruning Deep Neural Networks

This paper presents a survey of methods for pruning deep neural networks.It begins by categorising over 150 studies based on the underlying approach used and then focuses on three categories: methods that use magnitude based pruning, methods that utilise clustering to identify redundancy, and methods that use sensitivity analysis to assess the effect of pruning.Some of the key influencing studies within these categories are presented to highlight the underlying approaches and results achieved.Most studies present results which are distributed in the literature as new architectures, algorithms and data sets have developed with time, making comparison across different studied difficult.The paper therefore provides a resource for the community that can be used to quickly compare the results from many different methods on a variety of data sets, and a range of architectures, including AlexNet, ResNet, DenseNet and VGG.The resource is illustrated by comparing the results published for pruning AlexNet and ResNet50 on ImageNet and ResNet56 and VGG16 on the CIFAR10 data to reveal which pruning methods work well in terms of retaining accuracy whilst achieving good compression rates.The paper concludes by identifying some research gaps and promising directions for future research.
Open-Mindedness Culture, Knowledge-Sharing, Financial Performance, and Industry 4.0 in SMEs

Due to limited resources, the implementation of Industry 4.0 solutions is a big challenge in small- and medium-sized enterprises (SMEs). Therefore, it is important to identify the factors that facilitate such implementation. However, empirical research in this area is insufficient. The purpose of this article is to investigate whether open-mindedness culture, knowledge-sharing, and financial performance have a significant impact on willingness to implement Industry 4.0 in SMEs. To achieve the purpose of the article, quantitative research was conducted. Data from 562 Polish manufacturing SMEs were collected and analyses were performed using partial least squares path modeling (PLS-PM). The study results reveal that open-mindedness culture has both a direct and indirect (through knowledge sharing) impact on willingness to implement Industry 4.0 in SMEs. Results suggest that in terms of micro-enterprises, open-mindedness is more important and knowledge-sharing is less important for willingness to implement Industry 4.0 than in larger enterprises. Moreover, regardless of the company size, financial performance is positively related to willingness to implement Industry 4.0. The research results provide SME managers with guidelines on what activities within the company should be strengthened if the goal of the company is to implement Industry 4.0 solutions.
Exploring Critical Success Factors for Blockchain-based Intelligent Transportation Systems

Utilization of Intelligent Transportation Systems (ITS) provides increasing demands on decreasing traffic congestions, transportation safety, and environmental problems as well as increasing transportation capacity. Over the past few years, one of the most disruptive and transformational technologies, blockchain technology, has emerged. As a result of offering more security, privacy, traceability, transparency, and decentralized authority, the utilization of blockchain technology in ITS have increased drastically. Despite this increase, there is a lack of a good understanding of what are the Critical Success Factors (CSF) to support IT providers for designing their products properly, and to provide industry leaders to focus on the main drivers of their outstanding projects. This study aims to explore and analyze CSFs that can facilitate the success of blockchain-based ITS, which has so far been neglected despite its remarkable relevance. In doing so, eleven primary studies, identified as a result of conducting a systematic literature review (SLR), were taken as a baseline to develop the model of CSFs. Additionally, since the number of existing studies related to CSFs for blockchain and ITS are limited, an expert panel was formed to evaluate and contribute to the model. The developed model of CSF for blockchain-based ITS, having a comprehensive approach, consists of 29 CSFs defined under five main contexts, namely technical, management, governance, quality of service, and quality of life. Doi: 10.28991/esj-2020-SP1-03 Full Text: PDF
EDP: An Efficient Decomposition and Pruning Scheme for Convolutional Neural Network Compression

Model compression methods have become popular in recent years, which aim to alleviate the heavy load of deep neural networks (DNNs) in real-world applications. However, most of the existing compression methods have two limitations: 1) they usually adopt a cumbersome process, including pretraining, training with a sparsity constraint, pruning/decomposition, and fine-tuning. Moreover, the last three stages are usually iterated multiple times. 2) The models are pretrained under explicit sparsity or low-rank assumptions, which are difficult to guarantee wide appropriateness. In this article, we propose an efficient decomposition and pruning (EDP) scheme via constructing a compressed-aware block that can automatically minimize the rank of the weight matrix and identify the redundant channels. Specifically, we embed the compressed-aware block by decomposing one network layer into two layers: a new weight matrix layer and a coefficient matrix layer. By imposing regularizers on the coefficient matrix, the new weight matrix learns to become a low-rank basis weight, and its corresponding channels become sparse. In this way, the proposed compressed-aware block simultaneously achieves low-rank decomposition and channel pruning by only one single data-driven training stage. Moreover, the network of architecture is further compressed and optimized by a novel Pruning & Merging (PM) module which prunes redundant channels and merges redundant decomposed layers. Experimental results (17 competitors) on different data sets and networks demonstrate that the proposed EDP achieves a high compression ratio with acceptable accuracy degradation and outperforms state-of-the-arts on compression rate, accuracy, inference time, and run-time memory.
Pre-training for Abstractive Document Summarization by Reinstating Source Text

Abstractive document summarization is usually modeled as a sequence-to-sequence (SEQ2SEQ) learning problem.Unfortunately, training large SEQ2SEQ based summarization models on limited supervised summarization data is challenging.This paper presents three sequence-to-sequence pre-training (in shorthand, STEP) objectives which allow us to pre-train a SEQ2SEQ based abstractive summarization model on unlabeled text.The main idea is that, given an input text artificially constructed from a document, a model is pre-trained to reinstate the original document.These objectives include sentence reordering, next sentence generation and masked document generation, which have close relations with the abstractive document summarization task.Experiments on two benchmark summarization datasets (i.e., CNN/DailyMail and New York Times) show that all three objectives can improve performance upon baselines.Compared to models pre-trained on large-scale data (≥160GB), our method, with only 19GB text for pre-training, achieves comparable results, which demonstrates its effectiveness.Code and models are public
Blockchain and Healthcare: Opportunities and Prospects for the EHR

Health protection has always been a primary concern for mankind. Despite its important social role, current systems for managing the health records are slow, complicated, sometimes expensive and exposed to human errors and misunderstandings. In the health sector, the Medicalchain project seems to have the potential to become a new standard for managing health records using blockchain technology as a platform. In this paper, we propose a new model consisting of a permissioned blockchain to manage and store the electronic health records (EHR) of registered patients. This system guarantees transparency and especially immutability, which are essential for secure management and storage, ensuring a system that is efficient both for doctors and patients and, hopefully, bringing about renewed trust in the public health system. Our aim is that our work may contribute to gain momentum on the application of the blockchain technology to EHR and stimulate further discussion with health institutions to fully exploit the potential of the technology.
Text Graph Transformer for Document Classification

Text classification is a fundamental problem in natural language processing.Recent studies applied graph neural network (GNN) techniques to capture global word co-occurrence in a corpus.However, previous works are not scalable to large-sized corpus and ignore the heterogeneity of the text graph.To address these problems, we introduce a novel Transformer based heterogeneous graph neural network, namely Text Graph Transformer (TG-Transformer).Our model learns effective node representations by capturing structure and heterogeneity from the text graph.We propose a mini-batch text graph sampling method that significantly reduces computing and memory costs to handle large-sized corpus.Extensive experiments have been conducted on several benchmark datasets, and the results demonstrate that TG-Transformer outperforms state-of-the-art approaches on text classification task.
A Finnish news corpus for named entity recognition

We present a corpus of Finnish news articles with a manually prepared named entity annotation. The corpus consists of 953 articles (193,742 word tokens) with six named entity classes (organization, location, person, product, event, and date). The articles are extracted from the archives of Digitoday, a Finnish online technology news source. The corpus is available for research purposes. We present baseline experiments on the corpus using a rule-based and two deep learning systems on two, in-domain and out-of-domain, test sets.
High-Throughput Onboard Hyperspectral Image Compression With Ground-Based CNN Reconstruction

Compression of hyperspectral images onboard of spacecrafts is a tradeoff between the limited computational resources and the ever-growing spatial and spectral resolution of the optical instruments. As such, it requires low-complexity algorithms with good rate-distortion performance and high throughput. In recent years, the Consultative Committee for Space Data Systems (CCSDS) has focused on lossless and near-lossless compression approaches based on predictive coding, resulting in the recently published CCSDS 123.0-B-2 recommended standard. While the in-loop reconstruction of quantized prediction residuals provides excellent rate-distortion performance for the near-lossless operating mode, it significantly constrains the achievable throughput due to data dependencies. In this paper, we study the performance of a faster method based on prequantization of the image followed by a lossless predictive compressor. While this is well known to be suboptimal, one can exploit powerful signal models to reconstruct the image at the ground segment, recovering part of the suboptimality. In particular, we show that convolutional neural networks can be used for this task and that they can recover the whole SNR drop incurred at a bitrate of 2 bits per pixel.
The Evaluation of Virtual Reality Fire Extinguisher Training

The aim of this research was to explore trainees’ perceptions and evaluation of Virtual Reality fire extinguisher training. Virtual Reality technology is being adopted by many industries for various purposes including safety training for safety critical industries. The future direction of Virtual Reality training requires an understanding of trainees’ evaluation of it; this fact motivated this research. Data were collected from 85 participants using a questionnaire after the training. Observation notes were taken to provide a better understanding of the context. Qualitative research with a thematic analysis was used to analyse the data. The results of this analysis revealed that the most salient themes reflect on issues surrounding the realism of the Virtual Reality simulation, namely different emotional and bodily experiences during the training, mean the benefits of the training (health, safety, environmental advantages, efficiency and convenience, repeatability and variety of scenarios) make it a good supplement. Nevertheless, improved realism is needed to make it more effective and enhance transfer and acceptance. This study encourages the consideration of important matters (such as realism and emotions) when using Virtual Reality for fire training. It also describes the positive perceptions of this type of training (repeatability of training, safety and environmental concerns).
Minimize Exposure Bias of Seq2Seq Models in Joint Entity and Relation Extraction

Joint entity and relation extraction aims to extract relation triplets from plain text directly. Prior work leverages Sequence-to-Sequence (Seq2Seq) models for triplet sequence generation. However, Seq2Seq enforces an unnecessary order on the unordered triplets and involves a large decoding length associated with error accumulation. These methods introduce exposure bias, which may cause the models overfit to the frequent label combination, thus limiting the generalization ability. We propose a novel Sequence-to-Unordered-Multi-Tree (Seq2UMTree) model to minimize the effects of exposure bias by limiting the decoding length to three within a triplet and removing the order among triplets. We evaluate our model on two datasets, DuIE and NYT, and systematically study how exposure bias alters the performance of Seq2Seq models. Experiments show that the state-of-the-art Seq2Seq model overfits to both datasets while Seq2UMTree shows significantly better generalization. Our code is available at https://github.com/WindChimeRan/OpenJERE.
BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover’s Distance

Pre-trained language models (e.g., BERT) have achieved significant success in various natural language processing (NLP) tasks.However, high storage and computational costs obstruct pre-trained language models to be effectively deployed on resourceconstrained devices.In this paper, we propose a novel BERT distillation method based on many-to-many layer mapping, which allows each intermediate student layer to learn from any intermediate teacher layers.In this way, our model can learn from different teacher layers adaptively for various NLP tasks.In addition, we leverage Earth Mover's Distance (EMD) to compute the minimum cumulative cost that must be paid to transform knowledge from teacher network to student network.EMD enables the effective matching for many-to-many layer mapping.Furthermore, we propose a cost attention mechanism to learn the layer weights used in EMD automatically, which is supposed to further improve the model's performance and accelerate convergence time.Extensive experiments on GLUE benchmark demonstrate that our model achieves competitive performance compared to strong competitors in terms of both accuracy and model compression.For reproducibility, we release the code and data at https:
Multi-Task Pre-Training of Deep Neural Networks for Digital Pathology

In this work, we investigate multi-task learning as a way of pre-training models for classification tasks in digital pathology. It is motivated by the fact that many small and medium-size datasets have been released by the community over the years whereas there is no large scale dataset similar to ImageNet in the domain. We first assemble and transform many digital pathology datasets into a pool of 22 classification tasks and almost 900k images. Then, we propose a simple architecture and training scheme for creating a transferable model and a robust evaluation and selection protocol in order to evaluate our method. Depending on the target task, we show that our models used as feature extractors either improve significantly over ImageNet pre-trained models or provide comparable performance. Fine-tuning improves performance over feature extraction and is able to recover the lack of specificity of ImageNet features, as both pre-training sources yield comparable performance.
Capsule network based analysis of histopathological images of oral squamous cell carcinoma

Oral cancer is one of the most prevalent malignancy affecting oral cavity. Determining the correct type of oral cancer at the early stages is important in designing a detailed treatment plan and predicting the response of the patient to the treatment being adopted. A major challenge lies in the detection of oral cancer from histopathological images. In oral malignancy diagnosis, the main visual features are generally extracted from the architectural differences of epithelial layers and the appearance of keratin pearls. This paper proposes a new approach for classifying oral cancer using a deep learning technique known as capsule network. Dynamic routing and routing by agreement of capsule network makes it more robust for rotation and affine transformation of augmented oral dataset. This network’s capability of handling pose, view and orientation makes it suitable for analysis of oral cancer histopathological images at an early stage. The performance of cross-validation indicate that the proposed methodology can efficiently classify the histopathological images of Oral Squamous Cell Carcinoma (OSCC) with 97.78% sensitivity, 96.92% specificity and 97.35% accuracy.
Load What You Need: Smaller Versions of Mutililingual BERT

Pre-trained Transformer-based models are achieving state-of-the-art results on a variety of Natural Language Processing data sets.However, the size of these models is often a drawback for their deployment in real production applications.In the case of multilingual models, most of the parameters are located in the embeddings layer.Therefore, reducing the vocabulary size should have an important impact on the total number of parameters.In this paper, we propose to generate smaller models that handle fewer number of languages according to the targeted corpora.We present an evaluation of smaller versions of multilingual BERT on the XNLI data set, but we believe that this method may be applied to other multilingual transformers.The obtained results confirm that we can generate smaller models that keep comparable results, while reducing up to 45% of the total number of parameters.We compared our models with DistilmBERT (a distilled version of multilingual BERT) and showed that unlike language reduction, distillation induced a 1.7% to 6% drop in the overall accuracy on the XNLI data set.The presented models and code are publicly available.
Deep Co-Training with Task Decomposition for Semi-Supervised Domain Adaptation

Semi-supervised domain adaptation (SSDA) aims to adapt models trained from a labeled source domain to a different but related target domain, from which unlabeled data and a small set of labeled data are provided. Current methods that treat source and target supervision without distinction overlook their inherent discrepancy, resulting in a source-dominated model that has not effectively use the target supervision. In this paper, we argue that the labeled target data needs to be distinguished for effective SSDA, and propose to explicitly decompose the SSDA task into two sub-tasks: a semi-supervised learning (SSL) task in the target domain and an unsupervised domain adaptation (UDA) task across domains. By doing so, the two sub-tasks can better leverage the corresponding supervision and thus yield very different classifiers. To integrate the strengths of the two classifiers, we apply the well established co-training framework, in which the two classifiers exchange their high confident predictions to iteratively "teach each other" so that both classifiers can excel in the target domain. We call our approach Deep Co-training with Task decomposition (DeCoTa). DeCoTa requires no adversarial training and is easy to implement. Moreover, DeCoTa is well founded on the theoretical condition of when co-training would succeed. As a result, DeCoTa achieves state-of-the-art results on several SSDA datasets, outperforming the prior art by a notable 4% margin on DomainNet. Code is available at https://github.com/LoyoYang/DeCoTa.
Punctuation Restoration using Transformer Models for High-and Low-Resource Languages

Punctuation restoration is a common postprocessing problem for Automatic Speech Recognition (ASR) systems.It is important to improve the readability of the transcribed text for the human reader and facilitate NLP tasks.Current state-of-art address this problem using different deep learning models.Recently, transformer models have proven their success in downstream NLP tasks, and these models have been explored very little for the punctuation restoration problem.In this work, we explore different transformer based models and propose an augmentation strategy for this task, focusing on high-resource (English) and low-resource (Bangla) languages.For English, we obtain comparable state-of-the-art results, while for Bangla, it is the first reported work, which can serve as a strong baseline for future work.We have made our developed Bangla dataset publicly available for the research community.
Benchmarking Adaptive Variational Quantum Eigensolvers

By design, the variational quantum eigensolver (VQE) strives to recover the lowest-energy eigenvalue of a given Hamiltonian by preparing quantum states guided by the variational principle. In practice, the prepared quantum state is indirectly assessed by the value of the associated energy. Novel adaptive derivative-assembled pseudo-trotter (ADAPT) ansatz approaches and recent formal advances now establish a clear connection between the theory of quantum chemistry and the quantum state ansatz used to solve the electronic structure problem. Here we benchmark the accuracy of VQE and ADAPT-VQE to calculate the electronic ground states and potential energy curves for a few selected diatomic molecules, namely H 2 , NaH, and KH. Using numerical simulation, we find both methods provide good estimates of the energy and ground state, but only ADAPT-VQE proves to be robust to particularities in optimization methods. Another relevant finding is that gradient-based optimization is overall more economical and delivers superior performance than analogous simulations carried out with gradient-free optimizers. The results also identify small errors in the prepared state fidelity which show an increasing trend with molecular size.
Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps

Texts appearing in daily scenes that can be recognized by OCR (Optical Character Recognition) tools contain significant information, such as street name, product brand and prices. Two tasks -- text-based visual question answering and text-based image captioning, with a text extension from existing vision-language applications, are catching on rapidly. To address these problems, many sophisticated multi-modality encoding frameworks (such as heterogeneous graph structure) are being used. In this paper, we argue that a simple attention mechanism can do the same or even better job without any bells and whistles. Under this mechanism, we simply split OCR token features into separate visual- and linguistic-attention branches, and send them to a popular Transformer decoder to generate answers or captions. Surprisingly, we find this simple baseline model is rather strong -- it consistently outperforms state-of-the-art (SOTA) models on two popular benchmarks, TextVQA and all three tasks of ST-VQA, although these SOTA models use far more complex encoding mechanisms. Transferring it to text-based image captioning, we also surpass the TextCaps Challenge 2020 winner. We wish this work to set the new baseline for these two OCR text related applications and to inspire new thinking of multi-modality encoder design. Code is available at https://github.com/ZephyrZhuQi/ssbaseline