Reviewerly/exHarmony huggingface dataset:
Handwriting recognition using cohort of LSTM and lexicon verification with extremely large lexicon

In this article, a handwriting recognition model whose complexity does not depend on the lexicon size is proposed. It is an alternative to lexicon-driven decoding, based on a lexicon verification process that allows to deal with millions of words, without any time consuming decoding stage. This lexicon verification is included in a cascade framework that uses complementary LSTM RNN classifiers. An original and very efficient method to obtain hundreds of complementary LSTM RNN extracted from a single training, called cohort, is proposed. The proposed approach achieves new state-of-the art performance on the Rimes and IAM datasets, and provides 90% of accuracy on the Rimes dataset when dealing with a gigantic lexicon record of 3 millions of words. The last contribution extends the idea of cohort and lexicon verification in a ROVER combination for handwriting line recognition, and achieves state-of-the-art results on the Rimes dataset.
Features of Application of Machine Learning Methods for Classification of Network Traffic (Features, Advantages, Disadvantages)

The problem of traffic classification has been studied for a long time: a significant amount of research work has been devoted to its analysis and the search for effective solutions under various conditions and limitations, including in recent years. Classification of network traffic is widely required for many network management tasks, such as stream prioritization, traffic shaping/control, and diagnostic monitoring. When solving network management problems, problems arise such as characterization and modelling of workload, capacity planning, route preparation, and accurate identification of network traffic. This article provides a description of network traffic and an overview of machine learning methods for classifying it, such as Naive Bayes, C4.5, AdaBoost and Random Forest, their advantages, disadvantages and features.
Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient

The strong {\it lottery ticket hypothesis} (LTH) postulates that one can approximate any target neural network by only pruning the weights of a sufficiently over-parameterized random network. A recent work by Malach et al. \cite{MalachEtAl20} establishes the first theoretical analysis for the strong LTH: one can provably approximate a neural network of width $d$ and depth $l$, by pruning a random one that is a factor $O(d^4l^2)$ wider and twice as deep. This polynomial over-parameterization requirement is at odds with recent experimental research that achieves good approximation with networks that are a small factor wider than the target. In this work, we close the gap and offer an exponential improvement to the over-parameterization requirement for the existence of lottery tickets. We show that any target network of width $d$ and depth $l$ can be approximated by pruning a random network that is a factor $O(\log(dl))$ wider and twice as deep. Our analysis heavily relies on connecting pruning random ReLU networks to random instances of the \textsc{SubsetSum} problem. We then show that this logarithmic over-parameterization is essentially optimal for constant depth networks. Finally, we verify several of our theoretical insights with experiments.
Multi-facet user preference learning for fine-grained item recommendation

Existing recommendation methods mainly learn user preference from historical user-item interaction data, while ignoring the extent of interactions, i.e., diverse user experience and user intention. To be more specific, for new users with little or no experience, they may turn to popular items preferred by majority users. In terms of those medium-level users who already have interacted with some items, they may require and expect items to meet their personal preferences. As pro-active users are likely to leave rich behavior (action) feedback (e.g., view, like) on the items they interacted with, the system will have good chance to better interpret users’ intention, and thus generate more accurate and elaborate recommendations to hit their preferences. In this paper, we propose a generic Multi-facet User Preference Learning (MUPL) framework for fine-grained item recommendation. By considering diverse user experience and intention, MUPL captures user preference in the level of group-, individual- and action-facet. Besides, the importance of user preference in different facets can be automatically learnt by MUPL. Extensive experiments on two real-world datasets (Xing, Sobazaar) demonstrate the superiority of our proposed approach over other state-of-the-art methods.
Convolutional Neural Network-Based In-Vehicle Occupant Detection and Classification Method using Second Strategic Highway Research Program Cabin Images

This paper describes an approach for automatic detection and localization of drivers and passengers in automobiles using in-cabin images. We used a convolutional neural network (CNN) framework and conducted experiments based on the Faster R-CNN and Cascade R-CNN detectors. Training and evaluation were performed using the Second Strategic Highway Research Program (SHRP 2) naturalistic dataset. In SHRP 2, the cabin images have been blurred to maintain privacy. After detecting occupants inside the vehicle, the system classifies each occupant as driver, front-seat passenger, or back-seat passenger. For one SHRP 2 test set, the system detected occupants with an accuracy of 94.5%. Those occupants were correctly classified as front-seat passenger with an accuracy of 97.3%, as driver with 99.5% accuracy, and as back-seat passenger with 94.3% accuracy. The system performed slightly better for daytime images than for nighttime images. Unlike previous work, this method is capable of presence classification and location prediction of occupants. By fine-tuning the object detection model, there is also significant improvement in detection accuracy as compared with pretrained models. The study also provides a fully annotated dataset of in-cabin images. This work is expected to facilitate research involving interactions between drivers and passengers, particularly related to driver attention and safety.
Deep Neural Embedding for Software Vulnerability Discovery: Comparison and Optimization

Due to multitudinous vulnerabilities in sophisticated software programs, the detection performance of existing approaches requires further improvement. Multiple vulnerability detection approaches have been proposed to aid code inspection. Among them, there is a line of approaches that apply deep learning (DL) techniques and achieve promising results. This paper attempts to utilize CodeBERT which is a deep contextualized model as an embedding solution to facilitate the detection of vulnerabilities in C open-source projects. The application of CodeBERT for code analysis allows the rich and latent patterns within software code to be revealed, having the potential to facilitate various downstream tasks such as the detection of software vulnerability. CodeBERT inherits the architecture of BERT, providing a stacked encoder of transformer in a bidirectional structure. This facilitates the learning of vulnerable code patterns which requires long-range dependency analysis. Additionally, the multihead attention mechanism of transformer enables multiple key variables of a data flow to be focused, which is crucial for analyzing and tracing potentially vulnerable data flaws, eventually, resulting in optimized detection performance. To evaluate the effectiveness of the proposed CodeBERT-based embedding solution, four mainstream-embedding methods are compared for generating software code embeddings, including Word2Vec, GloVe, and FastText. Experimental results show that CodeBERT-based embedding outperforms other embedding models on the downstream vulnerability detection tasks. To further boost performance, we proposed to include synthetic vulnerable functions and perform synthetic and real-world data fine tuning to facilitate the model learning of C-related vulnerable code patterns. Meanwhile, we explored the suitable configuration of CodeBERT. The evaluation results show that the model with new parameters outperform some state-of-the-art detection methods in our dataset.
Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation

We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings.
AB-LaBSE: Uyghur Sentiment Analysis via the Pre-Training Model with BiLSTM

In recent years, more and more attention has been paid to text sentiment analysis, which has gradually become a research hotspot in information extraction, data mining, Natural Language Processing (NLP), and other fields. With the gradual popularization of the Internet, sentiment analysis of Uyghur texts has great research and application value in online public opinion. For low-resource languages, most state-of-the-art systems require tens of thousands of annotated sentences to get high performance. However, there is minimal annotated data available about Uyghur sentiment analysis tasks. There are also specificities in each task—differences in words and word order across languages make it a challenging problem. In this paper, we present an effective solution to providing a meaningful and easy-to-use feature extractor for sentiment analysis tasks: using the pre-trained language model with BiLSTM layer. Firstly, data augmentation is carried out by AEDA (An Easier Data Augmentation), and the augmented dataset is constructed to improve the performance of text classification tasks. Then, a pretraining model LaBSE is used to encode the input data. Then, BiLSTM is used to learn more context information. Finally, the validity of the model is verified via two categories datasets for sentiment analysis and five categories datasets for emotion analysis. We evaluated our approach on two datasets, which showed wonderful performance compared to some strong baselines. We close with an overview of the resources for sentiment analysis tasks and some of the open research questions. Therefore, we propose a combined deep learning and cross-language pretraining model for two low resource expectations.
Contrastive Hebbian Feedforward Learning for Neural Networks

This paper addresses the biological plausibility of both backpropagation (BP) and contrastive Hebbian learning (CHL) used in the Boltzmann machines. The main claim of this paper is that CHL is a general learning algorithm that can be used to steer feedforward networks toward desirable outcomes, and steer them away from undesirable outcomes without any need for the specialized feedback circuit of BP or the symmetric connections used by the Boltzmann machines. After adding perturbations during the learning phase to all the neurons in the network, multiple feedforward outcomes are classified into Hebbian and anti-Hebbian sets based on the network predictions. The algorithm is applied to networks when optimizing a loss objective where BP excels and is also applied to networks with stochastic binary outputs where BP cannot be easily applied. The power of the proposed algorithm lies in its simplicity where both learning and gradient estimation through stochastic binary activations are combined into a single local Hebbian rule. We will also show that both Hebbian and anti-Hebbian correlations are evaluated from the readily available signals that are fundamentally different from CHL used in the Boltzmann machines. We will demonstrate that the new learning paradigm where Hebbian/anti-Hebbian correlations are based on correct/incorrect predictions is a powerful concept that separates this paper from other biologically inspired learning algorithms.
Abstract Semantic Dependency

Dependency is a prevalent notion in computer science. There have been numerous informal or formal attempts to define viable syntactic and semantic concepts of dependency in programming languages with subtle variations and limitations. We develop a new value dependency analysis defined by abstract interpretation of a trace semantics. A sound approximate dependency algorithm is formally derived by calculational design. Further abstractions provide information flow, slicing, non-interference, dye, and taint analyses.
Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-ID.

Domain adaptive object re-ID aims to transfer the learned knowledge from the labeled source domain to the unlabeled target domain to tackle the open-class re-identification problems. Although state-of-the-art pseudo-label-based methods have achieved great success, they did not make full use of all valuable information because of the domain gap and unsatisfying clustering performance. To solve these problems, we propose a novel self-paced contrastive learning framework with hybrid memory. The hybrid memory dynamically generates source-domain class-level, target-domain cluster-level and un-clustered instance-level supervisory signals for learning feature representations. Different from the conventional contrastive learning strategy, the proposed framework jointly distinguishes source-domain classes, and target-domain clusters and un-clustered instances. Most importantly, the proposed self-paced method gradually creates more reliable clusters to refine the hybrid memory and learning targets, and is shown to be the key to our outstanding performance. Our method outperforms state-of-the-arts on multiple domain adaptation tasks of object re-ID and even boosts the performance on the source domain without any extra annotations. Our generalized version on unsupervised person re-ID surpasses state-of-the-art algorithms by considerable 16.2% and 14.6% on Market-1501 and DukeMTMC-reID benchmarks. Code is available at this https URL.
Component detection in piping and instrumentation diagrams of nuclear power plants based on neural networks

Piping and Instrumentation Diagrams (P&IDs) are the most commonly used engineering drawings to describe components and their relationships, and they are one of the most important inputs for data analysis in Nuclear Power Plants (NPP). In traditional analysis, the information related to the components is extracted manually from the P&IDs. This usually takes large amounts of effort and is error-prone. With the rapid development in the area of computer vision and deep learning, automatically detecting components and their relationships becomes possible. In this paper, we aim to use the latest neural network models to automatically extract information on components and their identifications, from the P&IDs in NPPs. We use a Faster Regional Convolutional Neural Networks (Faster RCNN) architecture called ResNet-50, to detect the components in the P&IDs. Compared to common object detection, object detection for P&IDs poses unique challenges to these methods. For example, the P&IDs symbols are much smaller than the background, and detecting such small objects remains a challenging task for modern neural networks. To address these challenges, we 1) propose several techniques for data augmentation that effectively solve the problem of training data shortage, and 2) propose a feature grouping strategy for detecting components with distinct features. Besides, we introduce a SegLink model for text detection, which can automatically extract components’ identifications from P&IDs. We also develop a method for building a data structure to reflect the relationships between components (e.g., to which pipe a component is connected, or what are the downstream or upstream components of one specific component) based on the extracted information. This data structure can be further used for plant safety analysis, and operation and maintenance cost optimization. Sensitivity analysis and comparison with other Convolutional Neural Networks (CNNs) are performed. The results of these analyses are also discussed in this paper. This analysis framework has been tested on the P&IDs from a commercial NPP. The Average Precision for components, which is used to measure the performance of the proposed method, is about 98%. The success rates of component-text mapping and component-pipe mapping are 270/275 and 319/319, respectively. It is worth noting that this framework is generic and can also be applied to P&IDs of non-nuclear industries.
Deep Learning Based Vehicle Detection From Aerial Images

With the help of today's developing technology; many applications such as traffic monitoring, target detection, empty parking lot assignment etc. have become applicable with machine learning and deep learning methods. These applications are aimed to determine the desired object from the photographs. In this study, it is aimed to develop an approach that able to identify the vehicles through aerial images by using the YOLO (You Only Look Once) algorithm, feeding it with a trained convolutional neural network structure. As a result of the study, an application that can detect vehicles was developed, increased the performance rate of the YOLO algorithm by 3.2%.
Blockchain Technology between Nakamoto and Supply Chain Management: Insights from Academia and Practice

Purpose: This paper aims to participate in answering the following research question: Will blockchain become the norm in supply chain management? Valuable insights from academia and industry about the use of blockchain technology in supply chain management are presented.Design/methodology/approach: I review the literature that mainly examined the use of blockchain technology in supply chain management from 2008 to 2020. Additionally, I explore two blockchain projects: Nimble and Carrefour.Findings: The mainstream research on the use of blockchain technology in supply chain management exhibits a positive trend. This study shows that relevant academic research increased by 174% in only 11 months (from February 2018 to January 2019). Countries and supply chains, to the same extent, are making bold moves toward using blockchain on a large scale. The European Commission, for example, funded a project to create a federated, multi-sided business ecosystem based on cloud services. The project developed a platform, named Nimble, to enable supply chain logistics to benefit from internet platforms and blockchain services. In another example, Carrefour has implemented blockchain in nine sectors (chicken, tomatoes, eggs, oranges, etc.) with a target of 300 sectors by 2022.Originality/value: Although this area of research becomes incessantly extensive, there is confusion regarding what type of blockchain or what part of the supply chain is addressed. This paper emphasizes some essential points to deal with much of the dubiety under consideration.
Self-supervised Longitudinal Neighbourhood Embedding

Longitudinal MRIs are often used to capture the gradual deterioration of brain structure and function caused by aging or neurological diseases. Analyzing this data via machine learning generally requires a large number of ground-truth labels, which are often missing or expensive to obtain. Reducing the need for labels, we propose a self-supervised strategy for representation learning named Longitudinal Neighborhood Embedding (LNE). Motivated by concepts in contrastive learning, LNE explicitly models the similarity between trajectory vectors across different subjects. We do so by building a graph in each training iteration defining neighborhoods in the latent space so that the progression direction of a subject follows the direction of its neighbors. This results in a smooth trajectory field that captures the global morphological change of the brain while maintaining the local continuity. We apply LNE to longitudinal T1w MRIs of two neuroimaging studies: a dataset composed of 274 healthy subjects, and Alzheimer's Disease Neuroimaging Initiative (ADNI, N = 632). The visualization of the smooth trajectory vector field and superior performance on downstream tasks demonstrate the strength of the proposed method over existing self-supervised methods in extracting information associated with normal aging and in revealing the impact of neurodegenerative disorders. The code is available at https://github.com/ouyangjiahong/longitudinal-neighbourhood-embedding.
Communication-Efficient Federated Edge Learning for NR-U-Based IIoT Networks

As a key infrastructural technology, Industrial Internet of Things (IIoT) and its related techniques have emerged in the age of Industrial Internet. Among them, an increasing popular and attractive federated edge learning (FEL) mechanism, which performs data analysis and inference at the edge devices distributedly, and aggregates local FEL units at a centralized controller, is introduced to meet the stringent data privacy and low-latency requirements for high-stake IIoT devices. Due to the bandwidth limitation, only parts of the IIoT devices can be selected to transmit their local FEL models to the centralized controller at each learning step. However, the centralized controller prefers to collect all the local FEL models to generate the global FL model since each IIoT device has a differential data set. Existing works mainly focus on selecting an appropriate subset of IIoT devices through advanced scheduling mechanisms without extending the resource bandwidth. However, the new radio in unlicensed spectrum (NR-U) technology in the 5G network opens up new possibilities for FEL since it is a privately owned network with fruitful bandwidth resources. We thus propose a novel communication-efficient FEL mechanism for NR-U-based IIoT networks, which aims to select data importance IIoT devices for local training under relatively sufficient unlicensed resources. The objective function is formulated as a tradeoff between total FEL data importance and the transmission latency via joint learning, device selection, and resource management scheduling, which is a mixed-integer nonlinear programming (MINLP). To deal with this problem, an alternating direction method of multipliers and block coordinate update (ADMM-BCU) algorithm with low computational complexity has been used. Closed-form expressions for both optimal device selection and resource management are derived, which highlighted significant insights. Numerical results demonstrate the algorithmic advantages and structural benefits of the proposed strategies.