Reviewerly/exHarmony huggingface dataset:
Semi-supervised anomaly detection with dual prototypes autoencoder for industrial surface inspection

Anomaly detection in the automated optical quality inspection is of great important for guaranteeing the surface quality of industrial products. Most related methods are based on supervised learning techniques, which require a large number of normal and anomalous samples to obtain a robust classifier. However, the diversity of potential defects and low availability of defective samples during manufacturing bring more challenges to anomaly detection. Based on the encoder-decoder-encoder paradigm, a semi-supervised anomaly detection method Dual Prototype Auto-Encoder (DPAE) is proposed in this paper. At the training stage, the dual prototype loss and reconstruction loss are introduced to encourage the latent vectors generated by the encoders to keep closer to their own prototype. Therefore, two latent vectors of the normal image tend to be closer, and large distance between the latent vectors indicates an anomaly. And we also construct the Aluminum Profile Surface Defect (APSD) dataset for the anomaly detection task. Finally, extensive experiments on four datasets show that DPAE is effective and outperforms state-of-the-art methods.
Monte Carlo Gradient Estimation in Machine Learning

This paper is a broad and accessible survey of the methods we have at our disposal for Monte Carlo gradient estimation in machine learning and across the statistical sciences: the problem of computing the gradient of an expectation of a function with respect to parameters defining the distribution that is integrated; the problem of sensitivity analysis. In machine learning research, this gradient problem lies at the core of many learning problems, in supervised, unsupervised and reinforcement learning. We will generally seek to rewrite such gradients in a form that allows for Monte Carlo estimation, allowing them to be easily and efficiently used and analysed. We explore three strategies--the pathwise, score function, and measure-valued gradient estimators--exploring their historical development, derivation, and underlying assumptions. We describe their use in other fields, show how they are related and can be combined, and expand on their possible generalisations. Wherever Monte Carlo gradient estimators have been derived and deployed in the past, important advances have followed. A deeper and more widely-held understanding of this problem will lead to further advances, and it is these advances that we wish to support.
BioMegatron: Larger Biomedical Domain Language Model

There has been an influx of biomedical domain-specific language models, showing language models pre-trained on biomedical text perform better on biomedical domain benchmarks than those trained on general domain text corpora such as Wikipedia and Books.Yet, most works do not study the factors affecting each domain language application deeply.Additionally, the study of model size on domain-specific models has been mostly missing.We empirically study and evaluate several factors that can affect performance on domain language applications, such as the sub-word vocabulary set, model size, pre-training corpus, and domain transfer.We show consistent improvements on benchmarks with our larger BioMegatron model trained on a larger domain corpus, contributing to our understanding of domain language model applications.We demonstrate noticeable improvements over the previous state-ofthe-art (SOTA) on standard biomedical NLP benchmarks of question answering, named entity recognition, and relation extraction.Code and checkpoints to reproduce our experiments are available at github.com/NVIDIA/NeMo.
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models

Recent Transformer-based large-scale pre-trained models have revolutionized vision-and-language (V+L) research. Models such as ViLBERT, LXMERT and UNITER have significantly lifted state of the art across a wide range of V+L benchmarks. However, little is known about the inner mechanisms that destine their impressive success. To reveal the secrets behind the scene, we present Value (Vision-And-Language Understanding Evaluation), a set of meticulously designed probing tasks (e.g., Visual Coreference Resolution, Visual Relation Detection) generalizable to standard pre-trained V+L models, to decipher the inner workings of multimodal pre-training (e.g., implicit knowledge garnered in individual attention heads, inherent cross-modal alignment learned through contextualized multimodal embeddings). Through extensive analysis of each archetypal model architecture via these probing tasks, our key observations are: (i) Pre-trained models exhibit a propensity for attending over text rather than images during inference. (ii) There exists a subset of attention heads that are tailored for capturing cross-modal interactions. (iii) Learned attention matrix in pre-trained models demonstrates patterns coherent with the latent alignment between image regions and textual words. (iv) Plotted attention patterns reveal visually-interpretable relations among image regions. (v) Pure linguistic knowledge is also effectively encoded in the attention heads. These are valuable insights serving to guide future work towards designing better model architecture and objectives for multimodal pre-training. (Code is available at https://github.com/JizeCao/VALUE ).
A deep learning diagnostic platform for diffuse large B-cell lymphoma with high accuracy across multiple hospitals

Diagnostic histopathology is a gold standard for diagnosing hematopoietic malignancies. Pathologic diagnosis requires labor-intensive reading of a large number of tissue slides with high diagnostic accuracy equal or close to 100 percent to guide treatment options, but this requirement is difficult to meet. Although artificial intelligence (AI) helps to reduce the labor of reading pathologic slides, diagnostic accuracy has not reached a clinically usable level. Establishment of an AI model often demands big datasets and an ability to handle large variations in sample preparation and image collection. Here, we establish a highly accurate deep learning platform, consisting of multiple convolutional neural networks, to classify pathologic images by using smaller datasets. We analyze human diffuse large B-cell lymphoma (DLBCL) and non-DLBCL pathologic images from three hospitals separately using AI models, and obtain a diagnostic rate of close to 100 percent (100% for hospital A, 99.71% for hospital B and 100% for hospital C). The technical variability introduced by slide preparation and image collection reduces AI model performance in cross-hospital tests, but the 100% diagnostic accuracy is maintained after its elimination. It is now clinically practical to utilize deep learning models for diagnosis of DLBCL and ultimately other human hematopoietic malignancies.
Human-level saccade detection performance using deep neural networks

Saccades are ballistic eye movements that rapidly shift gaze from one location of visual space to another. Detecting saccades in eye movement recordings is important not only for studying the neural mechanisms underlying sensory, motor, and cognitive processes, but also as a clinical and diagnostic tool. However, automatically detecting saccades can be difficult, particularly when such saccades are generated in coordination with other tracking eye movements, like smooth pursuits, or when the saccade amplitude is close to eye tracker noise levels, like with microsaccades. In such cases, labeling by human experts is required, but this is a tedious task prone to variability and error. We developed a convolutional neural network to automatically detect saccades at human-level accuracy and with minimal training examples. Our algorithm surpasses state of the art according to common performance metrics and could facilitate studies of neurophysiological processes underlying saccade generation and visual processing. NEW &amp; NOTEWORTHY Detecting saccades in eye movement recordings can be a difficult task, but it is a necessary first step in many applications. We present a convolutional neural network that can automatically identify saccades with human-level accuracy and with minimal training examples. We show that our algorithm performs better than other available algorithms, by comparing performance on a wide range of data sets. We offer an open-source implementation of the algorithm as well as a web service.
Structured Disentangled Representations

Deep latent-variable models learn representations of high-dimensional data in an unsupervised manner. A number of recent efforts have focused on learning representations that disentangle statistically independent axes of variation by introducing modifications to the standard objective function. These approaches generally assume a simple diagonal Gaussian prior and as a result are not able to reliably disentangle discrete factors of variation. We propose a two-level hierarchical objective to control relative degree of statistical independence between blocks of variables and individual variables within blocks. We derive this objective as a generalization of the evidence lower bound, which allows us to explicitly represent the trade-offs between mutual information between data and representation, KL divergence between representation and prior, and coverage of the support of the empirical data distribution. Experiments on a variety of datasets demonstrate that our objective can not only disentangle discrete variables, but that doing so also improves disentanglement of other variables and, importantly, generalization even to unseen combinations of factors.
Preventing Posterior Collapse with delta-VAEs

Due to the phenomenon of collapse, current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires augmenting the objective so it does not only maximize the likelihood of the data. In this paper, we propose an alternative that utilizes the most powerful generative models as decoders, whilst optimising the variational lower bound all while ensuring that the latent variables preserve and encode useful information. Our proposed $\delta$-VAEs achieve this by constraining the variational family for the posterior to have a minimum distance to the prior. For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis. We demonstrate the efficacy of our approach at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet $32\times 32$.
Machine Learning on Mainstream Microcontrollers

This paper presents the Edge Learning Machine (ELM), a machine learning framework for edge devices, which manages the training phase on a desktop computer and performs inferences on microcontrollers. The framework implements, in a platform-independent C language, three supervised machine learning algorithms (Support Vector Machine (SVM) with a linear kernel, k-Nearest Neighbors (K-NN), and Decision Tree (DT)), and exploits STM X-Cube-AI to implement Artificial Neural Networks (ANNs) on STM32 Nucleo boards. We investigated the performance of these algorithms on six embedded boards and six datasets (four classifications and two regression). Our analysis—which aims to plug a gap in the literature—shows that the target platforms allow us to achieve the same performance score as a desktop machine, with a similar time latency. ANN performs better than the other algorithms in most cases, with no difference among the target devices. We observed that increasing the depth of an NN improves performance, up to a saturation level. k-NN performs similarly to ANN and, in one case, even better, but requires all the training sets to be kept in the inference phase, posing a significant memory demand, which can be afforded only by high-end edge devices. DT performance has a larger variance across datasets. In general, several factors impact performance in different ways across datasets. This highlights the importance of a framework like ELM, which is able to train and compare different algorithms. To support the developer community, ELM is released on an open-source basis.
More trainable inception-ResNet for face recognition

In recent years, applications of face recognition have increased significantly. Despite the successful application of deep convolutional neural network (DCNN), training such networks is still a challenging task that needs a lot of experience and carefully tuning. Based on the Inception-ResNet network, we propose a novel method to mitigate the difficulty of training such deep convolutional neural network and improve its performance simultaneously. The residual scaling factor used in the Inception-ResNet module is a manually set fixed value. We believe that changing the value to a trainable parameter and initializing it to a small value can improve the stability of the model training. We further adopted a small trick of alternating the ReLU activation function with the Leaky ReLU and PReLU. The proposed model slightly increased the number of training parameters but improved training stability and performance significantly. Extensive experiments are conducted on VGGFace2, MS1MV2, IJBB and LFW datasets. The results show that the proposed trainable residual scaling factor (TRSF) and PReLU can promote the accuracy notably while stabilizing training process.
Principles, fundamentals, and applications of programmable integrated photonics

Programmable integrated photonics is an emerging new paradigm that aims at designing common integrated optical hardware resource configurations, capable of implementing an unconstrained variety of functionalities by suitable programming, following a parallel but not identical path to that of integrated electronics in the past two decades of the last century. Programmable integrated photonics is raising considerable interest, as it is driven by the surge of a considerable number of new applications in the fields of telecommunications, quantum information processing, sensing, and neurophotonics, calling for flexible, reconfigurable, low-cost, compact, and low-power-consuming devices that can cooperate with integrated electronic devices to overcome the limitation expected by the demise of Moore’s Law. Integrated photonic devices exploiting full programmability are expected to scale from application-specific photonic chips (featuring a relatively low number of functionalities) up to very complex application-agnostic complex subsystems much in the same way as field programmable gate arrays and microprocessors operate in electronics. Two main differences need to be considered. First, as opposed to integrated electronics, programmable integrated photonics will carry analog operations over the signals to be processed. Second, the scale of integration density will be several orders of magnitude smaller due to the physical limitations imposed by the wavelength ratio of electrons and light wave photons. The success of programmable integrated photonics will depend on leveraging the properties of integrated photonic devices and, in particular, on research into suitable interconnection hardware architectures that can offer a very high spatial regularity as well as the possibility of independently setting (with a very low power consumption) the interconnection state of each connecting element. Integrated multiport interferometers and waveguide meshes provide regular and periodic geometries, formed by replicating unit elements and cells, respectively. In the case of waveguide meshes, the cells can take the form of a square, hexagon, or triangle, among other configurations. Each side of the cell is formed by two integrated waveguides connected by means of a Mach–Zehnder interferometer or a tunable directional coupler that can be operated by means of an output control signal as a crossbar switch or as a variable coupler with independent power division ratio and phase shift. In this paper, we provide the basic foundations and principles behind the construction of these complex programmable circuits. We also review some practical aspects that limit the programming and scalability of programmable integrated photonics and provide an overview of some of the most salient applications demonstrated so far.
Turning 30: New Ideas in Inductive Logic Programming

Common criticisms of state-of-the-art machine learning include poor generalisation, a lack of interpretability, and a need for large amounts of training data. We survey recent work in inductive logic programming (ILP), a form of machine learning that induces logic programs from data, which has shown promise at addressing these limitations. We focus on new methods for learning recursive programs that generalise from few examples, a shift from using hand-crafted background knowledge to learning background knowledge, and the use of different technologies, notably answer set programming and neural networks. As ILP approaches 30, we also discuss directions for future research.
Histopathological Classification of Breast Cancer Images Using a Multi-Scale Input and Multi-Feature Network

Diagnosis of pathologies using histopathological images can be time-consuming when many images with different magnification levels need to be analyzed. State-of-the-art computer vision and machine learning methods can help automate the diagnostic pathology workflow and thus reduce the analysis time. Automated systems can also be more efficient and accurate, and can increase the objectivity of diagnosis by reducing operator variability. We propose a multi-scale input and multi-feature network (MSI-MFNet) model, which can learn the overall structures and texture features of different scale tissues by fusing multi-resolution hierarchical feature maps from the network's dense connectivity structure. The MSI-MFNet predicts the probability of a disease on the patch and image levels. We evaluated the performance of our proposed model on two public benchmark datasets. Furthermore, through ablation studies of the model, we found that multi-scale input and multi-feature maps play an important role in improving the performance of the model. Our proposed model outperformed the existing state-of-the-art models by demonstrating better accuracy, sensitivity, and specificity.
Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!

Topic models are a useful analysis tool to uncover the underlying themes within document collections.The dominant approach is to use probabilistic topic models that posit a generative story, but in this paper we propose an alternative way to obtain topics: clustering pretrained word embeddings while incorporating document information for weighted clustering and reranking top words.We provide benchmarks for the combination of different word embeddings and clustering algorithms, and analyse their performance under dimensionality reduction with PCA.The best performing combination for our approach performs as well as classical topic models, but with lower runtime and computational complexity.
Short-Term Power Prediction of Building Integrated Photovoltaic (BIPV) System Based on Machine Learning Algorithms

One of the biggest challenges is towards ensuring large-scale integration of photovoltaic systems into buildings. This work is aimed at presenting a building integrated photovoltaic system power prediction concerning the building’s various orientations based on the machine learning data science tools. The proposed prediction methodology comprises a data quality stage, machine learning algorithm, weather clustering assessment, and an accuracy assessment. The results showed that the application of linear regression coefficients to the forecast outputs of the developed photovoltaic power generation neural network improved the PV power generation’s forecast output. The final model resulted from accurate forecasts, exhibiting a root mean square error of 4.42% in NN, 16.86% in QSVM, and 8.76% in TREE. The results are presented with the building facade and roof application such as flat roof, south façade, east façade, and west façade.
Information technology governance and cybersecurity at the board level

Security breaches are very costly in the USA, followed very closely by the Middle East. Shareholders and investors demand that their firms mitigate all kinds of risks, and it is the responsibility of the BOD to gain and maintain their confidence. In view of this scenario, MENA companies need to protect their data, while the BODs need to embed a culture of cybersecurity in the firm. The aim of this paper is to examine the relationship between information technology governance (ITG) and the level of cybersecurity by MENA listed firms. The study used a checklist to collect data from a sample of 94 firms listed in the financial stock markets of the MENA countries for the year ended 2018. The study found that there is a significant and direct relationship between ITG and the level of a firm's cybersecurity. This indicates the importance of appointing board members with IT knowledge and experience. This leads to better decisions taken by the BODs when faced with cyber-threats and challenges. In addition, IT expertise on the BODs can be important to understand what the Heads of IT are doing on the inside and, thus being knowledgeable enough to challenge their actions.
Generate Adjective Sentiment Dictionary for Social Media Sentiment Analysis Using Constrained Nonnegative Matrix Factorization

Although sentiment analysis has attracted a lot of research, little work has been done on social media data compared to product and movie reviews. This is due to the low accuracy that results from the more informal writing seen in social media data. Currently, most of sentiment analysis tools on social media choose the lexicon-based approach instead of the machine learning approach because the latter requires the huge challenge of obtaining enough human-labeled training data for extremely large-scale and diverse social opinion data. The lexicon-based approach requires a sentiment dictionary to determine opinion polarity. This dictionary can also provide useful features for any supervised learning method of the machine learning approach. However, many benchmark sentiment dictionaries do not cover the many informal and spoken words used in social media. In addition, they are not able to update frequently to include newly generated words online. In this paper, we present an automatic sentiment dictionary generation method, called Constrained Symmetric Nonnegative Matrix Factorization (CSNMF) algorithm, to assign polarity scores to each word in the dictionary, on a large social media corpus — digg.com. Moreover, we will demonstrate our study of Amazon Mechanical Turk (AMT) on social media word polarity, using both the human-labeled dictionaries from AMT and the General Inquirer Lexicon to compare our generated dictionary with. In our experiment, we show that combining links from both WordNet and the corpus to generate sentiment dictionaries does outperform using only one of them, and the words with higher sentiment scores yield better precision. Finally, we conducted a lexicon-based sentiment analysis on human-labeled social comments using our generated sentiment dictionary to show the effectiveness of our method.