Reviewerly/exHarmony huggingface dataset:
Mish: A Self Regularized Non-Monotonic Neural Activation Function

The concept of non-linearity in a Neural Network is introduced by anactivation function which serves an integral role in the training andperformance evaluation of the network. Over the years of theoretical research,many activation functions have been proposed, however, only a few are widelyused in mostly all applications which include ReLU (Rectified Linear Unit),TanH (Tan Hyperbolic), Sigmoid, Leaky ReLU and Swish. In this work, a novelneural activation function called as Mish is proposed. The experiments showthat Mish tends to work better than both ReLU and Swish along with otherstandard activation functions in many deep networks across challengingdatasets. For instance, in Squeeze Excite Net- 18 for CIFAR 100 classification,the network with Mish had an increase in Top-1 test accuracy by 0.494% and1.671% as compared to the same network with Swish and ReLU respectively. Thesimilarity to Swish along with providing a boost in performance and itssimplicity in implementation makes it easier for researchers and developers touse Mish in their Neural Network Models.
Temporal Convolution for Real-Time Keyword Spotting on Mobile Devices

Keyword spotting (KWS) plays a critical role in enabling speech-based user interactions on smart devices. Recent developments in the field of deep learning have led to wide adoption of convolutional neural networks (CNNs) in KWS systems due to their exceptional accuracy and robustness. The main challenge faced by KWS systems is the trade-off between high accuracy and low latency. Unfortunately, there has been little quantitative analysis of the actual latency of KWS models on mobile devices. This is especially concerning since conventional convolution-based KWS approaches are known to require a large number of operations to attain an adequate level of performance. In this paper, we propose a temporal convolution for real-time KWS on mobile devices. Unlike most of the 2D convolution-based KWS approaches that require a deep architecture to fully capture both low- and high-frequency domains, we exploit temporal convolutions with a compact ResNet architecture. In Google Speech Command Dataset, we achieve more than \textbf{385x} speedup on Google Pixel 1 and surpass the accuracy compared to the state-of-the-art model. In addition, we release the implementation of the proposed and the baseline models including an end-to-end pipeline for training models and evaluating them on mobile devices.
Nonlinear Information Bottleneck

Information bottleneck (IB) is a technique for extracting information in one random variable X that is relevant for predicting another random variable Y. IB works by encoding X in a compressed “bottleneck” random variable M from which Y can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete X and Y with small state spaces, and continuous X and Y with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and/or continuous X and Y, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed “variational IB” method on several real-world datasets.
Choice of clear-sky model in solar forecasting

This paper is concerned with the choice of clear-sky model in solar forecasting. This issue is discussed from three perspectives: (1) accessibility, (2) forecast performance, and (3) statistical properties. Accessibility refers to the time and effort involved in obtaining clear-sky irradiance through a clear-sky model. Forecast performance is analyzed through a new concept called “mean square error (MSE) scaling,” which allows one to decompose the MSE of reference irradiance forecasts into three terms, each carrying a notion of predictability. The decomposition, however, resides on the assumption that the clear-sky index time series is stationary. In this regard, the stationarity assumption is investigated using statistical hypotheses. It is found that even the best clear-sky models, such as the REST2 model, are not able to produce a stationary clear-sky index time series. Instead, the time series is only local stationary, which, in the present context, means that its statistical properties change slowly with the value of clear-sky irradiance. Contrary to the common belief that a better clear-sky model leads to better forecasts, no evidence suggests that the more intricate REST2 has an advantage over the simpler Ineichen–Perez model, in terms of forecast performance. In that, accessibility becomes the primary concern when opting a clear-sky model for forecasting purposes. At this point, the McClear model, available as a web service for worldwide locations at 1-, 15-, and 60-min resolutions, is highly recommended.
LoRAS: an oversampling approach for imbalanced datasets

Abstract The Synthetic Minority Oversampling TEchnique (SMOTE) is widely-used for the analysis of imbalanced datasets. It is known that SMOTE frequently over-generalizes the minority class, leading to misclassifications for the majority class, and effecting the overall balance of the model. In this article, we present an approach that overcomes this limitation of SMOTE, employing Localized Random Affine Shadowsampling (LoRAS) to oversample from an approximated data manifold of the minority class. We benchmarked our algorithm with 14 publicly available imbalanced datasets using three different Machine Learning (ML) algorithms and compared the performance of LoRAS, SMOTE and several SMOTE extensions that share the concept of using convex combinations of minority class data points for oversampling with LoRAS. We observed that LoRAS, on average generates better ML models in terms of F1-Score and Balanced accuracy. Another key observation is that while most of the extensions of SMOTE we have tested, improve the F1-Score with respect to SMOTE on an average, they compromise on the Balanced accuracy of a classification model. LoRAS on the contrary, improves both F1 Score and the Balanced accuracy thus produces better classification models. Moreover, to explain the success of the algorithm, we have constructed a mathematical framework to prove that LoRAS oversampling technique provides a better estimate for the mean of the underlying local data distribution of the minority class data space.
Software Defect Prediction Using Supervised Machine Learning and Ensemble Techniques: A Comparative Study

An essential objective of software development is to locate and fix defects ahead of schedule that could be expected under diverse circumstances.Many software development activities are performed by individuals, which may lead to different software bugs over the development to occur, causing disappointments in the not-so-distant future.Thus, the prediction of software defects in the first stages has become a primary interest in the field of software engineering.Various software defect prediction (SDP) approaches that rely on software metrics have been proposed in the last two decades.Bagging, support vector machines (SVM), decision tree (DS), and random forest (RF) classifiers are known to perform well to predict defects.This paper studies and compares these supervised machine learning and ensemble classifiers on 10 NASA datasets.The experimental results showed that, in the majority of cases, RF was the best performing classifier compared to the others.
Adaptive Gradient-Based Meta-Learning Methods

We build a theoretical framework for designing and understanding practical meta-learning methods that integrates sophisticated formalizations of task-similarity with the extensive literature on online convex optimization and sequential prediction algorithms. Our approach enables the task-similarity to be learned adaptively, provides sharper transfer-risk bounds in the setting of statistical learning-to-learn, and leads to straightforward derivations of average-case regret bounds for efficient algorithms in settings where the task-environment changes dynamically or the tasks share a certain geometric structure. We use our theory to modify several popular meta-learning algorithms and improve their training and meta-test-time performance on standard problems in few-shot and federated learning.
V-NAS: Neural Architecture Search for Volumetric Medical Image Segmentation

Deep learning algorithms, in particular 2D and 3D fully convolutional neural networks (FCNs), have rapidly become the mainstream methodology for volumetric medical image segmentation. However, 2D convolutions cannot fully leverage the rich spatial information along the third axis, while 3D convolutions suffer from the demanding computation and high GPU memory consumption. In this paper, we propose to automatically search the network architecture tailoring to volumetric medical image segmentation problem. Concretely, we formulate the structure learning as differentiable neural architecture search, and let the network itself choose between 2D, 3D or Pseudo-3D (P3D) convolutions at each layer. We evaluate our method on 3 public datasets, i.e., the NIH Pancreas dataset, the Lung and Pancreas dataset from the Medical Segmentation Decathlon (MSD) Challenge. Our method, named V-NAS, consistently outperforms other state-of-the-arts on the segmentation tasks of both normal organ (NIH Pancreas) and abnormal organs (MSD Lung tumors and MSD Pancreas tumors), which shows the power of chosen architecture. Moreover, the searched architecture on one dataset can be well generalized to other datasets, which demonstrates the robustness and practical use of our proposed method.
t-viSNE: Interactive Assessment and Interpretation of t-SNE Projections

t-Distributed Stochastic Neighbor Embedding (t-SNE) for the visualization of multidimensional data has proven to be a popular approach, with successful applications in a wide range of domains. Despite their usefulness, t-SNE projections can be hard to interpret or even misleading, which hurts the trustworthiness of the results. Understanding the details of t-SNE itself and the reasons behind specific patterns in its output may be a daunting task, especially for non-experts in dimensionality reduction. In this work, we present t-viSNE, an interactive tool for the visual exploration of t-SNE projections that enables analysts to inspect different aspects of their accuracy and meaning, such as the effects of hyper-parameters, distance and neighborhood preservation, densities and costs of specific neighborhoods, and the correlations between dimensions and visual patterns. We propose a coherent, accessible, and well-integrated collection of different views for the visualization of t-SNE projections. The applicability and usability of t-viSNE are demonstrated through hypothetical usage scenarios with real data sets. Finally, we present the results of a user study where the tool's effectiveness was evaluated. By bringing to light information that would normally be lost after running t-SNE, we hope to support analysts in using t-SNE and making its results better understandable.
Blockchain for next generation services in banking and finance: cost, benefit, risk and opportunity analysis

Purpose The purpose of this paper is to help in providing a better understanding of the application of blockchain technology in the context of the banking and finance sectors. The aim is to outline blockchain's benefits, opportunities, costs, risks as well as challenges of the technology in the context of banking and finance services. Design/methodology/approach Careful examination of the extant literature, including utilising relevant academic-based research databases has been carried out. It covered reviewing various research contributions published in peer-reviewed journals, academic reports, as well as technical reports to help in identifying related benefits, opportunities, costs and risks. Findings The findings reveal that there are limited contributions in utilising blockchain in the banking and finance sectors when compared with other sectors. As such, the study highlighted the relevant perspective of benefits, opportunities, costs and risks within such sectors. Practical implications This study helps in offering a focal point to banking and financial sector managers and decision-makers for realising the value and offerings of blockchain technology as well as associated strategies and programmes. Originality/value This study highlights the need for a holistic understanding of the various aspects of cost, benefits, risk and opportunities to create blockchain applications that work for banking and finance sectors
Practical trapped-ion protocols for universal qudit-based quantum computing

This work uses multi-level qudit systems for quantum computation in the context of trapped ions and show all of the necessary operations for qudit quantum computation with trapped ions along with realistic error estimates.
Computer-Aided Diagnosis in Histopathological Images of the Endometrium Using a Convolutional Neural Network and Attention Mechanisms

Uterine cancer (also known as endometrial cancer) can seriously affect the female reproductive system, and histopathological image analysis is the gold standard for diagnosing endometrial cancer. Due to the limited ability to model the complicated relationships between histopathological images and their interpretations, existing computer-aided diagnosis (CAD) approaches using traditional machine learning algorithms often failed to achieve satisfying results. In this study, we develop a CAD approach based on a convolutional neural network (CNN) and attention mechanisms, called HIENet. In the ten-fold cross-validation on ~3,300 hematoxylin and eosin (H&E) image patches from ~500 endometrial specimens, HIENet achieved a 76.91 ± 1.17% (mean ± s. d.) accuracy for four classes of endometrial tissue, i.e., normal endometrium, endometrial polyp, endometrial hyperplasia, and endometrial adenocarcinoma. Also, HIENet obtained an area-under-the-curve (AUC) of 0.9579 ± 0.0103 with an 81.04 ± 3.87% sensitivity and 94.78 ± 0.87% specificity in a binary classification task that detected endometrioid adenocarcinoma. Besides, in the external validation on 200 H&E image patches from 50 randomly-selected female patients, HIENet achieved an 84.50% accuracy in the four-class classification task, as well as an AUC of 0.9829 with a 77.97% (95% confidence interval, CI, 65.27%~87.71%) sensitivity and 100% (95% CI, 97.42%~100.00%) specificity. The proposed CAD method outperformed three human experts and five CNN-based classifiers regarding overall classification performance. It was also able to provide pathologists better interpretability of diagnoses by highlighting the histopathological correlations of local pixel-level image features to morphological characteristics of endometrial tissue.
Factual Probing Is [MASK]: Learning vs. Learning to Recall

Petroni et al. (2019) demonstrated that it is possible to retrieve world facts from a pretrained language model by expressing them as cloze-style prompts and interpret the model's prediction accuracy as a lower bound on the amount of factual information it encodes.Subsequent work has attempted to tighten the estimate by searching for better prompts, using a disjoint set of facts as training data.In this work, we make two complementary contributions to better understand these factual probing techniques.First, we propose OPTIPROMPT, a novel and efficient method which directly optimizes in continuous embedding space.We find this simple method is able to predict an additional 6.4% of facts in the LAMA benchmark.Second, we raise a more important question: Can we really interpret these probing results as a lower bound?Is it possible that these prompt-search methods learn from the training data too?We find, somewhat surprisingly, that the training data used by these methods contains certain regularities of the underlying fact distribution, and all the existing prompt methods, including ours, are able to exploit them for better fact prediction.We conduct a set of control experiments to disentangle "learning" from "learning to recall", providing a more detailed picture of what different prompts can reveal about pre-trained language models.
Answering Complex Open-domain Questions Through Iterative Query Generation

It is challenging for current one-step retrieveand-read question answering (QA) systems to answer questions like "Which novel by the author of 'Armada' will be adapted as a feature film by Steven Spielberg?" because the question seldom contains retrievable clues about the missing entity (here, the author).Answering such a question requires multi-hop reasoning where one must gather information about the missing entity (or facts) to proceed with further reasoning.We present GOLDEN (Gold Entity) Retriever, which iterates between reading context and retrieving more supporting documents to answer open-domain multi-hop questions.Instead of using opaque and computationally expensive neural retrieval models, GOLDEN Retriever generates natural language search queries given the question and available context, and leverages off-the-shelf information retrieval systems to query for missing entities.This allows GOLDEN Retriever to scale up efficiently for open-domain multi-hop reasoning while maintaining interpretability.We evaluate GOLDEN Retriever on the recently proposed open-domain multi-hop QA dataset, HOTPOTQA, and demonstrate that it outperforms the best previously published model despite not using pretrained language models such as BERT.* Equal contribution, order decided by a random number generator.
Faster Randomized Block Kaczmarz Algorithms

The Kaczmarz algorithm is a simple iterative scheme for solving consistent linear systems. At each step, the method projects the current iterate onto the solution space of a single constraint. Hence, it requires low cost per iteration and storage, and it has a linear rate of convergence. Distributed implementations of Kaczmarz have recently become the de facto architectural choice for large-scale linear systems. Therefore, in this paper we develop a family of randomized block Kaczmarz algorithms that uses at each step a subset of the constraints and extrapolated stepsizes, and can be deployed on distributed computing units. Our approach is based on several new ideas and tools, including stochastic selection rules for the blocks of rows, stochastic conditioning of linear systems, and novel strategies for designing extrapolated stepsizes. We prove that randomized block Kaczmarz algorithms converge linearly in expectation, with a rate depending on the geometric properties of the matrix and its submatrices and on the size of the blocks. Our convergence analysis reveals that the algorithm is most effective when it is given a good sampling of the rows into well-conditioned blocks. Besides providing a general framework for the design and analysis of randomized block Kaczmarz methods, our results resolve an open problem in the literature related to the theoretical understanding of observed practical efficiency of extrapolated block Kaczmarz methods. We also propose an accelerated block Kaczmarz scheme, that is, acceleration in the sense of Chebyshev semi-iterative methods, where the stepsize is chosen based on the roots of Chebyshev polynomials, and we derive convergence rates depending on the square root of the geometric properties of the matrix. Finally, numerical examples illustrate the benefits of the new algorithms.
Self-Supervised Learning for Generalizable Out-of-Distribution Detection

The real-world deployment of Deep Neural Networks (DNNs) in safety-critical applications such as autonomous vehicles needs to address a variety of DNNs' vulnerabilities, one of which being detecting and rejecting out-of-distribution outliers that might result in unpredictable fatal errors. We propose a new technique relying on self-supervision for generalizable out-of-distribution (OOD) feature learning and rejecting those samples at the inference time. Our technique does not need to pre-know the distribution of targeted OOD samples and incur no extra overheads compared to other methods. We perform multiple image classification experiments and observe our technique to perform favorably against state-of-the-art OOD detection methods. Interestingly, we witness that our method also reduces in-distribution classification risk via rejecting samples near the boundaries of the training set distribution.
NVAE: A Deep Hierarchical Variational Autoencoder

Normalizing flows, autoregressive models, variational autoencoders (VAEs), and deep energy-based models are among competing likelihood-based frameworks for deep generative learning. Among them, VAEs have the advantage of fast and tractable sampling and easy-to-access encoding networks. However, they are currently outperformed by other models such as normalizing flows and autoregressive models. While the majority of the research in VAEs is focused on the statistical challenges, we explore the orthogonal direction of carefully designing neural architectures for hierarchical VAEs. We propose Nouveau VAE (NVAE), a deep hierarchical VAE built for image generation using depth-wise separable convolutions and batch normalization. NVAE is equipped with a residual parameterization of Normal distributions and its training is stabilized by spectral regularization. We show that NVAE achieves state-of-the-art results among non-autoregressive likelihood-based models on the MNIST, CIFAR-10, CelebA 64, and CelebA HQ datasets and it provides a strong baseline on FFHQ. For example, on CIFAR-10, NVAE pushes the state-of-the-art from 2.98 to 2.91 bits per dimension, and it produces high-quality images on CelebA HQ. To the best of our knowledge, NVAE is the first successful VAE applied to natural images as large as 256$\times$256 pixels. The source code is available at https://github.com/NVlabs/NVAE .
&lt;p&gt;COVID-19 Contact-Tracing Technology: Acceptability and Ethical Issues of Use&lt;/p&gt;

The contact-tracing COVID-19 technology allows for tracing people that come in contact to individuals with COVID-19 wherever they are located. The number of tracing COVID-19 infection technology and devices is rapidly increasing. This has prompted many researchers to study the acceptability and ethical issues related to the implementation of such technology.The purpose of this study was to determine the acceptability of COVID-19 contact-tracing technology and ethical issues of use.A cross-sectional questionnaire-based study was used. The target population was Jordanian adults (>18 years). The survey was distributed to a convenience sample of 2000 general public in Jordan.The results found that the number of people who accept to use COVID-19 contact-tracing technology was 71.6%. However, the percentage of people who were using this technology was 37.8. The main ethical concerns for many of participants were privacy, voluntariness, and beneficence of the data. Only income and living area were predictors for acceptability and use of tracing technology (p≤ 0.01).The majority of Jordanians accept the implementation of contact-tracing technology for COVID-19 infection. Among ethical concerns of the implementation of such technology were privacy, beneficence and voluntariness.The results of this study would help in improving the state of science regarding acceptability to use contact-tracing technology for health purposes. Moreover, the present findings provide evidence of predictors of acceptance and ethical concerns among Jordanian population about COVID-19 contact-tracing technology.