Query: machine learning and parallel computing

Candidate papers:

abstract: Tremendous progress has been witnessed in artificial intelligence within the domain of neural network backed deep learning systems and its applications. As we approach the post Moore's Law era, the limit of semiconductor fabrication technology along with a rapid increase in data generation rates have lead to an impending growing challenge of tackling newer and more modern machine learning problems. In parallel, quantum computing has exhibited rapid development in recent years. Due to the potential of a quantum speedup, quantum based learning applications have become an area of significant interest, in hopes that we can leverage quantum systems to solve classical problems. In this work, we propose a quantum deep learning architecture; we demonstrate our quantum neural network architecture on tasks ranging from binary and multi class classification to generative modelling. Powered by a modified quantum differentiation function along with a hybrid quantum classic design, our architecture encodes the data with a reduced number of qubits and generates a quantum circuit, loading it onto a quantum platform where the model learns the optimal states iteratively. We conduct intensive experiments on both the local computing environment and IBM Q quantum platform. The evaluation results demonstrate that our architecture is able to outperform Tensorflow Quantum by up to 12.51% and 11.71% for a comparable classic deep neural network on the task of classification trained with the same network settings. Furthermore, our GAN architecture runs the discriminator and the generator purely on quantum hardware and utilizes the swap test on qubits to calculate the values of loss functions. In comparing our quantum GAN, we note our architecture is able to achieve similar performance with 98.5% reduction on the parameter set when compared to classical GANs. With the same number of parameters, additionally, QuGAN outperforms other quantum based GANs in the literature for up to 125.0% in terms of similarity between generated distributions and original data sets. Keywords Quantum Machine Learning; Quantum Deep Learning; Quantum Generative Adversarial Networks; Quantum State Fidelity Preprints (www.preprints.org) NOT PEER REVIEWED Posted: 24 March 2021 doi:10.20944/preprints202103.0583.v1
author_names: Samuel A Stein
title: Quantum Computing Aided Machine Learning Through Quantum State Fidelity
year: 2021

abstract: As the training dataset size and the model size of machine learning increase rapidly, more computing resources are consumed to speedup the training process. However, the scalability and performance reproducibility of parallel machine learning training, which mainly uses stochastic optimization algorithms, are limited. In this paper, we demonstrate that the sample difference in the dataset plays a prominent role in the scalability of parallel machine learning algorithms. We propose to use statistical properties of dataset to measure sample differences. These properties include the variance of sample features, sample sparsity, sample diversity, and similarity in sampling sequences. We choose four types of parallel training algorithms as our research objects: (1) the asynchronous parallel SGD algorithm (Hogwild! algorithm) (2) the parallel model average SGD algorithm (minibatch SGD algorithm) (3) the decentralization optimization algorithm, and (4) the dual coordinate optimization (DADM algorithm) Our results show that the statistical properties of training datasets determine the scalability upper bound of these parallel training algorithms.
author_names: Daning Cheng

Shigang Li

Hanping Zhang

Fen Xia

Yunquan Zhang
title: Why Dataset Properties Bound the Scalability of Parallel Machine Learning Training Algorithms
venue: IEEE Transactions on Parallel and Distributed Systems
year: 2021

abstract: This special issue of Concurrency and Computation: Practice and Experience comprises four papers extending the original workshop publications accepted and presented at MPP 2019 (8th Workshop on Parallel Programming Models Special Issue on IoT and Machine Learning) held in Rio de Janeiro in conjunction with IPDPS 2019. The papers represent interesting research ideas of parallel programming models, tools, and optimizations suited for being applied to IoT based applications. The paper titled "Enabling Heterogeneous Ray Tracing Acceleration in Edge/Cloud Architectures" represents a very interesting research on reconfigurable accelerators for Ray Tracing, specialized in computing ray triangle intersections at the network edge of a heterogeneous cloud computing environment. The authors validated their approach on the Xilinx Zynq FPGA platform. The paper titled "An Incremental Reinforcement Learning Scheduling Strategy for Data Intensive Scientific Workflows in the Cloud" is a research work proposing a new scheduling algorithm based on Reinforcement Learning for scientific workflows on HPC distributed resources and Clouds. The paper titled "Latency aware Adaptive Micro Batching Techniques for Streamed Data Compression on GPUs" proposes a set of Autonomic Computing strategies for configuring the optimal parallelism degree and batch size on streaming data compression parallel applications on GPU architectures. Finally, the paper titled "Gamma General Abstract Model for Multiset mAnipulation and Dynamic Dataflow Model: an Equivalence Study" is an interesting research study on new applications of the Gamma Model (General Abstract Model for Multiset mAnipulation) and its joint utilization with the Dataflow programming model. As Guest Editors, we would like to express our gratitude for the valuable contributions made by all authors. We would like to thank all the reviewers who helped us during the thorough review process based on several review rounds. Finally, we would like to thank all the editorial board members and all staff of Concurrency and Computation: Practice and Experience for allowing us to publish our Special Issue and for their invaluable support during the whole publication process.
author_names: Cristiana Barbosa Bentes

Felipe Maia Galvao Franca

Leandro Augusto Justen Marzulo

Gabriele Mencagli

Mauricio Lima Pilla
title: Novel parallel processing techniques for IoT based machine learning applications
venue: Concurr. Comput. Pract. Exp.
year: 2021

abstract: Forecasting the dynamics of large complex networks from previous time series data is important in a wide range of contexts. Here we present a machine learning scheme for this task using a parallel architecture that mimics the topology of the network of interest. We demonstrate the utility and scalability of our method implemented using reservoir computing on a chaotic network of oscillators. Two levels of prior knowledge are considered: (i) the network links are known; and (ii) the network links are unknown and inferred via a data driven approach to approximately optimize prediction.
author_names: Keshav Srinivasan

Nolan J Coble

Joyce L Hamlin

Thomas M Antonsen

Edward Ott

Michelle Girvan
title: Parallel Machine Learning for Forecasting the Dynamics of Complex Networks
venue: ArXiv
year: 2021

abstract: We observe a continuously increased use of Deep Learning (DL) as a specific type of Machine Learning (ML) for data intensive problems (i.e. 'big data' that requires powerful computing resources with equally increasing performance. Consequently, innovative heterogeneous High Performance Computing (HPC) systems based on multi core CPUs and many core GPUs require an architectural design that addresses end user communities' requirements that take advantage of ML and DL. Still the workloads of end user communities of the simulation sciences (e.g. using numerical methods based on known physical laws) needs to be equally supported in those architectures. This paper offers insights into the Modular Supercomputer Architecture (MSA) developed in the Dynamic Exascale Entry Platform (DEEP) series of projects to address the requirements of both simulation sciences and data intensive sciences such as High Performance Data Analytics (HPDA) It shares insights into implementing the MSA in the Julich Supercomputing Centre (JSC) hosting Europe No. 1 Supercomputer Julich Wizard for European Leadership Science (JUWELS) We augment the technical findings with experience and lessons learned from two application communities case studies (i.e. remote sensing and health sciences) using the MSA with JUWELS and the DEEP systems in practice. Thus, the paper provides details into specific MSA design elements that enable significant performance improvements of ML and DL algorithms. While this paper focuses on MSA based HPC systems and application experience, we are not losing sight of advances in Cloud Computing (CC) and Quantum Computing (QC) relevant for ML and DL.
author_names: Morris Riedel

Rocco Sedona

Chadi Barakat

Petur Einarsson

Reza Hassanian

Gabriele Cavallaro

Matthias Book

Helmut Neukirchen

Andreas Lintermann
title: Practice and Experience in using Parallel and Scalable Machine Learning with Heterogenous Modular Supercomputing Architectures
venue: 2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
year: 2021

abstract: Today's systems, especially in the age of federated learning, rely on sending all the data to the cloud, and then use complex algorithms, such as Deep Neural Networks, which require billions of parameters and many hours to train a model. In contrast, the human brain can do much of this learning effortlessly. Hyperdimensional (HD) Computing aims to mimic the behavior of the human brain by utilizing high dimensional representations. This leads to various desirable properties that other Machine Learning (ML) algorithms lack such as: robustness to noise in the system and simple, highly parallel operations. In this paper, we propose HyDREA, a HD computing system that is Robust, Efficient, and Accurate. To evaluate the feasibility of HyDREA in a federated learning environment with wireless communication noise, we utilize NS 3, a popular network simulator that models a real world environment with wireless communication noise. We found that HyDREA is 48x more robust to noise than other comparable ML algorithms. We additionally propose a Processing in Memory (PIM) architecture that adaptively changes the bitwidth of the model based on the signal to noise ratio (SNR) of the incoming sample to maintain the robustness of the HD model while achieving high accuracy and energy efficiency. Our results indicate that our proposed system loses less than 1% classification accuracy, even in scenarios with an SNR of 6.64. Our PIM architecture is also able to achieve 255x better energy efficiency and speed up execution time by 28x compared to the baseline PIM architecture.
author_names: Justin Morris

Kazim Ergun

Behnam Khaleghi

Mohsen Imani

Baris Aksanli

Tajana Simunic
title: HyDREA: Towards More Robust and Efficient Machine Learning Systems with Hyperdimensional Computing
venue: 2021 Design, Automation Test in Europe Conference Exhibition (DATE)
year: 2021

abstract: Elastic scaling in/out of operator parallelism degree is needed for processing real time dynamic data streams under low latency and high stability requirements. Usually the operator parallelism degree is set when a streaming application is submitted to a stream computing system and kept intact during runtime. This may substantially affect the performance of the system due to the fluctuation of input streams and availability of system resources. To address the problems brought by the static parallelism setting, we propose and implement a machine learning based elastic strategy for operator parallelism (named MeStream) in big data stream computing systems. The architecture of Me Stream and its key models are introduced, including parallel bottleneck identification, parameter plan generation, parameter migration and conversion, and instances scheduling. Metrics of execution latency and process latency of the proposed scheduling strategy are evaluated on the widely used big data stream computing system Apache Storm. The experimental results demonstrate the efficiency and effectiveness of the proposed strategy.
author_names: Dawei Sun

Rajkumar Buyya
title: A Machine Learning based Elastic Strategy for Operator Parallelism in a Big Data Stream Computing System
year: 2021

abstract: We present a CMOS imager for low latency face detection empowered by parallel imaging and computing of machine learning (ML) classifiers. The energy efficient parallel operation and multi scale detection eliminate image capture delay and significantly alleviate backend computational loads. The proposed pixel architecture, composed of dynamic samplers in a global shutter (GS) pixel array, allows for energy efficient in memory charge domain computing of feature extraction and classification. The illumination invariant detection was realized by using log Haar features. A prototype 240x240 imager achieved an on chip face detection latency of 5.1ms with a 97.9% true positive rate and 2% false positive rate at 120fps. Moreover, a dynamic nature of in memory computing allows an energy efficiency of 419pJ/pixel for feature extraction and classification, leading to the smallest latency energy product of 3.66msnJ/pixel with digital backend processing.
author_names: Juan Salinas

Euisik Yoon
title: A5.1ms Low Latency Face Detection Imager with In Memory Charge Domain Computing of Machine Learning Classifiers
venue: 2021 Symposium on VLSI Technology
year: 2021

abstract: We present a CMOS imager for low latency face detection empowered by parallel imaging and computing of machine learning (ML) classifiers. The energy efficient parallel operation and multi scale detection eliminate image capture delay and significantly alleviate backend computational loads. The proposed pixel architecture, composed of dynamic samplers in a global shutter (GS) pixel array, allows for energy efficient in memory charge domain computing of feature extraction and classification. The illumination invariant detection was realized by using log Haar features. A prototype 240x240 imager achieved an on chip face detection latency of 5.1ms with a 97.9% true positive rate and 2% false positive rate at 120fps. Moreover, a dynamic nature of in memory computing allows an energy efficiency of 419pJ/pixel for feature extraction and classification, leading to the smallest latency energy product of 3.66msnJ/pixel with digital backend processing.
author_names: Hyunsoo Song

Sungjin Oh

Juan Salinas

Euisik Yoon
title: A 5.1ms Low Latency Face Detection Imager with In Memory Charge Domain Computing of Machine Learning Classifiers
venue: 2021 Symposium on VLSI Circuits
year: 2021

abstract: To detect defect is an important concept in machine leaning techniques and ambiguous dataset which develops into a challenging issue, as the software product expands in terms of size and its complexity. This chapter reveals an applied novel multi learner model which is ensembled to predict software metrics using classification algorithms and propose algorithm applied in parallel method for detection on ambiguous data using density sampling and develop an implementation running on both GPUs and multi core CPUs. The defect on the NASA PROMISE defect dataset is adequately predicted and classified using these models and implementing GPU computing. The performance compared to the traditional learning models improved algorithm and parallel implementation on GPUs shows less processing time in ensemble model compared to decision tree algorithm and effectively optimizes the true positive rate.
author_names: Sivakumar S

Sreedevi E

PremaLatha V

Haritha D
title: Parallel Defect Detection Model on Uncertain Data for GPUs Computing by a Novel Ensemble Learning
year: 2021