pinecone/core-2020-05-10-deduplication huggingface dataset:
Large improvements in application throughput of long-running multi-component applications using batch grids

Computational grids with multiple batch systems (batch grids) can be powerful infrastructures for executing long-running multi-component parallel applications. In this paper, we evaluate the potential improvements in throughput of long-running multi-component applications when the different components of the applications are executed on multiple batch systems of batch grids. We compare the multiple batch executions with executions of the components on a single batch system without increasing the number of processors used for executions. We perform our analysis with a foremost long-running multi-component application for climate modeling, the Community Climate System Model (CCSM). We have built a robust simulator that models the characteristics of both the multi-component application and the batch systems. By conducting large number of simulations with different workload characteristics and queuing policies of the systems, processor allocations to components of the application, distributions of the components to the batch systems and inter-cluster bandwidths, we show that multiple batch executions lead to 55% average increase in throughput over single batch executions for long-running CCSM. We also conducted real experiments with a practical middleware infrastructure and showed that multi-site executions lead to effective utilization of batch systems for executions of CCSM and give higher simulation throughput than single-site executions. Copyright (c) 2011 John Wiley & Sons, Ltd
Resource Analyses for Parallel and Distributed Coordination

Predicting the resources that are consumed by a program component is crucial for many parallel or distributed systems. In this context, the main resources of interest are execution time, space and communication/synchronisation costs. There has recently been significant progress in resource analysis technology, notably in type-based analyses and abstract interpretation. At the same time, parallel and distributed computing are becoming increasingly important. This paper synthesises progress in both areas to survey the state-of-the-art in resource analysis for parallel and distributed computing. We articulate a general model of resource analysis and describe parallel/distributed resource analysis together with the relationship to sequential analysis. We use three parallel or distributed resource analyses as examples and provide a critical evaluation of the analyses. We investigate why the chosen analysis is effective for each application and identify general principles governing why the resource analysis is effective.PostprintPeer reviewe
A new solution algorithm for solving rule-sets based bilevel decision problems

Bilevel decision addresses compromises between two interacting decision entities within a given hierarchical complex system under distributed environments. Bilevel programming typically solves bilevel decision problems. However, formulation of objectives and constraints in mathematical functions is required, which are difficult, and sometimes impossible, in real-world situations because of various uncertainties. Our study develops a rule-set based bilevel decision approach, which models a bilevel decision problem by creating, transforming and reducing related rule sets. This study develops a new rule-sets based solution algorithm to obtain an optimal solution from the bilevel decision problem described by rule sets. A case study and a set of experiments illustrate both functions and the effectiveness of the developed algorithm in solving a bilevel decision problem. © 2012 John Wiley & Sons, Ltd
Evaluation of intra-group optimistic data replication in P2P groupware systems

This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving http://olabout.wiley.com/WileyCDA/Section/id-820227.htmlPeer-to-peer (P2P) computing systems have become very popular during the last years because of their ability to scale to a large number of users and efficient communication among peers. They can support complex computational processes, beyond simple file sharing, while offering advantages of decentralized distributed systems. However, such systems may suffer from availability and reliability. To increase availability and reliability, and therefore, improve the perception of peers, yielding to fast response times and rich experience, data replication techniques are the foremost means in such systems. Indeed, in many P2P applications, for example, in a groupware, documents generated along application life cycle can change over time. The need is then to efficiently replicate dynamic documents and data to support group processes and collaboration. In this paper, we propose a replication system for documents structured as XML files and evaluate it under different scenarios. The proposed system has a super-peer architecture that provides fast consistency for late joining peers. It uses optimistic replication techniques with propagating update operations from source to destination node in push mode. The system is suitable for asynchronous collaboration in online collaborative teams accomplishing a common project in a P2P environment.Peer ReviewedPostprint (author's final draft
Evaluating the effectiveness of the abstract transaction model in testing Web services transactions

Web services transactions are used to build efficient and reliable web applications which are distributed across the Internet and are accessed by multiple simultaneous users. Current research develops various models and protocols in order to improve the performance and reliability of web services transactions. However, there is little research on testing the different models and protocols of web services transactions. This paper presents an abstract transaction model that patterns different web services transactions standards. This model is capable of deriving concrete models in order to automatically generate test cases for different web services transactions standards. The proposed model is implemented as a prototype system and is evaluated using a case of the Jboss Transaction. The evaluation shows that the proposed system has the capability to automatically generate test cases and detect possible failures of transactions running under different web services transactions standard
Using Premia and Nsp for Constructing a Risk Management Benchmark for Testing Parallel Architecture

International audienceFinancial institutions have massive computations to carry out overnight which are very demanding in terms of the consumed CPU. The challenge is to price many different products on a cluster-like architecture. We have used the Premia software to valuate the financial derivatives. In this work, we explain how Premia can be embedded into Nsp, a scientific software like Matlab, to provide a powerful tool to valuate a whole portfolio. Finally, we have integrated an MPI toolbox into Nsp to enable to use Premia to solve a bunch of pricing problems on a cluster. This unified framework can then be used to test different parallel architectures
Parallel classification and feature selection in microarray data using SPRINT.

The statistical language R is favoured by many biostatisticians for processing microarray data. In recent times, the quantity of data that can be obtained in experiments has risen significantly, making previously fast analyses time consuming or even not possible at all with the existing software infrastructure. High performance computing (HPC) systems offer a solution to these problems but at the expense of increased complexity for the end user. The Simple Parallel R Interface is a library for R that aims to reduce the complexity of using HPC systems by providing biostatisticians with drop‐in parallelised replacements of existing R functions. In this paper we describe parallel implementations of two popular techniques: exploratory clustering analyses using the random forest classifier and feature selection through identification of differentially expressed genes using the rank product method
Matrix inversion on CPU–GPU platforms with applications in control theory

In this paper, we tackle the inversion of large-scale dense matrices via conventional matrix factorizations (LU, Cholesky, and LDLT) and the Gauss–Jordan method on hybrid platforms consisting of a multicore CPU and a many-core graphics processor (GPU). Specifically, we introduce the different matrix inversion algorithms by using a unified framework based on the notation from the FLAME project; we develop hybrid implementations for those matrix operations underlying the algorithms, alternative to those in existing libraries for single GPU systems; and we perform an extensive experimental study on a platform equipped with state-of-the-art general-purpose architectures from Intel (Santa Clara, CA, USA) and a ‘Fermi’ GPU from NVIDIA (Santa Clara, CA, USA) that exposes the efficiency of the different inversion approaches. Our study and experimental results show the simplicity and performance advantage of the Gauss–Jordan elimination-based inversion methods and the difficulties associated with the symmetric indefinite case
The Speedup-Test: A Statistical Methodology for Program Speedup Analysis and Computation

Article first published online: 15 OCT 2012International audienceIn the area of high performance computing and embedded systems, numerous code optimisation methods exist to accelerate the speed of the computation (or optimise another performance criteria). They are usually experimented by doing multiple observations of the initial and the optimised execution times of a program in order to declare a speedup. Even with fixed input and execution environment, program execution times vary in general. Hence different kinds of speedups may be reported: the speedup of the average execution time, the speedup of the minimal execution time, the speedup of the median, etc. Many published speedups in the literature are observations of a set of experiments. In order to improve the reproducibility of the experimental results, this article presents a rigorous statistical methodology regarding program performance analysis. We rely on well known statistical tests (Shapiro-wilk's test, Fisher's F-test, Student's t-test, Kolmogorov-Smirnov's test, Wilcoxon-Mann-Whitney's test) to study if the observed speedups are statistically significant or not. By fixing 0Y]>1/2, the probability that an individual execution of the optimised code is faster than the individual execution of the initial code. In addition, we can compute the confidence interval of the probability to get a speedup on a randomly selected benchmark that does not belong to the initial set of tested benchmarks. Our methodology defines a consistent improvement compared to the usual performance analysis method in high performance computing. We explain in each situation what are the hypothesis that must be checked to declare a correct risk level for the statistics. The Speedup-Test protocol certifying the observed speedups with rigorous statistics is implemented and distributed as an open source tool based on R software
Distribution of Random Streams for Simulation Practitioners

International audienceThere is an increasing interest in the distribution of parallel random number streamsin the high-performance computing community particularly, with the manycore shift. Even ifwe have at our disposal statistically sound random number generators according to the latestand thorough testing libraries, their parallelization can still be a delicate problem. Indeed, aset of recent publications shows it still has to be mastered by the scientific community. Withthe arrival of multi-core and manycore processor architectures on the scientist desktop, modelerswho are non-specialists in parallelizing stochastic simulations need help and advice in distributingrigorously their experimental plans and replications according to the state of the art in pseudo-random numbers parallelization techniques. In this paper, we discuss the different partitioningtechniques currently in use to provide independent streams with their corresponding software. Inaddition to the classical approaches in use to parallelize stochastic simulations on regular processors,this paper also presents recent advances in pseudo-random number generation for general-purposegraphical processing units. The state of the art given in this paper is written for simulationpractitioners
Interoperating Grid Infrastructures with the GridWay\ud
Metascheduler

This paper describes the GridWay Metascheduler and exposes its latest and future developments, mainly related to interoperability and interoperation. GridWay enables large-scale, reliable and efficient sharing of computing resources over grid middleware. To favor interoperability, it shows a modular architecture based on drivers, which access middleware services for resource discovery and monitoring, job execution and management, and file transfer. This paper presents two new execution drivers for BES and CREAM services, and introduces a remote BES interface for GridWay. This interface allows users to access GridWay’s job metascheduling capabilities, using the BES implementation of GridSAM. Thus, GridWay now provides to end-users more possibilities of interoperability and interoperation
Accelerating R-based Analytics on the Cloud

This paper addresses how the benefits of cloud-based infrastructure can be
harnessed for analytical workloads. Often the software handling analytical
workloads is not developed by a professional programmer, but on an ad hoc basis
by Analysts in high-level programming environments such as R or Matlab. The
goal of this research is to allow Analysts to take an analytical job that
executes on their personal workstations, and with minimum effort execute it on
cloud infrastructure and manage both the resources and the data required by the
job. If this can be facilitated gracefully, then the Analyst benefits from
on-demand resources, low maintenance cost and scalability of computing
resources, all of which are offered by the cloud. In this paper, a Platform for
Parallel R-based Analytics on the Cloud (P2RAC) that is placed between an
Analyst and a cloud infrastructure is proposed and implemented. P2RAC offers a
set of command-line tools for managing the resources, such as instances and
clusters, the data and the execution of the software on the Amazon Elastic
Computing Cloud infrastructure. Experimental studies are pursued using two
parallel problems and the results obtained confirm the feasibility of employing
P2RAC for solving large-scale analytical problems on the cloud.Comment: Concurrency and Computation, 201
TomusBlobs: Scalable Data-intensive Processing on Azure Clouds

International audienceThe emergence of cloud computing has brought the opportunity to use large-scale compute infrastructures for a broader and broader spectrum of applications and users. As the cloud paradigm gets attractive for the "elasticity'' in resource usage and associated costs (the users only pay for resources actually used), cloud applications still suffer from the high latencies and low performance of cloud storage services. As Big Data analysis on clouds becomes more and more relevant in many application areas, enabling high-throughput massive data processing on cloud data becomes a critical issue, as it impacts the overall application performance. In this paper we address this challenge at the level of cloud storage. We introduce a concurrency-optimized data storage system (called TomusBlobs) which federates the virtual disks associated to the Virtual Machines running the application code on the cloud. We demonstrate the performance benefits of our solution for efficient data-intensive processing by building an optimized prototype MapReduce framework for Microsoft's Azure cloud platform based on TomusBlobs. Finally, we specifically address the limitations of state-of-the-art MapReduce frameworks for reduce-intensive workloads, by proposing MapIterativeReduce as an extension of the MapReduce model. We validate the above contributions through large-scale experiments with synthetic benchmarks and with real-world applications on the Azure commercial cloud, using resources distributed across multiple data centers: they demonstrate that our solutions bring substantial benefits to data intensive applications compared to approaches relying on state-of-the-art cloud object storage
Provenance and data differencing for workflow reproducibility analysis

One of the foundations of science is that researchers must publish the
methodology used to achieve their results so that others can attempt to
reproduce them. This has the added benefit of allowing methods to be adopted
and adapted for other purposes. In the field of e-Science, services -- often
choreographed through workflow, process data to generate results. The
reproduction of results is often not straightforward as the computational
objects may not be made available or may have been updated since the results
were generated. For example, services are often updated to fix bugs or improve
algorithms. This paper addresses these problems in three ways. Firstly, it
introduces a new framework to clarify the range of meanings of
"reproducibility". Secondly, it describes a new algorithm, \PDIFF, that uses a
comparison of workflow provenance traces to determine whether an experiment has
been reproduced; the main innovation is that if this is not the case then the
specific point(s) of divergence are identified through graph analysis,
assisting any researcher wishing to understand those differences. One key
feature is support for user-defined, semantic data comparison operators.
Finally, the paper describes an implementation of \PDIFF that leverages the
power of the e-Science Central platform which enacts workflows in the cloud. As
well as automatically generating a provenance trace for consumption by \PDIFF,
the platform supports the storage and re-use of old versions of workflows, data
and services; the paper shows how this can be powerfully exploited in order to
achieve reproduction and re-use
Analysis of power-saving techniques over a large multi-use cluster with variable workload.

Reduction of power consumption for any computer system is now an important issue, although this should be carried out in a manner that is not detrimental to the users of that computer system. We present a number of policies that can be applied to multi-use clusters where computers are shared between interactive users and high-throughput computing. We evaluate policies by trace-driven simulations to determine the effects on power consumed by the high-throughput workload and impact on high-throughput users. We further evaluate these policies for higher workloads by synthetically generating workloads based around the profiled workload observed through our system. We demonstrate that these policies could save 55% of the currently used energy for our high-throughput jobs over our current cluster policies without affecting the high-throughput users’ experience
Applying the dynamics of evolution to achieve reliability in master-worker computing

We consider Internet-based master-worker task computations, such as SETI@home, where a master process sends tasks, across the Internet, to worker processes; workers execute and report back some result. However, these workers are not trustworthy, and it might be at their best interest to report incorrect results. In such master-worker computations, the behavior and the best interest of the workers might change over time. We model such computations using evolutionary dynamics, and we study the conditions under which the master can reliably obtain task results. In particular, we develop and analyze an algorithmic mechanism based on reinforcement learning to provide workers with the necessary incentives to eventually become truthful. Our analysis identifies the conditions under which truthful behavior can be ensured and bounds the expected convergence time to that behavior. The analysis is complemented with illustrative simulations.This work is supported by the Cyprus Research Promotion Foundation grant TΠE/ΠΛHPO/0609(BE)/05, the National Science Foundation (CCF-0937829, CCF-1114930), Comunidad de Madrid grants S2009TIC-1692 and MODELICO-CM, Spanish PRODIEVO and RESINEE grants and MICINN grant EC2011-29688-C02-01, and National Natural Science Foundation of China grant 61020106002.Publicad
Approaching the Internet of things (IoT): a modelling, analysis and abstraction framework

Summary The evolution of communication protocols, sensory hardware, mobile and pervasive devices, alongside social and cyber‐physical networks, has made the Internet of things (IoT) an interesting concept with inherent complexities as it is realised. Such complexities range from addressing mechanisms to information management and from communication protocols to presentation and interaction within the IoT. Although existing Internet and communication models can be extended to provide the basis for realising IoT, they may not be sufficiently capable to handle the new paradigms that IoT introduces, such as social communities, smart spaces, privacy and personalisation of devices and information, modelling and reasoning. With interaction models in IoT moving from the orthodox service consumption model, towards an interactive conversational model, nature‐inspired computational models appear to be candidate representations. Specifically, this research contests that the reactive and interactive nature of IoT makes chemical reaction‐inspired approaches particularly well suited to such requirements. This paper presents a chemical reaction‐inspired computational model using the concepts of graphs and reflection, which attempts to address the complexities associated with the visualisation, modelling, interaction, analysis and abstraction of information in the IoT. Copyright © 2013 John Wiley & Sons, Ltd
CloudMon: a resource-efficient IaaS cloud monitoring system based on networked intrusion detection system virtual appliances

The networked intrusion detection system virtual appliance (NIDS-VA), also known as virtualized NIDS, plays an important role in the protection and safeguard of IaaS cloud environments. However, it is nontrivial to guarantee both of the performance of NIDS-VA and the resource efficiency of cloud applications because both are sharing computing resources in the same cloud environment. To overcome this challenge and trade-off, we propose a novel system, named CloudMon, which enables dynamic resource provision and live placement for NIDS-VAs in IaaS cloud environments. CloudMon provides two techniques to maintain high resource efficiency of IaaS cloud environments without degrading the performance of NIDS-VAs and other virtual machines (VMs). The first technique is a virtual machine monitor based resource provision mechanism, which can minimize the resource usage of a NIDS-VA with given performance guarantee. It uses a fuzzy model to characterize the complex relationship between performance and resource demands of a NIDS-VA and develops an online fuzzy controller to adaptively control the resource allocation for NIDS-VAs under varying network traffic. The second one is a global resource scheduling approach for optimizing the resource efficiency of the entire cloud environments. It leverages VM migration to dynamically place NIDS-VAs and VMs. An online VM mapping algorithm is designed to maximize the resource utilization of the entire cloud environment. Our virtual machine monitor based resource provision mechanism has been evaluated by conducting comprehensive experiments based on Xen hypervisor and Snort NIDS in a real cloud environment. The results show that the proposed mechanism can allocate resources for a NIDS-VA on demand while still satisfying its performance requirements. We also verify the effectiveness of our global resource scheduling approach by comparing it with two classic vector packing algorithms, and the results show that our approach improved the resource utilization of cloud environments and reduced the number of in-use NIDS-VAs and physical hosts