Package: datacentertracesdatasets-cli
Datacentertracesdatasets_clicli: Command-line interface for datacentertracesdatasets packageTable of ContentsDescriptionInstallationUsageLicenseAcknowledgementsDescriptionDatacentertracesdatasets-cli is a command-line interface tool that act as an interface of thedatacentertracesdatasets package. Datacentertracesdatasets package facilitates the access to three dataset traces: Alibaba2018, Azure_v2 and Google2019.This command-line interface is OS independent and can be easily installed and used.Available original datasetsPublic datasets organized for machine learning or artificial intelligence usage. The following dasets can be used:Alibaba 2018 machine usageProcessed from the original files found at:https://github.com/alibaba/clusterdata/tree/master/cluster-trace-v2018This repository dataset of machine usage includes the following columns:+--------------------------------------------------------------------------------------------+|Field|Type|Label|Comment|+--------------------------------------------------------------------------------------------+|cpu_util_percent|bigint||[0,100]||mem_util_percent|bigint||[0,100]||net_in|double||normarlizedincomingnetworktraffic,[0,100]||net_out|double||normarlizedoutgoingnetworktraffic,[0,100]||disk_io_percent|double||[0,100],abnormalvaluesareof-1or101|+--------------------------------------------------------------------------------------------+Three sampled datasets are found: average value of each column grouped every 10 seconds as original, and downsampled to 30 seconds and 300 seconds as well.
Every column includes the average utilization of the whole data center.FiguresSome figures were generated from these datasetsFigure: CPU utilization sampled every 10 secondsFigure: Memory utilization sampled every 300 secondsFigure: Net in sampled every 300 secondsFigure: Net out sampled every 300 secondsFigure: Disk io sampled every 300 secondsGoogle 2019 instance usageProcessed from the original dataset and queried using Big Query. More information available at:https://research.google/tools/datasets/google-cluster-workload-traces-2019/This repository dataset of instance usage includes the following columns:+--------------------------------------------------------------------------------------------+|Field|Type|Label|Comment|+--------------------------------------------------------------------------------------------+|avg_cpu|double||[0,1]||avg_mem|double||[0,1]||avg_assigned_mem|double||[0,1]||avg_cycles_per_instruction|double||[0,_]|+--------------------------------------------------------------------------------------------+One sampled dataset is found: average value of each column grouped every 300 seconds as original.
Every column includes the average utilization of the whole data center.FiguresSome figures were generated from these datasetsFigure: CPU usage day 26 sampled every 300 secondsFigure: Mem usage day 26 sampled every 300 secondsFigure: Assigned mem day 26 sampled every 300 secondsFigure: Cycles per instruction day 26 sampled every 300 secondsAzure v2 virtual machine workloadProcessed from the original dataset. More information available at:https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetV2.mdThis repository dataset of instance usage includes the following columns:+--------------------------------------------------------------------------------------------+|Field|Type|Label|Comment|+--------------------------------------------------------------------------------------------+|cpu_usage|double||[0,_]||assigned_mem|double||[0,_]|+--------------------------------------------------------------------------------------------+One sampled dataset is found: sum value of each column grouped every 300 seconds as original. For computing CPU_usage, we used core_count usage of each virtual machine.
Every column includes the total consumption of the whole data center virtual machines.
There is a version of each file including timestamp (from 0 to 2591700, in 300 seconds timestep), and other version without timestampFiguresSome figures were generated from these datasetsFigure: CPU total usage by virtual machines sampled every 300 seconds.Figure: Total assigned memory for virtual machines sampled every 300 seconds.Available sythetic datasetsMoreover, for dataset augmentation and deep learning purposes, the datasets have been augmented using TimeGAN (https://github.com/DamianUS/timegan-pytorch) trained models.The augmented datasets are composed of time series of lenght 288 and sampled to 300 seconds, that would correspond to one operational day of the data center.InstallationTo install the tool in your local environment, just run follow command:pipinstalldatacentertracesdatasets-cliBasic usage examples:Some examples for obtaining the datasets are shown below.Full Azure_V2 original dataset sampled at 300 seconds:datacentertracesdatasets-cli-traceazure_v2The resulting file will be found at:Full Alibaba2018 original dataset sampled at 10 seconds:datacentertracesdatasets-cli-tracealibaba2018-stride10A synthetic sample of 1 day for Google2019 sampled at 300 seconds:datacentertracesdatasets-cli-tracegoogle2019-generationsyntheticA synthetic sample of 1 day for Google2019 sampled at 300 seconds providing a filename:datacentertracesdatasets-cli-tracegoogle2019-generationsynthetic-filemy_dataset.csvLicenseDatacentertracesdatasets-cli is free and open-source software licensed under theMIT license.AcknowledgementsProject PID2021-122208OB-I00, PROYEXCEL_00286 and  TED2021-132695B-I00 project, funded by MCIN / AEI / 10.13039 / 501100011033, by Andalusian Regional Government, and by the European Union - NextGenerationEU.