Package: datacentertracesdatasets
DataCenter-Traces-DatasetsPip package that makes available the datasets published in:https://github.com/alejandrofdez-us/DataCenter-Traces-Datasets. Please, check the mentioned repository for a deeper understanding.Alibaba 2018 machine usageProcessed from the original files found at:https://github.com/alibaba/clusterdata/tree/master/cluster-trace-v2018This repository dataset of machine usage includes the following columns:+--------------------------------------------------------------------------------------------+|Field|Type|Label|Comment|+--------------------------------------------------------------------------------------------+|cpu_util_percent|bigint||[0,100]||mem_util_percent|bigint||[0,100]||net_in|double||normarlizedincomingnetworktraffic,[0,100]||net_out|double||normarlizedoutgoingnetworktraffic,[0,100]||disk_io_percent|double||[0,100],abnormalvaluesareof-1or101|+--------------------------------------------------------------------------------------------+Three sampled datasets are found: average value of each column grouped every 10 seconds as original, and downsampled to 30 seconds and 300 seconds as well.
Every column includes the average utilization of the whole data center.FiguresSome figures were generated from these datasetsFigure: CPU utilization sampled every 10 secondsFigure: Memory utilization sampled every 300 secondsFigure: Net in sampled every 300 secondsFigure: Net out sampled every 300 secondsFigure: Disk io sampled every 300 secondsGoogle 2019 instance usageProcessed from the original dataset and queried using Big Query. More information available at:https://research.google/tools/datasets/google-cluster-workload-traces-2019/This repository dataset of instance usage includes the following columns:+--------------------------------------------------------------------------------------------+|Field|Type|Label|Comment|+--------------------------------------------------------------------------------------------+|cpu_util_percent|double||[0,100]||mem_util_percent|double||[0,100]||assigned_mem_percent|double||[0,100]||avg_cycles_per_instruction|double||[0,_]|+--------------------------------------------------------------------------------------------+One sampled dataset is found: average value of each column grouped every 300 seconds as original.
Every column includes the average utilization of the whole data center.FiguresSome figures were generated from these datasetsFigure: CPU usage day 26 sampled every 300 secondsFigure: Mem usage day 26 sampled every 300 secondsFigure: Assigned mem day 26 sampled every 300 secondsFigure: Cycles per instruction day 26 sampled every 300 secondsAzure v2 virtual machine workloadProcessed from the original dataset. More information available at:https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetV2.mdThis repository dataset of instance usage includes the following columns:+--------------------------------------------------------------------------------------------+|Field|Type|Label|Comment|+--------------------------------------------------------------------------------------------+|cpu_usage|double||[0,_]||assigned_mem|double||[0,_]|+--------------------------------------------------------------------------------------------+One sampled dataset is found: sum value of each column grouped every 300 seconds as original.
Every column includes the total consumption of the whole data center virtual machines.FiguresSome figures were generated from these datasetsFigure: CPU total usage by virtual machines sampled every 300 seconds.Figure: Total assigned memory for virtual machines sampled every 300 seconds.InstallationpipinstalldatacentertracesdatasetsUsageTo load the original Alibaba's 2018 machine usage, with the mean usage of all machines for each timestamp (8 days, 10 seconds timestep) as a Pandas DataFrame:fromdatacentertracesdatasetsimportloadtracesalibaba_2018_original_machine_usage_df=loadtraces.get_trace(trace_name='alibaba2018',trace_type='machine_usage',stride_seconds=10)If, instead of a Pandas DataFrame, a numpy NDArray is needed, theformatparameter can be used:azure_v2_machine_usage_ndarray=loadtraces.get_trace(trace_name='azure_v2',trace_type='machine_usage',stride_seconds=300,format='ndarray')Or, for Google 2019 machine usage:azure_v2_machine_usage_ndarray=loadtraces.get_trace(trace_name='google2019',trace_type='machine_usage',stride_seconds=300,format='ndarray')In addition to the original Alibaba 2018 machine usage dataset, which has a 10-seconds timestep, two additional downsampled versions of 30 and 300 seconds timesteps are provided, which can be retrieved by using thestride_secondsargument:alibaba_2018_machine_usage_300_timestep_df=loadtraces.get_trace(trace_name='alibaba2018',trace_type='machine_usage',stride_seconds=300,format='ndarray')Dataset metadataThe dataset structure and metadata can be retrieved with theget_dataset_infomethod:dataset_info=get_dataset_info(trace_name='alibaba2018',trace_type='machine_usage',stride_seconds=300)Which returns:dataset_info={"timestamp_frequency_secs":300,"column_config":{"cpu_util_percent":{"column_index":0,"y_axis_min":0,"y_axis_max":100},"mem_util_percent":{"column_index":1,"y_axis_min":0,"y_axis_max":100},"net_in":{"column_index":2,"y_axis_min":0,"y_axis_max":100},"net_out":{"column_index":3,"y_axis_min":0,"y_axis_max":100},"disk_io_percent":{"column_index":4,"y_axis_min":0,"y_axis_max":100}},"metadata":{"fields":{"cpu_util_percent":{"type":"numerical","subtype":"float"},"mem_util_percent":{"type":"numerical","subtype":"float"},"net_in":{"type":"numerical","subtype":"float"},"net_out":{"type":"numerical","subtype":"float"},"disk_io_percent":{"type":"numerical","subtype":"float"}}}}Currently only Alibaba 2018's, Google 2019's and Azure_V2 (virtual) machine usage traces are available. In the future, we plan to add the following:Alibaba's 2018 batch_task workload trace.Google's 2019 batch_task workload trace.