Package: tsbenchmarks
TSBenchmarksTime-series benchmarking a service. TSBenchmarks is an SDK for benchmarking models on public datasets.Since we take care of the whole infrastructure, benchmarking becomes as easy as running a line in your python nootebooks or calling an API.Why?We build TSBenchmarks because we wanted a standarized solution for benchmarking time-series forecasting models. We evaluate provided forecasts on well known public datasets against benchmark models.Table of contentsTSBenchmarksInstallHow to useData Format1. Request free trial2. Runtsbenchmarkson a private S3 Bucket2.1 Upload to S3 from python2.2 Run the evaluation process2.3 Download your results from s3Available Benchmark DatasetsM4-DailyAvailable Benchmark modelsNaive1Naive2ETSThetaARIMAMLPRNNESRNNFFORMANBEATSAvailable Metrics and PlotsMASEsMAPEAverage loss by time-seriesAverage loss by timestampInstallpip install tsbenchmarksHow to useYou can use TSBenchmarks by either using a completely public S3 bucket or by uploading a file to your own S3 bucket provided by us.Data FormatCurrently we only support.csvfiles. These files must include at least 3 columns, with a unique_id (identifier of each time-series) a date stamp and a value. The unique_id and ds must coincide with the test set of the selected benchmark dataset.1. Request free trialRequest a free trial sending an email to:fede.garza.ramirez@gmail.comand get yourBUCKET_NAME,API_IDandAPI_KEY,AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY.2. Runtsbenchmarkson a private S3 BucketIf you don´t want other people to potentially have access to your data you can runtsbenchmarkson a private S3 Bucket. For that you have to upload your data to a private S3 Bucket that we will provide for you; you can do this inside of python.2.1 Upload to S3 from pythonYou will need theAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYthat we provided.Import and InstantiateTSBenchmarksintroducing yourBUCKET_NAME,API_IDandAPI_KEY,AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY.fromtsbenchmarks.coreimportTSBenchmarkstsbenchmarks=TSBenchmarks(bucket_name='<BUCKET_NAME>',api_id='<API_ID>',api_key='<API_KEY>',aws_access_key_id='<AWS_ACCESS_KEY_ID>',aws_secret_access_key='<AWS_SECRET_ACCESS_KEY>')Upload your local file introducing its name.s3_uri=tsbenchmarks.upload_to_s3(file='<YOUR FILE NAME>')Run the evaluation processTo run the process specify:s3_uri: S3 uri provided after callingtsbenchmarks.upload_to_s3().dataset: Name of datasetds_column: Name of the unique id column.y_column: Name of the target column.#Run Evaluationresponse_tmp_ft=tsbenchmarks.evaluate_my_model(s3_uri="<PRIVATE S3 URI HERE>",dataset='M4-Daily',unique_id_column="<NAME OF ID COLUMN>",ds_column="<NAME OF DATESTAMP COLUMN>",y_column="<NAME OF TARGET COLUMN>")response_tmp_ftstatusmessageid_jobdest_url0200Check job status at GET /tsbenchmarks/jobs/{jo...d8d2ae2f-ac53-4b81-87f5-49520782365as3://ts-benchmarks-api-public/M4-Daily-benchma...|Monitor the process with the following code. Once it's done, access to your bucket to download the generated features.job_id=response_tmp_ft['id_job'].item()tsbenchmarks.get_status(job_id)statusprocessing_time_seconds0InProgress32.2 Download your results from s3Once the process is done you can explore and download the results from s3.ToDosOptimizing writing and reading speed with Parquet filesNan HandlingCheck data integrity before UploadInformative error messagesInformative Status