Package: ts-eval
ts-evalTime Series analysis and evaluation toolsA set of tools to make time series analysis easier.🧩 Current featuresN-step ahead rolling origin time series evaluation– using a Jupyter widget.Friedman / Nemenyi rank test (posthoc)– to see which model statistically performs better.Relative Metrics– rMSE, rMAE + Forecasted Value analogues.Prediction Interval Metrics– MIS, rMIS, FVrMISFixed fourier series generation– fixed in time according to pandas indexNaive/Seasonal models for baseline predictions(with prediction intervals)Statsmodels n-step evaluation– helper functions to evaluate n-step ahead forecasts using Statsmodels models or naive/seasonal naive models.👩🏾‍🎨 Widget PreviewIn:TSMetrics(target, sm_seas, default)
.use_reference(snaive)
.for_horizons(0, 1, 5, 23)
.for_time_slices(time_slices.all, time_slices.weekend)
.with_description()
.with_prediction_rankings(mtx.FVrMSE, mtx.FVrMIS)
.with_predictions_plot()
.show()Out:👩🏾‍🚒 DemoFor a more elaborate example, please check out theDemo Notebook.Alternatively, check outinteractive Binder demo🤦🏾‍ MotivationWhile working on a long term time series analysis project, I had a need to summarize and store performance metrics
of different models and compare them. As it's daunting to do this across dozens of notebooks, I huddled over some code
to do what I want in a few lines of code.👩🏾‍🚀 Installationpip install ts-eval📋 Release Planning:Release 0.3use pandas better for dataframe styles / vizhttps://pandas.pydata.org/pandas-docs/stable/user_guide/style.htmlapi like (viz1 | viz2 | viz3 ) / viz4 (patchwork R package)CRPS evaluationdynamic insample forecast for statsmodelsPI coverage estimation is really needed (in %)predictions based on loess decompositionhttps://github.com/jrmontag/STLDecomposedataset description: first index - last index sample (dates or ids)transform with callbackaltair-like API where you can combine components with +success rate of prediction intervals like herehttp://freerangestats.info/blog/2016/01/30/hybrid-forecastsdescribe case when MIS rankings are better for one dataset, but its mean is worse (due to huge outliers)wrapper around xarray datasets, which always returns non-NaN data and/or statistics that I need are computed
inside of this wrapper. NaNs are always inside. Doable?boxplots by timestep visualization (with boxplot, outliers for each step)remove collection of deps in style [tests_and_bla_bla] to [tests,bla]links to papers – AvgRelMAE (Davydenko and Fildes, 2013); link to Nemenyi paper / implementationsmake graphs with PIs more narrow on 0,1,.. steps as there's too much space left (with an option to turn this off).better API for the end user – minimize interaction with xarraypep517 build / wheels / better setup.py as per Hynektravis: add 3.8 default python when it's availabledocs: supported metrics & API optionsMaybe use api like Summary in statsmodels MLEModel class, it has extend methods and warn/info messagespretty legend for lots like herehttps://studywolf.wordpress.com/2017/11/21/matplotlib-legends-for-mean-and-confidence-interval-plots/Look for TODOschangable colorsturn off colored display optiona nicer API for raw metrics containercodacy badgeboxplots to compare models (as an alternative)violin plots to compare predictions – areas can be colored, different metrics on left and right (like relative...)Release 0.4holiday/fourier features modelfix viz module to have less of important stuffa gif with project visualizationcheck shapes of input arrays (target vs preds), now it doesn't raise an errorBaseline prediction using target dataset (without explicit calculation, but losing some time points)💡 IdeascomponentsGraph: Visualize outliers from confidence intervalMulti-comparison component: scikit_posthocs lib or homecooked?inspect true confidence interval coverage via sampling (was done in postings around bayesian dropout sampling)xarrays: compare if compared datasets are actually equal (offets by dates, shapes, maybe even hashing)bin together step performance, like steps 0-1, 2-5, 6-12, 13-24highlight regions using a mask (holidays, etc.)option to view interactively points using widget (plotly)?diagnostics: bias to over / underestimate pointsanimated graphs for change in seasonalityfeaturesexample notebook for fourier?tests for fouriernint generationutils:model adaptor (for different models, generic) which generates 3d prediction dataset. For stastmodels using dyn forecast or kalman filterfuture importance calculator, but only if I can manipulate input featuresfeature selection using PACF / prewhiten?projectmore defensive style (add arg checks, so it's easier to understand what is going on)docstringscirclecihttps://timothycrosley.github.io/portray/for docssMAPE & MASE can be added for the jupyter evaluation tables? Residual stats: since I have residuals => Ljung-Box, Heteroscedasticity test, Jarque-Bera – like in statsmodels results,
but probably these stats were inspected already by the user... and on which step should they be computed then?See alsohttps://arch.readthedocs.io/en/latest/multiple-comparison/multiple-comparison_examples.htmlRMC as an anlogue for nemenyihttps://github.com/config-i1/greybox/blob/master/R/rmc.R🤹🏼‍♂️ DevelopmentRecommended development workflow:pipenv install -e .[dev]
pipenv shellThe library doesn't use Flit/Poetry, so the suggested workflow is based on Pipenv (as perhttps://github.com/pypa/pipenv/issues/1911).
Pipfile* are ignored in the .gitignore.Changelog0.2.3 (2021-06-03)Fixes (unreleased changes from 2019, doh)Fix results for 1 time series (noop)0.2.2 (2019-10-22)FixesFix nan values propagated to Friedman Nemenyi test.Critical distance is returned alongside Friedman Nemenyi test.0.2.1 (2019-10-18)FixesOutdated import in wheel version of the package.0.2.0 (2019-10-16)FeaturesMultiple prediction ranking with Friedman Nemenyi posthoc.Visualization of prediction intervalsIndication of prediction ranking in a colorful tableRewrite of the internal computation machinery0.1.0 (2019-10-04)FeaturesN-step ahead evaluation widget for JupyterAbsolute & relative metrics for point forecasts and prediction intervals (MSE, MAE, rMSE, rMAE, MIS, rMIS)Naive/Seasonal models for baseline (with prediction intervals)Helper functions to evaluate n-step ahead forecasts using Statsmodels models or naive/seasonal naive models.Holiday features generation and model evaluation on holiday datetimes.0.0.1 (2019-09-18)FeaturesFixed fourier series generation (fixed in time according to pandas index)