Package: great-expectations-time-series-expectations
TimeSeriesExpectationsExpectations for detecting trends, seasonality, outliers, etc. in time series dataAuthor: Abe Gong (abegong)OverviewThis package currently contains...Expectations for detecting trends, seasonality, outliers, etc. in time series dataAn abstract base class for creating additional time series Expectations based on column aggregate metrics.Methods for generating time series data for testing purposes.WarningThis package is experimental, a work in progress.ExpectationsThis package currently contains 3 new Expectations. These are examples of the primary patterns that future time series Expectations will follow.expect_batch_row_count_to_match_prophet_date_model
expect_column_max_to_match_prophet_date_model
expect_column_pair_values_to_match_prophet_date_modelexpect_batch_row_count_to_match_prophet_date_modelandexpect_column_max_to_match_prophet_date_modelare Batch-level Expectations: each Batch corresponds to a single timestamp-value pair in a time series. In typical usage, you would expect to validate a single Batch (and therefore a single timestamp-value) at a time.expect_column_pair_values_to_match_prophet_date_modelis a row-level Expectation: a Batch will typically contain many timestamp-values, which can be evaluated together.Backend support:All three Expectations work in pandasBoth batch-level Expectations work in SQL. The supporting metric forexpect_column_pair_values_to_match_prophet_date_modelis not yet implemented in SQL.All three Expectations work in Spark, butexpect_column_pair_values_to_match_prophet_date_modelrelies on a UDF that may be slow for large data sets.Methods for generating synthetic time series datafrom time_series_expectations.generator import DailyTimeSeriesGenerator

generator = DailyTimeSeriesGenerator()
df = generator.generate_df()

df = generator.generate_df(
    size=180,
    noise=5.0,
    start_date="2022-01-01",
    weekday_dummy_params=[0.0, 1.0, 4.0, 3.0, 2.0, -3.0, -4.5],
)

# etc.See the script that creates examples (assets/generate_test_time_series_data.py) and tests(link) for additional examplesFuture workAdd support for additional time series grains (not just daily)Add more options for time series models, other thanprophet(e.g.statsmodels.tst.arima,pdarima,NeuralProphet)Add methods for pulling illustrative time series data from real-world sourcesDefinerequirementsand documentation for installationAdd SQL implementation forexpect_column_pair_values_to_match_prophet_date_modeland other row-level metrics.Add Data Assistants for creating time series ExpectationsAdd better renderers for time series Expectations, including graphs produced byaltairPublish package to pypiDesign notes on future class hierarchyAs all of those use cases are realized, we imagine the full class hierarchy for time series Expectations to evolve into this:*BatchExpectation* (ABC)
    *BatchAggregateStatisticExpectation* (ABC)
        BatchAggregateStatisticTimeSeriesExpectation (ABC)
            ExpectBatchAggregateStatisticToMatchProphetDateModel (ABC)
                expect_batch_row_count_to_match_prophet_date_model (:white_check_mark:)
                expect_batch_most_recent_update_to_match_prophet_date_model

            ExpectBatchAggregateStatisticToMatchProphetTimestampModel (ABC)
                expect_batch_row_count_to_match_prophet_timestamp_model
                expect_batch_most_recent_update_to_match_prophet_timestamp_model

            ExpectBatchAggregateStatisticToMatchArimaModel (ABC)
                expect_batch_row_count_to_match_arima_model
                expect_batch_most_recent_update_to_match_arima_model
            
            ... for other types of models

    *ColumnAggregateExpectation* (ABC)
        ColumnAggregateTimeSeriesExpectation (ABC, :white_check_mark:)
            expect_column_max_to_match_prophet_date_model (:white_check_mark:)
            expect_column_{property}_to_match_{model}_model
            ...

    *ColumnPairMapExpectation* (ABC, :white_check_mark:)
        ColumnPairTimeSeriesExpectation (ABC)
            expect_column_pair_values_to_match_prophet_date_model (:white_check_mark:)
            expect_column_pair_values_to_match_prophet_timestamp_model
            expect_column_pair_values_to_match_arima_modelFormatting conventions for the nested hierarchy above:Abstract base classes are in CamelCase, and marked with (ABC).Concrete classes are in snake_case.Classes that live/will live in the core Great Expectations library are initalics.Classes that have already been implemented are marked with :white_check_mark:About Abstract Base Classes:The most important ABCs areBatchAggregateStatisticTimeSeriesExpectation,ColumnAggregateTimeSeriesExpectation, andColumnPairTimeSeriesExpectation. They allow time series models to be applied to data in a variety of shapes and formats. Like most ABCs, these classes won't be executable themselves, but will hold shared logic to make it easier to create and maintain Expectations that follow certain patterns.AboutColumnAggregateTimeSeriesExpectations:We expect these to be annbykmatrix ofnmetrics andkmodels. Once we've fully established the right patterns for these Expectations (and their Renderers and Profilers), we should be able to code-gen most or all of these Expectations, to quickly expand coverage.metrics:
    max
    min
    stdev
    sum
    percent_missingness
    percent_matching_regex
    ...

models:
    prophet_date
    prophet_timestamp
    arima
    ...