Package: ts2ml
ts2mlInstallpipinstallts2mlHow to useimportpandasaspdfromts2ml.coreimportadd_missing_slotsfromts2ml.coreimporttransform_ts_data_into_features_and_targetdf=pd.DataFrame({'pickup_hour':['2022-01-01 00:00:00','2022-01-01 01:00:00','2022-01-01 03:00:00','2022-01-01 01:00:00','2022-01-01 02:00:00','2022-01-01 05:00:00'],'pickup_location_id':[1,1,1,2,2,2],'rides':[2,3,1,1,2,1]})df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>pickup_hourpickup_location_idrides02022-01-01 00:00:001212022-01-01 01:00:001322022-01-01 03:00:001132022-01-01 01:00:002142022-01-01 02:00:002252022-01-01 05:00:0021Let’s fill the missing slots with zerosdf=add_missing_slots(df,datetime_col='pickup_hour',entity_col='pickup_location_id',value_col='rides',freq='H')df100%|██████████| 2/2 [00:00<00:00, 907.86it/s]<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>pickup_hourpickup_location_idrides02022-01-01 00:00:001212022-01-01 01:00:001322022-01-01 02:00:001032022-01-01 03:00:001142022-01-01 04:00:001052022-01-01 05:00:001062022-01-01 00:00:002072022-01-01 01:00:002182022-01-01 02:00:002292022-01-01 03:00:0020102022-01-01 04:00:0020112022-01-01 05:00:0021Now, let’s build features and targets to predict the number of rides for
the next hour for each location_id, by using the historical number of
rides for the last 3 hoursfeatures,targets=transform_ts_data_into_features_and_target(df,n_features=3,datetime_col='pickup_hour',entity_col='pickup_location_id',value_col='rides',n_targets=1,step_size=1,step_name='hour')100%|██████████| 2/2 [00:00<00:00, 597.86it/s]features<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>rides_previous_3_hourrides_previous_2_hourrides_previous_1_hourpickup_hourpickup_location_id02.03.00.02022-01-01 03:00:00113.00.01.02022-01-01 04:00:00120.01.02.02022-01-01 03:00:00231.02.00.02022-01-01 04:00:002targets<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>target_rides_next_hour01.010.020.030.0Xy_df=pd.concat([features,targets],axis=1)Xy_df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>rides_previous_3_hourrides_previous_2_hourrides_previous_1_hourpickup_hourpickup_location_idtarget_rides_next_hour02.03.00.02022-01-01 03:00:0011.013.00.01.02022-01-01 04:00:0010.020.01.02.02022-01-01 03:00:0020.031.02.00.02022-01-01 04:00:0020.0Another ExampleMontly spaced time seriesimportpandasaspdimportnumpyasnp# Generate timestamp index with monthly frequencydate_rng=pd.date_range(start='1/1/2020',end='12/1/2022',freq='MS')# Create list of city codescities=['FOR','SP','RJ']# Create dataframe with random sales data for each city on each monthdf=pd.DataFrame({'date':date_rng,'city':np.repeat(cities,len(date_rng)//len(cities)),'sales':np.random.randint(1000,5000,size=len(date_rng))})df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>datecitysales02020-01-01FOR494412020-02-01FOR343522020-03-01FOR454332020-04-01FOR387942020-05-01FOR260152020-06-01FOR292262020-07-01FOR454272020-08-01FOR133882020-09-01FOR293892020-10-01FOR2695102020-11-01FOR4065112020-12-01FOR3864122021-01-01SP2652132021-02-01SP2137142021-03-01SP2663152021-04-01SP1168162021-05-01SP4523172021-06-01SP4135182021-07-01SP3566192021-08-01SP2121202021-09-01SP1070212021-10-01SP1624222021-11-01SP3034232021-12-01SP4063242022-01-01RJ2297252022-02-01RJ3430262022-03-01RJ2903272022-04-01RJ4197282022-05-01RJ4141292022-06-01RJ2899302022-07-01RJ4529312022-08-01RJ3612322022-09-01RJ1856332022-10-01RJ4804342022-11-01RJ1764352022-12-01RJ4425FOR city only have data for 2020 year, RJ only for 2022 and SP only for
2021. Let’s also simulate more missing slots between the years.# Generate random indices to dropdrop_indices=np.random.choice(df.index,size=int(len(df)*0.2),replace=False)# Drop selected rows from dataframedf=df.drop(drop_indices)df.reset_index(drop=True,inplace=True)df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>datecitysales02020-01-01FOR494412020-02-01FOR343522020-03-01FOR454332020-04-01FOR387942020-05-01FOR260152020-06-01FOR292262020-07-01FOR454272020-08-01FOR133882020-09-01FOR293892020-11-01FOR4065102020-12-01FOR3864112021-01-01SP2652122021-02-01SP2137132021-03-01SP2663142021-07-01SP3566152021-08-01SP2121162021-10-01SP1624172021-11-01SP3034182021-12-01SP4063192022-01-01RJ2297202022-02-01RJ3430212022-03-01RJ2903222022-04-01RJ4197232022-05-01RJ4141242022-06-01RJ2899252022-09-01RJ1856262022-10-01RJ4804272022-11-01RJ1764282022-12-01RJ4425Now lets fill the missing slots with zero values. The function will
complete the missing slots with zeros:df_full=add_missing_slots(df,datetime_col='date',entity_col='city',value_col='sales',freq='MS')df_full100%|██████████| 3/3 [00:00<00:00, 843.70it/s]<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>datecitysales02020-01-01FOR494412020-02-01FOR343522020-03-01FOR454332020-04-01FOR387942020-05-01FOR2601............1032022-08-01RJ01042022-09-01RJ18561052022-10-01RJ48041062022-11-01RJ17641072022-12-01RJ4425108 rows × 3 columnsLet’s build a dataset for training a machine learning model to predict
the sales for the next 3 months, for each city, based on historical data
of sales for the previous 6 months.features,targets=transform_ts_data_into_features_and_target(df_full,n_features=3,datetime_col='date',entity_col='city',value_col='sales',n_targets=1,step_size=1,step_name='month')100%|██████████| 3/3 [00:00<00:00, 205.58it/s]pd.concat([features,targets],axis=1)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>sales_previous_3_monthsales_previous_2_monthsales_previous_1_monthdatecitytarget_sales_next_month04944.03435.04543.02020-04-01FOR3879.013435.04543.03879.02020-05-01FOR2601.024543.03879.02601.02020-06-01FOR2922.033879.02601.02922.02020-07-01FOR4542.042601.02922.04542.02020-08-01FOR1338.0.....................914197.04141.02899.02022-07-01RJ0.0924141.02899.00.02022-08-01RJ0.0932899.00.00.02022-09-01RJ1856.0940.00.01856.02022-10-01RJ4804.0950.01856.04804.02022-11-01RJ1764.096 rows × 6 columnsEmbedding on Sklearn Pipelinesfromsklearn.pipelineimportmake_pipelinefromsklearn.preprocessingimportFunctionTransformeradd_missing_slots_transformer=FunctionTransformer(add_missing_slots,kw_args={'datetime_col':'date','entity_col':'city','value_col':'sales','freq':'MS'})transform_ts_data_into_features_and_target_transformer=FunctionTransformer(transform_ts_data_into_features_and_target,kw_args={'n_features':3,'datetime_col':'date','entity_col':'city','value_col':'sales','n_targets':1,'step_size':1,'step_name':'month','concat_Xy':True})ts_data_to_features_and_target_pipeline=make_pipeline(add_missing_slots_transformer,transform_ts_data_into_features_and_target_transformer)ts_data_to_features_and_target_pipeline<style>#sk-container-id-3 {color: black;background-color: white;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: "▸";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: "▾";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: "";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: "";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: "";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style>Pipeline(steps=[('functiontransformer-1',FunctionTransformer(func=&lt;function add_missing_slots at 0x11f8f49d0&gt;,
                                 kw_args={&#x27;datetime_col&#x27;: &#x27;date&#x27;,
                                          &#x27;entity_col&#x27;: &#x27;city&#x27;,
                                          &#x27;freq&#x27;: &#x27;MS&#x27;,
                                          &#x27;value_col&#x27;: &#x27;sales&#x27;})),
            (&#x27;functiontransformer-2&#x27;,
             FunctionTransformer(func=&lt;function transform_ts_data_into_features_and_target at 0x11f925ca0&gt;,
                                 kw_args={&#x27;concat_Xy&#x27;: True,
                                          &#x27;datetime_col&#x27;: &#x27;date&#x27;,
                                          &#x27;entity_col&#x27;: &#x27;city&#x27;,
                                          &#x27;n_features&#x27;: 3, &#x27;n_targets&#x27;: 1,
                                          &#x27;step_name&#x27;: &#x27;month&#x27;,
                                          &#x27;step_size&#x27;: 1,
                                          &#x27;value_col&#x27;: &#x27;sales&#x27;}))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class="sk-container" hidden><div class="sk-item sk-dashed-wrapped"><div class="sk-label-container"><div class="sk-label sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually" id="sk-estimator-id-7" type="checkbox" ><label for="sk-estimator-id-7" class="sk-toggleable__label sk-toggleable__label-arrow">Pipeline</label><div class="sk-toggleable__content"><pre>Pipeline(steps=[(&#x27;functiontransformer-1&#x27;,
             FunctionTransformer(func=&lt;function add_missing_slots at 0x11f8f49d0&gt;,
                                 kw_args={&#x27;datetime_col&#x27;: &#x27;date&#x27;,
                                          &#x27;entity_col&#x27;: &#x27;city&#x27;,
                                          &#x27;freq&#x27;: &#x27;MS&#x27;,
                                          &#x27;value_col&#x27;: &#x27;sales&#x27;})),
            (&#x27;functiontransformer-2&#x27;,
             FunctionTransformer(func=&lt;function transform_ts_data_into_features_and_target at 0x11f925ca0&gt;,
                                 kw_args={&#x27;concat_Xy&#x27;: True,
                                          &#x27;datetime_col&#x27;: &#x27;date&#x27;,
                                          &#x27;entity_col&#x27;: &#x27;city&#x27;,
                                          &#x27;n_features&#x27;: 3, &#x27;n_targets&#x27;: 1,
                                          &#x27;step_name&#x27;: &#x27;month&#x27;,
                                          &#x27;step_size&#x27;: 1,
                                          &#x27;value_col&#x27;: &#x27;sales&#x27;}))])</pre></div></div></div><div class="sk-serial"><div class="sk-item"><div class="sk-estimator sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually" id="sk-estimator-id-8" type="checkbox" ><label for="sk-estimator-id-8" class="sk-toggleable__label sk-toggleable__label-arrow">FunctionTransformer</label><div class="sk-toggleable__content"><pre>FunctionTransformer(func=&lt;function add_missing_slots at 0x11f8f49d0&gt;,
                kw_args={&#x27;datetime_col&#x27;: &#x27;date&#x27;, &#x27;entity_col&#x27;: &#x27;city&#x27;,
                         &#x27;freq&#x27;: &#x27;MS&#x27;, &#x27;value_col&#x27;: &#x27;sales&#x27;})</pre></div></div></div><div class="sk-item"><div class="sk-estimator sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually" id="sk-estimator-id-9" type="checkbox" ><label for="sk-estimator-id-9" class="sk-toggleable__label sk-toggleable__label-arrow">FunctionTransformer</label><div class="sk-toggleable__content"><pre>FunctionTransformer(func=&lt;function transform_ts_data_into_features_and_target at 0x11f925ca0&gt;,
                kw_args={&#x27;concat_Xy&#x27;: True, &#x27;datetime_col&#x27;: &#x27;date&#x27;,
                         &#x27;entity_col&#x27;: &#x27;city&#x27;, &#x27;n_features&#x27;: 3,
                         &#x27;n_targets&#x27;: 1, &#x27;step_name&#x27;: &#x27;month&#x27;,
                         &#x27;step_size&#x27;: 1, &#x27;value_col&#x27;: &#x27;sales&#x27;})</pre></div></div></div></div></div></div></div>Xy_df=ts_data_to_features_and_target_pipeline.fit_transform(df)Xy_df100%|██████████| 3/3 [00:00<00:00, 715.47it/s]
100%|██████████| 3/3 [00:00<00:00, 184.12it/s]<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>sales_previous_3_monthsales_previous_2_monthsales_previous_1_monthdatecitytarget_sales_next_month04944.03435.04543.02020-04-01FOR3879.013435.04543.03879.02020-05-01FOR2601.024543.03879.02601.02020-06-01FOR2922.033879.02601.02922.02020-07-01FOR4542.042601.02922.04542.02020-08-01FOR1338.0.....................914197.04141.02899.02022-07-01RJ0.0924141.02899.00.02022-08-01RJ0.0932899.00.00.02022-09-01RJ1856.0940.00.01856.02022-10-01RJ4804.0950.01856.04804.02022-11-01RJ1764.096 rows × 6 columns