Package: price-forecast-suite-package
price_forecast_suite_packageA library containing common tools for data processing, feature engineering and model training and prediction.package:  - /dist/price_forecast_suite_package-0.0.31-py3-none-any.whlPick a supported environment and install the package.pip install price_forecast_suite_package-0.0.31-py3-none-any.whlDocument:document for using the libraryFunctions included in data processing:1. remove_outlierHelps filter outlier from a specified column, and will return a new dataframedef remove_outlier(df, column_name, n_outlier=0.25):
    ...
    return new_dfArguments tableArgument NameData TypeRequired/OptionalDefault ValuedfDataFrameRequired-column_namestringRequired-n_outlierfloatOptional0.252. add_period_to_timeUsed to add 'period' column as a time interval into the 'date' columndef add_period_to_time(df, 
                       date_column_name='DATE', 
                       period_column_name='PERIOD', 
                       period_minutes=30):
    ...
    return new_dfArguments tableArgument NameData TypeRequired/OptionalDefault ValuedfdataframeRequired-date_column_namestringOptional'DATE'period_column_namestringOptional'PERIOD'period_minutesintOptional303. show_draft_plotUsed to display the plot for a list of data, this function has no return.def show_draft_plot(datas, x_label, title, legend, picture_size=[18, 5]):
    ...Argument NameData TypeRequired/OptionalDefault Valuedatasnparray[]Required-x_labelnparray[]Requiredsame length as the list of data-titlestringOptional''picture_size[int, int]Optional[18,5]example:from ds_common_tool import suite_data  
  
lstmModel = dasuite_data.show_draft_plot(datas = [df['column1'], df['column2']], x_label = df.index, title='Compare 2 features values', legend=['column1_data', 'column2_data'], picture_size=[18, 5])4. get_n_rollingGet N days rolling mean/max/min/mediandef get_n_rolling(df, target_column_name, n = 30, method = 'mean') :
    ...
    return new_dfArgument NameData TypeRequired/OptionalDefault ValuedfDataFrameRequired-target_column_namestringRequired-nintOptional30methodstringOptional, accpect: 'mean', 'max', 'min', 'median', 'std''mean'5. read_data_externalUsed for reading external dataframesdef read_data_external(df,  main_df, date_column = 'DATE', fillna_limit_n=5):
    ...
    return new_dfArgument NameData TypeRequired/OptionalDefault ValuedfDataFrameRequired-main_dfDataFrameRequired-date_columnstring[]Optional'DATE'fillna_limit_nintOptional56. merge_dfsUsed for merging multiple dataframesdef merge_dfs(df_list = [], on_column = 'DATE', method_na = 'dropna', fill_customize_value = 0):
    ...
    return new_dfArgument NameData TypeRequired/OptionalDefault Valuedf_listDataFrame[]RequiredNOTE:first item should be the main target df-on_columnstringOptional'DATE'method_nastringOptional, accept: 'dropna', 'fill_customize''dropna'fill_customize_valueintOptional, only needed if method_na='fill_customize'07. shift_rowAdd a new shifted column to the dataframe based on an existing columndef shift_row(df, target_columns, shift_n_list = [], method_na = 'dropna'):
    ...
    return new_dfArgument NameData TypeRequired/OptionalDefault ValuedfdataframeRequired-target_columnsstringRequired-shift_n_listint[]Optional,Note:Number of shifts required, e.g. [-20, -30][]method_nadataframeOptional, accept: 'bfill', 'ffill''dropna'8. get_trend_meanAdd 'Month', 'WeekOfYear' and 'DayOfWeek' as dummy variables to dataframedef get_trend_mean(df, date_column_name = 'DATE'):
    ...
    return new_dfArgument NameData TypeRequired/OptionalDefault ValuedfdataframeRequired-date_column_namestr[]Optional'DATE'9. get_dummyAdd dummy variables based on a specified columndef get_dummy(df, dummy_columns = ['PERIOD']):
    ...
    return new_dfArgument NameData TypeRequired/OptionalDefault ValuedfdataframeRequired-dummy_columnsstr[]Optional['Period']10. switch_y_columnMove the label_column to the last columndef switch_y_column(df, column_name):
    ...
    return new_dfArgument NameData TypeRequired/OptionalDefault ValuedfdataframeRequired-column_namestringRequired-11. time_split_datasetTimeSeriesSplit the test data into test and validation datatime_split_dataset(X, y, split_n = 30, print_shape=True):
    ...
    return X_train_seq, y_train_seq, X_val_seq, y_val_seqArgument NameData TypeRequired/OptionalDefault ValueXnparrayRequired-ynparrayRequired-Split_nintOptional30print_shapeboolOptionalTrue12. get_df_sqlGet data from SQLget_df_sql(query, engine_path):
    ...
    return result_dfArgument NameData TypeRequired/OptionalDefault ValuequerystringRequired-engine_pathstringRequired-Functions included in model:1. model_with_data_splitdef model_with_data_split(df, 
                          label_column, 
                          train_start, train_end, 
                          look_back, look_forward, 
                          column_set_index = 'DATE', split_n = 30, 
                          n_neurons = [128], 
                          transformer_args = [5, 256, 256, 256], 
                          print_model_summary = True, 
                          dropout = 0.5, 
                          epochs = 30, patience = 5, 
                          early_stop = True, 
                          save_model = False, model_path = '', checkpoint_path = '', 
                          model_name = 'lstm', 
                          enable_optuna = False, epochs_each_try = 10, n_trials = 10, 
                          show_loss = True)Arguments TableArgument NameData TypeRequired/OptionalDefault ValuedfDataframeRequired-label_columnstringRequired-train_startSame as indexRequired-train_endSame as indexRequired-look_backintRequired-look_forwardintRequired-column_set_indexstringOptional'DATE'split_nintOptional30n_neuronsint[]Optional[128]transformer_argsint[]Optional[5, 256, 256, 256]print_model_summaryboolOptionalTruedropoutfloatOptional0.5epochsintOptional30patienceintOptional5early_stopboolOptionalTruesave_modelboolOptionalNote:If True, both model_path and checkpoint_path must be givenFalsemodel_pathstringOptional''checkpoint_pathstringOptional''model_namestringOptional'lstm'enable_optunaboolOptionalFalseepcochs_each_tryintOptional10n_trialsintOptional10show_lossboolOptionalTrue2. linear_model_with_optunalinear_model_with_optuna(df, 
                         label_column, 
                         column_set_index = 0, 
                         start_index = '', 
                         end_index = '', 
                         print_data_shape = True, 
                         save_model = False, 
                         model_path = '', 
                         enable_optuna = True, 
                         n_trials = 20, 
                         model_name = 'elasticnet', 
                         alpha = 0.1, 
                         l1 = 0.5)Arguments TableArgument NameData TypeRequired/OptionalDefault ValuedfDataFrameRequired-column_set_indexstringOptional0start_indexstringOptional''end_indexstringOptional''print_data_shapeboolOptionalTruesave_modelboolOptionalFalsemodel_pathboolOptional, Required if save_model=True''enable_optunaboolOptionalTruen_trialsintOptional20model_namestringOptional'elasticnet'alphaintOptional0.1l1intOptional0.53. xgb_with_optunaxgb_with_optuna(df, 
                label_column, 
                column_set_index = 0, 
                start_index = '0', 
                end_index = '1', 
                save_model = False, 
                model_path = '', 
                enable_optuna = True, 
                n_trials =10)Arguments TableArgument NameData TypeRequired/OptionalDefault ValuedfDataFrameRequired-label_columnstringRequired-column_set_indexstringOptional0start_indexstringOptional'0'end_indexstringOptional'1'save_modelboolOptionalFalsemodel_pathboolOptional, Required if save_model=True''enable_optunaboolOptionalTruen_trialsintOptional104. test_resulttest_result(df, 
            target_column , 
            model_path=[], 
            model_type=[], 
            look_back = 1, 
            look_forward = 1, 
            display_metrix = True, 
            display_plot = True, 
            display_title = '', 
            display_legend = ['pred', 'real'])Arguments TableArgument NameData TypeRequired/OptionalDefault ValuedfDataFrameRequired-target_columnstringRequired-model_pathmodelOptional[]model_typestr[]Optional, accpet: 'lstm', 'tcn', 'transformer'[]look_backintOptional1look_forwardintOptional1display_metrixboolOptionalTruedisplay_plotboolOptionalTruedisplay_titlestringOptional''display_legendstr[]Optional['pred', 'real']5. predict_resultpredict_result(predict_data_list = [], 
                model_path=[], 
                model_type=['lstm'])Arguments TableArgument NameData TypeRequired/OptionalDefault Valuepredict_data_listint[]Optional[]model_pathmodelOptional[]model_typestr[]Optional, accept: ['lstm', 'tcn', 'transformer']['lstm']Functions included in feature_engineer:1. sg_long_term_feature_engineeringsg_long_term_feature_engineering(df, 
                                brent_df, 
                                gas_df, 
                                weather_df, 
                                filter_columns, 
                                target_column)Arugments TableArgument NameData TypeRequired/OptionalDefault ValuedfDataFrameRequired-brent_dfDataFrameRequired-gas_dfDataFrameRequired-weather_dfDataFrameRequired-filter_columnsstringRequired-target_columnstringRequired-2. sg_short_term_feature_engineeringsg_short_term_feature_engineering(dpr_df, 
                                    reg_df, 
                                    advisory_df_dummy, 
                                    targetvariable, 
                                    main_features)Arugments TableArgument NameData TypeRequired/OptionalDefault Valuedpr_dfDataFrameRequired-reg_dfDataFrameRequired-advisory_df_dummyDataFrameRequired-targetvariablestringRequired-main_featureslistRequired-3. shandong_feature_engineeringshandong_feature_engineering(data, 
                            feature_columns, 
                            start_index, 
                            end_index, 
                            predict=False)Arugments TableArgument NameData TypeRequired/OptionalDefault ValuedataDataFrameRequired-feature_columnsstr[]Required-start_indexstringRequired-end_indexstringRequired-predictboolOptionalFalseOLD1. lstm model customizedsuite_model.lstm_model_custmize(look_back, look_forward, n_features, dropout=0.5, print_summary=False, n_neurons = [128])parameterdata typedescriptiondefault valuelook_backintinput size-look_forwardintoutput size-n_featuresintnumber of features-dropoutfloatrange (0,1)0.5n_neuronsint[]number of neurons in each layer, eg. [256, 128, 64][128]print_summarybooleanTrue: print model summary.FalsereturnModellstm model-example:from ds_common_tool import suite_model  
  
lstmModel = suite_model.lstm_model_custmize(look_back=30, look_forward=30, n_features=4, dropout=0.5, print_summary=False, n_neurons = [128])2. train the modelsuite_model.train_model(model, X_train_seq, y_train_seq, X_val_seq, y_val_seq, epochs=100, early_stop = True, patience=10, save_model = False, model_path='', checkpoint_path='', show_loss = True)parameterdata typedescriptiondefault valuemodelintinput size-X_train_seqsame type as model-y_train_seqsame type as model-X_val_seqsame type as model-y_val_seqsame type as model-epochsint epochs for traning100early_stopbooleanTrue: to use early_stopTruepatienceintrange(10, +), patience needed if early_stop=True10save_modelbooleanTrue: save model and weightFalsemodel_pathstringpath where the model is saved, e.g. model_path = '../../model/LSTM_longterm_usep.hdf5'''checkpoint_pathstringpath where the checkpoint(weight) is saved, e.g. checkpoint_path = '../../model/checkpoint/cp-lstm-longterm-usep.ckpt'''show_lossbooleanTrue: plot loss for each epochTruereturnModelupdated model-example:from ds_common_tool import suite_model  
  
lstmModel = suite_model.train_model(lstmModel, X_train_seq, y_train_seq, X_val_seq, y_val_seq, 
                          epochs=3, 
                          early_stop = True, 
                          patience=3, 
                          save_model = True, 
                          model_path = '../../model/LSTM_longterm_usep.hdf5',
                          show_loss = True)3. LSTM model with split sequencessuite_model.lstm_model_with_data_split(df, label_column, train_start, train_end, test_start, test_end, look_back, look_forward, column_set_index = 0,split_n = 30, lstm_n_neurons = [128], print_model_summary = True, dropout = 0.5, epochs = 30, patience = 5,early_stop = True, save_model = False, model_path = '', checkpoint_path = '', show_loss = True, show_result_plot = True, plot_title = '',evauate_metric = ['mape'])parameterdata typedescriptiondefault valuedfDataframedf to be processed-label_columnnparray[]target column for prediction-train_startstringStarting train date-train_endstringEnding train date-test_startstringStarting test date-test_endstringEnding test date-look_backintinput size-look_forwardintoutput size-column_set_indexintSet an index column to specified index0split_nintNumber of splits for TimeSeriesSplit30lstm_n_neuronsint[]Neurons needed per layer[128]print_model_summaryboolPrint summary if TrueTruedropoutfloatrange(0, 1)0.5epochsintNumber of epochs30patienceintPatience for early_stop5early_stopboolEnable early_stop if TrueTruesave_modelboolEnable to save modelNote:If True, both model_path and checkpoint_path must be givenFalsemodel_pathstringPath to save model''checkpoint_pathstringPath to save checkpoint''show_lossboolLoss plot will be printed if TrueTrueshow_result_plotboolResult plot will be shown if TrueTrueplot_titlestringInclude plot title''evaluate_metricstr[]Metrics to be evaluate['mape']returnModel, mae, mape, prediction--example:from ds_common_tool import suite_model 
  
lstmModel, mae, mape, prediction_data = suite_model.lstm_model_with_data_split(df = all_data, 
                                                                               label_column = 'mean_30', 
                                                                               train_start = '2017-02-01', 
                                                                               train_end = '2020-12-31', 
                                                                               test_start = '2020-12-01', 
                                                                               test_end = '2021-01-31', 
                                                                               look_back = 31, 
                                                                               look_forward = 31, 
                                                                               column_set_index = 'DATE',
                                                                               split_n = 30, 
                                                                               lstm_n_neurons = [128, 64],
                                                                               print_model_summary = True,
                                                                               dropout = 0.5,
                                                                               epochs = 30,
                                                                               patience = 5,
                                                                               early_stop = True,
                                                                               save_model = False,
                                                                               show_loss = True,
                                                                               show_result_plot = True,
                                                                               plot_title = '',
                                                                               evauate_metric = ['mape'])4. Check metrics: mae, mapecheck_result_metric(y_true, y_pred, methods=['mae'], print_result=True)parameterdata typedescriptiondefault valuey_truenparraySame type as model-y_prednparraySame type as model-methodsstr[]Specify metrics in a list. Available metrics: ['mae', 'mape']['mae']print_resultboolPrint result if TrueTrue5. TCNtcn_model(look_back, n_features, look_forward, batch_size=None, print_summary=True, n_neurons=[128])parameterdata typedescriptiondefault valuelook_backintInput size-n_featuresintNumber of features-look_forwardintOutput size-batch_sizeintSpecifies batch sizeNoneprint_summaryboolPrint if TrueTruen_neuronsint[]To specify number of neurons[128]6. LSTM with Optunaoptuna_lstm(epochs_each_try, n_trials, look_back, look_forward, n_features, X_train_seq, y_train_seq, X_val_seq, y_val_seq, print_best_trail = True)parameterdata typedescriptiondefault valueepochs_each_tryintNumber of epochs to run during each trial-n_trialsintNumber of trials to run for Optuna-look_backintInput size-look_forwardintOutput size-n_featuresintNumber of features-X_train_seqSame type as model-y_train_seqSame type as model-X_val_seqSame type as model-y_val_seqSame type as model-print_best_trialboolPrint out the best trial if TrueTruereturnList of neurons, dropout--example:from ds_common_tool import suite_model 
  
n_neurons, dropout = suite_model.optuna_lstm(epochs_each_try = 5, 
                                 n_trials = 10, 
                                 look_back = 31, 
                                 look_forward = 31, 
                                 n_features = 82, 
                                 X_train_seq = X_train_seq, 
                                 y_train_seq = y_train_seq, 
                                 X_val_seq = X_val_seq, 
                                 y_val_seq = y_val_seq, 
                                 print_best_trail = True)pypi