Package: rearden
ReardenReardenis a Python package designed for usage inside Jupyter Notebooks that provides a faster and more convenient way of carrying out data science and getting insights from machine learning algorithms. Making use of the functionality of the most popular libraries for data analysis (pandas,numpy,statsmodels), data visualization (matplotlib,seaborn) and grid search (scikit-learn), it enables reaching the conclusions about the data in a quicker and clearer manner.Modules and APIThe package is designed to aid data scientists in quickly getting insights about the data during the following stages of data analysis/machine learning:Data preprocessingData visualizationTime-series analysisGrid searchHence, the data structures which make up theReardenpackage have been logically divided into the following Python modules (moduledecorators.pyserves as an auxiliary file supplying the decorators for some data structures includes in the other modules):preprocessings.pyvizualizations.pytime_series.pygrid_search.pymetrics.pyData preprocessingFunctions included inpreprocessings.pyare basically programmed to help with missing values and data preparation for machine learning algorithms (e.g.data split into sets). For instance, currently the following functions and classes are included in the module:NameKindDescriptionDataSplitterclassData split into sets depending on target name and sets proportionsidentify_missing_valuesfunctionDisplay of the number and share of missing valuesThe module and the associated functions can be called like so:fromrearden.preprocessingsimportDataSplitterSplitting dataFor instance, if we have a pandas DataFrame which includes both features and target, we can easily separate them into distinct entities and at the same time split them into training/test sets:fromrearden.preprocessingsimportDataSplittersplitter=DataSplitter(set_shares=(0.75,0.25),random_seed=1)(features_train,target_train,features_test,target_test)=splitter.split_data(data=some_dataframe,target="target_name")Likewise, if we need to split the DataFrame into training/validation/test sets:splitter=DataSplitter(set_shares=(0.6,0.2,0.2),random_seed=1)(features_train,target_train,features_valid,target_valid,features_test,target_test)=splitter.split_data(data=some_dataframe,target="target_name")Afterwards, it is possible to check the splits via a specialsubset_info_attribute:splitter.subset_info_We will get the following table:obs_numset_sharestrain5,4540.6valid1,8180.2test1,8190.2In case we have intended to split the data only into training and test sets, the DataFrame contained insubset_info_attribute becomes as follows:obs_numset_sharestrain6,8180.75test2,2730.25All data subsets obtained as a result of a DataFrame split are contained in the according attributes of the class and can be used at any time:features_train=splitter.features_train_features_valid=splitter.features_valid_features_test=splitter.features_test_target_train=splitter.target_train_target_valid=splitter.target_valid_target_test=splitter.target_test_Thanks to decorators provided indecorators.py, the potential errors which can be made during the instantiation of the class object or during the data split can be caught faster. For instance,check_proportionsdecorator fromDataSplitterDecoratorsclass will check the correctness of set shares specification inset_sharesattribute when defining an object. Decoratorcheck_dimensionswill make sure that we specify two or three values inset_sharesattribute. Lastly,check_splitwill catch the user's mistake of accessingsubset_info_attribute before data split has occurred.Missing values calculationUsing a usefulidentify_missing_valuesfunction, it is possible to quickly calculate the number and share of missing values in a DataFrame. Furthermore, it also outputs the data type of a column where missing values have been found. For instance:identify_missing_values(data=some_dataframe)The resulting DataFrame looks as follows:dtypemissing_countmissing_fractioncol1float644000.8444col3object190.4512col19object40.0313In the case that no missing values have been detected, the function will not return anything.Data visualizationEnhanced data vizualizations tools are located invizualizations.pymodule. The functions here are as follows:NameKindDescriptionplot_model_comparisonfunctionVisualization of ML models performances based on their names and scoresplot_corr_heatmapfunctionPlotting correlation matrix heatmap in one goplot_class_structurefunctionPlotting the shares of different classes for a target vector in classification problemsModels performance comparisonUsingplot_model_comparisonfunction, it is very easy to conveniently showcase how models perform according to some metric. One would just run:importseabornassnsfromrearden.vizualizationsimportplot_model_comparisonsns.set_theme()models_performance=[("Decision Tree",30.8343),("Random Forest",29.3127),("Catboost",26.4651),("Xgboost",26.7804),("LightGBM",26.6084),]plot_model_comparison(results=models_performance,metric_name="RMSE",title_name="Grid search results",save_fig=True,)The result is the following figure which is automatically saved in the newly createdimagesdirectory upon specifyingsave_fig=True(setting this parameter toTrueautomatically activatesrearden.vizualizations.save_plot_in_dirfunction that createsimages/directory and saves the image there):Correlation matrix heatmapIt is possible to quickly plot the heatmap of the correlation matrix for the data usingplot_corr_heatmapfunction. Here is how we would do that:importpandasaspdimportseabornassnsfromrearden.vizualizationsimportplot_corr_heatmapsns.set_theme()test_data=pd.read_csv("datasets/test_data.csv")plot_corr_heatmap(data=test_data,heatmap_coloring="Blues",annotation=True,lower_triangle=True,save_fig=True,)The code above results in the following plot:Plotting the class structureThe functionplot_class_structureenables quickly making inquiries into the balancedness/unbalancedness of the target variable:fromrearden.vizualizationsimportplot_class_structureplot_class_structure(target_var=target_train,xlabel_name="Classes",ylabel_name="Share",title_name="Target variable structure",save_fig=True,)The resulting plot looks, for instance, as follows:Time-series analysisTools for time-series analysis fromtime_series.pyare pretty straightforward:NameKindDescriptionTimeSeriesFeaturesExtractorclassExtraction of time variables from a one-dimensional time-series depending on lag and rolling mean order valuesTimeSeriesSplitterclassTime-series data split into sets with the same functionality asDataSplitterTimeSeriesPlotterclassPlotting the original time-series or a decomposed oneOne can, for example, want to firstly look at the graph viaTimeSeriesPlotter, recover time variables from time-series byTimeSeriesFeaturesExtractorand then divide the data into sets withTimeSeriesSplitter.Time-series plot and its decompositionTimeSeriesPlotterclass provides two additional ways we could plot a time-series:Plain time-series (plot_time_seriesmethod)Decomposed time-series: trend, seasonality, residual (plot_decomposedmethod)For example, we can plot the resampled time-series data (with 1 hour periodicity by default):importpandasaspdimportseabornassnsfromrearden.time_seriesimportTimeSeriesPlottersns.set_theme()ts_data_test=pd.read_csv("datasets/ts_data_test.csv",parse_dates=[0],index_col=[0])ts_plotter=TimeSeriesPlotter()plotter.plot_time_series(data=taxi_data,resample=True,period=("2018-03-01","2018-03-03"),ylabel_name="Number of orders",title_name="Time-series plot",save_fig=True,)In this case we plot the evolution of the number of orders against time. We obtain the following plot:If we want to plot the time-series for the same period of time without resampling, then we just setresample=False:plotter.plot_time_series(data=taxi_data,resample=False,period=("2018-03-01","2018-03-03"),ylabel_name="Number of orders",title_name="Time-series plot",save_fig=True,)We could also decompose this time series which in this case should be resampled in any case (applied by default):plotter.plot_decomposed(data=taxi_data,period=("2018-03-01","2018-03-03"),ylabel_name="Number of orders",save_fig=True,)The result is as follows:Creating time featuresGiven the one-dimensional time-series data, it is actually possible to retrieve time variables from it: quarter, month, day or month. Additionally, we can also add lags as well as rolling mean of different orders. It can be done usingTimeSeriesFeaturesExtractor:fromrearden.time_seriesimportTimeSeriesFeaturesExtractorasTSFEfeatures_extractor=TSFE(col_name="num_orders")features_extractor.get_params()# {'col_name': 'num_orders', 'max_lag': 1, 'rolling_mean_order': 1}Hence, we can choose the column with the time-series as well as which maximum order of lag and rolling mean we want to use when generating new features. For instance, if we decide to use the default parameter values, we can easily configure the class and then use it on the data:importpandasaspdts_data_test=pd.read_csv("datasets/ts_data_test.csv",parse_dates=[0],index_col=[0])ts_data_test=ts_data_test.resample("1H").sum()ts_dataset=features_extractor.fit_transform(ts_data_test)The above code will add quarter, month, day and hour variables to the DataFrame as well as a column with rolling mean and the lag columns according tomax_lagandrolling_mean_orderattribute values.Grid searchIngrid_search.pymodule,RandomizedSearchCVbase class fromsklearn.model_selectionwas used, which has been wrapped with two additional classes with some additional methods, custom defaults and other functionality:NameKindDescriptionRandomizedHyperoptRegressionclassWrapper forRandomizedSearchCVwith functionality to quickly compute regression metrics and conveniently display tuning processRandomizedHyperoptClassificationclassWrapper forRandomizedSearchCVwith functionality to quickly compute classification metrics, conveniently display tuning process and fastly plot confusion matrixSimpleHyperparamsOptimizerclassCustom class that implements a simple grid search algorithm without cross-validationThe module also includes one customSimpleHyperparamsOptimierclass written from scratch that enables quickly running a grid search when, for instance, we need to optimize the hyperparameters of the model on one distinct validation set.Hyperparameter tuningWe can useRandomizedHyperoptClassificationorRandomizedHyperoptRegressionwrappers for quickly making conclusions about the results of the grid search. For instance, let's imagine that we have managed to split the data intofeatures_trainandfeatures_testas well astarget_trainandtarget_test. We can now run the grid search algorithms and immediately get the plot of the confusion matrix:fromsklearn.treeimportDecisionTreeClassifierfromrearden.grid_searchimportRandomizedHyperoptClassificationasRHCdtc_model=DecisionTreeClassifier(random_state=12345)param_grid_dtc={"max_depth":np.arange(2,12)}dtc_grid_search=RHC(estimator=dtc_model,param_distributions=param_grid_dtc,train_dataset=(features_train,target_train),eval_dataset=(features_test,target_test),random_state=12345,cv=5,n_iter=5,scoring="f1",n_jobs=None,return_train_score=True,)dtc_grid_search.train_crossvalidate()After the completing grid search, the methodtrain_crossvalidatewill notify us of the completion:Grid search for DecisionTreeClassifier completed.
Time elapsed: 0.6 sTuning process displayWe can then get a convenient representation of the tuning progress by usingdisplay_tuning_processmethod:dtc_grid_search.display_tuning_process()max_depthmean_test_scoremean_train_score020.49700.5065190.54480.7347250.50590.53793110.54020.8257480.54320.6869Since this class inherits fromsklearn.model_selection.RandomizedSearchCV, it also has access to usualbest_estimator_,best_score_,best_params_attributes.Metrics computationThanks to the additionaleval_datasetattribute, the resulting plot is already a confusion matrix for the best model after cross-validation which has been used for making predictions on the test data:It is also possible to compute all classification metrics usingclassif_stats_attribute that just outputs the result of applyingsklearn.metrics.classification_reportto data contained ineval_datasetattribute.Other functionality of the wrappers for classification and regression can be consulted in thegrid_search.pymodule.Custom grid search without CVA simple grid search algorithm has been implemented inSimpleHyperparamsOptimizerclass. The usage is similar tosklearngrid search but without crossvalidation:fromsklearn.metricsimportmean_squared_errorfromrearden.grid_searchimportSimpleHyperparamsOptimizerasSHOdecision_tree_model=DecisionTreeRegressor(random_state=12345)decision_tree_params_grid={"max_depth":np.arange(1,5)}sho_search=SHO(model=decision_tree_model,param_grid=decision_tree_params_grid,train_dataset=(features_train,target_train),eval_dataset=(features_test,target_test),scoring_function=mean_squared_error,display_search=True,higher_better=False,)sho_search.train()Since we setdisplay_search=True, we will see the following grid search progress information:{'max_depth': 1}: mean_squared_error=(train=1087.6845, valid=5115.3861)
{'max_depth': 2}: mean_squared_error=(train=915.5997, valid=4915.6283)
{'max_depth': 3}: mean_squared_error=(train=772.2890, valid=3686.3358)
{'max_depth': 4}: mean_squared_error=(train=696.7085, valid=3352.8290)We can use the class attribute after callingtrainto obtain the best model settings and results:sho_search.best_config_# {'max_depth': 4}sho_search.best_result_# 3352.8289830072163InstallationPackage dependenciesReardenlibrary requires the following dependencies:PackageVersionMatplotlib>= 3.3.4Pandas>= 1.2.4NumPy>= 1.24.3Scikit-learn>= 1.1.3Seaborn>= 0.11.1Statsmodels>= 0.13.2NOTE:The package currently requires Python 3.9 or higher.Installation usingpipThe package is available onPyPI Indexand can be easily installed usingpip:pip install reardenThe dependencies are automatically downloaded when executing the above command or can be installed manually using (after cloning the repo):pip install -r requirements.txtBuilding the packageThanks to the build system requirements and other metadata specified inpyproject.tomlit is easy to build and install the package. Firstly, clone the repository:git clone https://github.com/spolivin/rearden.git

cd reardenThen, one can simply run the following:pip install -e .Automatic code style checksInstallation ofpre-commitBefore pushing the changed code to the remote Github repository, the code undergoes numerous checks conducted with the help ofpre-commit hooksspecified in.pre-commit-config.yaml. Before making use of this feature, it is important to first downloadpre-commitpackage to the system:pip install pre-commitor ifreardenpackage has already been installed (pre-commitis installed together with the linters):pip install rearden[linters]Afterwards, in the git-repository run the following command for installation:pre-commit installNow, thepre-commit hookscan be easily used for verifying the code style.Pre-commit hooksAfter runninggit commit -m "<Commit message>"in the terminal, the file to be committed goes through a few checks before being enabled to be committed. As specified in.pre-commit-config.yaml, the following hooks are used:HooksVersionPre-commit-hooks4.3.0Pyupgrade3.10.1Autoflake2.1.1Isort5.12.0Black23.3.0Flake86.0.0Codespell2.2.4NOTE:Check.pre-commit-config.yamlfor more information about the repos and hooks used.