Package: atlantic
Atlantic - Automated Data Preprocessing Framework for Supervised Machine LearningFramework ContextualizationTheAtlanticproject constitutes an comprehensive and objective approach to simplify and automate data processing through the integration and objectively validated application of various preprocessing mechanisms, ranging from feature engineering, automated feature selection, multiple encoding versions and null imputation methods. The optimization methodology of this framework follows a evaluation structured in tree based models ensembles.This project aims at providing the following application capabilities:General applicability on tabular datasets: The developed preprocessing procedures are applicable on any data table associated with Supervised Machine Learning scopes regardless of the properties or specifications of the Dataset features.Automated treatment of tabular data associated with predictive analysis: It implements a global and carefully validated tested data treatment based on the characteristics of each Dataset input columns, assuming its identification with the associated target column.Robustness and improvement of predictive results: The implementation of theatlanticautomated data preprocessing pipeline aims at improving predictive performance directly associated with the processing methods implemented based on the Dataset properties.Main Development ToolsMajor frameworks used to built this project:H2O.aiScikit-learnXGBoostOptunaPandasFramework ArchitectureWhere to get itBinary installer for the latest released version is available at the Python Package IndexPyPI.The source code is currently hosted on GitHub at:https://github.com/TsLu1s/AtlanticInstallationTo install this package from Pypi repository run the following command:pip install atlanticUsage Examples1. Atlantic - Automated Data Preprocessing PipelineIn order to be able to apply the automated preprocessingatlanticpipeline you need first to import the package.
The following needed step is to load a dataset and define your to be predicted target column name into the variabletargetand define split ratio for your Train and Validation subsets.
You can customize thefit_processingmethod by altering the following running pipeline parameters:split_ratio: Division ratio in which the preprocessing methods will be evaluated within the loaded Dataset.relevance: Minimal value of the total sum of relative variable\feature importance percentage selected in theH2O AutoML feature selectionstep.h2o_fs_models: Quantity of models generated for competition in stepH2O AutoML feature selectionto evaluate the relative importance of each feature (only leaderboard model is selected for evaluation).encoding_fs: You can choose if you want to encond your features in order to reduce loading time inH2O AutoML feature selectionstep. If inTruemode label encoding is applied to categorical features.vif_ratio: This value defines the minimalthresholdfor Variance Inflation Factor filtering (default value=10).Importante Notes:Default predictive evaluation metric for regression contexts isMean Absolute Errorand classification isAccuracy.Although functional,Atlanticdata processing is not optimized for big data purposes yet.Major update is now available inversions>=1.1.0fromatlantic.pipelineimportATLpipelineimportpandasaspdfromsklearn.model_selectionimporttrain_test_splitimportwarningswarnings.filterwarnings("ignore",category=Warning)# -> For a clean consoledata=pd.read_csv('csv_directory_path',encoding='latin',delimiter=',')# Dataframe Loading Exampletrain,test=train_test_split(data,train_size=0.8)train,test=train.reset_index(drop=True),test.reset_index(drop=True)# -> Required### Fit Data Processingatl=ATLpipeline(X=train,# X:pd.DataFrame, target:str="Target_Column"target="Target Column")atl.fit_processing(split_ratio=0.75,# split_ratio:float=0.75, relevance:float=0.99 [0.5,1]relevance=0.99,# h2o_fs_models:int [1,50], encoding_fs:bool=True\Falseh2o_fs_models=7,# vif_ratio:float=10.0 [3,30]encoding_fs=True,vif_ratio=10.0)### Transform Data Processingtrain=atl.data_processing(X=train)test=atl.data_processing(X=test)### Export Atlantic Preprocessing Metadataimportpickleoutput=open("fit_atl.pkl",'wb')pickle.dump(atl,output)2. Atlantic - Preprocessing Data2.1 Encoding VersionsThere are multiple preprocessing methods available to direct use. This package provides upgrated encodingLabelEncoder,OneHotEncoderandIDFmethods with an automatic multicolumn application.fromatlantic.processingimportAutoLabelEncoder,AutoIdfEncoder,AutoOneHotEncoderimportpandasaspdfromsklearn.model_selectionimporttrain_test_splittrain,test=train_test_split(data,train_size=0.8)train,test=train.reset_index(drop=True),test.reset_index(drop=True)# Requiredtarget="Target_Column"# -> target feature namecat_cols=[colforcolindata.select_dtypes(include=['object']).columnsifcol!=target]### Encoders## Create Label Encoderencoder=AutoLabelEncoder()## Create IDF Encoderencoder=AutoIdfEncoder()## Create One-hot Encoderencoder=AutoOneHotEncoder()## Fitencoder.fit(train[cat_cols])# Transform the DataFrame using Label\IDF\One-hot Encodingtrain=encoder.transform(X=train)test=encoder.transform(X=test)# Label Encoding : Perform an inverse transform to convert it back the categorical columns valuestest=encoder.inverse_transform(X=test)# IDF & One-hot Encoding : Perform an inverse transform to convert it back the categorical columns values# Note: Only decodes the last transformed Dataframetest=encoder.inverse_transform()2.2 Feature Selection MethodsYou can get filter your most valuable features from the dataset via this 2 feature selection methods:H2O AutoML Feature Selection- This method is based of variable importance evaluation and calculation for tree-based models in H2Os AutoML and it can be customized by use of the following parameters:relevance: Minimal value of the total sum of relative variable\feature importance percentage selected.h2o_fs_models: Quantity of models generated for competition to evaluate the relative importance of each feature (only leaderboard model will be selected for evaluation).encoding_fs: You can choose if you want to encond your features in order to reduce loading time. If inTruemode label encoding is applied to categorical features.VIF Feature Selection (Variance Inflation Factor)- Variance inflation factor aims at measuring the amount of multicollinearity in a set of multiple regression variables or features, therefore for this filtering method to be applied all input variables need to be of numeric type. It can be customized by changing the column filtering tresholdvif_thresholddesignated with a default value of 10.fromatlantic.selectorimportSelectorfs=Selector(X=train,target="Target_Column")cols_vif=fs.feature_selection_vif(vif_threshold=10.0)# X: Only numerical values allowed & No nans allowed in VIFselected_cols,selected_importance=fs.feature_selection_h2o(relevance=0.99,# relevance:float [0.5,1], h2o_fs_models:int [1,50]h2o_fs_models=7,# encoding_fs:bool=True/Falseencoding_fs=True)2.3 Null Imputation Auxiliar MethodsSimplified and automated multivariate null imputation methods based fromSklearnare also provided and applicable, as following:## Simplified Null Imputation (Only numeric features)fromatlantic.imputationimportAutoSimpleImputer,AutoKNNImputer,AutoIterativeImputer# Example usage of AutoSimpleImputersimple_imputer=AutoSimpleImputer(strategy='mean')simple_imputer.fit(train)# Fit on the Train DataFramedf_imputed=simple_imputer.transform(train.copy())# Transform the Train DataFramedf_imputed_test=simple_imputer.transform(test.copy())# Transform the Test DataFrame# Example usage of AutoKNNImputerknn_imputer=AutoKNNImputer(n_neighbors=3,weights="uniform")knn_imputer.fit(train)# Fit on the Train DataFramedf_imputed=knn_imputer.transform(train.copy())# Transform the Train DataFramedf_imputed_test=knn_imputer.transform(test.copy())# Transform the Test DataFrame# Example usage of AutoIterativeImputeriterative_imputer=AutoIterativeImputer(max_iter=10,random_state=0,initial_strategy="mean",imputation_order="ascending")iterative_imputer.fit(train)# Fit on the Train DataFramedf_imputed=iterative_imputer.transform(train.copy())# Transform the Train DataFramedf_imputed_test=iterative_imputer.transform(test.copy())# Transform the Test DataFrameCitationFeel free to cite Atlantic as following:@article{SANTOS2023100532,
  author = {Luis Santos and Luis Ferreira}
  title = {Atlantic - Automated data preprocessing framework for supervised machine learning},
  journal = {Software Impacts},
  volume = {17},
  year = {2023},
  issn = {2665-9638},
  doi = {http://dx.doi.org/10.1016/j.simpa.2023.100532},
  url = {https://www.sciencedirect.com/science/article/pii/S2665963823000696}
}LicenseDistributed under the MIT License. SeeLICENSEfor more information.ContactLuis Santos - LinkedIn