Package: atgfe
ATgfe (Automated Transparent Genetic Feature Engineering)What is ATgfe?ATgfe stands for Automated Transparent Genetic Feature Engineering. ATgfe is powered by genetic algorithm to engineer new features. The idea is to compose new interpretable features based on interactions between the existing features. The predictive power of the newly constructed features are measured using a pre-defined evaluation metric, which can be custom designed.ATgfe applies the following techniques to generate candidate features:Simple feature interactions by using the basic operators (+, -, *, /).(petalwidth * petallength)Scientific feature interactions by applying transformation operators (e.g. log, cosine, cube, etc. as well as custom operators which can be easily implemented using user defined functions).squared(sepalwidth)*(log_10(sepalwidth)/squared(petalwidth))-cube(sepalwidth)Weighted feature interactions by adding weights to the simple and/or scientific feature interactions.(0.09*exp(petallength)+0.7*sepallength/0.12*exp(petalwidth))+0.9*squared(sepalwidth)Complex feature interactions by applying groupBy on the categorical features.(0.56*groupByYear0TakeMeanOfFeelslike*0.51*feelslike)+(0.45*temp)Why ATgfe?ATgfe allows you to deal withnon-linearproblems by generating newinterpretablefeatures from existing features. The generated features can then be used with a linear model, which is inherently explainable. The idea is to explore potential predictive information that can be represented using interactions between existing features.When compared with non-linear models (e.g. gradient boosting machines, random forests, etc.), ATgfe can achieve comparable results and in some cases over-perform them.
This is demonstrated in the following examples:BMI,Rational differenceandIRIS.ResultsGeneratedExpressionLinear RegressionLightGBM RegressorLinear Regression + ATgfeBMI = weight/height^2RMSE: 3.268r^2: 0.934RMSE: 0.624r^2: 0.996RMSE:0.0r^2:1.0Y = (X1 - X2) / (X3 - X4)RMSE: 141.261r^2: -0.068RMSE: 150.642r^2: -0.52RMSE:0.0r^2:1.0Y = (Log10(X1) + Log10(X2)) / X5RMSE: 0.140r^2: 0.899RMSE: 0.102r^2: 0.895RMSE:0.0r^2:1.0Y = 0.4X2^2 + 2X4 + 2RMSE: 30077.269r^2: 0.943RMSE: 980.297r^2: 1.0RMSE:0.0r^2:1.0ClassificationDatasetLogistic RegressionLightGBM ClassifierLogistic Regression + ATgfeIRIS  (4 features)10-CV Accuracy: 0.926Test-data Accuracy: 0.911ROC_AUC: 0.9910-CV Accuracy: 0.946Test-data Accuracy: 0.977ROC_AUC: 1.010-CV Accuracy:0.98Test-data Accuracy:1.0ROC_AUC:1.0RegressionDatasetLinear RegressionLightGBM RegressorLinear Regression + ATgfeConcrete  (8 features)10-CV RMSE: 11.13Test-data RMSE: 10.38r^2: 0.64410-CV RMSE: 6.44Test-data RMSE:4.23r^2:0.94110-CV RMSE:6.00Test-data RMSE: 5.45r^2: 0.899Boston  (13 features)10-CV RMSE: 4.796Test-data RMSE: 4.714r^2: 0.76510-CV RMSE: 3.38Test-data RMSE: 3.63r^2: 0.8610-CV RMSE:3.125Test-data RMSE:2.758r^2:0.920Get startedRequirementsPython ^3.6DEAP ^1.3Pandas ^0.25.2Scipy ^1.3Numpy ^1.17Sympy ^1.4Install ATgfepipinstallatgfeUpgrade ATgfepipinstall-UatgfeUsageExamplesTheExamplesare grouped under the following two sections:Generatedexamples test ATgfe against hand-crafted non-linear problems where we know there is information that can be captured using feature interactions.Toy Examplesshow how to use ATgfe in solving a mix of regression and classification problems from publicly available benchmark datasets.Pre-processing for column namesATgfe requires column names that are free from special characters and spaces (e.g. @, $, %, #, etc.)# exampledefprepare_column_names(columns):return[col.replace(' ','').replace('(cm)','_cm')forcolincolumns]columns=prepare_column_names(df.columns.tolist())df.columns=columnsConfiguring the parameters of GeneticFeatureEngineerGeneticFeatureEngineer(model,x_train:pandas.core.frame.DataFrame,y_train:pandas.core.frame.DataFrame,numerical_features:List[str],number_of_candidate_features:int,number_of_interacting_features:int,evaluation_metric:Callable[...,Any],minimize_metric:bool=True,categorical_features:List[str]=None,enable_grouping:bool=False,sampling_size:int=None,cv:int=10,fit_wo_original_columns:bool=False,enable_feature_transformation_operations:bool=False,enable_weights:bool=False,enable_bias:bool=False,max_bias:float=100.0,weights_number_of_decimal_places:int=2,shuffle_training_data_every_generation:bool=False,cross_validation_in_objective_func:bool=False,objective_func_cv:int=3,n_jobs:int=1,verbose:bool=True)modelATgfe works with any model or pipeline that follows scikit-learn API (i.e. the model should implement thefit()andpredict()methods).x_trainTraining features in a pandas Dataframe.y_trainTraining labels in a pandas Dataframe to also handle multiple target problems.numerical_featuresThe list of column names that represent the numerical features.number_of_candidate_featuresThe maximum number of features to be generated.number_of_interacting_featuresThe maximum number of existing features that can be used in constructing new features.
These features are selected from those passed in thenumerical_featuresargument.evaluation_metricAny of thescitkit-learn metricsor a custom-made evaluation metric to be used by the genetic algorithm to evaluate the predictive power of the newly generated features.importnumpyasnpfromsklearn.metricsimportmean_squared_errordefrmse(y_true,y_pred):returnnp.sqrt(mean_squared_error(y_true,y_pred))minimize_metricA boolean flag, which should be set toTrueif the evaluation metric is to be minimized; otherwise set toFalseif the evaluation metric is to be maximized.categorical_featuresThe list of column names that represent the categorical features. The parameterenable_groupingshould be set toTruein order for thecategorical_featuresto be utilized in grouping.enable_groupingA boolean flag, which should be set toTrueto construct complex feature interactions that usepandas.groupBy.sampling_sizeThe exact size of the sampled training dataset. Use this parameter to run the optimization using the specified number of observations in the training data. If thesampling_sizeis greater than the number of observations, then ATgfe will create a sample with replacement.cvThe number of folds for cross validation. Every generation of the genetic algorithm, ATgfe evaluates the current best solution using k-fold cross validation. The default number of folds is 10.fit_wo_original_columnsA boolean flag, which should be set toTrueto fit the model without the original features specified innumerical_features. In this case, ATgfe will only use the newly generated features together with any remaining original features inx_train.enable_feature_transformation_operationsA boolean flag, which should be set toTrueto enable scientific feature interactions on thenumerical_features.
The pre-defined transformation operators are listed as follows:np_log(), np_log_10(), np_exp(), squared(), cube()You can easily remove from or add to the existing list of transformation operators. Check out the next section for examples.enable_weightsA boolean flag, which should be set toTrueto enable weighted feature interactions.weights_number_of_decimal_placesThe number of decimal places (i.e. precision) to be applied to the weight values.enable_biasA boolean flag, which enables the genetic algorithm to add a bias to the expressions generated. For example:0.43*log(cement) + 806.8557595548646max_biasThe value of the bias will be between-max_biasandmax_bias.
If themax_biasis 100 then the bias value will be between -100 and 100.shuffle_training_data_every_generationA boolean flag, if enabled thetrain_test_splitmethod in the objective function uses the generation number as its random seed. This can prevent over-fitting.This option is only available ifcross_validation_in_objective_funcis set toFalse.cross_validation_in_objective_funcA boolean flag, if enabled thetrain_test_splitmethod will not be used in the objective function. Instead of usingtrain_test_split, the genetic algorithm will use cross validation to evaluate the generated features.The default number of folds is3. The number of folds can modified using theobjective_func_cvparameter.objective_func_cvThe number of folds to be used whencross_validation_in_objective_funcis enabled.verboseA boolean flag, which should be set toTrueto enable the logging functionality.n_jobsTo enable parallel processing, setn_jobsto the number of CPUs that you would like to utilise. Ifn_jobsis set to-1, all the machine's CPUs will be utilised.Configuring the parameters of fit()gfe.fit(number_of_generations:int=100,mu:int=10,lambda_:int=100,crossover_probability:float=0.5,mutation_probability:float=0.2,early_stopping_patience:int=5,random_state:int=77)number_of_generationsThe maximum number of generations to be explored by the genetic algorithm.muThe number of solutions to select for the next generation.lambda_The number of children to produce at each generation.crossover_probabilityThe crossover probability.mutation_probabilityThe mutation probability.early_stopping_patienceThe maximum number of generations to be explored before early the stopping criteria is satisfied when the validation score is not improving.Configuring the parameters of transform()X=gfe.transform(X)Where X is the pandas dataframe that you would like to append the generated features to.Transformation operationsGet current transformation operationsgfe.get_enabled_transformation_operations()The enabled transformation operations will be returned.['None', 'np_log', 'np_log_10', 'np_exp', 'squared', 'cube']Remove existing transformation operationsgfe.remove_transformation_operationaccepts string or a list of stringsgfe.remove_transformation_operation('squared')gfe.remove_transformation_operation(['np_log_10','np_exp'])Add new transformation operationsnp_sqrt=np.sqrtdefsome_func(x):return(x*2)/3gfe.add_transformation_operation('sqrt',np_sqrt)gfe.add_transformation_operation('some_func',some_func)