Package: collinearity
IntroductionThis library implements some functionf for removing collinearity from a dataset of features. It can be used both for supervised and for unsupervised machine learning problems.Collinearity is evaluated calculatingPearson's linear correlation coefficientbetween the features. The user sets athreshold, which is the maximum absolute value allowed for the correlation coefficients in the correlation matrix.Forunsupervised problems, the algorithm selects only those features that produce a correlation matrix whose off-diagonal elements are, in absolute value, less than the threshold.Forsupervised problems, the importance of the features with respect to the target variable is calculated using a univariate approach. Then, the features are added with the same unsupervised approach, starting from the most important ones.ObjectsThe main object isSelectNonCollinear. It can be imported this way:fromcollinearityimportSelectNonCollinearcollinearity.SelectNonCollinear(correlation_threshold=0.4, scoring=f_classif)Parameters:correlation_threshold :float (between 0 and 1), default = 0.4Only those features that produce a correlation matrix with off-diagonal elements that are, in absolute value, less than this threshold will be chosen.scoring :callable, default=f_classifThe scoring function for supervised problems. It must be the same accepted bysklearn.feature_selection.SelectKBest.MethodsThis object supports the main methods of scikit-learn Estimators:fit(X,y=None)Identifies the features to consider. For supervised problems,yis the target array and the algorithm is:Sort the features by scoring descendingTake the most important feature (i.e. the first feature)Take the next feature if it shows a linear correlation coefficient with the already selected feature that is, in absolute value, lower than the thresholdKeep adding features as long as the correlation constraint holdsFor unsupervised problems, we havey = Noneand the algorithm is:Take the couple of features that have the lowest absolute value of the linear correlation coefficient.If it's lower than the threshold, consider these featuresKeep adding features as long as the correlation matrix doesn't show off-diagonal elements whose absolute value is greater than the threshold.transform(X)Selects the features according to the result offit. It must be called afterfit.fit_transform(X,y=None)Callsfitand thentransformget_support()Returns an array ofTrueandFalseof size X.shape[1]. A feature is selected if the value on this array corresponding to its index isTrue, otherwise it's not selected.ExamplesThe following examples explain how the main objects work. The code to run in advance for initializing the environment is:fromcollinearityimportSelectNonCollinearfromsklearn.feature_selectionimportf_regressionimportnumpyasnpfromsklearn.datasetsimportload_diabetesX,y=load_diabetes(return_X_y=True)Unsupervised problemsThis example shows how to perform selection according to minimum collinearity in unsupervised problems.Let's consider, for this example, a threshold equal to 0.3.selector=SelectNonCollinear(0.3)If we apply the selection to the features and calculate the correlation matrix, we have:np.corrcoef(selector.fit_transform(X),rowvar=False)# array([[1.       , 0.1737371 , 0.18508467, 0.26006082],#       [0.1737371 , 1.        , 0.0881614 , 0.03527682],#       [0.18508467, 0.0881614 , 1.        , 0.24977742],#       [0.26006082, 0.03527682, 0.24977742, 1.        ]])As we can see, no off-diagonal element is greater than the threshold.Supervised problemsFor this problem, we must set the value of thescoringargument in the constructor.Let's consider a threshold equal to 0.4 and a scoring equal tof_regression.selector=SelectNonCollinear(correlation_threshold=0.4,scoring=f_regression)selector.fit(X,y)The correlation matrix is:np.corrcoef(selector.transform(X),rowvar=False)# array([[ 1.       ,  0.1737371 ,  0.18508467,  0.33542671,  0.26006082,#        -0.07518097,  0.30173101],#       [ 0.1737371 ,  1.        ,  0.0881614 ,  0.24101317,  0.03527682,#        -0.37908963,  0.20813322],#       [ 0.18508467,  0.0881614 ,  1.        ,  0.39541532,  0.24977742,#        -0.36681098,  0.38867999],#       [ 0.33542671,  0.24101317,  0.39541532,  1.        ,  0.24246971,#        -0.17876121,  0.39042938],#       [ 0.26006082,  0.03527682,  0.24977742,  0.24246971,  1.        ,#         0.05151936,  0.32571675],#       [-0.07518097, -0.37908963, -0.36681098, -0.17876121,  0.05151936,#         1.        , -0.2736973 ],#       [ 0.30173101,  0.20813322,  0.38867999,  0.39042938,  0.32571675,#        -0.2736973 ,  1.        ]])Again, no off-diagonal element is greater than the threshold in absolute value.Use in pipelinesIt's possible to useSelectNonCollinearinside a pipeline, if necessary.pipeline=make_pipeline(SelectNonCollinear(correlation_threshold=0.4,scoring=f_regression),LinearRegression())Contact the authorFor any questions, you can contact me atgianluca.malato@gmail.com