Package: rfcc
Python RFCC - Data understanding, clustering and outlier detection for regression and classification tasksRandom forests are invariant and robust estimators that can fit complex interactions between input data of different types and binary, categorical, or continuous outcome variables, including those with multiple dimensions. In addition to these desirable properties, random forests impose a structure on the observations from which researchers and data analysts can infer clusters or groups of interest.You can use these clusters to:structure your data,elucidate new patterns of how features influence outcomes,define subgroups for further analysis,derive prototypical observations,identify outlier observations,catch mislabeled data,evaluate the performance of the estimation model in more detail.Random Forest Consensus Clustering and implement is implemented in the Scikit-Learn / SciPy data science ecosystem. This algorithm differs from prior approaches by making use of the entire tree structure. Observations become proximate if they follow similar decision paths across trees of a random forest.More info in here:Marquart, Ingo and Koca Marquart, Ebru, RFCC: Random Forest Consensus Clustering for Regression and Classification (March 19, 2021). Available at SSRN: https://ssrn.com/abstract=3807828 or http://dx.doi.org/10.2139/ssrn.3807828InstallationInstall via pip!pipinstallrfccUsageLet's illustrate the approach with a simple example. We will be regression the miles-per-gallon in the city (cty) performance of a set of
cars on the class (compact, pick-up etc.), the number of cylinders and the engine displacement.The data is available in the pydataset packagedataset=data("mpg")y_col=["cty"]x_col=['displ','class','cyl']Y=dataset[y_col]X=dataset[x_col]print(X.head(5))displclasscyl11.8compact421.8compact432.0compact442.0compact452.8compact6We wantclassandcylto be treated as categorical variable, so we'll keep track of these columns.Initialization and model choiceThe first step is to initialize the model, much like one would initialize an scikit-learn model.
The main class iscluster_modelfrom the rfcc package.
We only need to pass an appropriate ensemble model (RandomForestClassifier, RandomForestRegressor) and specify the options we'd like to use.Since miles-per-gallon is a continuous measure, we'll be using a random forest regression.fromsklearn.ensembleimportRandomForestRegressorfromrfcc.modelimportcluster_modelmodel=cluster_model(model=RandomForestRegressor,max_clusters=20,random_state=1)We have two options to specify the size and number of clusters to be returned.The parametermax_clusterssets the maximum amount of leafs in each decision tree. It ensures that the model does not return too many or too few clusters, but it does change the estimation of the random forest.Another option is to setmax_clustersto a high value, or leave it unspecified, and use the hierarchical clustering algorithm to extract clusters of the desired size. See below fort_paramin the fit method.Fitting and optional parametersNow we need to fit our model to the data.model.fit(X,Y)The following optional parameters can be passedencode(list): A list of columns that we'd like to encode before fitting the model. Note that all non-numerical columns will be encoded automatically. However, you can also encode numerical data by passing it in theencodeparameter.encode_y(bool): You can choose to ordinally encode the outcome variables. If you do a classification, scikit learn will choose how to encode the outcome variables. If the variable is continuous, this will usually lead to a rather bad fit, in which case you may want to encode.linkage_method(str): Linkage method used in the clustering algorithm (average, single, complete, ward)clustering_type(str): "rfcc" (default) our path based clustering, or "binary" as in prior approachest_param(float): If None, number of clusters corresponds to average number of leafs. Ift_paramis specified,
pick that level of clustering hierarchy where distance between members of the group is less thant_param. The higher the value, the larger average size of a cluster.Let's check how well our model does on our training setmodel.score(X,Y)0.9231321010907177Cluster compositionsOnce the model is fit, we can extract the composition of clusters.
Let's see which car types and cylinders have the best and worst miles-per-gallon performance.First, we use the cluster_descriptions method to return the compositions for each cluster.clusters=model.cluster_descriptions(variables_to_consider=['class','manufacturer'],continuous_measures="mean")The optional parameters are:variables_to_consider(list): List of columns in X to take into account.continuous_measures(str, list): Measures to compute for each continuous feature (mean, std, median, max, min, skew)We will sort our clusters by the average mpg and return the clusters with the two highest and two lowest mpg performances.clusters=clusters.sort_values(by="cty-mean")print(clusters.head(2))print(clusters.tail(2))Nr_Obscty-meanclassmanufacturer711.85suv:1.0%ford:0.29%,landrover:0.57%,mercury:0.14%4912.02pickup:0.35%,suv:0.63%chevrolet:0.18%,dodge:0.43%,ford:0.12%,jeep:0.1%,lincoln:0.06%,mercury:0.02%,nissan:0.02%,toyota:0.06%Nr_Obscty-meanclassmanufacturer1524.4compact:0.33%,midsize:0.13%,subcompact:0.53%honda:0.53%,toyota:0.33%,volkswagen:0.13%332.3compact:0.33%,subcompact:0.67%volkswagen:1.0%Decision Path AnalysisCluster descriptions return the proportions of values for any feature we are interested in. However, we also may want to know how a decision tree classifies an observation. For example, it may be that the featuremanufacturerhas
no predictive value, whereas the number of cylinders or the displacement does.Another reason to do a decision path analysis is to check whetherCurrently, path analyses are queried for each estimator in the random forest.
In the future patch, the path analysis will be available for the entire random forest.Let's see how the first decision tree (index 0) classifies the observations with the lowest miles-per-gallon performancepaths=model.path_analysis(estimator_id=0)paths.sort_values(by="Output_cty")print(paths.head(5))Nr_ObsOutput_ctyclassdisplmanufacturer17[11.4]classisnot:2seater,compactdisplbetween5.25and4.4manufacturer:audi,chevrolet,dodge21[12.4]class:suvdispllargerthan:4.4manufacturerisnot:audi,chevrolet,dodge5[12.6]class:midsize,minivan,pickupdispllargerthan:4.4manufacturerisnot:audi,chevrolet,dodge13[12.6]classisnot:2seater,compactdispllargerthan:5.25manufacturer:audi,chevrolet,dodge5[13.4]class:minivandisplbetween3.75and3.15-22[14.1]-displbetween4.4and3.85-Detection of outliers and mislabelled dataOutliers are observations that are unusual - not necessarily because their features differ, but rather because their implications for the outcome variable are different from other comparable observations. Mislabelled data may appear as
outlier, since the relationships between outcome and feature values may not make much sense.Since outliers follow distinct decision paths in the random forest, RFCC does not cluster them with other observations.
We can therefore find outliers by analyzing clusters that have very few observations.Let's see what outliers exist in the mpg data.clusters=model.cluster_descriptions(continuous_measures="mean")clusters=clusters.sort_values(by="Nr_Obs")outliers=clusters.head(2)print(outliers)Cluster_IDNr_Obscty-meanclasscylmanufacturerdispl-mean16116.0minivan:1.0%6:1.0%dodge:1.0%4.03218.0midsize:1.0%6:1.0%hyundai:1.0%2.5It seems we have one cluster (id=16) with a dodge minivan, and a cluster (id=3) with two observations.
We can get the constituent observations directly from our model.ids=model.get_observations(cluster_id=16)print(dataset.iloc[ids,:])ids=model.get_observations(cluster_id=3)print(dataset.iloc[ids,:])manufacturermodeldisplyearcyltrans48dodgecaravan2wd4.020086auto(l6)manufacturermodeldisplyearcyltrans113hyundaisonata2.519996auto(l4)114hyundaisonata2.519996manual(m5)