Package: discover-feature-relationships
discover_feature_relationshipsAttempt to discover 1D relationships between all columns in a DataFrame using scikit-learn (RandomForests) and standard correlation tests (Pearson, Spearman and Kendall via Pandas).The goal is to see if we can better understand the data in a DataFrame by learning which features (1 column at a time) predict each other column. This code attempts to learn a predictive relationship between the Cartesian product (all pairs) of all columns.Rather than just learning which column(s) predict a target column, we might want to know what other relationships exist (e.g. during Exploratory Data Analysis) and whether some predictive features are driven by other less-predictive features (to help us find new & better features or data sources). We might also sense-check out data by checking that certain relationships exist.By default it assumes every target column is a regression challenge. You can provide a list of columns to treat as classification challenges. For regression we cap negative scores at 0 (r^2 can be arbitrarily negative, we cap at 0 to make this a little easier to interpret).Text-encoded columns are automatically LabelEncoded (this is a sensible default but may not reveal information in your case, you might need to provide your own smarter encoding). This adds to the correlation plots in YellowBrick and Pandas Profiling where the text columns are not auto-encoded.We might want to use this tool alongside:https://github.com/ResidentMario/missingnohttps://github.com/pandas-profiling/pandas-profilingThe project (and the examples) live on GitHub:https://github.com/ianozsvald/discover_feature_relationshipsTitanic exampleTitanic NotebookEmbarked (classification) is predicted well by Fare, also by AgePclass (regression) is predicted by Fare but Fare (regression) is poorly predicted by PclassSex (classification) is predicted well by SurvivedSurvived (classification) is predicted well by Sex, Fare, Pclass, SibSpParchPredicting this feature at circa 0.62 is equivalent to "no information" as 0.62 is the mean of SurvivedSibSpParch is predicted by both SibSp and Parch (SibSpParch is the sum of both - it is an engineered additional feature) - it is also predicted by FareSibSp and Parch are also predicted by Fare (but less well so than by SibSpParch)This is generated using:df = pd.read_csv("titanic_train.csv")
...

import discover
df_results = discover.discover(cols, classifier_overrides, df)

df_results.pivot(index='target', columns='feature', values='score').fillna(1) \
.style.background_gradient(cmap="viridis", low=0.3, high=0.0, axis=1) \
.set_precision(2)Boston exampleBoston NotebookNOX predicts AGE and DIS (but not the other way around)target predicts LSTAT, LSTAT weakly predicts target, LSTAT weakly predicts RMDIS predicts AGE, AGE weakly predicts DISINDUS predicts CRIM and somewhat AGE, Btarget weakly predicts RM, RM weakly predicts targetRequirementspython 3.6+scikit-learn (0.19+)pandasjupyter notebookmatplotlibseabornconda install scikit-learn pandas jupyter pytest seaborn
conda install -c conda-forge watermarkSetupInstall from PyPIpip install discover_feature_relationshipshttps://pypi.org/project/discover-feature-relationships/Install from sourceFirst check-out from GitHub, then install withpython setup.py install, thencdinto theexamplesfolder and run the Notebooks.TestsRundiscover.pyfor a simple test that the code is workingRunpytestto runtest_discover.pyfor a single unit test (usepytest -sto seeprintoutputs)Note to Ian for DevelopmentEnvironment:. ~/anaconda3/bin/activate discover_feature_relationshipsInstallerTo push to PyPI I need to followhttps://docs.python.org/3/distributing/index.html#distributing-index- specificallypython se tup.py sdist bdist_wheelandtwine upload dist/*. This useshttps://pypi.org/project/twine/.TODOconsider version info:https://github.com/pallets/flask/blob/master/setup.pyconsider python_requires line in setupconsider adding dependenciesadd dependencies to setupclean up the list of linksbump version nbradd conda installer