Package: data-science-utils
Data Science Utils: Frequently Used Methods for Data ScienceData Science Utils extends the Scikit-Learn API and Matplotlib API to provide simple methods that simplify task and
visualization over data.Code Examples and DocumentationLet's see some code examples and outputs.You can read the full documentation with all the code examples from:https://datascienceutils.readthedocs.io/en/latest/In the documentation you can find more methods and more examples.The API of the package is build to work with Scikit-Learn API and Matplotlib API. Here are some of capabilities of this
package:MetricsPlot Confusion MatrixComputes and plot confusion matrix, False Positive Rate, False Negative Rate, Accuracy and F1 score of a classification.fromds_utils.metricsimportplot_confusion_matrixplot_confusion_matrix(y_test,y_pred,[0,1,2])Plot Metric Growth per Labeled InstancesReceives a train and test sets, and plots given metric change in increasing amount of trained instances.fromds_utils.metricsimportplot_metric_growth_per_labeled_instancesplot_metric_growth_per_labeled_instances(x_train,y_train,x_test,y_test,{"DecisionTreeClassifier":DecisionTreeClassifier(random_state=0),"RandomForestClassifier":RandomForestClassifier(random_state=0,n_estimators=5)})Visualize Accuracy Grouped by ProbabilityReceives test true labels and classifier probabilities predictions, divide and classify the results and finally
plots a stacked bar chart with the results.Original codefromds_utils.metricsimportvisualize_accuracy_grouped_by_probabilityvisualize_accuracy_grouped_by_probability(test["target"],1,classifier.predict_proba(test[selected_features]),display_breakdown=False)Without breakdown:With breakdown:PreprocessVisualize FeatureReceives a feature and visualize its values on a graph:If the feature is float then the method plots the distribution plot.If the feature is datetime then the method plots a line plot of progression of amount thought time.If the feature is object, categorical, boolean or integer then the method plots count plot (histogram).fromds_utils.preprocessimportvisualize_featurevisualize_feature(X_train["feature"])Feature TypePlotFloatIntegerDatetimeCategory / ObjectBooleanGet Correlated FeaturesCalculate which features correlated above a threshold and extract a data frame with the correlations and correlation to
the target feature.fromds_utils.preprocessimportget_correlated_featurescorrelations=get_correlated_features(train,features,target)level_0level_1level_0_level_1_corrlevel_0_target_corrlevel_1_target_corrincome_category_Lowincome_category_Medium1.00.11821656093586500.11821656093586504term_ 36 monthsterm_ 60 months1.00.11821656093586500.11821656093586504interest_payments_Highinterest_payments_Low1.00.11821656093586500.11821656093586504Visualize CorrelationsCompute pairwise correlation of columns, excluding NA/null values, and visualize it with heat map.Original codefromds_utils.preprocessimportvisualize_correlationsvisualize_correlations(data)Plot Correlation DendrogramPlot dendrogram of a correlation matrix. This consists of a chart that that shows hierarchically the variables that are
most correlated by the connecting trees. The closer to the right that the connection is, the more correlated the
features are.Original codefromds_utils.preprocessimportplot_correlation_dendrogramplot_correlation_dendrogram(data)Plot Features' InteractionPlots the joint distribution between two features:If both features are either categorical, boolean or object then the method plots the shared histogram.If one feature is either categorical, boolean or object and the other is numeric then the method plots a boxplot chart.If one feature is datetime and the other is numeric or datetime then the method plots a line plot graph.If one feature is datetime and the other is either categorical, boolean or object the method plots a violin plot (combination of boxplot and kernel density estimate).If both features are numeric then the method plots scatter graph.fromds_utils.preprocessimportplot_features_interactionplot_features_interaction("feature_1","feature_2",data)NumericCategoricalBooleanDatetimeNumericCategoricalBooleanDatetimeStringsAppend Tags to FrameExtracts tags from a given field and append them as dataframe.A dataset that looks like this:x_train:article_namearticle_tags1ds,ml,dl2ds,mlx_test:article_namearticle_tags3ds,ml,pyUsing this code:importpandasaspdfromds_utils.stringsimportappend_tags_to_framex_train=pd.DataFrame([{"article_name":"1","article_tags":"ds,ml,dl"},{"article_name":"2","article_tags":"ds,ml"}])x_test=pd.DataFrame([{"article_name":"3","article_tags":"ds,ml,py"}])x_train_with_tags,x_test_with_tags=append_tags_to_frame(x_train,x_test,"article_tags","tag_")will be parsed into this:x_train_with_tags:article_nametag_dstag_mltag_dl11112110x_test_with_tags:article_nametag_dstag_mltag_dl3110Extract Significant Terms from SubsetReturns interesting or unusual occurrences of terms in a subset. Based on theelasticsearch significant_text aggregation.importpandasaspdfromds_utils.stringsimportextract_significant_terms_from_subsetcorpus=['This is the first document.','This document is the second document.','And this is the third one.','Is this the first document?']data_frame=pd.DataFrame(corpus,columns=["content"])# Let's differentiate between the last two documents from the full corpussubset_data_frame=data_frame[data_frame.index>1]terms=extract_significant_terms_from_subset(data_frame,subset_data_frame,"content")And the following table will be the output forterms:thirdoneandthistheisfirstdocumentsecond1.01.01.00.670.670.670.50.250.0UnsupervisedCluster CardinalityCluster cardinality is the number of examples per cluster. This method plots the number of points per cluster as a bar
chart.importpandasaspdfrommatplotlibimportpyplotaspltfromsklearn.clusterimportKMeansfromds_utils.unsupervisedimportplot_cluster_cardinalitydata=pd.read_csv(path/to/dataset)estimator=KMeans(n_clusters=8,random_state=42)estimator.fit(data)plot_cluster_cardinality(estimator.labels_)plt.show()Plot Cluster MagnitudeCluster magnitude is the sum of distances from all examples to the centroid of the cluster. This method plots the
Total Point-to-Centroid Distance per cluster as a bar chart.importpandasaspdfrommatplotlibimportpyplotaspltfromsklearn.clusterimportKMeansfromscipy.spatial.distanceimporteuclideanfromds_utils.unsupervisedimportplot_cluster_magnitudedata=pd.read_csv(path/to/dataset)estimator=KMeans(n_clusters=8,random_state=42)estimator.fit(data)plot_cluster_magnitude(data,estimator.labels_,estimator.cluster_centers_,euclidean)plt.show()Magnitude vs. CardinalityHigher cluster cardinality tends to result in a higher cluster magnitude, which intuitively makes sense. Clusters
are anomalous when cardinality doesn't correlate with magnitude relative to the other clusters. Find anomalous
clusters by plotting magnitude against cardinality as a scatter plot.importpandasaspdfrommatplotlibimportpyplotaspltfromsklearn.clusterimportKMeansfromscipy.spatial.distanceimporteuclideanfromds_utils.unsupervisedimportplot_magnitude_vs_cardinalitydata=pd.read_csv(path/to/dataset)estimator=KMeans(n_clusters=8,random_state=42)estimator.fit(data)plot_magnitude_vs_cardinality(data,estimator.labels_,estimator.cluster_centers_,euclidean)plt.show()Optimum Number of Clustersk-means requires you to decide the number of clusterskbeforehand. This method runs the KMean algorithm and
increases the cluster number at each try. The Total magnitude or sum of distance is used as loss.Right now the method only works withsklearn.cluster.KMeans.importpandasaspdfrommatplotlibimportpyplotaspltfromscipy.spatial.distanceimporteuclideanfromds_utils.unsupervisedimportplot_loss_vs_cluster_numberdata=pd.read_csv(path/to/dataset)plot_loss_vs_cluster_number(data,3,20,euclidean)plt.show()XAIGenerate Decision PathsReceives a decision tree and return the underlying decision-rules (or 'decision paths') as text (valid python syntax).Original codefromsklearn.treeimportDecisionTreeClassifierfromds_utils.xaiimportgenerate_decision_paths# Create decision tree classifier objectclf=DecisionTreeClassifier(max_depth=3)# Train modelclf.fit(x,y)print(generate_decision_paths(clf,feature_names,target_names.tolist(),"iris_tree"))The following text will be printed:def iris_tree(petal width (cm), petal length (cm)):
    if petal width (cm) <= 0.8000:
        # return class setosa with probability 0.9804
        return ("setosa", 0.9804)
    else:  # if petal width (cm) > 0.8000
        if petal width (cm) <= 1.7500:
            if petal length (cm) <= 4.9500:
                # return class versicolor with probability 0.9792
                return ("versicolor", 0.9792)
            else:  # if petal length (cm) > 4.9500
                # return class virginica with probability 0.6667
                return ("virginica", 0.6667)
        else:  # if petal width (cm) > 1.7500
            if petal length (cm) <= 4.8500:
                # return class virginica with probability 0.6667
                return ("virginica", 0.6667)
            else:  # if petal length (cm) > 4.8500
                # return class virginica with probability 0.9773
                return ("virginica", 0.9773)Plot Features` Importanceplot feature importance as a bar chart.importpandasaspdfrommatplotlibimportpyplotaspltfromsklearn.treeimportDecisionTreeClassifierfromds_utils.xaiimportplot_features_importancedata=pd.read_csv(path/to/dataset)target=data["target"]features=data.columns.to_list()features.remove("target")clf=DecisionTreeClassifier(random_state=42)clf.fit(data[features],target)plot_features_importance(features,clf.feature_importances_)plt.show()Excited?Read about all the modules here and see more abilities:Metrics- The module of metrics contains methods that help to calculate and/or visualize evaluation performance of an algorithm.Preprocess- The module of preprocess contains methods that are processes that could be made to data before training.Strings- The module of strings contains methods that help manipulate and process strings in a dataframe.Unsupervised- The module od unsupervised contains methods that calculate and/or visualize evaluation performance of an unsupervised model.XAI- The module of xai contains methods that help explain a model decisions.ContributingInterested in contributing to Data Science Utils? Great! You're welcome,  and we would love to have you. We follow
thePython Software Foundation Code of ConductandMatplotlib Usage Guide.No matter your level of technical skill, you can be helpful. We appreciate bug reports, user testing, feature
requests, bug fixes, product enhancements, and documentation improvements.Thank you for your contributions!Find a Bug?Check if there's already an openissueon the topic. If
needed, file an issue.Open SourceData Science Utils license isMIT License.Installing Data Science UtilsData Science Utils is compatible with Python 3.6 or later. The simplest way to install Data Science Utils and its
dependencies is from PyPI with pip, Python's preferred package installer:pipinstalldata-science-utilsNote that this package is an active project and routinely publishes new releases with more methods.  In order to
upgrade Data Science Utils to the latest version, use pip as follows:pipinstall-Udata-science-utilsAlternatively you can install from source by cloning the repo and running:gitclonehttps://github.com/idanmoradarthas/DataScienceUtils.gitcdDataScienceUtils
pipinstall.Or installation using pip from source:pipinstallgit+https://github.com/idanmoradarthas/DataScienceUtils.gitIf you're using Anaconda, you can install using conda:condainstallidanmorad::data-science-utils