Package: mars-profiling
Pandas ProfilingDocumentation|Slack|Stack OverflowGenerates profile reports from a pandasDataFrame.The pandasdf.describe()function is great but a little basic for serious exploratory data analysis.pandas_profilingextends the pandas DataFrame withdf.profile_report()for quick data analysis.For each column the following statistics - if relevant for the column type - are presented in an interactive HTML report:Type inference: detect thetypesof columns in a dataframe.Essentials: type, unique values, missing valuesQuantile statisticslike minimum value, Q1, median, Q3, maximum, range, interquartile rangeDescriptive statisticslike mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewnessMost frequent valuesHistogramCorrelationshighlighting of highly correlated variables, Spearman, Pearson and Kendall matricesMissing valuesmatrix, count, heatmap and dendrogram of missing valuesText analysislearn about categories (Uppercase, Space), scripts (Latin, Cyrillic) and blocks (ASCII) of text data.File and Image analysisextract file sizes, creation dates and dimensions and scan for truncated images or those containing EXIF information.AnnouncementsVersion v2.9.0 releasedThe release candidate for v2.9.0 was already out for a while, now v2.9.0 is finally released. See the changelog below to know what has changed.Spark backend in progressWe can happily announce that we're working on a Spark backend for generating profile reports.
Stay tuned.Supportpandas-profilingThe development ofpandas-profilingrelies completely on contributions.
If you find value in the package, we welcome you to support the project throughGitHub Sponsors!
It's extra exciting that GitHubmatches your contributionfor the first year.Find more information here:Changelog v2.9.0Sponsor the project on GitHubSeptember 2, 2020 💘Contents:Examples|Installation|Documentation|Large datasets|Command line usage|Advanced usage|Support|Types|How to contribute|Editor Integration|DependenciesExamplesThe following examples can give you an impression of what the package can do:Census Income(US Adult Census data relating income)NASA Meteorites(comprehensive set of meteorite landings)Titanic(the "Wonderwall" of datasets)NZA(open data from the Dutch Healthcare Authority)Stata Auto(1978 Automobile data)Vektis(Vektis Dutch Healthcare data)Colors(a simple colors dataset)Specific features:Russian Vocabulary(demonstrates text analysis)Cats and Dogs(demonstrates image analysis from the file system)Celebrity Faces(demonstrates image analysis with EXIF information)Website Inaccessibility(demonstrates URL analysis)Orange pricesandCoal prices(showcases report themes)Tutorials:Tutorial: report structure using Kaggle data (advanced)(modify the report's structure)InstallationUsing pipYou can install using the pip package manager by runningpipinstallpandas-profiling[notebook]Alternatively, you could install the latest version directly from Github:pipinstallhttps://github.com/pandas-profiling/pandas-profiling/archive/master.zipUsing condaYou can install using the conda package manager by runningcondainstall-cconda-forgepandas-profilingFrom sourceDownload the source code by cloning the repository or by pressing'Download ZIP'on this page.Install by navigating to the proper directory and running:pythonsetup.pyinstallDocumentationThe documentation forpandas_profilingcan be foundhere. Previous documentation is still availablehere.Getting startedStart by loading in your pandas DataFrame, e.g. by using:importnumpyasnpimportpandasaspdfrommars_profilingimportProfileReportdf=pd.DataFrame(np.random.rand(100,5),columns=["a","b","c","d","e"])To generate the report, run:profile=ProfileReport(df,title="Pandas Profiling Report")Explore deeperYou can configure the profile report in any way you like. The example code below loads theexplorative configuration file, that includes many features for text (length distribution, unicode information), files (file size, creation time) and images (dimensions, exif information). If you are interested what exact settings were used, you can compare with thedefault configuration file.profile=ProfileReport(df,title='Pandas Profiling Report',explorative=True)Learn more about configuringpandas-profilingon theAdvanced usagepage.Jupyter NotebookWe recommend generating reports interactively by using the Jupyter notebook.
There are two interfaces (see animations below): through widgets and through a HTML report.This is achieved by simply displaying the report. In the Jupyter Notebook, run:profile.to_widgets()The HTML report can be included in a Jupyter notebook:Run the following code:profile.to_notebook_iframe()Saving the reportIf you want to generate a HTML report file, save theProfileReportto an object and use theto_file()function:profile.to_file("your_report.html")Alternatively, you can obtain the data as JSON:# As a stringjson_data=profile.to_json()# As a fileprofile.to_file("your_report.json")Large datasetsVersion 2.4 introduces minimal mode.This is a default configuration that disables expensive computations (such as correlations and dynamic binning).Use the following syntax:profile=ProfileReport(large_dataset,minimal=True)profile.to_file("output.html")Command line usageFor standard formatted CSV files that can be read immediately by pandas, you can use thepandas_profilingexecutable.Run the following for information about options and arguments.mars_profiling-hAdvanced usageA set of options is available in order to adapt the report generated.title(str): Title for the report ('Pandas Profiling Report' by default).pool_size(int): Number of workers in thread pool. When set to zero, it is set to the number of CPUs available (0 by default).progress_bar(bool): If True,pandas-profilingwill display a progress bar.More settings can be found in thedefault configuration file,minimal configuration fileanddark themed configuration file.Exampleprofile=df.profile_report(title='Pandas Profiling Report',plot={'histogram':{'bins':8}})profile.to_file("output.html")Supporting open sourceMaintaining and developing the open-source code for pandas-profiling, with millions of downloads and thousands of users, would not be possible with support of our gracious sponsors.Lambda workstations, servers, laptops, and cloud services power engineers and researchers at Fortune 500 companies and 94% of the top 50 universities.Lambda Cloudoffers 4 & 8 GPU instances starting at $1.50 / hr. Pre-installed with TensorFlow, PyTorch, Ubuntu, CUDA, and cuDNN.We would like to thank our generous Github Sponsors supporters who make pandas-profiling possible:Martin Sotir, Joseph Yuen, Brian Lee, Stephanie Rivera, nscsekhar, abdulAzizMore info if you would like to appear here:Github Sponsor pageTypesTypes are a powerful abstraction for effective data analysis, that goes beyond the logical data types (integer, float etc.).pandas-profilingcurrently recognizes the following types:Boolean, Numerical, Date, Categorical, URL, Path, FileandImage.We have developed a type system for Python, tailored for data analysis:visions.
Selecting the right typeset drastically reduces the complexity the code of your analysis.
Future versions ofpandas-profilingwill have extended type support throughvisions!ContributingRead on getting involved in theContribution Guide.A low threshold place to ask questions or start contributing is by reaching out on the pandas-profiling Slack.Join the Slack community.Editor integrationPyCharm integrationInstallpandas-profilingvia the instructions aboveLocate yourpandas-profilingexecutable.On macOS / Linux / BSD:$whichmars_profiling(example)/usr/local/bin/mars_profilingOn Windows:$wherepandas_profiling(example)C:\ProgramData\Anaconda3\Scripts\pandas_profiling.exeIn PyCharm, go toSettings(orPreferenceson macOS) >Tools>External toolsClick the+icon to add a new external toolInsert the following valuesName: Pandas ProfilingProgram:The location obtained in step 2Arguments:"$FilePath$" "$FileDir$/$FileNameWithoutAllExtensions$_report.html"Working Directory:$ProjectFileDir$To use the PyCharm Integration, right click on any dataset file:External Tools>Pandas Profiling.Other integrationsOther editor integrations may be contributed via pull requests.DependenciesThe profile report is written in HTML and CSS, which meanspandas-profilingrequires a modern browser.You needPython 3to run this package. Other dependencies can be found in the requirements files:FilenameRequirementsrequirements.txtPackage requirementsrequirements-dev.txtRequirements for developmentrequirements-test.txtRequirements for testingsetup.pyRequirements for Widgets etc.