Package: ydata-profiling
ydata-profilingDocumentation|Discord|Stack Overflow|Latest changelogDo you like this project? Show us your love andgive feedback!ydata-profilingprimary goal is to provide a one-line Exploratory Data Analysis (EDA) experience in a consistent and fast solution. Like pandasdf.describe()function, that is so handy, ydata-profiling delivers an extended analysis of a DataFrame while allowing the data analysis to be exported in different formats such ashtmlandjson.The package outputs a simple and digested analysis of a dataset, includingtime-seriesandtext.Looking for a scalable solution that can fully integrate with your database systems?Leverage YData Fabric Data Catalog to connect to different databases and storages (Oracle, snowflake, PostGreSQL, GCS, S3, etc.) and leverage an interactive and guided profiling experience in Fabric. Check out theCommunity Version.▶️ QuickstartInstallpip install ydata-profilingorconda install -c conda-forge ydata-profilingStart profilingStart by loading your pandasDataFrameas you normally would, e.g. by using:importnumpyasnpimportpandasaspdfromydata_profilingimportProfileReportdf=pd.DataFrame(np.random.rand(100,5),columns=["a","b","c","d","e"])To generate the standard profiling report, merely run:profile=ProfileReport(df,title="Profiling Report")📊 Key featuresType inference: automatic detection of columns' data types (Categorical,Numerical,Date, etc.)Warnings: A summary of the problems/challenges in the data that you might need to work on (missing data,inaccuracies,skewness, etc.)Univariate analysis: including descriptive statistics (mean, median, mode, etc) and informative visualizations such as distribution histogramsMultivariate analysis: including correlations, a detailed analysis of missing data, duplicate rows, and visual support for variables pairwise interactionTime-Series: including different statistical information relative to time dependent data such as auto-correlation and seasonality, along ACF and PACF plots.Text analysis: most common categories (uppercase, lowercase, separator), scripts (Latin, Cyrillic) and blocks (ASCII, Cyrilic)File and Image analysis: file sizes, creation dates, dimensions, indication of truncated images and existence of EXIF metadataCompare datasets: one-line solution to enable a fast and complete report on the comparison of datasetsFlexible output formats: all analysis can be exported to an HTML report that can be easily shared with different parties, as JSON for an easy integration in automated systems and as a widget in a Jupyter Notebook.The report contains three additional sections:Overview: mostly global details about the dataset (number of records, number of variables, overall missigness and duplicates, memory footprint)Alerts: a comprehensive and automatic list of potential data quality issues (high correlation, skewness, uniformity, zeros, missing values, constant values, between others)Reproduction: technical details about the analysis (time, version and configuration)🎁 Latest featuresWant to scale? Check the latest release with ⭐⚡Spark support!Looking for how you can do an EDA for Time-Series 🕛 ? Checkthis blogpost.You want to compare 2 datasets and get a report? Checkthis blogpost✨ SparkSpark support has been released, but we are always looking for an extra pair of hands 👐.Check current work in progress!.📝 Use casesYData-profiling can be used to deliver a variety of different use-case. The documentation includes guides, tips and tricks for tackling them:Use caseDescriptionComparing datasetsComparing multiple version of the same datasetProfiling a Time-Series datasetGenerating a report for a time-series dataset with a single line of codeProfiling large datasetsTips on how to prepare data and configureydata-profilingfor working with large datasetsHandling sensitive dataGenerating reports which are mindful about sensitive data in the input datasetDataset metadata and data dictionariesComplementing the report with dataset details and column-specific data dictionariesCustomizing the report's appearanceChanging the appearance of the report's page and of the contained visualizationsProfiling DatabasesFor a seamless profiling experience in your organization's databases, checkFabric Data Catalog, which allows to consume data from different types of storages such as RDBMs (Azure SQL, PostGreSQL, Oracle, etc.) and object storages (Google Cloud Storage, AWS S3, Snowflake, etc.), among others.Using inside Jupyter NotebooksThere are two interfaces to consume the report inside a Jupyter notebook: through widgets and through an embedded HTML report.The above is achieved by simply displaying the report as a set of widgets. In a Jupyter Notebook, run:profile.to_widgets()The HTML report can be directly embedded in a cell in a similar fashion:profile.to_notebook_iframe()Exporting the report to a fileTo generate a HTML report file, save theProfileReportto an object and use theto_file()function:profile.to_file("your_report.html")Alternatively, the report's data can be obtained as a JSON file:# As a JSON stringjson_data=profile.to_json()# As a fileprofile.to_file("your_report.json")Using in the command lineFor standard formatted CSV files (which can be read directly by pandas without additional settings), theydata_profilingexecutable can be used in the command line. The example below generates a report namedExample Profiling Report, using a configuration file calleddefault.yaml, in the filereport.htmlby processing adata.csvdataset.ydata_profiling--title"Example Profiling Report"--config_filedefault.yamldata.csvreport.htmlAdditional details on the CLI are availableon the documentation.👀 ExamplesThe following example reports showcase the potentialities of the package across a wide range of dataset and data types:Census Income(US Adult Census data relating income with other demographic properties)NASA Meteorites(comprehensive set of meteorite landing - object properties and locations)Titanic(the "Wonderwall" of datasets)NZA(open data from the Dutch Healthcare Authority)Stata Auto(1978 Automobile data)Colors(a simple colors dataset)Vektis(Vektis Dutch Healthcare data)UCI Bank Dataset(marketing dataset from a bank)Russian Vocabulary(100 most common Russian words, showcasing unicode text analysis)Website Inaccessibility(website accessibility analysis, showcasing support for URL data)Orange pricesandCoal prices(simple pricing evolution datasets, showcasing the theming options)USA Air Quality(Time-series air quality dataset EDA example)HCC(Open dataset from healthcare, showcasing compare between two sets of data, before and after preprocessing)🛠️ InstallationAdditional details, including information about widget support, are availableon the documentation.Using pipYou can install using thepippackage manager by running:pipinstall-Uydata-profilingExtrasThe package declares "extras", sets of additional dependencies.[notebook]: support for rendering the report in Jupyter notebook widgets.[unicode]: support for more detailed Unicode analysis, at the expense of additional disk space.[pyspark]: support for pyspark for big dataset analysisInstall these with e.g.pipinstall-Uydata-profiling[notebook,unicode,pyspark]Using condaYou can install using thecondapackage manager by running:condainstall-cconda-forgeydata-profilingFrom source (development)Download the source code by cloning the repository or click onDownload ZIPto download the latest stable version.Install it by navigating to the proper directory and running:pipinstall-e.The profiling report is written in HTML and CSS, which means a modern browser is required.You needPython 3to run the package. Other dependencies can be found in the requirements files:FilenameRequirementsrequirements.txtPackage requirementsrequirements-dev.txtRequirements for developmentrequirements-test.txtRequirements for testingsetup.pyRequirements for widgets etc.🔗 IntegrationsTo maximize its usefulness in real world contexts,ydata-profilinghas a set of implicit and explicit integrations with a variety of other actors in the Data Science ecosystem:Integration typeDescriptionOther DataFrame librariesHow to compute the profiling of data stored in libraries other than pandasGreat ExpectationsGeneratingGreat Expectationsexpectations suites directly from a profiling reportInteractive applicationsEmbedding profiling reports inStreamlit,DashorPanelapplicationsPipelinesIntegration with DAG workflow execution tools likeAirfloworKedroCloud servicesUsingydata-profilingin hosted computation services likeLambda,Google CloudorKaggleIDEsUsingydata-profilingdirectly from integrated development environments such asPyCharm🙋 SupportNeed help? Want to share a perspective? Report a bug? Ideas for collaborations? Reach out via the following channels:Stack Overflow: ideal for asking questions on how to use the packageGitHub Issues: bugs, proposals for changes, feature requestsDiscord: ideal for projects discussions, ask questions, collaborations, general chatNeed Help?Get your questions answered with a product owner bybooking a Pawsome chat! 🐼❗ Before reporting an issue on GitHub, check outCommon Issues.🤝🏽 ContributingLearn how to get involved in theContribution Guide.A low-threshold place to ask questions or start contributing is theData Centric AI Community's Discord.A big thank you to all our amazing contributors!Contributors wall made withcontrib.rocks.