Package: ydata-profiling-thorrester
ydata-profilingDocumentation|Discord|Stack Overflow|Latest changelogDo you like this project? Show us your love andgive feedback!ydata-profilingprimary goal is to provide a one-line Exploratory Data Analysis (EDA) experience in a consistent and fast solution. Like pandasdf.describe()function, that is so handy, ydata-profiling delivers an extended analysis of a DataFrame while allowing the data analysis to be exported in different formats such ashtmlandjson.The package outputs a simple and digested analysis of a dataset, includingtime-seriesandtext.▶️ QuickstartInstallpip install ydata-profilingorconda install -c conda-forge ydata-profilingStart profilingStart by loading your pandasDataFrameas you normally would, e.g. by using:importnumpyasnpimportpandasaspdfromydata_profilingimportProfileReportdf=pd.DataFrame(np.random.rand(100,5),columns=["a","b","c","d","e"])To generate the standard profiling report, merely run:profile=ProfileReport(df,title="Profiling Report")Key featuresType inference: automatic detection of columns' data types (Categorical,Numerical,Date, etc.)Warnings: A summary of the problems/challenges in the data that you might need to work on (missing data,inaccuracies,skewness, etc.)Univariate analysis: including descriptive statistics (mean, median, mode, etc) and informative visualizations such as distribution histogramsMultivariate analysis: including correlations, a detailed analysis of missing data, duplicate rows, and visual support for variables pairwise interactionTime-Series: including different statistical information relative to time dependent data such as auto-correlation and seasonality, along ACF and PACF plots.Text analysis: most common categories (uppercase, lowercase, separator), scripts (Latin, Cyrillic) and blocks (ASCII, Cyrilic)File and Image analysis: file sizes, creation dates, dimensions, indication of truncated images and existence of EXIF metadataCompare datasets: one-line solution to enable a fast and complete report on the comparison of datasetsFlexible output formats: all analysis can be exported to an HTML report that can be easily shared with different parties, as JSON for an easy integration in automated systems and as a widget in a Jupyter Notebook.The report contains three additional sections:Overview: mostly global details about the dataset (number of records, number of variables, overall missigness and duplicates, memory footprint)Alerts: a comprehensive and automatic list of potential data quality issues (high correlation, skewness, uniformity, zeros, missing values, constant values, between others)Reproduction: technical details about the analysis (time, version and configuration)🎁 Latest featuresWant to scale? Check the latest release with ⭐⚡Spark support!Looking for how you can do an EDA for Time-Series 🕛 ? Checkthis blogpost.You want to compare 2 datasets and get a report? Checkthis blogpost⚡ SparkSpark support has been released, but we are always looking for an extra pair of hands 👐.Check current work in progress!.📝 Use casesYData-profiling can be used to deliver a variety of different use-case. The documentation includes guides, tips and tricks for tackling them:Use caseDescriptionComparing datasetsComparing multiple version of the same datasetProfiling a Time-Series datasetGenerating a report for a time-series dataset with a single line of codeProfiling large datasetsTips on how to prepare data and configureydata-profilingfor working with large datasetsHandling sensitive dataGenerating reports which are mindful about sensitive data in the input datasetDataset metadata and data dictionariesComplementing the report with dataset details and column-specific data dictionariesCustomizing the report's appearanceChanging the appearance of the report's page and of the contained visualizationsUsing inside Jupyter NotebooksThere are two interfaces to consume the report inside a Jupyter notebook: through widgets and through an embedded HTML report.The above is achieved by simply displaying the report as a set of widgets. In a Jupyter Notebook, run:profile.to_widgets()The HTML report can be directly embedded in a cell in a similar fashion:profile.to_notebook_iframe()Exporting the report to a fileTo generate a HTML report file, save theProfileReportto an object and use theto_file()function:profile.to_file("your_report.html")Alternatively, the report's data can be obtained as a JSON file:# As a JSON stringjson_data=profile.to_json()# As a fileprofile.to_file("your_report.json")Using in the command lineFor standard formatted CSV files (which can be read directly by pandas without additional settings), thepandas_profilingexecutable can be used in the command line. The example below generates a report namedExample Profiling Report, using a configuration file calleddefault.yaml, in the filereport.htmlby processing adata.csvdataset.pandas_profiling--title"Example Profiling Report"--config_filedefault.yamldata.csvreport.htmlAdditional details on the CLI are availableon the documentation.👀 ExamplesThe following example reports showcase the potentialities of the package across a wide range of dataset and data types:Census Income(US Adult Census data relating income with other demographic properties)NASA Meteorites(comprehensive set of meteorite landing - object properties and locations)Titanic(the "Wonderwall" of datasets)NZA(open data from the Dutch Healthcare Authority)Stata Auto(1978 Automobile data)Colors(a simple colors dataset)Vektis(Vektis Dutch Healthcare data)UCI Bank Dataset(marketing dataset from a bank)Russian Vocabulary(100 most common Russian words, showcasing unicode text analysis)Website Inaccessibility(website accessibility analysis, showcasing support for URL data)Orange pricesandCoal prices(simple pricing evolution datasets, showcasing the theming options)USA Air Quality(Time-series air quality dataset EDA example)HCC(Open dataset from healthcare, showcasing compare between two sets of data, before and after preprocessing)🛠️ InstallationAdditional details, including information about widget support, are availableon the documentation.Using pipYou can install using thepippackage manager by running:pipinstall-Uydata-profilingExtrasThe package declares "extras", sets of additional dependencies.[notebook]: support for rendering the report in Jupyter notebook widgets.[unicode]: support for more detailed Unicode analysis, at the expense of additional disk space.[pyspark]: support for pyspark for big dataset analysisInstall these with e.g.pipinstall-Uydata-profiling[notebook,unicode,pyspark]Using condaYou can install using thecondapackage manager by running:condainstall-cconda-forgeydata-profilingFrom source (development)Download the source code by cloning the repository or click onDownload ZIPto download the latest stable version.Install it by navigating to the proper directory and running:pipinstall-e.The profiling report is written in HTML and CSS, which means a modern browser is required.You needPython 3to run the package. Other dependencies can be found in the requirements files:FilenameRequirementsrequirements.txtPackage requirementsrequirements-dev.txtRequirements for developmentrequirements-test.txtRequirements for testingsetup.pyRequirements for widgets etc.🔗 IntegrationsTo maximize its usefulness in real world contexts,pandas-profilinghas a set of implicit and explicit integrations with a variety of other actors in the Data Science ecosystem:Integration typeDescriptionOther DataFrame librariesHow to compute the profiling of data stored in libraries other than pandasGreat ExpectationsGeneratingGreat Expectationsexpectations suites directly from a profiling reportInteractive applicationsEmbedding profiling reports inStreamlit,DashorPanelapplicationsPipelinesIntegration with DAG workflow execution tools likeAirfloworKedroCloud servicesUsingpandas-profilingin hosted computation services likeLambda,Google CloudorKaggleIDEsUsingpandas-profilingdirectly from integrated development environments such asPyCharm🙋 SupportNeed help? Want to share a perspective? Report a bug? Ideas for collaborations? Reach out via the following channels:Stack Overflow: ideal for asking questions on how to use the packageGitHub Issues: bugs, proposals for changes, feature requestsDiscord: ideal for projects discussions, ask questions, collaborations, general chatEmail: project collaborations or sponsoring❗ Before reporting an issue on GitHub, check outCommon Issues.🤝🏽 ContributingLearn how to get involved in theContribution Guide.A low-threshold place to ask questions or start contributing is theData Centric AI Community's Discord.A big thank you to all our amazing contributors!Contributors wall made withcontrib.rocks.