Package: tensorflow-data-validation
TensorFlow Data ValidationTensorFlow Data Validation(TFDV) is a library for exploring and validating
machine learning data. It is designed to be highly scalable
and to work well with TensorFlow andTensorFlow Extended (TFX).TF Data Validation includes:Scalable calculation of summary statistics of training and test data.Integration with a viewer for data distributions and statistics, as well
as faceted comparison of pairs of features (Facets)Automateddata-schemageneration to describe expectations about data
like required values, ranges, and vocabulariesA schema viewer to help you inspect the schema.Anomaly detection to identifyanomalies,
such as missing features,
out-of-range values, or wrong feature types, to name a few.An anomalies viewer so that you can see what features have anomalies and
learn more in order to correct them.For instructions on using TFDV, see theget started guideand try out theexample notebook.
Some of the techniques implemented in TFDV are described in atechnical paper published in SysML'19.Installing from PyPIThe recommended way to install TFDV is using thePyPI package:pipinstalltensorflow-data-validationNightly PackagesTFDV also hosts nightly packages on Google Cloud. To install the latest nightly
package, please use the following command:exportTFX_DEPENDENCY_SELECTOR=NIGHTLY
pipinstall--extra-index-urlhttps://pypi-nightly.tensorflow.org/simpletensorflow-data-validationThis will install the nightly packages for the major dependencies of TFDV such
as TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).Sometimes TFDV uses those dependencies' most recent changes, which are not yet
released. Because of this, it is safer to use nightly versions of those
dependent libraries when using nightly TFDV. Export theTFX_DEPENDENCY_SELECTORenvironment variable to do so.NOTE: These nightly packages are unstable and breakages are likely to happen.
The fix could often take a week or more depending on the complexity involved.Build with DockerThis is the recommended way to build TFDV under Linux, and is continuously
tested at Google.1. Install DockerPlease first installdockeranddocker-composeby following the directions:docker;docker-compose.2. Clone the TFDV repositorygitclonehttps://github.com/tensorflow/data-validationcddata-validationNote that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass-b <branchname>to thegit clonecommand.3. Build the pip packageThen, run the following at the project root:sudodocker-composebuildmanylinux2010
sudodocker-composerun-ePYTHON_VERSION=${PYTHON_VERSION}manylinux2010wherePYTHON_VERSIONis one of{38, 39, 310}.A wheel will be produced underdist/.4. Install the pip packagepipinstalldist/*.whlBuild from source1. PrerequisitesTo compile and use TFDV, you need to set up some prerequisites.Install NumPyIf NumPy is not installed on your system, install it now by followingthese
directions.Install BazelIf Bazel is not installed on your system, install it now by followingthese
directions.2. Clone the TFDV repositorygitclonehttps://github.com/tensorflow/data-validationcddata-validationNote that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass-b <branchname>to thegit clonecommand.3. Build the pip packageTFDVwheel is Python version dependent -- to build the pip package that
works for a specific Python version, use that Python binary to run:pythonsetup.pybdist_wheelYou can find the generated.whlfile in thedistsubdirectory.4. Install the pip packagepipinstalldist/*.whlSupported platformsTFDV is tested on the following 64-bit operating systems:macOS 12.5 (Monterey) or later.Ubuntu 20.04 or later.Windows 7 or later.Notable DependenciesTensorFlow is required.Apache Beamis required; it's the way that efficient
distributed computation is supported. By default, Apache Beam runs in local
mode but can also run in distributed mode usingGoogle Cloud Dataflowand other Apache
Beamrunners.Apache Arrowis also required. TFDV uses Arrow to
represent data internally in order to make use of vectorized numpy functions.Compatible versionsThe following table shows the  package versions that are
compatible with each other. This is determined by our testing framework, but
otheruntestedcombinations may also work.tensorflow-data-validationapache-beam[gcp]pyarrowtensorflowtensorflow-metadatatensorflow-transformtfx-bslGitHub master2.47.010.0.0nightly (2.x)1.14.0n/a1.14.01.14.02.47.010.0.02.131.14.0n/a1.14.01.13.02.40.06.0.02.121.13.1n/a1.13.01.12.02.40.06.0.02.111.12.0n/a1.12.01.11.02.40.06.0.01.15 / 2.101.11.0n/a1.11.01.10.02.40.06.0.01.15 / 2.91.10.0n/a1.10.11.9.02.38.05.0.01.15 / 2.91.9.0n/a1.9.01.8.02.38.05.0.01.15 / 2.81.8.0n/a1.8.01.7.02.36.05.0.01.15 / 2.81.7.0n/a1.7.01.6.02.35.05.0.01.15 / 2.71.6.0n/a1.6.01.5.02.34.05.0.01.15 / 2.71.5.0n/a1.5.01.4.02.32.04.0.11.15 / 2.61.4.0n/a1.4.01.3.02.32.02.0.01.15 / 2.61.2.0n/a1.3.01.2.02.31.02.0.01.15 / 2.51.2.0n/a1.2.01.1.12.29.02.0.01.15 / 2.51.1.0n/a1.1.11.1.02.29.02.0.01.15 / 2.51.1.0n/a1.1.01.0.02.29.02.0.01.15 / 2.51.0.0n/a1.0.00.30.02.28.02.0.01.15 / 2.40.30.0n/a0.30.00.29.02.28.02.0.01.15 / 2.40.29.0n/a0.29.00.28.02.28.02.0.01.15 / 2.40.28.0n/a0.28.10.27.02.27.02.0.01.15 / 2.40.27.0n/a0.27.00.26.12.28.00.17.01.15 / 2.30.26.00.26.00.26.00.26.02.25.00.17.01.15 / 2.30.26.00.26.00.26.00.25.02.25.00.17.01.15 / 2.30.25.00.25.00.25.00.24.12.24.00.17.01.15 / 2.30.24.00.24.10.24.10.24.02.23.00.17.01.15 / 2.30.24.00.24.00.24.00.23.12.24.00.17.01.15 / 2.30.23.00.23.00.23.00.23.02.23.00.17.01.15 / 2.30.23.00.23.00.23.00.22.22.20.00.16.01.15 / 2.20.22.00.22.00.22.10.22.12.20.00.16.01.15 / 2.20.22.00.22.00.22.10.22.02.20.00.16.01.15 / 2.20.22.00.22.00.22.00.21.52.17.00.15.01.15 / 2.10.21.00.21.10.21.30.21.42.17.00.15.01.15 / 2.10.21.00.21.10.21.30.21.22.17.00.15.01.15 / 2.10.21.00.21.00.21.00.21.12.17.00.15.01.15 / 2.10.21.00.21.00.21.00.21.02.17.00.15.01.15 / 2.10.21.00.21.00.21.00.15.02.16.00.14.01.15 / 2.00.15.00.15.00.15.00.14.12.14.00.14.01.140.14.00.14.0n/a0.14.02.14.00.14.01.140.14.00.14.0n/a0.13.12.11.0n/a1.130.12.10.13.0n/a0.13.02.11.0n/a1.130.12.10.13.0n/a0.12.02.10.0n/a1.120.12.10.12.0n/a0.11.02.8.0n/a1.110.9.00.11.0n/a0.9.02.6.0n/a1.9n/an/an/aQuestionsPlease direct any questions about working with TF Data Validation toStack Overflowusing thetensorflow-data-validationtag.LinksTensorFlow Data Validation Getting Started GuideTensorFlow Data Validation NotebookTensorFlow Data Validation API DocumentationTensorFlow Data Validation Blog PostTensorFlow Data Validation PyPITensorFlow Data Validation PaperTensorFlow Data Validation Slides