Package: pycodehash
PyCodeHashData pipelines are of paramount importance in data science, engineering and analysis.
Often, there are parts of the pipeline that have not changed.
Recomputing these nodes is wasteful, especially for larger datasets.
PyCodeHash is a generic data and code hashing library that facilitates downstream caching.Read more on thedocumentation site.Detecting changes in data pipelinesThe canonical way to check if two things are equal is to compare their hashes.
Learn more on how PyCodeHash detects changes in:Python FunctionsSQL QueriesDatasets: Files, Directories, S3, HivePython dependenciesInstallationPyCodeHash is available fromPyPI:pipinstallpycodehashExamplesPythonUse the FunctionHasher to obtain the hash of a Python function object:frompycodehashimportFunctionHasherfromtlibaimportcompute_momentsfromtliba.etlimportadd_bernoulli_samples,combine_random_samplesfh=FunctionHasher()# Hash the function `add_bernoulli_samples`h1=fh.hash_func(add_bernoulli_samples)print("Hash for `add_bernoulli_samples`",h1)# Hash the function `compute_moments`h2=fh.hash_func(compute_moments)print("Hash for `compute_moments`",h2)# Hash the function `combine_random_samples`h3=fh.hash_func(combine_random_samples)print("Hash for `combine_random_samples`",h3)Python Usage ExampleSQLHash SQL queries and files using theSQLHasher(requirespip install pycodehash[sql]):frompathlibimportPathfrompycodehash.sql.sql_hasherimportSQLHasher# First queryquery_1="SELECT * FROM db.templates"# The second query is equivalent, but has additional whitespacequery_2="SELECT\n*\nFROM\ndb.templates"# Write the second query to a filequery_2_file=Path("/tmp/query.sql")query_2_file.write_text(query_2)# Create the SQLHasher object for SparkSQLhasher=SQLHasher(dialect="sparksql")# We can hash a stringprint(hasher.hash_query(query_1))# Or pass a pathprint(hasher.hash_file(query_2_file))SQL Usage ExampleDatasetsHash data, such as files, directories, database tables:frompathlibimportPathfrompycodehash.datasetsimportLocalFileHash,LocalDirectoryHash# Hash a single filefh=LocalFileHash()print(fh.collect_metadata("example.py"))# {'last_modified': datetime.datetime(2023, 11, 24, 23, 38, 17, 524024), 'size': 543}print(fh.compute_hash("example.py"))# 6189721d3ecdf86503a82c07eed82743069ebbf39e974f33ca684809e67e9e0e# Hash a directorydh=LocalDirectoryHash()# Recursively hash all files in a directoryprint(len(dh.collect_partitions(Path(__file__).parent/"src")))# 29Dataset Usage ExamplePython Package DependenciesHash a user-provided list of Python packages your code depends on. This may be a selection of the total list of dependencies.
For example the most important libraries your code depends on, that you want to track in order to trigger a rerun of your pipeline in case of version changes.
The hasher retrieves the installed package versions and creates a hash of those. We emphasize it is up to the user to provide the list of relevant dependencies.frompycodehash.dependencyimportPythonDependencyHash# hash a list of dependencieshasher=PythonDependencyHash()print(hasher.collect_metadata(dependencies=["pycodehash","rope"],add_python_version=True))# hasher retrieves the installed package versions found# {'pycodehash': '0.2.0', 'rope': '1.11.0', 'Python': '3.11'}print(hasher.compute_hash(dependencies=["pycodehash","rope"],add_python_version=True))# cecb8036ad61235c2577db9943f519b824f7a25e449da9cd332bc600fb5dccf0Dependency Usage ExampleLicensePyCodeHash is completely free, open-source and licensed under theMIT license.