Package: pynock
pynockThe following describes a proposed standardNOCKfor a Parquet
format that supports efficient distributed serialization of multiple
kinds of graph technologies.This librarypynockprovides Examples for working with low-level
Parquet read/write efficiently in Python.Our intent is to serialize graphs in a way which aligns the data
representations required for popular graph technologies and related
data sources:semantic graphs (e.g., W3C formats RDF, TTL, JSON-LD, etc.)labeled property graphs (e.g., openCypher)probabilistic graphs (e.g., PSL)spreadsheet import/export (e.g., CSV)dataframes (e.g., Pandas, Dask, Spark, etc.)edge lists (e.g., NetworkX, cuGraph, etc.)This approach also efficient distributed partitions based on Parquet,
which can scale on a cluster to very large (+1 T node) graphs.For details about the proposed format in Parquet files, see theFORMAT.mdfile.If you have questions, suggestions, or bug reports, please openan issueon our public GitHub repo.CaveatsNote that thepynocklibrary does not provide any support for graph
computation or querying, merely for manipulating and validating
serialization formats.Our intent is to provide examples where others from the broader open
source developer community can help troubleshoot edge cases in
Parquet.DependenciesThis code has been tested and validated using Python 3.8, and we make
no guarantees regarding correct behaviors on other versions.The Parquet file formats depend on Arrow 5.0.x or later.For the Python dependencies, the library versioning info is listed in therequirements.txtfile.Set upTo install via PIP:python3 -m pip install -U pynockTo set up this library locally:python3 -m venv venv
source venv/bin/activate

python3 -m pip install -U pip wheel
python3 -m pip install -r requirements.txtUsage via CLITo run examples from CLI:python3 cli.py load-parq --file dat/recipes.parq --debugpython3 cli.py load-rdf --file dat/tiny.ttl --save-csv foo.csvFor further information:python3 cli.py --helpUsage programmatically in PythonTo construct a partition file programmatically, see theexamplesfor Jupyter notebooks with sample code and debugging.BackgroundFor more details about using Arrow and Parquet see:"Apache Arrow homepage""Finer-grained Reading and Writing""Apache Arrow: Read DataFrame With Zero Memory"Dejan SimicTowards Data Science(2020-06-25)Why the name?Anockis the English word for the end of an arrow opposite its point.If you must have an acronym, the proposed standardNOCKstands forNetworkObjects forConsistentKnowledge.Also, the library name had minimal namespace collisions on GitHub and
PyPi :)Developer updatesTo set up the build environment locally, also run:python3 -m pip install -U pip setuptools wheel
python3 -m pip install -r requirements-dev.txtNote that we require the use ofpre-commithooksand to configure that locally:pre-commit install
git config --local core.hooksPath .git/hooks/Package releasesFirst, verify thatsetup.pywill run correctly for the package
release process:python3 -m pip install -e .
python3 -m pytest -rx tests/
python3 -m pip uninstall pynockNext, update the semantic version number insetup.pyand create a
release on GitHub, and make sure to update the local repo:git stash
git checkout main
git pullMake sure that you have set up your 2FA authentication for generating
an API token on PyPi:https://pypi.org/manage/account/token/Then run our PyPi push script:./bin/push_pypi.shStar History