Package: dispatches-data-packages
DISPATCHES data packagesIntroductionWhat is it?A simple way to distribute and refer to data files that cannot be included directly in the DISPATCHES repository.GoalsProvide a reliable way to access the location of data files from DISPATCHES client code, regardless of the specifics of how DISPATCHES is installed (editable vs non-editable installation, local development vs CI, etc)Leverage as much as possible the built-in Python package distribution infrastructure to distribute collections of related non-code-files of small to moderate size (< 100 MB compressed)Allow using multiple repositories/package distributions to be used in a seamless way, so that the size limits only apply to each data package independentlyNon-goalsManage the packaged data in any way beyond the file-system leveli.e. the data package infrastructure only provide paths, which the client code uses to load the data in memory according to its specific needsManage and/or expose metadata beyond the name of the package and the Python package distribution used to installed itAutomatically enforce data distribution compliance requirements (LICENSE, COPYRIGHT, etc)ThisMUSTstill be done, but the process shall be manual rather than automaticRequirements and ConventionsDISPATCHES data packages SHALL be available on GitHub as repositories owned by thehttps://github.com/gmlc-dispatchesorganizationDISPATCHES data packages MAY be available on PyPIThe naming scheme SHOULD be consistent and follow this convention (usingmy-exampleas a placeholder):Repository URL:https://github.com/gmlc-dispatches/my-example-dataPython package distribution name:dispatches-my-example-dataThe repository SHOULD register itself by adding thedispatches-data-packagetopic so that all data packages repositories can be browsed at the URLhttps://github.com/topics/dispatches-data-packageThe repository MUST follow this directory structure:my-example-data/
`- .git/
`- pyproject.toml
`- src/
  `- dispatches_data/
    `- packages/
      `- my_example/
        `- __init__.py
        `- README.mdOnce installed, the data files SHALL be stored within the Python environment'ssite-packagesdirectory as.../lib/python3.8/site-packages/dispatches_data/packages/my_example, i.e. the data package directoryThe name of the data package directory (my_example) SHALL be used to refer to the data packageUsers should access the data package and its contents using the functions available in thedispatches_data.apimoduleThe Python package directory (i.e..../lib/python3.8/site-packages/dispatches_data/packages/my_example) MUST contain ALL information required for distribution of the dataThis includes, but is not limited to:LicenseCopyrightThe same information MAY berepeatedat the top level of the repository, but it MUST be in the package directoryThis is to ensure that all required information is always present when the data files are installed (which might not be the case if the information is stored at the top level of the repository)More than one data packages MAY be distributed together (i.e. as part of the same repository and/or Python package distribution)In this case, all of the above requirements apply to each data package individually (i.e. each separate data package directory MUST contain the appropriate required information)UsageStep 1Locate the data package(s) required by your application. In general, unless otherwise indicated, the naming conventions described above apply.Using the samemy_exampleplaceholder as above, the data package repository will be located athttps://github.com/gmlc-dispatches/my-example-data.gitStep 2Install the data package(s) required by your application, usingpip.pipinstallgit+https://github.com/gmlc-dispatches/my-example-data.gitStep 3Verify that the data packages where installed correctly, e.g.:pipshowdispatches-my-example-dataStep 4It should now be possible to access the data package from the client code, i.e. the DISPATCHES code that will load and use the data files, using the functions exposed in thedispatches_data.apimodule. These are simple functions that typically take the data package name (my_example) as astrargument.ExampleLet's assume we want to create a dataframe from a file namedmydata.csvin themy_exampledata package.In a Python file or Jupyter notebook:importpandasaspdfromdispatches_data.apiimportpathdefload_data()->pd.DataFrame:path_to_csv_file=path("my_example")/"mydata.csv"df=pd.read_csv(path_to_csv_file)# process df as neededreturndfdefmain():df=load_data()...# rest of the codeAPI ReferenceSee the documentation for thedispatches_data.apimodule onReadTheDocs.