Package: catmaid-publish
catmaid_publishFor the latest version, see here:https://github.com/clbarnes/catmaid_publish;
for docs see here:https://clbarnes.github.io/catmaid_publishScripts for publishing data from CATMAID.
Also useful for taking a snapshot of a particular set of data for further reproducible analysis (but be careful not to mix exported data with live data from the server).Originally created usingcookiecutterandclbarnes/python-template-sci.InstallationFirst, ensure you're working in a virtual environment:# create a virtual environment if you don't have onepython-mvenv--promptcatmaid_publishvenv# activate itsourcevenv/bin/activateThen install the package, using one of:# from githubpipinstallgit+https://github.com/clbarnes/catmaid_publish.git# a local copy of the repo, from within the parent directorypipinstall-e.Usagecatmaid_publishfetches data from a CATMAID instance based on a configuration file.If the instance requires authentication, credentials can be passed with environment variables or a separate TOML file.The workflow looks like this:# Empty config files will be created at these pathscatmaid_publish_initmy_config.toml--toml-credentialsmy_credentials.toml# Edit my_config.toml for your export needs# Edit my_credentials.toml with your login details (do not share or version control this file!)catmaid_publishmy_config.tomlmy_export/my_credentials.toml# optionally, compress export into a single zip file for transferzip-rmy_export.zipmy_exportcatmaid_publish_initUse this to initialise an empty config and optional credentials files.
If you have already set your credentials using environment variables starting withCATMAID_, the credentials files will be filled in.usage: catmaid_publish_init [-h] [--toml-credentials TOML_CREDENTIALS]
                            [--env-credentials ENV_CREDENTIALS] [--ignore-env]
                            [--no-http-basic] [--no-token]
                            config

Write an empty config file and, optionally, credentials files.

positional arguments:
  config                Path to write TOML config

options:
  -h, --help            show this help message and exit
  --toml-credentials TOML_CREDENTIALS, -t TOML_CREDENTIALS
                        Path to write TOML file for credentials. Will be
                        populated by CATMAID_* environment variables if set.
  --env-credentials ENV_CREDENTIALS, -e ENV_CREDENTIALS
                        Path to write env file for credentials. Will be
                        populated by CATMAID_* environment variables if set.
  --ignore-env, -i      Ignore CATMAID_* environment variables when writing
                        credential files.
  --no-http-basic, -H   Omit HTTP basic auth from credentials file.
  --no-token, -T        Omit CATMAID API token from credentials file.If you would prefer to write the config and credentials files yourself, see theexamples here.ConfigurationFill in the config file usingTOML formatting.Citation information will be included with the export.
Project information will be used to connect to CATMAID (but sensitive credentials should be stored elsewhere).For other data types,all = truemeans export all data of that type.
Note that this can take a very long time for common data types (e.g. neurons) in large projects.
Ifall = false, you can list the names of specific objects to be exported.
You can also rename specific objects by mapping the old name to the new one (objects to be renamed will be added to the list of objects to export).Some objects can be annotated.
In this case, you can instead list annotations for which annotated objects will be exported.
Indirectly annotated ("sub-annotated") objects, e.g. the relationship between A and C inannotation "A" -> annotation "B" -> neuron "C"will also be exported.All exported data have a pre-writtenREADME.mdfile detailing the data format and structure.
You can add additional information to the README using thereadme_footerkey.
This string will have leading and trailing whitespace stripped, and, if still non-empty, will be appended to the default README below a thematic break.AuthenticationIf your CATMAID instance requires authentication (with a CATMAID account and/or HTTP Basic authentication), fill in these details in a separate TOML file, or as environment variables (which can be loaded from a shell script file).Passwords, API tokens etc.MUST NOTbe tracked with git.A credentials file simply looks like this:# If your instance requires login to browseapi_token="y0urc47ma1d70k3n"# If your instance uses HTTP Basic authentication to accesshttp_user="myuser"http_password="mypassword"Or use environment variablesCATMAID_API_TOKEN,CATMAID_HTTP_USER, andCATMAID_HTTP_PASSWORD.catmaid_publishOnce you have filled in the config file, use thecatmaid_publishcommand to fetch and write the data, e.g.# leave out the credentials path if you are using environment variablescatmaid_publishpath/to/config.tomlpath/to/output_dirpath/to/credentials.tomlFull usage details are here:usage: catmaid_publish [-h] config out [credentials]

Export data from CATMAID in plaintext formats with simple configuration.

positional arguments:
  config       Path to TOML config file.
  out          Path to output directory. Must not exist.
  credentials  Optional path to TOML file containing CATMAID credentials
               (http_user, http_password, api_token as necessary).
               Alternatively, use environment variables with the same names
               upper-cased and prefixed with CATMAID_.

options:
  -h, --help   show this help message and exitOutputREADME files in the output directory hierarchy describe the formats of the included data.
All data are sorted deterministically and in plain text, and are highly compressible.ReadingAs detailed in the top-level README of the exported data, this package contains a utility for reading an export into common python data structures for neuronal analysis.For example:fromcatmaid_publishimportDataReader,ReadSpec,Locationimportnetworkxasnximportnavisreader=DataReader("path/to/exported/data")annotation_graph:nx.DiGraph=reader.annotations.get_graph()neuron:navis.TreeNeuron=reader.neurons.get_by_name("my neuron",ReadSpec(nodes=True,connectors=False,tags=True),)landmark_locations:list[Location]=list(reader.landmarks.get_all())volume:navis.Volume=reader.volumes.get_by_name("my volume")TipsIn general, it's most robust to use the CATMAID UI to make an annotation specifically for your export; ideally namespaced and timestamped (e.g.cbarnes_export_2023-02-15).
Later exports can be a superset of this one.PublicationConsider running the export once to find which objects are exported,
and determine whether any objects need renaming.
Then update your configuration with these renames.Analysis snapshotIn large CATMAID projects, there are relatively few landmarks, volumes, and annotations compared to neurons.
As these are all helpful for mining data, consider exporting withall = truefor everything except neurons.
Use the CATMAID UI (e.g. connectivity widget, graph widget, volume intersection) to annotate a superset of your neurons of interest for the export.It is easier to bounce between local analysis and use of the CATMAID UI if you do not rename any objects in this case.ContainerisationThis project can be containerised withapptainer(formerly called Singularity)
(bundling it with a python environment and full OS) on linux,
so that it can be run on any system with apptainer installed.Just runmake container(requires sudo).The python files are installed in the container at/project.Depending on where your config and credentials files are stored, they may beaccessible to the container by default.
Otherwise, you can manuallybind mountthe containing directories inside the container at runtime:# Find the data path your environment is using, defaulting to the local ./dataDATA_PATH="$(pwd)/data"CREDS_PATH="$(pwd)/credentials"# Execute the command `/bin/bash` (i.e. get a terminal inside the container),# mounting the data directory and credentials you're already using.# Container file (.sif) must already be builtapptainerexec\--bind"$DATA_PATH:/data"\--bind"$CREDS_PATH:/credentials"\catmaid_publish.sif/bin/bash# Now you're inside the container...>>>catmaid_publish_init/data/config.toml-t/credentials/my_instance.toml
>>>catmaid_publish/data/config.toml/data/my_export/credentials/my_instance.toml