Package: schematicpy
SchematicTable of contentsIntroductionInstallationInstallation RequirementsInstallation guide for data curator appInstallation guide for developers/contributorsOther Contribution GuidelinesUpdate readthedocs documentationCommand Line UsageTestingUpdating Synapse test resourcesCode StyleContributorsIntroductionSCHEMATIC is an acronym forSchema Engine for Manifest Ingress and Curation. The Python based infrastructure provides anovelschema-based, metadata ingress ecosystem, that is meant to streamline the process of biomedical dataset annotation, metadata validation and submission to a data repository for various data contributors.InstallationInstallation RequirementsPython version 3.9.0≤x<3.11.0Note: You need to be a registered and certified user onsynapse.org, and also have the right permissions to download the Google credentials files from Synapse.Installation guide for data curator appCreate and activate a virtual environment within which you can install the package:python3 -m venv .venv
source .venv/bin/activateNote: Python 3 has a built-in support for virtual environmentvenvso you no longer need to install virtualenv.Install and update the package usingpip:python3 -m pip install schematicpyIf you run into error: Failed building wheel for numpy, the error might be able to resolve by upgrading pip. Please try to upgrade pip by:pip3 install --upgrade pipInstallation guide for developers/contributorsWhen contributing to this repository, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository before making a change.Please note we have acode of conduct, please follow it in all your interactions with the project.Development environment setupClone theschematicpackage repository.git clone https://github.com/Sage-Bionetworks/schematic.gitInstallpoetry(version 1.3.0 or later) using either theofficial installerorpipx. If you have an older installation of Poetry, we recommend uninstalling it first.Start the virtual environment by doing:poetry shellInstall the dependencies by doing:poetry installThis command will install the dependencies based on what we specify in poetry.lock. If this step is taking a long time, try to go back to step 2 and check your version of poetry. Alternatively, you could also try deleting the lock file and regenerate it by doingpoetry install(Please note this method should be used as a last resort because this would force other developers to change their development environment)If you want to install the API you will need to install those dependencies as well:poetry install --extras "api"If you want to install the uwsgi:poetry install --extras "api"Fill in credential files:Note: If you won't interact with Synapse, please ignore this section.There are two main configuration files that need to be edited:
config.yml
andsynapseConfigConfigure .synapseConfig FileDownload a copy of the.synapseConfigfile, open the file in the
editor of your choice and edit theusernameandauthtokenattribute under theauthenticationsectionNote: You could also visitconfigparserdoc to see the format that.synapseConfigmust have. For instance:[authentication]username = ABCauthtoken = abcConfigure config.yml FileThere are some defaults in schematic that can be configured. These fields are inconfig_example.yml:# This is an example config for Schematic.
# All listed values are those that are the default if a config is not used.
# Save this as config.yml, this will be gitignored.
# Remove any fields in the config you don't want to change
# Change the values of any fields you do want to change


# This describes where assets such as manifests are stored
asset_store:
  # This is when assets are stored in a synapse project
  synapse:
    # Synapse ID of the file view listing all project data assets.
    master_fileview_id: "syn23643253"
    # Path to the synapse config file, either absolute or relative to this file
    config: ".synapseConfig"
    # Base name that manifest files will be saved as
    manifest_basename: "synapse_storage_manifest"

# This describes information about manifests as it relates to generation and validation
manifest:
  # Location where manifests will saved to
  manifest_folder: "manifests"
  # Title or title prefix given to generated manifest(s)
  title: "example"
  # Data types of manifests to be generated or data type (singular) to validate manifest against
  data_type:
    - "Biospecimen"
    - "Patient"

# Describes the location of your schema
model:
  # Location of your schema jsonld, it must be a path relative to this file or absolute
  location: "tests/data/example.model.jsonld"

# This section is for using google sheets with Schematic
google_sheets:
  # The Synapse id of the Google service account credentials.
  service_acct_creds_synapse_id: "syn25171627"
  # Path to the synapse config file, either absolute or relative to this file
  service_acct_creds: "schematic_service_account_creds.json"
  # When doing google sheet validation (regex match) with the validation rules.
  #   true is alerting the user and not allowing entry of bad values.
  #   false is warning but allowing the entry on to the sheet.
  strict_validation: trueIf you want to change any of these copyconfig_example.ymltoconfig.yml, change any fields you want to, and remove any fields you don't.For example if you wanted to change the folder where manifests are downloaded your config should look like:manifest:
  manifest_folder: "my_manifest_folder_path"Note:config.ymlis ignored by git.Note: Paths can be specified relative to theconfig.ymlfile or as absolute paths.Login to Synapse by using the command line
On the CLI in your virtual environment, run the following command:synapse login -u <synapse username> -p <synapse password> --rememberMePlease make sure that you run the command before runningschematic initbelowObtain Google credential Files
To obtainschematic_service_account_creds.json, please run:schematic init --config ~/path/to/config.ymlAs v22.12.1 version of schematic, usingtokenmode of authentication (in other words, usingtoken.pickleandcredentials.json) is no longer supported due to Google's decision to move away from using OAuth out-of-band (OOB) flow. Clickhereto learn more.Notes: Use theschematic_service_account_creds.jsonfile for the service
account mode of authentication (for Google services/APIs). Service accounts
are special Google accounts that can be used by applications to access Google APIs
programmatically via OAuth2.0, with the advantage being that they do not require
human authorization.Background: schematic uses Google’s API to generate google sheet templates that users fill in to provide (meta)data.
Most Google sheet functionality could be authenticated with service account. However, more complex Google sheet functionality
requires token-based authentication. As browser support that requires the token-based authentication diminishes, we are hoping to deprecate
token-based authentication and keep only service account authentication in the future.Development process instructionFor new features, bugs, enhancementsPull the latest code fromdevelop branch in the upstream repoCheckout a new branch develop-<feature/fix-name> from the develop branchDo development on branch develop-<feature/fix-name>
a. may need to ensure that schematic poetry toml and lock files are compatible with your local environmentAdd changed files for tracking and commit changes usingbest practicesHave granular commits: not “too many” file changes, and not hundreds of code lines of changesCommits with work in progress are encouraged:
a. add WIP to the beginning of the commit message for “Work In Progress” commitsKeep commit messages descriptive but less than a page long, see best practicesPush code to develop-<feature/fix-name> in upstream repoBranch out off develop-<feature/fix-name> if needed to work on multiple features associated with the same code baseAfter feature work is complete and before creating a PR to the develop branch in upstream
a. ensure that code runs locally
b. test for logical correctness locally
c. wait for git workflow to complete (e.g. tests are run) on githubCreate a PR from develop-<feature/fix-name> into the develop branch of the upstream repoRequest a code review on the PROnce code is approved merge in the develop branchDelete the develop-<feature/fix-name> branchNote: Make sure you have the latest version of thedevelopbranch on your local machine.Installation Guide - DockerInstall docker fromhttps://www.docker.com/.Identify docker image of interest fromSchematic DockerHubExdocker pull sagebionetworks/schematic:latestfrom the CLI or, rundocker compose upafter cloning the schematic github repoin this case,sagebionetworks/schematic:latestis the name of the image chosenRun Schematic Command withdocker run <flags> <schematic command and args>.- For more information on flags fordocker runand what they do, visit theDocker Documentation- These example commands assume that you have navigated to the directory you want to run schematic from. To specify your working directory, use$(pwd)on MacOS/Linux or%cd%on Windows.- If not using the latest image, then the full name should be specified: iesagebionetworks/schematic:commit-e611e4a- If using local image created bydocker compose up, then the docker image name should be changed: i.e.schematic_schematic- Using the--nameflag sets the name of the container running locally on your machineExample For REST APIUse file path ofconfig.ymlto run API endpoints:docker run --rm -p 3001:3001 \
  -v $(pwd):/schematic -w /schematic --name schematic \
  -e SCHEMATIC_CONFIG=/schematic/config.yml \
  -e GE_HOME=/usr/src/app/great_expectations/ \
  sagebionetworks/schematic \
  python /usr/src/app/run_api.pyUse content ofconfig.ymlandschematic_service_account_creds.jsonas an environment variable to run API endpoints:save content ofconfig.ymlas to environment variableSCHEMATIC_CONFIG_CONTENTby doing:export SCHEMATIC_CONFIG_CONTENT=$(cat /path/to/config.yml)Similarly, save the content ofschematic_service_account_creds.jsonasSERVICE_ACCOUNT_CREDSby doing:export SERVICE_ACCOUNT_CREDS=$(cat /path/to/schematic_service_account_creds.json)PassSCHEMATIC_CONFIG_CONTENTandschematic_service_account_credsas environment variables by usingdocker rundocker run --rm -p 3001:3001 \
  -v $(pwd):/schematic -w /schematic --name schematic \
  -e GE_HOME=/usr/src/app/great_expectations/ \
  -e SCHEMATIC_CONFIG_CONTENT=$SCHEMATIC_CONFIG_CONTENT \
  -e SERVICE_ACCOUNT_CREDS=$SERVICE_ACCOUNT_CREDS \
  sagebionetworks/schematic \
  python /usr/src/app/run_api.pyExample For Schematic on mac/linuxTo run example below, first clone schematic into your home directorygit clone https://github.com/sage-bionetworks/schematic ~/schematicThen update .synapseConfig with your credentialsdocker run \
  -v ~/schematic:/schematic \
  -w /schematic \
  -e SCHEMATIC_CONFIG=/schematic/config.yml \
  -e GE_HOME=/usr/src/app/great_expectations/ \
  sagebionetworks/schematic schematic model \
  -c /schematic/config.yml validate \
  -mp /schematic/tests/data/mock_manifests/Valid_Test_Manifest.csv \
  -dt MockComponent \
  -js /schematic/tests/data/example.model.jsonldExample For Schematic on Windowsdocker run -v %cd%:/schematic \
  -w /schematic \
  -e GE_HOME=/usr/src/app/great_expectations/ \
  sagebionetworks/schematic \
  schematic model \
  -c config.yml validate -mp tests/data/mock_manifests/inValid_Test_Manifest.csv -dt MockComponent -js /schematic/data/example.model.jsonldOther Contribution GuidelinesUpdating readthedocs documentationcd docsAfter making relevant changes, you could run themake htmlcommand to re-generate thebuildfolder.Please contact the dev team to publish your updatesOther helpful resources:Getting started with SphinxInstalling SphinxUpdate toml file and lock fileIf you install external libraries by usingpoetry add <name of library>, please make sure that you includepyproject.tomlandpoetry.lockfile in your commit.Reporting bugs or feature requestsYou cancreate bug and feature requeststhroughSage Bionetwork's FAIR Data service desk. Providing enough details to the developers to verify and troubleshoot your issue is paramount:Provide a clear and descriptive title as well as a concise summaryof the issue to identify the problem.Describe the exact steps which reproduce the problemin as many details as possible.Describe the behavior you observed after following the stepsand point out what exactly is the problem with that behavior.Explain which behavior you expected to seeinstead and why.Provide screenshots of the expected or actual behaviourwhere applicable.Command Line UsagePlease visit more documentationhereTestingAll code added to the client must have tests. The Python client uses pytest to run tests. The test code is located in thetestssubdirectory.You can run the test suite in the following way:pytest -vs tests/Updating Synapse test resourcesDuplicate the entity being updated (or folder if applicable).Edit the duplicates (e.g.annotations, contents, name).Update the test suite in your branch to use these duplicates, including the expected values in the test assertions.Open a PR as per the usual process (see above).Once the PR is merged, leave the original copies on Synapse to maintain support for feature branches that were forked fromdevelopbefore your update.If the old copies are problematic and need to be removed immediately (e.g.contain sensitive data), proceed with the deletion and alert the other contributors that they need to merge the latestdevelopbranch into their feature branches for their tests to work.Code stylePlease consult theGoogle Python style guideprior to contributing code to this project.Be consistent and follow existing code conventions and spirit.ContributorsMain contributors and developers:Milen NikolovMialy DeFeliceSujay PatilBruno GrandeRobert AllawayGianna JordanLingling Peng