Package: nci-cidc-api-modules
NCI CIDC APIThe next generation of the CIDC API, reworked to use Google Cloud-managed services. This API is built with the Flask REST API framework backed by Google Cloud SQL, running on Google App Engine.DevelopmentInstall Python dependenciesDatabase ManagementSetting up a local development databaseConnecting to a Cloud SQL database instanceRunning database migrationsServing LocallyTestingCode FormattingDeploymentCI/CDDeploying by handConnecting to the APIProvisioning the system from scratchDocker ComposeInstall Python dependenciesPython versions tested include 3.9 and 3.10. The current App Engine is using version 3.9 (seeapp.prod.yaml). You can usehttps://github.com/pyenv/pyenvto manage your python versions. Homebrew will also work, but you will have to be specific when you install packages with pip outside of virtual environments. On that note, it is recommended that you install your python dependencies in an isolated environment. For example,# make a virtual environment in the current direcory called "venv"python3-mvenvvenvsourcevenv/bin/activate# optionally add an alias to your shell rc file# alias activate='source venv/bin/activate'Install both the production and development dependencies.pipinstall-rrequirements.dev.txtInstall and configure pre-commit hooks for code formatting and commit message standardization.pre-commitinstallDatabase ManagementSetting up a local development databaseIn production, the CIDC API connects to a PostgreSQL instance hosted by Google Cloud SQL, but for local development, you should generally use a local PostgreSQL instance.To do so, first install and start PostgreSQL:brewinstallpostgresql@11
brewservicesstartpostgresql@11# launches the postgres service whenever your computer launchesThe cidc-devops repo creates a Cloud SQL instance using postgresql@9.6 which is disabled through homebrew. The earliest non-deprecated version through homebrew is postgresql@11.Troubleshooting Homebrew PostgreSQL InstallationIf you already have a different version of postgresql installed via homebrew, see this gist for upgrading helphttps://gist.github.com/olivierlacan/e1bf5c34bc9f82e06bc0. You could also trybrew postgresql-upgrade-database.If you see issues regarding "postmaster.pid already exists", see this posthttps://stackoverflow.com/questions/36436120/fatal-error-lock-file-postmaster-pid-already-exists.You may see thatbrew services start postgresql@11succeeds, but when you list the servicesbrew servicesyou see an error for postgresql.$brewservices
NameStatusUserFile
postgresql@11errorcrouchcd~/Library/LaunchAgents/homebrew.mxcl.postgresql@11.plistIn that case, use the plist file referenced in the command output to locate where the postgresql logs are written. The postgresql log file will help you detect the issue. If you see something regarding incompatible versions of postgresql, see issue #1.Homebrew will install the psql client (psql) under /opt/homebrew/Cellar/postgresql@11/11.20_2/bin/ ; you may want to create a symlink from here to somewhere in your path.By default, the postgres service listens on port 5432. Next, create thecidcdevuser, your localcidcdevelopment database, and a localcidctestdatabase that the unit/integration tests will use:psqlpostgresec-c"create user cidcdev with password '1234'"# Database to use for local developmentpsqlpostgres-c"create database cidc"psqlcidc-c"grant all privileges on database cidc to cidcdev"psqlcidc-c"create extension citext"psqlcidc-c"create extension pgcrypto"# Database to use for automated testingpsqlpostgres-c"create database cidctest"psqlcidctest-c"grant all privileges on database cidctest to cidcdev"psqlcidctest-c"create extension citext"psqlcidctest-c"create extension pgcrypto"Now, you should be able to connect to your development database with the URIpostgresql://cidcdev:1234@localhost:5432/cidc. Or, in the postgres REPL:psqlcidcInstall the gcloud CLIhttps://cloud.google.com/sdk/docs/install. You will use this CLI to authenticate with GCP for development purposes.Setting Environment VariablesThe app will use the environment variables defined in the.envfile to connect to Auth0 and GCP. You will need to update those values with either the dev or staging Auth0 and GCP instances values provided by Essex or CBIIT/Cloud2. The followingflask db upgradecommand relies on these environment variables.Each time the backend starts up (including cloud-functions), an object that pulls secrets from Secret Manager is used to initialize the google client libraries. You need to authenticate with gcloud beforehand so that the API can fetch the secrets. First, you will need to ensure that your account has the "Secret Manager Secrets Accessor" role. Then, generate Application Default Credentials by running the following,gcloudauthapplication-defaultloginTo check that your gcloud config is set to the right config rungcloud config list.The application reads environment variables that map to the appropriate secret ids for the selected project.  As of this writing, the env variables and secret ids are listed below (replace[env_tag]with the appropriate environment tag):env_variables:APP_ENGINE_CREDENTIALS_ID:"cidc_app_engine_credentials_[envtag]"AUTH0_CLIENT_SECRET_ID:"cidc_auth0_client_secret_[envtag]"CLOUD_SQL_DB_PASS_ID:"cidc_cloud_sql_db_pass_[envtag]"CSMS_BASE_URL_ID:"cidc_csms_base_url_[envtag]"CSMS_CLIENT_ID_ID:"cidc_csms_client_id_[envtag]"CSMS_CLIENT_SECRET_ID:"cidc_csms_client_secret_[envtag]"CSMS_TOKEN_URL_ID:"cidc_csms_token_url_[envtag]"INTERNAL_USER_EMAIL_ID:"cidc_internal_user_email_[envtag]"PRISM_ENCRYPT_KEY_ID:"cidc_prism_encrypt_key_[envtag]"To ensure you are using the correct secret ids, you may list the secrets for the active project using the following command:gcloud secrets listAdding new secrets to the applicationTo add new secrets to the application, follow the instructions below:Choose a name for the secret.  It should be snake case, and end in the relevant environment tag, as seen above.Reach out to the DevOps team to add the secret to Secret Manager.  Include the secret value for each environment.Choose an environment variable to map to the secret id.  It should end in_ID. This is because the secret will be passed in directly under testing conditions, using the same environment variable minus the_IDpostfix to avoid conflicts.Add the env variable mapping to the GAE environment configurations under thesecretssection. These are theapp.[env].yamlfiles.Load the secret into the SETTINGS dictionary via thecidc_api/config/settings.pymodule. See the# CSMS Integration Valuessection for how to do this.Next, you'll need to set up the appropriate tables, indexes, etc. in your local database. To do so, run:FLASK_APP=cidc_api.app:appflaskdbupgradeConnecting to a Cloud SQL database instanceInstall theCloud SQL Proxy:curl-o/usr/local/bin/cloud_sql_proxyhttps://dl.google.com/cloudsql/cloud_sql_proxy.darwin.amd64
chmod+x/usr/local/bin/cloud_sql_proxy
mkdir~/.cloudsqlProxy to the staging Cloud SQL instance:cloud_sql_proxy-instancescidc-dfci-staging:us-central1:cidc-postgresql-staging-dir~/.cloudsqlIn your.envfile, comment outPOSTGRES_URIand uncomment all environment variables prefixed withCLOUD_SQL_. Restart your local API instance, and it will connect to the staging Cloud SQL instance via the local proxy.If you wish to connect to the staging Cloud SQL instance via the postgres REPL, download and run the CIDC sql proxy tool (a wrapper forcloud_sql_proxy):# Download the proxycurlhttps://raw.githubusercontent.com/NCI-CIDC/cidc-devops/master/scripts/cidc_sql_proxy.sh-o/usr/local/bin/cidc_sql_proxy# Prepare the proxychmod+x/usr/local/bin/cidc_sql_proxy
cidc_sql_proxyinstall# Run the proxycidc_sql_proxystaging# or cidc_sql_proxy prodRunning database migrationsThis project usesFlask Migratefor managing database migrations. To create a new migration and upgrade the database specified in your.envconfig:exportFLASK_APP=cidc_api/app.py# Generate the migration scriptflaskdbmigrate-m"<a message describing the changes in this migration>"# Apply changes to the databaseflaskdbupgradeTo revert an applied migration, run:flaskdbdowngradeIf you're updatingmodels.py, you should create a migration and commit the resultingServing LocallyOnce you have a development database set up and running, run the API server:ENV=devgunicorncidc_api.app:appTestingThis project usespytestfor testing.To run the tests, simply run:pytestCode FormattingThis project usesblackfor code styling.We recommend setting up autoformatting-on-save in your IDE of choice so that you don't have to worry about runningblackon your code.DeploymentCI/CDThis project usesGitHub Actionsfor continuous integration and deployment. To deploy an update to this application, follow these steps:Create a new branch locally, commit updates to it, then push that branch to this repository.Make a pull request from your branch intomaster. This will trigger GitHub Actions to run various tests and report back success or failure. You can't merge your PR until it passes the build, so if the build fails, you'll probably need to fix your code.Once the build passes (and pending approval from collaborators reviewing the PR), merge your changes intomaster. This will trigger GitHub Actions to re-run tests on the code then deploy changes to the staging project.Try out your deployed changes in the staging environment once the build completes.If you're satisfied that staging should be deployed into production, make a PR frommasterintoproduction.Once the PR build passes, mergemasterintoproduction. This will trigger GitHub Actions to deploy the changes on staging to the production project.For more information or to update the CI workflow, check out the configuration in.github/workflows/ci.yml.Deploying by handShould you ever need to deploy the application to Google App Engine by hand, you can do so by running the following:gcloudappdeploy<app.staging.yamlorapp.prod.yaml>--project<gcloudprojectid>That being said, avoid doing this! Deploying this way circumvents the safety checks built into the CI/CD pipeline and can lead to inconsistencies between the code running on GAE and the code present in this repository. Luckily, though, GAE's built-in versioning system makes it hard to do anything catastrophic :-)Connecting to the APICurrently, the staging API is hosted athttps://api.cidc-stage.nci.nih.govand the production instance is hosted athttps://api.cidc.nci.nih.gov.To connect to the staging API withcurlor a REST API client like Insomnia, get an id token from cidc-stage.nci.nih.gov, and include the headerAuthorization: Bearer YOUR_ID_TOKENin requests you make to the staging API. If your token expires, generate a new one following this same procedure.To connect to the production API locally, follow the same procedure, but instead get your token from cidc.nci.nih.gov.Provisioning the system from scratchFor an overview of how to set up the CIDC API service from scratch, see the step-by-step guide inPROVISION.md.Setting up Docker ComposeIf you would like to run this project as a docker container.  We have dockerized the cidc-api-gae and cidc-ui so that you don't have to install all the requirements above.  Included in the docker-compose file are postgres:14 with data and test user login, bigquery-emulator, fake-gcs-server with buckets and data to match postgres, and gcs-oauth2-emulator to generate faked presigned urls.NOTE:You must have docker installed and have this repository and cidc-ui in the same directory (~/git/cidc/cidc-ui and ~/git/cidc/cidc-api-gae), or you can download each and build the image with the commanddocker build .NOTE:Having issues with the cidc-ui docker container. You'll have to start that manually using the instructions in the repo.NOTE:You can't use Docker while simultaneously running your NIH VPN. This is due to a quirk with self-hosting a google secrets bucket. More work is required to make the docker containers work while the VPN is on.This repo has hot code reloading. However, you will need to build the image again if there is an update to python libraries. Make sure you don't use a cached image when rebuilding.Make sure you add this line to your /etc/hosts file:127.0.0.1       host.docker.internalTo run everything simply run the following commands:vim.env# uncomment the docker section in the .env file.  Comment out any overlaping variable defintions(POSTGRES)cp~/.config/gcloud/application_default_credentials.json.cddocker
dockercomposeupNOTE:You still need to install and signin to gcloud CLI. The application_default_credentials.json should be under the cidc-api-gae directory next to the Dockerfile. We have mocked most of the connection points to GCP but at startup it still checks for a valid user account. This is very similar behavior to aws's localstack. It requires a realistic token at the start even though it doesn't make connections to aws.TODO:The application_default_credentials.json I think could be faked and pointed to the gcs-oauth2-emulator for startup.  In this case a gcloud cli wouldnt be needed at all and a faked application_default_credentials.json could be uploaded under the docker folder.JIRA IntegrationTo set-up the git hook for JIRA integration, run:ln-s../../.githooks/commit-msg.git/hooks/commit-msg
chmod+x.git/hooks/commit-msg
rm.git/hooks/commit-msg.sampleThis symbolic link is necessary to correctly link files in.githooksto.git/hooks. Note that setting thecore.hooksPathconfiguration variable would lead topre-commit failing. Thecommit-msghookruns afterthepre-commithook, hence the two are de-coupled in this workflow.To associate a commit with an issue, you will need to reference the JIRA Issue key (For eg 'CIDC-1111') in the corresponding commit message.API FAQHow is the API repo structured?At the top-level, there are a handful of files related to how and where the API code runs:app.prod.yamlandapp.staging.yamlare the App Engine config files for the prod and staging API instances - these specify instance classes, autoscaling settings, env variables, and what command App Engine should run to start the app.gunicorn.conf.pycontains config for thegunicornserver that runs the API’s flask app in production.The migrations/versions directorycontains SQLAlchemy database migrations generated using flask-sqlalchemy.The core API code lives in a python module in thecidc_apisubdirectory. In this subdirectory, theapp.pyfile contains the code that instantiates/exports the API’s flask app. Stepping through that file top to bottom is probably the best way to get an overall picture of the structure of the API code:get_loggerinstantiates a logger instance based on whether the app is running in a flask development server or gunicorn production server. We need this helper function (or something like it), because logs must be routed in a particular manner for them to show up in stderr when the app is running as a gunicorn server. Any python file in the cidc_api module that includes logging should call this get_logger helper at the top of the file.Next, the Flask app instance is created and configured using settings loaded fromsettings.py. This file contains a handful of constants used throughout the app code. Additionally, it contains code forsetting up the temporary directorieswhere empty manifest/assay/analysis templates will live.This lineat the bottom of the file builds a settings dictionary mapping variable names to values for all constants (i.e., uppercase variables) defined above it.Next,CORSis enabled. CORS allows the API to respond to requests originating from domains other than the API’s domain. If we didn’t do this, then an API instance running at “api.cidc.nci.nih.gov” would be prohibited from responding to requests from a UI instance running at “cidc.nci.nih.gov”.Next,init_dbconnects the flask-sqlalchemy package to a given API app instance. Moreover, it sets up our database migration utility,flask-migrate, which provides CLI shortcuts for generating migrations based on changes to the API’s sqlalchemy models. Currently, db migrations arerun every time init_db is called, but this is arguably tech debt, since it slows down app startup for no good reason - it might be better to try running db migrations as part of CI. (All other code in this file is related to building database connections based on the environment in which the app is running).Next,register_resources“wires up” all of the REST resources in the API, which are organized as independent flaskblueprints. Each resource blueprint is a collection of flask endpoints. Resources are split up into separate blueprints solely for code organization purposes.Next,validate_api_authenforces that all endpoints configured in the API are explicitly flagged as public or private using thepublicandrequires_authdecorators, respectively. This is intended to help prevent a developer from accidentally making a private endpoint public by forgetting to include the requires_auth decorator. If this validation check fails, the app won’t start up.Next,register_dashboardswires up our plot.ly dash dashboards.Next,handle_errorsadds generic code for formatting any error thrown while a request is being handled as JSON.Finally, if the app submodule is being run directly via “python -m cidc_api.app”,this codewill start a flask (non-gunicorn) server.For diving deeper into the API code’s structure, the next place to look is theresourcesdirectory. Endpoint implementations string together code for authenticating users, loading and validating JSON input from requests, looking up or modifying database records, and dumping request output to JSON. For some endpoints, a lot of this work is handled using generic helper decorators - e.g., theupdate_userendpoint uses nearly every helper available in therest_utils.pyfile. For others, like theupload_analysisendpoint, the endpoint extracts request data and builds response data in an endpoint-specific way. Most endpoints will involve some interaction with sqlalchemy models, either directly in the function body or via helper decorators.How do I add a new resource to the API?Create a new file in theresourcesdirectory named “<resource>.py”.Create a flask blueprint for the resource, named “<resource>_bp” (example).Add the blueprint to theregister_resourcesfunction. The resource’s url_prefix should generally be “/<resource>”.How do I add a new endpoint to the API?If you want to add an endpoint to an existing REST resource, open the file in the resources directory related to that resource. You can build an endpoint using some (almost definitely not all) of these steps:Wire up the endpoint. Find theblueprintfor the resource, usually named like “<resource>_bp”. You add an endpoint to the blueprint by decorating a python function using theroutemethod on the blueprint instance.Configure endpoint auth. Either flag the endpoint as public using thepublicdecorator, or configure authentication and authorization using therequires_authdecorator. The requires_auth decorator takes a unique string identifier for this endpoint as its required first argument (potentially used for endpoint-specific role-based access control logichere- the string ID is passed to the “resource” argument), and an optional list of allowed roles as its second argument (if no second arg is provided, users with all roles will be able to access the endpoint).Configure custom URL query params.The API uses thewebargslibrary for validating and extracting URL query param data. For example, the “GET /permissions/” endpointconfigures a query param "user_id"for filtering the resulting permissions list by user id.Look up a database record associated with the request.Use thewith_lookupdecorator to load a database record based on a URL path parameter. The with_lookup decorator takes three arguments: the first is the sqlalchemy model class, the second is the name of the URL path parameter that will contain the ID of the database record to look up, and the third is whether or not to check that the client making the request has seen the most recent version of the object (an “etag” is a hash of a database record’s contents - set check_etag=True to ensure that the client’s provided etag is up-to-date in order to, e.g., prohibit updates based on stale data). See, for example, usage forlooking up a particular user by ID- note that “user” is the name of the URL path parameter in the argument to user_bp.route.Deserialize the request body. POST and PATCH endpoints generally expect some JSON data in the request body. Such endpoints should validate that this data has the expected structure and, if appropriate, load that data into a sqlalchemy model instance. Theunmarshal_requestdecorator makes it easy to do this. The unmarshal request decorator takes three arguments: the first is amarshmallowschema defining the expected request body structure, the second is the argument name through which the deserialized result data should be passed to the endpoint function (e.g.,“permission”andpermission), and the third is whether to try loading the request body into a sqlalchemy model or to just leave it as a python dictionary. For schemas autogenerated from sqlalchemy models, see theschemas.pyfile - we use themarshmallow-sqlalchemylibrary for this.Serialize the response body. If an endpoint returns a database record (or a list of records), use themarshal_responsedecorator to convert a sqlalchemy model instance (or list of instances) into JSON. Seethis example.Note: when you add a new endpoint to the API, you’ll also need to add that endpoint to thetest_endpoint_urlstest. This test ensures that CIDC developers are aware of every endpoint that the API exposes (since under certain configurations Flask might expose unwanted default endpoints).How does API authentication and authorization work?First off - what’s the difference between authentication and authorization? Authentication is concerned with verifying a user’s identity. Authorization is concerned with restricting the actions a user is allowed to perform within an application based on their identity. Since we need to know a user’s identity in order to execute logic based on their identity, user authentication is required for authorization.AuthenticationWe use a protocol called OpenID Connect to leverage Auth0/Google for verifying the identity of users accessing data from the API (rather than maintaining user identities and login sessions ourselves). Here’sa talkthat might help in learning about OAuth 2.0 and OpenID Connect - I highly recommend watching it before making any non-trivial update to authentication-related logic or configuration.API authentication relies onidentity tokensgenerated by Auth0 to verify that the client making the request is logged in. An identity token is aJWTcontaining information about a user’s account (like their name, their email, their profile image, etc.) and metadata (like an expiry time after which the token should be considered invalid). Here’s the part that makes JWTs trustworthy and useful for authentication: they include a cryptographic signature from a trusted identity provider service (Auth0, in our case). So, an identity token represents a currently authenticated user if:It is a well-formatted JWT.It has not yet expired.Its cryptographic signature is valid.JWTs are a lot like passports - they convey personal information, they’re issued by a trusted entity, and they expire after a certain time. Moreover, like passports, JWTscan be stolenand used to impersonate someone. As such, JWTs should be kept private and treated sort of like short-lived passwords.AuthorizationThe CIDC API takes arole-based access controlapproach to implementing its authorization policy. Each user is assigned a role (likecidc-admin,cimac-biofx-user, etc.), and the actions they’re allowed to take in the system are restricted based on that role. For the most part, any two users with the same role will be allowed to take the same actions in the CIDC system.The one exception to the role-based access control rule is file access authorization, which is configured at the specific user account level for non-admin users via trial/assay permissions.ImplementationHere’s where this happens in the code.check_authis the workhorse authentication and authorization function (this is whatrequires_authcalls under the hood). check_auth first authenticates the current requesting user’s identity then performs authorization checks based on that identity:Here’s what theauthenticatefunction does:Tries to extractthe identity token from the request’s HTTP headers. It expects a header with the structure “Authorization: Bearer <id token>”. If the expected “Authorization” header is not present, it looks for an identity token in the request’s JSON body (this is specific to the way our plotly dash integration handles authentication).Gets a public key from Auth0that it will use to verify that the identity token was signed by Auth0.Decodes the identity tokenand verifies its signature using the public key obtained in the previous step. If the JWT is malformed, expired, or otherwise invalid, this step will respond to the requesting user with HTTP 401 Unauthorized.Initializes and returns a sqlalchemy model instance for the current user.Next, check_auth passes the user info parsed from the identity token to theauthorizefunction. The function implements some access control logic based on whether the requesting user’s account is registered and their role gives them permission to access the endpoint they are trying to access.Note: this function encapsulates simple, generic RBAC operations (“only users with these roles can perform this HTTP action on this endpoint”) and does not encapsulate more complicated, endpoint-specific role-based access control logic (e.g.,this logic for listing file access permissions). As things currently stand, understanding the RBAC policy for a particular endpoint requires reviewing that endpoint’s implementation in its entirety.How do I propagate SQLAlchemy model updates to the database?Updates to SQLAlchemy model python classes do not automatically update the corresponding tables in the database. Rather, you need to create a “migration” script to apply any model class updates to the database. We use theflask-migrateplugin for managing our migration scripts. See this briefoverviewof creating, running, and undoing migrations.Note:although flask-migrate and alembic, the tool flask-migrate uses under the hood, can automatically pick up certain sqlalchemy model class changes (e.g., adding/removing models, adding/removing columns, column data type changes), there are other changes that it can’t pick up automatically. Two examples I’ve encountered are adding/removing values from enum types and adding/updatingCHECK constraints. For this reason, always review the auto-generated migration file before applying it, making any required manually edits/additions.How can I check that a database migration works?First, I run the “flask db upgrade” against my local database - this can catch basic errors, like syntax or type issues, even if there’s no data currently stored in the local database.Next, I’ll try running the migration against the staging database from my local computer (since the staging db generally has representative data in it, this can catch further errors you might miss in a local db test). To do this, you need toset up a connection to the staging dband toupdate your .env fileto tell your local api code to use this connection.Make sure that no one else is using the staging db for anything critical, then run the db upgrade. If you encounter new errors, fix them. Once the upgrade succeeds, undo it with “flask db downgrade”, then make a PR to deploy the new migration.What happens when a database migration fails, and what should I do to remediate the situation?Because database migrations are run when the app starts up, failed database migrations manifest as the API failing to start up. This usually looks like the “failed to load account information” error message appearing 5-10 seconds after trying to load the portal.Remediating a failing migration requires two steps:Redirect traffic to a previous app engine version that does not include the failing migration code. You can select a known-good version fromthis pagein the GCP console.Debug and fix the migration locally following a process like the one described above.Note:when you want to undo a migration thatdid not fail to run, but has some other issue with it, the solution is different. If you try to simply send traffic to a previous app engine version without the migration you want to undo included in it, you’ll get an error on app startup (something like “alembic.util.CommandError: Can't locate revision identified by '31b8ab83c7d'”). In order to undo this migration, you’ll need tomanually connect to the cloud sql instance,update your .env fileto tell your local api code to use this connection, then run “flask db downgrade”. Once that command succeeds, you’ve rolled back the unwanted migration, and you can safely send traffic to a previous app engine version that doesn’t include the migration.What’s packaged up in cidc-api-modules pypi package?The cidc-api-modules package includes only the submodules used in the cidc-cloud-functions module. Here’s thefull list. Notably, the “cidc_api.app” and “cidc_api.resources” submodules are excluded, since these pertain only to the API. To be perfectly honest, I don’t remember the issue that led to the decision to not simply package up and public the top-level “cidc_api” module (it’s possible even if it’s not necessary). Anyhow, this means that bumping the cidc-api-modules version is only necessary when making changes to the included submodules that you want to propagate to the cloud functions repo.Relatedly, it could be worth looking into combining the cloud functions repo into the cidc-api-gae repo. There’s no great reason for them to be separate. In fact, since they share code related to interacting with the database and with GCP, the decision to separate the two repos likely creates more friction than it alleviates.