Package: ata-db-models
ata-db-modelsDatabase models and migrations for Automating the Ask.UsageAs a packageWe useSQLModel, a layer on top of SQLAlchemy with Pydantic, to define our tables.
This is useful because we can import this package to interact with the tables AND have Pydantic objects in Python
that correspond to a row in the table.To install the package from PyPi, run:pip install ata-db-models. Check existing versionshere.Initialize a new clusterIf you want to initialize a fresh database cluster, pass in the env vars to connect to the cluster and runinit_db.
If the target cluster has IP restrictions, make sure your IP address is a valid access point.An example run with fake credentials (from the root dir of this project with the virtual env
activated):HOST=fakehost USER=fakeuser PASSWORD=fakepw DB_NAME=postgres python ata_db_models/db_init_stages/_0_init_db.pyThis should run the most up-to-date SQLModel definitions of the tables, which means you are
safe to then run any additional changes in role, access, and policy changes. So you can
run the rest of the steps indb_init_stages, one after the other in ascending numerical order.NoPORTis passed because the default port is 5432, the standard for Postgres.MigrationsSo you made some changes to what tables there are, what columns there are, indices, etc. and you'd like to
update the databases. This is what alembic is for!To generate a new revision after you've updated the models:Run this from the root of the project:DB_CONNECTION_STRING='postgresql://user:password@host:port/db_name' alembic revision --autogenerate -m "message"Check the/alembic/versions/directory for the new revision and verify that it does what you want it toRun this from the root of the project:DB_CONNECTION_STRING='postgresql://user:password@host:port/db_name' alembic upgrade headNote that you only need to generate the revision file (step 1)oncebecause we want the same content in each environment's database, but you do need to run theupgrade headcommand oncefor eachdatabase (change the DB_NAME to the desired target).To make new users, grant privileges, etc., follow the patterns used in db_init_stages along with the
helpers under ata_db_models.Create a new file under db_init_stages that does what you want and is prefixed with_X_, whereXis the next number (it has no function, it's just nice to keep track of the step order).Run the file. You can run it like so:HOST=fakehost USER=fakeuser PASSWORD=fakepw DB_NAME=postgres python ata_db_models/db_init_stages/_X_fake_file.pyI'd recommend that you then connect to the cluster and verify your changes took place.Note that you must provide valid host, user, password, and database name environment variables for it to work. ThePORTenv var has a default value of 5432, so it is omitted here. The only other env var you might need
(if you are creating new roles/users that have credentials) is theENABLE_SSMenv var. By default
it isFALSEbut if you set it toTRUEthen it will make sure to upload any new credentials to the
SSM parameter store.DevelopmentThis project usesPoetryto manage dependencies. It also helps with pinning dependency and python
versions. We also usepre-commitwith hooks forisort,black, andflake8for consistent code style and
readability. Note that this means code that doesn't meet the rules will fail to commit until it is fixed.We usemypyfor static type checking. This can be runmanually,
and the CI runs it on PRs to themainbranch. We also usepytestto run our tests.
This can be runmanuallyand the CI runs it on PRs to themainbranch.SetupInstall Poetry.Runpoetry install --no-rootRunsource $(poetry env list --full-path)/bin/activate && pre-commit install && deactivateto set uppre-commitYou're all set up! Your local environment should include all dependencies, including dev dependencies likeblack.
This is done with Poetry via thepoetry.lockfile.Run Code Format and LintingTo manually run isort, black, and flake8 all in one go, simply runpre-commit run --all-files. Explore thepre-commitdocs (linked above)
to see more options.Run Static Type CheckingTo manually run mypy, simply runmypyfrom the root directory of the project. It will use the default configuration
specified inpyproject.toml.Update DependenciesTo update dependencies in your local environment, make changes to thepyproject.tomlfile then runpoetry updatefrom the root directory of the project.Run TestsTo manually run rests, you need to have a Postgres instance running locally on port 5432. One way to do this
is to run a Docker container, then run the tests while it is active.(If you don't already have the image locally) Rundocker pull postgresRundocker run --rm --name postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_HOST_AUTH_METHOD=trust -p 127.0.0.1:5432:5432/tcp postgresRunDB_NAME=postgres pytest testsfrom the root directory of the project. Explore thepytestdocs (linked above)
to see more options.Note that if you decide to run the Postgres container with different credentials (a different password, port, etc.) or
via a different method, you will likely need to update the test file to point to the correct Postgres instance.Additionally, if you want to re-run the tests, you want to make sure you start over from a fresh Postgres
instance. If you run Postgres via Docker, you can simplyctrl-Cto stop the image and start a new one.