Package: thoth-storage
This library provides a library calledthoth-storagesused in projectThoth.  The library exposes core queries and methods
for PostgreSQL database as well as adapters for manipulating with Ceph via its
S3 compatible API.Installation and UsageThe library can be installed via pip or Pipenv fromPyPI:pipenv install thoth-storagesThe library does not provide any CLI, it is rather a low level library
supporting other parts of Thoth.You can run prepared testsuite via the following command:pipenv install --dev
pipenv run python3 setup.py test#Togeneratedocs:pipenv run python3 setup.py build_sphinxRunning PostgreSQL locallyYou can usedocker-compose.yamlpresent in this repository to run a local PostgreSQL instance, (make sure you installedpodman-compose):$podman-composeupAfter running the command above, you should be able to access a local PostgreSQL instance atlocalhost:5432. This is also the default configuration for PostgreSQL’s adapter - you don’t need to provideGRAPH_SERVICE_HOSTexplicitly. The default configuration uses database namedpostgreswhich can be accessed usingpostgresuser andpostgrespassword (SSL is disabled).The provideddocker-compose.yamlhas also PGweb enabled for to have an UI for the database content. To access it visithttp://localhost:8081/.The provideddocker-compose.yamldoes not use any volume. After you containers restart, the content will not be available anymore.If you would like to experiment with PostgreSQL programatically, you can use the following code snippet as a starting point:fromthoth.storagesimportGraphDatabasegraph=GraphDatabase()graph.connect()# To clear database:# graph.drop_all()# To initialize schema in the graph database:# graph.initialize_schema()Generating migrations and schema adjustment in deploymentIf you make any changes to data model of the main PostgreSQL database, you need
to generate migrations. These migrations state how to adjust already existing
database with data in deployments. For this purpose,Alembic migrationsare used. Alembic can (partially)
automatically detect what has changed and how to adjust already existing
database in a deployment.Alembic uses incremental version control, where each migration is versioned and
states how to migrate from previous state of database to the desired next state - these
versions are present inalembic/versionsdirectory and are automatically
generated with procedure described bellow.If you make any changes, follow the following steps which will generate version
for you:make sure your local PostgreSQL instance is running (followRunning
PostgreSQL locallyinstructions above):$podman-composeupRun Alembic CLI to generate versions for you:#Makesureyouhaveyourenvironmentsetup:#pipenvinstall--dev#Makesureyouarerunningthemostrecentversionofschema:$PYTHONPATH=.pipenvrunalembicupgradehead#Actuallygenerateanewversion:$PYTHONPATH=.pipenvrunalembicrevision--autogenerate-m"Added row to calculate sum of sums which will be divided by 42"Review migrations generated by Alembic. NoteNOT all changes are
automatically detected by Alembic.Make sure generated migrations are part of your pull request so changes are
propagated to deployments:$gitaddthoth/storages/data/alembic/versions/In a deployment, use Management API and its/graph/initializeendpoint to
propagate database schema changes in deployment (Management API has to have
recent schema changes present which are populated with newthoth-storagesreleases).If running locally and you would like to propagate changes, run the following Alembic command to update migrations to the latest version:$PYTHONPATH=.pipenvrunalembicupgradeheadIf you would like to update schema programmatically run the following Python code:fromthoth.storagesimportGraphDatabasegraph=GraphDatabase()graph.connect()graph.initilize_schema()Generate schema imagesYou can use shipped CLIthoth-storagesto automatically generate schema images out of the current models:#First,makesureyouhavedevpackagesinstalled:pipenv install --dev
PYTHONPATH=. pipenv run python3 ./thoth-storages generate-schemaThe command above will produce 2 images namedschema.pngandschema_cache.png. The first PNG file shows schema for the main PostgreSQL
instance and the latter one, as the name suggests, shows how cache schema looks
like.If the command above fails with the following exception:FileNotFoundError:[Errno2]"dot"notfoundinpath.make sure you havegraphvizpackage installed:dnf install -y graphvizCreating own performance indicatorsYou can create your own performance indicators. To create own performance
indicator, create a script which tests desired functionality of a library. An
example can be matrix multiplication script present inperformancerepository. This script can be supplied to Dependency Monkey to validate
certain combination of libraries in desired runtime and buildtime environment
or directly on Amun API which will run the given script using desired software
and hardware configuration. Please follow instructions on how to create a
performance script shown in theREADME of performance repo.To create relevant models, adjustthoth/storages/graph/models_performance.pyfile
and add your model. Describe parameters (reported in@parameterssection of
performance indicator result) and result (reported in@result). The name of
class should matchnamewhich is reported by performance indicator run.classPiMatmul(Base,BaseExtension,PerformanceIndicatorBase):"""A class for representing a matrix multiplication micro-performance test."""# Device used during performance indicator run - CPU/GPU/TPU/...device=Column(String(128),nullable=False)matrix_size=Column(Integer,nullable=False)dtype=Column(String(128),nullable=False)reps=Column(Integer,nullable=False)elapsed=Column(Float,nullable=False)rate=Column(Float,nullable=False)All the models useSQLAchemy.
Seedocsfor more info.Online debugging of queriesYou can print to logger all the queries that are performed to a PostgreSQL instance. To do so, set the following environment variable:export THOTH_STORAGES_DEBUG_QUERIES=1Online debugging of queriesYou can print information about PostgreSQL adapter together with statisics on
the graph cache and memory cache usage to logger (it has to have at least levelINFOset). To do so, set the following environment variable:export THOTH_STORAGES_LOG_STATS=1These statistics will be printed once the database adapter is destructed.Creating backups from Thoth deploymentYou can usepg_dumpandpsqlutilities to create dumps and restore
the database content from dumps. This tool is pre-installed in the container image
which is running PostgreSQL so the only thing you need to do is executepg_dumpin Thoth’s deployment in a PostgreSQL container to create a dump, useoc cpto retrieve dump (or directly useoc execand create the dump from the
cluster) and subsequentlypsqlto restore the database content. The
prerequisite for this is to have access to the running container (edit rights).#ExecutethefollowingcommandsfromtherootofthisGitrepo:#ListPostgreSQLpodsrunning:$ocgetpod-lname=postgresqlNAME                 READY     STATUS    RESTARTS   AGE
postgresql-1-glwnr   1/1       Running   0          3d#OpenremoteshelltotherunningcontainerinthePostgreSQLpod:$ocrsh-tpostgresql-1-glwnrbash#Performdumpofthedatabase:(cluster-postgres)$pg_dump>pg_dump-$(date+"%s").sql(cluster-postgres)$lspg_dump-*.sql# Remember the current dump name(cluster-postgres)pg_dump-1569491024.sql(cluster-postgres)$exit#Copythedumptothecurrentdir:$occpthoth-test-core/postgresql-1-glwnr:/opt/app-root/src/pg_dump-1569491024.sql.#StartlocalPostgreSQLinstance:$podman-composeup--detach<logs will show up>$psql-hlocalhost-p5432--username=postgres<pg_dump-1569491024.sqlpassword: <type password "postgres" here>
<logs will show up>Syncing results of jobs run in the clusterEach job in the cluster reports a JSON which states necessary information about
the job run (metadata) and actual job results. These results of jobs are stored
on object storageCephvia S3 compatible API and later on
synced via graph syncs to the knowledge graph. The component responsible for
graph syncs isgraph-sync-jobwhich is written generic
enough to sync any data and report metrics about synced data so you don’t need
to provide such logic on each new workload registered in the system. To sync
your own results of job results (workload) done in the cluster, implement
related syncing logic in thesync.pyand register handler in the_HANDLERS_MAPPINGin the same file. The mapping
maps prefix of the document id to the handler (function) which is responsible
for syncing data into the knowledge base (please mind signatures of existing
syncing funcions to automatically integrate withsync_documentsfunction
which is called fromgraph-sync-job).