Package: thoth-storages
This library provides a library calledthoth-storagesused in projectThoth.  The library exposes core queries and methods
forPostgreSQL databaseas well as adapters
for manipulating withCephvia its S3 compatible API.Quick StartPre-requisites:make sure you havepodmanandpodman-composeinstalled. You can install those tools by runningdnf install-ypodmanpodman-composemake sure you are in an environment created withpipenv install--devTo develop locally the first time:Have a pg dump that you canretrieve from aws s3Get the latest PostgreSQL container image from:https://catalog.redhat.com/software/containers/rhel8/postgresql-13/5ffdbdef73a65398111b8362?container-tabs=gti&gti-tabs=red-hat-loginRunpodman-composeupto scale up pods for database and pgweb. For more detail, refer to theRunning PostgreSQL locally sectionRun this command to sync the pg dump into the local database:psql -h localhost -p 5432 --username=postgres < pg_dump.sqlNow you are ready to test new queries orcreate new migrationsIf you already have a local database, make sure it is not outdated and rember to follow theGenerating migrations and schema adjustment in deploymentsection before testing any changes.Installation and UsageThe library can be installed via pip or Pipenv fromPyPI:pipenv install thoth-storagesThe library provides a CLI that can assist you with exploring schema and data
storing:thoth-storages --help#Inaclonedrepo,run:PYTHONPATH=. pipenv run python3 thoth-storages --helpYou can run prepared test-suite via the following command:pipenv install --dev
pipenv run python3 setup.py testRunning PostgreSQL locallyYou can usedocker-compose.yamlpresent in this repository to run a local
PostgreSQL instance, (make sure you installedpodman-compose):$dnfinstall-ypodmanpodman-compose$# Also available from PyPI: pip install podman-compose$podman-composeupAfter running the commands above, you should be able to access a local
PostgreSQL instance atlocalhost:5432. This is also
the default configuration for PostgreSQL’s adapter that connects to localhost
unlessKNOWLEDGE_GRAPH_HOSTis supplied explicitly (see also other
environment variables in the adapter constructor for more info on configuring
the connection). The default configuration uses database namedpostgreswhich can be accessed usingpostgresuser andpostgrespassword (SSL is
disabled).The provideddocker-compose.yamlhas alsoPGwebenabled to enable data exploration using
UI. To access it visitlocalhost:8081.The provideddocker-compose.yamldoes not use any volume. After you
containers restart, the content will not be available anymore.You can sync your local instance usingpgsql:$psql-hlocalhost-p5432--username=postgres<pg_dump.sqlIf you would like to experiment with PostgreSQL programmatically, you can use
the following code snippet as a starting point:fromthoth.storagesimportGraphDatabasegraph=GraphDatabase()graph.connect()# To clear database:# graph.drop_all()# To initialize schema in the graph database:# graph.initialize_schema()Generating migrations and schema adjustment in deploymentIf you make any changes to data model of the main PostgreSQL database, you need
to generate migrations. These migrations state how to adjust already existing
database with data in deployments. For this purpose,Alembic migrationsare used. Alembic can (partially)
automatically detect what has changed and how to adjust already existing
database in a deployment.Alembic uses incremental version control, where each migration is versioned and
states how to migrate from previous state of database to the desired next state
- these versions are present inalembic/versionsdirectory and are
automatically generated with procedure described bellow.If you make any changes, follow the following steps which will generate version
for you:Make sure your local PostgreSQL instance is running (followRunning
PostgreSQL locallyinstructions above):$podman-composeupRun Alembic CLI to generate versions for you:#Makesureyouhaveyourenvironmentsetup:#pipenvinstall--dev#Makesureyouarerunningthemostrecentversionofschema:$PYTHONPATH=.pipenvrunalembicupgradehead#Actuallygenerateanewversion:$PYTHONPATH=.pipenvrunalembicrevision--autogenerate-m"Added row to calculate sum of sums which will be divided by 42"Review migrations generated by Alembic. NoteNOT all changes are
automatically detected by Alembic.Make sure generated migrations are part of your pull request so changes are
propagated to deployments:$gitaddthoth/storages/data/alembic/versions/In a deployment, use Management API and its/graph/initializeendpoint to
propagate database schema changes in deployment (Management API has to have
recent schema changes present which are populated with newthoth-storagesreleases).If running locally and you would like to propagate changes, run the following
Alembic command to update migrations to the latest version:$PYTHONPATH=.pipenvrunalembicupgradeheadIf you would like to update schema programmatically run the following Python
code:fromthoth.storagesimportGraphDatabasegraph=GraphDatabase()graph.connect()graph.initilize_schema()When updating a deployment, make sure all the components use the same database
schema. Metrics exposed from a deployment should state schema version of all
the components in a deployment.Generate schema imagesYou can use shipped CLIthoth-storagesto automatically generate schema
images out of the current models:#First,makesureyouhavedevpackagesinstalled:$pipenvinstall--dev$PYTHONPATH=.pipenvrunpython3./thoth-storagesgenerate-schemaThe command above will produce an image namedschema.png. Check--helpto get more info on available options.If the command above fails with the following exception:FileNotFoundError: [Errno 2] "dot" not found in path.make sure you havegraphvizpackage installed:dnf install -y graphvizCreating own performance indicatorsPerformance indicators report performance aspect of a library onAmunand results can be automatically
synced if the following procedure is respected.To create own performance
indicator, create a script which tests desired functionality of a library. An
example can be matrix multiplication script present inthoth-station/performancerepository. This script can be supplied toDependency Monkeyto validate certain combination of libraries in desired runtime and buildtime
environment. Please follow instructions on how to create a performance script
shown in theREADME of performance repo.To create relevant models, adjustthoth/storages/graph/models_performance.pyfile and add your model.
Describe parameters (reported in@parameterssection of performance
indicator result) and result (reported in@result). The name of class
should matchnamewhich is reported by performance indicator run.classPiMatmul(Base,BaseExtension,PerformanceIndicatorBase):"""A class for representing a matrix multiplication micro-performance test."""# Device used during performance indicator run - CPU/GPU/TPU/...device=Column(String(128),nullable=False)matrix_size=Column(Integer,nullable=False)dtype=Column(String(128),nullable=False)reps=Column(Integer,nullable=False)elapsed=Column(Float,nullable=False)rate=Column(Float,nullable=False)All the models useSQLAchemy.  Seedocsfor more info.Online debugging of queriesYou can print to logger all the queries that are performed to a PostgreSQL
instance. To do so, set the following environment variable:export THOTH_STORAGES_DEBUG_QUERIES=1Memory usage statistictsYou can print information about PostgreSQL adapter together with statistics on
the adapter in-memory cache usage to logger (it has to have at least levelINFOset). To do so, set the following environment variable:export THOTH_STORAGES_LOG_STATS=1These statistics will be printed once the database adapter is destructed.Automatic backups of Thoth deploymentIn each deployment, an automatic knowledgegraph backup cronjobis run, usually once a
day. Results of automatic backups are stored on Ceph - you can find them ins3://<bucket-name>/<prefix>/<deployment-name>/graph-backup/pg_dump-<timestamp>.sql.
Refer to deployment configuration for expansion of parameters in the path.To create a database instance out of this backup file, run a fresh local
PostgreSQL instance and fill it from the backup file:$cdthoth-station/storages$awss3--endpoint<ceph-s3-endpoint>cps3://<bucket-name>/<prefix>/<deployment-name>/graph-backup/pg_dump-<timestamp>pg_dump-<timestamp>.sql$podman-composeup$psql-hlocalhost-p5432--username=postgres<pg_dump-<timestamp>.sqlpassword: <type password "postgres" here>
<logs will show up>Manual backups of Thoth deploymentYou can usepg_dumpandpsqlutilities to create dumps and restore the
database content from dumps. This tool is pre-installed in the container image
which is running PostgreSQL so the only thing you need to do is executepg_dumpin Thoth’s deployment in a PostgreSQL container to create a dump,
useoc cpto retrieve dump (or directly useoc execand create the dump
from the cluster) and subsequentlypsqlto restore the database content.
The prerequisite for this is to have access to the running container (edit
rights).#ExecutethefollowingcommandsfromtherootofthisGitrepo:#ListPostgreSQLpodsrunning:$ocgetpod-lname=postgresqlNAME                 READY     STATUS    RESTARTS   AGE
postgresql-1-glwnr   1/1       Running   0          3d#OpenremoteshelltotherunningcontainerinthePostgreSQLpod:$ocrsh-tpostgresql-1-glwnrbash#Performdumpofthedatabase:(cluster-postgres)$pg_dump>pg_dump-$(date+"%s").sql(cluster-postgres)$lspg_dump-*.sql# Remember the current dump name(cluster-postgres)pg_dump-1569491024.sql(cluster-postgres)$exit#Copythedumptothecurrentdir:$occpthoth-test-core/postgresql-1-glwnr:/opt/app-root/src/pg_dump-1569491024.sql.#StartlocalPostgreSQLinstance:$podman-composeup--detach<logs will show up>$psql-hlocalhost-p5432--username=postgres<pg_dump-1569491024.sqlpassword: <type password "postgres" here>
<logs will show up>You can ignore error messages related to an owner error like this:STATEMENT:  ALTER TABLE public.python_software_stack OWNER TO thoth;
ERROR:  role "thoth" does not existThe PostgreSQL container uses user “postgres” by default which is different
from the one run in the cluster (“thoth”). The role assignment will simply not
be created but data will be available.Syncing results of a workflow run in the clusterEach workflow task in the cluster reports a JSON which states necessary
information about the task run (metadata) and actual results. These results of
workflow tasks are stored on object storageCephvia S3
compatible API and later on synced via graph syncs to the knowledge graph. The
component responsible for graph syncs isgraph-sync-jobwhich is written generic
enough to sync any data and report metrics about synced data so you don’t need
to provide such logic on each new workload registered in the system. To sync
your own results of job results (workload) done in the cluster, implement
related syncing logic in thesync.pyand register handler in theHANDLERS_MAPPINGin the same file. The mapping
maps prefix of the document id to the handler (function) which is responsible
for syncing data into the knowledge base (please mind signatures of existing
syncing functions to automatically integrate withsync_documentsfunction
which is called fromgraph-sync-job).Query Naming conventions in ThothFor query naming conventions, please read all the docs inconventions for
query name.Accessing data on CephTo access data on Ceph, you need to knowaws_access_key_idandaws_secret_access_keycredentials
of endpoint you are connecting to.Absolute file path of data you are accessing is constructed as:s3://<bucket_name>/<prefix_name>/<file_path>There are two ways to initialize the data handler:Configure environment variablesVariable nameContentS3_ENDPOINT_URLCeph Host nameCEPH_BUCKETCeph Bucket nameCEPH_BUCKET_PREFIXCeph PrefixCEPH_KEY_IDCeph Key IDCEPH_SECRET_KEYCeph Secret Keyfromthoth.storages.cephimportCephStoreceph=CephStore()Initialize the object directly with parametersfromthoth.storages.cephimportCephStoreceph=CephStore(key_id=<aws_access_key_id>,secret_key=<aws_secret_access_key>,prefix=<prefix_name>,host=<endpoint_url>,bucket=<bucket_name>)After initialization, you are ready to retrieve dataceph.connect()try:# For dictionary stored as jsonjson_data=ceph.retrieve_document(<file_path>)# For general blobblob=ceph.retrieve_blob(<file_path>)exceptNotFoundError:# File does not existAccessing Thoth Data on the Operate-First Public BucketA public instance of Thoth’s database is available on theOperate-First Public Bucketfor external contributors to start developing components of Thoth.Instructions for accessing the bucket are available in thedocumentationof thethoth/datasetsrepository.Be careful not to store any confidential or valuable information in this bucket as its content can be wiped out at any time.