Package: ert-storage
ERT Storage ServerThis is the permanent storage solution for theERTproject. It is written in Python 3.6+
using FastAPI, Pydantic, SQLAlchemy and Azure Blob Storage.Development EnvironmentThis section describes how to use ERT Storage for testing or development purposes. It uses an in-memory SQLite database with no Azure Blob Storage.First, install ERT Storage into a Python virtual environment:# Clone this repogitclonehttps://github.com/equinor/ert-storagecdert-storage# Create the virtual environment (assuming you have Python 3.6+)python3-mvenvmy-venv# Enable the environmentsourcemy-venv/bin/activate# sh/bash/zsh# Install ERT Storage into the environment from current working directorypipinstall-e.Then start ERT Storage simply by using the newert-storagecommand. This command will tell you that you are using an in-memory SQLite database. Now, using your browser, navigate tohttp://localhost:8000to bring up the Swagger documentation page. Pages of note are:http://localhost:8000/docSwaggerhttp://localhost:8000/redocReDoc (alternative to Swagger)Persisting dataSpecifying theERT_STORAGE_DATABASE_URLenvironment variable it is possible to
specify a non-in-memory database. Refer toSQLAlchemy
Documentationfor all possible database URLs. However, only SQLite and PostgreSQL with default
SQLAlchemy drivers are officially supported at this time.SQLiteTo persist a SQLite database (note the number of forward slashes/):exportERT_STORAGE_DATABASE_URL="sqlite:///ert.db"# Use ert.db in current working directoryexportERT_STORAGE_DATABASE_URL="sqlite:////tmp/ert.db"# Use /tmp/ert.dbNOTE: SQLite does not support the same features as PostgreSQL, and as such
migrations (viaalembic) are not supported and some data may be stored or
fetched in an inefficient manner. You will have to delete your data whenever we
update the database schema, which happens relatively often.PostgreSQLWe use PostgreSQL in production and we usealembicfor migrations. The following will set everything up.SetERT_STORAGE_DATABASE_URL# Example postgres settingsHOST="localhost"PORT=5432# default PostgreSQL portDATA="ert_store"USER="my-usrnam"PASS="my-passwd"# The connection string is then:exportERT_STORAGE_DATABASE_URL="postgresql://${USER}:${PASS}@${HOST}:${PORT}/${DATA}"# orexportERT_STORAGE_DATABASE_URL="postgresql://my-usrnam:my-passwd@localhost:5432/ert_store"Then install a compatible SQLAlchemy PostgreSQL driver, such aspsycopg2. We provide the extras
requirementpostgreswhich installs the necessary requirements.pipinstall-e.[postgres]Thepsycopg2package requires the postgres client development headers to be
installed on your machine. Thepsycopg2-binaryis a precompiled
version of this package that might be compatible with your OS and PostgreSQL
version.Database setup: ContainerYou can use either Docker orpodmanto easily start a postgres instance as a
container. On modern Linux distributions it is easier to use thepodmanproject.See the docker.io page for postgres for more informationThe following instructions are forpodman. Replacingpodmanwithdockerwill# Start the container in the background, with password 'password', forwarding the port 5432 to the host.podmanrun--nameert-storage-pg-ePOSTGRES_PASSWORD=password-p5432:5432-ddocker.io/postgres# Useful commandspodmanstopert-storage-pg
podmanstartert-storage-pg
podmankillert-storage-pg# SIGTERMpodmanrmert-storage-pg# Remove the container entirely# Use with ERT StorageexportERT_STORAGE_DATABASE_URL="postgresql://postgres:password@localhost:5432/postgres"To reset the database, do:podmanexecert-storage-pgsh-c"dropdb -U postgres postgres; createdb -U postgres postgres"ert-storagealembicupgradeheadDatabase setup: Linux serviceFirst, install the dependencies:# RHEL / CentOS / Fedorasudoyuminstallpostgresql-serverpostgresql-devel# Ubuntusudoaptinstallpostgresqlpostgresql-clientThen, initialize the database and configure a user with a password.# Initialise stuff in /var (OPTIONAL)sudo-i-upostgres--initdb# Create superusersudo-i-upostgres--createuser-sert_user# Give yourself a password (default is to disallow password login)sudo-i-upostgres--psql-c"alter user ert_user with password '12345';"# Create a database with you as its ownercreatedb-Uert_userert_db# Set the database URL so that ERT Storage can connect to it# You will need to redo this every time you close your shellexportERT_STORAGE_DATABASE_URL="postgresql://ert_user:12345@localhost:5432/ert_db"To recreate the database, do:dropdbert_db&&createdb-Oert_userert_db
ert-storagealembicupgradeheadDatabase setup: Local user setup on RHEL 7First, download the required version of postgresql. For instance, for RHEL-7 you can checkhere.
Extract and copy postgresql to a custom directory; eg.:rpm2cpiopostgresql-<version>-x86_64.rpm|cpio-idvAppend the postgres bin path to your actual PATH; eg.exportPATH=$PATH:/data/install/postgres/usr/pgsql-<version>/binIn the directory where you want to host the database runcd<path-to-your-project>
initdb-D~/dbdirwhere dbdir represents the directory name where the database is hosted.Open the config file~/dbdir/postgresql.confand find the entryunix_socket_directoriesand set it to
unix_socket_directories = '/tmp/'Start the server viapostgres-D~/dbdir-k/tmpin other terminal, whenever you want to make use of the database, setexportPGHOST=/tmpNow everything should be good to go. The logs can be found in~/dbdir/log/directory.
To create a user and a database, you can use the following commands:createuser-hlocalhost-p5432-sert_user
createdb-hlocalhost-Uert_userert_dbAfter setupOnce a PostgreSQL server is running, you need to apply alembic migrations.ert-storageprovides a wrapper around thealembiccommand which
automatically selects the correct alembic configuration.# Upgrade to latest migrationert-storagealembicupgradeheadAzure Blob StorageERT Storage supports Azure Blob Storage for storing opaque data. This feature is invisible to the user. Install theazureextras withpip install ert-storage[azure].Set theERT_STORAGE_AZURE_CONNECTION_STRINGenvironment variable to the connection string found in
your Azure configuration to have ERT Storage use Azure Blob Storage.Local instance for developmentIt is possible to useAzurite, an open source Azure Storage implementation, for testing and development purposes.As a container:# Podmanpodmanrun--nameert-storage-az-p10000:10000-p10001:10001-dmcr.microsoft.com/azure-storage/azurite# Let ERT Storage know about AzuriteexportERT_STORAGE_AZURE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;"DevelopmentFor development, install thetestextraspip install ert-storage[test],
which installsblack,pytestandmypy. Run tests usingpytest.All integration tests must be able to pass when using a non-empty production
database. In practice that means that each test needs to create a unique
"experiment".During test debugging it might be helpful to have the tests persist data in the
database. The following environment variable tells ERT Storage not to rollback the data after each test.exportERT_STORAGE_NO_ROLLBACK=1