Package: opal-fetcher-postgres
OPAL Fetcher for PostgresCheck out OPAL main repo here.What's in this repo?An OPALcustom fetch providerto bring authorization state fromPostgres.This fetcher is both:A fully functional fetch-provider for Postgres:can be used by OPAL to fetch data from Postgres DB.Serving as an examplehow to write custom fetch providers for OPAL and how to publish them as pip packages.How to try this custom fetcher in one command? (Example docker-compose configuration)You can test this fetcher with the example docker compose file in this repository root. Clone this repo,cdinto the cloned repo, and then run:docker compose upthis docker compose configuration already correctly configures OPAL to load the Postgres Fetch Provider, and correctly configuresOPAL_DATA_CONFIG_SOURCESto include an entry that uses this fetcher.✏️ How to use this fetcher in your OPAL Setup1) Build a custom opal-clientDockerfileThe official docker image only contains the built-in fetch providers. You need to create your ownDockerfile(that is based on the official docker image), that includes this fetcher's pip package.YourDockerfileshould look like this:FROM permitio/opal-client:latest
RUN pip install --no-cache-dir --user opal-fetcher-postgres2) Build your custom opal-client containerSay your special Dockerfile from step one is calledcustom_client.Dockerfile.You must build a customized OPAL container from this Dockerfile, like so:docker build -t yourcompany/opal-client -f custom_client.Dockerfile .3) When running OPAL, setOPAL_FETCH_PROVIDER_MODULESPass the following environment variable to the OPAL client docker container (comma-separated provider modules):OPAL_FETCH_PROVIDER_MODULES=opal_common.fetcher.providers,opal_fetcher_postgres.providerNotice that OPAL receives a list from where to search for fetch providers.
The list in our case includes the built-in providers (opal_common.fetcher.providers) and our custom postgres provider.4) Using the custom provider in your DataSourceEntry objectsYour DataSourceEntry objects (either inOPAL_DATA_CONFIG_SOURCESor in dynamic updates sent via the OPAL publish API) can now include this fetcher's config.Example value ofOPAL_DATA_CONFIG_SOURCES(formatted nicely, but in env var you should pack this to one-line and no-spaces):{"config":{"entries":[{"url":"postgresql://postgres@example_db:5432/postgres","config":{"fetcher":"PostgresFetchProvider","query":"SELECT * from city;","connection_params":{"password":"postgres"}},"topics":["policy_data"],"dst_path":"cities"}]}}Notice howconfigis an instance ofPostgresFetcherConfig(code is inopal_fetcher_postgres/provider.py).Values for this fetcher config:Theurlis actually a postgres dsn. You can set the postgres password in the dsn itself if you want.connection_paramsare optional, if you want to include certain overrides outside the dsn.Yourconfigmust include thefetcherkey to indicate to OPAL that you use a custom fetcher.Yourconfigmust include thequerykey to indicate what query to run against postgres.🚩 Possible User IssuesWhile trying to send requests to a Postgres data source, you may encounter that the request fails. This can be caused by the format of the config entry URL for which the standard is:postgresql://<user>:<password>@<host>/<db>It might be most common that this request fails due to the password field being incorrectly parsed by the underlying library calledasyncpg, which is one of the required libraries used within our OPAL custom data fetcher.In order to solve the issue, you need to change the data source config entry URL to the format shown below:postgresql://<host>/<db>?user=<user>&password=<password>📖 About OPAL (Open Policy Administration Layer)OPALis an administration layer for Open Policy Agent (OPA), detecting changes to both policy and policy data in realtime and pushing live updates to your agents.OPAL brings open-policy up to the speed needed by live applications. As your application state changes (whether it's via your APIs, DBs, git, S3 or 3rd-party SaaS services), OPAL will make sure your services are always in sync with the authorization data and policy they need (and only those they need).Check out OPAL's main site atOPAL.ac.