Package: i2b2rls
i2b2rlsThis package provides a cli and python api to manage users and projects
in the dockerizedi2b2clinical data warehousing platform
using PostgreSQL Row Level Security (RLS) templates.The key functionality is registering a single database schema
as multiple Data Repositories (CRC) in i2b2.
Each CRC connects to the same database as a different user with its own RLS policies defined.
This enables the usage of a single datasource for multiple projects to
circumvent the costly and complex replication of data into project specific databases,
by utilising PostgreSQL's Row Level Security feature.In the examples each project connects with a role that has RLS policies enabled,
which allows access only to patients with theprovider_idset in the project configuration.
The required data is defined in and derived fromi2b2.yaml, the central config file.The corresponding docker-compose based project is thei2b2-stackGetting StartedPrerequisitesDependencies# On Ubuntu / Debiansudoaptinstall-ypython3python3-pipgccpostgresqlpostgresql-server-dev-all# On Manjaro / Archsudopacman-Spython3python-pipgccpostgresqlpostgresql-libslibffiInstall repository# pip install from pypipython3-mpipinstalli2b2rls# pip install from gitlab repositorypython-mpipinstallgit+https://gitlab.com/mds-imbi-freiburg/i2b2/i2b2rls.gitUsageCreate a folder and initiate the i2b2 project dir:mkdiri2b2-demo&&cdi2b2-demo/
i2b2rls--conf=.--initCreate a new project ini2b2.yaml:subdepartment1:# Name of projectcrc:# Register a new CRCds-user:dep1# Create role in database & datasourceds-pass:demouser# Password for databse roleprovider-filter:LCS-I2B2:D000109075%# Filter for observation_facts providermembers:user1:Register the project:i2b2rlsprojectregistersubdepartment1This creates all required resources for an i2b2 project:i2b2Projectsubdepartment1i2b2Frontend Useruser1i2b2Project Rolesforuser1(based on default-project-roles ini2b2.yaml)PostgresDatabase Userdep1with passworddemouser(based onexample sql files)WildflyDatasourceusingdep1user to connect to the Database (based ontemplate datasource)NOTE:You may need to restart the stack for all settings to apply.Manual stepsTo manually run the steps for a project one can use the API from the command line:# Run SQL scripts in user folderi2b2rlsprojectrun_rlssubdepartment1--script=main.sql# Generate and deploy datasource to Wildfly containeri2b2rlsprojectgenerate_dssubdepartment1deploy=True# Register project in PM celli2b2rlsprojectadd_pm_project_datasubdepartment1# Register project members and their rolesi2b2rlsprojectregister_memberssubdepartment1# Add Project roles for Service user (AGG_SERVICE_ACCOUNT)i2b2rlsprojectadd_service_rolessubdepartment1# Register Cells (crc, ont, im, work) for new Projecti2b2rlsprojectcreate_db_lookupssubdepartment1HelpCLI and API are built with the awesomeFirelibrary.
It provides a help page for every possible command:i2b2rls---helpFor example:$i2b2rlsprojectregister--help
>NAME
>i2b2rlsprojectregister-CreateallresourcesforaRLSproject
>
>SYNOPSIS
>i2b2rlsprojectregisterPROJECT_ID<flags>
>
>DESCRIPTION
>Steps:
>*(Optional)CreateSQLscriptfromtemplate
>*RunSQLfilesinrolefolder
>*Create&deployDatasourcefileforCRCcell
>*Createtheprojectini2b2pm.pm_project_data
>*Createmandatoryi2b2-rolesforAGG_SERVICE_ACCOUNT
>*Registerprojectdb_lookupsini2b2hive.{im,ont,work,crc}_db_lookup
>
>POSITIONALARGUMENTS
>PROJECT_ID
>Type:str
>IDofprojectasini2b2.yaml
>
>FLAGS
>--cell_id=CELL_ID
>Type:str
>Default:'crc'>Thecelldatatodeploy(Default:'crc')>--force=FORCE
>Type:bool
>Default:False
>DeleteExistingresourcesandrecreatethem
>--ignore_errors=IGNORE_ERRORS
>Type:bool
>Default:False
>Ignoreerrorsandcontinuecreatingresources
>--add_i2b2=ADD_I2B2
>Type:bool
>Default:True
>Add'i2b2'usertoprojectwithdefaultroles
>--deploy_ds=DEPLOY_DS
>Type:bool
>Default:True
>CopyDatasourcefiletowildflyserverandreload
>--extract_ds=EXTRACT_DS
>Type:bool
>Default:True
>PullDatasourcefilefromwildflyservertolocalfolderConfigurationFolder layoutThe init command creates the template files for an i2b2rls instance structure:i2b2-demo/
├──i2b2.yaml# Central project definition and configuration├──template-ds.xml# Template for XML Datasource Objects├──datasources/# Temporary folder for wildfly datasources├──roles/# Each RLS user has a folder here│├──template# Template folder other roles inherit from││└──main.sql# Main file, write RLS struff or reference other files│├──example# demouser example RLS files││└──main.sql# Main file, includes other sql files with \ir││├──drop.sql# Script for dropping all sql resources││├──test.sql# Contains PgTAP tests└──└──└──rls.sql# Creates roles and RLS policiesStart by editing the filei2b2.yamland change the basic settings for your i2b2 instance in thedefaultsection.
These settings will be inherited by any further project you define and can be overwritten by setting the value explicitly.Generallog-level:DEBUG# Application logging level: INFO, WARN, ERROR, DEBUGlog-file:/tmp/i2b2rls.log# Log to file - Off by defaultchangeby-user:i2b2rls# Will appear in database entries at changeby_userwildfly-container:i2b2-wildfly# Name of wildfly docker containerwildfly-datasources:/opt/jboss/wildfly/standalone/deployments/# Path to datasources in containerConfiguration LevelsAny configuration option fromi2b2.yamlhas sane defaults configured inconf.pyto ensure every possible configuration option has a value. This is thecode-levelconfig.
The values there are recursively overwritten by settings you make ini2b2.yaml.Ini2b2.yamlis a default configuration tree for projects, thedefaultproject.
Every option there is inherited by other projects, and can be overwritten by them.
The hierarchy of overriding values for projects looks like this:code-level < default-project-level < project-levelFor the deeper nested LDAP-settings of a user:code-level < default-project-level < project-level < user-levelDefault ProjectAs mentioned above the default project holds configurations that apply to all projects,
and can be overridden by settings them explicitly on a project.domain:i2b2demo# i2b2 domain (c_domain_id in db_lookups)owner:"@"# Owner (c_owner_id in db_lookups)default-project-roles:[USER,EDITOR,...]# Default roles each user in each project gets, if not overriddencrc:host:other-pg-hostCell ConfigurationsEach project needs access to the core modules (cells) of i2b2.
For that, each project's cells are registered in the correspondingdb_lookuptable in thei2b2hiveschema.Each Cell type (except Project Management) has its own db_lookup table:Ontology: ont_db_lookupWorkplace: work_db_lookupIdentity Management (IM): im_db_lookupData Repository (CRC): crc_db_lookupProject Management (PM): Tables in schemai2b2pmAs these can be hosted on various postgres servers, the connection details from
the wildfly server need to be configured in the datasources.If the Datasource already exists in wildfly,ds-userandds-passcan be omitted.
If the Datasource should be created,admin-userandadmin-passmust be set,
so the application can connect to the postgres instance and create the db_lookup entry.ont:host:i2b2-pg# Postgres serverport:5432database:i2b2schema:i2b2metadatapool-name:OntologyDemoDSjndi-name:java:/OntologyDemoDSds-user:i2b2metadatads-pass:demouserUsers, Roles and MembersThere are several types of roles, users and groups involved, so here is an overview:Postgres RolesPostgres does not differentiate between users and groups, both are summarized in the conceptrole.
This way a user can inherit privileges from any other user and group, or vice versa.
The only difference is that some roles have theLOGINflag and a password set,
which technically makes them users.In context ofi2b2rlsthere are the config objectsds-userandadmin-user,
each referring to arolein the Postgres database:ds-user: Role that is created in the target Data Repository (Postgres server) by the application.
It enables the overlying project to connect to the CRC through a Wildfly Datasource, limited by your RLS rules.admin-user: Used to connect to the Postgres server to create the Datasource user above.
Must be highly privileged to create resources like roles, policies and grants.i2b2 UsersUsers of the i2b2 frontend are registered in the tablepm_user_dataof the Project Management cell.
They are identified by theiruser_id(username) and may have a password, full name, email address and status (active/disabled).
Ini2b2.yamlusers of a project are defined in thememberssection:your_project:members:rls:# Usernamepassword:demouser# Login passwordfullname:RLS Demo User# Users full nameemail:demouser@example.com# User email addressproject-roles:[USER,DATA_OBFSC,MANAGER]If a user already exists in the database it can simply be referred by the user_id as a key without values:other_project:members:rls:demo:i2b2:Project RolesTo login to a project a user needsProject Roles, defined by the keyproject-roles.
Roles are inherited by a project'sdefault-project-rolesif a user has no explicit project-roles set.
The commandì2b2rls project register_members yourproject user_namecollects these settings and creates
the database resources inpm_user_dataandpm_project_user_roles.User LDAPTo enable LDAP authentication for a project, make sure you have the correct ldap settings in your Postgrespg_hba.confand the ldap object in youri2b2.yamlpopulated with corresponding values.Just like any setting LDAP can be configured for all projects in the default project
and overridden by your specific project.
The project-level settings can then be overridden by user-level settings:projects:your_project:ldap:# LDAP Settings used in i2b2pm.pm_user_paramsauthentication-method:LDAP# LDAP / LDAPSconnection-url:ldap.mycompany.com# LDAP Serverdistinguished-name:uid=# Field to user for loginsearch-base:ou=people,dc=mycompany,dc=com# Where to find userssecurity-authentication:simple# Auth mechanism (none/simple/sasl)members:rls:ldap:# Override project-level search-basesearch-base:ou=department,ou=people,dc=example,dc=orgTo enable the LDAP login for a useron project registrationset at least an emptyldap:key.To enable LDAP login manually use:i2b2rls project set_user_ldap rlsdemo rlsTo manually register LDAP for a user while setting a certain value explicitly:i2b2rlsprojectset_user_ldaprlsdemorls--data='{"search-base":""}'SQL TemplatesThe example folder features a set of scripts that split the creation of Postgres Roles and its RLS policies into multiple steps:main.sql:
Incorporates the .sql files into a single transaction. A SAVEPOINT is created before the tests and rolled back to after the tests.
If any of the tests fail, the whole transaction is rolled back.drop.sql:
Used to remove any (preexisting) objects in the database that should be created by the application.
This should be called before the Role and the Policies are created, and can be called if the Role is to be removed.init.sql:
Initialisation script that ensures the extensionpgtapexists and the viewv_observation_factsis present and contains the required fields.rls.sql:
Contains the statements for creating the Role and its privileges:CREATE ROLE, GRANT, CREATE POLICY.test.sql:
PgTAP tests that validate the role is created with necessary privileges and the RLS policies filter correctly.The SQL scripts are called with thepsqlCommandline client for postgres.
This enables the usage of variables in the scripts that are passed by the application (e.g. :variable_name).
By default the following variables are passed to the client and can be used in scripts:project_id:Project id, the object name of a projectrole_name:Name of postgres role defined inds-userrole_pass:Password of postgres role, defined inds-passdb:Database name of datasource defined incrc > database(default i2b2)cell_schema:Schema of i2b2 data defined incrc > schema(default i2b2demodata)Anykey: valuepair inpg-vars. The keysmustonly contain lower dashes as seperator!
Also, any of the above values can be overwritten by setting it inpg-vars!A full call from the application to psql may look like this:psqlpostgresql://i2b2:demouser@i2b2-pg:5432/i2b2\-vproject_id=rlsdemo\-vrole_name=rlsuser\-vrole_pass=rlsuserpass\-vdb=i2b2\-vcell_schema=i2b2demodata\-vprovider_filter=LCS-I2B2:D000109061%\-f"/path/to/config/roles/example/main.sql"NOTE:The viewv_observation_factis required for the RLS policies.
It joins the clinic's departments tableprovider_dimensionto the patients observationsobservation_factby theprovider_id.RessourcesTo sum up, following resources are managed by the tool:i2b2 Project and User Management (i2b2pm)Frontend users (pm_user_data)Frontent user LDAP settings (pm_user_params)Project roles (pm_project_user_roles)Project data (pm_project_data)i2b2 CellsDatasource's manipulation and deployment to WildflyCell db_lookups in i2b2hivePostgreSQL resourcesExecutor for SQL templates viapsqlclientInstantiation of custom SQL templatesDefinition of postgres database rolesSimple CLIThe cli features a set of simplified commands for management of users and groups.
These commands are accessed through the subcommandi2b2rls cmd [user|group]and will return 1 on error or 0 on success.
These commands are designed for use in CI processes.Docker.Asi2b2rlsis built to work with a dockerized setup it needs access to the docker daemon.
For this work, the image is built on top ofdocker-in-dockerand requires
the docker socket as a volume:dockerrun\-v/tmp/data:/data\-v/var/run/docker.sock:/var/run/docker.sock\--networki2b2-stack_i2b2-net\-tii2b2rls\i2b2rls--init--conf=/dataDevelopmentpipinstall-rrequirements.dev.txt
pre-commitinstall
pre-commitinstall--hook-typecommit-msgTestStart the i2b2 postgres backend:dockerrun-p5432:5432i2b2/i2b2-pg:p1Run tests:python-mpytest--cov=.To simulate the hostname of the postgres server on your machine for testing it in/etc/hosts:sudoecho"127.0.0.1    i2b2-pg">>/etc/hostsDebuggingEnable logging in Wildflydockerexec-it-urooti2b2-wildflybash-c"sed -i 's/INFO/DEBUG/g' /opt/jboss/wildfly/standalone/configuration/standalone.xml"Logging in Postgres backendclient_min_messages = debug1
log_line_prefix = '%m [%p] %q%u@%d '
log_statement = 'all'How it worksFirst the application parses the i2b2.yaml file and creates a full configuration tree for every project.GlossaryShortLongExplanationRLSRow Level SecurityPostgresSQL access policies based on row values. Allows granular filtering of user access to patient/observation informationCRCData RepositoryDatabase schema and REST API (cell) for i2b2 medical data, e.g. the tablesobservation_facts, patient_dimension etc.PMProject ManagementDatabase schema (cell) for i2b2 project data: users, projects, project_roles  etc.ONTOntologyDatabase schema and REST API (cell) for i2b2 ontologies. Used to query patient data by diagnosisDSDatasourcesXML object (e.g. crc-ds.xml) that defines a database connection for the i2b2 wildfly server