Repository: rucio/rucio

Issue:
Change m-VO tests to use postgres
Motivation
----------
PostgreSQL is a more widely used DB dialect than the currently used SQLite. It should still be sufficient to only test m-VO explicitly against one of the supported DB dialects however.


Modification
------------
Alter the `.cfg`s used in the test suite to use PostgreSQL, and rename files appropriately.




Discussion:
I think SQLAlchemy does a pretty good job of abstracting the actual DB away and since SQLite is somewhat faster to setup and less error prone on its own, because it is much simpler, I'd suggest to think about it again. What is the gain from this?
There is some peculiarities with the different databases, e.g. in handling constraints, compound indices, partitioning etc. which sqlalchemy does not entirely cover. It's better to test multi-VO with a DB which is actually being used by clients in production. Postgres is a good choice for that. I don't think we need to test all the other ones (like we do for the single-vo mode), but running the multi-vo tests on postgres has just a bigger potential to uncover an issue than with sqlite.
I'm not knowledgeable on the pros/cons of the different DBs myself (initially I chose SQLite because the client suite used it so it seemed sensible to pick that one). Martin brought up the points above at a m-VO development meeting on the 23rd of June, so I made a note to make the change from SQLite to PostgreSQL. The change wasn't included in the main m-VO PR, but the work was done a while ago.
I'm happy to make this change or not make it depending on what people think is best, but personally I don't know enough to contribute to the discussion.

Resolving patch:
diff --git a/lib/rucio/web/rest/webpy/v1/did.py b/lib/rucio/web/rest/webpy/v1/did.py
--- a/lib/rucio/web/rest/webpy/v1/did.py
+++ b/lib/rucio/web/rest/webpy/v1/did.py
@@ -732,7 +732,7 @@ def DELETE(self, scope, name):
                 raise generate_http_error(404, 'KeyNotFound', 'No key provided to remove')
 
         try:
-            delete_metadata(scope=scope, name=name, key=key)
+            delete_metadata(scope=scope, name=name, key=key, vo=ctx.env.get('vo'))
         except KeyNotFound as error:
             raise generate_http_error(404, 'KeyNotFound', error.args[0])
         except DataIdentifierNotFound as error:


Test patch:
diff --git a/etc/docker/test/extra/rucio_multi_vo_ts2_sqlite.cfg b/etc/docker/test/extra/rucio_multi_vo_ts2_postgres12.cfg
similarity index 97%
rename from etc/docker/test/extra/rucio_multi_vo_ts2_sqlite.cfg
rename to etc/docker/test/extra/rucio_multi_vo_ts2_postgres12.cfg
--- a/etc/docker/test/extra/rucio_multi_vo_ts2_sqlite.cfg
+++ b/etc/docker/test/extra/rucio_multi_vo_ts2_postgres12.cfg
@@ -19,10 +19,13 @@ request_retries = 3
 vo = ts2
 
 [database]
-default = sqlite:////tmp/rucio.db
+default = postgresql://postgres:secret@postgres12/postgres
+schema = dev
 pool_recycle=3600
 echo=0
 pool_reset_on_return=rollback
+pool_size = 20
+max_overflow = 20
 
 [bootstrap]
 # Hardcoded salt = 0, String = secret, Python: hashlib.sha256("0secret").hexdigest()
diff --git a/etc/docker/test/extra/rucio_multi_vo_tst_sqlite.cfg b/etc/docker/test/extra/rucio_multi_vo_tst_postgres12.cfg
similarity index 97%
rename from etc/docker/test/extra/rucio_multi_vo_tst_sqlite.cfg
rename to etc/docker/test/extra/rucio_multi_vo_tst_postgres12.cfg
--- a/etc/docker/test/extra/rucio_multi_vo_tst_sqlite.cfg
+++ b/etc/docker/test/extra/rucio_multi_vo_tst_postgres12.cfg
@@ -19,10 +19,13 @@ request_retries = 3
 vo = tst
 
 [database]
-default = sqlite:////tmp/rucio.db
+default = postgresql://postgres:secret@postgres12/postgres
+schema = dev
 pool_recycle=3600
 echo=0
 pool_reset_on_return=rollback
+pool_size = 20
+max_overflow = 20
 
 [bootstrap]
 # Hardcoded salt = 0, String = secret, Python: hashlib.sha256("0secret").hexdigest()
diff --git a/etc/docker/test/matrix.yml b/etc/docker/test/matrix.yml
--- a/etc/docker/test/matrix.yml
+++ b/etc/docker/test/matrix.yml
@@ -22,4 +22,4 @@ suites:
   - id: client
     RDBMS: sqlite
   - id: multi_vo
-    RDBMS: sqlite
+    RDBMS: postgres12
diff --git a/tools/run_multi_vo_tests_docker.sh b/tools/run_multi_vo_tests_docker.sh
--- a/tools/run_multi_vo_tests_docker.sh
+++ b/tools/run_multi_vo_tests_docker.sh
@@ -40,8 +40,7 @@ do
     r) activate_rse="true";;
   esac
 done
-
-cp /opt/rucio/etc/rucio_multi_vo_tst.cfg /opt/rucio/etc/rucio.cfg
+export RUCIO_HOME=/opt/rucio/etc/multi_vo/tst
 
 echo 'Clearing memcache'
 echo flush_all > /dev/tcp/127.0.0.1/11211
@@ -130,7 +129,7 @@ if [ $? != 0 ]; then
 fi
 
 echo 'Tests on first VO successful, preparing second VO'
-cp /opt/rucio/etc/rucio_multi_vo_ts2.cfg /opt/rucio/etc/rucio.cfg
+export RUCIO_HOME=/opt/rucio/etc/multi_vo/ts2
 
 echo 'Clearing memcache'
 echo flush_all > /dev/tcp/127.0.0.1/11211
diff --git a/tools/test/before_script.sh b/tools/test/before_script.sh
--- a/tools/test/before_script.sh
+++ b/tools/test/before_script.sh
@@ -134,17 +134,18 @@ elif [[ $RDBMS == "postgres12" ]]; then
         echo Could not connect to Postgres in time.
         exit 1
     fi
+    if [[ $SUITE == "multi_vo" ]]; then
+        docker exec rucio mkdir -p /opt/rucio/etc/multi_vo/tst/etc
+        docker exec rucio mkdir -p /opt/rucio/etc/multi_vo/ts2/etc
+        docker exec rucio cp /usr/local/src/rucio/etc/docker/test/extra/rucio_multi_vo_tst_postgres12.cfg /opt/rucio/etc/multi_vo/tst/etc/rucio.cfg
+        docker exec rucio cp /usr/local/src/rucio/etc/docker/test/extra/rucio_multi_vo_ts2_postgres12.cfg /opt/rucio/etc/multi_vo/ts2/etc/rucio.cfg
+    fi
     docker exec rucio cp /usr/local/src/rucio/etc/docker/test/extra/rucio_postgres12.cfg /opt/rucio/etc/rucio.cfg
     docker exec rucio cp /usr/local/src/rucio/etc/docker/test/extra/alembic_postgres12.ini /opt/rucio/etc/alembic.ini
     docker exec rucio httpd -k restart
 
 elif [[ $RDBMS == "sqlite" ]]; then
-    if [[ $SUITE == "multi_vo" ]]; then
-        docker exec rucio cp /usr/local/src/rucio/etc/docker/test/extra/rucio_multi_vo_tst_sqlite.cfg /opt/rucio/etc/rucio_multi_vo_tst.cfg
-        docker exec rucio cp /usr/local/src/rucio/etc/docker/test/extra/rucio_multi_vo_ts2_sqlite.cfg /opt/rucio/etc/rucio_multi_vo_ts2.cfg
-    else
-        docker exec rucio cp /usr/local/src/rucio/etc/docker/test/extra/rucio_sqlite.cfg /opt/rucio/etc/rucio.cfg
-    fi
+    docker exec rucio cp /usr/local/src/rucio/etc/docker/test/extra/rucio_sqlite.cfg /opt/rucio/etc/rucio.cfg
     docker exec rucio cp /usr/local/src/rucio/etc/docker/test/extra/alembic_sqlite.ini /opt/rucio/etc/alembic.ini
     docker exec rucio httpd -k restart
 fi