Project: datamanagement
Issue DM-516: Fix race condition when creating db (and elsewhere?) in client/qserv_admin_impl.py
This came up in the review of CSS, see DM-225: QservAdminImpl: so, if there's a problem while inserting the various keys for a database, and another exception during cleanup, you can end up with a partially constructed database. How do you plan on handling this? This is perhaps another argument for trying to batch metadata more - if it's all in a single key, this worry goes away (at least in this particular case). The createDb implementations seem racy. Consider what happens if 2 admins try to create the same database D. Suppose admin A does the _dbExists(D) check first; it returns false. Then admin B comes along, and does the same check, which returns false again. Let's further suppose that admin B successfully issues all the key creates for D without being interrupted by A. Back to A: when A tries to create all the keys required for D, an exception will be raised immediately since D exists. At this point A will dutifully try to clean up (to avoid leaving around a partially constructed D), deleting D as created by B. End result: D was successfully created, no explicit drop was issued, but D does not exist. Database drops can also race with table creates: zookeeper doesn't have a native recursive delete. In other words, it is perfectly possible for an admin A to issue a database drop while someone else is issuing a table create. The recursive delete might descend into the table "directory" while the create is populating it with child znodes. The recursive delete will get an incomplete list of children, remove them, and finally fail to remove the table directory because the table create has since added more children, and it is illegal to remove a non-empty directory. At this stage, the drop will report success even though it has failed (that's the recursive delete behavior that kazoo gives you), and the table create will report success. But the database will still exist, and table keys may be partially present in zookeeper.
Story points: 10