reorganize test after init vs open/close operating modes
check if sqlfs is open before trying to rekey database

Technically I think it is possible to sqlite3_rekey() on an open and active
database.  But for now, only run it when the user is not using it at all.

refs #3174 https://dev.guardianproject.info/issues/3174