reorganize test after init vs open/close operating modes
check if sqlfs is open before trying to rekey database

Technically I think it is possible to sqlite3_rekey() on an open and active
database.  But for now, only run it when the user is not using it at all.

refs #3174 https://dev.guardianproject.info/issues/3174
purge reference to non-existent c_thread_api_password.c
tests: fix not-exist test, sqlfs functions return negative error values

errno is not used at all with sqlfs_proc_* functions, instead the correct
thing is to check the return value.  It is 0 if successful, and negative if
not.  The negative values are negative versions of the standard errors.