Removing the files[] entries from the .info file makes the conditions plugins stop appearing as an option. (Read: the whole module stops doing what it's supposed to.)
The reason the conditions files needed to be in the .info file was because of a typo in the file path used in context_flag_context_plugins().
Recommendation from joachim: Only check flag_get_flags if this is already shown to be a user page.