Stop encoder in file only mode when the output file cannot be opened

This may or may not break some other combination of hotkey/button/invalid
setting, we shall see