Don't show "error opening file" message if the file is currently open in a custom viewer.

This fixes issue #5718. On Windows, this also prevents the file from being open as gibberish text.