Handle flagging binary files properly

Flagging offending paths with `file_match` blows up for binary files.