dir.c: encoding check

* dir.c (file_s_fnmatch, fnmatch_brace): encoding-incompatible pattern
  and string do not match, instead of exception.  [ruby-dev:47069]
  [Bug #7911]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@39413 b2dd03c8-39d4-4d8f-98ff-823fe69b080e