Handle unexpected errors better

This lets us avoid stats in all cases, and just infer directory-ness
based on the errors that we see.
loosen graceful-fs optional dep