Only check is_dot() on directories.

While I don't notice a performance increase, gprof claims we spent 30%
of out time there, and now it's zero.