fspoll: don't compare directory times and sizes.

Fixes contant rebuilding on Linux due to modifying out/ directory in the
root directory, which then has its ModTime updated.