twitter/scalding — pull request #772 by isnotinvain (2014-02-04 07:48:01+00:00)
Title: Use hidden path filter in pathIsGood, add tests for pathIsGood

This is to address issue #764

Unfortunately, it looks like a bigger issue to make pathIsGood exhaustive in checking for validity of paths.

So this PR just addresses the one corner case of handing directories with only hidden files to FileInputFormat which doesn't like that.

There were no tests for pathIsGood, so I've added some.

Before applying the changes to FileSource.scala, here's what the new tests output:
Before:

[error] x default pathIsGood should
[info] + accept a directory with data in it
[info] + accept a directory with data and _SUCCESS in it
[info] + reject an empty directory
[error] x reject a directory with only _SUCCESS when specified as a glob
[error] 'true' is not equal to 'false' (FileSourceTest.scala:116)
[info] + accept a directory with only _SUCCESS when specified without a glob
[error] x success file source pathIsGood should
[info] + reject a directory with data in it but not _SUCCESS file
[info] + accept a directory with data and _SUCCESS in it when specified as a glob
[info] + reject a directory with data and _SUCCESS in it when specified without a glob
[info] + reject an empty directory
[error] x reject a directory with only _SUCCESS when specified as a glob
[error] 'true' is not equal to 'false' (FileSourceTest.scala:146)
[info] + reject a directory with only _SUCCESS when specified without a glob

And after applying the fixes to FileSource.scala, they all pass. So I left most of the behavior of pathIsGood in-tact, with the exception of the two failed tests above.