Add parameter to make AsyncWalker recursing partition directory configurable

Found different behavior between Hive and Presto:
if we have:

/table/partition/file1
/table/partition/directory/file2

and the hive metastore points partition to /table/partition

Hive will process only /table/partition/file1

While, Presto will recursively processing all files in any subdirectories, it will process both file1 and file2

This is to make AsyncWalker configurable on whether recursing partition directory or not.