Allow you to crawl without a file size limit, and allow these to be configurable (depending on the types of files your crawling).

I also added two small javadoc comments to describe these changes.