Merge branch 'hq' of ../h3hq
project layout change to accommodate hq subproject. moved existing tools code to tools sub-project, and added a new aggregate project crawltools
let PetaboxFileSystem reopen prematurelly closed stream. removed MetaManager-based item lookup code into a separate class (historical record keeping purposes only)
catch ConnectionClosedException due to servers-side closure and perform resume.
fix for over-crawling issue related to robots.txt redirect.
added OnlineCrawlMapperTest (required update to dependency: JUnit 3->JUnit 4)
catching IOException from reading search engine response (ex. SocketTimeoutException).
touch up for robustness of item search. "m" value can be null for not-yet-published/being-derived items.
PetaboxFS: made retryDelay confiurable.
disabled logging stacktrace upon metadata API read timeout.
added FromJsonFunc, opposite of ToJsonFunc.
updated CollectionIndexItemSearcher.seviceUri default so that it uses archive.org/~kenji/getitems.php.
ignore-missing-items for ignoring items found in item search result but in fact non-existent (item db out of sync, or metadata API has got confused).
robustness against pathological cases.
fixed a bug found during test with "hadoop fs"
* item/file timestamp is in seconds
* getFileStatus() assumes Path is always relative for depth==2 case.
added test case for PetaboxFielSystem.
fixed: CollectionIndexItemSearcher returns mtime in seconds.
reflect solo status to replication number.
print Path passed to getFileStatus() for easier diagnosis.
added PetaboxCredentialProvider interface and impl, being in use but not added to repo.
deleted classes not moved to ia-hadoop-tools
moved main and test into src dir to match final structure
imported hadoop-related classes from crawltools (kenji's private repo).