Bugfix logic around tika.convert.markup
Merge pull request #39 from mumrah/master

Fix field mappings for Solr and new unit test
Applied format to SOLRWriter and its tester + access private fieldmapping via get method
Issue #38 Warn when input is not available for CorpusGenerator
Tika : can force detection of Mime type with option tika.forceMimetypeDetect=true
Improvements to LanguageID : conditional reducer for filtering + static processor for JVM reuse
Fixed tests for DocFilter
Tika process returns -1 if exception + LanguageId to work on non-default HDFS
LanguageID filters in map and does not use reducer
Added document.filter.md.mode which can be AND or OR and determines whether all the constraints must match
Tika : catch CountersExceededExceptions
ContentExtractor uses Path and relies on Hadoop serialisation so that can write output straight to S3 or HDFS as well as locally
ContentExtractor generates index file + specify naming node via param
Content extractor puts files in archives
Flush index file after each archive
Update script.sh
Mahout : upgrade to 0.7 + catch runtime exceptions
Merge branch 'master' of github.com:DigitalPebble/behemoth
Mahout : added ClusterDocIDDumper
SparseVectorsFromBehemoth : specifiation of Lucene analyzer is optional