Added information about the profiles directory.
Added langdetect dependency.
Now takes as input a directory; changed logic to account for language detection.
Removed some debugging steps; made containsHtml more compact; added English detection; method to removed extraneous whitespace.
Profiles directory in the base dir.
Removed tidbit about the profiles directory being in the jar.
Renamed WarcEntry to WarcRecord, added a LOT of comments about the new warc format.
Renaming WarcEntry to WarcRecord.
changed the format of the comments, fields; only mandatory fields now.
Initial commit.
Updated scala, sentencer dependencies; removed configuration.
Pretty much rewrote this file for better scala style and to account for changes in WarcRecordIterator.
Some comments about the new WARC; removed unecessary fields.
Update for WARC 1.0. Huge overhaul, but general logic remains the same. Changed how WARC record headers are processed; instead of assuming static fields, use a map and just take out fields that will always be there.
Added filter for HTTP response headers.
Added model file for sentencer.
Didn't end up using this.
Update for new way to pass input.
Downgrade scala to 2.9.2, no longer need Try.
Added ability to pass input through standard in; moved opening of source logic to the main class.