Hadoop InputFormats for ARC and WARC files + simple demos.
Tweaked browser; added MR programs for simple content analysis.
Improved error checking for dates.
Extract links and Lucene FST for URLs.
Simple MapReduce program to count number of unique URLs.
modify UriMappingBuilder to read all files under given directory
check text/html type and modify Jsoup.parse charset as ISO-8859-1
build hadoop job using Maven Assembly Plugin, modified pom.xml, added hadoop-job.xml
modify UriMappingBuilder in pom.xml
new hbase structure for ingest files
servlet updated with the new hbase structure.
Update README.md
Update README.md
Merge branch 'extract-links' of https://github.com/Jeffyrao/warcbase into working
Whitespace.
Tweaked POM
More POM tweaks to get MR jobs running correctly.
Update README.
Merge branch 'working'
Merge branch 'new_hbase_structure' of https://github.com/milad621/warcbase into new_hbase_schema