Hadoop InputFormats for ARC and WARC files + simple demos.
Tweaked browser; added MR programs for simple content analysis.
Improved error checking for dates.
started a new branch to extract text from warcbase data and organize the data to run topic modeling
Started a new branch to extract text from warcbase data and organize the data to run topic modeling.
removed dead code
Extract links and Lucene FST for URLs.
Simple MapReduce program to count number of unique URLs.
modify UriMappingBuilder to read all files under given directory
check text/html type and modify Jsoup.parse charset as ISO-8859-1
build hadoop job using Maven Assembly Plugin, modified pom.xml, added hadoop-job.xml
modify UriMappingBuilder in pom.xml
new hbase structure for ingest files
servlet updated with the new hbase structure.
Update README.md
Update README.md
Merge branch 'extract-links' of https://github.com/Jeffyrao/warcbase into working
Whitespace.
Tweaked POM
More POM tweaks to get MR jobs running correctly.