Initial commit
First commit with dependencies and class files
Play application with first version of scrapper.
DOD:
1. MongoLab account tied to application
2. Heroku App launched
3. App linked through GIT
4. Crawler first version with HTMLunit built.
5. Configured to go through top 500 sites to get top .sg web domains.
Mongo through Jackson Mapper as well as the first version of Scraper modifications.
Procfile changes to prevent heroku crash.
Procfile corrected with app name.
Remodified procfile.
Memory exceeded error without java_opts.
Modified procfile again.
Remove JAVA_OPTS, instead created java_opts from heroku env variables.
Forcing JVM 7 through
a. build.sbt as target and source.
b. system.properties.
working in progress with craw4j
testing crawler
testing craw4j
testing craw4j
Delete .classpath
Delete .project
remove eclipse settings
using the crawlControleler
correct conf