Initial directory structure.

git-svn-id: http://ldspider.googlecode.com/svn/trunk@1 e43db140-ba3d-11de-8be4-e1425cda3908
initialising of the repository. Checked in version 0.1 of the ldspider

git-svn-id: http://ldspider.googlecode.com/svn/trunk@2 e43db140-ba3d-11de-8be4-e1425cda3908
link extraction hook. (dis)allow certain links to extract new URIs. 
This link filter is not included in the Main method so far. Needs to add new cli option to enable it

git-svn-id: http://ldspider.googlecode.com/svn/trunk@3 e43db140-ba3d-11de-8be4-e1425cda3908


git-svn-id: http://ldspider.googlecode.com/svn/trunk@4 e43db140-ba3d-11de-8be4-e1425cda3908
some small refactoring, added skeleton query processor based on ARQ

git-svn-id: http://ldspider.googlecode.com/svn/trunk@7 e43db140-ba3d-11de-8be4-e1425cda3908
added second test case

git-svn-id: http://ldspider.googlecode.com/svn/trunk@8 e43db140-ba3d-11de-8be4-e1425cda3908
removed query code

git-svn-id: http://ldspider.googlecode.com/svn/trunk@9 e43db140-ba3d-11de-8be4-e1425cda3908
updated to httpclient 4.0.1, synchronized set in linkfilter, proxy support, moved system.out/err to use logger, added redirects handling, parameter -uri to -u, added -o for output file name

git-svn-id: http://ldspider.googlecode.com/svn/trunk@15 e43db140-ba3d-11de-8be4-e1425cda3908
max_connections set to 1024 gets rid of hangs for now

git-svn-id: http://ldspider.googlecode.com/svn/trunk@17 e43db140-ba3d-11de-8be4-e1425cda3908
connections used up in tld and robots lookup

git-svn-id: http://ldspider.googlecode.com/svn/trunk@18 e43db140-ba3d-11de-8be4-e1425cda3908
max connections dependent on threads, merged lookupmanager with crawler (was essentially duplicate code)

git-svn-id: http://ldspider.googlecode.com/svn/trunk@19 e43db140-ba3d-11de-8be4-e1425cda3908
more comments on queue

git-svn-id: http://ldspider.googlecode.com/svn/trunk@20 e43db140-ba3d-11de-8be4-e1425cda3908
minor cleanups, added max no of uris per pld per round

git-svn-id: http://ldspider.googlecode.com/svn/trunk@21 e43db140-ba3d-11de-8be4-e1425cda3908
minor changes

git-svn-id: http://ldspider.googlecode.com/svn/trunk@23 e43db140-ba3d-11de-8be4-e1425cda3908
refactored FetchQueue

git-svn-id: http://ldspider.googlecode.com/svn/trunk@24 e43db140-ba3d-11de-8be4-e1425cda3908
added header fields output

git-svn-id: http://ldspider.googlecode.com/svn/trunk@25 e43db140-ba3d-11de-8be4-e1425cda3908
now with common.log

git-svn-id: http://ldspider.googlecode.com/svn/trunk@26 e43db140-ba3d-11de-8be4-e1425cda3908
check if redirected uri already in queue

git-svn-id: http://ldspider.googlecode.com/svn/trunk@27 e43db140-ba3d-11de-8be4-e1425cda3908
check if redirected uri already in queue

git-svn-id: http://ldspider.googlecode.com/svn/trunk@28 e43db140-ba3d-11de-8be4-e1425cda3908
logging now in errorhandler

git-svn-id: http://ldspider.googlecode.com/svn/trunk@29 e43db140-ba3d-11de-8be4-e1425cda3908