removed catch statements
filterclass should now correctly parse robots files
handling of absolute links
added link with less cluttered robots.txt file for debugging
rearrangement of functioncalls in main method
correct mergemethod for merging to linked lists
some renaming
changed link because of problems with XMLLoader on the stackoverflow pages
implementation of normalized tf-idf¨
creating a query and getting a ranked list of results
more general Tokenize method
Usage of newly created methods in the crawler