added url normalisation, extended tests, added godocs to crawler package