Added crawlable/non-crawlable status for every file.
Made URL generation produce better URLs.