Revert "considers robots.txt and normalizing all urls"

This reverts commit d643731cd3d7d445af56d5694e94d2d5f2cec9c5.
first test file, added overview to README