detected many problems after implementing my own crawler (for establishing the cache)