Updating crawler to handle simple rate limiting (without explicit module changing)