small modifications in crawler and network requests