Computation of robots.txt block and redirect percentage
(backend/ingest functionality)