Corrected handling of a missing robots.txt (now returns 503 to make the gateway able to put the server " offline" )

Also corrected the handling of file not found