Fixed robos.txt not blocking non-trailing slashes.