Fix robots.txt parsing to handle empty disallow tags properly