Fix robots parsing with some non-standard sites