The crawler now takes a look at the content-type header.