modified to extract googlebot's crap