Introduced black and whit lists to augment robots.txt functionality