the crawler will not download links in black list.