ignore select bots while selecting urls