added some exceptions to steer crawler away from crawler traps\n also added logic to improve database insertion stability