try to fix some more exceptions that would only arise from robots requesting bad urls