update endpoint spider logging
drop non-visiting requests early to reduce jobdir size