updates to crawler, roblyparser to handle 404s.