Improvements to the crawler and mark as read function