Improvements for crawling repositories