adds minor fixes and additional comments
Reduces the number of times the crawler performs a commit() operation