updated to enhance detection of end of crawls
merge: