Reinstate John's fixes to crawler

Back to state of commit: b9d4f260054067b6f7c3813dcc062ef2c55f9ab6