General progress on implementing crawler