add crawler and some fixes to the rest