crawler now a class; files consolidated