fixed a bunch of bugs, and (hopefully) fixed all NPEs in crawler client (ugh lazy vals)