more cunning crawler

Signed-off-by: Tom Everett <tome@criticalmass.com>