adapting crawlers to subsets

also fixed csv output bug