added rules for logn page detect (upon forced redirect). randomized linkchecking for reduced sequential curl buffer waiting.). also added optional gzip for faster retriveal of sites.