simplify meta tag parse, getting robots. remove .reject. adding time tracking for some debug performance. actually run through skip on steps.