remove nokogiri requirement. add in parsing for meta content and links. push_to_queue, error marking, more TOOD notes