added code to not revisit a page and only follow next links. lemurproject header also added