largely refactor scrap process + add full pagination discovery and automatic scrapping