Setting default headers, rotating user agents and setting up delays is easy with web scraping tools. Some of the best web scraping tools which are open source or free are:
Puppeteer
BeautifulSoup
Cheerio
Diffbot
ScrapeSimple
Scrape API