Web scraping with rvest (R Case Study).  Use RVEST to scrape and crawl websites then parse the HTML.
Channel: John Little
Date: 2021-03-10

An introduction to web crawling and HTML parsing.  Use RVEST to scrape, harvest, and crawl websites then parse the HTML.  In this case study in R learn how to deconstruct a target site into structured data by combining limited knowledge of HTML specifications with a very limited appreciation of the HTTP protocol along with basic Tidyverse-style iteration. 

0:00 - Start
14:30 - Overview of the functions, HTML, CSS
22:45 - Initial Q/A
25:30 - Demonstration of inspecting/identifying HTML element 
36:30  - Set up: scraping goal/scenario AND manually investigate the site's structure
35:45 - Code demonstration begins 
36:55 - read_html()  
38:11 - Demonstration of selector gadget
41:08 - html_nodes() & html_text()
41:37 - html_attr()
43:17 - Create a crawling plan - moving through site navigation
52:50 - purrr::map()  iteration
1:03:20 - Q&A
1:05:00 - Javascript buttons, RSelenium, WebScraper.io
1:12:25 - HTML tables
1:14:00 - Iterating over PDF documents
1:19:00 - More on html tables

This Rfun Case Study demonstrate the utility R / Tidyverse workflows.  You can use the Tidyverse as a universal reproducible interface for your analysis projects.

More Rfun at https://Rfun.library.duke.edu/
Part of the DVS Workshop Series:  http://library.duke.edu/data/

LINKS
https://rvest.tidyverse.org
https://selectorgadget.com/

CODE:
https://github.com/libjohn/workshop_webscraping
First Release: https://github.com/libjohn/workshop_webscraping/releases/tag/v1.0