1 Introduction

2 Using Selenium server browser

2.1 Install and use Selenium

2.1.1 The Selenium RLibrary

2.1.2 Run a Selenium server

2.1.3 Connect to Selenium Server

2.1.4 Basic command for RSelenium

2.2 From analysis to automation

2.2.1 Analysis of website HTML structure

2.2.2 Resolve infinite loading using XPath

2.3 Exercices

3 Scrap using XHR requests

3.1 Analyse Network data in browser

3.2 Generate a custom GET Query

3.3 Exercices

4 Automation of scraping using Docker containers !

4.1 Why ?

4.2 What is Docker ?

4.3 Installing Docker

4.3.1 Linux way

4.3.2 Windows and Mac Way

4.4 Tutorial Docker / Linux

4.5 Tutorial Docker / Windows & Mac

Learn to scrap dynamic content using Docker and
Selenium

library(rvest)
library(tidyverse)
library(knitr)
library(plyr)
library(dplyr)
library(jsonlite)
library(lubridate)
library(RSelenium)

1 Introduction

We see two methods to capture departures and arrivals data for airport on **FlightRadar** website, using a browser running on a server, and using an XHR request.

For each airports page, FlightRadar website offer the possibility to see general informations, departures and arrivals flights information. For this tutorial we try to scrape the Bordeaux Mérignac Airport BOD (https://www.flightradar24.com/data/airports/bod/) arrival flights page (https://www.flightradar24.com/data/airports/bod/arrivals)

As you could see if you go to departures pages, you have two interesting buttons, one at the top of the page, and one at the bottom of the page.

Load earlier flights

Load later flights

To display all data available (something like 24h of past and future departures/arrivals), we simulate multiples clic on this two buttons, and we stop this behavior only when these buttons disapears from the page.

At the end of this tutorial, we present how to use Docker (https://www.docker.com/) container technology to export our webscrapping script on some remote
server for an autonomous execution 24/24h.

2 Using Selenium server browser

2.1 Install and use Selenium

2.1.1 The Selenium RLibrary
# Webscrapping avec R

Lise Vaudor, Sébastien Rey-Coyrehourcq, Fabien Pfaender

► To cite this version:

Lise Vaudor, Sébastien Rey-Coyrehourcq, Fabien Pfaender. Webscrapping avec R. École thématique. Florence Villa Finaly, Italy. 2018. cel-02285503

HAL Id: cel-02285503

https://shs.hal.science/cel-02285503

Submitted on 13 Sep 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
Due to some defense created by webmaster to protect their data (test javascript, user-agent, infinite loading, etc.), you need to simulate an human behavior, if possible using a real browser.

To be short, Selenium (https://docs.seleniumhq.org/) is a multi-tools project focusing on task automation to test web application. It works with lots of Internet browsers, and lot of operating systems.

Selenium Webdriver give to developer an API to interact/pilot an headless internet browser without opening it. So, you, developer, you could use this API with your favorite language (Java, Python, R, etc.) to send commands to browser in order to navigate, move your mouse, click on DOM element, send keyboard output to input forms, inject javascript, capture image of the page, extract html, etc.

First, you need to install and load RSelenium package, the R bindings library for Selenium Webdriver API :

```
install.packages("devtools")
devtools::install_github("ropensci/RSelenium")
```

Depending of your existing configuration and OS you probably need to install some dependent software packages.

It's possible to use directly Selenium in connection with your browser, but we prefer to use directly a server version. Why ? Because using server version of Selenium, you have the possibility :

a. to send command on local or remote server running Selenium

b. which run a different browsers and/or OS,

c. to distribute tests over multiple machines.

Selenium is a fast moving project, and some release are really buggy, so try to choose a stable version, and don't desperate.

## 2.1.2 Run a Selenium server

**!! Before continuing, read the documentation on Docker at the bottom of this document, it explain what is really Docker/images/container, and it explain how to install images/containers on your system !!**

When it's done, we pull and run one of (https://github.com/SeleniumHQ/docker-selenium) Docker Selenium-Server image using terminal. For this tutorial we use **Firefox** !

In classic context (good internet connection), we pull images directly from the Docker Hub server, a central repository like CRAN for R.

```
sudo docker pull selenium/standalone-firefox:3.14.0-arsenic
```

**But, because the image is heavy in size (1 GO for the two images used in this tutorial), we prefer to directly load the image given by USB key by your teachers. Open a terminal on the folder where located the images.**

```
sudo docker load --input=r-alpine.tar
sudo docker load --input=rSelenium.tar
```

Create the Selenium container which contain Firefox :

```
sudo docker run --shm-size=2g --name selenium -d -p 4445:4444 selenium/standalone-firefox:3.14.0-arsenic
```

Type `sudo docker ps` to see if server correctly run and listen to port **4445**

## 2.1.3 Connect to Selenium Server

Connect and open the browser on the server.

```R
user_agent_list = c("Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0", "Mozilla/5.0 (Windows NT 6.1; rv:27.3) Gecko/20130101 Firefox/27.3", "Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0", "Mozilla/5.0 (Windows NT 6.2; Win64; x64;) Gecko/20100101 Firefox/20.0", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2226.0 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1664.3 Safari/537.36", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.90 Safari/537.36", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Ubuntu/11.04 Chrome/17.0.963.56 Chrome/17.0.963.56 Safari/535.11")
```

```R
fprof <- makeFirefoxProfile(list(general.useragenthamid=sample(user_agent_list,1)))
```

```R
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, extraCapabilities = fprof )
remDr$open()
```
# Web scraping with R

Lise Vaudor / Sébastien Rey-Coyrehourcq / Fabien Pfaender
## [1] "Connecting to remote server"
## $acceptInsecureCerts
## [1] FALSE
##
## $browserName
## [1] "firefox"
##
## $browserVersion
## [1] "61.0.1"
##
## $`moz:accessibilityChecks`
## [1] FALSE
##
## $`moz:headless`
## [1] FALSE
##
## $`moz:processID`
## [1] 73
##
## $`moz:profile`
## [1] "/tmp/rust_mozprofile.hYVGLKxPdfkd"
##
## $`moz:useNonSpecCompliantPointerOrigin`
## [1] FALSE
##
## $`moz:IESelectAll`
## [1] TRUE
##
## $pageLoadStrategy
## [1] "normal"
##
## $platformName
## [1] "linux"
##
## $platformVersion
## [1] "4.15.0-34-generic"
##
## $rotatable
## [1] FALSE
##
## $timeouts
## $timeouts$implicit
## [1] 0
##
## $timeouts$pageLoad
## [1] 300000
##
## $timeouts$script
## [1] 30000
##
##
## $IESelectAll
## [1] "413c56f3-b518-4c34-8aeb-37f2fb3cd9c9"
##
## $id
## [1] "413c56f3-b518-4c34-8aeb-37f2fb3cd9c9"

remDr$maxWindowSize()

remDr$executeScript("return navigatorompuserAgent;", list("") )

## [[1]]

## [1] "Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0"

2.1.4 Basic command for RSelenium

Johnd Harrison (https://github.com/johndharrison), the creator and first commiter of RSelenium binding library for Selenium, create a big tutorial with lots of
commands covered : https://rpubs.com/johndharrison/RSelenium-Basics (https://rpubs.com/johndharrison/RSelenium-Basics)

Some of them :

* remDr$maxWindowSize() : maximize windows of the browser.

- remDr$navigate("https://www.google.fr") : navigate to url

- remDr$screenshot (display = TRUE) : take a screenshot of the webpage and display it in RStudio Viewer

- remDr$findElement(...) : Find and element in the html structure, using different method : xpath, css, etc.

- remDr$executeScript(...) : Execute a js script in the remote browser

• remDr$clickElement(...) : Click on element

2.2 From analysis to automation

2.2.1 Analysis of website HTML structure
# Today !

* Webscraping, what is this ?

* Some ethical considerations

* A theoritical HTML page

* Read HTML using rvest

* How manipulate data efficiently (purrr, string)

* How-to extract link & data from HTML (Regex, CSS, XPath)

* A cat and mouse game :)

* Tutorial & Uses cases
Open Web Developer tools in your favorite browser on the arrivals webpage of BOD :
https://www.fliightradar24.com/data/airports/bod/arrivals

We investigate what happens in the html code when we clic the **load earlier** or **load later** button. Why we do that ? To understand how we could automate things later.

Because we want to automate clic on this two buttons, we need to understand WHEN we need to stop clicking :) If we clic an infinite number of time, an error probably trigger when one of the two button disapear.

Select the **Selector tools** and click on the **load earlier** button.

<figure><img src="image_1.png" /></figure>

If you clic the right thing, normally you have highlighted some part of the html code which interest us :

<figure><img src="image_2.png" /></figure>

Now, if you highlight and clic with the web tool selector on the load later flights button, you have something like that :

<figure><img src="image_3.png" /></figure>

Things are not so very differences between this two buttons objects. It seems that only the timestamp, the data page number and the button text change ...

Highlight and clic one more time on the **load earlier flights** button. Clic another time to load a new page of data. You see that the html code change during the data load to desactivate clic on the button. Not so interesting. Now repeat the clic and stop only when the button disapear on your screen.

<figure><img src="image_4.png" /></figure>

Great, a new css style attribute appear to indicate that now this button object is hidden : `style="display: none;"`

How could we re-use this important information during data harvesting to detect if the button is activated/desactivated ? The best solution was to use XPath query !

Load the page in the selenium server

```python
remDr$navigate("https://www.fliightradar24.com/data/airports/bod/arrivals")
Sys.sleep(5) # time to load !
remDr$screenshot(file = "screenshot.png")
```

## 2.2.2 Resolve infinite loading using XPath
# Webscraping

* definition

* some elements of web architecture
Building XPATH correct expression could be difficult. A good way to test validity of your XPATH expressions was to use an interactive way, using the web developer console. There are some good cheatsheet pages which resume all the possibilities of XPATH : 1 (https://devhints.io/xpath) 2 (https://gist.github.com/LeCoupa/8c305ec8c713aad07b14)

Clic on console tab :

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

Inspector

<figure><img src="image_7.png" /></figure>

Console

<figure><img src="image_8.png" /></figure>

Debugger

<figure><img src="image_9.png" /></figure>

} Style Editor

<figure><img src="image_10.png" /></figure>

Performance

<figure><img src="image_11.png" /></figure>

Memory

<figure><img src="image_12.png" /></figure>

Network

<figure><img src="image_13.png" /></figure>

Storage

<figure><img src="image_14.png" /></figure>

HTTPS Everywhere

Type this in the console : `$x("//button[@class='btn.btn-table-action.btn-flights-load']")`

The result is an interactive array you could develop as a tree if you want.

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

Inspector

Console

<figure><img src="image_18.png" /></figure>

Debugger

<figure><img src="image_19.png" /></figure>

} Style Editor

<figure><img src="image_20.png" /></figure>

Performance

<figure><img src="image_21.png" /></figure>

Memory

<figure><img src="image_22.png" /></figure>

Network

<figure><img src="image_23.png" /></figure>

Storage

<figure><img src="image_24.png" /></figure>

HTTPS Everywhere

<figure><img src="image_25.png" /></figure>

…

<figure><img src="image_26.png" /></figure>

x

<figure><img src="image_27.png" /></figure>

x

<figure><img src="image_28.png" /></figure>

x

<figure><img src="image_29.png" /></figure>

x

<figure><img src="image_30.png" /></figure>

x

<figure><img src="image_31.png" /></figure>

x

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

<figure><img src="image_61.png" /></figure>

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>

<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>

<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

<figure><img src="image_78.png" /></figure>

```html
>> $x("//button[@class='btn.btn-table-action.btn-flights-load']")
```

<figure><img src="image_79.png" /></figure>

<figure><img src="image_80.png" /></figure>

<figure><img src="image_81.png" /></figure>

<figure><img src="image_82.png" /></figure>

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /></figure>

<figure><img src="image_85.png" /></figure>

<figure><img src="image_86.png" /></figure>

<figure><img src="image_87.png" /></figure>

<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>

<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /></figure>

<figure><img src="image_94.png" /></figure>

<figure><img src="image_95.png" /></figure>

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /></figure>

<figure><img src="image_98.png" /></figure>

<figure><img src="image_99.png" /></figure>

<figure><img src="image_100.png" /></figure>

<figure><img src="image_101.png" /></figure>

<figure><img src="image_102.png" /></figure>

<figure><img src="image_103.png" /></figure>

<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /></figure>

<figure><img src="image_106.png" /></figure>

<figure><img src="image_107.png" /></figure>

<figure><img src="image_108.png" /></figure>

<figure><img src="image_109.png" /></figure>

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>

<figure><img src="image_115.png" /></figure>

<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>

<figure><img src="image_119.png" /></figure>

<figure><img src="image_120.png" /></figure>

<figure><img src="image_121.png" /></figure>

<figure><img src="image_122.png" /></figure>

<figure><img src="image_123.png" /></figure>

<figure><img src="image_124.png" /></figure>

<figure><img src="image_125.png" /></figure>

<figure><img src="image_126.png" /></figure>

<figure><img src="image_127.png" /></figure>

<figure><img src="image_128.png" /></figure>

<figure><img src="image_129.png" /></figure>

<figure><img src="image_130.png" /></figure>

<figure><img src="image_131.png" /></figure>

<figure><img src="image_132.png" /></figure>

<figure><img src="image_133.png" /></figure>

<figure><img src="image_134.png" /></figure>

<figure><img src="image_135.png" /></figure>

<figure><img src="image_136.png" /></figure>

<figure><img src="image_137.png" /></figure>

<figure><img src="image_138.png" /></figure>

<figure><img src="image_139.png" /></figure>

<figure><img src="image_140.png" /></figure>

<figure><img src="image_141.png" /></figure>

<figure><img src="image_142.png" /></figure>

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /></figure>

<figure><img src="image_145.png" /></figure>

<figure><img src="image_146.png" /></figure>

<figure><img src="image_147.png" /></figure>

<figure><img src="image_148.png" /></figure>

<figure><img src="image_149.png" /></figure>

<figure><img src="image_150.png" /></figure>

<figure><img src="image_151.png" /></figure>

<figure><img src="image_152.png" /></figure>

<figure><img src="image_153.png" /></figure>

<figure><img src="image_154.png" /></figure>

<figure><img src="image_155.png" /></figure>

<figure><img src="image_156.png" /></figure>

<figure><img src="image_157.png" /></figure>

<figure><img src="image_158.png" /></figure>

<figure><img src="image_159.png" /></figure>

<figure><img src="image_160.png" /></figure>

<figure><img src="image_161.png" /></figure>

<figure><img src="image_162.png" /></figure>

<figure><img src="image_163.png" /></figure>

<figure><img src="image_164.png" /></figure>

<figure><img src="image_165.png" /></figure>

<figure><img src="image_166.png" /></figure>

<figure><img src="image_167.png" /></figure>

<figure><img src="image_168.png" /></figure>

<figure><img src="image_169.png" /></figure>

<figure><img src="image_170.png" /></figure>

<figure><img src="image_171.png" /></figure>

<figure><img src="image_172.png" /></figure>

<figure><img src="image_173.png" /></figure>

<figure><img src="image_174.png" /></figure>

<figure><img src="image_175.png" /></figure>

<figure><img src="image_176.png" /></figure>

<figure><img src="image_177.png" /></figure>

<figure><img src="image_178.png" /></figure>

<figure><img src="image_179.png" /></figure>

<figure><img src="image_180.png" /></figure>

<figure><img src="image_181.png" /></figure>

<figure><img src="image_182.png" /></figure>

<figure><img src="image_183.png" /></figure>

<figure><img src="image_184.png" /></figure>

<figure><img src="image_185.png" /></figure>

<figure><img src="image_186.png" /></figure>

<figure><img src="image_187.png" /></figure>

<figure><img src="image_188.png" /></figure>

<figure><img src="image_189.png" /></figure>

<figure><img src="image_190.png" /></figure>

<figure><img src="image_191.png" /></figure>

<figure><img src="image_192.png" /></figure>

<figure><img src="image_193.png" /></figure>

<figure><img src="image_194.png" /></figure>

<figure><img src="image_195.png" /></figure>

<figure><img src="image_196.png" /></figure>

<figure><img src="image_197.png" /></figure>

<figure><img src="image_198.png" /></figure>

<figure><img src="image_199.png" /></figure>

<figure><img src="image_200.png" /></figure>

<figure><img src="image_201.png" /></figure>

<figure><img src="image_202.png" /></figure>

<figure><img src="image_203.png" /></figure>

<figure><img src="image_204.png" /></figure>

<figure><img src="image_205.png" /></figure>

<figure><img src="image_206.png" /></figure>

<figure><img src="image_207.png" /></figure>

<figure><img src="image_208.png" /></figure>

<figure><img src="image_209.png" /></figure>

<figure><img src="image_210.png" /></figure>

<figure><img src="image_211.png" /></figure>

<figure><img src="image_212.png" /></figure>

<figure><img src="image_213.png" /></figure>

<figure><img src="image_214.png" /></figure>

<figure><img src="image_215.png" /></figure>

<figure><img src="image_216.png" /></figure>

<figure><img src="image_217.png" /></figure>

<figure><img src="image_218.png" /></figure>

<figure><img src="image_219.png" /></figure>

<figure><img src="image_220.png" /></figure>

<figure><img src="image_221.png" /></figure>

<figure><img src="image_222.png" /></figure>

<figure><img src="image_223.png" /></figure>

<figure><img src="image_224.png" /></figure>

<figure><img src="image_225.png" /></figure>

<figure><img src="image_226.png" /></figure>

<figure><img src="image_227.png" /></figure>

<figure><img src="image_228.png" /></figure>

<figure><img src="image_229.png" /></figure>

<figure><img src="image_230.png" /></figure>

<figure><img src="image_231.png" /></figure>

<figure><img src="image_232.png" /></figure>

<figure><img src="image_233.png" /></figure>

<figure><img src="image_234.png" /></figure>

<figure><img src="image_235.png" /></figure>

<figure><img src="image_236.png" /></figure>

<figure><img src="image_237.png" /></figure>

<figure><img src="image_238.png" /></figure>

<figure><img src="image_239.png" /></figure>

<figure><img src="image_240.png" /></figure>

<figure><img src="image_241.png" /></figure>

<figure><img src="image_242.png" /></figure>

<figure><img src="image_243.png" /></figure>

<figure><img src="image_244.png" /></figure>

<figure><img src="image_245.png" /></figure>

<figure><img src="image_246.png" /></figure>

<figure><img src="image_247.png" /></figure>

<figure><img src="image_248.png" /></figure>

<figure><img src="image_249.png" /></figure>

<figure><img src="image_250.png" /></figure>

<figure><img src="image_251.png" /></figure>

<figure><img src="image_252.png" /></figure>

<figure><img src="image_253.png" /></figure>

<figure><img src="image_254.png" /></figure>

<figure><img src="image_255.png" /></figure>

<figure><img src="image_256.png" /></figure>

<figure><img src="image_257.png" /></figure>

<figure><img src="image_258.png" /></figure>

<figure><img src="image_259.png" /></figure>

<figure><img src="image_260.png" /></figure>

<figure><img src="image_261.png" /></figure>

<figure><img src="image_262.png" /></figure>

<figure><img src="image_263.png" /></figure>

<figure><img src="image_264.png" /></figure>

<figure><img src="image_265.png" /></figure>

<figure><img src="image_266.png" /></figure>

<figure><img src="image_267.png" /></figure>

<figure><img src="image_268.png" /></figure>

<figure><img src="image_269.png" /></figure>

<figure><img src="image_270.png" /></figure>

<figure><img src="image_271.png" /></figure>

<figure><img src="image_272.png" /></figure>

<figure><img src="image_273.png" /></figure>

<figure><img src="image_274.png" /></figure>

<figure><img src="image_275.png" /></figure>

<figure><img src="image_276.png" /></figure>

<figure><img src="image_277.png" /></figure>

<figure><img src="image_278.png" /></figure>

<figure><img src="image_279.png" /></figure>

<figure><img src="image_280.png" /></figure>

<figure><img src="image_281.png" /></figure>

<figure><img src="image_282.png" /></figure>

<figure><img src="image_283.png" /></figure>

<figure><img src="image_284.png" /></figure>

<figure><img src="image_285.png" /></figure>

<figure><img src="image_286.png" /></figure>

<figure><img src="image_287.png" /></figure>

<figure><img src="image_288.png" /></figure>

<figure><img src="image_289.png" /></figure>

<figure><img src="image_290.png" /></figure>

<figure><img src="image_291.png" /></figure>

<figure><img src="image_292.png" /></figure>

<figure><img src="image_293.png" /></figure>

<figure><img src="image_294.png" /></figure>

<figure><img src="image_295.png" /></figure>

<figure><img src="image_296.png" /></figure>

<figure><img src="image_297.png" /></figure>

<figure><img src="image_298.png" /></figure>

<figure><img src="image_299.png" /></figure>

<figure><img src="image_300.png" /></figure>

<figure><img src="image_301.png" /></figure>

<figure><img src="image_302.png" /></figure>

<figure><img src="image_303.png" /></figure>

<figure><img src="image_304.png" /></figure>

<figure><img src="image_305.png" /></figure>

<figure><img src="image_306.png" /></figure>

<figure><img src="image_307.png" /></figure>

<figure><img src="image_308.png" /></figure>

<figure><img src="image_309.png" /></figure>

<figure><img src="image_310.png" /></figure>

<figure><img src="image_311.png" /></figure>

<figure><img src="image_312.png" /></figure>

<figure><img src="image_313.png" /></figure>

<figure><img src="image_314.png" /></figure>

<figure><img src="image_315.png" /></figure>

<figure><img src="image_316.png" /></figure>

<figure><img src="image_317.png" /></figure>

<figure><img src="image_318.png" /></figure>

<figure><img src="image_319.png" /></figure>

<figure><img src="image_320.png" /></figure>

<figure><img src="image_321.png" /></figure>

<figure><img src="image_322.png" /></figure>

<figure><img src="image_323.png" /></figure>

<figure><img src="image_324.png" /></figure>

<figure><img src="image_325.png" /></figure>

<figure><img src="image_326.png" /></figure>

<figure><img src="image_327.png" /></figure>

<figure><img src="image_328.png" /></figure>

<figure><img src="image_329.png" /></figure>

<figure><img src="image_330.png" /></figure>

<figure><img src="image_331.png" /></figure>

<figure><img src="image_332.png" /></figure>

<figure><img src="image_333.png" /></figure>

<figure><img src="image_334.png" /></figure>

<figure><img src="image_335.png" /></figure>

<figure><img src="image_336.png" /></figure>

<figure><img src="image_337.png" /></figure>

<figure><img src="image_338.png" /></figure>

<figure><img src="image_339.png" /></figure>

<figure><img src="image_340.png" /></figure>

<figure><img src="image_341.png" /></figure>

<figure><img src="image_342.png" /></figure>

<figure><img src="image_343.png" /></figure>

<figure><img src="image_344.png" /></figure>

<figure><img src="image_345.png" /></figure>

<figure><img src="image_346.png" /></figure>

<figure><img src="image_347.png" /></figure>

<figure><img src="image_348.png" /></figure>

<figure><img src="image_349.png" /></figure>

<figure><img src="image_350.png" /></figure>

<figure><img src="image_351.png" /></figure>

<figure><img src="image_352.png" /></figure>

<figure><img src="image_353.png" /></figure>

<figure><img src="image_354.png" /></figure>

<figure><img src="image_355.png" /></figure>

<figure><img src="image_356.png" /></figure>

<figure><img src="image_357.png" /></figure>

<figure><img src="image_358.png" /></figure>

<figure><img src="image_359.png" /></figure>

<figure><img src="image_360.png" /></figure>

<figure><img src="image_361.png" /></figure>

<figure><img src="image_362.png" /></figure>

<figure><img src="image_363.png" /></figure>

<figure><img src="image_364.png" /></figure>

<figure><img src="image_365.png" /></figure>

<figure><img src="image_366.png" /></figure>

<figure><img src="image_367.png" /></figure>

<figure><img src="image_368.png" /></figure>

<figure><img src="image_369.png" /></figure>

<figure><img src="image_370.png" /></figure>

<figure><img src="image_371.png" /></figure>

<figure><img src="image_372.png" /></figure>

<figure><img src="image_373.png" /></figure>

<figure><img src="image_374.png" /></figure>

<figure><img src="image_375.png" /></figure>

<figure><img src="image_376.png" /></figure>

<figure><img src="image_377.png" /></figure>

<figure><img src="image_378.png" /></figure>

<figure><img src="image_379.png" /></figure>

<figure><img src="image_380.png" /></figure>

<figure><img src="image_381.png" /></figure>

<figure><img src="image_382.png" /></figure>

<figure><img src="image_383.png" /></figure>

<figure><img src="image_384.png" /></figure>

<figure><img src="image_385.png" /></figure>

<figure><img src="image_386.png" /></figure>

<figure><img src="image_387.png" /></figure>

<figure><img src="image_388.png" /></figure>

<figure><img src="image_389.png" /></figure>

<figure><img src="image_390.png" /></figure>

<figure><img src="image_391.png" /></figure>

<figure><img src="image_392.png" /></figure>

<figure><img src="image_393.png" /></figure>

<figure><img src="image_394.png" /></figure>

<figure><img src="image_395.png" /></figure>

<figure><img src="image_396.png" /></figure>

<figure><img src="image_397.png" /></figure>

<figure><img src="image_398.png" /></figure>

<figure><img src="image_399.png" /></figure>

<figure><img src="image_400.png" /></figure>

<figure><img src="image_401.png" /></figure>

<figure><img src="image_402.png" /></figure>

<figure><img src="image_403.png" /></figure>

<figure><img src="image_404.png" /></figure>

<figure><img src="image_405.png" /></figure>

<figure><img src="image_406.png" /></figure>

<figure><img src="image_407.png" /></figure>

<figure><img src="image_408.png" /></figure>

<figure><img src="image_409.png" /></figure>

<figure><img src="image_410.png" /></figure>

<figure><img src="image_411.png" /></figure>

<figure><img src="image_412.png" /></figure>

<figure><img src="image_413.png" /></figure>

<figure><img src="image_414.png" /></figure>

<figure><img src="image_415.png" /></figure>

<figure><img src="image_416.png" /></figure>

<figure><img src="image_417.png" /></figure>

<figure><img src="image_418.png" /></figure>

<figure><img src="image_419.png" /></figure>

<figure><img src="image_420.png" /></figure>

<figure><img src="image_421.png" /></figure>

<figure><img src="image_422.png" /></figure>

<figure><img src="image_423.png" /></figure>

<figure><img src="image_424.png" /></figure>

<figure><img src="image_425.png" /></figure>

<figure><img src="image_426.png" /></figure>

<figure><img src="image_427.png" /></figure>

<figure><img src="image_428.png" /></figure>

<figure><img src="image_429.png" /></figure>

<figure><img src="image_430.png" /></figure>

<figure><img src="image_431.png" /></figure>

<figure><img src="image_432.png" /></figure>

<figure><img src="image_433.png" /></figure>

<figure><img src="image_434.png" /></figure>

Clic Clic Clic to make disappear one of the loading button, and now we trying to select only the available button. XPATH understand boolean operator (or, and, etc.) so we filter by @class and style :

```html
$x("//button[@class='btn.btn-table-action.btn-flights-load' and contains(@style,'display:none;')]")
```

Great, this query return only the valid button. We use later this query to stop our loop of infernal button clic.

Now we try to build this query using R Selenium with `findElement()` function :

```javascript
loadMoreButton <- remDr$findElements(using = 'xpath', "/button[@class='btn.btn-table-action.btn-flights-load' and not(co ntains(@style,'display:none;'))]")
```

Display the text of each element retrieved by function `findElements()` using the ` elementText()` function

```javascript
unlist(lapply.loadMoreButton, function(x){x$elementText()}))
```

```javascript
## [1] "Load earlier flights" "Load later flights"
```

Now, how to simulate a click on one of this button ?

An easy way was to call `clickElement()` function on the first `loadMoreButton` webelement :

```javascript
tryCatch(
  suppressMessages({
    loadMoreButton[[1]]$clickElement()
  }
  error = function(e) {
    loadMoreButton[[1]]$errorDetails()$message
  })
```

This command return an error message (if not, you're lucky !), not very explicit, so if you want more details, you could call the function `errorDetails()` like our `trycatch` block.

An element of the webpage overlapp our button, so browser say us that's not possible to clic on this webelement. Use snapshot function to see the page :

```javascript
remDr$snapshot(file = 'screenshoot_overlap.png')
```

If we hide these elements using XPath and javascript injection, everything goes to normal. First we accept cookies.

```javascript
hideCookie <- function (x) {
  cookiesButton <- x$findElement(using = 'xpath', "/div[@class='importantbanner__close']")
  cookiesButton$clickElement()
}

hideCookie(remDr)
remDr$snapshot(file = 'screenshoot_hidden.png')
```

The navbar element create problem, so we hide it using javascript injection :

```javascript
hideNavBar <- function (x) {
  script <- "document.getElementById('navContainer').hidden = true;"
  x$executeScript script
}

hideNavBar(remDr)
```

```javascript
## list()
```

Now you can `clickElement()` without problem :)

```javascript
tryCatch(
  suppressMessages({
    loadMoreButton[[1]]$clickElement()
  }
  error = function(e) {
    loadMoreButton[[1]]$errorDetails()$message
  })
```
# Ethics, some elements of reflexion.

**In two words**

* Grey Zone !! Heavily dependent of your country and the site you scrap !

* Law is clearly not so clear, but now RGPD include **derogations** for researchers

* Contact CNIL (France) or similar institute in your country, they will help you !

* **Don't publish any data on the web** before anonymisation !

* Collect data limited to your project

* Protect personal data very seriously (data-center, encrypted, firewall protection, etc.)

**In more than two words ...**

* Read new European Law : GPDR or RGPD (not a joke, not so hard to read)
See changes before and after using `remDr$ screenshot (display = TRUE)` command

## 2.3 Exercices

* Create a function which click on button until they all disappears :)

Show/hide

* Extract data for only one day using XPATH and Rvest !

Show/hide

# 3 Scrap using XHR requests

## 3.1 Analyse Network data in browser

Sometimes, a defense is also a point of vulnerability. Many site use some sort of internal API to query and feed their website, it's an easy way for developers to distribute the data. But for us, this is also a perfect data SOF (Single Point Of Failure) (https://en.wikipedia.org/wiki/Single_point_of_failure).

We try to see if this is the case with flight radar :)

Open the dev tools in the browser, click on **Network** tab, then **XHR** tab.

<figure><img src="image_435.png" /></figure>

Lucky guy/girl, do you see it ? Each GET query call an `airport.json` file on the server :

https://api.flightradar24.com/common/v1/airport.json?code=bod&plugin []=&plugin-setting[schedule][mode]=&plugin-setting[schedule][timesta]

If we decompose the query, we have :

* an airport code : `bod`

* a timestamp : `1537297562`

* a page number : `1`

* a limit by page : `100`

Copy paste this url in your browser to see how the result json is structured. Interesting data is located into schedule
`result > response > airport > arrivals :`

* **item**: number of total items

* **page**: actual page and number of page

* **timestamp**: date of capture

* **data**: a list of 100 flights corresponding to actual page

## 3.2 Generate a custom GET Query

We have the capacity to generate custom query to download data at custom timestamp. This query return data structured in json, so we try to convert this data to `data.frame` using the `jsonlite` wonderful package :) Why wonderful ? Because `jsonlite` had an option to flatten the structure of json which normally contain `data.frame` into `data.fram` into `data.frame` ...

timestamp <- as.numeric(as.POSIXct(now()))

url <- paste("https://api.flightradar24.com/common/v1/airport.json?code=bod&plugin []=&plugin-setting[schedule][mode]=&plugin-setting[schedule][timestamp]=" , timestamp , "&page=1&limit=100&token=" , sep="")

# https://cran.r-project.org/web/packages/jsonlite/vignettes/json-aaquickstart.html

json <- jsonlite::fromJSON(url, flatten = T)

We extract information for the first page of Arrivals data collected by airport from json.

pageOfData <- json result $response $airport $pluginData $schedule $arrivals $data
filteredData <- pageOfData %>% select(flight.airline. code.icao, flight.airline.name, flight.airport.origin.name, flight.a
irport.origin. code.icao, flight.airport.origin. position. latitude, flight.airport.origin. position. longitude)

filteredData <- rename(filteredeData, c(flight.airline. code.icao = "ICAO", flight.airline.name = "Name", flight.airport.ori
gin.name = "Origin", flight.airport.origin. code.icao = "Origin ICAO", flight.airport.origin. position. latitude = "Latitude",
flight.airport.origin. position. longitude = "Longitude" ))

knitr::kable(filteredeData, caption = "page 1 of arrival for BOD")

page 1 of arrival for BOD

<table><thead><tr><th>ICAO</th><th>Name</th><th>Origin</th><th>Origin ICAO</th><th>Latitude</th><th>Longitude</th></tr></thead><tbody><tr><td>CLG</td><td>Chalair Aviation</td><td>Nantes Atlantique Airport</td><td>LFRS</td><td>47.15694</td><td>-1.607770</td></tr><tr><td>AFR</td><td>Air France</td><td>Lyon Saint Exupery Airport</td><td>LFLL</td><td>45.71964</td><td>5.089108</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Orly Airport</td><td>LFPO</td><td>48.72333</td><td>2.379444</td></tr></tbody></table>
* Read this starting paper written on this subject for this school :
https://hackmd.io/g9JrWURjTHyLr9EWQqKoiQ?
both

* Be inspired by existing case of scrapping or other discipline (medecine)
<table><thead><tr><th>ICAO</th><th>Name</th><th>Origin</th><th>Origin ICAO</th><th>Latitude</th><th>Longitude</th></tr></thead><tbody><tr><td>CLG</td><td>Chalair Aviation</td><td>Brest Bretagne Airport</td><td>LFRB</td><td>48.44722</td><td>-4.421660</td></tr><tr><td>TUI</td><td>TUI</td><td>Casablanca Mohammed V International Airport</td><td>GMMN</td><td>33.36746</td><td>-7.589960</td></tr><tr><td>EZY</td><td>easyJet</td><td>Marseille Provence Airport</td><td>LFML</td><td>43.43666</td><td>5.215000</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Charles de Gaulle Airport</td><td>LFPG</td><td>49.01252</td><td>2.555752</td></tr><tr><td>KLM</td><td>KLM</td><td>Amsterdam Schiphol Airport</td><td>EHAM</td><td>52.30861</td><td>4.763889</td></tr><tr><td>HOP</td><td>HOP!</td><td>Lille Airport</td><td>LFQQ</td><td>50.56333</td><td>3.086944</td></tr><tr><td>VOE</td><td>Volotea</td><td>Bastia Poretta Airport</td><td>LFKB</td><td>42.55000</td><td>9.484722</td></tr><tr><td>VOE</td><td>Volotea</td><td>Palermo Falcone-Borsellino Airport</td><td>LICJ</td><td>38.17595</td><td>13.091010</td></tr><tr><td>VOE</td><td>Volotea</td><td>Luqa Malta International Airport</td><td>LMML</td><td>35.85749</td><td>14.477500</td></tr><tr><td>FPO</td><td>ASL Airlines France</td><td>Paris Charles de Gaulle Airport</td><td>LFPG</td><td>49.01252</td><td>2.555752</td></tr><tr><td>HOP</td><td>HOP!</td><td>Marseille Provence Airport</td><td>LFML</td><td>43.43666</td><td>5.215000</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Orly Airport</td><td>LFPO</td><td>48.72333</td><td>2.379444</td></tr><tr><td>CLG</td><td>Chalair Aviation</td><td>Rennes Saint-Jacques Airport</td><td>LFRN</td><td>48.07194</td><td>-1.732220</td></tr><tr><td>HOP</td><td>HOP!</td><td>Lyon Saint Exupery Airport</td><td>LFLL</td><td>45.71964</td><td>5.089108</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Orly Airport</td><td>LFPO</td><td>48.72333</td><td>2.379444</td></tr><tr><td>VOE</td><td>Volotea</td><td>Strasbourg Airport</td><td>LFST</td><td>48.54361</td><td>7.637222</td></tr><tr><td>CLG</td><td>Chalair Aviation</td><td>Nantes Atlantique Airport</td><td>LFRS</td><td>47.15694</td><td>-1.607770</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Nice Cote d'Azur Airport</td><td>LFMN</td><td>43.66527</td><td>7.215000</td></tr><tr><td>IBE</td><td>Iberia</td><td>Madrid Barajas Airport</td><td>LEMD</td><td>40.49355</td><td>-3.566760</td></tr><tr><td>SWR</td><td>Swiss</td><td>Zurich Airport</td><td>LSZH</td><td>47.46472</td><td>8.549167</td></tr><tr><td>BAW</td><td>British Airways</td><td>London Gatwick Airport</td><td>EGKK</td><td>51.14805</td><td>-0.190270</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Charles de Gaulle Airport</td><td>LFPG</td><td>49.01252</td><td>2.555752</td></tr><tr><td>HOP</td><td>HOP!</td><td>Lille Airport</td><td>LFQQ</td><td>50.56333</td><td>3.086944</td></tr><tr><td>HOP</td><td>HOP!</td><td>Lyon Saint Exupery Airport</td><td>LFLL</td><td>45.71964</td><td>5.089108</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Lyon Saint Exupery Airport</td><td>LFLL</td><td>45.71964</td><td>5.089108</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Orly Airport</td><td>LFPO</td><td>48.72333</td><td>2.379444</td></tr><tr><td>CLG</td><td>Chalair Aviation</td><td>Brest Bretagne Airport</td><td>LFRB</td><td>48.44722</td><td>-4.421660</td></tr><tr><td>CLG</td><td>Chalair Aviation</td><td>Montpellier Mediterranee Airport</td><td>LFMT</td><td>43.58333</td><td>3.961389</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Charles de Gaulle Airport</td><td>LFPG</td><td>49.01252</td><td>2.555752</td></tr><tr><td>EZY</td><td>EasyJet</td><td>London Gatwick Airport</td><td>EGKK</td><td>51.14805</td><td>-0.190270</td></tr><tr><td>KLM</td><td>KLM</td><td>Amsterdam Schiphol Airport</td><td>EHAM</td><td>52.30861</td><td>4.763889</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Orly Airport</td><td>LFPO</td><td>48.72333</td><td>2.379444</td></tr><tr><td>PBD</td><td>Pobeda</td><td>Moscow Vnukovo International Airport</td><td>UUWW</td><td>55.59153</td><td>37.261478</td></tr><tr><td>VOE</td><td>Volotea</td><td>Split Airport</td><td>LDSP</td><td>43.53894</td><td>16.297960</td></tr><tr><td>TSC</td><td>Air Transat</td><td>Montreal Pierre Elliott Trudeau Airport</td><td>CYUL</td><td>45.47055</td><td>-73.740799</td></tr><tr><td>CLG</td><td>Chalair Aviation</td><td>Nantes Atlantique Airport</td><td>LFRS</td><td>47.15694</td><td>-1.607770</td></tr><tr><td>RYR</td><td>Ryanair</td><td>Brussels South Charleroi Airport</td><td>EBCI</td><td>50.46000</td><td>4.452778</td></tr><tr><td>VLG</td><td>Vueling</td><td>Palma de Mallorca Airport</td><td>LEPA</td><td>39.55167</td><td>2.738808</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Bristol Airport</td><td>EGGD</td><td>51.38266</td><td>-2.719080</td></tr><tr><td>THY</td><td>Turkish Airlines</td><td>Istanbul Ataturk International Airport</td><td>LTBA</td><td>40.97692</td><td>28.814600</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Belfast International Airport</td><td>EGAA</td><td>54.65750</td><td>-6.215830</td></tr><tr><td>VLG</td><td>Vueling</td><td>Malaga Costa Del Sol Airport</td><td>LEMG</td><td>36.67490</td><td>-4.499100</td></tr><tr><td>BEE</td><td>Flybe</td><td>Birmingham Airport</td><td>EGBB</td><td>52.45385</td><td>-1.748020</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Charles de Gaulle Airport</td><td>LFPG</td><td>49.01252</td><td>2.555752</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Milan Malpensa Airport</td><td>LIMC</td><td>45.63060</td><td>8.728111</td></tr><tr><td>HOP</td><td>HOP!</td><td>Nice Cote d'Azur Airport</td><td>LFMN</td><td>43.66527</td><td>7.215000</td></tr><tr><td>VOE</td><td>Volotea</td><td>Santorini Thira National Airport</td><td>LGSR</td><td>36.39916</td><td>25.479330</td></tr><tr><td>JAF</td><td>TUI fly Belgium</td><td>Corfu International Airport</td><td>LGKR</td><td>39.60194</td><td>19.911659</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Lisbon Humberto Delgado Airport</td><td>LPPT</td><td>38.78131</td><td>-9.135910</td></tr></tbody></table>
# Packages

<figure><img src="image_2.png" /></figure>

* `rvest`: web scraping

<figure><img src="image_3.png" /></figure>

* `dplyr`: table-wrangling

<figure><img src="image_4.png" /></figure>

* `stringr`: strings

<figure><img src="image_5.png" /></figure>

* `purrr`: iteration
<table><thead><tr><td>ICAO</td><td>Name</td><td>Origin</td><td>Origin ICAO</td><td>Latitude</td><td>Longitude</td></tr></thead><tbody><tr><td>EZY</td><td>EasyJet</td><td>Palma de Mallorca Airport</td><td>LEPA</td><td>39.55167</td><td>2.738808</td></tr><tr><td>VOE</td><td>Volotea</td><td>Mahon Menorca Airport</td><td>LEMH</td><td>39.86259</td><td>4.218647</td></tr><tr><td>HOP</td><td>HOP!</td><td>Dusseldorf International Airport</td><td>EDDL</td><td>51.28945</td><td>6.766775</td></tr><tr><td>VLG</td><td>Vueling</td><td>Barcelona El Prat Airport</td><td>LEBL</td><td>41.29707</td><td>2.078463</td></tr><tr><td>RYR</td><td>Ryanair</td><td>London Stansted Airport</td><td>EGSS</td><td>51.88500</td><td>0.235000</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Orly Airport</td><td>LFPO</td><td>48.72333</td><td>2.379444</td></tr><tr><td>VOE</td><td>Volotea</td><td>Dubrovnik Airport</td><td>LDDU</td><td>42.56135</td><td>18.268240</td></tr><tr><td>TAP</td><td>TAP Portugal</td><td>Lisbon Humberto Delgado Airport</td><td>LPPT</td><td>38.78131</td><td>-9.135910</td></tr><tr><td>EIN</td><td>Aer Lingus</td><td>Dublin Airport</td><td>EIDW</td><td>53.42138</td><td>-6.270000</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Orly Airport</td><td>LFPO</td><td>48.72333</td><td>2.379444</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Barcelona El Prat Airport</td><td>LEBL</td><td>41.29707</td><td>2.078463</td></tr><tr><td>HOP</td><td>HOP!</td><td>Marseille Provence Airport</td><td>LFML</td><td>43.43666</td><td>5.215000</td></tr><tr><td>KLM</td><td>KLM</td><td>Amsterdam Schiphol Airport</td><td>EHAM</td><td>52.30861</td><td>4.763889</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Rhodes International Airport</td><td>LGRP</td><td>36.40541</td><td>28.086189</td></tr><tr><td>CLG</td><td>Chalair Aviation</td><td>Rennes Saint-Jacques Airport</td><td>LFRN</td><td>48.07194</td><td>-1.732220</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Basel Mulhouse-Freiburg EuroAirport</td><td>LFSB</td><td>47.59890</td><td>7.528300</td></tr><tr><td>EZY</td><td>EasyJet</td><td>London Luton Airport</td><td>EGGW</td><td>51.87472</td><td>-0.368330</td></tr><tr><td>VOE</td><td>Volotea</td><td>Strasbourg Airport</td><td>LFST</td><td>48.54361</td><td>7.637222</td></tr><tr><td>HOP</td><td>HOP!</td><td>Lyon Saint Exupery Airport</td><td>LFLL</td><td>45.71964</td><td>5.089108</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Lille Airport</td><td>LFQQ</td><td>50.56333</td><td>3.086944</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Charles de Gaulle Airport</td><td>LFPG</td><td>49.01252</td><td>2.555752</td></tr><tr><td>HOP</td><td>HOP!</td><td>Lyon Saint Exupery Airport</td><td>LFLL</td><td>45.71964</td><td>5.089108</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Berlin Schönefeld Airport</td><td>EDDB</td><td>52.38000</td><td>13.522500</td></tr><tr><td>IBE</td><td>Iberia</td><td>Madrid Barajas Airport</td><td>LEMD</td><td>40.49355</td><td>-3.566760</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Orly Airport</td><td>LFPO</td><td>48.72333</td><td>2.379444</td></tr><tr><td>DLH</td><td>Lufthansa</td><td>Frankfurt Airport</td><td>EDDF</td><td>50.02642</td><td>8.543125</td></tr><tr><td>BEL</td><td>Brussels Airlines</td><td>Brussels Airport</td><td>EBBR</td><td>50.90138</td><td>4.484444</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Geneva International Airport</td><td>LSGG</td><td>46.23806</td><td>6.108950</td></tr><tr><td>CLG</td><td>Chalair Aviation</td><td>Brest Bretagne Airport</td><td>LFRB</td><td>48.44722</td><td>-4.421660</td></tr><tr><td>VOE</td><td>Volotea</td><td>Malaga Costa Del Sol Airport</td><td>LEMG</td><td>36.67490</td><td>-4.499100</td></tr><tr><td>HOP</td><td>HOP!</td><td>Lille Airport</td><td>LFQQ</td><td>50.56333</td><td>3.086944</td></tr><tr><td>FPO</td><td>ASL Airlines France</td><td>Faro Airport</td><td>LPFR</td><td>37.01442</td><td>-7.965910</td></tr><tr><td>SWR</td><td>Swiss</td><td>Zurich Airport</td><td>LSZH</td><td>47.46472</td><td>8.549167</td></tr><tr><td>CLG</td><td>Chalair Aviation</td><td>Brest Bretagne Airport</td><td>LFRB</td><td>48.44722</td><td>-4.421660</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Lyon Saint Exupery Airport</td><td>LFLL</td><td>45.71964</td><td>5.089108</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Orly Airport</td><td>LFPO</td><td>48.72333</td><td>2.379444</td></tr><tr><td>CLG</td><td>Chalair Aviation</td><td>Montpellier Mediterranee Airport</td><td>LFMT</td><td>43.58333</td><td>3.961389</td></tr><tr><td>HOP</td><td>HOP!</td><td>Marseille Provence Airport</td><td>LFML</td><td>43.43666</td><td>5.215000</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Nice Cote d'Azur Airport</td><td>LFMN</td><td>43.66527</td><td>7.215000</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Charles de Gaulle Airport</td><td>LFPG</td><td>49.01252</td><td>2.555752</td></tr><tr><td>EZY</td><td>easyJet (Europcar Livery)</td><td>Amsterdam Schiphol Airport</td><td>EHAM</td><td>52.30861</td><td>4.763889</td></tr><tr><td>VOE</td><td>Volotea</td><td>Toulon-Hyeres Airport</td><td>LFTH</td><td>43.09734</td><td>6.146031</td></tr><tr><td>AFR</td><td>Air France</td><td>Paris Orly Airport</td><td>LFPO</td><td>48.72333</td><td>2.379444</td></tr><tr><td>HOP</td><td>HOP!</td><td>Lyon Saint Exupery Airport</td><td>LFLL</td><td>45.71964</td><td>5.089108</td></tr><tr><td>EZY</td><td>EasyJet</td><td>Lyon Saint Exupery Airport</td><td>LFLL</td><td>45.71964</td><td>5.089108</td></tr><tr><td>FPO</td><td>ASL Airlines France</td><td>Paris Charles de Gaulle Airport</td><td>LFPG</td><td>49.01252</td><td>2.555752</td></tr><tr><td>CLG</td><td>Chalair Aviation</td><td>Nantes Atlantique Airport</td><td>LFRS</td><td>47.15694</td><td>-1.607770</td></tr><tr><td>EZY</td><td>easyJet</td><td>Brussels Airport</td><td>EBBR</td><td>50.90138</td><td>4.484444</td></tr></tbody></table>
# A theoretical example of Web scraping

<figure><img src="image_6.png" /></figure>

Package rvest:

<figure><img src="image_7.png" /></figure>

To collect the contents of a web page, one has to:

## HTML Document

* read an html page in R

Understanding html documents' structure

* get the element of interest in the page (basic html, SelectorGadget)

Tools SelectorGadget or html inspector

* parse the page

* select an element of the page

* get the contents of the element

<figure><img src="image_8.png" /></figure>
## 3.3 Exercices

* Get all pages of arrivals data by generating the correct query to API :)

# 4 Automation of scraping using Docker containers !

## 4.1 Why ?

This is the ultimate and probably the most complex part of this big tutorial.

In real webscraping project, there are two possible use case :

* a one shoot data harvesting or

* a daily/monthly/etc. data harvesting.

Take a very practical example, if you need to collect **one year of data on a daily basis** you cannot use your personal computer. You need to connect and run your program from a distant server (somewhere on internet).

## 4.2 What is Docker ?

To be really really short on subject, Docker (https://en.wikipedia.org/wiki/Docker_(software)) is a technology which encapsulate one or multiple software with all theirs dependencies into an isolated (and if possible immutable) container which run on top of any system interoperable with Docker tools (Window/Linux/Mac). You could run multiple isolated containers (A, B, ...), with capacity to exchange informations using a common dedicated local network, on the same machine.

<figure><img src="image_436.png" /></figure>

It was a very interesting technology because if you develop a program in a container on your local machine, it works on any server compatible with **Docker**. If you know the concept of Virtual Machine (VM) (https://en.wikipedia.org/wiki/Virtual_machine), the idea is the same, but Docker is a more efficient technology (see this comparison (https://www.docker.com/resources/what-container) on Docker official documentation).

<figure><img src="image_437.png" /></figure>

Lena create an independent container which contain :

* the software

* the dependencies, libraries, etc.

* the data collected or needed by software

Two use case are possible :

* If Lena want to share container **MyContainer** to paul, she (command docker export --output myContainer.tar myContainer) the container on a USB key. Later, Paul copy and load the container **MyContainer** on his machine (command docker load ).

* If Lena want to run **MyContainer** container on some server on the web which contain docker program, she export the container(command docker --output myContainer.tar myContainer), copy this container on the server (using FTP protocol for example) and load this container (command docker load --input myContainer.tar) on the server.

**Be careful 1** if your container contain a Volume you need a special procedure to migrate it.

**Be careful 2** If your container need another container to work, you need to export both.

Here we are, we use this **Docker** (https://www.docker.com/) **technology** to encapsulate our webscrapping script into one **portable** container. After that you could save yours and launch it on any webserver which run Docker.

There are three big step to understand in containers lifecycle:

* First, we describe the composition of an **image** into a **Dockerfile** file using a special Docker syntax. **It's like a recipe into cookbook.** For example, you could find lot of recipes for general software on this site : DockerHub (https://hub.docker.com/explore/).
<figure><img src="image_9.png" /></figure>
* Next, like a recipe in the real life, you need to concretize this recipe into some delicious cake. Image need to be built before usage. From an **Image/Recipe of a cake** we create a **Container/Cake**

* Finally, you **run** the built image.

You could find lot of other informations on the web, but also on this online tutorial (https://hackmd.io/s/SylAFG_Kf#) which resume lot of commands.

In this tutorial, we give you the corresponding `Dockerfile` which contain the recipe to build a `Container` ready to scrape the flightradar website. All script to do that are in `docker-scripts` folder.

At the end of the tutorial you have this architecture, with two container (RSelenium and Flightscrapradar) which communicate to scrap data on flightradar website.

<figure><img src="image_438.png" /></figure>

## 4.3 Installing Docker

### 4.3.1 Linux way

On linux Ubuntu, you found documentation here (https://docs.docker.com/install/linux/docker-ce/ubuntu/). First step, install the key and repository.

```bash
sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
```

Add key and repository :

```bash
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"

sudo apt-get update
```

Install `docker-ce`:

```bash
sudo apt-get install docker-ce
```

You are ready to jump to the tutorial which correspond to your OS.

### 4.3.2 Windows and Mac Way

There are two way to install Docker for Windows and Mac, a new way Windows (https://docs.docker.com/docker-for-windows/) / Mac (https://docs.docker.com/docker-for-mac/) and an old way. For this tutorial we use the the old way due to better compatibility.

Install Docker Tools for windows using the `DockerToolbox.exe` (or `.dmg` for mac) file. You could find the official documentation is available here Windows (https://docs.docker.com/toolbox/toolbox_install_windows/) and Mac (https://docs.docker.com/toolbox/toolbox_install_mac/)
# html tags

```html
<html>
<style>
h1 {background-color: powderblue;}
.image {margin-left: 50%;}
 comment{border-style:solid; background-color: black;}
 comment-author{font-style: italic;}
</style>
<h1> MA VIE A LA FERME </h1>
<div class="ingredients">
 <b> INGREDIENTS </b>
 <ul>
  <li> >1 cochon(s) </li>
  <li> >1 légume(s) </li>
 </ul>
</div>
<div class="right"><div class="image"><p> Je fais de la bouillie pour mes petit <p> Pour un cochon, pour deux cochons, po <b>comments</b></p></div>
<div class="comment">Et pour une poule su </div>
<div class="comment-author">Emma, 22 ans, <div class="comment">Je vois que vous êtes <div class="comment-author">Michel, 56 an </div></html>
```

## MA VIE A LA FERME

### INGREDIENTS

* >1 cochon(s)

* >1 légume(s)

<figure><img src="image_10.png" /></figure>

Je fais de la bouillie pour mes petits cochons.

Pour un cochon, pour deux cochons, pour trois cochons, pour quatre, puis pour cinq, pour six, pour sept, pour huit, pour neuf, boeuf!

### Commentaires

Et pour une poule sur un mur qui picoterait du pain dur? c'est bien beau de nourrir les petits cochons mais qu'en est-il des autres?

Emma, 22 ans, Limoges

Je vois que vous êtes, telle la petite poule rousse, bien aimable. Avez-vous pu compter sur l'aide du chat et du canard (et des cochons eux-mêmes!) pour semer vos 5 grains de blé? Les cochons se sont-ils régalés?
<figure><img src="image_439.png" /></figure>

After that you could launch Docker quickstart terminal directly after installation or using the icon in start menu.

<figure><img src="image_440.png" /></figure>

Docker first download an iso, and after that test if your system is ready to run containers. If you see an error like this, you need to run another step.

<figure><img src="image_441.png" /></figure>

Restart your computer, and try to activate an option in the BIOS (Del key during initialization of your computer) probably named "Vanderpool technology" or "VT-X technology" or "Virtualization technology". Save and restart. Some pictore for UEFI Bios on HP, DELL, ASUS motherboard/systems.

Asus
Read an html page

Read html page in R:

```r
library(rvest)
html=read_html("data/blog_de_ginette.html")
## {xml_document}
## <html>
## [1] <head>
## [2] <body>
```
<figure><img src="image_442.png" /></figure>

Dell

<figure><img src="image_443.png" /></figure>
# Extract some elements in the html page

```html
<html>
<style>
h1 {background-color: powderblue;}
.image {margin-left: 50%;}
(comment{border-style:solid; background-color: italic;}
</style>
<h1> MA VIE A LA FERME </h1>
<div class="ingredients">
<b> INGREDIENTS </b>
<ul>
<li> >1 cochon(s) </li>
<li> >1 légume(s) </li>
</ul>
</div>
<div class="right"><div class="image"><img src="" alt=""/>
<p> Je fais de la bouillie pour mes petits. Pour un cochon, pour deux cochons, pour des comments </p>
<b>comments</b>
<div class="comment">Et pour une poule suivi</div>
<div class="comment-author">Emma, 22 ans, <br/>Je vois que vous êtes Michel, 56 an</div>
</div>
```

## Extract some elements ("nodes" or "nodesets"):

```javascript
html_nodes/html, "b")
## {xml_nodeset (2)}
## [1] <b> INGREDIENTS </b>
## [2] <b>Commentaires</b>
html_nodes/html, ".comment-author")
## {xml_nodeset (2)}
## [1] <div class="comment-author">Emma</div>
## [2] <div class="comment-author">Michel</div>
html_nodes/html, ".ingredients") %>%
html_children()
## {xml_nodeset (2)}
## [1] <b> INGREDIENTS </b>
## [2] <ul>
```
<figure><img src="image_444.png" /></figure>

You are ready to jump to the tutorial which correspond to your OS.

## 4.4 Tutorial Docker / Linux

Copy the folder `docker-images` on the USB Key (ask teachers) into the `scrap-flightradar` folder of this tutorial.

Now, go to this folder using terminal command (`cd pathofthefolder`), and load the two images on your system.

```
sudo docker load --input=r-alpine.tar
sudo docker load --input=rSelenium.tar
```

### BUILD image

Go to `docker-scripts` folder into the folder which contain this tutorial on your disk.

The building of this image take lot of times (ten minutes), this is due to the huge dplyr library. Run the `docker build` command in the folder which contain the `Dockerfile` description of the image.

```
docker build . --tag=rflightscraps
```

### LAUNCH Container

a. Using a **binded volume**, this is the easiest way actually. Open your terminal, go to the folder of your script. Create a new folder named `localbackup` and run the container `rflightscraps` with correct path.

```
mkdir localbackup
docker run --name rflightscraps -d -e UID=1000 -e GID=1000 --mount type=bind,source=$(pwd)/localbackup,destination=/usr/local/src/flight-scrap/docker-scripts/data rflightscraps
```

To see if your container is running and consult the logs of execution :

```
sudo docker ps
sudo docker logs rflightscraps
```

To consult the result of automatic harvesting, consult the `docker-scripts/localbackup` folder using `ls` unix command. You see a list of `csv` which correspond to harvest made every minute. If you want to change this, you need to modify the `crontab` file following the cron syntax (https://crontab.guru/#*_*_*_*_*_*), and rebuild/relaunch the image (it take less time, because you only modify one file, no need to recompile).

b. Same thing, but using a **named volume**, a more portable way to share data between docker container, but it lacks some features on permissions to correctly export data.

Create a named volume, independent from filesystem

```
docker volume create --name myDataVolume
docker volume ls
```

Mount the volume :

```
docker run --mount type=volume,source=myDataVolume,destination=/usr/local/src/flight-scrap/docker-scripts/data rflightscraps
```

Export data :
Extract the type of some
elements

Extract the type of
nodes or nodesets:

html_nodes/html,".image") %>%
html_name()

## [1] "div"
* Using a alpine image, we mount the named volume (`myDataVolume`) to a `/alpine_data` folder inside the alpine container.

* Then, we create a new folder inside the alpine container named `/alpine_backup`.

* We then create an archive containing the contents of the `/alpine_data` folder and we store it inside the `/alpine_backup` folder (inside the container).

* We also mount the `/alpine_backup` folder from the container to the docker host (your local machine) in a folder named `/local_backup` inside the current directory.

```bash
docker run --rm -v myDataVolume:/alpine_data -v $(pwd)/local_backup:/alpine_backup alpine:latest tar cvf /alpine_backup/scrap_data "$(dir $(pwd))"
```

## EXPORT Containers

<figure><img src="image_445.png" /></figure>

If you don't have a server yet, you could buy one with docker already installed for cheap / month : - https://www.digitalocean.com/products/one-click-apps/docker/ (https://www.digitalocean.com/products/one-click-apps/docker/) - https://www.ovh.com/fr/vps/vps-cloud.xml (https://www.ovh.com/fr/vps/vps-cloud.xml)

<figure><img src="image_446.png" /></figure>

00:00

<figure><img src="image_447.png" /></figure>

# 4.5 Tutorial Docker / Windows & Mac

## PREPARE image

Copy `docker-scripts` and `docker-images` folders into `c:\Program Files\Docker Toolbox`
## Extract the content of some elements

```html
<html>
<style>
h1 {background-color: powderblue;}
.image {margin-left: 50%;}
 comment{border-style: solid; background-color: italic;}
</style>
<h1> MA VIE A LA FERME </h1>
<div class="ingredients">
  <b> INGREDIENTS </b>
  <ul>
    <li> >1 cochon(s) </li>
    <li> >1 légume(s) </li>
  </ul>
</div>
<div class="right"><div class="image"><img src="" alt="A right side of a person with a hat and a綾子"></div></div>
<p> Je fais de la bouillie pour mes petit <img src="" alt="A person with a hat and a綾子, sitting on a chair, having a small plate of food and a cup of tea"></p>
<p> Pour un cochon, pour deux cochons, po <b>comments</b></p>
<div class="comment">Et pour une poule su <img src="" alt="A person with a hat and a綾子, sitting on a chair, having a small plate of food and a cup of tea"></div>
<div class="comment-author">Emma, 22 ans, <img src="" alt="A person with a hat and a綾子, sitting on a chair, having a small plate of food and a cup of tea"></div>
<div class="comment">Je vois que vous êtes <img src="" alt="A person with a hat and a綾子, sitting on a chair, having a small plate of food and a cup of tea"></div>
<div class="comment-author">Michel, 56 an</div>
</html>
```

```html
<name attr.name="attr.value"> text </name>
html_name( ) => name
html_text( ) => text
html_attrs( ) => attr.name
"attr.value"
html_attr( , "attr.name") => "attr.value"
```

## Extract the content of nodes or nodesets:

```html
html_nodes/html,"b") %>%
html_text()
## [1] " INGREDIENTS" "Commentaires"
```
<figure><img src="image_448.png" /></figure>

After that, into Terminal of Docker Toolbox you see this folders.

```
docker images
$ ls -l
total 1217
drwxr-xr-x 1 reyse 197609 0 sept. 21 17:43 docker-images/
drwxr-xr-x 1 reyse 197609 0 sept. 22 11:54 docker-scripts/
drwxr-xr-x 1 reyse 197609 0 sept. 21 12:20 docker-test/
drwxr-xr-x 1 reyse 197609 0 sept. 18 18:29 images/
-rw-r--r-- 1 reyse 197609 1212081 sept. 21 17:30 scrap_fliightradar.html
-rw-r--r-- 1 reyse 197609 19370 sept. 21 17:57 scrap_fliightradar.Rmd
-rw-r--r-- 1 reyse 197609 205 sept. 20 15:34 scrap_fliightradar.Rproj
```

Go to `docker-images` folder using `cd` command, and load the two images:

```
docker load --input=r-alpine.tar
docker load --input=rSelenium.tar
```

```
docker images
$ docker load -i r-alpine.tar
cd7100a724140: Loading layer [==================================================>] 4.403MB/4.403MB
385866c7f983: Loading layer [==================================================>] 290.1MB/290.1MB
Loaded image: artemklevtsov/r-alpine:latest

reyse@INSMOUTH MINGW64 /c/Program Files/Docker Toolbox
$ docker load -i rSelenium.tar
0a42ee6cecb: Loading layer [==================================================>] 118.8MB/118.8MB
c2af38e6b250: Loading layer [==================================================>] 15.87kB/15.87kB
6e95929b2798: Loading layer [==================================================>] 14.85kB/14.85kB
2166dba7c95b: Loading layer [==================================================>] 5.632kB/5.632kB
bcff331e13e3: Loading layer [==================================================>] 3.072kB/3.072kB
5eba6110ccc4: Loading layer [==================================================>] 3.072kB/3.072kB
70f675152cda: Loading layer [==================================================>] 152.4MB/152.4MB
0e3b1af8adf7: Loading layer [==================================================>] 1.304MB/1.304MB
8c63cb20b8cd: Loading layer [==================================================>] 353.8kB/353.8kB
385f1fee572c: Loading layer [==================================================>] 23.25MB/23.25MB
48b1b5ef5604: Loading layer [==================================================>] 103MB/103MB
1ab56b003e8b: Loading layer [==================================================>] 22.58MB/22.58MB
402ffa13597a: Loading layer [==================================================>] 38.79MB/38.79MB
195cbc461a59: Loading layer [==================================================>] 5.632kB/5.632kB
738eacc64f56: Loading layer [==================================================>] 280.3MB/280.3MB
cfbe9d798bbe: Loading layer [==================================================>] 10.37MB/10.37MB
5bfc0c2f26ee: Loading layer [==================================================>] 3.584kB/3.584kB
cf2e1eac6e59: Loading layer [==================================================>] 8.704kB/8.704kB
9e56646bb1c7: Loading layer [==================================================>] 3.584kB/3.584kB
Loaded image: selenium/standalone-firefox:3.14.0-arsenic
```

## BUILD image

Go to `docker-scripts` folder into the folder which contain this tutorial on your disk.

The building of this image take lot of times (ten minutes), this is due to the huge `dplyr` library. Run the `docker build` command in the folder which contain the `Dockerfile` description of the image.
## Extract the attributes of some elements

```html
<html>
<style>
h1 {background-color: powderblue;}
.image {margin-left: 50%;}
 comment{border-style:solid; background-c
 .comment-author{font-style: italic;}
</style>
<h1> MA VIE A LA FERME </h1>
<div class="ingredients">
 <b> INGREDIENTS </b>
 <ul>
  <li> >1 cochon(s) </li>
  <li> >1 légume(s) </li>
 </ul>
</div>
<div class="right"><div class="image"><im
 <p> Je fais de la bouillie pour mes petit
 <p> Pour un cochon, pour deux cochons, po
 <b> comments </b>
 <div class="comment">Et pour une poule su
 </div>
<div class="comment-author">Emma, 22 ans,
<div class="comment">Je vois que vous êtes
<div class="comment-author">Michel, 56 an
</html>
```

```html
<name attr.name="attr.value"> text </name>
html_name( ) => name
html_text( ) => text
html_attrs( ) => attr.name
"attr.value"
html_attr( , "attr.name") => "attr.value"
```

## Extract the attributes of nodes or nodesets:

```html
html_nodes/html, "div") %>%
html_attrs()
## [[1]]
##       class
## "ingredients"
##
## [[2]]
##       class
## "right"
##
## [[3]]
##       class
## "image"
##
## [[4]]
##       class
## "comment"
##
## [[5]]
##       class
## "comment-author"
```
```
docker build . --tag=rflightscraps
```

## LAUNCH container

We use a **bound volume**, this is the easiest way actually.

First, create a new folder named `localbackup` into your users folder on windows : `C:\Users\yourname` After that, change the path by yours in this command and run it.

```
docker run --name rflightscraps -d -e UID=1000 -e GID=1000 --mount type=bind,source=/c/Users/reyse/localbackup,destination=/usr/local/src/flight-scrap/docker-scripts/data rflightscraps
```

The end, close the session !

```
remDr$close()
```
##
## [[6]]
## class
## "comment"
##
## [[7]]
## class
## "comment-author"
# Rectangular format, and function-ization

<figure><img src="image_11.png" /></figure>

Extract data and make it into a table:

```json
page="data/blog_de_ginette.htm"
html=read_html(page, encoding="UTF-8")
texte=html %>% html_nodes(".comment") %>%
auteur=html %>% html_nodes(".comment-auth
tib_commments=tibble(texte,auteur)
tib_commments
```

## # A tibble: 2 x 2

## texte

## <chr>

## 1 Et pour une poule sur un mur qui pic

## 2 Je vois que vous êtes, telle la peti

It is actually a good idea to make all this into a fonction that would have the page's url as input and the tibble as output:

```javascript
extract_commments=function(page) {
  html=read_html(page, encoding="UTF-8")
  texte=html %>% html_nodes(".comment")
  auteur=html %>% html_nodes(".comment-tib_commments=tibble(doc=rep(page,leng
texte,
auteur)
  return(tib_commments)
}
```

```javascript
extract_commments("data/blog_de_ginette.")
```

## # A tibble: 2 x 3

## doc texte

## <chr> <chr>

## 1 data/blog_de_ginette.htm Et pour u

## 2 data/blog_de_ginette.htm Je vois q

```javascript
extract_commments("data/blog_de_jean-mar")
```

## # A tibble: 6 x 3

## doc texte

## <chr> <chr>

## 1 data/blog_de_jean-marc.htm Les tho
## 2 data/blog_de_jean-marc.htm Pourquo

## 3 data/blog_de_jean-marc.htm Tout ça

## 4 data/blog_de_jean-marc.htm Je préf

## 5 data/blog_de_jean-marc.htm On ne c

## 6 data/blog_de_jean-marc.htm Et pend
Iteration

<figure><img src="image_12.png" /></figure>

Now, let's imagine that we actually have to deal with several pages with a common structure.

We would like to apply `extract_commments()` iteratively to all these pages.

The `purrr` package enables us to **apply a function iteratively** to all elements of a list or of a vector (... of course this is just a more straightforward way to loop through a for structure...).

<figure><img src="image_13.png" /></figure>
Iteration with purrr

pages=c("data/blog_de_ginette.htm",
"data/blog_de_jean-marc.htm",
"data/blog_de_norbert.htm")

list_commments=map(pages, extract_commments
list_commments

## [[1]]
## # A tibble: 2 x 3
##   doc             texte
##   <chr>           <chr>
## 1 data/blog_de_ginette.htm Et pour une
## 2 data/blog_de_ginette.htm Je vois que
##
## [[2]]
## # A tibble: 6 x 3
##   doc             texte
##   <chr>           <chr>
## 1 data/blog_de_jean-marc.htm Les thons
## 2 data/blog_de_jean-marc.htm Pourquoi
## 3 data/blog_de_jean-marc.htm Tout ça m
## 4 data/blog_de_jean-marc.htm Je préfère
## 5 data/blog_de_jean-marc.htm On ne com
## 6 data/blog_de_jean-marc.htm Et pendan
d

## [[3]]
## # A tibble: 4 x 3
##   doc             texte
##   <chr>           <chr>
## 1 data/blog_de_norbert.htm "A quel mom
## 2 data/blog_de_norbert.htm Norbert, mo
## 3 data/blog_de_norbert.htm L'ambiance
## 4 data/blog_de_norbert.htm Vous devez

tibtot_commments <- list_commments %>%
  bind_rows()
tibtot_commments

## # A tibble: 12 x 3
##   doc             texte
##   <chr>           <chr>
## 1 data/blog_de_ginette.htm Et pou
## 2 data/blog_de_ginette.htm Je voi
## 3 data/blog_de_jean-marc.htm Les th
## 4 data/blog_de_jean-marc.htm Pourqu
## 5 data/blog_de_jean-marc.htm Tout ç
## 6 data/blog_de_jean-marc.htm Je pré
## 7 data/blog_de_jean-marc.htm On ne
## 8 data/blog_de_jean-marc.htm Et pen
## 9 data/blog_de_norbert.htm "A que
## 10 data/blog_de_norbert.htm Norber
## 11 data/blog_de_norbert.htm L'ambi
## 12 data/blog_de_norbert.htm Vous d
# Manipulate strings: package stringr

Package **stringr** Blog post (in French) [here]().

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>
# Strings: concatenate, replace pattern

`str_c()` to combine strings

<figure><img src="image_27.png" /></figure>

str_c("abra", "ca", "dabra")

## [1] "abracadabra"

str_c("Les jeux", "de mots laids", "sont po")

## [1] "Les jeux de mots laids sont pour"

`str_detect()` detects a pattern

<figure><img src="image_28.png" /></figure>

str_detect(c("Quarante", "carottes", "cru
"croient", "que", "croquer",
"crée", "des", "crampes."),
pattern="cr")

## [1] FALSE FALSE TRUE TRUE FALSE T
# Strings: replace pattern, extract pattern

`str_replace()` replaces a pattern with another

<figure><img src="image_29.png" /></figure>

```python
str_replace(c("All we hear is",
"Radio ga ga",
"Radio goo goo",
"Radio ga ga"),
pattern="goo",
replacement="ga")
```

```python
## [1] "All we hear is" "Radio ga ga"
```

`str_extract()` extracts the pattern (if it's there!)

<figure><img src="image_30.png" /></figure>

```text
str_extract(c("L'âne", "Trotro", "trotte"
             pattern="tr")
```

```text
## [1] NA "tr" "tr" NA "tr" "tr"
```
# Regular expressions

Regular expressions are used to define patterns through **rules of construction**. A tutorial here

## Character classes and groups

<table><tr><td><pre><code>.</code></pre></td><td>any</td><td><pre><code>|</code></pre></td><td>or</td></tr><tr><td><pre><code>...</code></pre></td><td>list</td><td><pre><code>(...)</code></pre></td><td>grouping</td></tr><tr><td><pre><code>[^...]</code></pre></td><td>excluded</td><td><pre><code>\1</code></pre></td><td><pre><code>\2</code></pre></td><td>...</td></tr><tr><td><pre><code>[...]</code></pre></td><td>range</td><td></td><td></td><td>backreference</td></tr></table>

## Quantifiers

<figure><img src="image_31.png" /></figure>

## Predefined character classes

<figure><img src="image_32.png" /></figure>

## Anchors

<figure><img src="image_33.png" /></figure>

## Look-arounds

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>
# Regexp: character classes and groups

```
.      any          |      or
[...]  list         (...) grouping
[^...]  excluded      \1 \2 ...
[......]  range       backreference
```

A **character class** corresponds to the notation `[...]`.

For instance, to detect all voyels in the string:

```
str_view_all("youp la boum",
             "[aeiou]")
```

youp la boum

See the difference:

```
str_view_all("A132-f445-e34-C308-M9-E18
"[308]")
```

A132-f445-e34-C308-M9-E18

```
str_view_all("A132-f445-e34-C308-M9-E18
"308")
```

A132-f445-e34-C308-M9-E18

**Any character** can be noted ..

For instance, the pattern "any character followed by a letter" can be searched through

```
str_view_all("32a-B44-552-98eEf",
             ".[a-z]")
```

32a-B44-552-98eEf
<figure><img src="image_37.png" /></figure>
# Regexp: special characters

<figure><img src="image_38.png" /></figure>

To find dots, question marks, exclamation marks:

```c
str_view_all(c("Allô, John-John? Ici Joe
"[\\.\\?\\!]"))
```

<figure><img src="image_39.png" /></figure>

```
. (as we saw before),
as well as ? and ! are
special characters.
So, to point out actual
dots or
interrogation/exclamation
marks, one has to use
the escape character
\.
The regular
expression hence
becomes [\.\?\!]
...
```

Allô, John-John? Ici Joe ta frite! Surprise!

Note that we don't write "[.?!]", but "[\\.\\?\\!]".

But it does not end here... as it is not *directly* the regular expression that is passed to the function, but a *string* that is *interpreted as a regular expression*. Thus each escape character \ has to be escaped through a \.
So that the pattern
passed to the function
is actually
" [\\.\\?\\!] ".
# Regexp: excluded characters, character ranges

A character class can be defined as all characters **excluding the ones listed**. This is noted as `[^...]`:

For instance, all characters that are neither a vowel nor a blank space:

`str_view_all("turlututu chapeau pointu", "[^aeiou ]")`

`turlututu chapeau pointunains`

Ranges of characters are noted `[......]`

For instance, all numbers between 1 and 5:

`str_view_all(c("3 petits cochons", "101 "[1-5]"))`

3 petits cochons

101 dalmations

... or all those between A-F or a-e:

`str_view_all("A132-f445-e34-C308-M2244- "[A-Fa-e]")`

A132-f445-e34-C308-M2244-Z449-E18
<figure><img src="image_40.png" /></figure>
# Regexp: predefined classes

Some character classes are predefined for instance **digits**, **punctuation characters**, **lower-case alphabetical characters**, etc.

<figure><img src="image_41.png" /></figure>
# Regexp: quantifiers

Quantifiers are used to specify **how many consecutive times** a particular class or group occurs.

<figure><img src="image_42.png" /></figure>

**zero or one**: the pattern of interest is followed by ?.

<figure><img src="image_43.png" /></figure>

file1990-fileB1990-fileAbis2005

**zero or more**: O the pattern of interest is followed by *.

<figure><img src="image_44.png" /></figure>

file1990-fileB1990-fileAbis2005

**one or more**: the pattern of interest is followed by +.

<figure><img src="image_45.png" /></figure>

file1990-fileB1990-fileAbis2005
# XPath

XPATH is another way to query the DOM, it's very powerfull, but also a little complex. Use some **cheatsheet** to remember principal operators.

## Some examples :

```xml
html_nodes/html, xpath = "//div[@class='ingredients']//ul//li"]
## {xml_nodeset (2)}
## [1] <li> &gt;1 cochon(s) </li>
## [2] <li> &gt;1 légume(s) </li>
html_nodes/html, xpath = "//text()[contains(.,'cochons') and not(contains(.,'poule'))]"
## {xml_nodeset (2)}
## [1] Je fais de la bouillie pour mes petits cochons.
## [2] Pour un cochon, pour deux cochons, pour trois cochons, pour quatre, ...
html_nodes/html, xpath = "//div[@class='comment']"
## {xml_nodeset (2)}
## [1] <div class="comment">Et pour une poule sur un mur qui picoterait du ...
## [2] <div class="comment">Je vois que vous êtes, telle la petite poule ro ...
```
# A 🐱 and 😊 game (1)

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

## How to defend from webscraping :

Legend : {★: cost, difficulty to implement} , {★: difficulty to bypass} , { 😊 or 😞 : user happiness}

- change or randomize some internal structure of HTML pages (★★ / ★★★ / 😊)

- sample, limits data displayed in time/quantity (★★
★★ / ★★ / 😞)

- use really weird Captcha (★ / ★ / 😞)

- insert all informations into image, remove textual information (★ / ★★ / 😞)

- ask authentification before display any informations
(★ / ★ / 😊)

- add a signature invisible in the content (honey pot)
(★★/ ★★/ ★★/ 😊)

- slowing at infinite the download of the content (★★
/ ★★★ / 😊)
* device/browser fingerprinting (★ ★ / ★ / 😊)

* pay company to do part of this, ex: cloudflare (★ / ★ / 😊)
# A 😂 and 😂 game (2)

## How to attack :

One rule : If you see it on your browser, so you can get it

## Basics :

* using lot of slave people in parallel (joke!?)

* analyze webpage structure and network data (search XHR/API)

* use random (for time delay, query, location, etc.)

* use random user-agent

## Advanced :

* use your imagination (limited to 1000 result for any reason ? => np, find another query with no limitation)

* use OCR / AI (image to text ? no problem !)

* use Proxy/VPN or TOR network (fast Hiding ! fast Jumping !)
* use cookie

* pay for online tools
made by others

* ask dev community

* use (headless)
browsers (to
simulate human
behavior)

* use a framework
with lot of options
(Scrapy & Python !)

* use a framework +
lot of headless
browser on the
cloud (an army of
false human)
# Uses cases

<figure><img src="image_48.png" /></figure>

* First tutorial
  (beginer), open
  project
  projet_leboncoin.Rproj
  or directly the html
  page online

<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /></figure>

* Second tutorial
  (advanced), open
  project
  scrap_flightradar.Rmd
  or directly the html
  page online

**!Warning!** You need
Docker installed

<figure><img src="image_51.png" /></figure>