WhyiloveR: Learn 1 Thing to Lean Toward Data Science-Dplyr & More
By: Mr Ding
Tags: Database, Data Cleaning, Data Science, Dplyr, Beginner

The first writing for #whyiloveR series

There is a tongue of reasons why I would recommend dplyr packages to new R users instead of the base packages. The package is created with the objective for the newbie to quickly master data manipulation(so-called “cleaning”) and start to do cool things with it. The benefit includes The intuitive coding syntax and pipe operator %>%. With dplyr, I have transformed smoothly from a SQL programmer to an R programmer. Today, I am introducing the extra hidden benefit of learning dplyr that some senior programmer may not know.

Here are three scenarios you can use dplyr for data manipulation:

1, Preparing data (most common scenario)

2, Querying data in a database

3, Preparing ”Big Data” with Spark running behind

0) Data Use

This data is downloaded the same way as you download and load a package:

install.packages(“nycflights”)

library(nycflights)

weather_df <- nycflights13::weather

1）Preparing data (a most common scenario)

Reference from Restudio official website https://dplyr.tidyverse.org/

First, we import the library:

install.packages(“dplyr”)

library(dplyr)

or

install.packages(“tidyverse”)

library(tidyverse)

Tidyverse is a family of packages published by RStudio. Dplyr is one of them. By importing tidyverse library, dplyr is imported as well.

First, it is a good habit to use glimpse to check the data structure and data types:

> glimpse(weather_df)

Observations: 26,115

Variables: 15

$ origin <chr> "EWR", "EWR", "EWR", "EWR", "EWR", "EWR", "EWR", "EWR", "EWR", "EWR", ...

$ year <dbl> 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013...

$ month <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1...

$ day <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2...

$ hour <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21,...

$ temp <dbl> 39.02, 39.02, 39.02, 39.92, 39.02, 37.94, 39.02, 39.92, 39.92, 41.00, ...

$ dewp <dbl> 26.06, 26.96, 28.04, 28.04, 28.04, 28.04, 28.04, 28.04, 28.04, 28.04, ...

$ humid <dbl> 59.37, 61.63, 64.43, 62.21, 64.43, 67.21, 64.43, 62.21, 62.21, 59.65, ...

$ wind_dir <dbl> 270, 250, 240, 250, 260, 240, 240, 250, 260, 260, 260, 330, 280, 290, ...

$ wind_speed <dbl> 10.35702, 8.05546, 11.50780, 12.65858, 12.65858, 11.50780, 14.96014, 1...

$ wind_gust <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 20.71404, NA, ...

$ precip <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...

$ pressure <dbl> 1012.0, 1012.3, 1012.5, 1012.2, 1011.9, 1012.4, 1012.2, 1012.2, 1012.7...

$ visib <dbl> 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10...

$ time_hour <dttm> 2013-01-01 01:00:00, 2013-01-01 02:00:00, 2013-01-01 03:00:00, 2013-0...

Checking Null value:

> weather_df %>% summarise_all(funs(sum(is.na(.)))) %>% gather

# A tibble: 15 x 2

key value

<chr> <int>

1 origin 0

2 year 0

3 month 0

4 day 0

5 hour 0

6 temp 1

7 dewp 1

8 humid 1

9 wind_dir 460

10 wind_speed 4

11 wind_gust 20778

12 precip 0

13 pressure 2729

14 visib 0

15 time_hour 0

Checking the distinct value in origin columns:

> weather_df %>% select(origin) %>% table

.

EWR JFK LGA

8703 8706 8706

Let’s try to count the number of row with some conditions:

> #count the number of day where the temp is between the avereage temperture and the median temperture

> weather_df %>%

+ select(temp) %>%

+ filter(temp >= mean(temp,na.rm=TRUE) &

+ temp <= median(temp,na.rm=TRUE)

+ ) %>%

+ summarise(weather_between=n())

# A tibble: 1 x 1

weather_between

<int>

1 41

With a similar case, but this time we use the case_when operator inside the mutate function to count three types of data groups.

> #count the number of day where the temp is split by mean and medium temperture

> weather_df %>%

+ mutate( split_by_temp =

+ case_when(temp < mean_temp ~ 'lower_than_mean',

+ temp >= mean_temp &

+ temp <= median_temp ~ 'between',

+ temp > median_temp ~ 'higher_than_median')

+ ) %>%

+ group_by(split_by_temp) %>%

+ summarise(count=n())

# A tibble: 4 x 2

split_by_temp count

<chr> <int>

1 between 41

2 higher_than_median 13019

3 lower_than_mean 13054

4 NA 1

2, Querying data in a database

Reference from Restudio official website https://db.rstudio.com/dplyr/ https://blog.rstudio.com/2017/06/27/dbplyr-1-1-0/

In you R interface(for example Rstudio), it looks like you are still using R to do all the calculation. But that is partially true, you are actually writing some SQL code for querying database, most of the calculation will be done in the database application and export to Rstudio for you.

To use databases with dplyr , you need to first install dbplyr :

install.packages("dbplyr")

library( dbplyr )

Then you create a connection call con, providing the database backend, the hostname, the username and the password.

con <- DBI::dbConnect(RSQLite::SQLite(),

host = "database.com",

user = "username",

password = rstudioapi::askForPassword("Database password") )

Five commonly used backends are: RMySQL connects to MySQL and MariaDB RPostgreSQL connects to Postgres and Redshift. RSQLite embeds a SQLite database. odbc connects to many commercial databases via the open database connectivity protocol. bigrquery connects to Google’s BigQuery.

from: https://db.rstudio.com/dplyr/

You can use the copy_to function to import data into the connected database, but most of the time. The best part about doing so is the ease of assigned data type according to data frame datatypes.

copy_to(con, #choose the connection

weather_df, #choose the data

“weather”, #assign table name

temporary = FALSE,

indexes = list(...) #the colomn name

Now we can assign an object to take a reference of a table on the database, you can treat this object like datasets, but actually, they are writing SQL query for you to do the work.

weather_db <- tbl(con, #connection name

“weather”)#table name

flights_db <- tbl(con,

"flights") > #count the number of day where the temp is between the avereage temperture and the median temperture

> weather_df %>%

+ select(temp) %>%

+ filter(temp >= mean(temp,na.rm=TRUE) &

+ temp <= median(temp,na.rm=TRUE)

+ ) %>%

+ summarise(weather_between=n())

The querying work fine, you are writing dplyr syntax instead of SQL but you can get the result you want. Personally, writing dplyr syntax is so much faster for me. There are many similar functions like case_when() in dplyr must have got the idea from CASE in SQL.

However, I have not seen any resources mentioned using dplyr to join two tables directly from the database. In addition, you can only use the available dplyr verbs against the tables, while the R base functions will not work. For both, you will need to use collect() function to store your table in R and apply the calculation in R instead.

3, Preparing ”Big Data” with Spark running behind

Reference from Restudio official website https://spark.rstudio.com/

In you R interface(for example Rstudio), it looks like you are still using R to do all the calculation. But that is partially true, you are actually running Spark behind, most of the calculation will be done in the Spark, with the result exported to Rstudio for analysis and visualization. Quoted from the official website, the Sparklyr allows you to “Create extensions that call the full Spark API and provide interfaces to Spark packages.”

To use play around “big data” with dplyr , you need to first install sparklyr and install spark into your computer:

install.packages("sparklyr")

library(sparklyr)

library(dplyr)

sparklyr::spark_install()

Making a connection:

con <- spark_connect(master = "local")

The code above connects to a local spark machine, ideally, you are using spark for parallel computing and enhancing your computing capability for “big data”. For more info of how to connect to your spark see here: https://spark.rstudio.com/deployment/

Copy data:

flights_tbl <- copy_to(con, #choose the connection

nycflights13::flights, #choose the data

“flights”) #assign a name for the cluster in spark

Again, now just treat flights_tbl as a datasets in R. Just remember that we have the same constraint: You can only use the available dplyr verbs against the tables, while the R base functions will not work. However, the join functions provided by dplyr all works pretty well.

See you all next time~

Stay tuned for a detailed example of using sparklyr in my blog very soon.

If you like this Blog feel free to followed my account and clap it page.