Python for data science

Alexandre Chevallier, Jérémy Richard, Geneviève Michaud, Baptiste Rouxel

► To cite this version:

Alexandre Chevallier, Jérémy Richard, Geneviève Michaud, Baptiste Rouxel. Python for data science. Doctoral. Summer School 2017, Trivedi Centre for Political Data (TCPD) Ashoka University, India. 2017. hal-03923285

HAL Id: hal-03923285

https://sciencespo.hal.science/hal-03923285

Submitted on 4 Jan 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Python for data science
# Introduction to Python and Jupyter

Trivedi Center for Political Data (TCPD) 2017 Summer School,
Technical module, July 11th 2017

Alexandre Chevallier (CDSP, SciencesPo and CNRS)

Jérémy Richard (SciencesPo)
# Outline

Introduction

Adopt Python

Jupyter
## Who are we?

Jérémy Richard, Sysadmin engineer.

Joined Sciences Po in 2012.

Missions: Building IT infrastructures for two laboratories of the scientific department:

* CDSP - Center For Socio-Political Data

* Médialab
# Who are we?

Alexandre Chevallier, Web Developer.

Joined the CDSP of Sciences Po in 2014

Missions: Develop tools/applications for several projects, among which:

* ELIPSS project (Internet online panel for the Social Sciences)

* Bequali (Qualitative data bank)

* support for CDSP data and metadata related activities
## Who are we?

- CDSP - Center for Socio Political Data

- CNRS and Sciences Po mixed service unit for social science researchers

- Helps researchers with their data and metadata, collect, curate and disseminate data and metadata.

- Involved in several projects such as:

- ELIPSS (Digital panel for quantitative surveys)

- beQuali (Archive & share qualitative studies)

- Vizlab (Visualization for French election data)
# Why are we here?

Dispelling a few myths

* Myth 01: I have to go to university to learn how to code.

* Myth 02: I need to be a genius to code

Share IT knowledge

Helping you decide if our techs suit your use
# Adopt Python - Brief

- Invented by Guido van Rossum, in the Netherlands early 90s

- Python is a general purpose open source programming language

- Often used as a scripting tool

- It could be also called an interpreted language (≠ compiled language)

- Swissknife language
# Adopt Python - Scope

## Data science

* Data analysis

* Data visualization

## System information

* Scripting

* Deployed on most of Unix systems

## Software development

* Web applications

* Testing scripts

## Education

* Teaching programming
# Adopt Python - Why use it ?

## Object-oriented

### Native indentation restriction

* Very clear readable syntax

### Powerful

* Very high level data types

* Other language interoperation (e.g. C/C++)

* Python Package Index (pip)

## Community support

* Open-Source

* Portable

* Free
## Adopt Python - Data Structures

- List → [1, 2, 3, 4, 5, “hello”] : Ordered series of values

- add data list.append(1)

- Dictionary → { “key” : “value”, “hello” : 1 } : Key/Value data structure

- add data dict['key'] = 'value'

- Tuple → (1, 2, 3, 4, “hello”) : Like list but immutable
# Jupyter - Python libraries we use

## Web Scraping: **BeautifulSoup**

* HTML/XML Parser

* Built on top of popular Python parsers like `xml` and `html5lib`

* Designed for quick turnaround projects

## Data Visualization: **Pandas**

* Similar to R, MATLAB, SAS

* Built on top of `Numpy`, `Scipy` and `Matplotlib`

* Handle a vast majority of typical use cases in finance, statistics, social science, and many areas of engineering
# Jupyter - Brief

Web platform for Data Science.

Support for over 40 programming languages.

IPython is an interactive shell for Python.

Create and share notebooks that contains:

* live code

* equations

* visualizations

* text
# Jupyter - Notebooks

Interactive way to learn, experiment and share your work.

Tool of choice for Data scientists.

Runs in every recent web browser.

<figure><img src="image_2.png" /></figure>
# Jupyter - Notebook: input and output file formats

<figure><img src="image_3.png" /></figure>
Alexandre Chevallier - Jérémy Richard
itcdsp-scpolst@sciencespo.fr
# Python for data science
## #2 Web Scraping

Trivedi Center for Political Data (TCPD) 2017 Summer School,
Technical module, July 11th 2017

Alexandre Chevallier (CDSP, SciencesPo and CNRS)

Jérémy Richard (SciencesPo)
# Outline

- Web scraping

- Web page

- BeautifulSoup Library

- Practical Works
# Web Scraping - What is it?

## Data Scraping?

* Automated process

* Explore and download raw data

* Grab content

* Convert data in usable format for analysis

* Store data in database or text file

Web Scraping = Data Scraping of web pages
# Web Scraping - What is a web page ?

Components of a web page

* HTML - Organizes and contains the main content of a web page

* CSS - Add styling to make the page looks nicer

* JS - Javascript files add interactivity to web pages

* Media files - Images, Sounds, Videos, etc.

Interesting content for web scraping = **HTML**
# Web Scraping - HTML

HTML is used to create documents on the Web

Very simple and logical

NOT a programming language but a **markup** language that uses `<tags>` like this

The websites you view are basically HTML files rendered by web browsers

1<!DOCTYPE html>

2<html>

3 <head>

4 <title>Example<title>

5 <link rel="stylesheet" href="style"/>

6 </head>

7 <body>

8 <h1>

9 <a href="/">Header</a>

10 </h1>

11 <nav>

12 <a href="one/">One</a>

13 <a href="two/">Two</a>

14 <a href="three/">Three</a>

15 </nav>
# Web Scraping - HTML

<figure><img src="image_4.png" /></figure>

HTML is organized like a hierarchical tree
# Web Scraping - Inspect the source

Inspect element

Find HTML nodes

`<table>` defines a table

`<tr>` defines a row in a table

`<th>` defines a table header cell

`<td>` defines a cell in table

Use beautifulSoup to grab it

<figure><img src="image_5.png" /></figure>
# Web Scraping - BeautifulSoup

* A Python library

* Pull out data out of HTML/XML files

* Designed for quick turnaround projects

* Charged with some superb methods

* Open-source, free & well documented
# Web Scraping - Jump into the code

```python
#Grab node with bs
from bs.BeautifulSoup import bs
import urllib

raw_html = 
urllib.urlopen('http://www.elections.in/delhi/mcd-elections/').read()

soup = bs.BeautifulSoup(raw_html)

attrs = { 'class':'tableizer-table' }
tables = soup FindAll attrs=attrs)
table = tables[0]
rows = table FindAll('tr')
```

<figure><img src="image_6.png" /></figure>

Import libraries

<figure><img src="image_7.png" /></figure>

Download data

<figure><img src="image_8.png" /></figure>

Initialize
BeautifulSoup object

<figure><img src="image_9.png" /></figure>

Access the data
Use grabbed data to write a CSV file

# Web Scraping - Jump into the code

```python
import csv
```

```python
with open('export.csv', 'wb') as f:
    writer = csv writer(f, delimiter=';')
for row in rows:
```

```python
csv_row = []
headers = rowærAll('th')
for header in headers:
    csv_row.append (header.text)
cells = row. ALL('td')
for cell in cells:
    csv_row.append (cell.text)
writer. writerow (csv_row)
```

```python
} Import the CSV library
```

```python
} Open a file with write permissions
Handle it with CSV lib's methods
```

```python
} Make loops for selecting data
inside table cells.
Write it in a python list
```

```python
} Write list in CSV handle file
```
# Web Scraping - Jump into the code

<figure><img src="image_10.png" /><figcaption>## Extraction Result</figcaption></figure>

<figure><img src="image_11.png" /></figure>
Alexandre Chevallier - Jérémy Richard
itcdsp-scpolst@sciencespo.fr
# Python for data science
## #3 Data Visualization

Trivedi Center for Political Data (TCPD) 2017 Summer School, Technical module, July 11th 2017

Alexandre Chevallier (CDSP, SciencesPo and CNRS)

Jérémy Richard (SciencesPo)
## Outline

- Pandas library

- Visualization

- Practical works
# Data Visualization - Brief

Pandas - Panel Data System

Used in production in many companies, especially in financial industries

Suitable for many different kinds of data

Two primary data structures:

* Series (1 dimensional)

*DFrame (2 dimensional). For R's users, it's like R's data.frame on steroids.
# Data Visualization - Series

## 1 dimensional

from pandas import Series

data = {'a' : 0., 'b' : 1., 'c' : 2.}

s = Series(data)

print(s)

a 0.0

b 1.0

c 2.0

<figure><img src="image_12.png" /></figure>

- Import pandas library

<figure><img src="image_13.png" /></figure>

- Create python ordered dictionary with data

<figure><img src="image_14.png" /></figure>

- Instantiate Series object

<figure><img src="image_15.png" /></figure>

- Show variable content
Data Visualization - Series

import matplotlib.pyplot as plt
s.plot()
plt.show()

<figure><img src="image_16.png" /></figure>
Data Visualization - Series

s = s.reindex(['c','a','b'])

print(s)

c 2.0

a 0.0

b 1.0

s.plot()

plt.show()

<figure><img src="image_17.png" /></figure>
# Data Visualization - DataFrame

2 dimensional table data structure

Like data.frame in R

Data manipulation with integrated indexing

Support heterogeneous type of columns
# Data Visualization - Frame

#File input/output

import pandas as pd

data = pd.read_csv('2012-electoral-college.csv', sep='; ', index_col='State')

data .head()

<table>
  <thead>
    <tr>
      <th></th>
      <th>Name</th>
      <th>Electors</th>
      <th>Population</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>State</td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td>AK</td>
      <td>Alaska</td>
      <td>3</td>
      <td>710000</td>
    </tr>
    <tr>
      <td>AL</td>
      <td>Alabama</td>
      <td>9</td>
      <td>4780000</td>
    </tr>
    <tr>
      <td>AR</td>
      <td>Arkansas</td>
      <td>6</td>
      <td>2916000</td>
    </tr>
    <tr>
      <td>AZ</td>
      <td>Arizona</td>
      <td>11</td>
      <td>6392000</td>
    </tr>
    <tr>
      <td>CA</td>
      <td>California</td>
      <td>55</td>
      <td>37254000</td>
    </tr>
  </tbody>
</table>

<figure><img src="image_18.png" /></figure>
# Data Visualization - DataFrame

## Analysis

>>> data.Electors.mean()

10.549019607843137

>>> data.Electors.max()

55

>>> data.locinia[ data.Electors .argmax() , 'Name']

'California'

>>> data.Population .sum()

308746000

>>> data['ratio'] =

data['Electors'] /data['Population']

>>> data

State

AK

AL

AR

AZ

[...]

<table><thead><tr><td>Name</td><td>Electors</td><td>Population</td><td>ratio</td></tr></thead><tbody><tr><td>Alaska</td><td>3</td><td>710000</td><td>0.000004</td></tr><tr><td>Alabama</td><td>9</td><td>4780000</td><td>0.000002</td></tr><tr><td>Arkansas</td><td>6</td><td>2916000</td><td>0.000002</td></tr><tr><td>Arizona</td><td>11</td><td>6392000</td><td>0.000002</td></tr></tbody></table>

0.000004

0.000002

0.000002

0.000002
Data Visualization - DataFrame

#Visualization with matplotlib

import matplotlib.pyplot() as plt

data.Electors.bar()

plt.show()

<figure><img src="image_19.png" /></figure>
# Data Visualization - Go further

And much more

* Group By

* Merge, join, aggregation

* Reshaping and Pivot Tables

* Time based series, date functions

* Multi-index

* ...
Alexandre Chevallier - Jérémy Richard
itcdsp-scpolst@sciencespo.fr