Data Science(lecture)- Simona Vasilache University of Tsukuba

“Data Science” course at UT• Mandatory course for all freshmen• 10 weeks, double-period class (75 min x 2)
• Each class: two parts - lecture and exercises• English version (for undergraduate Englishprograms)

Course overview and keywords
WeekOverview

Keywords

1

Overview of Data Science

course overview, video lecture (“Introduction of Data Science”)

2

Data Type and Collection (1)data types and scale levels, data collection and survey methods,
basic usage of Excel

3

Data Type and Collection (2)basic process of data science, data preprocessing, datareusability; Excel (statistics)

4

Data Type and Collection (3)data management, separation of document structure from visual
representation; Excel (data operation)

5

Advanced Data Management

video lecture (“Big Data”)

6

Data Visualization and Qualitative significance and purpose of visualization, choice of visual
Data Analysis
representations, visualization of data with Excel

7

One Variable Quantitative Data

8

Two Variable Quantitative Data and understanding two variable quantitative data, correlation andTime Series Datacorrelation coefficient, time series data (regression line)

9Analysis of Actual of Data

10 Advanced Data Management

understanding one variable quantitative data, histograms,
descriptive statistics; statistical tests

analysis of various data, network data, correlation and causality
video lecture (“Artificial Intelligence”)

Visualizing Data
Data Science(lecture)Goals of the lecture• Data visualization– Understand significance of visualization and thevisualization process– Learn how to choose visual attributes and visual
representations (graphs)– Understand suitable / unsuitable graphs– Learn how to draw basic graphs in Excel• Analysis of qualitative data
– Categorize and understand frequency distribution table– Understand qualitative data statistics– Understand and utilize cross tabulations and odds ratio– Understand stacked bar charts
5Significance of visualization andVisualization processing model

6

Famous example: Anscombe’s quartet• Four data sets with almost identical simple descriptivestatistics• Proposed in 1973 by Francis Anscombe (English statistician)• Purpose: to show importance of visualization andcounteract some common beliefs- “Numerical calculations are exact, but graphs arerough”- “Performing intricate calculations is virtuous, whereas
actually looking at the data is cheating”

“Anscombe’s Quartet”
Ixy10.0 8.04
8.0 6.95
13.0 7.58
9.0 8.81
11.0 8.33
14.0 9.96
6.0 7.24
4.0 4.26
12.0 10.84
7.0 4.82
5.0 5.68

IIx
10.0
8.0
13.0
9.0
11.0
14.0
6.0
4.0
12.0
7.0
5.0

III
y
9.14
8.14
8.74
8.77
9.26
8.10
6.13
3.10
9.13
7.26
4.74

xy
10.0 7.46
8.0 6.77
13.0 12.74
9.0 7.11
11.0 7.81
14.0 8.84
6.0 6.08
4.0 5.39
12.0 8.15
7.0 6.42
5.0 5.73

IVx
8.0
8.0
8.0
8.0
8.0
8.0
8.0
19.0
8.0
8.0
8.0

y
6.58
5.76
7.71
8.84
8.47
7.04
5.25
12.50
5.56
7.91
6.89

Source: F. J. Anscombe, Graphs in Statistical Analysis, The American Statistician, Vo. 27, No. 1, pp. 17-21, 1973 8

Features of the four datasets• Each dataset has 11 pairs– mean of x: 9.0– variance of x: 11.0
– mean of y: 7.5– variance of y: 4.1– correlation coefficient : 0.82

128?

410• Mean, variance, correlation coefficient are the samefor all datasets• Q: Are all four datasets the same? What is thedifference between them?
9Features of the four datasets

Scatter plot visualization• Example: visualization ofkg
quantitative data for two
Weight 90
variables - height and weight
80

Quantitative data
Target(two variables):Data
Height, Weight• Record ⇒ Dot• Variable 1 (Height) ⇒Horizontal coordinates• Variable 2 (Weight) ⇒ Verticalcoordinates

70
60
50
140

150

160

170

180cm

Height10

Quiz: Find corresponding scatter plot foreach of the 4 Anscombe’s data sets
(data given in excel file on manaba)

14.00

14.00

(a)

12.00
10.00

10.00

8.00

8.00

6.00

6.00

4.00

4.00

2.00

2.00

0.00

0.00
0.0

14.00

5.0

10.0

15.0

20.0

0.0

14.00

(c)

12.00

10.00

8.00

8.00

6.00

6.00

4.00

4.00

2.00

2.00

0.00

0.00
5.0

10.0

15.0

20.0

5.0

10.0

15.0

20.0

5.0

10.0

15.0

1120.0

(d)

12.00

10.00

0.0

(b)

12.00

0.0

Visualization processing model• Generalized processing flow in visualization
Data
Raw Data

Visual Form

Processed DataData Transformation

VisualStructure

Visual Mapping

View

View Transformation

Human Interaction
Source:Figure 1.23 in Stuart K. Card et al., Readings in Information Visualization: Using Vision to Think, Morgan Kaufmann, 1999.

12

Components of visual representation

VISUAL ATTRIBUTES

13

Representing data
80

80

60

70

40

60

20

180

cm

02000

2002

2004

年

0

golf

170

football160

tennis150

baseball50
140

Popularitykg
90

14

What is used to represent values?
• What is used torepresent values in thegraph?
• Hue (blue, magenta)• Size of circles (area)• Horizontal positions• Vertical positions

kg90

80
70

6050
140

150

160

170180cm

15

Visual Attributes
Visual attributes that can represent values
Position y0Huex

TextureSize (area, length, width)

Orientation

Brightness

Shape16

Nota Bene• The graph consists of a combination of basic shapes(e.g. points) and visual attributes (e.g. positions)• The importance of clarifying the rules cannot be
underestimated!
• In order for the visual representation to convey
information, the author and the reader must sharethe same rules!

ANALYSIS OF QUALITATIVE DATA

(one qualitative variable)- Categorization, frequency distribution table- Bar chart, stacked bar chart- Statistics of qualitative data18

Categorization (1)• Categorizing records by a qualitative attribute– Count records with the same value when onlyinterested in the number of records (frequency)• E.g. Number of people who like a certain kind of sportFrequency table
Judo

FootballMarathon
FootballBaseballKarateJudo

Categorization

Class
BaseballFootballMarathonJudo
KarateFrequency
1212
1
19

Categorization (2)• Categorizing records by a qualitative attribute– May be grouped into general categories• E.g. Baseball, Football → Ball game
Frequency table
JudoFootballMarathon
FootballBaseballKarateCategorization

ClassBall game
Athletic
Martial arts

Frequency
313Judo20

Frequency and relative frequency (ratio)• Frequency– Number of occurrences of each observed value• Relative frequency (ratio)– Percentage of number of occurrences (frequency) of eachobserved value when the entire data (no. of cases) is used as
denominator• Cumulative relative frequency– Sum of relative frequencies up to the observed value(cumulative sum)
Category

Frequency

Relativefrequency

Cumulative relative
frequency

Ball game

3=3/7(≒0.43)≒0.43

Athletic

1=1/7(≒0.14)≒0.43+0.14=0.57

Martial arts

3=3/7(≒0.43)≒0.57+0.43 = 121

Remaining points…
• Data visualization– Understand significance of visualization and thevisualization process– Learn how to choose visual attributes and visual
representations (graphs)– Understand suitable / unsuitable graphs– Learn how to draw basic graphs in Excel• Analysis of qualitative data
– Categorize and understand frequency distribution table– Understand qualitative data statistics– Understand and utilize cross tabulations and odds ratio– Understand stacked bar charts
22

Thank you for listening!