powered by

DATA ANALYSIS AND STATISTICS
The purpose of this tutorial is to show that Scilab can beconsidered as a powerful data mining tool, able to perform thewidest possible range of important data mining tasks.
Level

This work is licensed under a Creative Commons Attribution-NonComercial-NoDerivs 3.0 Unported License.

www.openeering.com

Step 1: The purpose of this tutorial
Nowadays it is really frequent to have availability of large amounts of data
describing aspects of our world or work in a deeply detailed way. It could
be difficult to get useful information without the support of data mining, adiscipline that describes the process of extracting meaningful patterns
from these complex data sets.
The purpose of this tutorial is to show that Scilab can be considered as a
powerful data mining tool, able to perform the widest possible range ofimportant data mining tasks.

Step 2: Roadmap
In this tutorial, after a description of the database we are going to use andof the commands used to extract the data, we describe the charts we cancreate in Scilab in order to analyze our data.

A Scilab data mining tutorialwww.openeering.com

Descriptions
Database description
Data extraction
Data mining charts
Conclusions and remarks

Steps3
4
5-13
14-15

page 2/15

Step 3: Database description
The database presented in this tutorial regards the most relevantcharacteristics of the United Nations in terms of population, latitude,longitude, ages, per capita GDP and so on. It has been created byextracting data from the website“http://www.un.org/”
The database collects all the United Nations with any row containing 23
specific elements, listed on the right.

Once the data are well organized in a table, Scilab helps the users ingetting the relations between the data that are not visible at a first glance
due to the quantity of data and/or the high dimensionality of the problem.

1.
2
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.

A Scilab data mining tutorial

www.openeering.com

A unique identification code for the state
Name of the state
Average latitude of the state
Average longitude of the state
Total population (in thousands)
Number of women (in thousands)
Number of men (in thousands)
Number of women every 100 men
Annual population growth rate
Percentage of population under 15 years
Percentage of men over 60 years
Percentage of women over 60 years
Number of men every 100 women of the over 60
population
Number of annual maternal deaths every 100.000 livebirths
Number of annual infants dying before reaching the age of
one year per 1.000 live births
Life expectancy at birth for womenLife expectancy at birth for men
Life expectancy at the age of 60 years for womenLife expectancy at the age of 60 years for men
Total school life expectancy (in years)School life expectancy for men (in years)
School life expectancy for women (in years)
Per capita GDP (in US$)

page 3/15

Step 4: How to extract the data
The first step in data mining is to input raw data in an appropriate way. InScilab, loading and filtering the data is really easy. During the importphase, the user can remove rows and columns containing useless data.

Our dataset is stored in a comma-separated values (CSV) file, which
stores tabular data (numbers and text) in plain-text form.

We load the data using the function csvRead, which returns thecorresponding Scilab matrix of strings or doubles. In particular, typing in
the Scilab Console(Loading data as matrix of doubles)

D = csvRead('data_UN.csv');
we create a matrix of doubles, where data that cannot be read as a double
are substituted with NaN, while typing

S = csvRead('data_UN.csv',',','.', 'string');
we get a matrix in which every data is read as a string.
In the following examples we will always restrict our set of data of type“double” from the second to the last line of the matrix D:

data = D(2:$,:);(Loading data as matrix of strings)

A Scilab data mining tutorial

www.openeering.com

page 4/15

Once our dataset has been loaded, it is possible to select on “VariableBrowser” the cell “data” with a double click, which will show the table withthe data stored as shown in the figure on the right.

In the Variable Editor it is possible to select a subset of the data in thetable and to create automatically a chart, choosing among the available
charts listed by an icon.

(Table of data selected from the Variable Browser)

A Scilab data mining tutorial

www.openeering.com

page 5/15

Step 5: The history plot
We can start analyzing the database in Scilab with the help of available
charts beginning with the simplest chart, the history plot. The figure on theright reports the history plot of the maternal mortality ratio of the 30
poorest states, the red lines point out the incomes gap between thehighest (in Somalia with 1000 deaths every 100000 live births) and lowest(in North Korea with 81) maternal mortality ratio.

This chart points out the peculiarity of this data in North Korea: in spite ofbeing the twelfth poorest state of the world, it boasts of a pretty small
maternal mortality ratio.

// Getting rid of the NaN entries[gdp,k] = thrownan(data(:,23));[rationonan,kk] = thrownan(data(k,14));
gdpnonan = gdp(kk);// History plot[p,i] = gsort(gdpnonan,'g','i');scf(1); clf(1);plot([1:30],rationonan(i(1:30)),'bo-')[m,im] = min(rationonan(i(1:30)));plot([0,im],[m,m],'r')[M,iM] = max(rationonan(i(1:30)));plot([0,iM],[M,M],'r')set(gca(),"grid",[1 1]*color('gray'));set(gca(),"data_bounds",[0 30 0 1200]);
title('Maternal mortality ratio in the poor states');A Scilab data mining tutorial(History plot of the maternal mortality ratio in the poor states)

www.openeering.com

page 6/15

Similarly, we can even plot a multi-history chart, putting in the same plot
the maternal mortality ratio and the infant mortality rate by adding the lineplot([1:30],data(k(kk(i(1:30))),15),'go-')as shown in the figure on the top-right, but this chart does not give us agood comparison between the two lines, because we are plotting in blue
the annual maternal deaths every 100.000 live births, while the green line
represents the number of annual infants dying before reaching the age of
one year per 1.000 live births.
A good solution to compare this two datasets is to normalize them; thisoperation allows us to get the unique information we are interested in: thetrends of the two lines, which are very similar.

(History plot of the maternal and infant mortality ratio in the poorstates)// Data normalization
dnorm = [rationonan(i(1:30)),data(k(kk(i(1:30))),15)];
for i = 1:size(dnorm,2)dmin(i) = min(dnorm(:,i));dmax(i) = max(dnorm(:,i));dnorm(:,i) = (dnorm(:,i)-dmin(i)) ./ (dmax(i)dmin(i));
end// Plotscf(2); clf(2);plot([1:30],dnorm(:,1),'bo-')plot([1:30],dnorm(:,2),'go-')set(gca(),"grid",[1 1]*color('gray'));set(gca(),"data_bounds",[0 30 0 1]);

(History plot of the normalized maternal and infant mortality ratio
in the poor states)

A Scilab data mining tutorial

www.openeering.com

page 7/15

Step 6: The pie chart
In the pie chart a value is associated with the area of a slice of pie,possibly colored, as shown in the figure on the right.
This kind of chart is maybe the most frequent chart in the business world
and the most used by the mass media. Unfortunately with the pie chart it
could be difficult to compare different sections of the pie or to compare
data across different pie charts. These charts can be effective to correlate
the size of a slice with the whole pie, rather than comparing the slices witheach other, e.g., they can be convenient for visualizing percentages.

In the figure on the right we have plotted the size of the population of the
10 biggest states and the population of the rest of the world (41% in lightblue). It is really interesting to see that China (19%) and India (18%)
together cover the 37% of the global population.

We get a pie chart using the Scilab command pie.

// Sorting the population size and plotting the Pie Chart[p,ind] = gsort(data(:,5));
scf(3); clf(3);pie([p(1:10);sum(p(11:$))])
legend([S(ind(1:10)+1,2);"Rest of the world"]);A Scilab data mining tutorial

www.openeering.com(Pie chart of the size of the population of the states)

page 8/15

Step 7: The bar chart
A bar chart consists of rectangular bars with lengths proportional to thevalues that they represent.

In order for the bars to be clearly visible, their number has to be limited.
Each bar is characterized by a label and a length: they can therefore beused for plotting data with a discrete set of labels, while the data assigned
to the length can be continuous. Bar charts are optimal when we want toassociate nominal values along the X axis to numerical values along the Yaxis.

In the figure on the right, using the command bar, we have plotted thepercentage of the global per capita GDP for the 10 states with highest percapita GDP, while the red line shows the cumulative sum of these values:
it points out that these 10 states cover the 30% of the global per capita
GDP.

// Bar Chart and cumulative
scf(4); clf(4);[gdpnonan,i] = thrownan(data(:,23));[gdp,j] = gsort(gdpnonan);
tot_gdp = sum(gdp);
cum_gdp = cumsum(gdp(1:10));bar(gdp(1:10)*100/tot_gdp)
plot([1:10],cum_gdp*100/tot_gdp,'ro-')set(gca(),"grid",[1 1]*color('gray'));h = get("current_entity");h.parent.x_ticks.labels = S(i(j(1:10))+1,1);h.parent.y_ticks.labels = h.parent.y_ticks.labels+'%';A Scilab data mining tutorial(Bar chart and cumulative of the per capita GDP)

www.openeering.com

page 9/15Step 8: Further bar charts
If we want to compare more than one bar chart, the barhomogenize
function allows to homogenize the width and style of all the bars.

In the figures on the right we have plotted for 10 states the percentages of
population composed by under 15 (in yellow), men over 60 (in blue) andwomen over 60 (in red).

In the subplot on the left we have set the option 'stacked'.

(Homogenized bar charts)

Step 9: The bubble chart
In a bubble chart a point is substituted by a “bubble”, a circle whose size
(diameter or area) is proportional to a selected parameter. Finally, byadding a color we get a very powerful and effective way to show four
distinct coordinates on a single plot.
In the figure on the right we have plotted the latitude and the longitude of
the states along the X and the Y axes, while the bubbles are given by thesize of the population of every state. In this way, we get a nice view of theworld. The red bubbles point out the 10 states with the biggest population.

The code is available in the file data_mining.sce.

A Scilab data mining tutorial(Bubble chart of the size of the population of every state, given
the latitude and the longitude)

www.openeering.com

page 10/15

Step 10: Histograms
When dealing with numerical data corresponding to measurements, auseful type of information is related to the data distribution. We might beinterested in knowing if the values are nicely concentrated around acentral value; or it may be interesting to know how many cases are fallingin a given interval.
In a histogram, a simple descriptive analysis can be done by partitioning
the interval between two specific values (usually the minimum andmaximum values) into a set of equally-sized segments, also called bins,
and by counting how many values fall in the different segments, as shownin the figure on the right.

In the figure, using the command histplot, we have plotted the number ofstates that have the life expectancy in the given age groups.

// Histograms
LEW = thrownan(data(:,16));
LEM = thrownan(data(:,17));classes = 45:5:90; lW = length(LEW); lM = length(LEM);
scf(5); clf(5);subplot(2,1,1)
histplot(classes, LEW, normalization=%f, style=5)
xtitle("Life expectancy at birth: Women");subplot(2,1,2)
histplot(classes, LEM, normalization=%f, style=2)
xtitle("Life expectancy at birth: Men");(Histograms of the life expectancy for women and men)

The command hist3d allows to create histograms in three dimensions.

A Scilab data mining tutorial

www.openeering.com

page 11/15

Step 11: The box-and-whisker plot
Also the box-and-whisker plot is useful if we are interested in the datadistribution.
The four horizontal lines of the boxes are the lower quartile Q1, the mean(in red), the mode (in green) and the upper quartile Q3. In order tounderstand the graphic we give the notion of interquartile range, which is ameasure of statistical dispersion, being equal to the difference between
the upper and lower quartiles: IQR = Q3−Q1. The lowest datum of every
whisker under the boxes is placed within 1.5 IQR of the lower quartile andthe highest datum of every whisker on top of the boxes is placed within 1.5
IQR of the upper quartile. Any point not included between the whiskers iscalled outlier, because it is an observation that is numerically distant fromthe rest of the data. In our chart, outliers are plotted as red points.
In the figure on the right we have plotted the same data used for plotting
the histograms above: the life expectancy of women and men respectively.

The function box_whiskers.sci is provided with the source code.
// Box-and-whisker
scf(6); clf(6);subplot(1,2,1)[outUp, outDown]=box_whiskers(LEW)
a=get("current_axes");a.data_bounds=[0,45;2,90];
xtitle("Life expectancy at birth: Women");subplot(1,2,2)[outUp, outDown]=box_whiskers(LEM)
a=get("current_axes");a.data_bounds=[0,45;2,90];
xtitle('Life expectancy at birth: Men')

A Scilab data mining tutorial(Box-and-whisker plot of the life expectancy for women andmen)

www.openeering.com

page 12/15

Step 12: The parallel coordinates plot
In the parallel coordinates plot, coordinates are represented by equallyspaced parallel vertical lines and each data point is assigned to a polyline(a continuous line composed of a sequence of segments) that intersects
each vertical line at the specific value received by that coordinate.

In the figures on the right we have plotted the data related to thepopulation (considering its composition and evolution), hence the parallel
coordinates correspond to the elements of our database starting from thefifth column up to the fifteenth one and each polyline corresponds to astate.
In the first chart we have pointed out the polyline defined by the UnitedStates, while in the second one we have chosen to point out the UnitedStates, China and Afghanistan.

(Parallel plot pointing out the United States)

The function plot_parallel_chart.sci is provided with the source code.

// Parallel coordinates[i1,j1] = find(S=='US'); // United States[i2,j2] = find(S=='CN'); // China[i3,j3] = find(S=='AF'); // Afghanistan
plot_parallel_chart(9,data(:,5:15),i1-1,S(1,5:15))
plot_parallel_chart(10,data(:,5:15),[i1-1,i2-1,i31],S(1,5:15))(Parallel plot pointing out three nations)

A Scilab data mining tutorial

www.openeering.com

page 13/15

Step 13: The correlation matrix chart and scatterplots
The correlation matrix of n random variables
whose , entry is,…,is thematrix,
whereis the mean of the variable

andis its standard deviation.

corr is +1 in the case of a perfect positive linear relationship (correlation)
between two variables, -1 in the case of a perfect negative linear
relationship (anticorrelation), and some value between -1 and 1 in all othercases, indicating the degree of linear dependence between the variables.
As it approaches zero the variables are closer to uncorrelated. The closer
the coefficient is to either -1 or 1, the stronger the correlation. Theandcorrelation matrix is symmetric because the correlation between
is the same as the correlation between and .

(Correlation matrix)

In the figure on the top-right we have written in the main diagonal ourelements (the correlation of an element with itself is 1) and the colors map
the degree of correlation (dark blue stands for -1, dark red for +1).
In the figure on the bottom-right we have the scatterplot of the data thatrefer to the value of the matrix -0.83 in the white circle, i.e. the number ofannual infants dying before reaching the age of one year per 1000 live
births and the school life expectancy for women (in years). These twoelements are strongly anticorrelated, hence when we have many infant
deaths we also have a brief school life expectancy, which is typical of poorstates. The red line is obtained using the command reglin, which performs
the linear regression between two sets of data.
The
files
data_mining.sce,
linear_correlation.sci
linear_corr_matrix.sci are provided with the source code.

A Scilab data mining tutorialandwww.openeering.com(Scatterplot with linear regression)

page 14/15

Step 14: Concluding remarks and References

1. Scilab Web Page: Available: www.scilab.org.

In this tutorial we have shown that Scilab can be considered as a powerfuldata mining tool, well-equipped to perform the widest possible range ofimportant data mining tasks.

2. Openeering: www.openeering.com.
3. United Nations web site: http://www.un.org/

On the right-hand column you may find a list of references for furtherstudies.

Step 15: Software content
To report a bug or suggest some improvement please contact Openeeringteam at the web site www.openeering.com.
Thank you for your attention,Anna Bassi, Giovanni Borzi

A Scilab data mining tutorial----------------------------A SCILAB DATA MINING TUTORIAL-----------------------------------------------Directory: functions-------------------box_whiskers.sci
linear_correlation.sci
linear_corr_matrix.sci
plot_parallel_chart.sci: box and whiskers plot
: computes the linear correlation: linear correlation matrix plot
: parallel coordinates plot-------------Main directory-------------data_mining.sce
data_UN.csv
license.txt: main scilab program: dataset: the license filewww.openeering.com

page 15/15