I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Other Python data structures (ones without an index) need to be the same length as the DataFrame:
Can you write Python code for it?

month = ['Jan', 'Feb', 'Mar', 'Apr']
# data['month'] = month # throws error (done on puropse)

data['month'] = ['Jan']*len(data)
data


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Pandas' data alignment places NaN values for labels that do not overlap in the two Series. In fact, there are only 6 players that occur in both years.
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Rather than omitting missing data from an analysis, in some cases it may be suitable to fill the missing value in, either with a default value (such as zero) or a value that is either imputed or carried forward/backward from similar data points. We can do this programmatically in Pandas with the fillna argument.
Can you write Python code for it?

bacteria2.fillna(0)

data.fillna({'year': 2013, 'treatment':2})


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Recall earlier we imported some microbiome data using two index columns. This created a 2-level hierarchical index:
Can you write Python code for it?

mb = pd.read_csv("Data/microbiome.csv", index_col=['Taxon','Patient'])

mb.head(10)


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Sometimes we may not want to ignore missing values, and allow the nan to propagate.
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Sorting and Ranking
Pandas objects include methods for re-ordering data.
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
The dtypes attribute reveals the data type for each column in our DataFrame.

int64 is numeric integer values
object strings (letters and numbers)
float64 floating-point values
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
The indexing field loc allows us to select subsets of rows and columns in an intuitive way:
Can you write Python code for it?

baseball_newind.loc['gonzalu01ARI2006', ['h','X2b', 'X3b', 'hr']]

baseball_newind.loc[:'myersmi01NYA2006', 'hr']


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
The most important consequence of a non-unique index is that indexing by label will return multiple values for some labels:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
The sep argument can be customized as needed to accomodate arbitrary separators. For example, we can use a regular expression to define a variable amount of whitespace, which is unfortunately very common in some data formats:
sep='\s+'

For a more useful index, we can specify the first two columns, which together provide a unique index to the data.
Can you write Python code for it?

mb = pd.read_csv("Data/microbiome.csv", index_col=['Patient','Taxon'])
mb.head()


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
The to_csv method writes a DataFrame to a comma-separated values (csv) file. You can specify custom delimiters (via sep argument), how missing values are written (via na_rep argument), whether the index is writen (via index argument), whether the header is included (via header argument), among other options.
An efficient way of storing data to disk is in binary format. Pandas supports this using Python’s built-in pickle serialization.
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
These can be specified on a column-wise basis using an appropriate dict as the argument for na_values.
Microsoft Excel
Since so much financial and scientific data ends up in Excel spreadsheets (regrettably), Pandas' ability to directly import Excel spreadsheets is valuable. This support is contingent on having one or two dependencies (depending on what version of Excel file is being imported) installed: xlrd and openpyxl (these may be installed with either pip or easy_install).
The read_excel convenience function in pandas imports a specific sheet from an Excel file
Can you write Python code for it?

mb = pd.read_excel('Data/microbiome/MID2.xls', sheetname='Sheet 1', header=None)
mb.head()


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
These labels can be used to refer to the values in the Series.
Can you write Python code for it?

bacteria['Actinobacteria']

bacteria[[name.endswith('bacteria') for name in bacteria.index]]

[name.endswith('bacteria') for name in bacteria.index]


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
This index is a MultiIndex object that consists of a sequence of tuples, the elements of which is some combination of the three columns used to create the index. Where there are multiple repeated values, Pandas does not print the repeats, making it easy to identify groups of values.
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
This is so that Index objects can be shared between data structures without fear that they will be changed.
Can you write Python code for it?

bacteria2.index = bacteria.index

bacteria2


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
This is typically used in time series applications, where there are repeated measurements that are incomplete for some subjects.
Exercise 5
Try using the axis argument to drop columns with missing values:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
This looks okay, but let's check:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Ties are assigned the mean value of the tied ranks, which may result in decimal values.
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Try retrieving the value corresponding to b2 in Colorado:

fetch b2 and then Colorado
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Try running corr on the entire baseball DataFrame to see what is returned:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Try using this hierarchical index to retrieve Julio Franco (francju01), who played for the Atlanta Braves (ATL) in 2007:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
We can also apply functions to each column or row of a DataFrame
Can you write Python code for it?

stats.apply(np.median)

def range_calc(x):
    return x.max() - x.min()

stat_range = lambda x: x.max() - x.min()
stats.apply(stat_range)


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
We can assign meaningful labels to the index, if they are available:
Can you write Python code for it?

bacteria = pd.Series([632, 1638, 569, 115],
    index=['Firmicutes', 'Proteobacteria', 'Actinobacteria', 'Bacteroidetes'])

bacteria


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
We can extract the underlying data as a simple ndarray by accessing the values attribute:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
We can use the drop method to remove rows or columns, which by default drops rows. We can be explicit by using the axis argument:
Can you write Python code for it?

data_nomonth = data.drop('month', axis=1)
data_nomonth


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
We will learn more about indexing below.
We can create a truly unique index by combining player, team and year:
Can you write Python code for it?

player_unique = baseball.player + baseball.team + baseball.year.astype(str)
baseball_newind = baseball.copy()
baseball_newind.index = player_unique
baseball_newind.head()

baseball_newind.index.is_unique


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
While we do want the operation to honor the data labels in this way, we probably do not want the missing values to be filled with NaN. We can use the add method to calculate player home run totals by using the fill_value argument to insert a zero for home runs where labels do not overlap:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
With a hierachical index, we can select subsets of the data based on a partial index:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
With this, we can do all sorts of custom indexing:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
Writing Data to Files
As well as being able to read several data input formats, Pandas can also export data to a variety of storage formats. We will bring your attention to just a couple of these.
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
and again but ordered
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/Intro to Pandas.ipynb".
The first task is:
describe can detect non-numeric data and sometimes yield useful information about it.
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
121 bodys got an number
home dest
Can you write Python code for it?

homedest = original_data['home.dest']
len(homedest.dropna().unique())


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
3. Calculate the same proportion, but by class and sex.
Can you write Python code for it?

# make use of the 'total' column (which is all 1's in the original_data)
survived_by_class = original_data[['pclass', 'sex', 'survived', 'total']].groupby(['pclass', 'sex']).sum()
survived_by_class

def combine_surv_total(row):
    #print(row)
    return 100.0/row.total * row.survived


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
369 different home destinations
Lets find the most common one
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
64.4% are male
age
How old the passenger is
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
Almost all women from class 1 and 2 survived, in class 3 about 50% survived
Can you write Python code for it?

sns.barplot(x="pclass", y="survived", hue="age-category", data=original_data[original_data['sex'] == 'male'])


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
And finally calculate the survive portion for all cases
Can you write Python code for it?

def surv_proportions(row):
    if row.total == 0:
        return np.nan
    return round(100.0/row.total * row.survived, 2)

grouped['survive-portion (%)'] = grouped.apply(surv_proportions, axis=1)

grouped


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
And the distribution of age plotted:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
Most come from New York
2. Use the groupby method to calculate the proportion of passengers that survived by sex:
First gather the numbers
Can you write Python code for it?

survived_by_sex = original_data[['survived', 'sex']].groupby('sex').sum()
nbr_males = len(original_data[original_data['sex'] == 'male'])
nbr_females = len(original_data[original_data['sex'] == 'female'])
nbr_total = len(original_data['sex'])
survived_by_sex

print(nbr_total == nbr_females + nbr_males) # to check if consistent


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
Plot histogram: Almost all traveled without siblings or spouses. there is apparently one family that traveled together (8 siblings are on board)
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
Then calcultate the percentages
Can you write Python code for it?

female_survived_percentage = (100/nbr_females) * survived_by_sex.at['female', 'survived']
male_survived_percentage = (100/nbr_males) * survived_by_sex.at['male', 'survived']
print('female surv: '+str(round(female_survived_percentage, 3))+'%')
print('male surv: '+str(round(male_survived_percentage, 3))+'%')


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
Then group the data in a sensible way to get the nice Table below.
Can you write Python code for it?

grouped = catsdata.groupby(['sex', 'age-category', 'pclass']).sum().fillna(0)
grouped


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
There are NaN values! But also floating point values, which is somewhat unusual but not a problem per se.
Can you write Python code for it?

age.min() # a baby?

age.max()

age.mean()


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
There are people that did not pay anything
Can you write Python code for it?

original_data[fare == 0]

fare.dtypes

original_data[fare.isnull()]


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
This are the children and the parents of the 'big' familly. Sadly all died :(
ticket
the ticketnbr the passanger had
Can you write Python code for it?

ticket = original_data['ticket']
len(ticket.unique())

ticket.dtype

len(ticket[ticket.isnull()])


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
apparently there are some with the same name
Can you write Python code for it?

len(name.values) - len(name.unique())

#lets find them
original_data[name.isin(name[name.duplicated()].values)]


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
create a new column with the apply method
Can you write Python code for it?

survived_by_class['survived in %'] = survived_by_class.apply(combine_surv_total, axis=1)
survived_by_class


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
embarked
Can you write Python code for it?

embarked = original_data['embarked']
embarked.unique()

len(embarked[embarked.isnull()])


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
most are in class 3, but surprisingly class 1 has more passagengers than class 2
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
parch
The number of parents or children on the ship
Can you write Python code for it?

parch = original_data['parch']
parch.unique()

parch.mean()


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
sex
the sex of the passenger
Can you write Python code for it?

sex = original_data['sex']
sex.unique()

nbr_males = len(sex[sex == 'male'])

nbr_females= len(sex[sex == 'female'])

100/len(sex) * nbr_males


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
sibsp
The number of siblings or spouses on the ship
Can you write Python code for it?

sipsp = original_data['sibsp']
sipsp.unique()

sipsp.mean()


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
some have several boats.
body
the identification number of a body
Can you write Python code for it?

body = original_data['body']
body.count()


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
survived
States if the passenger survived the titanic sinking
Can you write Python code for it?

surv = original_data['survived']
surv.unique() # to make sure there are only 1 and 0

#how many survived?
surv.sum()

#how many died?
len(surv[surv == 0])


I am working on the file "HW01-Intro_to_Pandas/data-wrangling-last-exo.ipynb".
The first task is:
there are 3 different classes
Can you write Python code for it?

for c in pclass.unique():
    print('nbr in class '+str(c)+': '+str(len(pclass[pclass == c])))


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
A problem is that the columnames and the variables are not the same over the 3 countries. So we harmonize it somewhat.
Can you write Python code for it?

# rename the columns
var_name = 'vars'
sl_relevant.rename(columns={'variable': var_name}, inplace=True)
guinea_relevant.rename(columns={'Description': var_name, 'Date': 'date'}, inplace=True)
liberia_relevant.rename(columns={'Variable': var_name, 'Date': 'date'}, inplace=True)

#rename the variables
new_infected = 'new_infected'
new_death= 'new_death'
sl_relevant[var_name][sl_relevant[var_name] == sl_variables_to_use[0]] = new_infected
sl_relevant[var_name][sl_relevant[var_name] == sl_variables_to_use[1]] = new_death

guinea_relevant[var_name][guinea_relevant[var_name] == guinea_variables_to_use[0]] = new_infected
guinea_relevant[var_name][guinea_relevant[var_name] == guinea_variables_to_use[1]] = new_death

liberia_relevant[var_name][liberia_relevant[var_name] == liberia_variables_to_use[0]] = new_infected
liberia_relevant[var_name][liberia_relevant[var_name] == liberia_variables_to_use[1]] = new_death

# rename the data
sl_clean = sl_relevant.copy()
guinea_clean = guinea_relevant.copy()
liberia_clean = liberia_relevant.copy()


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
Advanced Exo from 'Intro to Pandas'
Can you write Python code for it?

# load all data and parse the 'date' column
def load_data():
    sl_files=glob.glob('Data/ebola/sl_data/*.csv')
    guinea_files=glob.glob('Data/ebola/guinea_data/*.csv')
    liberia_files=glob.glob('Data/ebola/liberia_data/*.csv')
    sl = pd.concat((pd.read_csv(file,  parse_dates=['date']) for file in sl_files), ignore_index=True)
    guinea = pd.concat((pd.read_csv(file , parse_dates=['Date']) for file in guinea_files), ignore_index=True)
    liberia = pd.concat((pd.read_csv(file , parse_dates=['Date']) for file in liberia_files), ignore_index=True)
    return (sl, guinea, liberia)

(sl, guinea, liberia) = load_data()

# look at the sl data
sl.columns

sl['variable'].unique()


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
And sort the data:
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
Concat the dataframes
Can you write Python code for it?

final_data = pd.concat([sl_final, guinea_final, liberia_final], ignore_index=True)


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
Handle missing data
Can you write Python code for it?

#remove al rows and columns that consist only of NaNs
def remove_rows_and_cols_with_only_nan(data):
    return data.dropna(axis=1, how='all').dropna(axis=0, thresh=3)
sl_clean = remove_rows_and_cols_with_only_nan(sl_clean)
guinea_clean = remove_rows_and_cols_with_only_nan(guinea_clean)
liberia_clean = remove_rows_and_cols_with_only_nan(liberia_clean)


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
Show the data
Can you write Python code for it?

liberia_final.head()

guinea_final.head()

sl_final.head()


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
Then we can replace all NaN values with 0. We don't know anything about that data to put something else, and removing is no option since there would not be much left if we removed all rows/cols that contain at least one NaN
Can you write Python code for it?

# replace all NaNs with 0 (inplace)
sl_clean.fillna(value=0, inplace=True)
guinea_clean.fillna(value=0, inplace=True)
liberia_clean.fillna(value=0, inplace=True)


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
add 'country' col to distinguish the dataframes when they are put together
Can you write Python code for it?

sl_final['country'] = 'sl'
guinea_final['country'] = 'guinea'
liberia_final['country'] = 'liberia'

guinea_final.head()

liberia_final.head()

sl_final.head()


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
make all types numerical (excluding the date and variable columns)
Can you write Python code for it?

def change_to_numeric(data):
    col_list = list(data.columns)
    col_list.remove('date')
    col_list.remove(var_name)
    data[col_list] = data[col_list].apply(pd.to_numeric)

change_to_numeric(sl_clean)
change_to_numeric(guinea_clean)
change_to_numeric(liberia_clean)


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
not all values are numerical (most are objects)
Can you write Python code for it?


I am working on the file "HW01-Intro_to_Pandas/intro-to-pandas-last-exo.ipynb".
The first task is:
we decide to only take the 'confirmed' cases and not the suspected or probable ones since 'suspected' and 'probable' are very subjective terms and may not be the same over the 3 countries.
Can you write Python code for it?

sl_variables_to_use = ['new_confirmed', 'death_confirmed']

# look at the guinea data
guinea.columns

guinea['Description'].unique()

guinea_variables_to_use = ['New cases of confirmed', 'New deaths registered today (confirmed)']

# look at the liberia data
liberia.columns

liberia['Variable'].unique()

liberia_variables_to_use = ['New case/s (confirmed)', 'Total death/s in confirmed cases']

def select_features(data, var_name, features):
    return data[data[var_name].isin(features)]

# take the relevant variables
sl_relevant = select_features(sl, 'variable', sl_variables_to_use)
guinea_relevant = select_features(guinea, 'Description', guinea_variables_to_use)
liberia_relevant = select_features(liberia, 'Variable', liberia_variables_to_use)


I am working on the file "HW01-connect.ipynb".
The first task is:
Are all values numbers???  No.  You can represent anything in the computer.  How about strings.
So what operators are defined on strings?
Can you write Python code for it?

# Concatenation
"cal"+"rocks"

# Replication
"cal"*3

# let's try something weird
"cal"*pie

# string equality,
"cal" == "cal"

"cal" == "rocks"

# regardless of the kind of quotes
"cal" == 'cal'

# lexicographic order
'cal' < 'rocks'

# case matters
'Cal' == 'cal'

# Uppercase before lower case
'Cal' < 'cal'

# go on, it won't break anything
"cal"*"rocks"

# These are really different, but we can still ask if they are the same - they aren't
0 == "hello world"

# Not like C (what's 'C'? another programming language)
0 == ""

False == ""

# Not everything is comparable
0 < "cal"


I am working on the file "HW01-connect.ipynb".
The first task is:
Bit more about numbers
Can you write Python code for it?


I am working on the file "HW01-connect.ipynb".
The first task is:
Every datatype has a set of operators defined on it - not just values.  Try out some operators on sequences.
Can you write Python code for it?

# operators: concatenation
(1,2)+(3,4)

[1,2,3]+[4,5]

# replication
(1,2)*3

# does this make any sense?
(1,2)*(3,4)

# what about this?
[1,2]+(3,4)

# conversion
list((1,2))


I am working on the file "HW01-connect.ipynb".
The first task is:
Logical operations make sense on boolean values.  And you can build expressions
Can you write Python code for it?

1 < 2 and 2 < 3

1 < 2 or 2 > 3

1 != 2 or "cal" == "rocks"

not "cal" == "rocks"

not "cal" != "rocks"


I am working on the file "HW01-connect.ipynb".
The first task is:
So far everything you've seen looks the same as the arithmetic you did in elementary school.  But back then you used the equals sign in two different ways without even knowing it.  We need to be a little more careful.
When we mean the operator that tests for equality, we use ==.
Can you write Python code for it?

# Are these two the same?
2 == 3

# Are they 'not equal'?
2 != 3

3.141592 == 3.141592

# Nice to be able to do this too
1 == 1.0

# what's the order here?
1 / 2 == 3

# nonsensical expressions do funny things
1 / (2 == 3)

# makes more sense here
1/0

# what is the difference between asignment and equality?
x = 2 == 3

x


I am working on the file "HW01-connect.ipynb".
The first task is:
Strings are a very special type of sequence.  There are others.  We call these data structures.  For example, tuples, lists, arrays.
Can you write Python code for it?

# tuples - created by separating values by commas (often in parens)
1,2

type((1,2))

# You can even assign tuples
a,b = 1,2

a

# lists - are in brackets
[1,2,3]

type([1,2,3])


I am working on the file "HW01-connect.ipynb".
The first task is:
Values all have types.  Just ask them.
Can you write Python code for it?


I am working on the file "HW01-connect.ipynb".
The first task is:
Variables
Introduced in c8 as "names for values".
Assignment binds a value to a variable.
Can you write Python code for it?

pie = 3.1415

# use variables just like values
pie*2

# here's a better one
from math import pi
pi

pi-pie


I am working on the file "HW01-connect.ipynb".
The first task is:
What you should know:
  - two types: integers and floats
  - arithemetic operators: (+,-,*,/,//,%)
Expressions
follow 'operator precedence'
The operations are performed one at at time in a specific order.  Parentheses are used to specify order, just like in math class.
Can you write Python code for it?

# equal precedence, left to right
2 + 3 - 4 + 5

# order matters.  Parentheses are your friend.
2 + 3 - (4 + 5)

# make it explicit
(((2 + 3) - 4) + 5)

2 + 3 * 4

# * binds more tightly than + or -
2 + (3 * 4)

(2 + 3) * 4

# what about / ?
2 + 3/4 * 5

2 + 3*4 / 5

# left to right for same precedence
2 + ((3*4)/5)

# can you always tell?
2 + 3 * (4/5)

2 + (3*(4/5))

2 + (3*4)/5


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
1. Showing types, describing data and changing to categorical types
Can you write Python code for it?

data_titanic.dtypes

data_titanic.describe()

for col in ['pclass', 'survived', 'sex', 'embarked']:
    data_titanic[col] = data_titanic[col].astype('category')
data_titanic.dtypes


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
2.
Can you write Python code for it?

import matplotlib.pyplot as plt
fig, axes = plt.subplots(2, 2, figsize=(10,10))

data_titanic['pclass'].value_counts().plot(kind='bar', ax=axes[0, 0])
axes[0, 0].set_title('Number of passenger w.r.t class')

data_titanic['embarked'].value_counts().plot(kind='bar', ax=axes[0, 1])
axes[0, 1].set_title('Number of passenger w.r.t embarkment location')

data_titanic['sex'].value_counts().plot(kind='bar', ax=axes[1, 0])
axes[1, 0].set_title('Number of passenger w.r.t sex')

age_binned = pd.cut(data_titanic['age'], bins=[0, 10, 20, 30, 40, 50, 60, 70, 80], include_lowest=True)
axes[1, 1].set_title('Number of passenger w.r.t age range')
age_binned.value_counts(sort=False).plot(kind='bar', ax=axes[1, 1])

plt.show()


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
4.
Can you write Python code for it?

# Sierra Leone

# basic formating of the data, to be more readable for a human
data_cleaned_SL = data.query('Country == "Sierra Leone"').dropna(axis=1, how='all').set_index(['Date','variable'])['National'].apply(pd.to_numeric, errors='coerce')
data_cleaned_SL = pd.pivot_table(data_cleaned_SL.reset_index(), values='National', index='Date', columns='variable', aggfunc='sum')

# no specific problem for the new cases
new_cases_SL = data_cleaned_SL['new_confirmed'].resample('M').mean()

# new deaths
# check consisteny of cumulative data
data_cleaned_SL['death_confirmed'] = check_cumulative_entries(data_cleaned_SL, 'death_confirmed')
# create daily entries
data_cleaned_SL['death_confirmed'] = compute_daily_entries_from_cumulative(data_cleaned_SL, 'death_confirmed')

#compute the mean by month
new_deaths_SL = data_cleaned_SL['death_confirmed'].resample('M').mean()

# concatenate the two time series
data_cleaned_SL = pd.concat([new_cases_SL, new_deaths_SL], axis=1)
data_cleaned_SL

# display the values in a sexy manner
data_cleaned_SL.rename_axis({'new_confirmed': 'New cases (mean)', 'death_confirmed': 'Deaths (mean)'}, axis='columns', inplace=True)
data_cleaned_SL.index.names = ['Month']
data_cleaned_SL.index = data_cleaned_SL.index.month
data_cleaned_SL

# general overview

data_cleaned_guinea['Country'] = 'Guinea'
data_cleaned_liberia['Country'] = 'Liberia'
data_cleaned_SL['Country'] = 'Sierra Leone'

# concatenate the data from the three countries to have a general overview
overview = pd.concat([data_cleaned_guinea, data_cleaned_liberia,data_cleaned_SL])
overview.reset_index().set_index(['Country', 'Month'])


plt.figure(figsize=(8,8))
df = data_titanic[data_titanic['survived'] == 1]['pclass']
df.value_counts().plot.pie(autopct='%1.0f%%')
plt.title('Proportion of survivors for each class')
plt.show()


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
Comments: Same as the previous plot, the over-representation of the sex "male" in the dataset can be misleading and one has to be careful about the conclusion drawn from such plots.
Once again, the survival rate for each group will be further analyzed in the next section.

6.
Comments: This section demonstrate the importance of data interpretation and how plots can lead to incorrect conclusions when not interpreted with caution.
Indeed, by analyzing survival rate group by group and not merged, we contradict all the potential conclusions from the previous sections. To counteract this phenomena (class imbalance), we could also have resampled the classes equally before doing the previous plots.
Can you write Python code for it?

df = data_titanic[['age', 'pclass', 'sex', 'survived']]
df = df.sort_values('age')
# Take the 50% quantile (quantile) of the indices (since the median by age) will not produce two equally sized sets.
# The result will satisfy the conditions answered for the question 20) in the FAQ
# CAREFUL: df must be sorted by age before taking such a median by indices.
med_id = df.shape[0] // 2
print('Age of the median:', int(df.iloc[med_id]['age']), 'years old')

df['age_group'] = 'Older'
df.iloc[:med_id]['age_group'] = 'Younger'

df['age_group'] = df['age_group'].astype('category')
df['survived'] = df['survived'].astype(bool)

groups = pd.DataFrame(df.groupby(['age_group', 'pclass', 'sex', 'survived']).size())
# Following the instructions:  Present your results in a DataFrame with unique index.
groups.reset_index(inplace=True)
groups.columns = ['age category', 'class', 'sex', 'survived', 'count']

groups

fig, axes = plt.subplots(1, 2, figsize=(16,8))
groups[groups['age category'] == 'Younger'].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[0], labels=['', ''])
groups[groups['age category'] == 'Older'].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[1], labels=['', ''])
axes[0].set_title('Proportion of survivors for younger')
axes[1].set_title('Proportion of survivors for older')
axes[0].legend(labels=['Not survived', 'Survived'])
axes[1].legend(labels=['Not survived', 'Survived'])
plt.show()


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
Comments: We clearly see that most people were in the third class (~50%), and then almost equally splitted between the first and second class.
The age plot indicates a huge discrepancy between male and female people present on the boat.
On a side note, we can clearly see on the age plot that the imputed values for the NaN impacts the histogram by representing widely the median range, 20-30 range.

3.
Can you write Python code for it?

fig, axes = plt.subplots(1, 2, figsize=(16,8))

s = data_titanic['cabin']
s_nan = s.astype(str).str[0]
# We have to replace the N
s_nan = s_nan.replace('n', 'NaN')
s_no_nan = s.dropna().astype(str).str[0]

axes[0].set_title('Cabin repartition with undefined')
s_nan.value_counts().plot.pie(ax=axes[0], autopct='%1.0f%%')
axes[1].set_title('Cabin repartition without undefined')
s_no_nan.value_counts().plot.pie(ax=axes[1], autopct='%1.0f%%')
plt.show()


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
Comments: We see that except for Guinea, the number of new cases of Ebola increases among the months until December, where it decreases a little bit. We cannot define a general trend for the number of deaths. We have different behaviours depending on the country. This could depend on the health infrastructure of each country, the way the disease is treated or what it is done to prevent the citizens from the risk of disease. It is difficult to draw strong conclusions from this data for different reasons : the number of data points for each month is not the same depending on the country and some datasets have not been rigorously fullfilled, which means that we can have inaccurate or erroneous data.
Task 2. RNA Sequences
In the DATA_FOLDER/microbiome subdirectory, there are 9 spreadsheets of microbiome data that was acquired from high-throughput RNA sequencing procedures, along with a 10th file that describes the content of each.
Use pandas to import the first 9 spreadsheets into a single DataFrame.
Then, add the metadata information from the 10th spreadsheet as columns in the combined DataFrame.
Make sure that the final DataFrame has a unique index and all the NaN values have been replaced by the tag unknown.
Can you write Python code for it?

path_RNA = "Data/microbiome"

data_RNA = pd.DataFrame()

allFiles_RNA = glob.glob(path_RNA + "/MID*.xls")
metadata_RNA = path_RNA + "/metadata.xls"

# Concat all the MID* files
data_RNA = pd.concat((pd.read_excel(file, index_col=0, names=[file[16:-4]]) for file in allFiles_RNA), axis=1)

# transpose index/columns and sort indexes
data_RNA = data_RNA.T

# add last file as columns
data_RNA = pd.concat([data_RNA, pd.read_excel(metadata_RNA, index_col=0)], axis=1)

# replace NaN values by 'unknown'
data_RNA = data_RNA.fillna(value='unknown')

data_RNA


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
Data cleaning
Can you write Python code for it?

# We are only interested in handling a few of the columns and their missing values

# We drop the row with a NaN value for Embarked since there are only two of them
# We could also replace by the most frequent value policy, and thus fill with "S" in this case
data_titanic.dropna(subset=["embarked"], inplace=True)
# Fill NaN with median value policy for fare and age
data_titanic["fare"].fillna(data_titanic["fare"].median(), inplace=True)
data_titanic["age"].fillna(data_titanic["age"].median(), inplace=True)


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
For each of the following questions state clearly your assumptions and discuss your findings:
1. Describe the type and the value range of each attribute. Indicate and transform the attributes that can be Categorical.
2. Plot histograms for the travel class, embarkation port, sex and age attributes. For the latter one, use discrete decade intervals.
3. Calculate the proportion of passengers by cabin floor. Present your results in a pie chart.
4. For each travel class, calculate the proportion of the passengers that survived. Present your results in pie charts.
5. Calculate the proportion of the passengers that survived by travel class and sex. Present your results in a single histogram.
6. Create 2 equally populated age categories and calculate survival proportions by age category, travel class and sex. Present your results in a DataFrame with unique index.
Data loading
Can you write Python code for it?

# load data
data_titanic = pd.read_excel('Data/titanic.xls')
data_titanic.head()


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
Table of Contents

Task 1.  Compiling Ebola Data

Task 2.  RNA Sequences

Task 3.  Class War in Titanic

Can you write Python code for it?

DATA_FOLDER = 'Data' # Use the data folder provided in Tutorial 02 - Intro to Pandas.


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
Task 1. Compiling Ebola Data
The DATA_FOLDER/ebola folder contains summarized reports of Ebola cases from three countries (Guinea, Liberia and Sierra Leone) during the recent outbreak of the disease in West Africa. For each country, there are daily reports that contain various information about the outbreak in several cities in each country.
Use pandas to import these data files into a single Dataframe.
Using this DataFrame, calculate for each country, the daily average per year of new cases and deaths.
Make sure you handle all the different expressions for new cases and deaths that are used in the reports.

Assumption :

For the daily average new cases per month we decided to select the "new cases confirmed" (or +/- similar) for the three different countries.
For the daily average of deaths per month we chose to select the "new deaths registered" (or +/- similar) for the three different countries.
In both cases, we pay attention only to the "confirmed" data.

How we did it :

Create a new column containing the name of the country in the dataframe of the corresponding country, then join all three dataframes. To avoid complications in further work, we aggregated the different columns that represented the dates into a single one.
Guinea : no specific assumption have to be made as the data we process seem to be without ambiguity.
Liberia : here we have some ambiguity. For the data concerning the new cases, the daily entries are not consistent among all the sheets. We just recompute it based on the cumulative entries of the new cases, as they seem to be more reliable.
Sierra Leone : here we have a similar case to Liberia; for the data concerning the daily new deaths, the entries are written in a cumulative manner, so we will proceed in the exact same way as for the Liberia case, in order to retrieve the daily new deaths.

1.
Can you write Python code for it?

path = "Data/ebola"
countries = [("Guinea","guinea_data"), ("Liberia","liberia_data"), ("Sierra Leone","sl_data")]

data = pd.DataFrame()

# for each country dataframe, concatenate all the sheets and add a new column with the corresponding country
for country in countries:
    allFiles = glob.glob(path + "/" + country[1] + "/" + "/*csv")
    data_per_country = pd.concat((pd.read_csv(file) for file in allFiles), axis=0)
    data_per_country['Country'] = country[0]
    data = pd.concat([data, data_per_country])

# merge columns 'Date' and 'date'
data['Date'] = pd.concat([data['Date'].dropna(), data['date'].dropna()])
data = data.drop('date', axis=1)
data['Date'] = pd.to_datetime(data['Date'])


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
We can clearly notice that the survival rate decreases drastically as the class changes from first to "ecnonomy" class.
Can you write Python code for it?

fig, axes = plt.subplots(1, 2, figsize=(16,8))
groups[groups['sex'] == 'male'].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[0], labels=['', ''])
groups[groups['sex'] == 'female'].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[1], labels=['', ''])
axes[0].set_title('Proportion of survivors for male')
axes[1].set_title('Proportion of survivors for female')
axes[0].legend(labels=['Not survived', 'Survived'])
axes[1].legend(labels=['Not survived', 'Survived'])
plt.show()


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
We don't see any clear difference between the two groups in term of survival rate.
Can you write Python code for it?

fig, axes = plt.subplots(1, 3, figsize=(16,5))
groups[groups['class'] == 1].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[0], labels=['', ''])
groups[groups['class'] == 2].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[1], labels=['', ''])
groups[groups['class'] == 3].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[2], labels=['', ''])
axes[0].set_title('Proportion of survivors for class 1')
axes[1].set_title('Proportion of survivors for class 2')
axes[2].set_title('Proportion of survivors for class 3')
axes[0].legend(labels=['Not survived', 'Survived'])
axes[1].legend(labels=['Not survived', 'Survived'])
axes[2].legend(labels=['Not survived', 'Survived'])
plt.show()


I am working on the file "HW01/Homework 1.ipynb".
The first task is:
We notice a huge difference in the survival rate between genders. Indeed, women have a much higher survival rate than men.
We then chose the 12.5% lower quantile to exacerbate the motto "Women and children first"    (:
Can you write Python code for it?

df = data_titanic[['age', 'pclass', 'sex', 'survived']]
df = df.sort_values('age')
# Take the 12.5% quantile (quantile) of the indices (since the median by age) will not produce two equally sized sets.
# The result will satisfy the conditions answered for the question 20) in the FAQ
# CAREFUL: df must be sorted by age before taking such a median by indices.
med_id = df.shape[0] // 8
print('Age of the median:', int(df.iloc[med_id]['age']), 'years old')

df['age_group'] = 'Older'
df.iloc[:med_id]['age_group'] = 'Younger'

df['age_group'] = df['age_group'].astype('category')
df['survived'] = df['survived'].astype(bool)

groups = pd.DataFrame(df.groupby(['age_group', 'pclass', 'sex', 'survived']).size())
# Following the instructions:  Present your results in a DataFrame with unique index.
groups.reset_index(inplace=True)
groups.columns = ['age category', 'class', 'sex', 'survived', 'count']

fig, axes = plt.subplots(1, 2, figsize=(16,8))
groups[groups['age category'] == 'Younger'].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[0], labels=['', ''])
groups[groups['age category'] == 'Older'].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[1], labels=['', ''])
axes[0].set_title('Proportion of survivors for younger')
axes[1].set_title('Proportion of survivors for older')
axes[0].legend(labels=['Not survived', 'Survived'])
axes[1].legend(labels=['Not survived', 'Survived'])
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(16,8))
groups[groups['sex'] == 'male'].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[0], labels=['', ''])
groups[groups['sex'] == 'female'].groupby('survived').sum().plot.pie(y='count', autopct='%1.0f%%', ax=axes[1], labels=['', ''])
axes[0].set_title('Proportion of survivors for male')
axes[1].set_title('Proportion of survivors for female')
axes[0].legend(labels=['Not survived', 'Survived'])
axes[1].legend(labels=['Not survived', 'Survived'])
plt.show()


I am working on the file "HW02-Data_from_the_Web/bachelor_data_analysis.ipynb".
The first task is:
Collect the data
Create a function that will parse one request and return a dataFrame
Can you write Python code for it?

def parseRequest(u_a, p_a, p_p, h_e):

    #Send request
    url = 'http://isa.epfl.ch/imoniteur_ISAP/!GEDPUBLICREPORTS.html?ww_x_GPS=-1&ww_i_reportModel=133685247&ww_i_reportModelXsl=133685270&ww_x_UNITE_ACAD='+u_a[0]+'&ww_x_PERIODE_ACAD='+p_a[0]+'&ww_x_PERIODE_PEDAGO='+p_p[0]+'&ww_x_HIVERETE='+ h_e
    r = requests.get(url)
    soupe = BeautifulSoup(r.text, 'html.parser')

    #get the header , we also replace the space by '_' (easier to use the dataframe later)
    th_tag = soupe.find_all('th')
    th = [t.text.replace(' ', '_') for t in th_tag]
    #remove the first th that correspond to the title of the table
    th = th[1:]

    #save the size of the header
    header_size = len(th)
    #add new information (new columns): year_start, year_stop, semester number
    th = np.append(th, ['Year_start', 'Year_stop', 'Semester'])

    #Find all the 'tr' tag
    tr_tag = soupe.find_all('tr')
    #drop the 2 first tag that correspond to the title and the headers of the table
    tr_tag = tr_tag[2:]

    #Temporary dictionary that will collect all the entry of the dataframe
    data = []

    #Read the request line by line and fill the dataframe
    for tr in tr_tag:
        #create the new entry
        row = [r.text.replace('\xa0', ' ')  for r in tr]
        #one row contains 12 elements but the header has only 11-> drop the last one because it is always empty
        row = row[:header_size]

        ##add the  new information to the row
        #split the academic period
        year = p_a[1].split('-')
        #find the semester
        semester = p_p[1].split()[2]
        newCol = [int(year[0]), int(year[1]), semester]

        #concat the row with the new info
        row += newCol
        data.append(row)

    df = pd.DataFrame(data, columns= th)
    return df


I am working on the file "HW02-Data_from_the_Web/bachelor_data_analysis.ipynb".
The first task is:
Collect the data
Get the name of the 4 fields we have to select
Can you write Python code for it?

select = soupe.find_all('select')
select_name = [s.attrs['name'] for s in select]
select_name


I am working on the file "HW02-Data_from_the_Web/bachelor_data_analysis.ipynb".
The first task is:
Get the select field correspondind to the 4 names found before
Can you write Python code for it?

select_field = [soupe.find('select',{'name': name}) for name in select_name]


I am working on the file "HW02-Data_from_the_Web/bachelor_data_analysis.ipynb".
The first task is:
Get the value corresponding to the "Informatique"
Can you write Python code for it?

option_unite_acad = select_field[0].find_all('option')
#option_unite_acad[[opt.text == 'Informatique' for opt in option_unite_acad]]
option_unite_acad
unite_acad ={opt['value']: opt.text for opt in option_unite_acad if opt.text == 'Informatique'}

unite_acad


I am working on the file "HW02-Data_from_the_Web/bachelor_data_analysis.ipynb".
The first task is:
Nomber of person that complete the bachelor in computer science
Can you write Python code for it?


I am working on the file "HW02-Data_from_the_Web/bachelor_data_analysis.ipynb".
The first task is:
Number of person that tried at least the first years or last one
Can you write Python code for it?


I am working on the file "HW02-Data_from_the_Web/bachelor_data_analysis.ipynb".
The first task is:
Person that didn't complete the first year in compute Science, we don't consider them since we can't know when they begin their first year
Can you write Python code for it?


I am working on the file "HW02-Data_from_the_Web/bachelor_data_analysis.ipynb".
The first task is:
Show total number of student that made at least one semester
Can you write Python code for it?


I am working on the file "HW02-Data_from_the_Web/bachelor_data_analysis.ipynb".
The first task is:
We iterate over all the parameters. We decided to skip the 'Type de semestre' (HIVERETE) since it is a redundant information. An odd semester is always in Autumn and an even one is always in Spring
Can you write Python code for it?

list_df = []

for u_a in unite_acad.items():
    for p_a in period_acad.items():
        for p_p in period_pedago.items():
                print('Request for: ',u_a[1], p_a[1],  p_p[1])
                list_df.append(parseRequest(u_a,p_a, p_p, 'null'))

Student = pd.concat(list_df, ignore_index=True)
Student


I am working on the file "HW02-Data_from_the_Web/collect_and_persist_all_data.ipynb".
The first task is:
Collect the data
Get the select options
Can you write Python code for it?

reportModel = 133685247
full_form_url = 'http://isa.epfl.ch/imoniteur_ISAP/!GEDPUBLICREPORTS.filter?ww_i_reportModel=133685247'
r = requests.get(full_form_url)
soup = BeautifulSoup(r.text, 'html.parser')

# get the name of the dropdown menus
select = soup.find_all('select')
select_name = [s.attrs['name'] for s in select]
select_name

select_fields = [soup.find('select',{'name': name}) for name in select_name]
# the html for each <select> field

# find the value for the informatique section
unite_acad_options = select_fields[0].find_all('option')
#unite_acad_options

unite_acad_informatique ={opt['value']: opt.text for opt in unite_acad_options if opt.text == 'Informatique'}
unite_acad_informatique

# periode academic
#select_fields[1].find_all('option')
period_acad = {opt['value']: opt.text for opt in select_fields[1].find_all('option') if opt['value'] != 'null' and int(opt.text.split('-')[0]) >= 2007}
period_acad

# get all the pedagogic periods
option = select_fields[2].find_all('option')
period_pedago = {opt['value']: opt.text for opt in option if opt.text != '' }
period_pedago

option = select_fields[3].find_all('option')
hiverEte = {opt['value']: opt.text for opt in option if opt['value'] != 'null'}
hiverEte


I am working on the file "HW02-Data_from_the_Web/collect_and_persist_all_data.ipynb".
The first task is:
Note that the Mineur semestre X are always empty. That is why they dont appear here.
Can you write Python code for it?


I am working on the file "HW02-Data_from_the_Web/collect_and_persist_all_data.ipynb".
The first task is:
make one request for all permutations of (unite_acad, periode_acad, periode_pedago). We ignore hiverete because it is redundant.
Can you write Python code for it?

perm_list = list(itertools.product(list(unite_acad_informatique.items()), list(period_acad.items()), list(period_pedago.items())))

# get the data
header = ['Civilité', 'Nom_Prénom', 'Orientation_Bachelor', 'Orientation_Master', 'Spécialisation', 'Filière_opt.', 'Mineur', 'Statut', 'Type_Echange', 'Ecole_Echange', 'No_Sciper', 'title', 'periode_acad', 'periode_pedago']
dframes = []
for (ua, pa, pp) in perm_list:
    res = collect_dataframe(ua, pa, pp,('null', 'null'), header)
    [dframes.append(df) for df in res]

#concatenate
all_data = pd.concat([df for df in dframes])

# write to file
#all_data.to_csv('all_data.csv')

len(all_data)


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
... and remove them to
Can you write Python code for it?

sciper_to_remove = study_now['No_Sciper'].unique()
master_data = master_data[~master_data['No_Sciper'].isin(sciper_to_remove)]


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
A nice plot showing that there is a huge difference in the number of people taking the different Specialisations
Can you write Python code for it?


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
And per specialisation
We will take the 'added_semester' data since we think it is more accurate.
Note that the average number of semesters is only 0.3 higher than the one of all students.
Can you write Python code for it?

master_with_spe = master_added_sem[~pd.isnull(master_added_sem['Specialisation'])]
master_with_spe.describe()


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
Clean the data
Can you write Python code for it?


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
Compared to all master students, it seems that only the first two specs actually inply a longer stay at EPFL (their confidence interval just barely don't touch the general mean)
Can you write Python code for it?

def calc_diff_from_average(col, average):
    return (lambda row: row[col] - average)
master_with_spe['diff_from_average'] = master_with_spe.apply(calc_diff_from_average('nombre_semestres', master_added_sem['nombre_semestres'].mean()), axis=1)

sns.barplot(x='diff_from_average', y='Specialisation', data=master_with_spe);


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
Find all the students that are registered in the current semester and filter them.
Can you write Python code for it?

study_now = pd.DataFrame(columns=['Civilité', 'Nom_Prénom', 'Orientation_Bachelor', 'Orientation_Master',
      'Spécialisation', 'Filière_opt.', 'Mineur', 'Statut', 'Type_Echange',
      'Ecole_Echange', 'No_Sciper', 'title', 'periode_acad',
      'periode_pedago'])
for scip, group in grouped:
    if (group.periode_acad == '2016-2017').any():
        study_now = pd.concat([study_now, group])
len(study_now.No_Sciper.unique())


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
In the time taken to colmplete their studies they don't differ a lot.
(The red dotted line is the averarge of all students, the blue line the average of the students with specialisations, the black bar is the confidence interval)
Can you write Python code for it?

ax = sns.barplot(x='nombre_semestres', y='Specialisation', data=master_with_spe);
plt.axvline(master_with_spe['nombre_semestres'].mean(), color='b', linestyle='dashed', linewidth=2)
plt.axvline(master_added_sem['nombre_semestres'].mean(), color='r', linestyle='dashed', linewidth=2)


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
Master Data
Obtain all the data for the Bachelor students, starting from 2007. Keep only the students for which you have an entry for both Bachelor semestre 1 and Bachelor semestre 6. Compute how many months it took each student to go from the first to the sixth semester. Partition the data between male and female students, and compute the average -- is the difference in average statistically significant?
2) Perform a similar operation to what described above, this time for Master students. Notice that this data is more tricky, as there are many missing records in the IS-Academia database. Therefore, try to guess how much time a master student spent at EPFL by at least checking the distance in months between Master semestre 1 and Master semestre 2. If the Mineur field is not empty, the student should also appear registered in Master semestre 3. Last but not the least, don't forget to check if the student has an entry also in the Projet Master tables. Once you can handle well this data, compute the "average stay at EPFL" for master students. Now extract all the students with a Spécialisation and compute the "average stay" per each category of that attribute -- compared to the general average, can you find any specialization for which the difference in average is statistically significant?
Read the data from csv
Can you write Python code for it?

all_data = pd.read_csv('all_data.csv', usecols=['Civilité', 'Nom_Prénom', 'title', 'periode_acad', 'periode_pedago','Orientation_Master', 'Spécialisation', 'Filière_opt.', 'Mineur', 'Statut', 'Type_Echange', 'Ecole_Echange', 'No_Sciper'])

all_data.sort_values(by='No_Sciper', axis=0).head(10)

len(all_data)


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
Only Signals, Images and Interfaces and Internet computing have a pvalue < 0.05 (pvalue = 0.044 and 0.028 resp.) thus we can reject the null hypothesis and tell that the difference is significant.
For all other specialisation we cannot reject the null hypothesis of identical average scores (pvalue > 0.05), which means: for those specialisation we cannot say that the difference is in average statistically significant
Female vs Male (Optional)
Except for the number of students, male and females are quite similar
Can you write Python code for it?

Female = master_added_sem[master_added_sem.Sex == 'Madame']
Female.describe()

Male = master_added_sem[master_added_sem.Sex == 'Monsieur']
Male.describe()


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
Set the Sciper number as index
Can you write Python code for it?


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
Since the pvalue is > 0.05, we cannot reject the null hypothesis of identical average scores which means: we cannot say that the difference is in average statistically significant, in fact it is quite similar (p value close to 1)
Also over time nothing much changed. The females might have a small upwards trend, but the confidence intervall is quite big.
Can you write Python code for it?

sns.regplot(x=Male.last_year, y=Male.nombre_semestres, marker="o",ci=95, label='Male', scatter_kws={'s':100})
ax =sns.regplot(x=Female.last_year, y=Female.nombre_semestres, marker="d", ci= 95, label='Female', color='r')
ax.legend(loc="best")


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
Test if the difference in average is statistically significant for each spec
Can you write Python code for it?


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
We want to see if the difference of the average number of semesters for a particular specialisation and for all the master students in informatique are statistically significant with a threshold of 95%
We use a Welch's T-Test (which does not assume equal population variance): it measures whether the average  value differs significantly across samples.
Can you write Python code for it?

specialisation = master_with_spe.Specialisation.unique()

for spec in specialisation:
    print(spec)
    data_spec = master_with_spe[master_with_spe.Specialisation == spec]
    print (stats.ttest_ind(a = data_spec.nombre_semestres, b= master_added_sem.nombre_semestres, equal_var=False))
    print('\n')


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
We want to see if the difference of the average years for female and male are statistically significant with a threshold of 95%
We use a Welch's T-Test (which does not assume equal population variance): it measures whether the average  value differs significantly across samples.
Can you write Python code for it?

stats.ttest_ind(a = Female.nombre_semestres, b= Male.nombre_semestres, equal_var=False)


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
We will remove all students that did certainly not finish the master. That is, we remove students that:
- Have not done the Master 1 semestre. (There are not a lot of them (see later) and it is too cumbersome to track them and check if they eventually passed or not)
- Did only 1 semester. (They did not pass)
- Have a mineur or spec and did less than 3 semesters. (They also did not pass)
- Are registered in the current semester (have not passed (yet))
We also remove students that already have been studing in 2007 (the start of our data) because we can not tell how long they already have studied before.
Can you write Python code for it?

# find all students that have not done the 'master 1' semester -> have not studied enough to finish the master.
grouped = master_data.groupby(by='No_Sciper')
no_master_1 = pd.DataFrame(columns=['Civilité', 'Nom_Prénom', 'Orientation_Bachelor', 'Orientation_Master',
      'Spécialisation', 'Filière_opt.', 'Mineur', 'Statut', 'Type_Echange',
      'Ecole_Echange', 'No_Sciper', 'title', 'periode_acad',
      'periode_pedago'])
for scip, group in grouped:
    if (group.periode_pedago != 'Master semestre 1').all():
        no_master_1 = pd.concat([no_master_1, group])
len(no_master_1.No_Sciper.unique())

# all the students that already studied in the '2007-2008' year:
df_2006 = no_master_1[no_master_1.periode_acad == '2007-2008']
len(df_2006.No_Sciper.unique())


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
What are the different specialisations and how popular are they?
Can you write Python code for it?

# The different specialisations
master_with_spe['Specialisation'].unique()


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
add some columns that make the use of the data easier later
- store the start and end year of each entry ('2008-2009' -> 2008) and '2008-2009' -> 2009
- make a column for master1, master2, master3 and and project master and indicate if done it (true and false)
Can you write Python code for it?

master_data['periode_acad'].unique()

# store the year of the entry
def start_year(student):
    return int (student['periode_acad'].split('-')[0])
master_data['start_year'] = master_data.apply(start_year, axis=1)
master_data['end_year'] = master_data.apply(lambda st: start_year(st)+1, axis=1)

# make the indicator columns
new_cols_map = {
    'Master1': ['Master semestre 1'],
    'Master2': ['Master semestre 2'],
    'Master3': ['Master semestre 3'],
    'Project_Master' : ['Projet Master printemps', 'Projet Master automne']
}
for (new_col_name, match_list) in new_cols_map.items():
    master_data[new_col_name] = master_data.apply(lambda student: student['periode_pedago'] in match_list, axis=1)

# show the new master_data (sample randomly to keep it managable but still informative)
master_data.sample(n = 10, axis=0, replace=False)


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
almost 50% of all students do a minor or a specialisation:
Can you write Python code for it?

# how many percent of students do a specialisation
100/len(master_added_sem) * len(master_with_spe)

# how many percent of students do a minor
100/len(master_added_sem) * len(master_added_sem[~pd.isnull(master_added_sem['Mineur'])])


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
checkout what the Semester printemps and automne are
Can you write Python code for it?

all_data[all_data['periode_pedago'].isin(['Semestre printemps', 'Semestre automne'])]['title'].unique()


I am working on the file "HW02-Data_from_the_Web/master_data_analysis.ipynb".
The first task is:
remove the columns with only NaN (in case there is one)
Can you write Python code for it?


I am working on the file "HW02/Homework 2.ipynb".
The first task is:
Best universities in term of:
We selected the top 10 universities in point (a) and (b). For point (c) and (d), the top 200 universities were used in order to have more data.
(a) ratio between faculty members and students
Can you write Python code for it?

top = 10
top_universities_ratio = select_top_N(top_universities, 'overall_rank', top)

top_universities_ratio_sf = top_universities_ratio[['name', 'students_total', 'faculty_total']]
top_universities_ratio_sf = top_universities_ratio_sf.set_index(['name'])
top_universities_ratio_sf.index.name = None

fig, axes = plt.subplots(1, 1, figsize=(10,5), sharey=True)
top_universities_ratio_sf.plot.bar(stacked=True, color=colors_1, ax=axes)
axes.set_title('Top 10 ratio\'s between students and faculty members among universities')
axes.legend(labels=['students', 'faculty members'], bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
fig.autofmt_xdate()
plt.show()


top = 10
times_higher_education_ratio = select_top_N(times_higher_education, 'overall_rank', top)

times_higher_education_ratio_sf = times_higher_education_ratio[['name', 'students_total', 'faculty_total']]
times_higher_education_ratio_sf = times_higher_education_ratio_sf.set_index(['name'])
times_higher_education_ratio_normed = times_higher_education_ratio_sf.div(times_higher_education_ratio_sf.sum(1), axis=0).sort_values(by='faculty_total', ascending=False)
times_higher_education_ratio_normed.index.name = None

fig, axes = plt.subplots(1, 1, figsize=(10,5), sharey=True)
times_higher_education_ratio_normed.plot.bar(stacked=True, color=colors_1, ax=axes)
axes.set_title('Top 10 ratio\'s between students and faculty members among universities')
axes.legend(labels=['students', 'faculty members'], bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
axes.set_ylim([0.8,1])
fig.autofmt_xdate()
plt.show()


I am working on the file "HW02/Homework 2.ipynb".
The first task is:
Best university
First we have to transform ranking in some score. Here we assume a linear relation for the score given the ranking,
so we gave a score of 1 for the best ranking and 0 for the worst ranking with linear mapping between these two. We did it for each of the ranking (the two websites).
Also we don't really know if a website is most trustworthy than the other, so a good merging for the ranking would be to take the average of the two scores with equal weights for each score.
Finally we also took into account the ratio of staff member per students:
$finalScore = mean(score1, score2, staff per studiants)$
After computing these values, we found that Caltech is the best university (according to our assumptions).
Per Website ranking:
Caltech: top_universities -> 4 | times_higher_education = -> 3 | staff per student ratio -> 0.15 | => final score: 0.71
Can you write Python code for it?

r = merged[['name', 'overall_rank_x', 'overall_rank_y']]
r.head()

def lin(df):
    best_rank = df.min()
    worst_rank = df.max()
    a = 1 / (best_rank - worst_rank)
    b = 1 - a*best_rank
    return df.apply(lambda x: a*x + b)

r['stud_staff_ratio'] = merged[['faculty_international_x', 'faculty_international_y']].mean(axis=1) / \
                        merged[['students_total_x', 'students_total_y']].mean(axis=1)
r['score_x'] = lin(r['overall_rank_x'])
r['score_y'] = lin(r['overall_rank_y'])
r['overall_score'] = r[['score_x', 'score_y', 'stud_staff_ratio']].mean(axis=1)
r = r.dropna()

r[r['overall_score'] == r['overall_score'].max()]


I am working on the file "HW02/Homework 2.ipynb".
The first task is:
Comments: Aggregating the data by country, we see that Russia is the country offering more faculty members for its student, followed by Danemark and Saudi Arabia. The most international university in terms of students is Australia, followed by United Kingdom and Hong Kong. Switzerland is in the fifth position and India is the country with the lowest ratio of international students.
(d) same comparisons by region
Can you write Python code for it?

ratio_region_s = top_universities.groupby(['region'])['students_total', 'faculty_total'].sum()
ratio_region_s_normed = ratio_region_s.div(ratio_region_s.sum(1), axis=0).sort_values(by='faculty_total', ascending=False)
ratio_region_s_normed.index.name = None

fig, axes = plt.subplots(1, 1, figsize=(10,5), sharey=True)
ratio_region_s_normed.plot.bar(stacked=True, color=colors_1, ax=axes)
axes.set_title('Ratio of students and faculty members by region')
axes.legend(labels=['students', 'faculty members'], bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
axes.set_ylim([0.8,1])
axes.yaxis.grid(True)
fig.autofmt_xdate()
plt.show()

ratio_region_s = top_universities.groupby(['region'])['students_international', 'students_national'].sum()
ratio_region_s_normed = ratio_region_s.div(ratio_region_s.sum(1), axis=0).sort_values(by='students_international', ascending=False)
ratio_region_s_normed.index.name = None

fig, axes = plt.subplots(1, 1, figsize=(10,5), sharey=True)
ratio_region_s_normed.plot.bar(stacked=True, color=colors_2, ax=axes)
axes.set_title('Ratio of international and national students by region')
axes.legend(labels=['international students', 'national students'], bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
axes.set_ylim([0,0.4])
axes.yaxis.grid(True)
fig.autofmt_xdate()
plt.show()


I am working on the file "HW02/Homework 2.ipynb".
The first task is:
Comments: Asia is the region offering more faculty members to its students. It is followed by North America and Europe. The most international university in terms of students is Oceania. Europe is second.
Analysis of the two methods
We get consistent results comparing the results obtained by region or by country about the ratio of international students. By country, we get Australia and by region, Oceania. This makes sense as Australia owns nine of the eleven top_universities of Oceania.
www.timeshighereducation.com
Can you write Python code for it?

# we repeat the same procedure as for www.topuniversities.com
root_url_2 = 'https://www.timeshighereducation.com'
list_url_2 = root_url_2 + '/sites/default/files/the_data_rankings/world_university_rankings_2018_limit0_369a9045a203e176392b9fb8f8c1cb2a.json'

r = requests.get(list_url_2)
times_higher_education = pd.DataFrame()
times_higher_education = times_higher_education.from_dict(r.json()['data'])[['rank', 'location', 'location', 'name', 'url', 'stats_number_students', 'stats_pc_intl_students', 'stats_student_staff_ratio']]

# rename columns as is the first dataframe
times_higher_education.columns = ['overall_rank', 'location', 'region', 'name', 'url', 'students_total', 'ratio_inter', 'student_staff_ratio']

# as the ranks have different represetation we had to delete the '=' in front of universities that have the same rank,
# rewrite the rank when it is represented as an interval (ex: 201-250) and finally delete the '+' in the end for the last ones
times_higher_education['overall_rank'] = times_higher_education['overall_rank'].apply(lambda rank: re.sub('[=]', '', rank))
times_higher_education['overall_rank'] = times_higher_education['overall_rank'].apply(lambda rank: rank.split('–')[0])
times_higher_education['overall_rank'] = times_higher_education['overall_rank'].apply(lambda rank: re.sub('[+]', '', rank)).astype(int)

# remaps a ranking in order to make selection by ranking easier
# ex: 1,2,3,3,5,6,7 -> 1,2,3,3,4,5,6
def remap_ranking(rank):
    last=0
    for i in range(len(rank)):
        if last == rank[i]-1:
            #no problem
            last = rank[i]
        elif last != rank[i]:
            last = last+1
            rank[i] = last
            rank[(i+1):] = rank[(i+1):]-1
    return rank

times_higher_education['overall_rank'] = remap_ranking(times_higher_education['overall_rank'].copy())

# in the following lines we make the necessary transformation in order to get the right type or numbers for each column
times_higher_education['students_total'] =...


I am working on the file "HW02/Homework 2.ipynb".
The first task is:
Comments: In the first plot, we see that Africa is the region where there is more faculty members by students. The two following regions are very close to each other. In the second plot, Oceania is the more internationalized school in terms of its students and Europe is second. We had similar results by the other website concerning this last outcome.
Can you write Python code for it?

# Detects same universities with different names in the two dataframes before merging
# using Jaccard similarity and same location rule (seems to keep matching entry)
def t(x):
    # Compute Jaccard score (intersection over union)
    def jaccard(a, b):
        u = set(a.split(' '))
        v = set(b.split(' '))
        return len(u.intersection(v)) / len(u.union(v))

    names = top_universities['name'].tolist()
    locations = top_universities['location'].tolist()
    scores = np.array([jaccard(x['name'], n) for n in names])
    m = scores.max()
    i = scores.argmax()
    # Jaccard score for similarity and location match to filter out name with different locations
    if m > 0.5 and x['location'] == locations[i]:
        x['name'] = names[i]
    return x

# Match universities name in both dataframes
times_higher_education = times_higher_education.apply(t, axis=1)
# Intersection on the name column of the two datasets
merged = pd.merge(top_universities, times_higher_education, on='name', how='inner')

merged.head()


I am working on the file "HW02/Homework 2.ipynb".
The first task is:
Comments: Luxembourg has more international than national students which allows it to be in first position without difficulty. Switzerland is in the sixth position (versus fifth for top university website).
(d) same comparisons by region
Can you write Python code for it?

# Some countries have their field 'region' filled with 'N/A': this is due to the technique we used to write the
# correct region for each university. In the sample we are considering, let's see how many universities are concerned:
times_higher_education[times_higher_education['region'] == 'N/A']

# As there is only two universities concerned, we can rapidly write it by hand. Of course we should have develop a
# more generalized manner to do it, if we had a much larger sample.
times_higher_education.set_value(178, 'region', 'Europe')
times_higher_education.set_value(193, 'region', 'Europe')

ratio_region_s = times_higher_education.groupby(['region'])['students_total', 'faculty_total'].sum()
ratio_region_s_normed = ratio_region_s.div(ratio_region_s.sum(1), axis=0).sort_values(by='faculty_total', ascending=False)
ratio_region_s_normed.index.name = None

fig, axes = plt.subplots(1, 1, figsize=(10,5), sharey=True)
ratio_region_s_normed.plot.bar(stacked=True, color=colors_1, ax=axes)
axes.set_title('Ratio of students and faculty members by region')
axes.legend(labels=['students', 'faculty members'], bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
axes.set_ylim([0.8,1])
axes.yaxis.grid(True)
fig.autofmt_xdate()
plt.show()

ratio_region_s = times_higher_education.groupby(['region'])['students_international', 'students_national'].sum()
ratio_region_s_normed = ratio_region_s.div(ratio_region_s.sum(1), axis=0).sort_values(by='students_international', ascending=False)
ratio_region_s_normed.index.name = None

fig, axes = plt.subplots(1, 1, figsize=(10,5), sharey=True)
ratio_region_s_normed.plot.bar(stacked=True, color=colors_2, ax=axes)
axes.set_title('Ratio of international and national students by region')
axes.legend(labels=['international students', 'national students'], bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
axes.set_ylim([0,0.4])
axes.yaxis.grid(True)
fig.autofmt_xdate()
plt.show()


I am working on the file "HW02/Homework 2.ipynb".
The first task is:
Comments: The most international university, by its students, among the top 10 universities is the Imperial College London. Notice that ETHZ is in the third position.
(c) same comparisons by country
Can you write Python code for it?

ratio_country_sf = top_universities.groupby(['location'])['students_total', 'faculty_total'].sum()
ratio_country_sf_normed = ratio_country_sf.div(ratio_country_sf.sum(1), axis=0).sort_values(by='faculty_total', ascending=False)
ratio_country_sf_normed.index.name = None

fig, axes = plt.subplots(1, 1, figsize=(15, 5))
ratio_country_sf_normed.plot.bar(stacked=True, color=colors_1, ax=axes)
axes.set_title('Ratio of students and faculty members by country')
axes.legend(labels=['students', 'faculty members'], bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
axes.set_ylim([0.8,1])
fig.autofmt_xdate()
plt.show()

ratio_country_s = top_universities.groupby(['location'])['students_international', 'students_national'].sum()
ratio_country_s_normed = ratio_country_s.div(ratio_country_s.sum(1), axis=0).sort_values(by='students_international', ascending=False)
ratio_country_s_normed.index.name = None

fig, axes = plt.subplots(1, 1, figsize=(15, 5))
ratio_country_s_normed.plot.bar(stacked=True, color=colors_2, ax=axes)
axes.set_title('Ratio of international and national students by country')
axes.legend(labels=['international students', 'national students'], bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
axes.set_ylim([0, 0.4])
fig.autofmt_xdate()
plt.show()


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
Add a function to control the writes to csv file.
Can you write Python code for it?

# set to true if you want to write the data to csv
do_persist = True

def write_to_csv(df, file_name):
    if do_persist:
        df.to_csv(file_name)
        print('...done')
        return True
    else:
        return False


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
And finally group by the institutions and sum the grants.
We show the sorted dataframe
It is interesting that neither ETHZ nor EPFL has received the most grants.
Can you write Python code for it?

universities = grants.groupby(by='university', axis=0, as_index=False).sum()
universities.sort_values('amount', ascending=False)


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
How many did we match?
Can you write Python code for it?


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
If one of the two values is NaN, we can't use the entry, a grant without institution is as meanningless as a institution without grant. -> We drop it.
Note that there are entries with 'Nicht zuteilbar - NA' which are essentially also NaN values.
We drop almost 25% of all entries, which seems a lot, but what else can we do with uncomplete data?
Can you write Python code for it?

nbr_entries = len(grants)
grants = grants.replace(to_replace='Nicht zuteilbar - NA', value=np.nan)
grants = grants.dropna()
print('Dropped '+str((100/nbr_entries) * (nbr_entries - len(grants)))+'% of all entries.')


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
Load the data and check the types of the columns
Can you write Python code for it?

data = pd.read_csv('data/GrantExport.csv', delimiter=';')
data.dtypes


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
Look at the DF
Can you write Python code for it?


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
Map the institutions to cantons
In this section we will map each institution to the canton it belongs to with following steps:
1. split the institution name into a 'university_name' and a 'abbreviation' part
2. query the google places api with the university name (stripped of some special characters) and the keyword 'switzerland'
3. then take the coorinates returned by google and use the geonames api to find the canton of the coordinates
4. If for some reason no canton is found (google or geonames did not find any match) we look if the 'university_name' contains a canton name, if so we map it to that canton.
With that method we can map 61 out of the 76 institutions.
The rest we mapped manually.
One institution is present in 3 cantons, we just split the grants of it equally among the different cantons.
Other institutions are all over the place, those we did not match as it would be too dificult to find all cantons they have a presence in.
Note that there are several 'institutions' that can't be mapped such as 'Weitere Spitäler' (engl: 'other hospitals'). But they only account for ~4% of all grants, which is low enough for us to ignore.
Lets start:
Split the names
Can you write Python code for it?

delim = ' - '
universities['university_name'] = [fn.split(delim)[0].strip() for fn in universities['university'].values]
universities['abbrev'] = [fn.split(delim)[1].strip() if len(fn.split(delim)) > 1 else np.nan for fn in universities['university'].values]
universities.set_index('university', inplace=True)
universities.head(1)


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
Some numbers on how many we matched.
how many institutions did we match?
Can you write Python code for it?

# the 1+len(...) accounts for the mapping of EMPA
print(str(round((100/ len(universities) ) * (1+len(universities[~pd.isnull(universities['canton'])]))) )+ '%')


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
The functions for using the APIs are situated in the file 'map_universities.py', so we run it.
Can you write Python code for it?

# run the python file that defines the functions to access the api's
%run map_universities.py


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
The institution EMPA has presence in 3 cantons: BE, ZH, SG. So we split the grants for EMPA and add it to the 3 cantons (1/3 for each)
Can you write Python code for it?

grants_empa = universities.at['Eidg. Material und Prüfungsanstalt - EMPA', 'amount']
grants_empa_third = grants_empa / 3
empa_cantons = ['BE', 'ZH', 'SG']
for c in empa_cantons:
    canton_grants = canton_grants.set_value(c, 'amount', canton_grants.at[c, 'amount'] + grants_empa_third)


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
Then make the 'amount' column numeric in order to make the utilisation easier.
Can you write Python code for it?

grants['amount'] = pd.to_numeric(grants.amount, errors='coerce')


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
Then match each institution:
Can you write Python code for it?

# create a new column with the canton in it.
def canton_for_university_query(uni):
    # create the query
    q = str(uni.university_name) + ' ' + str(uni.abbrev) + ' Switzerland'
    # remove some special characters in the query
    to_remove = ['(', ')', ',', '.', '-', '+', '&']
    [q.replace(ch, ' ') for ch in to_remove]
    # execute the query
    return canton_for_university(q)

universities['canton'] = universities.apply(canton_for_university_query, axis=1)


I am working on the file "HW03-Interactive_Viz/HW03-collect.ipynb".
The first task is:
We only need the columns 'University' and 'Approved Amount'.
All other columns are not relevant for this homework.
Can you write Python code for it?

# take only the relevant cols and give them nicer names
grants = data[['University', 'Approved Amount']]
grants.rename(columns={'University': 'university', 'Approved Amount' : 'amount'}, inplace=True)
grants.dtypes


I am working on the file "HW03-Interactive_Viz/HW03-vizualisation.ipynb".
The first task is:
Bonus
Can you write Python code for it?


I am working on the file "HW03-Interactive_Viz/HW03-vizualisation.ipynb".
The first task is:
Group by language (FR, D, IT) and sum the amount. Then rename the amount columns to amount_by_language
Can you write Python code for it?

grant_language =   grant_rostigraben.groupby(by='language', axis=0, as_index=False).sum()
grant_language.rename(columns={'amount':'amount_by_language'}, inplace=True)


I am working on the file "HW03-Interactive_Viz/HW03-vizualisation.ipynb".
The first task is:
Join the the table group by language and the one containing the grant by canton
Can you write Python code for it?

grant_with_language = pd.merge(grant_rostigraben, grant_language, how='inner')
grant_with_language = pd.merge(grant_rostigraben, grant_language, on='language')
grant_with_language.head()

rostigraben_map = folium.Map(location=[46.5966, 7.9761],zoom_start=7)
rostigraben_map.choropleth(
                geo_path=swiss_canton,
                topojson='objects.cantons',
                data=grant_with_language,
                columns=['canton', 'amount_by_language'],
                key_on='id',
                #threshold_scale=[7,8,9],
                fill_color='YlOrRd', fill_opacity=0.7, line_opacity=0.6,
                legend_name='Grant money received by language (CHF)'
            )

rostigraben_map.save('rostigraben_map.html')
rostigraben_map
#the map is not rendered on Github. To see it, you can download rostigraben_map.html and open it on your browser.


I am working on the file "HW03-Interactive_Viz/HW03-vizualisation.ipynb".
The first task is:
Load the topojson with the Swiss cantons and the CSV containing the grants by canton
Can you write Python code for it?

swiss_canton = 'ch-cantons.topojson.json'
grant_data = pd.read_csv(r'all_canton_grants.csv')


I am working on the file "HW03-Interactive_Viz/HW03-vizualisation.ipynb".
The first task is:
Map each canton to its languages. We apply the function get_language (in map_university.py) that given an abreviation of canton, return a list of language. We didn't consider romanish since it'is only spoken by a small part of the Graubünden.
Can you write Python code for it?

grant_rostigraben['language'] = grant_rostigraben.canton.apply(get_language)
grant_rostigraben.head()


I am working on the file "HW03-Interactive_Viz/HW03-vizualisation.ipynb".
The first task is:
Some cantons have 2 languages (Fribourg, Berne and Valais). We need to split the list and create a second attribut language2. The array of language is sorted by main language. So language contains the main language first and language2 int the second place
Can you write Python code for it?

def split_if_two_language(row):
    l = row['language']
    row.language = l[0]
    if(len(l)>1):
        row['language2'] = l[1]
    return row

grant_rostigraben = grant_rostigraben.apply(split_if_two_language, 1)

grant_rostigraben.head()


I am working on the file "HW03-Interactive_Viz/HW03-vizualisation.ipynb".
The first task is:
We want to compare the difference between the area divided by the röstigraben
So we need:
1) Map each canton to its main language. Since the topojson allow us only to give one color for each canton, we only consider the main language of each canton. It means:
    * Fribourg -> French
    * Valais -> French
    * Bern -> German
    * Graubünden -> German)
2) Group by each language and sum the grant
3) Join the list of canton with the list of grant by language
4) Show the map with the color code mapped to the amount by language
Can you write Python code for it?


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
1. Observation and clean of the data
Can you write Python code for it?

print('Number of diad: ', len(data))
print('Number of players: ', len(data.playerShort.unique()))
print('Number of referees: ', len(data.refNum.unique()))


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
1.1. Missing values
Can you write Python code for it?

complete = len(data.dropna())
all_ = len(data_total)
print('Number of row with complete data: {} ({:.3f}%)'.format(complete, (complete/all_ ) * 100 ))
print('Number of row with missing data: {} ({:.3f}%)'.format(all_-complete, (all_ -complete)/all_  * 100 ))


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
1.2. Handling different ratings (rater1 vs rater2)

We check if there is player with only one rating
We also check if one rater always give the same rate to the same player
Now we need to take one single rating for each row. We'll need to investigate a bit more the different rating

Is there players with only one rating?
Can you write Python code for it?


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
Aggregate the data
One solution is to group the data by player name. Then we need to find a solution to correctly group the remaining features:
    - club: we have to check if a player appear in 2 different club (in case of a transfer during the winter mercato ) or if the transfer are not taking into account. (-> one (several) hot encoding. or majority dyads per club)
    - leagueCountry: same as club
    - position: test if the player as different -> position with the majority of game?
    - photoID: drop that information, the id is unique -> not relevant for our classification probleme
    - refNum: replace with the total of unique refs
    - refCountry: same as refNum
    - Alpha_3: remove: redundant information since it correspond to the refCountry
    - meanIAT: make new features
        - take mean
        - take weighted mean (weight with nIAT)
        - take weighted mean (weight with game numers)
    - meanExp: same as IAT
    - seAIT:
    - seExp:
First do some checks
Can you write Python code for it?

clubUnique = True
leagueUnique = True
positionUnique = True
def checkFunction(player):

    #check if the club is unique for one player
    if len(player.club.unique()) > 1:
        clubUnique = False
        print(player.player, 'plays for more than one team: ', player.club.unique())
    #check if the leagueCountry is unique
    if len(player.leagueCountry.unique()) > 1:
        leagueUnique = False
        print(player.player, 'plays for more than one league: ', player.leagueCountry.unique())
    #check if the position is unique
    if len(player.position.unique()) > 1:
        positionUnique = False
        print(player.player, 'plays for more than one position: ', player.position.unique())

data_group = pd.groupby(data_cleaned, by=data_cleaned.playerShort).apply(checkFunction)
print("Is the club for a player unique? ", clubUnique)
print("Is the league for a player unique? ", leagueUnique)
print("Is the position for a player unique? ", positionUnique)


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
Are the raters always agree
Can you write Python code for it?

print("percentage of players with different ratings: ", len(data[data['rater1'] != data['rater2']])*100 / len(data), "%")


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
Check now how many incomplete dayads our data contains now
Can you write Python code for it?

complete = len(data.dropna())
all_ = len(data)
print("After removing data without rating:")
print("-----------------------------------")
print('Number of row with complete data: {} ({:.3f}%)'.format(complete, (complete/all_ ) * 100 ))
print('Number of row with missing data: {} ({:.3f}%)'.format(all_-complete, (all_ -complete)/all_  * 100 ))


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
Encode the data for the Random Forest
The Random Forest can only handle integer or float attributes, so we have to encode the string attributes as numbers.
Can you write Python code for it?

def encode(data_frame, inplace=False):
    """
    encodes the non numerical columns with a LabelEncoder
    returns a new data_frame if inplace = False, otherwise changes the given data_frame
    """
    _df = data_frame if inplace else data_frame.copy()
    le = pp.LabelEncoder() # encoder
    for col_name in _df.columns:
        if _df.dtypes[col_name] == 'O':
            _df[col_name] = le.fit_transform(_df[col_name])
            print("encoded", col_name)
    return _df

data_cleaned_encoded = encode(data_cleaned, inplace=False)
data_cleaned_encoded.head(1)

data_cleaned_aggregated_encoded = encode(data_agregated, inplace=False)
data_cleaned_aggregated_encoded.head(1)

data_cleaned_encoded.to_csv('CrowdstormingDataJuly1st_preprocessed_encoded.csv')
data_cleaned_aggregated_encoded.set_index("playerShort").to_csv('CrowdstormingDataJuly1st_aggregated_encoded.csv')

data_cleaned_aggregated_encoded.set_index("playerShort").head()


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
For those player we will try to assign them a new position called 'Joker'
Can you write Python code for it?

data.position = data.position.fillna('Joker')

missing_col = find_col_nan(data)
missing_col

complete = len(data.dropna())
all_ = len(data)
print("After removing data without rating:")
print("-----------------------------------")
print('Number of row with complete data: {} ({:.3f}%)'.format(complete, (complete/all_ ) * 100 ))
print('Number of row with missing data: {} ({:.3f}%)'.format(all_-complete, (all_ -complete)/all_  * 100 ))


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
Investigation on the raters
Values of the rating
Can you write Python code for it?


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
Let's show some plost to visualize when the rater differe
Can you write Python code for it?

len(data)

fig, ax = plt.subplots(1, 4, figsize=(16, 4))
ax[0].hist([data['rater1'], data['rater2']], bins=5)
ax[0].set_title("1) Raters compared \n(blue: rater1, green: rater2)")

ax[1].hist(abs(data['rater1'] - data['rater2']), bins=3)
ax[1].set_title("2) Difference (Rater1 - Rater2)")

dissagree_data = data[data['rater1'] != data['rater2']]
ax[2].hist(dissagree_data['rater1'], bins=5)
ax[2].set_title("3) Dissagreed values Rater1")

ax[3].hist(dissagree_data['rater2'], bins=5, color='seagreen')
ax[3].set_title("4) Dissagreed values Rater2")


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
The remaining NaN represent only 0.1% and concern the columns about country and the test in those country which are hard to guess. So we deceide to drop the remaining dayad with NaN
Can you write Python code for it?


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
Then aggregate
Can you write Python code for it?

def meanCards(row, test):
    total_cards =   row.yellowCards + row.yellowReds + row.redCards
    if total_cards == 0:
            return 0
    else:
        if(test == 'IAT'):
            return (row.meanIAT * row.yellowCards) + (row.meanIAT * row.yellowReds)  \
             + (row.meanIAT * row.redCards) / total_cards
        else:
            return (row.meanExp * row.yellowCards) + (row.meanExp * row.yellowReds)  \
             + (row.meanExp * row.redCards) / total_cards

def aggreagtion(df):
    first_entry = df.head(1)
    # new aggregation entry
    new_entry = first_entry.copy()

    #sum of the info about the games
    new_entry.games = df.games.sum()
    new_entry.victories = df.victories.sum()
    new_entry.ties = df.ties.sum()
    new_entry.defeats = df.defeats.sum()
    new_entry.goals = df.goals.sum()
    new_entry.yellowCards = df.yellowCards.sum()
    new_entry.yellowReds = df.yellowReds.sum()
    new_entry.redCards = df.redCards.sum()

    #drop photoID and alpha_3
    new_entry.drop('photoID', inplace = True, axis=1)
    new_entry.drop('Alpha_3', inplace = True, axis=1)

    #refNum: number of unique ref
    new_entry = new_entry.rename(columns = {'refNum': 'refCount'})
    new_entry.refCount = len(df.refNum.unique())

    #refCountry: replace by number of unique country
    new_entry = new_entry.rename(columns = {'refCountry': 'refCountryCount'})
    new_entry.refCountryCount = len(df.refCountry.unique())

    #==Mean of the test result ===
    #- take mean
    #- take weighted mean (weight with nIAT)
    #- take weighted mean (weight with game numers)

    new_entry.meanIAT = df.meanIAT.mean()
    new_entry.meanExp = df.meanExp.mean()

    new_entry['meanIAT_nIAT'] = (df.meanIAT * df.nIAT).sum() / df.nIAT.sum()
    new_entry['meanExp_nExp'] = (df.meanExp * df.nExp).sum() / df.nExp.sum()

    new_entry['meanIAT_GameNbr'] = (df.meanIAT * df.games).sum() / df.games.sum()
    new_entry['meanExp_GameNbr'] = (df.meanExp * df.games).sum() /...


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
To clean the data, we will go step by step:
    - First of all we have to clean all dayads that don't have any rating, because those dayads are uneseful for our problem.
    - Then we will look again which columns contains missing values and how to deal with them
Can you write Python code for it?

data = data[ ~data.rater1.isnull() & ~data.rater2.isnull()]
print('Number of row with the 2 ratings {} ({:.3f}%)'.format(len(data), len(data)/len(data_total) * 100))
onlyOne = data[ ~data.rater1.isnull() ^ ~data.rater2.isnull()]
print('Number of row with only one ratings {} ({:.3f}%)'.format(len(onlyOne), len(onlyOne)/len(data_total) * 100))


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
We can see that rater1 and 2 disagree the most when they have to rate "white" people.
We can also see with graph2 that when there disagree it's ony of one category.
Now we will create a new columns: color_skin that will be our label to guess. To convert the values of rater 1 and 2 in one rate, we need to follow some rules that come from the graph:
    1. if rater1 and rater2 are agree, take that value
    2. We can see on graph 4 that when rater2 give 0, usually, rater1 agrees => so when rater2 give 0, we take that value as the color skin
    3. In graph 3, when rater1 give 1, rater2 usually agrees => when rater1 give 1, take that value as the color_skin
    4. In graph 3, we can see that when rater1 rate1 give 0.25, usually rater2 agrees => take rater1
    5. choose at random between both values
Then we can drop the features rater1, rater2 since we don't need them anymore
Can you write Python code for it?

data['color_rating'] = -1

def color_skin_rules(row):
    #Rule 1
    if row.rater1 == row.rater2:
        return row.rater1
    #Rule2
    elif row.rater2 == 0:
        return 0
    #Rule 3
    elif row.rater1 == 1:
        return 1
    #Rule 4
    elif row.rater1 == 0.25:
        return 0.25
    else:
        return np.random.choice([row.rater1, row.rater2])
data.color_rating = data.apply(color_skin_rules, axis=1)


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
We only 7.4% of incomplete dayads. Let's try to replace the nan in heights and weights with the median
Can you write Python code for it?

# replace no height and weight with the mean value
median_height = np.median(data['height'].dropna())
median_weight = np.median(data['weight'].dropna())
data['height'] = data['height'].fillna(value=median_height)
data['weight'] = data['weight'].fillna(value=median_weight)

complete = len(data.dropna())
all_ = len(data)
print("After removing data without rating:")
print("-----------------------------------")
print('Number of row with complete data: {} ({:.3f}%)'.format(complete, (complete/all_ ) * 100 ))
print('Number of row with missing data: {} ({:.3f}%)'.format(all_-complete, (all_ -complete)/all_  * 100 ))

missing_col = find_col_nan(data)
missing_col


I am working on the file "HW04-Applied ML/HW04-1-Preprocessing.ipynb".
The first task is:
Where are the missing values ?
Can you write Python code for it?

def find_col_nan(d):
    col = []
    for c in d.columns:
        if d[c].isnull().any():
            col = np.append(col, c)
    return col

missing_col = find_col_nan(data)
missing_col


I am working on the file "HW04-Applied ML/HW04-2-RandomForest.ipynb".
The first task is:
Drop the player unique features because they can't be usefull to classify since they are unique.
Can you write Python code for it?

data_aggregated = data_aggregated.drop(['playerShort', 'player', 'birthday'], axis=1)