I am working on the file "EEC2006/3_TransparencyPortal/UFRN Transparency Portal - Name gender analysis - Main.ipynb".
The first task is:
So after realizing the scrapping we have acquired the following informations:

cpf - CPF stands for 'Cadastro de Pessoas Único' that is a unique number for
cpf_region - The nascence region from the CPF
name - Name of the employee
emp_enrollment - Enrollment number
emp_responsability - Job responsability
emp_responsability_class - Job class number
emp_responsability_pattern - Job patter number
emp_responsability_reference - Job reference number
emp_responsability_level -
emp_UF - State from the location
emp_UORG -
emp_legal_regime -
emp_activit -
emp_absence_from_work -
emp_work_time -
emp_last_job_responsability_modification_date -
emp_job_responsability_nominee_date -
emp_job_responsability_nominee_act -
emp_last_job_responsability_modification_date_body -
emp_entrance_public_service_legal_document -
emp_entrance_public_service_legal_document_number -
emp_entrance_public_service_legal_document_date -
responsability_description -
responsability_activity -
responsability_activity_parcial -
responsability_UF -
responsability_UORG -
responsability_sup_body -
responsability_last_data_change_resp -
hrefLevel2 - URL for level two
url_level_three - URL for level three
emp_month - Month of the paycheck
emp_year - Year of the paycheck
emp_tot_paycheck - Total of the paycheck
emp_irrf - Income taxes
emp_rgps - Retirement taxes
emp_other_deduction - Other taxes
emp_paycheck_after_deduction - Income after deductions
emp_other_receivings - Other values that do not pay taxes
first_name - First name of employee
last_name -  Last name of employee
gender_namsor - Return from the NamSor service
gender_namsor_adjusted - Adjustment for unkown names 'manually' made
gender_pygenderbr - Return from the pygenderbr
gender_genderguesser - Return from the genderguesser
Can you write Python code for it?


I am working on the file "EEC2006/3_TransparencyPortal/UFRN Transparency Portal - Name gender analysis - Main.ipynb".
The first task is:
The Gender Guesser API utilizes more classification values as its return, so we made a adjustment to make all three tools/services similar in output. The Gender Guesser output andy, mostly_female and mostly_male will be categorized as unknown.
Can you write Python code for it?

dfUfrnComplete['gender_genderguesser_adjusted'] = dfUfrnComplete['gender_genderguesser']
dfUfrnComplete.loc[(dfUfrnComplete['gender_genderguesser_adjusted'] == 'andy') | (dfUfrnComplete['gender_genderguesser_adjusted'] =='mostly_female') | (dfUfrnComplete['gender_genderguesser_adjusted'] =='mostly_male'),'gender_genderguesser_adjusted'] = 'unknown'

dfUfrnComplete['gender_genderguesser_adjusted'].value_counts(sort=True)


I am working on the file "EEC2006/3_TransparencyPortal/UFRN Transparency Portal - Name gender analysis - Main.ipynb".
The first task is:
We will adjust the names that are unkown in the NAMSOR servise manually. For this, we will all first name finished in 'o', 'os', 'on', 'u', 'us' as male names. The other names, like finished in 'e', 'i', and etc do not follow a constant like male names do with o and u. However, there is a list of know names finished in 'o' that are female names and are:

Aiko

Amparo

Anunciação

Assunção

Calipso

Carmo

Carminho

Cléo

Charo

Clio

Conceição

Consolação

Consuelo

Dido

Erato

Índigo

Ino

Io

Izaro

Juno

Keiko

Lerato

Leto

Lilo

Lucero

Margô

Mirto

Purificação

Rocío

Rosário

Rosarinho

Socorro

Tamiko

Tariro

Temisto

Yoko

so for this we could seach for the first name:
Can you write Python code for it?

dfUfrnComplete[dfUfrnComplete['first_name'].str.contains("AIKO | AMPARO | ANUNCIAÇÃO | ASSUNÇÃO | CALIPSO | CARMO  | CARMINHO | CLÉO | CHARO | CLIO | CONCEIÇÃO | CONSOLAÇÃO | CONSUELO | DIDO | ERATO | ÍNDIGO | INO | IO | IZARO | JUNO | KEIKO | LERATO | LETO | LILO | LUCERO | MARGÔ | MIRTO | PURIFICAÇÃO | ROCÍO | ROSÁRIO | ROSARINHO | SOCORRO | TAMIKO | TARIRO | TEMISTO | YOKO")==True]


I am working on the file "EEC2006/3_TransparencyPortal/UFRN Transparency Portal - Name gender analysis - Main.ipynb".
The first task is:
as we can see there is no name of the given list se now we can generalize and say that the names that fall in the rule above described are male names
Can you write Python code for it?

dfUfrnComplete['gender_namsor_adjusted'] = dfUfrnComplete['gender_namsor']

dfUfrnComplete.loc[(dfUfrnComplete['first_name'].str[-1] == 'O') & (dfUfrnComplete['gender_namsor'] =='unknown'),'gender_namsor_adjusted']#= 'male'
dfUfrnComplete.loc[(dfUfrnComplete['first_name'].str[-2:] == 'ON') & (dfUfrnComplete['gender_namsor'] =='unknown'),'gender_namsor_adjusted'] #= 'male'
dfUfrnComplete.loc[(dfUfrnComplete['first_name'].str[-1] == 'U') & (dfUfrnComplete['gender_namsor'] =='unknown'),'gender_namsor_adjusted'] #= 'male'
dfUfrnComplete.loc[(dfUfrnComplete['first_name'].str[-2:] == 'US') & (dfUfrnComplete['gender_namsor'] =='unknown'),'gender_namsor_adjusted'] #= 'male'
dfUfrnComplete.loc[(dfUfrnComplete['first_name'].str[-2:] == 'OS') & (dfUfrnComplete['gender_namsor'] =='unknown'),'gender_namsor_adjusted'] = 'male'

#dfUfrnComplete[(dfUfrnComplete['first_name'].str[-1] == 'O') & (dfUfrnComplete['gender_namsor'] =='unknown')]
#dfUfrnComplete[(dfUfrnComplete['first_name'].str[-2:] == 'ON') & (dfUfrnComplete['gender_namsor'] =='unknown')]
#dfUfrnComplete[(dfUfrnComplete['first_name'].str[-1] == 'U') & (dfUfrnComplete['gender_namsor'] =='unknown')]
#dfUfrnComplete[(dfUfrnComplete['first_name'].str[-2:] == 'US') & (dfUfrnComplete['gender_namsor'] =='unknown')]
#dfUfrnComplete[(dfUfrnComplete['first_name'].str[-2:] == 'OS') & (dfUfrnComplete['gender_namsor'] =='unknown')]
dfUfrnComplete[(dfUfrnComplete['gender_namsor_adjusted'] =='unknown')]['first_name']

#dfUfrnComplete.loc[dfUfrnComplete['first_name'].str.contains("ACYNELLY | AISLANIA | ALCIONE | ALMARIA | AMALUSIA | ANEIDE | ANELLYSA | AURIGENA | AUZELIVIA | BARNORA | CLAUDIANNY | CRISLUCI | CRISTHIANNE | DERISCLEIA | EDILZA | EDZANA | SANZIA | SAONARA | SEMELY | SHEYLENA | SISLLEY | SONAYDY | SORANEIDE | SUELENE | SUENI | SUENIA | SULEMI | SUZERICA | TAIZA | THAISE | THAIZA | THATYANE | VALDECY | VALDENIA | WALANNE | WALDENICE | WANDERLEIA | WANUSIA | WICLIFFE | YULYANNA | ZORAIDE ")==True]['gender_namsor_adjusted'] =...


I am working on the file "EEC2006/5_FinalProject_Regression_KNN/.ipynb_checkpoints/ForestFirePredictionVictor-checkpoint.ipynb".
The first task is:
3. Start - Loading, Checking and Adjusting Data Set
Can you write Python code for it?

import math
#
import numpy as np
import pandas as pd

#
%matplotlib inline
import matplotlib.pyplot as plt

firedb = pd.read_csv("forestfires.csv")
firedb.columns


I am working on the file "EEC2006/5_FinalProject_Regression_KNN/.ipynb_checkpoints/ForestFirePredictionVictor-checkpoint.ipynb".
The first task is:

Principal Component Analisys

Experiments
Can you write Python code for it?

from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import cross_val_score, KFold
from sklearn import preprocessing

firedb.head()

hyper_params = range(1,20)
features_list = [ ['FFMC'], ['DMC'], ['DC'], ['ISI'], ['temp'], ['RH'], ['wind'], ['rain'],
                  ['X', 'Y', 'FFMC', 'DMC','DC','ISI'], ['X', 'Y', 'temp','RH','wind','rain'],
                  ['FFMC', 'DMC','DC','ISI'], ['temp','RH','wind','rain'],
                  ['DMC', 'wind'], ['DC','RH','wind'],
                  ['X', 'Y', 'DMC', 'wind'], ['X', 'Y', 'DC','RH','wind'],
                  ['FFMC', 'DMC','DC','ISI', 'temp','RH','wind','rain'],
                  ['X', 'Y', 'FFMC', 'DMC','DC','ISI', 'temp','RH','wind','rain']  ]

num_folds = [3, 5, 7, 9, 10, 11, 13, 15, 17, 19, 21, 23]

outputs = ['area', 'area_adjusted']

import csv

# initializing our file that will act like our database for the results

#open the file in the 'write' mode
file = open('results_db.csv','w')
writer = csv.writer(file)

db_row = ['HYPER_PARAM', 'FEATURES', 'K_FOLDS', 'OUTPUT',
          'AVG_RMSE', 'AVG_RMSE_%_AREA', 'STD_RMSE', 'CV_RMSE',
          'AVG_MAE', 'AVG_MAE_%_AREA', 'STD_MAE', 'CV_MAE']

#write the row mounted
writer.writerow(db_row)

#close the file
file.close()

from IPython.core.display import clear_output
from time import time

start_time = time()

k = 0
k_total = len(hyper_params) * len(features_list) * len(num_folds) * len(outputs)

for hp in hyper_params:
    for features in features_list:
        for fold in num_folds:
            for output in outputs:
                k += 1

                kf = KFold(fold, shuffle=True, random_state=1)
                model = KNeighborsRegressor(n_neighbors = hp, algorithm='auto')

                mses = cross_val_score(model, firedb[features],
                                       firedb[output], scoring="neg_mean_squared_error", cv=kf)
        ...


I am working on the file "EEC2006/5_FinalProject_Regression_KNN/.ipynb_checkpoints/ForestFirePredictionVictor-checkpoint.ipynb".
The first task is:

Pq a transformação?

Can you write Python code for it?

fig, axes = plt.subplots(nrows=1, ncols=2)
firedb['area'].plot.hist(ax=axes[0],figsize=(17,8))
firedb['area_adjusted'].plot.hist(ax=axes[1])


I am working on the file "EEC2006/5_FinalProject_Regression_KNN/.ipynb_checkpoints/ForestFirePredictionVictor-checkpoint.ipynb".
The first task is:
No null values.
Can you write Python code for it?

firedb.info()

firedb['area'] = firedb['area'].astype(np.float32)

firedb[['FFMC','ISI','temp','RH','wind','rain']].plot(figsize=(17,15))

firedb['month'].value_counts()

firedb[['area']].plot(figsize=(17,15))

firedb['area_adjusted'] = np.log(firedb['area']+1)


I am working on the file "EGU2015_FOSS_notebooks.ipynb".
The first task is:
Measured vs modelled discharge:
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(10, 6))
p1 = measured.plot(ax=ax, label='measured')
p2 = modeloutput1.plot(ax=ax, label='modelled')
t = ax.set_ylabel(r'Q m$^3$s$^{-1}$')
plt.legend(['measured', 'modelled'])

from scatter_hist2 import create_scatterhist, create_seasonOF

names = pars.columns
time=np.array(measured.index)
modelled.index = time

pars_name={}
for i in range(0, names.size):
    pars_name[names[i]]=i


I am working on the file "EGU2015_FOSS_notebooks.ipynb".
The first task is:
What about the model?

Parameter boundaries correct?
Optimal paramersets change periodically... Correct model structure?
NOT an optimization tool!

The function

Select parametersets based on:
Objective function (SSE, RMSE, RRMSE)
Time period of interest (whole year or specific season)
Relative threshold (scaled between 0 and 1)

Visualisation of a 2D parameter response surface of selected parametersets together with histograms

More interactive?
command:
interact(...)
Can you write Python code for it?

#Loading interact functionality
from IPython.html.widgets import interact, fixed


I am working on the file "EIT QuTiP.ipynb".
The first task is:
First define the $\hat{\sigma}_{ij}$ operators according to atomic levels shown in the following figure. Decay rates are also indicated.
Can you write Python code for it?

one, two, three = three_level_basis()
sig11 = one * one.dag()
sig22 = two * two.dag()
sig33 = three * three.dag()
sig13 = one * three.dag()
sig23 = two * three.dag()
sig12 = one * two.dag()

Image("fig/EIT_levels.png")

# Loop over the detuning Delta and generate optical response

# constants based on Jenkins EIT masters thesis
# It seems like the values they quote aren't right. They don't match the code.

# Decay rates
Gamma = 0.1  # decay rate on probe transition
gamma = 5.0  # decay rate on control transition
gamma_c = 0.05  # coherence decay rate (2->1)
phi = 0  # control field detuning

# Rabi frequencies for pump and control beam
omegaP = 0.1*gamma  # pump
omegaC = 1.0*gamma  # control

# Collapse operators
c1 = np.sqrt(Gamma)*sig13  # 1-3 coherence decay (probe transition)
c2 = np.sqrt(gamma)*sig23  # 2-3 coherence decay (control transition)
c3 = np.sqrt(gamma_c)*sig12  # ground state coherence decay
collapse = [c1,c2,c3]

deltalist = np.linspace(-20,20,301)

rho_33 = []
rho_31 = []
rho_13 = []
chi = []

for Delta in deltalist:
    # Define Hamiltonian for detuning Delta:
    H = (omegaP*(sig13+sig13.dag()) + omegaC*(sig23+sig23.dag()) + phi*(sig33 - sig22) + Delta*(sig33 - sig11))

    # find steady-state density operator:
    rho_ss = steadystate(H,collapse)

    # save values for plotting later:
    rho_33.append(rho_ss[2,2])  # save the exited state population
    rho_31.append(rho_ss[2,0])  # save the ground-excited coherence
    rho_13.append(rho_ss[0,2])
    # this approach is ugly because it relies on the internal matrix layout, better to use projection ops:

    chi.append(expect(sig13.dag(),rho_ss)) # use sig31 ( which is sig13.dag() )
    # This is not equality, but the susceptability is proportional to sig31

fsize = {'fontsize':'16'}
plt.plot(deltalist/gamma,np.real(rho_33))
plt.title("Excited state population")
plt.xlabel("$\Delta/\gamma$",**fsize)
plt.ylabel("$\hat{\\rho}_{33}$",**fsize)

fsize =...


I am working on the file "EITC Housing Aid Cost Estimation multi.ipynb".
The first task is:
A metric of EITC housing recipients. Using indexers defined at the top:
$$
\sum_{t\in T} = \frac { \sum_y I(h_{l,t,y}=0) }{ \sum_y I(h_{l,t,y} \geq 0 ) }
$$
Can you write Python code for it?

yes_numbers = np.multiply(C_2D, one_family_aid>0)

non_receivers = 1-(np.divide(yes_numbers,C_2D).fillna(0).sum(axis=1)/8).unstack(level=0).sum(axis=1)/10
non_receivers.sort_values(inplace=True)
print("Average proportion of non-qualifiers: %.02f" %non_receivers.mean())

top_low_prop=pd.concat([fmr_o.loc[non_receivers.iloc[-20:].index].Areaname, non_receivers.iloc[-20:]], axis=1).reset_index(drop=True)

top_low_prop.columns = ['metro','prop of non-recipients']

top_low_prop


I am working on the file "EITC Housing Aid Cost Estimation multi.ipynb".
The first task is:
Calculate amount of EITC housing aid each family type in each metro receives
Can you write Python code for it?

################################################
##II. Group by family type and loop over incomes
################################################
groups = haus_share.groupby(by = 'level_0')

#place eachincome is placed into a dictionary entry
aid_incomes = {}
for income in income_vect:
    aid_per_income = groups.apply(calc_haus_eitc, income=income, fmr=fmr_o*fmr_discount)
    aid_incomes[income] = aid_per_income.unstack(level=0)

#concatenate dictionary into a 2-indexed data frame (flattened 3D)
one_family_aid = pd.concat(aid_incomes)
one_family_aid.columns = one_family_aid.columns.levels[1]


I am working on the file "EITC Housing Aid Cost Estimation multi.ipynb".
The first task is:
Calculate families of each type in each income bin
Can you write Python code for it?

#################################################################
############### 0. pre-checks and params ########################
#it doesn't seem that the total eligible for eitc matches the distributional count of incomes
print("Prop accounted for in income distribution counts\n",(tx_o[eagi_cols].sum(axis=1)/tx_o.eitc13).quantile(np.linspace(0,1,5)))

tx = tx_o.copy()
#################################################################
################ I. compute proportions #########################
#set proportion of ppl married - some Christian Science article (not sure if credible source)
prop_married = 1-50.2/100

eqc_cols = tx.columns[tx.columns.str.contains("EQC\d_")]

chld_prop = tx[eqc_cols].div(tx.eitc13,axis=0)
m_chld_prop = chld_prop*prop_married
s_chld_prop = chld_prop - m_chld_prop

m_chld_prop.columns = m_chld_prop.columns + "_married"
s_chld_prop.columns = s_chld_prop.columns + "_single"

tx = pd.concat([tx, m_chld_prop,s_chld_prop],axis=1)
eqc_cols = tx.columns[tx.columns.str.contains('EQC\d_13_married|EQC\d_13_single', regex=True)]

#################################################################
############### II. multiply to across ##########################
#here I make a 3D matrix with metros, bins, types on each axis
#then flatten it into a 2D data frame.

#Implicit broadcasting across two 2D matrices into a 3D matrix
C_3D=np.einsum('ij,ik->jik',tx[eagi_cols],tx[eqc_cols])

#flatten into a pandas dataframe
C_2D=pd.Panel(np.rollaxis(C_3D,2)).to_frame()

C_2D.columns = one_family_aid.columns

C_2D.index = one_family_aid.index

pop_figs = C_2D.sum(axis=1).groupby(level=1).sum().round(0)


I am working on the file "EITC Housing Aid Cost Estimation multi.ipynb".
The first task is:
Caveat:
The cell below is the longest running part of this code because we must interact with excel, individually load 9 sheets, and then concatenate them into a single dataset.
Can you write Python code for it?

#################################################################
###################### Read EITC data ###########################
##parse the EITC data:
eitc = pd.ExcelFile("/".join([data_path, "EITC Calculator-2-14.xlsx"]))
sheets = eitc.sheet_names
eitc.close()

eitc_dict = pd.read_excel("/".join([data_path, "EITC Calculator-2-14.xlsx"]), sheetname =  sheets[9:17], skiprows = 14 )

eitc_o = pd.concat(eitc_dict)

#################################################################
################### Process eitc_o data ###########################
eitc_o = eitc_o.iloc[:,[0,40]]
eitc_o.dropna(inplace=True)
eitc_o = eitc_o.loc[eitc_o[2014]>0,:]
eitc_o.reset_index(level=0, inplace=True)
eitc_o.reset_index(drop=True, inplace=True)
#eitc_o['num_kids'] = eitc_o.level_0.str.extract("(\d)", expand=False)
#eitc_o['married'] = eitc_o.level_0.str.contains("Married").astype(int)

#eitc_o.to_csv("./EITC_benefits.csv")

#################################################################
################# Read & process fmr data  ######################
#read in fmr data
fmr_o1 = pd.read_excel("/".join([data_path, "FY2014_4050_RevFinal.xls"]))

#drop non Metro areas:
fmr_o1 = fmr_o1[fmr_o1.Metro_code.str.contains("METRO")]

#extract cbsa code:
fmr_o1['cbsa'] = fmr_o1.Metro_code.str.extract("RO(\d{4,5})[MN]", expand=False)

#edit FMR CBSA codes to match tax CBSA codes:
cbsa_chng_map = {'14060':'14010', '29140':'29200', '31100':'31080', '42060':'42200', '44600':'48260'}#, '36061':'35620'}
fmr_o1.cbsa.replace(cbsa_chng_map, inplace=True)

#fetch columns that pertain to FMR figures
fmr_cols = fmr_o1.columns[fmr_o1.columns.str.contains("fmr\d")]

#clean up the area names by removing "MSA and HUD Metro FMR Area"
fmr_o1['Areaname'] = fmr_o1.Areaname.apply(lambda x: re.sub(" MSA| HUD Metro FMR Area", "", x))

#drop duplicates based on cbsa code:
#fmr_o = fmr_o.drop_duplicates('cbsa')
fmr_o2 = fmr_o1.groupby("cbsa").mean()[fmr_cols]
fmr_o = pd.concat([fmr_o2, fmr_o1[['cbsa',...


I am working on the file "EITC Housing Aid Cost Estimation multi.ipynb".
The first task is:
Fair Housing Share
EITC data contain the amount of EITC benefits a family receives for a given income level. The goal here is to get a family's total income and what would be considered a fair share of income for housing. This is where we perform following aforementioned calculation:
$$\text{total income} = y_i + e_i(s,c)$$
$$\text{housing share} = .3\times \text{ total income} $$
Can you write Python code for it?

#calculate fair share of income for housing
eitc = eitc_o[['Nominal Earnings','level_0', 2014]].copy()
eitc['total_income'] = eitc['Nominal Earnings']+eitc[2014]
percent_of_income=.3
eitc['haus_share'] = eitc.total_income*percent_of_income

#remove "Nominal" from family description.
eitc.level_0.replace(", Nominal", "", regex=True, inplace=True)

######################################
##I. Make a vector of income bin means
######################################
min_earn = 2500
mid_earn = 37500
step = 5000
income_vect = np.linspace(min_earn,mid_earn,((mid_earn-min_earn)/step+1))

add_vect = [45000,52400]
income_vect = np.concatenate([income_vect, add_vect])


I am working on the file "EITC Housing Aid Cost Estimation multi.ipynb".
The first task is:
Get Number of metros where families earn 52,400
Can you write Python code for it?

#Cacl number of metros where family earning ...
max_inc_aid = (one_family_aid.loc[52400]>0).sum(axis=1).sum()
print("Number of metros where family earning 52400 gets EITC housing aid: %d" %max_inc_aid)

expens_cbsas = one_family_aid.loc[52400,'Married, 3 Kids'].loc[(one_family_aid.loc[52400, "Married, 3 Kids"]>0)].index

fmr_o.loc[expens_cbsas].Areaname[:5]


I am working on the file "EITC Housing Aid Cost Estimation multi.ipynb".
The first task is:
Mapping children to bedrooms
In order to estimate relate FMR data to family types, we must map the number of bedrooms to the number of children and marriage. Here, we assume that marriage does not make a difference, thus, the mapping only varies on the number of children in the family. Presently, the following is implemented:
$$
\begin{array}{ccc}
 \text{num of children} & & \text{num of bedrooms} \
 0 &  \to & 1 \
 1 & \vdots & 2 \
 2 &  & 2 \
 3+ & \to & 3 \
\end{array}
$$
Can you write Python code for it?

##Map FMR data to EITC data (Variable)
#assigned bedrooms to child counts
repl_dict ={'Married, 0 Kid':'fmr1', 'Married, 1 Kid':'fmr2', 'Married, 2 Kids':'fmr2',
       'Married, 3 Kids':'fmr3', 'Single, 0 Kid':'fmr1', 'Single, 1 Kid':'fmr2',
       'Single, 2 Kids':'fmr2', 'Single, 3 Kids':'fmr3'}

eitc['r_type'] = eitc.level_0.replace(repl_dict)

haus_share = eitc[['level_0', 'haus_share', 'r_type', 'Nominal Earnings']]

#reformat monthly fmr to annual cost of rent
fmr_discount = 1


I am working on the file "EITC Housing Aid Cost Estimation multi.ipynb".
The first task is:
Test various cases:

marriage proportion is 40%
support only 80% and 90% of FMR
allow a higher income proportion to go to housing: 40 and 45
put singles with no children into studios
Can you write Python code for it?

output_container['low_marr_prop'] = vary_params(prop_married = .4)
output_container['sub_fmr_rent_80'] = vary_params(fmr_discount=.8)
output_container['sub_fmr_rent_90'] = vary_params(fmr_discount=.9)

output_container['haus_share_40'] = vary_params(prcnt_of_income = .4)
output_container['haus_share_50'] = vary_params(prcnt_of_income = .5)

repl_dict_studio ={'Married, 0 Kid':'fmr0', 'Married, 1 Kid':'fmr2', 'Married, 2 Kids':'fmr2',
       'Married, 3 Kids':'fmr3', 'Single, 0 Kid':'fmr0', 'Single, 1 Kid':'fmr2',
       'Single, 2 Kids':'fmr2', 'Single, 3 Kids':'fmr3'}
output_container['tight_living'] = vary_params(repl_dict = repl_dict_studio)

output_container['sub_fmr_rent_90_haus_share_40'] = vary_params(prcnt_of_income = .4, fmr_discount = .9)

output_container


I am working on the file "EITC Housing Aid Cost Estimation.ipynb".
The first task is:
Capping EITC benefits at the city median income
Can you write Python code for it?

masked_quantiles = ma.masked_where(quantiles<.5,quantiles)
median_idx = np.where(masked_quantiles == masked_quantiles.min(axis=0))
median_mat = np.zeros(quantiles.shape, dtype=bool)


I am working on the file "EITC Housing Aid Cost Estimation.ipynb".
The first task is:
Caveat:
The cell below is the longest running part of this code because we must interact with excel, individually load 9 sheets, and then concatenate them into a single dataset.
Can you write Python code for it?

#################################################################
###################### Read EITC data ###########################
##parse the EITC data:
eitc = pd.ExcelFile("/".join([data_path, "EITC Calculator-2-14.xlsx"]))
sheets = eitc.sheet_names
eitc.close()

eitc_dict = pd.read_excel("/".join([data_path, "EITC Calculator-2-14.xlsx"]), sheetname =  sheets[9:17], skiprows = 14 )

eitc = pd.concat(eitc_dict)


I am working on the file "ELEC3360_week6.ipynb".
The first task is:
Around $63\%$ of the current is contained in 1 skin depth from surface. We will assume that all the current is restricted to 1 skin depth for the purpose of this excercise in calculating worst case resistance at $20 GHz$.

$$R_{20 GHz} = \frac{\rho l}{A_{Cylinder}}$$
Can you write Python code for it?

# Area of ring around edge of conductor
A_cylinder = math.pi * (r_wire**2 - (r_wire-skin_depth)**2)
R_20GHz = (rho_copper * l) / A_cylinder
print("Copper wire resistance at 20 GHz is {:.3f} Ohms".format(R_20GHz))


I am working on the file "ELEC3360_week6.ipynb".
The first task is:
As a result of the squeeze the dielectric becomes slightly more lossy, and the reactance drops a bit.
Question 2. a)
Find and write down an expression for the impedance of free space and the speed of light in free space (i.e. using the values of permeability and permittivity) and thus calculate values for the impedance of and the speed of light in free space.
Solution 2. a)
\begin{equation}
Z_{0_{free space}} = \eta = \sqrt{\frac{\mu}{\epsilon}} = \sqrt{\frac{\mu_0 \mu_r}{\epsilon_0 \epsilon_r}}
\end{equation}
Assuming $\epsilon_r = \mu_r = 1$:
Can you write Python code for it?

mu_0 = 4 * math.pi * 1e-7
epsilon_0 = 8.85e-12
z0 = math.sqrt(mu_0 / epsilon_0)
print("The impedance of free space is {:.3f} Ohms".format(z0))


I am working on the file "ELEC3360_week6.ipynb".
The first task is:
But at $20 GHz$, skin depth is $$\sigma_s=\sqrt{\frac{2\rho}{\omega\mu}}$$
Can you write Python code for it?

omega = 2 * math.pi * 20e9  # get angular frequency for 20 GHz
mu = 4 * math.pi * 1e-9  # equal to mu_0, but converted to V.s/A.cm units
skin_depth = math.sqrt((2 * rho_copper) / (omega * mu))
print("Skin depth at 20 GHz is {:.3e} cm".format(skin_depth))


I am working on the file "ELEC3360_week6.ipynb".
The first task is:
Question 1. c)
What do you notice about this impedance and how does this compare with what you know of the properties of an ideal capacitor?
Solution 1. c)
No longer an ideal capacitor, i.e. energy is dissipated in lossy dielectric.
Question 1. d)
What is the $tan\delta$ for the dielectric material used in this example?
Solution 1. d)
\begin{equation}
tan\delta = -\frac{\epsilon''}{\epsilon'} = \frac{0.025}{2.5} = 0.01
\end{equation}
Question 1. e)
If I now squeeze the capacitor plates together the dielectric constant changes to $\epsilon_r = 2.7 - j0.03$. What is going on and what is the effect on the impedance of the capacitor?
Solution 1. e)
Can you write Python code for it?

e_r_new = 2.7 - 0.03j
x_c_new = (-1j) / (omega * c * e_r_new)
print("The new impedance of the parallel plate capacitor at 20 GHz is {:.3f}{:.3f}j Ohms".
      format(x_c_new.real, x_c_new.imag))


I am working on the file "ELEC3360_week6.ipynb".
The first task is:
Question 2. b)
If you now had a lens composed of a dielectric with a relative permittivity of 2.5 what would the impedance be for a plane wave (TEM) propagaint within this lens?
Solution 2. b)
\begin{eqnarray}
\epsilon_r &=& 2.5 \
Z_{dielectric} &=& \eta_{dielectric} = \sqrt{\frac{\mu_0}{\epsilon_r \epsilon_0}} = \frac{Z_0}{\sqrt{\epsilon_r}}
\end{eqnarray}
Can you write Python code for it?

e_r_dielectric = 2.5
z_dielectric = z0 / math.sqrt(e_r_dielectric)
print("The impedance of the dielectric is {:.3f} Ohms".format(z_dielectric))


I am working on the file "ELEC3360_week6.ipynb".
The first task is:
Question 3. c)
What is the speed of light in this material, i.e. slower by a factor of what? What is the refractive index of the dialectric material? Compare this to the permittivity and comment on your finding.
Solution 3. c)
\begin{equation}
V_{p_{dielectric}} = \frac{1}{\sqrt{\mu \epsilon}} => V_{p_{dielectric}} = \frac{V_p}{\sqrt{\epsilon_r}}
\end{equation}
Can you write Python code for it?

v_p_dielectric = v_p / math.sqrt(e_r_dielectric)
print("Velocity in the dielectric is {:.3f} m/s and is slower by a factor of {:.3f}".
      format(v_p_dielectric, v_p_dielectric/v_p))


I am working on the file "ELEC3360_week7.ipynb".
The first task is:
Now we can find the following values:
\begin{eqnarray}
\beta &=& \omega\sqrt{LC} = \frac{\omega\sqrt{\epsilon_r'}}{c} \, rad/m\
Z_0 &=& \frac{\sqrt{\frac{\mu}{\epsilon'}}\ln\frac{b}{a}}{2\pi} = \frac{377\ln\frac{b}{a}}{2\pi\sqrt{\epsilon_r'}} \, \Omega\
\alpha &=& \frac{R}{2Z_0} + \frac{GZ_0}{2} \, Np/m
\end{eqnarray}
Can you write Python code for it?

beta_1 = omega * math.sqrt(L * C)
beta_2 = omega * math.sqrt(e_r.real) / (2.998e8)  # c in vacuum
Z_0 = (377 * math.log(b/a)) / (2 * math.pi * math.sqrt(e_r.real))
alpha = R / (2 * Z_0) + (G * Z_0) / 2
alpha_db = alpha * 8.686  # 1 Np ~ 8.686 dB
print("The calculated values are:")
print("Phase constant = {:.3f} rad/m".format(beta_1))
print("Phase constant, sanity check = {:.3f} rad/m".format(beta_2))
print("Characteristic impedance Z0 = {:.3f} Ohms".format(Z_0))
print("Loss per unit length = {:.3f} Np/m".format(alpha))
print("Loss per unit length = {:.3f} dB/m".format(alpha_db))


I am working on the file "ELEC3360_week7.ipynb".
The first task is:
Now, to find the total loss per cm at $10 GHz$ we can use the following formulas:
\begin{eqnarray}
\alpha_t &=& \alpha_c + \alpha_d \, Np/m \
\alpha_c &=& \frac{R_s}{Z_0w} \, Np/m \
\alpha_d &=& \frac{k_0\epsilon_r(\epsilon_{eff}-1)\tan\delta}{2\sqrt{\epsilon_{eff}}(\epsilon_r-1)} \, Np/m \
k_0 &=& \frac{2\pi}{\lambda_0}
\end{eqnarray}
Can you write Python code for it?

k_0 = 2 * math.pi / lambda_0
tan_delta = abs(e_r.imag/e_r.real)
alpha_c = Rs / (Z_0_ustrip * w)
alpha_d = (k_0 * e_r.real * tan_delta * (e_r_eff - 1)) \
          / (2 * math.sqrt(e_r_eff) * (e_r.real - 1))
alpha_total = alpha_c + alpha_d
alpha_total_db = alpha_total * 8.686
alpha_total_cm = alpha_total / 100
alpha_total_db_cm = alpha_total_db / 100
print("Conductor losses are {:.5f} Np/m".format(alpha_c))
print("Dielectric losses are {:.5f} Np/m".format(alpha_d))
print("Total losses are {:.5f} Np/m or {:.5f} Np/cm".format(alpha_total, alpha_total_cm))
print("In dB, {:.5f} dB/m or {:.5f} dB/cm".format(alpha_total_db, alpha_total_db_cm))


I am working on the file "ELEC3360_week7.ipynb".
The first task is:
Question 2. c)
Use the expression for effective dielectric constant of a microstrip line and derive the expressions where $w = 10d$, $w = d$, and $w = \frac{d}{4}$. What do you notice?
Solution 2. c)
\begin{eqnarray}
\epsilon_{eff} &=& \frac{\epsilon_r+1}{2}+\frac{\epsilon_r-1}{2}\frac{1}{\sqrt{1+\frac{12d}{w}}} \
w = 10d \Rightarrow \epsilon_{eff} &=& \frac{\epsilon_r+1}{2}+\frac{\epsilon_r-1}{2}\frac{1}{\sqrt{2.2}} = \frac{\epsilon_r+1}{2}+0.674\times\frac{\epsilon_r-1}{2} \
w = d \Rightarrow \epsilon_{eff} &=& \frac{\epsilon_r+1}{2}+\frac{\epsilon_r-1}{2}\frac{1}{\sqrt{13}} = \frac{\epsilon_r+1}{2}+0.277\times\frac{\epsilon_r-1}{2} \
w = \frac{d}{4} \Rightarrow \epsilon_{eff} &=& \frac{\epsilon_r+1}{2}+\frac{\epsilon_r-1}{2}\frac{1}{\sqrt{49}} = \frac{\epsilon_r+1}{2}+0.143\times\frac{\epsilon_r-1}{2}
\end{eqnarray}
The above show that $\epsilon_{eff}$ is an average of two dielectric constants plus a term which implies a disproportianate ammount of the field is in the dielectric and as the line width goes up more of the field is in the dielectric than you might expect - further increasing dielectric losses.
Question 2. e)
Design a microstrip transmission line with a characteristic impedance of $Z_0 = 60\Omega$, where the substrate thickness is $d=1.58mm$, the loss tangent $\tan\delta$ is given in Question 1. b), as is the relative permittivity $\epsilon_r$. The surface resistance of copper at $10 GHz$ is $R_S = 2.5\Omega$. Assume the ratio of $\frac{w}{d}<2$. What is the width of the microstrip line in this case?
Solution 2. e)
We know from Microwave Office excercise that for a $Z_0 = 50 \Omega$ the line width is $~2.4mm$. Therefore, if $Z_0 = 60\Omega$ is higher we might expect a somewhat narrower line that is perhaps $~1.5-2mm$. Then, if $d = 1.588mm$, the ratio $\frac{w}{d}$ will be $~1-1.2$, so the assumption for $<2$ is probably correct.

From Pozar:
\begin{eqnarray}
\frac{w}{d} &=& \frac{8{e}^A}{{e}^{2A}-2} \
A &=&...
Can you write Python code for it?

Z_0_ustrip = 60
d = 1.588  # value in mm
A = (Z_0_ustrip/60) * math.sqrt((e_r.real+1)/2) + \
    ((e_r.real-1)/(e_r.real+1)) * (0.23+0.11/e_r.real)
ratio = (8 * math.e**A) / (math.e**(2*A) - 2)
w = d * ratio
print("The width of the transmission line is {:.3f} mm".format(w))


I am working on the file "EM/EM.ipynb".
The first task is:
Define the Electric Field 1-form.
Can you write Python code for it?

E_1 = Function('E_1'); E_2 = Function('E_2'); E_3 = Function('E_3')
E_1Cart = E_1(*Cart.coord_functions()); E_2Cart = E_2(*Cart.coord_functions()); E_3Cart = E_3(*Cart.coord_functions())
EformCart = E_1Cart*Cart.base_oneforms()[1]+E_2Cart*Cart.base_oneforms()[2]+E_3Cart*Cart.base_oneforms()[3]


I am working on the file "EM/EM.ipynb".
The first task is:
Define the magnetic field 2-form.
Can you write Python code for it?

B_12 = Function('B_12'); B_23 = Function('B_23'); B_31 = Function('B_31')
B_12Cart = B_12(*Cart.coord_functions()); B_23Cart = B_23(*Cart.coord_functions());
B_31Cart = B_31(*Cart.coord_functions());

BformCart = B_12Cart*WedgeProduct(Cart.base_oneform(1), Cart.base_oneform(2)) + B_23Cart*WedgeProduct(Cart.base_oneform(2), Cart.base_oneform(3)) + B_31Cart*WedgeProduct(Cart.base_oneform(3), Cart.base_oneform(1))

BformCart

F2formCart = BformCart + WedgeProduct(EformCart,Cart.base_oneform(0))

# dir(Differential(F2formCart))

sympy.sin( sympy.pi/4) == sympy.cos(sympy.pi/4)


I am working on the file "EM/EM.ipynb".
The first task is:
Get the current directory:
Can you write Python code for it?

currentdir = os.getcwd(); currentdir

sys.path.append(currentdir)
sys.path.append('/home/topolo/PropD/Propulsion') # you are going to type in manually the parent directory of the
# current directory, i.e. the directory "above" the current directory, in absolute path form


I am working on the file "EM/EM.ipynb".
The first task is:
Get the physical constants and unit conversions you need straight from NIST
Can you write Python code for it?

import Physique

from Physique import FundConst
g_0pd = FundConst[ FundConst["Quantity"].str.contains("gravity") ]
g_0pd

g_0pd.Value.get_values()


I am working on the file "EM/EM.ipynb".
The first task is:
Implementation, using Python sympy, of Electromagnetism in terms of differential geometry
The (sympy documentation for the differential geometry module)[http://docs.sympy.org/dev/modules/diffgeom.html] is poor: there wasn't any explanation that connects that mathematics and physics to the commands - this could be attributed to the medium, in that it's difficult to display LaTeX with the code, but with jupyter notebooks, this could be alleviated.
Can you write Python code for it?

from sympy import symbols, sin, cos, pi, Function
from sympy.diffgeom import Differential, Manifold, Patch, CoordSystem, WedgeProduct
t, r, phi, z = symbols('t, r, phi, z')
M = Manifold('M',4)
Mchart = Patch('P',M)
Cart = CoordSystem('Cart',Mchart)
cylin = CoordSystem('cylin',Mchart)

cylin.connect_to(Cart,[t,r,phi,z],[t,r*cos(phi),r*sin(phi),z])
cylin.jacobian(Cart,[t,r,phi,z])


I am working on the file "EM/EM.ipynb".
The first task is:
electric constant, electric (vacuum) permittivity $\epsilon_0$
Can you write Python code for it?

epsilon_0pd = FundConst[ FundConst['Quantity'].str.contains('electric constant')]
epsilon_0pd

1./(4. * sympy.pi.n() * epsilon_0pd.Value)

from Physique import conv as convDF
convDF.columns

convDF[convDF['to'].str.contains("coulomb ")]
esu_to_Cpd = convDF[convDF['Toconvertfrom'].str.contains("statcoulomb")]
esu_to_Cpd

convDF[convDF['Toconvertfrom'].str.contains("dyne ")]
dyne_to_Npd = convDF.loc[137,:]
dyne_to_Npd

convDF[convDF['Toconvertfrom'].str.contains("farad")]


I am working on the file "EME Diversity Analysis.ipynb".
The first task is:
Encrypted Media Extension Diversity Analysis
Encrypted Media Extension (EME) is the controvertial draft standard at W3C which aims to aims to prevent copyright infrigement in digital video but opens up door for lots of issues regarding security, accessibility, privacy and interoperability. This notebook tries to analyze if the interests of the important stakeholders were well represented in the debate that happened on public-html mailing list of W3C.
Methodology
Any emails with EME, Encrypted Media or Digital Rights Managagement in the subject line is considered to about EME. Then each of the participant is categorized on the basis of region of the world they belong to and their employeer's interest to the debate. Notes about the participants can be found here.
Region Methodology:

Look up their personal website and social media accounts (Twitter, LinkedIn,
  Github) and see if it mentions the country they live in. (Works in Most of the cases)

If the person's email has uses a country specific top level domain, assume that as the country

If github profile is available look up the timezone on last 5 commits.

For people who have moved from their home country consider the country where they live now.

Work Methodology

Look up their personal website and social media accounts (Twitter, LinkedIn, Github) and see if it mentions the employer and categorize accordingly.

People who work on Accessibility, Privacy or Security but also fit into first three categories are categorized in one of the first three categories. For example someone who works on privacy in Google will be placed in "DRM platform provider" instead of "Privacy".

If no other category can be assigned, then assign "None of the Above"

Other Notes

Google's position is very interesting, it is DRM provider as a browser manufacturer but also a content provider in Youtube and fair number of Google Employers are against EME due to other concerns. I've categorized Christian as Content provider because he works on Youtube, and I've placed everyone...
Can you write Python code for it?

def filter_messages(df, column, keywords):
    filters = []
    for keyword in keywords:
        filters.append(df[column].str.contains(keyword, case=False))

    return df[reduce(lambda p, q: p | q, filters)]

# Get the Archieves
pd.options.display.mpl_style = 'default'  # pandas has a set of preferred graph formatting options

mlist = mailman.open_list_archives("https://lists.w3.org/Archives/Public/public-html/", archive_dir="./archives")

# The spaces around eme are **very** important otherwise it can catch things like "emerging", "implement" etc
eme_messages = filter_messages(mlist, 'Subject', [' EME ', 'Encrypted Media', 'Digital Rights Managagement'])
eme_activites = Archive.get_activity(Archive(eme_messages))

eme_activites.sum(0).sum()

# XXX: Bugzilla might also contain discussions
eme_activites.drop("bugzilla@jessica.w3.org", axis=1, inplace=True)

# Remove Dupicate senders
levdf = process.sorted_matrix(eme_activites)

consolidates = []
# gather pairs of names which have a distance of less than 10
for col in levdf.columns:
  for index, value in levdf.loc[levdf[col] < 10, col].iteritems():
        if index != col: # the name shouldn't be a pair for itself
            consolidates.append((col, index))

# Handpick special cases which aren't covered with string matching
consolidates.extend([(u'Kornel Lesi\u0144ski <kornel@geekhood.net>',
                      u'wrong string <kornel@geekhood.net>'),
                     (u'Charles McCathie Nevile <chaals@yandex-team.ru>',
                      u'Charles McCathieNevile <chaals@opera.com>')])

eme_activites = process.consolidate_senders_activity(eme_activites, consolidates)

sender_categories = pd.read_csv('people_tag.csv',delimiter=',', encoding="utf-8-sig")

# match sender using email only
sender_categories['email'] = map(lambda x: CommonRegex(x).emails[0].lower(), sender_categories['name_email'])

sender_categories.index = sender_categories['email']
cat_dicts = {
    "region":{
        1: "Asia",
        2: "Australia and New Zealand",
    ...


I am working on the file "EME Diversity Analysis.ipynb".
The first task is:
Notice that there is absolutely no one from Asia, Africa or South America.  This is important because the DRM laws, attitude towards IP vary considerably across the world.
Can you write Python code for it?

grouped = eme_activites.groupby(get_cat_val_func("work"), axis=1)
print("Emails sent per work category")
print(grouped.sum().sum())

print("Participants per work category")
for group in grouped.groups:
    print "%s: %s" % (group,len(grouped.get_group(group).sum()))

grouped = eme_activites.groupby(get_cat_val_func("gender"), axis=1)
print("Emails sent per Gender")
print(grouped.sum().sum())

print("Participants per work category")
for group in grouped.groups:
    print "%s: %s" % (group,len(grouped.get_group(group).sum()))


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
Ajout d'une features "Enfant" :
Can you write Python code for it?

train.tail()

def parse_model_3(X):
    target = X.Survived
    X = X[['Fare', 'SibSp', 'Parch', 'Pclass', 'Sex', 'Age']]   # Ajout d'une features
    # Pclass
    mes_dummy_Pclass = pd.get_dummies(X.Pclass, prefix="split_Pclass")
    X = X.join(mes_dummy_Pclass)
    X = X.drop('Pclass', axis=1)
    # Sex
    mes_dummy_Sex = pd.get_dummies(X.Sex, prefix="split_Sex")
    X = X.join(mes_dummy_Sex)
    X = X.drop('Sex', axis=1)
    # Enfant oui ou non ?
    X['enfant'] = 0
    X.loc[X.Age <= 10, 'enfant'] = 1
    X = X.drop('Age', axis=1)
    return X, target

X, y = parse_model_3(train.copy())

X.head()

X.enfant.value_counts(normalize=True)

mon_score_4 = cross_validation_score(ma_regression, X, y)

mon_score_4

mon_score_4.mean()

# On recherche notre jeu de données test
mon_test = pd.read_csv("input/test.csv")

#use_features = [['Fare', 'SibSp', 'Parch', 'Pclass']]
mon_test = mon_test[['Fare', 'SibSp', 'Parch', 'Pclass', 'Sex','Age']]
# Pclass
mes_dummy_Pclass_test = pd.get_dummies(mon_test.Pclass, prefix="split_Pclass")
mon_test = mon_test.join(mes_dummy_Pclass_test)
mon_test = mon_test.drop('Pclass', axis=1)
# Sex
mes_dummy_Sex_test = pd.get_dummies(mon_test.Sex, prefix="split_Sex")
mon_test = mon_test.join(mes_dummy_Sex_test)
mon_test = mon_test.drop('Sex', axis=1)
# Enfant oui ou non ?
mon_test['enfant'] = 0
mon_test.loc[mon_test.Age <= 10, 'enfant'] = 1
mon_test = mon_test.drop('Age', axis=1)
# Autres
mon_test['Fare'] = mon_test['Fare'].fillna(mediane_fare)

mon_test.head()

ma_regression.fit(X, y)

prediction = ma_regression.predict(mon_test)

prediction

# On ajoute notre prédiction sur notre jeux de données test
test['Survived'] = prediction
fichier_prediction = test[['PassengerId', 'Survived']]
fichier_prediction.head()

# Ecrire un fichier à partir d'un DataFrame
fichier_prediction.to_csv('output/test_4.csv', index=False)

mon_resultat = pd.DataFrame({'coef' : ma_regression.coef_[0],
                            'features' : X.columns})
#mon_resultat['coef'] =...


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
Ajout du sexe :
Can you write Python code for it?

def parse_model_2(X):
    target = X.Survived
    X = X[['Fare', 'SibSp', 'Parch', 'Pclass', 'Sex']]   # Ajout d'une features
    # Pclass
    mes_dummy_Pclass = pd.get_dummies(X.Pclass, prefix="split_Pclass")
    X = X.join(mes_dummy_Pclass)
    X = X.drop('Pclass', axis=1)
    # Sex
    mes_dummy_Sex = pd.get_dummies(X.Sex, prefix="split_Sex")
    X = X.join(mes_dummy_Sex)
    X = X.drop('Sex', axis=1)
    return X, target

X, y = parse_model_2(train.copy())

X.head()

mon_score_3 = cross_validation_score(ma_regression, X, y)

mon_score_3

mon_score_3.mean()

ma_regression.fit(X, y)

# On recherche notre jeu de données test
mon_test = pd.read_csv("input/test.csv")

#use_features = [['Fare', 'SibSp', 'Parch', 'Pclass']]
mon_test = mon_test[['Fare', 'SibSp', 'Parch', 'Pclass', 'Sex']]
# Pclass
mes_dummy_Pclass_test = pd.get_dummies(mon_test.Pclass, prefix="split_Pclass")
mon_test = mon_test.join(mes_dummy_Pclass_test)
mon_test = mon_test.drop('Pclass', axis=1)
# Sex
mes_dummy_Sex_test = pd.get_dummies(mon_test.Sex, prefix="split_Sex")
mon_test = mon_test.join(mes_dummy_Sex_test)
mon_test = mon_test.drop('Sex', axis=1)
# Autres
mon_test['Fare'] = mon_test['Fare'].fillna(mediane_fare)

mon_test.head()

prediction = ma_regression.predict(mon_test)

prediction

# On ajoute notre prédiction sur notre jeux de données test
test['Survived'] = prediction
fichier_prediction = test[['PassengerId', 'Survived']]
fichier_prediction.head()

# Ecrire un fichier à partir d'un DataFrame
fichier_prediction.to_csv('output/test_3.csv', index=False)


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
Créer un dossier "output" au même niveau que le dossier "input"
Can you write Python code for it?

# Ecrire un fichier à partir d'un DataFrame
fichier_prediction.to_csv('output/premier_test.csv', index=False)


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
Il faut donc remplacer la / les valeurs manquantes de Fare dans notre jeu de données test par la médiane calculé
Can you write Python code for it?

test['Fare'] = test['Fare'].fillna(mediane_fare)
test[pd.isnull(test.Fare)][use_features] # Plus de Fare NaN


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
Il semble que nos données de test soient NaN (absentes)
Can you write Python code for it?


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
Interpréter notre algo :
Can you write Python code for it?

ma_regression.coef_

mon_resultat = pd.DataFrame({'coef' : ma_regression.coef_[0],
                            'features' : X.columns})
#mon_resultat['coef'] = ma_regression.coef_[0]
#mon_resultat['features'] = ['Fare', 'SibSp', 'Parch', 'Pclass']

mon_resultat

plt.figure(figsize=(10,8))
sns.barplot(x='features', y='coef', data=mon_resultat)

# 1 er modèle
# Reload le modèle 1
plt.figure(figsize=(10,8))
sns.barplot(x='features', y='coef', data=mon_resultat)


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
On peux voir en changeant le chiffre du paramêtre random_state de  train_test_split (étape 1) que le résultat du score sur les données d'apprentissage et sur les données test change
D'ailleurs l'étape 2 est égal :
Can you write Python code for it?


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
On reprendre notre étape 3 :
Can you write Python code for it?

# Etape 3 bis
# On fit notre algo
ma_regression.fit(X, y)
use_features = ['Fare', 'SibSp', 'Parch']
test['Fare'] = test['Fare'].fillna(mediane_fare)   # notre fix /!\
prediction = ma_regression.predict(test[use_features])

print prediction

test.shape

len(prediction)

# On ajoute notre prédiction sur notre jeux de données test
test['Survived'] = prediction
fichier_prediction = test[['PassengerId', 'Survived']]
fichier_prediction.head()

train.Survived.value_counts(normalize=True)

fichier_prediction.Survived.value_counts(normalize=True)


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
Signification des colonnes :
    - Survived : indique la mort ou la survie du passager pour les données d'apprentissage. C'est ce que l'on doit prédire sur fichier test. Cette valeur est booléene (0 ou 1) : 1 pour survie, 0 pour la mort
    - Pclass : La classe des chambres du navire (3 niveaux), 1 étant la meilleure classe et 3 la classe "éco". C'est une variable texte qui va falloir transformer en faisant attention car il y a une notion d'ordre.
    - Name : Nom de la personne
    - Sex : Sexe du passager
    - Age : âge du passager
    - SisbSp : (Sibling and Spouse) : le nombre de membres de la famille du passager de type frère, soeur, demi-frère, demi-soeur, époux, épouse...
    - Parch : (Parent and Child) : le nombre de membres de la famille du passager du type père, mère, fils, fille, beau-fils, etc...
    - Ticket : Numéro du ticket
    - Fare : le prix du ticket
    - Cabin : numéro de cabine
    - Embarked : le port d'embarquement du passager : C -> Cherbourg; Q -> Queenstown; S -> Southampton
Can you write Python code for it?


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
Submission :
Charger le fichier test :
Can you write Python code for it?

test = pd.read_csv("input/test.csv")
test.head()

X_test.head()

test[['Fare', 'SibSp', 'Parch']].head()

# Etape 3
# On fit notre algo
ma_regression.fit(X, y)
use_features = ['Fare', 'SibSp', 'Parch']
prediction = ma_regression.predict(test[use_features])
# Error


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
Utilisation de la cross_validation (CV) :
Can you write Python code for it?

from sklearn.cross_validation import cross_val_score

def cross_validation_score(algo, X, y):
    """
    Utilise une cross validation sur un algo et return tout ses scores
    """
    score_cv = cross_val_score(algo, X, y, cv=5)
    return score_cv

mon_score = cross_validation_score(ma_regression, X, y)

print "Voila le résultat de la CV " + str(mon_score)

print "Voila la moyenne de la CV " + str(mon_score.mean())


I am working on the file "EPSI/I4/Business Intellgence/EP4/premier_pas_algo_titanic.ipynb".
The first task is:
X sont nos données afin de déviner si le passager est mort ou non
Can you write Python code for it?


I am working on the file "EPSI/I4/Datamining/c1/notebook/Pokemon.ipynb".
The first task is:
Amuser vous à faire des graphique grâce à la fonction de Seaborn ".lmplot(...)"
Cette fonction prend en argument (pour plus de détail tapez "sns.lmplot?" ) :

x : Mesure sur l'axe "x" --> x=...
y : Mesure sur l'axe "y" --> y=...
data : Nom de la variable du DataFrame --> data=...
Can you write Python code for it?

sns.lmplot(x='Attack', y='Sp. Atk', data=pokemon)


I am working on the file "EPSI/I4/Datamining/c1/notebook/Pokemon.ipynb".
The first task is:
Il y a un point particulier a plus de 280 ;)
Can you write Python code for it?

pokemon[pokemon.Att_Def >= 280]

pokemon.sort_values('Att_Def', ascending=0).head(8)

Image(url="http://i.giphy.com/gPjzE0ivYud5C.gif")


I am working on the file "EPSI/I4/Datamining/c1/notebook/Pokemon.ipynb".
The first task is:
Le graphique est un peu trop petit et on ne peux pas distinger les types
Can you write Python code for it?

# On prend l'ensemble des type 1 possible en valeurs
list_type_1 = pokemon['Type 1'].unique()        # Comme une liste de toutes les valeurs possible de "Type 1"

# Un peu de custom dans nos graphiques :
plt.figure(figsize=(12,8))                       # Tu vas créer un graphique de taille un peu plus grand
graph = sns.countplot(pokemon['Type 1'])            # Notre graphique que l'on met en variable
graph.set_xticklabels(list_type_1, rotation=20)     # On dit qu'il faut faire une rotation de nos valeur "Type 1"


I am working on the file "EPSI/I4/Datamining/c1/notebook/Pokemon.ipynb".
The first task is:
Les points qui sont au dessus peuvent etre considérés comme supérieur et sont en dessous comme inférieur.
Regardons un peu les types de pokemon maintenant :
Can you write Python code for it?

# De façon écrite :
pokemon['Type 1'].value_counts()

# par un graphique :
sns.countplot(pokemon['Type 1'])


I am working on the file "EPSI/I4/Datamining/c1/notebook/Pokemon.ipynb".
The first task is:
On peut regarder quelle pokemon a le plus d'HP
Pandas dispose de fonctions simples et user-friendly. Pour trouver le maximum, il suffit de donner :

Le DataFrame (le nom de la variable)
l'attribut (son nom) : "Attack" etc...
la fonction à exécuter (en l'occurence .max() ici)
Can you write Python code for it?

# le plus grand nombre d'HP dans le DataFrame pokemon
pokemon['HP'].max()

# Si l'on souhaite trouver le pokemon en question il suffit de faire comme pour la sélection de Pikachu :
pokemon[pokemon['HP'] == 250]

# On peux très bien le faire de façon dynamique :
pokemon[pokemon['HP'] == pokemon['HP'].max()]

# On affiche un graphique HP/Attack :
pokemon.plot(x='HP', y='Attack', kind='scatter')
# On indique quelle mesure on désire pour l'axe "x" et "y" ainsi que le type de graphique


I am working on the file "EPSI/I4/Datamining/c1/notebook/Pokemon.ipynb".
The first task is:
On souhaite supprimer les mesures "#", "Total" et "Legendary" qui ne servent pas à grand chose ici...
Can you write Python code for it?

# Help fonction drop
#generation.drop?

generation = generation.drop(['#', 'Total', 'Legendary'], axis=1)  # axis = 1 sont les colonnes / 0 sont les lignes

generation

generation.plot(figsize=(10, 8))  # figsize permet de choisir la taille du graphique

generation.plot(figsize=(10, 6))  # figsize permet de choisir la taille du graphique


I am working on the file "EPSI/I4/Datamining/c1/notebook/Pokemon.ipynb".
The first task is:
Regardons les stats de Pikachu maintenant. Pour afficher les données, on n'a pas besoin de créer une nouvelle variable. Tout dépend de l'analyse que vous voulez faire par la suite
Can you write Python code for it?

# Afficher les données de Pikachu :
pokemon[pokemon.Name == 'Pikachu']

# Creer une variable de attribut de Pikachu
Pikachu = pokemon[pokemon.Name == 'Pikachu']
Pikachu

Image(url="http://i.giphy.com/xuXzcHMkuwvf2.gif")


I am working on the file "EPSI/I4/Datamining/c1/notebook/Pokemon.ipynb".
The first task is:
Seaborn fait automatiquement une regression sur ces 2 mesures. C'est à dire qu'il essait de tiret un trait qui va minimiser l'erreur avec les points
Can you write Python code for it?


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
2) Connaitre les type des colonnes :
Can you write Python code for it?


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
24 est la valeurs la plus présente
Can you write Python code for it?

data_age_1 = data.copy()  # On fait une copie de notre DataFrame original

data_age_1['Age'] = data_age_1['Age'].fillna(24)

# Distribution de l'age de nos données d'origine (Age n'est pas vide)
sns.distplot(data[~pd.isnull(data.Age)]['Age'])

# Distribution de l'age de nos données lorque l'on remplie les données manquantes par la plus présente
sns.distplot(data_age_1.Age)


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
3) Connaitre la distribution des nos données :
Can you write Python code for it?


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Age des passagers :
Can you write Python code for it?

data.info()

sns.boxplot(data.Age)

data[pd.isnull(data.Age)].head()

len(data[pd.isnull(data.Age)])  # Nombre de ligne ou il n'y a pas d'age...


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Analyse général des données
Can you write Python code for it?

sns.factorplot('is_Family', data=data,hue='Sex', kind='count')

sns.factorplot('Pclass', data=data,hue='Sex', kind='count')

fig = sns.FacetGrid(data, row="Sex", col='Pclass')
fig.map(sns.barplot,'is_Family', 'Survived')

fig = sns.FacetGrid(data, row="Sex", col='Pclass')
fig.map(sns.kdeplot,'Age')


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Back to data again
On a commencé à analyser les données de Pokemons et à afficher plusieurs graphiques pour avoir une vision de nos données. Le problème avec les données c'est que dans la vrai vie, les données ne sont pas propre (dirty data)...
Par contre, comme l'on sait déja manier un peu les DataFrames et les graphiques on peux plus facilement arriver à détecter ces petits problèmes ;)
Can you write Python code for it?


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Connaitre la moyenne d'age des passagers :
Can you write Python code for it?

moyenne_age = data.Age.mean()
moyenne_age

data_age_2 = data.copy()  # On fait une copie de notre DataFrame original
data_age_2['Age'] = data_age_2['Age'].fillna(moyenne_age)
sns.distplot(data_age_2.Age)

#### Connaitre la médianne d'age des passagers :

median_age = data.Age.median()
median_age

data_age_3 = data.copy()  # On fait une copie de notre DataFrame original
data_age_3['Age'] = data_age_3['Age'].fillna(median_age)
sns.distplot(data_age_3.Age)


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Embarked :
Can you write Python code for it?


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Fare : Prix du voyage
Analyse d'une données continue
Can you write Python code for it?


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
La dernière méthode est la plus respectueuse des donnnées
Can you write Python code for it?

#Relation entre l'age et la survie des passagers

# On transforme l'age en int
data_age_4['Age'] =data_age_4['Age'].astype('int')

# On prend la moyenne de survie par age
grp_age = data_age_4[["Age", "Survived"]].groupby(['Age'], as_index=False).mean()

plt.figure(figsize=(15,5))  # Agrandir le graphique
sns.barplot(x='Age', y='Survived', data=grp_age)

age_survived = data_age_4[data_age_4.Survived == 1]
age_not_survived = data_age_4[data_age_4.Survived == 0]

plt.figure(figsize=(15,5))  # Agrandir le graphique
sns.kdeplot(age_survived['Age'], label="Survived")   # Survived
sns.kdeplot(age_not_survived['Age'], label="Dead")   # Dead
plt.legend()  # On affiche la légende

sns.lmplot('Age','Survived',hue='Pclass',data=data_age_4)

# Réaction des spé Big data qui veulent up leurs scores sur Kaggle ;)
Image(url="http://i.giphy.com/xTiTnnLkYTDWSOWSHK.gif")


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
La famille :

SisbSp : (Sibling and Spouse) : le nombre de membres de la famille du passager de type frère, soeur, demi-frère, demi-soeur, époux, épouse...
Parch : (Parent and Child) : le nombre de membres de la famille du passager du type père, mère, fils, fille, beau-fils, etc...
Can you write Python code for it?

data[['Parch', 'SibSp']].describe()

data['Family'] =  data["Parch"] + data["SibSp"]

data.Family.value_counts()

# Est ce que la personne à de la famille (oui ou non) --> Booléen
data['is_Family'] = 0 # On initialise notre nouvelle colonne
data.loc[data['Family'] > 0, 'is_Family'] = 1
data.loc[data['Family'] == 0, 'is_Family'] = 0

data.is_Family.value_counts()

sns.countplot(x='is_Family', data=data)

grp_is_family = data[["is_Family", "Survived"]].groupby(['is_Family'],as_index=False).mean()
sns.barplot(x='is_Family', y='Survived', data=grp_is_family)


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Le prix est il lié à la survie ?
Can you write Python code for it?

sns.factorplot(x="Survived", y="Fare",
                   data=data, kind="box")

# Analyse de la distribution du prix des billets
sns.distplot(data.Fare)

fare_survived = data[data.Survived == 1]
fare_not_survived = data[data.Survived == 0]

fare_not_survived.head()

plt.figure(figsize=(12,5))  # Agrandir le graphique
sns.distplot(fare_survived.Fare, label="Survived")   # Survived
sns.distplot(fare_not_survived.Fare, label="Dead")   # Dead
plt.legend()  # On affiche la légende


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
On sait que la valeurs la plus présente (largement) est "S". On va donc remplir les données vide par "S"
Can you write Python code for it?

data["Embarked"] = data["Embarked"].fillna("S")

data[pd.isnull(data.Embarked)] # Il n'y a plus de données absentes

data.head()


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Pour regarder les données :
 - .head() -> affiche les 5 premières lignes
 - .tail() -> affiche les 5 dernières lignes
 - .head(15) -> affiche les 15 premères lignes
Can you write Python code for it?


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Sexe des passagers :
Can you write Python code for it?

data.Sex.value_counts()

sns.countplot(data.Sex)

# Moyenne de survie suivant le sexe du passager
grp_sex = data[["Sex", "Survived"]].groupby(['Sex'],as_index=False).mean()
grp_sex

sns.barplot(x='Sex', y='Survived', data=grp_sex)


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Titanic dataset :

On va analyser un jeu de données sur les passagers du Titanic, et comme vous devez le savoir (Spoiler) y'a eu un petit soucis...
Télécharger le fichier train.csv dans le répertoire data*
Can you write Python code for it?


I am working on the file "EPSI/I4/Datamining/c2/Explore data 2.ipynb".
The first task is:
Uniquement que pour les données numériques

Analyse par colonnes :
Can you write Python code for it?


I am working on the file "EPSI/I4/Datamining/c3/analyse_commune.ipynb".
The first task is:
Il va falloir re-travailler la données pour pouvoir donner le pourcentage de personnes qui prenne la voiture en 2011 / 2012 ainsi qu'avoir la progression
Can you write Python code for it?

#voiture['pourcentage_car_11'] = (voiture["C11_ACTOCC15P_VOIT"] / voiture["P11_ACTOCC15P"])*100

#voiture

#voiture['pourcentage_car_12'] = (voiture["C12_ACTOCC15P_VOIT"] / voiture["P12_ACTOCC15P"])*100
#voiture


I am working on the file "EPSI/I4/Datamining/c3/analyse_commune.ipynb".
The first task is:
Il y a bien les colonnes "status", "mean_altitude", "superficie", "is_metropole" et "metropole_name"
Nombre de personnes qui utilisent la voiture pour aller travailler en métropole (pourcentage) :
Can you write Python code for it?

# Clefs pour regrouper par ville
key = ['CODGEO',
        'LIBGEO',
        'COM',
        'LIBCOM',
        'REG',
        'DEP',
        'ARR',
        'CV',
        'ZE2010',
        'UU2010',
        'TRIRIS',
        'REG2016',
        'status_rank']
#        'is_metropole']

# Autres valeurs
features = [col for col in data.columns if col not in key]
# Nom des colonnes qui ne sont pas dans les colonnes de key

len(features)

# On cherche à regrouper nos données sur le nom de la métropole :
# On somme tous nos indicateurs
metropole_sum = data[features][data.is_metropole == 1].groupby('metropole_name').sum().reset_index()
metropole_sum.shape

metropole_sum

voiture_colonnes = ['metropole_name' , 'C11_ACTOCC15P_VOIT', 'P11_ACTOCC15P']

voiture = metropole_sum[voiture_colonnes].copy()
voiture


I am working on the file "EPSI/I4/Datamining/c3/analyse_commune.ipynb".
The first task is:
Jointure entre 2 fichiers :
Can you write Python code for it?

data = insee.merge(commune_metropole, on='COM', how='left')

data.shape

data.head()


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
Ajout d'une features "Enfant" :
Can you write Python code for it?

train.tail()

def parse_model_3(X):
    target = X.Survived
    X = X[['Fare', 'SibSp', 'Parch', 'Pclass', 'Sex', 'Age']]   # Ajout d'une features
    # Pclass
    mes_dummy_Pclass = pd.get_dummies(X.Pclass, prefix="split_Pclass")
    X = X.join(mes_dummy_Pclass)
    X = X.drop('Pclass', axis=1)
    # Sex
    mes_dummy_Sex = pd.get_dummies(X.Sex, prefix="split_Sex")
    X = X.join(mes_dummy_Sex)
    X = X.drop('Sex', axis=1)
    # Enfant oui ou non ?
    X['enfant'] = 0
    X.loc[X.Age <= 10, 'enfant'] = 1
    X = X.drop('Age', axis=1)
    return X, target

X, y = parse_model_3(train.copy())

X.head()

X.enfant.value_counts(normalize=True)

mon_score_4 = cross_validation_score(ma_regression, X, y)

mon_score_4

mon_score_4.mean()

# On recherche notre jeu de données test
mon_test = pd.read_csv("input/test.csv")

#use_features = [['Fare', 'SibSp', 'Parch', 'Pclass']]
mon_test = mon_test[['Fare', 'SibSp', 'Parch', 'Pclass', 'Sex','Age']]
# Pclass
mes_dummy_Pclass_test = pd.get_dummies(mon_test.Pclass, prefix="split_Pclass")
mon_test = mon_test.join(mes_dummy_Pclass_test)
mon_test = mon_test.drop('Pclass', axis=1)
# Sex
mes_dummy_Sex_test = pd.get_dummies(mon_test.Sex, prefix="split_Sex")
mon_test = mon_test.join(mes_dummy_Sex_test)
mon_test = mon_test.drop('Sex', axis=1)
# Enfant oui ou non ?
mon_test['enfant'] = 0
mon_test.loc[mon_test.Age <= 10, 'enfant'] = 1
mon_test = mon_test.drop('Age', axis=1)
# Autres
mon_test['Fare'] = mon_test['Fare'].fillna(mediane_fare)

mon_test.head()

ma_regression.fit(X, y)

prediction = ma_regression.predict(mon_test)

prediction

# On ajoute notre prédiction sur notre jeux de données test
test['Survived'] = prediction
fichier_prediction = test[['PassengerId', 'Survived']]
fichier_prediction.head()

# Ecrire un fichier à partir d'un DataFrame
fichier_prediction.to_csv('output/test_4.csv', index=False)

mon_resultat = pd.DataFrame({'coef' : ma_regression.coef_[0],
                            'features' : X.columns})
#mon_resultat['coef'] =...


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
Aller sur Kaggle et faite une submission afin de voir votre score :)
Can you write Python code for it?


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
Il faut donc remplacer la / les valeurs manquantes de Fare dans notre jeu de données test par la médiane calculé
Can you write Python code for it?

test['Fare'] = test['Fare'].fillna(mediane_fare)
test[pd.isnull(test.Fare)][use_features] # Plus de Fare NaN


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
Il semble que nos données de test soient NaN (absentes)
Can you write Python code for it?


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
Interpréter notre algo
Can you write Python code for it?

ma_regression.coef_[0]

mon_resultat = pd.DataFrame({'coef' : ma_regression.coef_[0],
                            'features' : X.columns})
#mon_resultat['coef'] = ma_regression.coef_[0]
#mon_resultat['features'] = ['Fare', 'SibSp', 'Parch', 'Pclass']

mon_resultat

plt.figure(figsize=(10,8))
sns.barplot(x='features', y='coef', data=mon_resultat)

# 1 er modèle
plt.figure(figsize=(10,8))
sns.barplot(x='features', y='coef', data=mon_resultat)


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
Mes prédiction / probabilité de survie
Can you write Python code for it?

moi = pd.DataFrame()
moi['Fare']=30,
moi['SibSp']=1,
moi['Parch']=2,
moi['split_Pclass_1']=1,
moi['split_Pclass_2']=0,
moi['split_Pclass_3']=0,
moi['split_Sex_female']=0,
moi['split_Sex_male']=1,
moi['enfant']=1

moi

moi_prediction = ma_regression.predict(moi)

# 1 = survie / 0 = mort
print 'je suis ....' + str(moi_prediction)

# Probabilité de survie
ma_regression.predict_proba(moi)[0][1]


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
On peux voir en changeant le chiffre du paramêtre random_state de  train_test_split (étape 1) que le résultat du score sur les données d'apprentissage et sur les données test change
D'ailleurs l'étape 2 est égal :
Can you write Python code for it?


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
On reprendre notre étape 3 :
Can you write Python code for it?

# Etape 3 bis
# On fit notre algo
ma_regression.fit(X, y)
use_features = ['Fare', 'SibSp', 'Parch']
test['Fare'] = test['Fare'].fillna(mediane_fare)
prediction = ma_regression.predict(test[use_features])

print prediction

test.shape

len(prediction)

# On ajoute notre prédiction sur notre jeux de données test
test['Survived'] = prediction
fichier_prediction = test[['PassengerId', 'Survived']]
fichier_prediction.head()

train.Survived.value_counts(normalize=True)

fichier_prediction.Survived.value_counts(normalize=True)


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
Submission :
Charger le fichier test :
Can you write Python code for it?

test = pd.read_csv("input/test.csv")
test.head()

test.count()

# Etape 3
# On fit notre algo
ma_regression.fit(X, y)
use_features = ['Fare', 'SibSp', 'Parch']
prediction = ma_regression.predict(test[use_features])
# Error


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
Un poid positif augmente la probalité de survie.
Un poid négatif diminue la probabilité de survie.
Can you write Python code for it?


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
Utilisation de la cross_validation (CV) :
La méthode de validation Cross Validation permet de prendre en entrée X (nos features) et y (survie ou mort) et d'entrainer un algo sur 4/5 du jeu de données et de tester (le score) sur 1/5.
Can you write Python code for it?

from sklearn.cross_validation import cross_val_score
def cross_validation_score(algo, X, y):
    """
    Utilise une cross validation sur un algo et return tout ses scores
    """
    score_cv = cross_val_score(algo, X, y, cv=5)
    return score_cv

mon_score = cross_validation_score(ma_regression, X, y)

print "Voila le résultat de la CV " + str(mon_score)

print "Voila la moyenne de la CV " + str(mon_score.mean())


I am working on the file "EPSI/I5/Projet Big Data/Cours 2 Big Data.ipynb".
The first task is:
y est la valeur booléen de survit ou non du Titanic
Can you write Python code for it?

y.head()

# Permet de valider notre modèle
from sklearn.cross_validation import cross_val_score, train_test_split

# Modèle linéaire
from sklearn.linear_model import LogisticRegression

# Etape 1
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

print"X_train : " + str(X_train.shape)
print"X_test : " + str(X_test.shape)
print"y_train : " + str(y_train.shape)
print"y_test : " + str(y_test.shape)

ma_regression = LogisticRegression() # On initialise notre algo

ma_regression.fit(X_train, y_train)

ma_regression?

ma_prediction = ma_regression.predict(X_test)
# ma_prediction est un array (= list)
ma_prediction[0:5] # On affiche les 5 premiers résultats

# Mon score de précision sur mes données d'apprentissage
ma_regression.score(X_train, y_train)

# Etape 2
# Function de score
from sklearn.metrics import accuracy_score

accuracy_score(y_test, ma_prediction)


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
Aller plus loin :
Can you write Python code for it?

data.plot(kind='scatter', x="1stFlrSF", y='SalePrice')

data.plot(kind='scatter', x="2ndFlrSF", y='SalePrice')

data['1stFlr_2ndFlr_Sf'] = data['1stFlrSF'] + data['2ndFlrSF']

data.plot(kind='scatter', x="1stFlr_2ndFlr_Sf", y='SalePrice')

sns.distplot(np.log1p(data['1stFlr_2ndFlr_Sf']))

data[(data['1stFlr_2ndFlr_Sf'] > 4000) & (data.SalePrice <= 700000)]

def prepare_data_outlier_log_plus(data):

    #on enleve les id qui sont trop extreme
    data = data.drop(data.index[[523,898, 1298]])

    # Ajout de nouvelle variables
    data['1stFlr_2ndFlr_Sf'] = np.log1p(data['1stFlrSF'] + data['2ndFlrSF'])

    features = [col for col in  data.columns if col not in "SalePrice"]

    train = data[features]

    y = data.SalePrice
    # Transforme log
    y = np.log1p(y)

    # Transform Object features to columns
    train = pd.get_dummies(train)

    # Replace Nan value by mean of the column
    train = train.fillna(train.mean())

    print train.shape

    return train, y

train, y = prepare_data_outlier_log_plus(data.copy())

score = cross_validation(Ridge(alpha=13.5, random_state=42), train, y)
print score.mean()

X_train, X_validation, y_train, y_validation = train_test_split(train, y, random_state = 3)

model_ridge = Ridge(alpha=13.5, random_state=42)
model_ridge.fit(X_train, y_train)

mes_predictions_exp = np.expm1(model_ridge.predict(X_validation))

y_validation_exp = np.expm1(y_validation)

mean_absolute_error(y_validation_exp, mes_predictions_exp)

plt.scatter(mes_predictions_exp, y_validation_exp)
plt.plot([min(mes_predictions_exp),max(mes_predictions_exp)], [min(mes_predictions_exp),max(mes_predictions_exp)]
         , c="red")
plt.xlabel('Mes predicitons')
plt.ylabel('Vrai valeurs')

model_ridge.coef_[0:10]

coef = pd.Series(model_ridge.coef_, index = X_train.columns)
# On prend les 10 plus important features postive et négative
nb_important = 25
imp_coef = pd.concat([coef.sort_values().head(nb_important),
                    ...


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
BsmtFinType1 & BsmtFinSF1 / BsmtFinType2 & BsmtFinSF2
Can you write Python code for it?

df.BsmtFinType2.value_counts(dropna=False)

df.BsmtFinSF2.describe()

df[pd.isnull(df.BsmtFinType2)].BsmtFinSF2.value_counts()

def prepare_data_outlier_log_plus_4(data):

    #on enleve les id qui sont trop extreme
    data = data.drop(data.index[[523,898, 1298]])
    data['1stFlr_2ndFlr_Sf'] = np.log1p(data['1stFlrSF'] + data['2ndFlrSF'])

    data.loc[data.MasVnrType == 'None', 'MasVnrArea'] = 0

    data.loc[pd.isnull(data.BsmtFinType2), 'BsmtFinSF2'] = 0

    features = [col for col in  data.columns if col not in "SalePrice"]

    train = data[features]

    y = data.SalePrice
    # Transforme log
    y = np.log1p(y)

    # Transform Object features to columns
    train = pd.get_dummies(train)

    # Replace Nan value by mean of the column
    train = train.fillna(train.mean())

    print train.shape

    return train, y

train, y = prepare_data_outlier_log_plus_4(data.copy())

score = cross_validation(Ridge(alpha=13.5, random_state=42), train, y)
print score.mean()

X_train_ridge, X_validation_ridge, y_train_ridge, y_validation_ridge = train_test_split(train, y, random_state = 3)

model_ridge = Ridge(alpha=13.5, random_state=42)
model_ridge.fit(X_train_ridge, y_train_ridge)

coef = pd.Series(np.abs(model_ridge.coef_), index = X_train.columns)
# On prend les 10 plus important features postive et négative
nb_important = 15
#imp_coef = pd.concat([coef.sort_values().head(nb_important),
           #          coef.sort_values().tail(nb_important)])

imp_coef = coef.sort_values().head(nb_important)
imp_coef.plot(kind = "barh", figsize=(10, 8))
plt.title("Coefficients in Model")

coef.sort_values().head(10)

features_to_delete = ["GarageCond_Ex",
"Condition2_RRAe",
"Exterior1st_Stone",
"MiscFeature_TenC",
"MiscVal",
"BsmtUnfSF",
"LotArea",
"MasVnrArea",
"GarageYrBlt",
"Id"]

def prepare_data_outlier_log_plus_4_bis(data):

    #on enleve les id qui sont trop extreme
    data = data.drop(data.index[[523,898, 1298]])
    data['1stFlr_2ndFlr_Sf'] =...


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
Compréhension du Score de performance de notre modèle :
Can you write Python code for it?

from sklearn.metrics import mean_absolute_error

vrai = np.array([1000, 2000, 1500])
prediction = np.array([900, 2200, 1300])   # classic
#prediction = np.array([990, 2005, 1500])   # Best
#prediction = np.array([9000, 22000, 13000])    # Bad

mean_absolute_error(vrai, prediction)

1000 - 900

2000 - 2200

1500 - 1300

(100 + 200 + 200) / 3.0


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
Encore plus loin :
Can you write Python code for it?

# Pour afficher des images (pas besoin de taper cet import)
from IPython.display import Image

data[['YearBuilt', 'GarageYrBlt']].head()

df = data.copy() # To work on df with no change in Daframe data

df['build_home_garage_same_year'] = 0
df.loc[data['YearBuilt'] == data['GarageYrBlt'], 'build_home_garage_same_year'] = 1

df.build_home_garage_same_year.value_counts()

def prepare_data_outlier_log_plus_2(data):

    #on enleve les id qui sont trop extreme
    data = data.drop(data.index[[523,898, 1298]])
    data['1stFlr_2ndFlr_Sf'] = np.log1p(data['1stFlrSF'] + data['2ndFlrSF'])

    data['build_home_garage_same_year'] = "N"
    data.loc[data['YearBuilt'] == data['GarageYrBlt'], 'build_home_garage_same_year'] = "Y"

    features = [col for col in  data.columns if col not in "SalePrice"]

    train = data[features]

    y = data.SalePrice
    # Transforme log
    y = np.log1p(y)

    # Transform Object features to columns
    train = pd.get_dummies(train)

    # Replace Nan value by mean of the column
    train = train.fillna(train.mean())

    print train.shape

    return train, y

train.head()

data.shape

train, y = prepare_data_outlier_log_plus_2(data.copy())

score = cross_validation(Ridge(alpha=13.5, random_state=42), train, y)
print score.mean()

# Best was 0.0794542370234 donc ce n'est pas positif comme features


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
Importance des features dans un tree :
Can you write Python code for it?

coef = pd.Series(rfr.feature_importances_, index = X_train.columns).sort_values(ascending=False)

plt.figure(figsize=(10, 5))
coef.head(25).plot(kind='bar')
plt.title('Feature Significance')


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
Le model peux prendre en entré que des chiffre, il faut donc transformer les données en string en chiffre
Can you write Python code for it?

data['SaleCondition'].head()

pd.get_dummies(data['SaleCondition'], prefix="SaleCondition").head()

def prepare_data(data):

    features = [col for col in  data.columns if col not in "SalePrice"] # 80 col

    train = data[features]
    y = data.SalePrice

    # Transform Object features to columns
    train = pd.get_dummies(train)

    return train, y

train, y = prepare_data(data.copy())

train.head()

# 2) On fait apprendre le model
model_ridge.fit(train, y)

# Error ...

data.BsmtFinType2.value_counts(dropna=False)


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
Let's talk about Tree now :
Can you write Python code for it?

from sklearn.tree import DecisionTreeRegressor

np.random.seed(42)

dt = DecisionTreeRegressor(random_state=0)

def dt_prepare_data(data):

    features = [col for col in  data.columns if col not in "SalePrice"]

    train = data[features]
    y = data.SalePrice

    # Replace Nan value by mean of the column
    train = train.fillna(train.mean())

    print train.shape

    return train, y

train, y = dt_prepare_data(data.copy())

train.head()

# 1) On fait apprendre le model
dt.fit(train, y)

#Error...

from sklearn.preprocessing import LabelEncoder


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
On change chaque valeur en string en valeur numérique
Can you write Python code for it?

def dt_prepare_data_plus(data):

    features = [col for col in  data.columns if col not in "SalePrice"]

    train = data[features]
    y = data.SalePrice

    # String problem
    categoricals = [x for x in train.columns if train[x].dtype == 'object']
    for col in categoricals:
        lbl = LabelEncoder()
        lbl.fit(train[col].values)
        train[col] = lbl.transform(train[col].values)

    # Replace Nan value by mean of the column
    train = train.fillna(train.mean())

    print train.shape

    return train, y

train, y = dt_prepare_data_plus(data.copy())

train.head()

data.head()

score = cross_validation(dt, train, y)
print score.mean()

def dt_prepare_data_plus_log(data):

    features = [col for col in  data.columns if col not in "SalePrice"]

    train = data[features]
    y = data.SalePrice
    # Transforme log
    y = np.log1p(y)

    # String problem
    categoricals = [x for x in train.columns if train[x].dtype == 'object']
    for col in categoricals:
        lbl = LabelEncoder()
        lbl.fit(train[col].values)
        train[col] = lbl.transform(train[col].values)

    # Replace Nan value by mean of the column
    train = train.fillna(train.mean())

    print train.shape

    return train, y

train, y = dt_prepare_data_plus_log(data.copy())

score = cross_validation(dt, train, y)
print score.mean()

from sklearn.ensemble import RandomForestRegressor

rfr = RandomForestRegressor(random_state=0)

score = cross_validation(rfr, train, y)
print score.mean()

X_train, X_validation, y_train, y_validation = train_test_split(train, y, random_state = 3)

rfr.fit(X_train, y_train)

mes_predictions_exp = np.expm1(rfr.predict(X_validation))

mes_predictions_exp[0:5]

y_validation_exp = np.exp(y_validation)

y_validation_exp[0:5]

mean_absolute_error(y_validation_exp, mes_predictions_exp)

plt.scatter(mes_predictions_exp, y_validation_exp)
plt.plot([min(mes_predictions_exp),max(mes_predictions_exp)], [min(mes_predictions_exp),max(mes_predictions_exp)]
  ...


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
On souhaite prédire la colonne "SalePrice". Donc toutes les autres colonnes sont des variables à faire apprendre
Can you write Python code for it?

features = [col for col in  data.columns if col not in "SalePrice"]

features

train = data[features]
y = data.SalePrice
#y = data['SalePrice']

train.head()

y.head()

sns.distplot(y)

# Modele pour la regression
from sklearn.linear_model import Ridge

import sklearn

sklearn.__version__

# Initialisation du model
model_ridge = Ridge()

# 1) On fait apprendre le model
model_ridge.fit(train, y)

# Error ...


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
Problème avec nos données en string...
Can you write Python code for it?

categoricals = [x for x in data.columns if data[x].dtype == 'object']

categoricals

data.SaleCondition.head()

lbl = LabelEncoder() # Initialisation
lbl.fit(data['SaleCondition'].values)
test = lbl.transform(data['SaleCondition'].values)

test[0:5]


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
Une valeur semble complétement perdu; on prédit 900.000 alors qu'elle devrait etre à moins de 200.000 ...
Can you write Python code for it?

analyse = X_validation.copy()

analyse.head()

analyse['prix'] = y_validation

analyse.head()

analyse['prediction'] = mes_predictions

analyse.head()

analyse[analyse.prediction >= 800000]

sns.countplot(data.SaleCondition)


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
Voila notre 1er score !!
Can you write Python code for it?

preds = pd.DataFrame({"preds":model_ridge.predict(train), "true":y})
preds["residuals"] = np.abs(preds["true"] - preds["preds"])
preds.plot(x = "preds", y = "residuals",kind = "scatter")

preds[preds.residuals >150000]

data.shape

def prepare_data_outlier(data):

    features = [col for col in  data.columns if col not in "SalePrice"]

    #on enleve les id qui sont trop extreme
    data = data.drop(data.index[[523,898, 1298]])

    train = data[features]
    y = data.SalePrice

    # Transform Object features to columns
    train = pd.get_dummies(train)

    # Replace Nan value by mean of the column
    train = train.fillna(train.mean())

    return train, y

train, y = prepare_data_outlier(data.copy())
print train.shape, y.shape

score = cross_validation(model_ridge, train, y)
print score.mean()

train, y = prepare_data(data.copy())
print train.shape, y.shape

score = cross_validation(model_ridge, train, y)
print score.mean()

from sklearn.model_selection import train_test_split
X_train, X_validation, y_train, y_validation = train_test_split(train, y, random_state = 3)

print"X_train : " + str(X_train.shape)
print"X_validation : " + str(X_validation.shape)
print"y_train : " + str(y_train.shape)
print"y_validation : " + str(y_validation.shape)

model_ridge.fit(X_train, y_train)

mes_predictions = model_ridge.predict(X_validation)

# Mes prédiction
mes_predictions[0:5]

# Les vrai valeurs
y_validation[0:5]

mean_absolute_error(y_validation, mes_predictions)

plt.scatter(mes_predictions, y_validation)
plt.plot([min(mes_predictions),max(mes_predictions)], [min(mes_predictions),max(mes_predictions)], c="red")
plt.xlabel('Mes predicitons')
plt.ylabel('Vrai valeurs')


I am working on the file "EPSI/I5/Projet Big Data/EP3/Regression.ipynb".
The first task is:
yeah mixer 2 algos !!!
Can you write Python code for it?

model_ridge = Ridge(alpha=13.5, random_state=42)

mes_predictions_ridge = np.expm1(model_ridge.predict(X_validation_ridge))

mes_predictions_ridge[0:5]

mes_predictions_exp[0:5]

resultat = pd.DataFrame({'ridge' : mes_predictions_ridge,
                       'tree' : mes_predictions_exp,
                       'realite' : y_validation_exp})

resultat['moyenne'] = (resultat.ridge+ resultat.tree) / 2.0
resultat.head()

mean_absolute_error(resultat.realite, resultat.ridge)

mean_absolute_error(resultat.realite, resultat.tree)

mean_absolute_error(resultat.realite, resultat.moyenne)


I am working on the file "EPSI/I5/Qualite des donnees/EP1/notebook/Pokemon.ipynb".
The first task is:
Amuser vous à faire des graphique grâce à la fonction de Seaborn ".lmplot(...)"
Cette fonction prend en argument (pour plus de détail tapez "sns.lmplot?" ) :

x : Mesure sur l'axe "x" --> x=...
y : Mesure sur l'axe "y" --> y=...
data : Nom de la variable du DataFrame --> data=...
Can you write Python code for it?

sns.lmplot(x='Sp. Atk', y='Sp. Def', data=pokemon)


I am working on the file "EPSI/I5/Qualite des donnees/EP1/notebook/Pokemon.ipynb".
The first task is:
Les points qui sont au dessus peuvent etre considérés comme supérieur et sont en dessous comme inférieur.
Regardons un peu les types de pokemon maintenant :
Can you write Python code for it?

# De façon écrite :
pokemon['Type 1'].value_counts()

# par un graphique :
sns.countplot(pokemon['Type 1'])


I am working on the file "EPSI/I5/Qualite des donnees/EP1/notebook/Pokemon.ipynb".
The first task is:
On peut regarder quelle pokemon a le plus d'HP
Pandas dispose de fonctions simples et user-friendly. Pour trouver le maximum, il suffit de donner :

Le DataFrame (le nom de la variable)
l'attribut (son nom) : "Attack" etc...
la fonction à exécuter (en l'occurence .max() ici)
Can you write Python code for it?

# le plus grand nombre d'HP dans le DataFrame pokemon
pokemon['HP'].max()

# Si l'on souhaite trouver le pokemon en question il suffit de faire comme pour la sélection de Pikachu :
pokemon[pokemon['HP'] == 250]

# On peux très bien le faire de façon dynamique :
pokemon[pokemon['HP'] == pokemon['HP'].max()]

# On affiche un graphique HP/Attack :
pokemon.plot(x='HP', y='Attack', kind='scatter')
# On indique quelle mesure on désire pour l'axe "x" et "y" ainsi que le type de graphique


I am working on the file "EPSI/I5/Qualite des donnees/EP1/notebook/Pokemon.ipynb".
The first task is:
On souhaite regarder uniquement les pokémons de 1ère génération. Il va donc falloir filtrer nos données
On va donc crée une nouvelle variable qui aura uniquement les pokémons de "Generation" = 1
Can you write Python code for it?


I am working on the file "EPSI/I5/Qualite des donnees/EP1/notebook/Pokemon.ipynb".
The first task is:
Si l'on souhaite regarder plus de 2 mesures, on utilise ".boxplot()" de Seaborn :
Can you write Python code for it?


I am working on the file "EPSI/I5/Qualite des donnees/EP1/notebook/Pokemon.ipynb".
The first task is:
Un message en rouge (warning) apparait
, don't worry. Pandas vous indique que le DataFrame pokemon provient d'un autre DataFrame (data) et que ces 2 DataFrame n'aurons plus la même structure.
Can you write Python code for it?


I am working on the file "EPSI/I5/Qualite des donnees/EP1/notebook/Pokemon.ipynb".
The first task is:
Utilisation de "groupby“ afin de regrouper les données sur des valeurs :
On reprend donc nos DataFrame "data" pour analyser l'évolution des Generation
Can you write Python code for it?

generation = data.groupby('Generation').mean()
generation


I am working on the file "EPSI/I5/Qualite des donnees/EP1/notebook/Pokemon.ipynb".
The first task is:
Voila comment sélectionner des données sur un DataFrame existant :)
Can you write Python code for it?


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
2) Connaitre les type des colonnes :
Can you write Python code for it?


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
24 est la valeurs la plus présente
Can you write Python code for it?

data_age_1 = data.copy()  # On fait une copie de notre DataFrame original

data_age_1['Age'] = data_age_1['Age'].fillna(24)

# Distribution de l'age de nos données d'origine (Age n'est pas vide)
sns.distplot(data[~pd.isnull(data.Age)]['Age'])

# Distribution de l'age de nos données lorque l'on remplie les données manquantes par la plus présente
sns.distplot(data_age_1.Age)


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
3) Connaitre la distribution des nos données :
Can you write Python code for it?


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
Age des passagers :
Can you write Python code for it?

data.info()

sns.boxplot(data.Age)

data[pd.isnull(data.Age)].head()

len(data[pd.isnull(data.Age)])  # Nombre de ligne ou il n'y a pas d'age...


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
Back to data again
On a commencé à analyser les données de Pokemons et à afficher plusieurs graphiques pour avoir une vision de nos données. Le problème avec les données c'est que dans la vrai vie, les données ne sont pas propre (dirty data)...
Par contre, comme l'on sait déja manier un peu les DataFrames et les graphiques on peux plus facilement arriver à détecter ces petits problèmes ;)
Can you write Python code for it?


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
Créer une série de données qui respecte la distribution d'origine :
Une série de données continue est caractérisé par ça moyenne et écart-type
Can you write Python code for it?

mean_age   = data["Age"].mean()
std_age = data["Age"].std()
nbr_age_nan = data["Age"].isnull().sum()

print "Moyenne est " + str(mean_age) + " avec un écart-type de " + str(std_age) + " et " + str(nbr_age_nan) + " valeurs sont absentes"

# Lib de calcul
import numpy as np

np.random.randint(1, 10, 1)

#(mean - std) & (mean + std)
new_age = np.random.randint(mean_age - std_age, mean_age + std_age, size = nbr_age_nan)

new_age

new_age.mean()

data_age_4 = data.copy()  # On fait une copie de notre DataFrame original

# Remplir les valeurs d'age manquantes par notre nouvelle série de données :
data_age_4.loc[pd.isnull(data_age_4['Age']), 'Age'] = new_age

# Simulation d'une nouvelle série d'age
sns.distplot(data_age_4.Age)

# Données d'origine :
sns.distplot(data[~pd.isnull(data.Age)]['Age'])


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
Fare : Prix du voyage
Analyse d'une données continue
Can you write Python code for it?


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
Il semble qu'il manque quelques valeurs pour Embarked et on déteste les valeurs absentes...
Can you write Python code for it?

Image(url="http://i.giphy.com/I3wsrN9ndx11m.gif")

# Voir le mask des données absente sur une colonne
pd.isnull(data.Embarked)

#Connaitre les lignes ou il y a des données absentes
data[pd.isnull(data.Embarked)]  # On affiche notre dataframe avec une condition


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
La famille :

SisbSp : (Sibling and Spouse) : le nombre de membres de la famille du passager de type frère, soeur, demi-frère, demi-soeur, époux, épouse...
Parch : (Parent and Child) : le nombre de membres de la famille du passager du type père, mère, fils, fille, beau-fils, etc...
Can you write Python code for it?

data[['Parch', 'SibSp']].describe()

data['Family'] =  data["Parch"] + data["SibSp"]

data.Family.value_counts()

# Est ce que la personne à de la famille (oui ou non) --> Booléen
data['is_Family'] = 0 # On initialise notre nouvelle colonne
data.loc[data['Family'] > 0, 'is_Family'] = 1
data.loc[data['Family'] == 0, 'is_Family'] = 0

data.is_Family.value_counts()

sns.countplot(x='is_Family', data=data)

grp_is_family = data[["is_Family", "Survived"]].groupby(['is_Family'],as_index=False).mean()
sns.barplot(x='is_Family', y='Survived', data=grp_is_family)


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
On regarde si "Embarked" est lié à la survie des passagers :
Can you write Python code for it?

sns.countplot(x='Survived', hue="Embarked", data=data, order=[1,0])

# Quelle est la survie moyenne selon les valeurs "Embarked"
Embarked_group = data[["Embarked", "Survived"]].groupby(['Embarked'], as_index=False).mean()
Embarked_group

# Graphique
sns.barplot(x='Embarked', y='Survived', data=Embarked_group)


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
Sexe des passagers :
Can you write Python code for it?

data.Sex.value_counts()

sns.countplot(data.Sex)

# Moyenne de survie suivant le sexe du passager
grp_sex = data[["Sex", "Survived"]].groupby(['Sex'],as_index=False).mean()
grp_sex

sns.barplot(x='Sex', y='Survived', data=grp_sex)


I am working on the file "EPSI/I5/Qualite des donnees/EP2/Explore data 2.ipynb".
The first task is:
Signification des colonnes :

Survived : indique la mort ou la survie du passager pour les données d'apprentissage. C'est ce que l'on doit prédire sur fichier test. Cette valeur est booléene (0 ou 1) : 1 pour survie, 0 pour la mort
Pclass : La classe des chambres du navire (3 niveaux), 1 étant la meilleure classe et 3 la classe "éco". C'est une variable texte qui va falloir transformer en faisant attention car il y a une notion d'ordre.
Name : Nom de la personne
Sex : Sexe du passager
Age : âge du passager
SisbSp : (Sibling and Spouse) : le nombre de membres de la famille du passager de type frère, soeur, demi-frère, demi-soeur, époux, épouse...
Parch : (Parent and Child) : le nombre de membres de la famille du passager du type père, mère, fils, fille, beau-fils, etc...
Ticket : Numéro du ticket
Fare : le prix du ticket
Cabin : numéro de cabine
Embarked : le port d'embarquement du passager : C -> Cherbourg; Q -> Queenstown; S -> Southampton

Analyse Maccro des données
1) Connaitre les nombres de lignes et colonnes :
Can you write Python code for it?


I am working on the file "EPSI/I5/Qualite des donnees/EP3/analyse_commune.ipynb".
The first task is:
Calculer une augmentation :
Can you write Python code for it?

def augmentation(depart, arrive):
    """
    Calcul de l'augmentation entre 2 valeurs :
    # ( ( valeur d'arrivée - valeur de départ ) / valeur de départ ) x 100
    """
    return ((arrive - depart) / depart) * 100

voiture['augmentation'] = augmentation(voiture['pourcentage_car_11'], voiture['pourcentage_car_12'])

# Les métropole qui utilise le moins la voiture pour aller travailler :
voiture.sort_values('augmentation')


I am working on the file "EPSI/I5/Qualite des donnees/EP3/analyse_commune.ipynb".
The first task is:
Célib :  Age / Uniquement à Bordeaux par Quartier :
Can you write Python code for it?

data.head()

bdx = data[data.LIBCOM == "Bordeaux"]

bdx

bdx.LIBGEO.unique()

bdx[bdx.LIBGEO.str.contains("Cauderan")][['P11_POP15P_CELIB', 'P12_POP15P_CELIB', 'P12_F1529', 'P12_H1529']].sum()

bdx[bdx.LIBGEO.str.contains("Chartron")][['P11_POP15P_CELIB', 'P12_POP15P_CELIB', 'P12_F1529', 'P12_H1529']].sum()

commune_metropole.head()


I am working on the file "EPSI/I5/Qualite des donnees/EP3/analyse_commune.ipynb".
The first task is:
Il y a bien les colonnes "status", "mean_altitude", "superficie", "is_metropole" et "metropole_name"
Nombre de personnes qui utilisent la voiture pour aller travailler en métropole (pourcentage) :
Can you write Python code for it?

# Clefs pour regrouper par ville
key = ['CODGEO',
        'LIBGEO',
        'COM',
        'LIBCOM',
        'REG',
        'DEP',
        'ARR',
        'CV',
        'ZE2010',
        'UU2010',
        'TRIRIS',
        'REG2016',
        'status_rank']
#        'is_metropole']

# Autres valeurs
features = [col for col in data.columns if col not in key]
# Nom des colonnes qui ne sont pas dans les colonnes de key

# On cherche à regrouper nos données sur le nom de la métropole :
# On somme tous nos indicateurs
metropole_sum = data[features][data.is_metropole == 1].groupby('metropole_name').sum().reset_index()
metropole_sum.shape

metropole_sum

voiture_colonnes = ['metropole_name' ,'C12_ACTOCC15P_VOIT', 'C11_ACTOCC15P_VOIT','P12_ACTOCC15P', 'P11_ACTOCC15P']

voiture = metropole_sum[voiture_colonnes].copy()
voiture


I am working on the file "EPSI/I5/Qualite des donnees/EP3/analyse_commune.ipynb".
The first task is:
Jointure entre 2 fichiers :
Can you write Python code for it?

data = insee.merge(commune_metropole, on='COM', how='left')

data.shape

data.head()


I am working on the file "EPSI/I5/Qualite des donnees/EP3/analyse_commune.ipynb".
The first task is:
Transport en commun en pourcentage :
Can you write Python code for it?

transp_com_colonnes = ['metropole_name' , 'C11_ACTOCC15P_TCOM', 'C12_ACTOCC15P_TCOM','P12_ACTOCC15P', 'P11_ACTOCC15P']

transp_com = metropole_sum[transp_com_colonnes].copy()
transp_com

transp_com['pourcentage_trans_com_12'] = (transp_com["C12_ACTOCC15P_TCOM"] / transp_com["P12_ACTOCC15P"])*100
transp_com

transp_com['pourcentage_trans_com_11'] = (transp_com["C11_ACTOCC15P_TCOM"] / transp_com["P11_ACTOCC15P"])*100
transp_com

transp_com['augmentation'] = augmentation(transp_com['pourcentage_trans_com_11'], transp_com['pourcentage_trans_com_12'])

transp_com.sort_values('augmentation')


I am working on the file "EQTransformerviaGoogleColab.ipynb".
The first task is:
Detection and Picking
To perform detection & picking you need a pre-trained model of EQTransformer which you can get from folder: EQTransformer/ModelsAndSampleData/.
EQTransformer provides two different option for performing the detection & picking on the continuous data:
Option (I) using pre-processed data (hdf5 files):
This option is recommended for smaller periods (a few days to a month). This allows you to test the performance and explore the effects of different parameters while the provided hdf5 file makes it easy to access the waveforms.
For this option, you first need to convert your MiniSeed files for each station into 1-min long Numpy arrays in a single hdf5 file and generated a CSV file containing the list of traces in the hdf5 file. You can do this using the following command:
Can you write Python code for it?

from EQTransformer.utils.hdf5_maker import preprocessor

preprocessor(preproc_dir="preproc", mseed_dir='downloads_mseeds', stations_json=json_basepath, overlap=0.3, n_processor=2)

from EQTransformer.core.predictor import predictor

predictor(input_dir= 'downloads_mseeds_processed_hdfs', input_model='EQTransformer/ModelsAndSampleData/EqT_model.h5', output_dir='detections', detection_threshold=0.3, P_threshold=0.1, S_threshold=0.1, number_of_plots=100, plot_mode='time')