I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Functions for performing simple evolution
For the sake of simplicity, we've only made point mutations -- i.e no sequence shuffling, deletions, insertions, etc.
Handy global variables:
Can you write Python code for it?

bases = ['A','C','G','T']
base_dict = dict(zip(bases, range(4)))


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Load model
Can you write Python code for it?

model = keras.models.model_from_json(open(model_params_dir + 'model_arch.json').read())
model.load_weights(model_params_dir + 'model_weights.hdf5')
model.compile(loss='mean_squared_error', optimizer='adam')


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Load the model
The directory that contains information about the model parameters
Can you write Python code for it?

model_name = 'Random_UTR_CNN'
model_params_dir = '../Results/{0}.Hyperparam.Opt/'.format(model_name)


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Make mutations in sequence:
Can you write Python code for it?

def make_mutation(sequences, pos, base):

    return [seq[:pos] + base + seq[pos + 1:] if pos < len(seq) else seq for seq in sequences]


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Organize the data
reshape and combine the dataframes with the sequences and the predicted growth rates
Can you write Python code for it?

# I apologize in advance, this is pretty unreadable

# reshape the dataframes. Transpose, then convert into a Multiindex, then convert the index into new columns
seqs_df = mut_native_seqs[0].T.unstack().reset_index()
scores_df = mut_native_seqs[1].T.unstack().reset_index()

# rename the columns so that the names are somewhat descriptive
seqs_df.rename(columns = {'level_0' : 'UTR_id', 'level_1' : 'Round', 0 : 'Evolved_Seq'}, inplace = True)
scores_df.rename(columns = {'level_0' : 'UTR_id', 'level_1' : 'Round', 0 : 'Pred_Growth'}, inplace = True)

# merge two datasets, only include columns that aren't redundant
cols_to_use = scores_df.columns.difference(seqs_df.columns) # identify non-redundant columns
seqs_scores = pd.merge(seqs_df, scores_df[cols_to_use], right_index = True, left_index = True)


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Save data
Can you write Python code for it?


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Select UTRs to evolve
We only want 100, so from those that we are most confident in (i.e. those with >100 reads at time point 0), we choose a range of expression levels.
Can you write Python code for it?

native_data = native_data[native_data.t0 > 100]


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Select UTRs with a range of expressions -- we don't only want to try to improve the worst or the best ones
We do this by first making an array of 100 desired growth rates evenly spaced between the lowest and highest actual growth rates. To find a UTR that is closest to one of the desired growth rates, we take the absolute value of the difference between the UTR's growth rate and the desired growth rate. Then we take the UTR with the value closest to zero (i.e. it's growth rate is the closest to the desired rate).
Can you write Python code for it?

# define the range of growth rates
desired_growth_rates = np.linspace(native_data.growth_rate.min(),
                                   native_data.growth_rate.max(),
                                   106)

selected_native_UTRs = []

for i in range(len(desired_growth_rates)):

    # with each step we select the UTR with the growth rate closest to the desired growth rate
    selected_native_UTRs.append(abs(native_data.growth_rate - desired_growth_rates[i]).argmin())

# just take the unique ones
selected_native_UTRs = pd.Series(np.unique(selected_native_UTRs))

selected_native_UTRs_data = native_data.loc[selected_native_UTRs]


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Since we have quite a lot of UTRs still, we also choose to focus on sequences that come from 5' UTR "chunks" that originate from proximal to the start codon. Recall that to test native 5' UTR sequences in our HIS3 growth assay, we broke the sequences into "chunks" of 50 nucleotides or less. Thus, some of the segments originate from sequence adjacent to the start codon, and others come from regions further upstream.
Can you write Python code for it?

native_data = native_data.loc[native_data['UTR_name'].apply(lambda s: s.split(':')[-1]) == '0']


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
__ ___ ______
Make Super UTRs from random library:
Here, we just go through all the same steps we just performed with the native library
Load data
Can you write Python code for it?

random_data = pd.read_csv('../Data/Random_UTRs.csv', index_col=0)


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
add Gene/UTR name information
Can you write Python code for it?

index_to_name = dict(zip(range(100), selected_native_UTRs_data.UTR_name.values))

seqs_scores['UTR_name'] = seqs_scores['UTR_id'].map(index_to_name)


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
add initial measured growth rate
Can you write Python code for it?

index_to_measured_growth = dict(zip(range(100), selected_random_UTRs_data.growth_rate.values))

ran_seqs_scores['Round0_Measured_Growth'] = ran_seqs_scores['UTR_id'].map(index_to_measured_growth)


I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
add input sequence
Can you write Python code for it?

index_to_input_seq = dict(zip(range(100), seqs))

seqs_scores['Starting_Seq'] = seqs_scores['UTR_id'].map(index_to_input_seq)


index_to_input_seq = dict(zip(range(100), seqs))

ran_seqs_scores['Starting_Seq'] = ran_seqs_scores['UTR_id'].map(index_to_input_seq)


I am working on the file "Jupyterlab/An introduction to Jupyterlab notebooks.ipynb".
The first task is:
Sharing your notebook
To allow others to interact with your notebook, create a github repository of the notebook, data, and dependencies. Then share using mybinder
Widgets for easy interaction!
Can you write Python code for it?

from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

def f(x):
    return 3*x^2 - 2*x + 8

interact(f, x=10);

def g(mu,sigma):
    s = np.random.normal(mu, sigma, 1000)
    count, bins, ignored = plt.hist(s, 30, normed=True)
    plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) * np.exp( - (bins - mu)**2 / (2 * sigma**2) ),
             linewidth=2, color='r')
    plt.xlim((-5,5))
    plt.ylim((0,3))
    plt.show()

interact(g,
         mu=widgets.FloatSlider(min=-4,max=4,step=0.5,value=0),
         sigma=widgets.FloatSlider(min=0.1,max=3,step=0.1,value=0.1));

def random_plot(sigma=0.85):
    x = np.random.randint(0,50,100)
    s = sigma * np.random.randn(100)
    y = x + (10 * s)
    plt.scatter(x,y)

random_plot(0.1)

interact(random_plot,
         sigma=widgets.FloatSlider(min=0.1,max=1,step=0.1,value=0.5));


I am working on the file "Jupyterlab/An introduction to Jupyterlab notebooks.ipynb".
The first task is:
Text can be entered into cells by designating it for markdown, allowing simple formatting.
It is also possible to enter Latex formulae into markdown cells:
$$ p_{obs} = \frac{(k+1)\cdot p_{linear}}{k \cdot p_{linear}+1} $$
Graphical output can also be incorporated:
Can you write Python code for it?

%matplotlib inline

import matplotlib.pyplot as plt

# import a practice dataset

from sklearn import datasets

diabetes = datasets.load_diabetes()
bmi = diabetes.data[2]
bp = diabetes.data[3]

plt.scatter(bmi,bp);

# add a line of best fit:

import numpy as np

z = np.polyfit(x=bmi, y=bp, deg=1)
p = np.poly1d(z)
trendline = p(sorted(bmi))

plt.scatter(bmi,bp)
plt.plot(sorted(bmi),trendline);


I am working on the file "Jupytor.ipynb".
The first task is:
Ahora un ejemplo con javascript:
Can you write Python code for it?

%%jupytor --lang javascript
var a = 1;
var b = 1;
console.log(a + b);


I am working on the file "Jupytor.ipynb".
The first task is:
Una vez hecho esto ya deberiamos tener disponible la cell magic para ser usada:
Can you write Python code for it?

%%jupytor --lang python3
a = 1
b = 2

def add(x, y):
    return x + y

c = add(a, b)


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Age
Can you write Python code for it?

data_basic_age = data_basic['Age']
print_stats(data_basic_age, hist_bins=20)
print('Population Standard Deviation:')
print(data_basic_age.std(ddof=0))


data_yxc_basic_age = data_yxc_basic['Age']
print_stats(data_yxc_basic_age, hist_bins=20)
print('Population Standard Deviation:')
print(data_yxc_basic_age.std(ddof=0))


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Basic Data
Can you write Python code for it?

data_basic = data['Basic']

# Translating Genre from Portuguese to English
data_basic['Genre'].replace('Masculino','Male',inplace=True)
data_basic['Genre'].replace('Feminino','Female',inplace=True)
data_basic['Genre'].replace('Prefiro não responder','Prefer not to say',inplace=True)

# Translating Education from Portuguese to English
data_basic['Education'].replace('Educação Básica','Basic Education',inplace=True)
data_basic['Education'].replace('Educação Secundária','Secondary Education',inplace=True)
data_basic['Education'].replace('Educação Pós-Secundária','Post-Secondary Education',inplace=True)
data_basic['Education'].replace('Educação Pós-Secundária','Post-Secondary Education',inplace=True)
data_basic['Education'].replace('Licenciatura','Bachelor\'s Degree',inplace=True)
data_basic['Education'].replace('Mestrado','Master\'s Degree',inplace=True)
data_basic['Education'].replace('Doutoramento','Doctoral Degree',inplace=True)
data_basic['Education'].replace('Prefiro não responder','Prefer not to say',inplace=True)

# Translating Type of Devices from Portuguese to English
data_basic['Type of Devices'].replace({'Computador de secretária':'Desktop'}, regex=True ,inplace=True)
data_basic['Type of Devices'].replace({'Computador portátil':'Laptop'}, regex=True, inplace=True)

# Translating Test mode from Portuguese to English
data_basic['Test Mode'].replace('PRIMEIRO em modo "Multi-Dispositivo" e DEPOIS em modo "Dispositivo Único".','Single Device > Multi-Device',inplace=True)
data_basic['Test Mode'].replace('PRIMEIRO em modo "Dispositivo Único" e DEPOIS em modo "Multi-Disposito".','Multi-Device > Single Device',inplace=True)

#display(data_basic)


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Confidence Interval
Can you write Python code for it?


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Confidence Intervals
Can you write Python code for it?

for column in data_general_use:
    print(column,
          'Median', data_general_use[column].median(),
          'Mean', data_general_use[column].mean(),
          'Standard Deviation',  data_general_use[column].std(),
          'Confidence Interval', confidence_interval_t(data_general_use[column]))


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Domain Specific Questions
Can you write Python code for it?

data_yxc_domain_specific = data_yxc['Domain Specific']
#data_yxc_domain_specific

print_stats(data_yxc_domain_specific)


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Education
Can you write Python code for it?

data_basic_education = data_basic['Education']
print_stats(data_basic_education)


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
General Stats
Can you write Python code for it?


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
General Use
Questions (Portuguese)

Normalmente, eu utilizo dois ou mais dispositivos em simultâneo no meu quotidiano.
O tempo que a aplicação demorou a reagir ao meu "input" foi adequado.
A distância é uma boa medida do quão relacionados são dois dispositivos.
O tempo que o sistema demorou a reagir às alterações nas posições dos dispositivos foi adequado.
É fácil mudar a distribuição dos componentes da interface de acordo com as minhas necessidades.
Eu considero que a distribuição automática dos componentes da interface foi adequada.
Can you write Python code for it?

data_general_use = data['General Use'].copy().dropna()
data_general_use['Score'] = data_general_use.iloc[:,0:5].sum(axis=1)
data_general_use['Score 100'] = data_general_use['Score'] * (100/35)
#data_general_use

print_stats(data_general_use, hist_size=(10,8))


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Genre
Can you write Python code for it?

data_yxc_basic_genre = data_yxc_basic['Genre']
print_stats(data_yxc_basic_genre)


data_basic_genre = data_basic['Genre']
print_stats(data_basic_genre)


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Helper Functions
Can you write Python code for it?

def print_stats(data, hist_bins=10, hist_size=(8,4)):
    print('--- Statistics ----')
    display(data.describe())
    print('\n')
    print('--- Counting Unique Values ----')
    display(data.value_counts())
    print('\n')
    print('--- Basic Histogram ----')
    data.hist(bins=hist_bins, figsize=hist_size)
    plt.show()

def calculate_sus(data):
    for i in range(len(data.columns)):
        if i % 2:
            data.iloc[:,i] = 7 - data.iloc[:,i]
        else:
            data.iloc[:,i] = data.iloc[:,i] - 1

    data['Score'] = data.iloc[:,0:10].sum(axis=1)
    data['Score 100'] = data['Score'] * (100/60)
    return data

def confidence_interval_t(data, confidence_level=0.95):
    return stats.t.interval(confidence_level, data.count()-1, data.mean(), data.sem())


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Mean Score 100
Can you write Python code for it?


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Per-question Stats
Can you write Python code for it?

##Uncomment if you need these stats
#for column in data_sus_single_device:
    #print_stats(data_sus_single_device[column])


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Percentage of Users that User Multiple Devices Sometimes or More Often
Can you write Python code for it?


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Response Frequency
Can you write Python code for it?

data_general_use_freq = pd.concat([data_basic['Usage of Multiple-Devices'], data['General Use']], axis=1)
data_general_use_freq = data_general_use_freq.rename(columns={"Usage of Multiple-Devices": "GU0"})
data_general_use_freq_res = (data_general_use_freq.apply(pd.value_counts).fillna(0)/data_general_use_freq.count()*100).transpose()
data_general_use_freq_res

data_general_use_freq.to_csv('out/juxtboard-data_general_use_freq.csv')
data_general_use_freq.describe()

data_general_use_freq_res.iloc[::-1].plot.barh(stacked=True).legend(loc='center left',bbox_to_anchor=(1.0, 0.5))
plt.show()


data_sd_md_freq = data['SD vs. MD'].copy().dropna()
data_sd_md_freq.iloc[:,0:5] = data_sd_md_freq.iloc[:,0:5] - 4
data_sd_md_freq_res = (data_sd_md_freq.apply(pd.value_counts).fillna(0)/data_sd_md_freq.count()*100).transpose()
data_sd_md_freq_res

data_sd_md_freq_res.to_csv('out/juxtboard-data_sd_md_freq_res.csv')
data_sd_md_freq.describe()

data_sd_md_freq_res.iloc[::-1].plot.barh(stacked=True).legend(loc='center left',bbox_to_anchor=(1.0, 0.5))
plt.show()

# Multi-Device Preference vs. Neutrality + Single Device Preference
data_sd_md_md_pref = (data_sd_md > 0).sum()
data_sd_md_neutral_or_sd_pre = (data_sd_md <= 0).sum()
data_sd_md_md_pref_relative = data_sd_md_md_pref / (data_sd_md_md_pref + data_sd_md_neutral_or_sd_pre)

# Single Device Preference vs. Neutrality + Multi-Device Preference
data_sd_md_sd_pref = (data_sd_md < 0).sum()
data_sd_md_neutral_or_md_pre = (data_sd_md <= 0).sum()
data_sd_md_sd_pref_relative = data_sd_md_sd_pref / (data_sd_md_sd_pref + data_sd_md_neutral_or_md_pre)

print('Multi-Device Preference vs. Neutrality + Single Device Preference')
for question in data_sd_md:
    print(question,
          'Relative Frequency:',
          data_sd_md_md_pref_relative[question],
          'Confidence Interval:',
          smp.proportion_confint(data_sd_md_md_pref[question], data_sd_md_md_pref[question]+data_sd_md_neutral_or_sd_pre[question], alpha=0.05, method='wilson'))

print('\n')

print('Single Device Preference vs. Neutrality + Multi-Device Preference')
for question in data_sd_md:
    print(question,
          'Relative Frequency:',
          data_sd_md_sd_pref_relative[question],
          'Confidence Interval:',
          smp.proportion_confint(data_sd_md_sd_pref[question], data_sd_md_sd_pref[question]+data_sd_md_neutral_or_md_pre[question], alpha=0.05, method='wilson'))


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
SUS
Questions

I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.
Can you write Python code for it?

data_yxc_sus = calculate_sus(data_yxc['SUS'].copy().dropna())
data_yxc_sus


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
SUS - Multi-Device Mode
Questions

I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.
Can you write Python code for it?

data_sus_multi_device = calculate_sus(data['SUS Multi-Device'].copy().dropna())
#data_sus_multi_device


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
SUS - Single Device Mode
Questions

I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.
Can you write Python code for it?

data_sus_single_device = calculate_sus(data['SUS Single Device'].copy().dropna())
#data_sus_single_device


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
SUS Single Device vs. SUS Multi-Device
Dependent t-test for paired samples
Can you write Python code for it?

stats.ttest_rel(data_sus_multi_device['Score 100'], data_sus_single_device['Score 100'])


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Single Device vs. SUS Multi-Device
Questions (Portuguese)

Em que modo de utilização foi mais fácil manipular as colecções de notas na aplicação (por exemplo, seleccioná-las, adicioná-las, partilhá-las, renomeá-las ou removê-las).
Em que modo de utilização foi mais fácil gerir as notas na aplicação (por exemplo, seleccioná-las, visualizá-las, adicioná-las, renomeá-las ou removê-las).
Em que modo de utilização foi mais fácil colaborar e mostrar notas aos outros participantes?
Indique o grau de preferência entre os dois modos de interacção do JuxtBoard caso esteja a utilizar a aplicação para realizar uma tarefa sozinho.
Indique o grau de preferência entre os dois modos de interacção do JuxtBoard caso se encontre numa situação em que esteja a utilizar a aplicação e haja necessidade de colaborar com outros utilizadores.
Can you write Python code for it?

data_sd_md = data['SD vs. MD'].copy().dropna()
data_sd_md.iloc[:,0:5] = data_sd_md.iloc[:,0:5] - 4
#data_sd_md

print_stats(data_sd_md, hist_size=(10,8))


I am working on the file "JuxtBoardUserStudy.ipynb".
The first task is:
Type of Devices
Can you write Python code for it?

data_basic_type_of_devices = data_basic['Type of Devices'].map(lambda x: [i.strip() for i in x.split(",")])
print('--- Statistics ----')
display(data_basic_type_of_devices.describe())

#print('--- Basic Histogram ----')
#data_basic_type_of_devices.hist()

data_number_of_devices = data_basic_type_of_devices.apply(len)
print_stats(data_number_of_devices)

#print('Population Standard Deviation:')
#print(data_number_of_devices.std(ddof=0))

data_basic_usage_multiple_devices = data_basic['Usage of Multiple-Devices']
print_stats(data_basic_usage_multiple_devices)

#print('Population Standard Deviation:')
#print(data_basic_usage_multiple_devices.std(ddof=0))


I am working on the file "K Means Clustering/K Means Clustering.ipynb".
The first task is:
Create some data
Can you write Python code for it?

from sklearn.datasets import make_blobs

#create data
data = make_blobs(n_samples=200,n_features=2,centers=4,cluster_std=1.8,random_state=101)


I am working on the file "K Means Clustering/K Means Clustering.ipynb".
The first task is:
Creating Clusters
Can you write Python code for it?

from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=4)

kmeans.fit(data[0])

kmeans.cluster_centers_

kmeans.labels_

f, (ax1, ax2) = plt.subplots(1, 2, sharey=True,figsize=(10,6))
ax1.set_title('K Means')
ax1.scatter(data[0][:,0],data[0][:,1],c=kmeans.labels_,cmap='rainbow')
ax2.set_title("Original")
ax2.scatter(data[0][:,0],data[0][:,1],c=data[1],cmap='rainbow')


I am working on the file "K Means Clustering/K Means Clustering.ipynb".
The first task is:
Visualize data
Can you write Python code for it?


I am working on the file "K Means and the UCI Wholesale Data Set.ipynb".
The first task is:
Add a function to convert and join dummy variables to the model.  If source df = dest df then remember to delete the original catagorical variable.
Can you write Python code for it?

def get_dummies(source_df, dest_df, col):
    dummies = pd.get_dummies(source_df[col], prefix=col)

    print 'Quantities for %s column' % col
    for col in dummies:
        print '%s: %d' % (col, np.sum(dummies[col]))
    print

    dest_df = dest_df.join(dummies)
    return dest_df


I am working on the file "K Means and the UCI Wholesale Data Set.ipynb".
The first task is:
Create a feature for total customer size. Note: 'Delicassen' misspelled in original data file.
Can you write Python code for it?

df['Total'] = df['Fresh'] + df['Milk'] + df['Grocery'] + df['Frozen'] + df['Detergents_Paper'] + df['Delicassen']
print df.head()


I am working on the file "K Means and the UCI Wholesale Data Set.ipynb".
The first task is:
Distortion plots often have elbow points that are difficult to interpret, but k = 6 looks like it might make for an interesting set of clusters and plots.
Can you write Python code for it?

np.random.seed(555)  # sets seed, makes it repeatable
k = 6
kmeans = KMeans(n_clusters=k)  # , init='random')
kmeans.fit(X)
pred = kmeans.predict(X)

for i in range(0, k):
    x = len(pred[pred == i])
    print 'Cluster %d has %d members' % ((i + 1), x)


I am working on the file "K Means and the UCI Wholesale Data Set.ipynb".
The first task is:
Plot K Means result with centroids.
Can you write Python code for it?

centroids = sc.inverse_transform(kmeans.cluster_centers_)
plot_kmeans(pred, centroids, 'Frozen', 'Detergents_Paper', 3, 4, k)


I am working on the file "K Means and the UCI Wholesale Data Set.ipynb".
The first task is:
Plot a histogram of customer size, shows a small number of large customers, many smaller customers.
Can you write Python code for it?

plt.hist(df['Total'], bins=32)
plt.xlabel('Total Purchases')
plt.ylabel('Number of Customers')
plt.title('Histogram of Customer Size')
plt.show()
plt.close()


I am working on the file "K Means and the UCI Wholesale Data Set.ipynb".
The first task is:
Process dummy variables for the 'Channel' and 'Region' features.  Drop original categorical feature, plus drop one of the dummy variables for 'leave one out' encoding.
Can you write Python code for it?

df = get_dummies(df, df, 'Channel')
df.drop(['Channel', 'Channel_2'], axis=1, inplace=True)
df = get_dummies(df, df, 'Region')
df.drop(['Region', 'Region_3'], axis=1, inplace=True)
df.rename(index=str, columns={'Channel_1': 'Channel_Horeca', 'Region_1': 'Region_Lisbon', 'Region_2': 'Region_Oporto'},
                              inplace=True)
print df.head()


I am working on the file "K Means and the UCI Wholesale Data Set.ipynb".
The first task is:
Set up a plotting function for K Means output.
Can you write Python code for it?

def plot_kmeans(pred, centroids, x_name, y_name, x_idx, y_idx, k):
    for i in range(0, k):
        plt.scatter(df[x_name].loc[pred == i], df[y_name].loc[pred == i], s=6,
                    c=colors[i], marker=markers[i], label='Cluster %d' % (i + 1))

    centroids = sc.inverse_transform(kmeans.cluster_centers_)
    plt.scatter(centroids[:, x_idx], centroids[:, y_idx],
                marker='x', s=180, linewidths=3,
                color='k', zorder=10)

    plt.xlabel(x_name)
    plt.ylabel(y_name)
    plt.legend()
    plt.show()
    plt.close()


I am working on the file "K Means and the UCI Wholesale Data Set.ipynb".
The first task is:
Use an inverse transform on the centroids.  Needed because the centroids were calculated on the scaled data and we would like the centroids to plot correctly with the original data.
Can you write Python code for it?

centroids = sc.inverse_transform(kmeans.cluster_centers_)

plot_kmeans(pred, centroids, 'Frozen', 'Detergents_Paper', 3, 4, k)


I am working on the file "K Nearest Neighbors/KNN.ipynb".
The first task is:
Get the data
Can you write Python code for it?

df = pd.read_csv('Classified Data',index_col=0)
#set index_col=0 to use the first column as the index

df.head()


I am working on the file "K Nearest Neighbors/KNN.ipynb".
The first task is:
Predictions and Evaluations
Can you write Python code for it?

from sklearn.metrics import classification_report,confusion_matrix

print(confusion_matrix(y_test,pred))

print(classification_report(y_test,pred))


I am working on the file "K Nearest Neighbors/KNN.ipynb".
The first task is:
Train Test Split
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(scaled_features,df['TARGET CLASS'],
                                                    test_size=0.30)


I am working on the file "K Nearest Neighbors/KNN.ipynb".
The first task is:
Using KNN
We'll start with k=1.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=1)

knn.fit(X_train,y_train)

pred = knn.predict(X_test)


I am working on the file "KC_RecSys/project/notebook/Recommenders_5Star.ipynb".
The first task is:
Conversion and cleaning
Surprise forces you to use schema ["user_id", "doc_id", "rating"]
CF models are often sensitive to NA values -> replace NaN with 0 (TBD: originally 0 codes to non-rating, but doc already shown to user)
Can you write Python code for it?

# Convert df
data = raw_data.unstack().to_frame().reset_index()
data.columns = ["user", "doc_uri", "rating"]

# Missing value handling
data.fillna(0, inplace=True)

assert data.shape == (raw_data.shape[0] * raw_data.shape[1], 3), "Conversion error, df has false shape"
assert data.rating.max() <= 5., "Value error, max rating over upper bound"
assert data.rating.min() >= 0., "Value error, min rating under lower bound"
data.head()


I am working on the file "KC_RecSys/project/notebook/Recommenders_5Star.ipynb".
The first task is:
Final model and predictions
kNN Basic looks most promising so far, so go for this one
Train & evaluate final model
Can you write Python code for it?

# Train final model
trainset = ds.build_full_trainset()
knn.fit(trainset)

# RMSE of final model
testset = trainset.build_testset()
test_pred = knn.test(testset)
accuracy.rmse(test_pred, verbose=True) # should be very bad ;)


I am working on the file "KC_RecSys/project/notebook/Recommenders_5Star.ipynb".
The first task is:
Matrix factorization-based CF
Can you write Python code for it?

# Models - tune parameters, if you'd like ;)
svd = SVD() # Singular Value Decomposition
nmf = NMF() # Non-negative Matrix factorization


I am working on the file "KC_RecSys/project/notebook/Recommenders_5Star.ipynb".
The first task is:
Model Evaluation
Don't expect accurate models <- they are trained with random noise
Can you write Python code for it?

for algo in [baseline_model, svd, nmf]:
    cross_validate(algo, ds, measures=["RMSE", "MAE"], cv=5, verbose=True)


cross_validate(knn, ds, measures=["RMSE", "MAE"], cv=5, verbose=True)


I am working on the file "KC_RecSys/project/notebook/Recommenders_5Star.ipynb".
The first task is:
Recommendation Engines
Model-based Collaborative Filtering
Can you write Python code for it?

from surprise import SVD, Dataset, Reader, NMF, accuracy
from surprise.model_selection import cross_validate
from surprise.prediction_algorithms.random_pred import NormalPredictor

reader = Reader(rating_scale=(1, 5))
ds = Dataset.load_from_df(data[["user", "doc_uri", "rating"]], reader)

baseline_model = NormalPredictor() # Baseline model, predicts labels based on distribution of ratings


I am working on the file "KC_RecSys/project/notebook/Recommenders_5Star.ipynb".
The first task is:
k-NN-based CF
Can you write Python code for it?

from surprise.prediction_algorithms.knns import KNNBasic

sim_options = {"name": "pearson", # pearson's r
               "user_based": False  # item-based
               }

knn = KNNBasic(sim_options=sim_options)


I am working on the file "KC_RecSys/project/notebook/Recommenders_binary.ipynb".
The first task is:
Conversion and cleaning
Surprise forces you to use schema ["user_id", "doc_id", "rating"]
CF models are often sensitive to NA values -> replace NaN with 0 OR drop NaN. For demonstration purpose replacement used.
Can you write Python code for it?

# Convert df
data = raw_data.unstack().to_frame().reset_index()
data.columns = ["user", "doc_uri", "rating"]

# Missing value handling
data.fillna(0, inplace=True)

assert data.shape == (raw_data.shape[0] * raw_data.shape[1], 3), "Conversion error, df has false shape"
assert data.rating.max() <= 1., "Value error, max rating over upper bound"
assert data.rating.min() >= -1., "Value error, min rating under lower bound"
data.head()


I am working on the file "KC_RecSys/project/notebook/Recommenders_binary.ipynb".
The first task is:
Descriptive statistics of ratings
Not meaningful <- randomly generated
Can you write Python code for it?

data.rating.describe().to_frame().T

data.rating.value_counts(normalize=True).to_frame().T

# Plot distribution of (random) ratings

hist = data.rating.plot(kind="hist", grid=True,
                        bins=[-1.1,-0.9,-0.1,0.1,0.9,1.1])
hist.set(xlabel= "rating")

plt.tight_layout()
plt.savefig("plots/ratings_binary.png", orientation="landscape", dpi=120)


I am working on the file "KC_RecSys/project/notebook/Recommenders_binary.ipynb".
The first task is:
Evaluation
Don't expect accurate models <- they are trained with random noise.
User- & item-based CF are slightly better than baseline model (predicts labels based on distribution of ratings). User-based approach works surprisingly better than item-based CF and is faster.
Can you write Python code for it?

for algo_name, algo in zip(["Baseline", "User-based CF", "Item-based CF"],
                           [baseline_model, user_knn, item_knn]):
    history = cross_validate(algo, ds, measures=["RMSE", "MAE"], cv=5, verbose=False)
    print("***", algo_name, "***")
    print("RMSE: {:0.3f} (std {:0.4f}) <- {}".format(history["test_rmse"].mean(),
                                                     history["test_rmse"].std(),
                                                     history["test_rmse"]))
    print("MAE: {:0.3f} (std {:0.4f}) <- {}".format(history["test_mae"].mean(),
                                                     history["test_mae"].std(),
                                                     history["test_mae"]))
    print("Avg fit time: {:0.5f}s".format(np.array(history["fit_time"]).mean()))


I am working on the file "KC_RecSys/project/notebook/Recommenders_binary.ipynb".
The first task is:
Item-based CF
Can you write Python code for it?

sim_options = {"name": "cosine", # cosine similarity
               "user_based": False,  # item-based
               "min_support": 5 # min number of common users, else pred 0
               }

item_knn = KNNWithMeans(sim_options=sim_options)


I am working on the file "KC_RecSys/project/notebook/Recommenders_binary.ipynb".
The first task is:
Memory-based
Can we enhance performance of model by using memory-based techniques?
Matrix factorization-based CF
Can you write Python code for it?

# Models - tune parameters, if you'd like ;)
svd = SVD() # Singular value decomposition
pmf = SVD(biased=False) # Probabilistic matrix factorization
nmf = NMF() # Non-negative matrix factorization


I am working on the file "KC_RecSys/project/notebook/Recommenders_binary.ipynb".
The first task is:
Memory-based CF
User-based CF
Can you write Python code for it?

sim_options = {"name": "cosine", # cosine similarity
               "user_based": True,  # user-based
               "min_support": 10 # min number of common items, else pred 0
               }

user_knn = KNNWithMeans(sim_options=sim_options)


I am working on the file "KC_RecSys/project/notebook/Recommenders_binary.ipynb".
The first task is:
Nope, there isn't much of enhancement. But maybe finetuning on the two most promising models helps.
Finetuning models
Grid searching the best parameters -> This might take a while, time to brew some XPRESS0 ;)
Can you write Python code for it?

# SVD
param_svd = {"n_factors": [1, 100],
             "n_epochs": [5, 20],
             "reg_all": [0.02, 0.08], # regularization term for all param
             "lr_all": [0.001, 0.005]} # learning rate for all param

gs_svd = GridSearchCV(SVD, param_svd, measures=["rmse", "mae"], cv=5)
gs_svd.fit(ds)

print("Best RMSE:", gs_svd.best_score["rmse"])
best_params_svd = gs_svd.best_params["rmse"]
for param in best_params_svd:
    print(param, ":", best_params_svd[param])

# NMF
param_nmf = {"n_factors": [15, 100],
             "n_epochs": [50, 60],
             #"biased": [True, False],
             #"reg_pu": [0.04, 0.06, 0.08], # regularization term for users
             #"reg_qi": [0.04, 0.06, 0.08], # regularization term for items
             "lr_bu": [0.001, 0.005], # learning rate for user bias term
             "lr_bi": [0.001, 0.005]} # learning rate for item bias term

gs_nmf = GridSearchCV(NMF, param_nmf, measures=["rmse"], cv=5)
gs_nmf.fit(ds)

print("Best RMSE:", gs_nmf.best_score["rmse"])
best_params_nmf = gs_nmf.best_params["rmse"]
for param in best_params_nmf:
    print(param, ":", best_params_nmf[param])


I am working on the file "KC_RecSys/project/notebook/Recommenders_binary.ipynb".
The first task is:
Preprocess data
Can you write Python code for it?

# Import data
path = "../data/petdata_binary_1000_100.csv"
raw_data = pd.read_csv(path, index_col="doc_uri")
assert raw_data.shape == (1000,100), "Import error, df has false shape"


I am working on the file "KC_RecSys/project/notebook/Recommenders_binary.ipynb".
The first task is:
Recommendation Engines
Can you write Python code for it?

from surprise import KNNWithMeans, SVD, NMF, Dataset, Reader, accuracy
from surprise.prediction_algorithms.random_pred import NormalPredictor
from surprise.model_selection import cross_validate, GridSearchCV

reader = Reader(rating_scale=(-1, 1))
ds = Dataset.load_from_df(data[["user", "doc_uri", "rating"]], reader)

baseline_model = NormalPredictor() # Baseline model, predicts labels based on distribution of ratings


I am working on the file "KK_to_ABS_15apr15.ipynb".
The first task is:
!
Next is to use PyMySQLdb module and do some data manipulation as Somayaji did in the Rscript.
Can you write Python code for it?

from pandasql import sqldf
from pandasql import load_meat, load_births

meat = load_meat
births = load_births

print meat.head()

q = "SELECT * FROM meat LIMIT 10;"

print sqldf(q, locals())


I am working on the file "KK_to_ABS_15apr15.ipynb".
The first task is:
<hr>
</hr>
!
Now, we will use the Crime Data csv file for this exercise and it is here in this location:
Can you write Python code for it?

ls -l *.csv

df = pd.read_csv('Crimes_-_2001_to_present.csv')

df.dtypes # Tells us what data type each column is!

df.ID # Shows us the elements inside the column named, "ID"

# You can also check it's data type separately
df.ID.dtype

df.Date.dtype


I am working on the file "KK_to_ABS_15apr15.ipynb".
The first task is:
I do not know what happened by the above converting command, but I can see 2 changes, while one of them seems positive.

The dates got a - symbol in place /. (Positive)
The IUCR column seems to be totally different from the source csv file (?) Why is it showing NaT instead of an alphanumeric as in the csv file.

The second point may not be due to this command, perhaps it has been this way from when python started reading the file. But I am not sure how to correct this now.
!
Now, I would like to save this newly created dataframe as a new csv file using pandas. This is how I should do it as per the tutorial.
Can you write Python code for it?

# Here is the commnd to do it.
# But the parameter `index=False` will prevent
# the index column from being exported to the csv file.
df.to_csv('Crimes_-_2001_to_present_v2.csv', index=False)

ls -l *.csv


I am working on the file "KLscore.ipynb".
The first task is:
Basic visualization of hmm state evolutions
Can you write Python code for it?

myhmm.transmat_.sum(axis=1) # confirm orientation of transition prob matrix
A = myhmm.transmat_.copy()
fig, ax = plt.subplots(figsize=(3.5, 3))
im = ax.matshow(A, interpolation='none', cmap='RdPu')
# Make an axis for the colorbar on the right side
cax = fig.add_axes([0.9, 0.1, 0.03, 0.8])
fig.colorbar(im, cax=cax)

m = myhmm.n_components
Pi = myhmm.startprob_.copy()
Pi = np.reshape(Pi,(1,m))
fig, ax = plt.subplots(figsize=(6, 2))
ax.stem(np.transpose(Pi),':k')

fig, ax = plt.subplots(figsize=(6, 2))
ax.matshow(Pi)
PiA = np.dot(Pi,A)
ax.matshow(np.vstack([Pi,PiA,np.dot(PiA,A)]))

def advance_states(St,A,n):
    count = 1
    St = np.dot(St,A)
    while count <= n:
        yield St
        count += 1
        St = np.dot(St,A)

numsteps = 50
prior_evo = np.zeros((numsteps+1,m))
prior_evo[0,:] = Pi
for ii, S in enumerate(advance_states(Pi, A, numsteps)):
    prior_evo[ii+1,:] = S

fig, ax = plt.subplots(figsize=(10, 3))
ax.matshow(np.transpose(prior_evo))


I am working on the file "KLscore.ipynb".
The first task is:
First look at shuffle distribution for single sequence
Can you write Python code for it?

def KL(distr_matU, distr_matV):
    from scipy.stats import entropy as KLD
    num_bins = distr_matU.shape[0]

    KLarray = np.zeros(num_bins)

    for ii in np.arange(1,num_bins):
        KLarray[ii-1] = KLD(distr_matU[ii,:],distr_matV[ii,:])

    return KLarray.mean()

from random import shuffle

###########################################################3
stacked_data = ts_b
seq_id = 0
n_shuffles = 500
###########################################################3

tmpseqbdries = [0]; tmpseqbdries.extend(np.cumsum(stacked_data.sequence_lengths).tolist());
obs = stacked_data.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
ll, pp = myhmm.score_samples(obs)
num_bins = obs.shape[0]
ppp = advance_states_one(np.vstack([Pi,pp[:num_bins-1,:]]), A)
trj_shfl_idx = np.arange(0, num_bins);
KL_true = KL(pp,ppp)

KL_shuffles = np.zeros(n_shuffles)
for nn in np.arange(0,n_shuffles):
    shuffle(trj_shfl_idx)
    obs_shfl = np.array([obs[i] for i in trj_shfl_idx])
    ll_shfl, pp_shfl = myhmm.score_samples(obs_shfl)
    ppp_shfl = advance_states_one(np.vstack([Pi,pp_shfl[:num_bins-1,:]]), A)
    KL_shuffles[nn] = KL(pp_shfl,ppp_shfl)

fig, ax1 = plt.subplots(1, 1, figsize=(6, 2))

sns.distplot(KL_shuffles, ax=ax1, bins=40)
ax1.axvline(x=KL_true, ymin=0, ymax=1, linewidth=2, color = 'k', linestyle='dashed', label='true sequence')
x1.legend()


I am working on the file "KLscore.ipynb".
The first task is:
Load data
Can you write Python code for it?

datadirs = ['/home/etienne/Dropbox/neoReader/Data',
            'C:/etienne/Dropbox/neoReader/Data',
            '/Users/etienne/Dropbox/neoReader/Data']

fileroot = next( (dir for dir in datadirs if os.path.isdir(dir)), None)

animal = 'gor01'; month,day = (6,7); session = '11-26-53' # WARNING! POSITION DATA INCOMPLETE!
animal = 'gor01'; month,day = (6,7); session = '16-40-19' # 91 units

#animal = 'gor01'; month,day = (6,12); session = '15-55-31' # 55 units
#animal = 'gor01'; month,day = (6,12); session = '16-53-46' # 55 units

#animal = 'gor01'; month,day = (6,13); session = '14-42-6' # 58 units
#animal = 'gor01'; month,day = (6,13); session = '15-22-3' # 58 units

#animal = 'vvp01'; month,day = (4,9); session = '16-40-54' # ?? units
#animal = 'vvp01'; month,day = (4,9); session = '17-29-30' # ?? units

#animal = 'vvp01'; month,day = (4,10); session = '12-25-50' # lin1; ?? units
#animal = 'vvp01'; month,day = (4,10); session = '12-58-3'  # lin2; ?? units
#animal = 'vvp01'; month,day = (4,10); session = '19-11-57' # lin2; ?? units
#animal = 'vvp01'; month,day = (4,10); session = '21-2-40'  # lin1; ?? units

#animal = 'vvp01'; month,day = (4,18); session = '13-06-01' # lin1; ?? units
#animal = 'vvp01'; month,day = (4,18); session = '13-28-57' # lin2; ?? units
#animal = 'vvp01'; month,day = (4,18); session = '15-23-32' # lin1; ?? units
#animal = 'vvp01'; month,day = (4,18); session = '15-38-02' # lin2; ?? units

spikes = load_data(fileroot=fileroot, datatype='spikes',animal=animal, session=session, month=month, day=day, fs=32552, verbose=True)
eeg = load_data(fileroot=fileroot, datatype='eeg', animal=animal, session=session, month=month, day=day,channels=[0,1,2], fs=1252, starttime=0, verbose=True)
posdf = load_data(fileroot=fileroot, datatype='pos',animal=animal, session=session, month=month, day=day, verbose=True)
speed = klab.get_smooth_speed(posdf,fs=60,th=8,cutoff=0.5,showfig=True,verbose=True)

## bin ALL spikes
ds = 0.125 # bin spikes into 125 ms bins (theta-cycle inspired)
binned_spikes =...


I am working on the file "KLscore.ipynb".
The first task is:
Now look at entire data set, with shuffles for each sequence
In particular, we consider all the sequences in the test set of running behavior, ts_b, none of which have been seen by the HMM before.
Can you write Python code for it?

from random import shuffle

###########################################################3
stacked_data = ts_b
n_shuffles = 250 # shuffles PER sequence in data set
###########################################################3

num_sequences = len(stacked_data.sequence_lengths)
tmpseqbdries = [0]; tmpseqbdries.extend(np.cumsum(stacked_data.sequence_lengths).tolist());

KL_true = np.zeros(num_sequences)
KL_shuffles = np.zeros((num_sequences,n_shuffles))

for seq_id in np.arange(0,num_sequences):

    obs = stacked_data.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
    ll, pp = myhmm.score_samples(obs)
    num_bins = obs.shape[0]
    ppp = advance_states_one(np.vstack([Pi,pp[:num_bins-1,:]]), A)
    trj_shfl_idx = np.arange(0, num_bins);
    KL_true[seq_id] = KL(pp,ppp)

    for nn in np.arange(0,n_shuffles):
        shuffle(trj_shfl_idx)
        obs_shfl = np.array([obs[i] for i in trj_shfl_idx])
        ll_shfl, pp_shfl = myhmm.score_samples(obs_shfl)
        ppp_shfl = advance_states_one(np.vstack([Pi,pp_shfl[:num_bins-1,:]]), A)
        KL_shuffles[seq_id,nn] = KL(pp_shfl,ppp_shfl)

fig, ax1 = plt.subplots(1, 1, figsize=(6, 2))

sns.distplot(KL_true, ax=ax1, label='true sequences')
sns.distplot(KL_shuffles.flatten(), bins=80, ax=ax1, label='trajectory shuffled')
#ax1.axvline(x=KL_true, ymin=0, ymax=1, linewidth=2, color = 'k', linestyle='dashed', label='true sequence')
#ax1.set_xlim([0,5])
ax1.legend()


I am working on the file "KLscore.ipynb".
The first task is:
Remark: What if our pmfs contain any zeros? Problem!
One way to think about your problem is that you don't really have confidence in the PMF you have calculated from the histogram. You might need a slight prior in your model. Since if you were confident in the PMF, then the KL divergence should be infinity since you got values in one PMF that are impossible in the other PMF. If, on the other hand you had a slight, uninformative prior then there is always some small probability of seeing a certain outcome. One way of introducing this would be to add a vector of ones times some scalar to the histogram. The theoretical prior distribution you would be using is the dirichlet distribution, which is the conjugate prior of the categorical distribution. But for practical purposes you can do something like
python
    pmf_unnorm = scipy.histogram(samples, bins=bins, density=True)[0] +  w * scipy.ones(len(bins)-1)
    pmf = pmf_unnor / sum(pmf_unnorm)
where w is some positive weight, depending on how strong a prior you want to have.
Can you write Python code for it?

from scipy.stats import entropy as KLD

KLlist = []
KLlist_shfl = []

for ii in np.arange(1,len(pth)):
    KLlist.append(KLD(pp[ii,:],ppp[ii,:]))
    KLlist_shfl.append(KLD(pp_shfl[ii,:],ppp_shfl[ii,:]))

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 3))

ax1.plot(np.cumsum(KLlist), label='sequence', lw=1.5)
ax1.plot(np.cumsum(KLlist_shfl), label='trajectory shuffled', lw=1.5)
ax1.legend()
ax1.set_xlabel('bin')
ax1.set_title('Cumulative KL divergence')

seqscore = np.cumsum(KLlist) / np.arange(1,len(pth))
seqscore_shfl = np.cumsum(KLlist_shfl) / np.arange(1,len(pth))

ax2.plot(seqscore, label='sequence', lw=1.5)
ax2.plot(seqscore_shfl, label='trajectory shuffled', lw=1.5)
ax2.legend()
ax2.set_xlabel('bin')
ax2.set_title('Running average KL divergence')


I am working on the file "KMeansCluster.ipynb".
The first task is:
General information on the Gapminder data
Can you write Python code for it?

display(Markdown("Number of countries: {}".format(len(data))))


I am working on the file "KMeansCluster.ipynb".
The first task is:
If we set employment rate aside, the group 0 gathers the better countries (high urban rate, democratic system, high life expectancy, high internet use rate and high electricity consumption) when the groups 1 and 2 gather the countries not having those characteristics.
The difference between the group 1 and 2 is mainly on the employment rate. The group 1 having a important employment rate when the group 2 has a low one.
Validation on the test variable
The tree clusters will be tested against our test variable income per person. To confirm the accuracy of the clusters obtained.
Can you write Python code for it?

target_train, target_test = train_test_split(target,
                                             test_size=.3,
                                             random_state=123)
merged_train_all=merged_train.merge(target_train.to_frame(),
                                    left_index=True,
                                    right_index=True)
sub1 = merged_train_all[[test_var, 'cluster']].dropna()

import statsmodels.formula.api as smf
import statsmodels.stats.multicomp as multi

income = smf.ols(formula='{} ~ C(cluster)'.format(test_var), data=sub1).fit()
income.summary()


I am working on the file "KMeansCluster.ipynb".
The first task is:
Predictors selections and standardization
Can you write Python code for it?

explanatory_vars = ['employrate', 'urbanrate', 'polityscore', 'lifeexpectancy', 'internetuserate', 'relectricperperson']
test_var = 'incomeperperson'

constructor_dict = dict()
for var in explanatory_vars + [test_var, ]:
    constructor_dict[var] = pd.to_numeric(data[var], errors='coerce')

numeric_data = pd.DataFrame(constructor_dict, index=data.index).dropna()

display(Markdown("Number of countries after discarding countries with missing data: {}".format(len(numeric_data))))

predictors = numeric_data[explanatory_vars]

target = numeric_data[test_var]

# Standardize predictors to have mean=0 and std=1
std_predictors = predictors.copy()

for var in std_predictors.columns:
    std_predictors[var] = preprocessing.scale(std_predictors[var].astype('float64'))

# Check standardization
std_predictors.describe()


I am working on the file "KMeansCluster.ipynb".
The first task is:
The ANOVA test confirms there is a significant variation of income per person with cluster (p-value = 1.46e-10 << 0.05).
Can you write Python code for it?

display(Markdown('means for income by cluster :'))
sub1.groupby('cluster').mean()

display(Markdown('standard deviations for income by cluster : '))
sub1.groupby('cluster').std()


I am working on the file "KMeansCluster.ipynb".
The first task is:
The group 0 has the highest income per person (even if we substract the standard deviation, it stays higher than the average of groups 1 and 2).
Group 1 has a lower average than group 2. But due to their respective standard deviation, they have a large overlapping confirming the idea of merging them for a better description.
Can you write Python code for it?

mc1 = multi.MultiComparison(sub1[test_var], sub1['cluster'])
res1 = mc1.tukeyhsd()
res1.summary()


I am working on the file "KMeansCluster.ipynb".
The first task is:
The multiple comparison above confirms that the cluster analysis would be better keeping only two clusters as groups 1 and 2 have no significant difference.
Validation on the test subset
The clustering will be validated on the test data set by looking at the evolution of the average distance with the number of clusters and by looking at the clusters in the canonical variables space.
Can you write Python code for it?

meandist_test = list()

for k in clusters:
    model = KMeans(n_clusters=k+1)
    model.fit(cluster_train)
    meandist_test.append(sum(np.min(cdist(cluster_test,
                                     model.cluster_centers_,
                                     'euclidean'),
                               axis=1))
                    / cluster_test.shape[0])

plt.plot(clusters, meandist, label='Train data')
plt.plot(clusters, meandist_test, label='Test data')
plt.legend()
plt.xlabel('Number of clusters')
plt.ylabel('Average distance')
plt.title('Selecting k with the Elbow Method');


I am working on the file "KMeans_Tutorial.ipynb".
The first task is:
Analyzing the clusters
As you can see in the previous plots, K-means does a good job of separating the Setosa species (red) into its own cluster. It also does a reasonable job separating Versicolour (green) and Virginica (blue), although there is a considerable amount of overlap that it can't predict properly.
This is an example where it is important to understand your data (and visualize it whenever possible), as well as understand your machine learning model. In this example, you may want to use a different machine learning model that can separate the data more accurately. Alternatively, we could use all four features to see if that improves accuracy (remember, we aren't using petal length or width here for easier data visualization).
Changing the number of clusters
What would happen if you changed the number of clusters? What would the plot look like with 2 clusters, or 5? Based on the unlabeled data, how would you try to determine the number of classes to use?
In the next block of code, try changing the number of clusteres and seeing what happens. You may need to change the number of colors represented in cmap_bg to match the number of classes you are using.
Can you write Python code for it?


I am working on the file "KMeans_Tutorial.ipynb".
The first task is:
As you can see, example A is grouped into Cluster 2 and example B is grouped into Cluster 0. Remember, K-means does not use labels. It only clusters the data by feature similarity, and it's up to us to decide what the clusters mean (or if they don't mean anything at all).
Using different features
Try using different combinations of the four features and see what results you get. Does it make it any easier to determine how many clusters should be used?
Can you write Python code for it?


I am working on the file "KMeans_Tutorial.ipynb".
The first task is:
Below, we plot the first two features from the dataset (sepal length and width). Normally we would try to use all useful features, but sticking with two allows us to visualize the data more easily.
Then we plot the data to get a look at what we're dealing with. The colormap is used to determine what colors are used for each class when plotting.
Can you write Python code for it?

# Plot the data

# Sepal length and width
X_small = X[:,:2]
# Get the minimum and maximum values with an additional 0.5 border
x_min, x_max = X_small[:, 0].min() - .5, X_small[:, 0].max() + .5
y_min, y_max = X_small[:, 1].min() - .5, X_small[:, 1].max() + .5

plt.figure(figsize=(8, 6))

# Plot the training points
plt.scatter(X_small[:, 0], X_small[:, 1], c=y, cmap=cmap_bold)
plt.xlabel('Sepal length (cm)')
plt.ylabel('Sepal width (cm)')
plt.title('Sepal width vs length')

# Set the plot limits
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)

# Plot the legend
plt.legend(labelList, labelNames)

plt.show()


I am working on the file "KMeans_Tutorial.ipynb".
The first task is:
Cheating with labels
Because we do have labels for this dataset, let's see how well k-means did at separating the three species.
Can you write Python code for it?

h = .02  # step size in the mesh

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, m_max]x[y_min, y_max].
x_min, x_max = X_small[:, 0].min() - 1, X_small[:, 0].max() + 1
y_min, y_max = X_small[:, 1].min() - 1, X_small[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()]) # Make a prediction oat every point
                                               # in the mesh in order to find the
                                               # classification areas for each label

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(figsize=(8, 6))
plt.pcolormesh(xx, yy, Z, cmap=cmap_bg)

# Plot the training points
plt.scatter(X_small[:, 0], X_small[:, 1], c=y, cmap=cmap_bold)
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.title("KMeans (k = %i)"
         % (n_clusters))
plt.xlabel('Sepal length (cm)')
plt.ylabel('Sepal width (cm)')

# Plot the centroids as a black X
centroids = kmeans.cluster_centers_
plt.scatter(centroids[:, 0], centroids[:, 1],
           marker='x', s=169, linewidths=3,
           color='k', zorder=10)

# Plot the legend
plt.legend(labelList, labelNames)

plt.show()


I am working on the file "KMeans_Tutorial.ipynb".
The first task is:
Import the dataset
Import the dataset and store it to a variable called iris. Scikit-learn's explanation of the dataset is here. This dataset is similar to a python dictionary, with the keys: ['DESCR', 'target_names', 'target', 'data', 'feature_names']
The data features are stored in iris.data, where each row is an example from a single flower, and each column is a single feature. The feature names are stored in iris.feature_names. Labels are stored as the numbers 0, 1, or 2 in iris.target, and the names of these labels are in iris.target_names.
The dataset consists of measurements made on 50 examples from each of three different species of iris flowers (Setosa, Versicolour, and Virginica). Each example has four features (or measurements): sepal length, sepal width, petal length, and petal width. All measurements are in cm.
Below, we load the labels into y, the corresponding label names into labelNames, the data into X, and the names of the features into featureNames.
Can you write Python code for it?

# Import some data to play with
iris = datasets.load_iris()

# Store the labels (y), label names, features (X), and feature names
y = iris.target       # Labels are stored in y as numbers
labelNames = iris.target_names # Species names corresponding to labels 0, 1, and 2
X = iris.data
featureNames = iris.feature_names


I am working on the file "KMeans_Tutorial.ipynb".
The first task is:
K-means: training
Next, we train a K-means classifier on our data.
The first section chooses the number of clusters to use, and stores it in the variable n_clusters. We choose 3 because we know there are 3 species of iris, but we don't always know this when approaching a machine learning problem.
The last two lines create and train the classifier.
The first line creates a classifier (kmeans) using the KMeans() function, and tells it to use the number of neighbors stored in n_neighbors. The second line uses the fit() method to train the classifier on the features in X. Notice that because this is an unsupervised method, it does not use the labels stored in y.
Can you write Python code for it?

# Choose your number of clusters
n_clusters = 3

# we create an instance of KMeans Classifier and fit the data.
kmeans = KMeans(n_clusters=n_clusters)
kmeans.fit(X_small)


I am working on the file "KMeans_Tutorial.ipynb".
The first task is:
Making Predictions
Now, let's say we go out and measure the sepals of two iris plants, and want to know what group they belong to. We're going to use our classifier to predict the flowers with the following measurements:
Plant | Sepal length | Sepal width
------|--------------|------------
A     |4.3           |2.5
B     |6.3           |2.1
We can use our classifier's predict() function to predict the label for our input features. We pass in the variable examples to the predict() function, which is a list, and each element is another list containing the features (measurements) for a particular example. The output is a list of labels corresponding to the input examples.
We'll also plot them on the boundary plot, to show why they were predicted that way.
Can you write Python code for it?

# Add our new data examples
examples = [[4.3, 2.5], # Plant A
            [6.3, 2.1]] # Plant B

# Choose your number of clusters
n_clusters = 3

# we create an instance of KMeans Classifier and fit the data.
kmeans = KMeans(n_clusters=n_clusters)
kmeans.fit(X_small)

# Predict the labels for our new examples
labels = kmeans.predict(examples)

# Print the predicted species names
print('A: Cluster ' + str(labels[0]))
print('B: Cluster ' + str(labels[1]))

# Now plot the results
h = .02  # step size in the mesh

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, m_max]x[y_min, y_max].
x_min, x_max = X_small[:, 0].min() - 1, X_small[:, 0].max() + 1
y_min, y_max = X_small[:, 1].min() - 1, X_small[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))
Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(figsize=(8, 6))
plt.pcolormesh(xx, yy, Z, cmap=cmap_bg)

# Plot the training points
plt.scatter(X_small[:, 0], X_small[:, 1])
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.title("KMeans (k = %i)"
         % (n_clusters))
plt.xlabel('Sepal length (cm)')
plt.ylabel('Sepal width (cm)')

# Plot the centroids as a black X
centroids = kmeans.cluster_centers_
plt.scatter(centroids[:, 0], centroids[:, 1],
           marker='x', s=169, linewidths=3,
           color='k', zorder=10)

# Display the new examples as labeled text on the graph
plt.text(examples[0][0], examples[0][1],'A', fontsize=14)
plt.text(examples[1][0], examples[1][1],'B', fontsize=14)

plt.show()


I am working on the file "KMeans_Tutorial.ipynb".
The first task is:
Next, the datapoints are re-assigned to the new closest centroid.
Can you write Python code for it?

h = .02  # step size in the mesh

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, m_max]x[y_min, y_max].
x_min, x_max = data[:, 0].min() - 1, data[:, 0].max() + 1
y_min, y_max = data[:, 1].min() - 1, data[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()]) # Make a prediction at every point
                                               # in the mesh in order to find the
                                               # classification areas for each label

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(figsize=(8, 6))
plt.pcolormesh(xx, yy, Z, cmap=cmap_bg)

# Plot the training points
plt.scatter(data[:, 0], data[:, 1])
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.title("Re-assigned to New Clusters")

# Plot the centroids as a black X
plt.scatter(centroids[:, 0], centroids[:, 1],
           marker='x', s=169, linewidths=3,
           color='r', zorder=10)

plt.show()


I am working on the file "KMeans_Tutorial.ipynb".
The first task is:
Plot the classification boundaries
Now that we have our classifier, let's visualize what it's doing.
First we plot the decision boundaries, or the lines dividing areas assigned to the different clusters. The background shows the areas that are considered to belong to a certain cluster, and each cluster can then be assigned to a species of iris. They are plotted in grey, because the classifier does not assign labels to the clusters. The center of each cluster is plotted as a black x. Then we plot our examples onto the space, showing where each point lies in relation to the decision boundaries.
If we took sepal measurements from a new flower, we could plot it in this space and use the background shade to determine which cluster of data points our classifier would assign to it.
Can you write Python code for it?

h = .02  # step size in the mesh

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, m_max]x[y_min, y_max].
x_min, x_max = X_small[:, 0].min() - 1, X_small[:, 0].max() + 1
y_min, y_max = X_small[:, 1].min() - 1, X_small[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()]) # Make a prediction oat every point
                                               # in the mesh in order to find the
                                               # classification areas for each label

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(figsize=(8, 6))
plt.pcolormesh(xx, yy, Z, cmap=cmap_bg)

# Plot the training points
plt.scatter(X_small[:, 0], X_small[:, 1])
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.title("KMeans (k = %i)"
         % (n_clusters))
plt.xlabel('Sepal length (cm)')
plt.ylabel('Sepal width (cm)')

# Plot the centroids as a black X
centroids = kmeans.cluster_centers_
plt.scatter(centroids[:, 0], centroids[:, 1],
           marker='x', s=169, linewidths=3,
           color='k', zorder=10)

plt.show()


I am working on the file "KNN.ipynb".
The first task is:
3) Uma vez que temos os vizinhos mais próximos, precisamos contar a classe de cada um, para saber o que responder
Can you write Python code for it?

from collections import Counter

def get_majority_vote(neighbours):
    # presumo aqui que a neighbours tem o formato
    # [(item,classe), (item,classe)...]
    classes = [neighbour[1] for neighbour in neighbours]
    count = Counter(classes)
    return count.most_common()[0][0]


I am working on the file "KNN.ipynb".
The first task is:
Equal distance contours
Can you write Python code for it?

%run plot_normballs.py

def nearest(A,x, p=2):
    '''A: NxD data matrix, N - number of samples, D - the number of features
       x: test vector

       returns the distance and index of the the nearest neigbor
    '''
    N = A.shape[0]
    d = np.zeros((N,1))

    md = np.inf

    for i in range(N):
        d[i] = Divergence(A[i,:], x, p)
        if d[i]<md:
            md = d[i]
            min_idx = i

    return min_idx

def predict(A, c, X, p=2):
    L = X.shape[0]
    return [np.asscalar(c[nearest(A, X[i,:], p=p)]) for i in range(L)]

x_test = np.mat('[3.3, 2.5,5.5,1.7]')

#d, idx = distance(X, x_test, p=2)

cc = predict(X, c, x_test)

print(cc)
#float(c[idx])

def leave_one_out(A, c, p=2):

    N = A.shape[0]
    correct = 0
    for j in range(N):
        md = np.inf
        for i in range(N):
            if i != j:
                d = Divergence(A[i,:], A[j,:], p=p)
                if d<md:
                    md = d
                    min_idx = i
        if c[min_idx] == c[j]:
            correct += 1

    accuracy = 1.*correct/N
    return accuracy

leave_one_out(X, c, p=np.diag([1,1,1,1]))


I am working on the file "KNN.ipynb".
The first task is:
Python 2.7 中
- dict.iteritems() 返回迭代器
- dict.items() 返回字典的复制
Python 3 中
- dict.items() 返回迭代器
- dict.iteritems() 该函数在 Python 3 中不存在了
我用的 Python 3，所以下面的代码中，我用的是 dict.items()
operator.itemgetter 函数可以获取一个对象指定序号的数据

operator.itemgetter 获取的不是值，而是一个函数，通过该函数作用到对象上才能获取值。

一般该函数用在 sorted 函数中。

需要 import operator 模块
Can you write Python code for it?

a={'a':3,'b':2,'c':5,'d':1}
import operator
sorted(a.items(),key=operator.itemgetter(1),reverse=False)


I am working on the file "KNN.ipynb".
The first task is:
The choice of the distance function (divergence) can be important. In practice, a popular choice is the Euclidian distance but this is by no means the only one.
Can you write Python code for it?

def Divergence(x,y,p=2.):
    e = np.array(x) - np.array(y)
    if np.isscalar(p):
        return np.sum(np.abs(e)**p)
    else:
        return np.sum(np.matrix(e)*p*np.matrix(e).T)

Divergence([0,0],[1,1],p=2)
W = np.matrix(np.diag([2,1]))
Divergence([0,0],[1,1],p=W)
W = np.matrix([[2,1],[1,2]])
Divergence([0,0],[1,1],p=W)


I am working on the file "KNN.ipynb".
The first task is:
agora vamos brincar...
Can you write Python code for it?


I am working on the file "KNN.ipynb".
The first task is:
sum 求和
Can you write Python code for it?


I am working on the file "KNN.ipynb".
The first task is:
sum(0) 按列求和
sum(1) 按行求和
min() max() 两个函数同样0列1行
Can you write Python code for it?


I am working on the file "KNN.ipynb".
The first task is:
以上方法可以对字典按值排序<br>
排序从小到大，reverse=True则是从大到小
Can you write Python code for it?


I am working on the file "KNN.ipynb".
The first task is:
以上方法按键排序

程序清单 2-1
Can you write Python code for it?

def classify0(inX, dataSet, labels, k):
	dataSetSize = dataSet.shape[0]
	diffMat = tile(inX, (dataSetSize, 1)) - dataSet
	sqDiffMat = diffMat ** 2
	sqDistances = sqDiffMat.sum(axis=1)
	distances = sqDistances ** 0.5
	sortedDistIndicies = distances.argsort()
	classCount = {}
	for i in range(k):
		voteIlabel = labels[sortedDistIndicies[i]]
		classCount[voteIlabel] = classCount.get(voteIlabel, 0) + 1
	sortedClassCount = sorted(classCount.items(), key=operator.itemgetter(1), reverse=True)
	return sortedClassCount[0][0]


I am working on the file "KNN_Tutorial.ipynb".
The first task is:
Evaluating The Classifier
In order to evaluate a classifier, we need to split our dataset into training data, which we'll show to the classifier so it can learn, and testing data, which we will hold back from training and use to test its predictions.
Below, we create the training and testing datasets, using all four features. We then train our classifier on the training data, and get the predictions for the test data.
Can you write Python code for it?

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

# Choose your number of neighbors
n_neighbors_distance = 15

# we create an instance of Neighbours Classifier and fit the data.
clf_distance = neighbors.KNeighborsClassifier(n_neighbors_distance, weights='distance')
clf_distance.fit(X_train, y_train)

# Predict the labels of the test data
predictions = clf_distance.predict(X_test)


I am working on the file "KNN_Tutorial.ipynb".
The first task is:
Import the dataset
Import the dataset and store it to a variable called iris. Scikit-learn's explanation of the dataset is here. This dataset is similar to a python dictionary, with the keys: ['DESCR', 'target_names', 'target', 'data', 'feature_names']
The data features are stored in iris.data, where each row is an example from a single flower, and each column is a single feature. The feature names are stored in iris.feature_names. Labels are stored as the numbers 0, 1, or 2 in iris.target, and the names of these labels are in iris.target_names.
The dataset consists of measurements made on 50 examples from each of three different species of iris flowers (Setosa, Versicolour, and Virginica). Each example has four features (or measurements): sepal length, sepal width, petal length, and petal width. All measurements are in cm.
Below, we load the labels into y, the corresponding label names into labelNames, the data into X, and the names of the features into featureNames.
Can you write Python code for it?


I am working on the file "KNN_Tutorial.ipynb".
The first task is:
Making predictions
Now, let's say we go out and measure the sepals of two new iris plants, and want to know what species they are. We're going to use our classifier to predict the flowers with the following measurements:
Plant | Sepal length | Sepal width
------|--------------|------------
A     |4.3           |2.5
B     |6.3           |2.1
We can use our classifier's predict() function to predict the label for our input features. We pass in the variable examples to the predict() function, which is a list, and each element is another list containing the features (measurements) for a particular example. The output is a list of labels corresponding to the input examples.
Can you write Python code for it?

# Add our new data examples
examples = [[4.3, 2.5], # Plant A
            [6.3, 2.1]] # Plant B

# Reset our number of neighbors
n_neighbors = 15

# Create an instance of Neighbours Classifier and fit the original data.
clf = neighbors.KNeighborsClassifier(n_neighbors)
clf.fit(X_small, y)

# Predict the labels for our new examples
labels = clf.predict(examples)

# Print the predicted species names
print('A: ' + labelNames[labels[0]])
print('B: ' + labelNames[labels[1]])


I am working on the file "KNN_Tutorial.ipynb".
The first task is:
Nearest neighbors: training
Next, we train a nearest neighbor classifier on our data.
The first section chooses the number of neighbors to use, and stores it in the variable n_neighbors (another, more intuitive, name for the K variable mentioned previously).
The last two lines create and train the classifier. Line 1 creates a classifier (clf) using the KNeighborsClassifier() function, and tells it to use the number of neighbors stored in n_neighbors. Line 2 uses the fit() method to train the classifier on the features in X_small, using the labels in y.
Can you write Python code for it?

# Choose your number of neighbors
n_neighbors = 15

# we create an instance of Neighbours Classifier and fit the data.
clf = neighbors.KNeighborsClassifier(n_neighbors)
clf.fit(X_small, y)


I am working on the file "KNN_Tutorial.ipynb".
The first task is:
Now, see how this affects your plots using other features.
Can you write Python code for it?


I am working on the file "KNN_Tutorial.ipynb".
The first task is:
Plotting our predictions
Now let's plot our predictions to see why they were classified that way.
Can you write Python code for it?

# Now plot the results
h = .02  # step size in the mesh

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, m_max]x[y_min, y_max].
x_min, x_max = X_small[:, 0].min() - 1, X_small[:, 0].max() + 1
y_min, y_max = X_small[:, 1].min() - 1, X_small[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

Z = clf.predict(np.c_[xx.ravel(), yy.ravel()]) # Make a prediction oat every point
                                               # in the mesh in order to find the
                                               # classification areas for each label

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(figsize=(8, 6))
plt.pcolormesh(xx, yy, Z, cmap=cmap_light)

# Plot the training points
plt.scatter(X_small[:, 0], X_small[:, 1], c=y, cmap=cmap_bold)
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.title("3-Class classification (k = %i)"
         % (n_neighbors))
plt.xlabel('Sepal length (cm)')
plt.ylabel('Sepal width (cm)')

# Display the new examples as labeled text on the graph
plt.text(examples[0][0], examples[0][1],'A', fontsize=14)
plt.text(examples[1][0], examples[1][1],'B', fontsize=14)

# Plot the legend
plt.legend(labelList, labelNames)

plt.show()


I am working on the file "KNN_Tutorial.ipynb".
The first task is:
Select a Model
To select a model, we look at the average accuracy across all folds.
Can you write Python code for it?

for k in kvals:
    print('k=%i: %.2f' % (k, np.mean(accuracies[k])))


I am working on the file "KNN_Tutorial.ipynb".
The first task is:
Sidenote: Randomness and Results
Every time you run this notebook, you will get slightly different results. Why? Because data is randomly divided among the training/testing/validation data sets. Running the code again will create a different division of the data, and will make the results slightly different. However, the overall outcome should remain consistent and have approximately the same values. If you have drastically different results when running an analysis multiple times, it suggests a problem with your model or that you need more data.
If it's important that you get the exact same results every time you run the code, you can specify a random state in the random_state argument of train_test_split() and KFold.
Can you write Python code for it?

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)


I am working on the file "KNN_Tutorial.ipynb".
The first task is:
Using more than two features
Using two features is great for visualizing, but it's often not good for training a good classifier. Below, we will train a classifier using the 'distance' weighting method and all four features, and use that to predict plants A and B.
How do the predictions compare to our predictions using only two labels?
Can you write Python code for it?

# Add our new data examples
examples = [[4.3, 2.5, 1.5, 0.5], # Plant A
            [6.3, 2.1, 4.8, 1.5]] # Plant B

# Choose your number of neighbors
n_neighbors_distance = 15

# we create an instance of Neighbours Classifier and fit the data.
clf_distance = neighbors.KNeighborsClassifier(n_neighbors_distance, weights='distance')
clf_distance.fit(X, y)

# Predict the labels of the new examples
labels = clf_distance.predict(examples)

# Print the predicted species names
print('A: ' + labelNames[labels[0]])
print('B: ' + labelNames[labels[1]])


I am working on the file "KNN_Tutorial.ipynb".
The first task is:
What about our other features?
You may remember that our original dataset contains two additional features, the length and width of the petals.
What does the plot look like when you train on the petal length and width? How does it change when you change the number of neighbors?
How would you plot our two new plants, A and B, on these new plots? Assume we have all four measurements for each plant, as shown below.
Plant | Sepal length | Sepal width| Petal length | Petal width
------|--------------|------------|--------------|------------
A     |4.3           |2.5         | 1.5          | 0.5
B     |6.3           |2.1         | 4.8          | 1.5
Can you write Python code for it?

# Put your code here!

# Feel free to add as many code cells as you need


I am working on the file "KRR - TotalEnergy.ipynb".
The first task is:
Let's pick a descriptor. Allowed types are:

cnt: atom counts
bob: bag of bonds
soap: smooth overlap of atomic positions; choose from:
soap.sum - all atoms summed together
soap.mean - mean of all atom SOAP
soap.centre - computed at the central point

mbtr: many-body tensor representation
cm: Coulomb matrix
Can you write Python code for it?

# TYPE is the descriptor type
TYPE = "cnt"

#show descriptor details
print("\nDescriptor details")
desc = open("./data/descriptor."+TYPE.split('.')[0]+".txt","r").readlines()
for l in desc: print(l.strip())
print(" ")


I am working on the file "KRR - TotalEnergy.ipynb".
The first task is:
The evaluation function computes the distance between each validation sample $T_i$ and the training ones $M_i$:
$$ D_{ij} =  d(T_{i}, M_{j}) $$
The predicted energies $E\left(T\right)$ are then given by:
$$ E\left( T \right) = E\left( M \right) \cdot K^{-1} \cdot D^T $$
Can you write Python code for it?

# get a list of predicted outputs for the validation inputs
predict = solver.Evaluate(validIn)

print("Mean Abs Error (validation): " + str((numpy.abs(predict-validOut)).mean()))

# do the regression plot
plt.plot(validOut, predict, 'o')
plt.plot([numpy.min(validOut),numpy.max(validOut)], [numpy.min(validOut),numpy.max(validOut)], '-')
plt.xlabel('correct output')
plt.ylabel('KRR output')
plt.show()

# check the distribution of energies in the training set
plt.hist(validOut, bins=20, density=True, alpha=0.5, facecolor='gray')
plt.hist(trainOut, bins=20, density=True, alpha=0.2, facecolor='red')
plt.xlabel("Energy [H]")
plt.ylabel("Probability")
plt.show()


I am working on the file "KRR - TotalEnergy.ipynb".
The first task is:
and load the databases with the descriptors (input) and the correct charge densities (output). Databases are quite big, so we can decide how many samples to use for training.
Can you write Python code for it?

# load input/output data
trainIn = load_npz("./data/energy.input."+TYPE+".npz").toarray()
trainOut = numpy.load("./data/energy.output.npy")
trainIn = trainIn.astype(dtype=numpy.float64, casting='safe')

# decide how many samples to take from the database
samples  = min(trainIn.shape[0], 1000) # change the number here!
vsamples = min(trainIn.shape[0]-samples,1000)
print("training samples:   "+str(samples))
print("validation samples: "+str(vsamples))

# split between training and validation
validIn = trainIn[samples:samples+vsamples]
validOut = trainOut[samples:samples+vsamples]

trainIn  = trainIn[0:samples]
trainOut = trainOut[0:samples]

# show the first few descriptors
print("\nDescriptors for the first 5 molecules:")
print(trainIn[0:5])


I am working on the file "KSP_pid_tuning.ipynb".
The first task is:
Awesome! The controller drops the velocity to a stable oscillation around 0 m/s, and the position seems to flatten off, but it isn't perfect. Maybe it's because of the oscillations? In the game I can see the engine spurt on and off rythmically. It seems to try and stay at roughly 0 m/s, but the position is not 100m and it drifts.
The kOS script to run this test is located in hover2.ks and called on vspeed.txt by RUN hover2(vspeed.txt,20).
Position Control
Instead of gforce or velocity, lets try controlling for position next.
$$ \Delta thrott = K_p(altitude_{goal} - altitude_{current}) $$
Using $Kp = 0.05$ and  $altitude_{goal} = 100$.
LOCK dthrott_p TO Kp * (goal_altitude - SHIP:ALTITUDE).
LOCK dthrott TO dthrott_p.

Note: From now on we start the controller directly from the launch pad (instead of burning till passing 100m).
Can you write Python code for it?

# To run in kOS console: RUN hover3(pos0.txt,20,0.05,0).
data = loadData('collected_data\\pos0.txt')
plotData(data)


I am working on the file "KSP_pid_tuning.ipynb".
The first task is:
G-force Control
As a first test, we'll start from the launchpad, thrust at full throttle till we hit $altitude_{goal} > 100\ \text{meters}$ and then use a proportional gain of $Kp = 0.05$ to keep $gforce \sim 1.0$.
LOCK dthrott_p TO Kp * (1.0 - gforce).
LOCK dthrott TO dthrott_p.
Can you write Python code for it?

data = loadData('collected_data\\gforce.txt')
plotData(data)


I am working on the file "KSP_pid_tuning.ipynb".
The first task is:
Hmm, after trying a few other combinations, it seems like there's a conceptual error here keeping us from getting to a smooth point.
We've been trying to build our controller with $thrott = thrott + \Delta thrott$$ which means it takes time to overcome our previous throttle, introducing this lag between our current position/velocity and goal one, that shows up as an oscillation.
The kOS script used is hover3.ks and these tests are run by calling RUN hover3(posN.txt,20,Kp,Kd). For example for $Kp=0.08$ and $Kd=0.04$ the command is RUN hover3(pos2.txt,20,0.08,0.04), for data pos2.txt.
Hover set-point
Instead let's figure out how to set our throttle point for hovering, and set $thrott = thrott_{hover} + \Delta thrott$
We know our ship's current max thrust with 100%, which we will call $Thrust_{available}$. What we want to find is our $Throttle_\%$ which would be
$$Throttle_\% = \frac {Thrust_{desired}}{Thrust_{available}}$$
Where $Throttle_\%$ is between 0 and 1 (0 - 100%). Our $Thrust_{desired}$ is when we have a thrust to weight ratio of 1, matching gravitational acceleration.
$$
\begin{equation}
TWR = \frac{F}{mg} = 1\
F_{thrust} = mg \
g = \frac{\mu {planet}}{(PLANET:RADIUS)^2} \
Thrust{desired} = SHIP:MASS * g \
\end{equation}
$$
Finally that gives us
$$Throttle_\% = \frac {SHIP:MASS * \frac{\mu {planet}}{(PLANET:RADIUS)^2}}{Thrust{available}}$$
LOCK hover_throttle_level TO MIN(1, MAX(0, SHIP:MASS * g / MAX(0.0001, curr_engine:AVAILABLETHRUST))).

Okay, moment of truth. Let's start small with $Kp = 0.01,\ \ Kd = 0.001$ and set  $altitude_{goal} = 100\ \text{meters}$. Shown with the solid red line across.
Can you write Python code for it?

# To run in kOS console: RUN hover4(hover0.txt,60,0.01,0.001).
data = loadData('collected_data\\hover0.txt')
plotData(data)


I am working on the file "KSP_pid_tuning.ipynb".
The first task is:
It's stably oscillating! This is a good sign, showing our hover setpoint is doing it's job, the proportional gain is there, and there's barely any derivative gain.
Let's bump up the derivative gain to $Kd = 0.01$.
Can you write Python code for it?

# To run in kOS console: RUN hover4(hover1.txt,60,0.01,0.01).
data = loadData('collected_data\\hover1.txt')
plotData(data)


I am working on the file "KSP_pid_tuning.ipynb".
The first task is:
Woohoo! It overshoots a little but stablizes smoothly at 100m! Great to see this going in the game, looks a bit like the SpaceX grasshopper.

The kOS script used is hover4.ks and these tests are run by calling RUN hover4(hoverN.txt,20,Kp,Kd).
Some datasets:

hover0.txt
hover1.txt
hover2.txt
hover3.txt

Tweaking gains
Now let's try and optimize the gains to reduce the rise time.
We take a quick shot by increasing both gains to $Kp = 0.1,\ \ Kd = 0.1$
Can you write Python code for it?

# To run in kOS console: RUN hover4(hover2.txt,10,0.1,0.1).
data = loadData('collected_data\\hover2.txt')
plotData(data)


I am working on the file "K_NEAREST_IMPLEMENTATION.ipynb".
The first task is:
Normalizing the values in the Vacation,eCredit,Salary,Property so that all the values range in between 0 to 1.we use a formula to do  this.we are going to capture min an max for the categories which should be normalized so that we can use them for test data normalization also
Can you write Python code for it?

vacmaxval=0
vacminval=0
ecrmaxval=0
ecrminval=0
salmaxval=0
salminval=0
prpminval=0
prpmaxval=0

for attribute in list(df.columns.values):
    if attribute == 'Vacation' or attribute == 'eCredit' or attribute == 'Salary' or attribute == 'Property':
        if attribute == 'Vacation':
            vacmaxval=max(df[attribute])
            vacminval=min(df[attribute])
        elif attribute == 'eCredit':
            ecrmaxval=max(df[attribute])
            ecrminval=min(df[attribute])
        elif attribute == 'Salary':
            salmaxval=max(df[attribute])
            salminval=min(df[attribute])
        elif attribute == 'Property':
            prpmaxval=max(df[attribute])
            prpminval=min(df[attribute])
        maxValue = max(df[attribute])
        minValue = min(df[attribute])

        norm = []
        for i in df[attribute]:
            normalisedValue = (i - minValue + 0.0)/(maxValue - minValue + 0.0)
            norm.append(normalisedValue)
        df[attribute] = norm

df.head()

# uncomment the below line to get a csv file of the dataframe

#df.to_csv('sanitizedData.csv', sep=',', encoding='utf-8', header=False)


I am working on the file "K_NEAREST_IMPLEMENTATION.ipynb".
The first task is:
Removing the cluster names from the instances and adding them to lists.
Can you write Python code for it?

for i in train_data:
    train_set[i[-1]].append(i[:-1])

for i in test_data:
    test_set[i[-1]].append(i[:-1])

#print test_set

for i in test_datanew:
    test_setnew[i[-1]].append(i[:-1])

for i in train_datanew:
    train_setnew[i[-1]].append(i[:-1])

#print test_set

#print test_setnew


I am working on the file "K_NEAREST_IMPLEMENTATION.ipynb".
The first task is:
Sending our actual testing data to find the accuracy our model
Can you write Python code for it?

correct =0.0
total =0.0
for group in test_setnew:
    for data in test_setnew[group]:
        vote= k_nearest(train_set,data,k=5)
        print '*****'
        print group
        print vote
        #print '*****'
        if group==vote:
            correct+=1
        total +=1
print correct
print total
print ('Accuracy:', (correct)/total)


I am working on the file "K_NEAREST_IMPLEMENTATION.ipynb".
The first task is:
Sending our slpitted test and train from the given train data in order to cross validadate
Can you write Python code for it?

correct=0.0
total=0.0
for group in test_set:
    for data in test_set[group]:
        vote= k_nearest(train_set,data,k=5)
        #print '*****'
        #print group
        #print vote
        #print '*****'
        if group==vote:
            correct+=1
        total +=1
print correct
print total
print ('Accuracy:', correct/total)


I am working on the file "K_NEAREST_IMPLEMENTATION.ipynb".
The first task is:
changing the strings in the test data into to numbers so that they can be used while calculating euclidean distance
Can you write Python code for it?

dft['Type'] = dft.Type.map({'student':1,'engineer':2,'librarian':3,'professor':4,'doctor':5 })
# df.head()
dft['LifeStyle'] = dft.LifeStyle.map({'spend<<saving':1, 'spend<saving':2, 'spend>saving':3, 'spend>>saving':4})

dft['Label'] = dft.Label.map({'C1':1, 'C2':2 ,'C3':3 ,'C4':4 ,'C5':5})

# df['Vacation']=df['Vacation']/100
dft.head()


I am working on the file "K_NEAREST_IMPLEMENTATION.ipynb".
The first task is:
changing the strings in the train data into to numbers so that they can be used while calculating euclidean distance
Can you write Python code for it?

df['Type'] = df.Type.map({'student':1,'engineer':2,'librarian':3,'professor':4,'doctor':5 })
# df.head()
df['LifeStyle'] = df.LifeStyle.map({'spend<<saving':1, 'spend<saving':2, 'spend>saving':3, 'spend>>saving':4})

df['Label'] = df.Label.map({'C1':1, 'C2':2 ,'C3':3 ,'C4':4 ,'C5':5})

# df['Vacation']=df['Vacation']/100
df.head()


I am working on the file "K_NEAREST_IMPLEMENTATION.ipynb".
The first task is:
converting all the data too float into make calculations accurate
Can you write Python code for it?

dataframe= df.astype(float).values.tolist()
dataframet= dft.astype(float).values.tolist()


I am working on the file "K_NEAREST_IMPLEMENTATION.ipynb".
The first task is:
developing our own k nearest algorithm with respective to our requirments, we use similarity matrix to take decision between then two parameters, and the rest we can go with euclidean distance.
Can you write Python code for it?

def k_nearest(data,predict,k=5):
    #if(len(data)>=k):
    #    warnings.warn('bye')
    distances =[]
    for group in data:
        for features in  data[group]:
            euclidean_distance=0
            if(predict[0]!=features[0]):
                euclidean_distance+=1
            if(predict[1]!=features[1]):
                euclidean_distance+=1
            euclidean_distance += ((predict[2]-features[2])**2 + (predict[3]-features[3])**2 +(predict[4]-features[4])**2 + (predict[5]-features[5])**2)
            #euclidean_distance = np.linalg.norm(np.array(features)-np.array(predict))
            euclidean_distance=sqrt(euclidean_distance)
            distances.append([euclidean_distance,group])
    votes = [i[1] for i in sorted(distances) [:k]]
    #print distances
    print votes
    #print (Counter(votes).most_common(1))
    vote_result = Counter(votes).most_common(1)[0][0]
    #print vote_result

    return vote_result


I am working on the file "K_NEAREST_IMPLEMENTATION.ipynb".
The first task is:
reading the testing data into a data frame and assigning headings
Can you write Python code for it?

dft = pd.read_table('test.csv', sep=',', header=None, names=['Type', 'LifeStyle', 'Vacation', 'eCredit', 'Salary', 'Property', 'Label'])
dft.head()


I am working on the file "K_NEAREST_IMPLEMENTATION.ipynb".
The first task is:
reading the training data into a data frame and assigning headings
Can you write Python code for it?

df = pd.read_table('train.csv', sep=',', header=None, names=['Type', 'LifeStyle', 'Vacation', 'eCredit', 'Salary', 'Property', 'Label'])
df.head()


I am working on the file "KaggelChallenges/titanic/.ipynb_checkpoints/explore-checkpoint.ipynb".
The first task is:
Interesting, actually those from C had higher rate of survival.  So, knowing more people from your home town didn't help.
Next, did how much they paid have an effect?
Can you write Python code for it?

train_df['Fare'].astype(int).plot(kind='hist',bins=100, xlim=(0,50))

# get fare for survived & didn't survive passengers
fare_not_survived = train_df["Fare"].astype(int)[train_df["Survived"] == 0]
fare_survived     = train_df["Fare"].astype(int)[train_df["Survived"] == 1]

# get average and std for fare of survived/not survived passengers
avgerage_fare = DataFrame([fare_not_survived.mean(), fare_survived.mean()])
std_fare      = DataFrame([fare_not_survived.std(), fare_survived.std()])

avgerage_fare.index.names = std_fare.index.names = ["Survived"]
avgerage_fare.plot(yerr=std_fare,kind='bar',legend=False)


I am working on the file "KaggelChallenges/titanic/.ipynb_checkpoints/explore-checkpoint.ipynb".
The first task is:
Let's see where they got on
Can you write Python code for it?

train_df['Embarked'].head()

train_df.info()

train_df['Embarked'].isnull().sum()

train_df["Embarked"].count()

sns.countplot(x="Embarked",data=train_df)

sns.countplot(x='Survived',hue='Embarked',data=train_df,order=[0,1])


I am working on the file "KaggelChallenges/titanic/.ipynb_checkpoints/explore-checkpoint.ipynb".
The first task is:
OK, so clearly there were more people who got on at S, and it seems their survival is disproportional.  Let's check that.
Can you write Python code for it?

embark_survive_perc = train_df[["Embarked", "Survived"]].groupby(['Embarked'],as_index=False).mean()
sns.barplot(x='Embarked', y='Survived', data=embark_survive_perc,order=['S','C','Q'])


I am working on the file "KaggelChallenges/titanic/explore.ipynb".
The first task is:
 NEXT: TRY TO PERFORM BOOSTING WITH SKLEARN, NOT XGBoost to see how it changes above results.  THEN TRY TO BUILD SINGLE LAYER NEURAL NETWORKS TO SEE HOW THEY PERFORM, THEN TRY MULTI-LAYER NEURAL NETWORKS.

XGBoost stuff
Can you write Python code for it?

if False:
    submission = pd.DataFrame({
            "PassengerId": test_df["PassengerId"],
            "Survived": Y_pred
        })
    submission.to_csv('submission.csv', index=False)

if False:
    ### Using XGboost
    #X_train = train_df.drop("Survived",axis=1)
    #train_X = train_df.drop("Survived",axis=1).as_matrix()
    X_train_new
    #X_train_new, Y_train, X_test_new
    #train_y = train_df["Survived"]
    Y_train
    #test_X = test_df.drop("PassengerId",axis=1).copy().as_matrix()
    X_test_new
    model = xgb.XGBClassifier(max_depth=10, n_estimators=300, learning_rate=0.05)
    model.fit(X_train_new, Y_train)
    predictions = model.predict(X_test_new)
    # plot feature importance
    plot_importance(model)
    plt.show()
    #X_train, X_test, y_train, y_test = train_test_split(X_train_new, Y_train, test_size=0.33)
    #accuracy = accuracy_score(y_test, predictions)
    #print("Accuracy: %.2f%%" % (accuracy * 100.0))

# basic try at iterative training with XGboost
#train_X = train_df.drop("Survived",axis=1).as_matrix()
#train_y = train_df["Survived"]
#test_X = test_df.drop("PassengerId",axis=1).copy().as_matrix()
# fit model on all training data
acc = []
mx_v = 0
mx_e = 0
ests = range(10,500,10)
if False:
    for est in ests:
        #print est
        model = xgb.XGBClassifier(max_depth=5, n_estimators=ests, learning_rate=0.05)
        X_train, X_test, y_train, y_test = train_test_split(X_train_new, Y_train, test_size=0.33)#, random_state=7)
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        accuracy = accuracy_score(y_test, predictions)
        accuracy *= 100.0
        acc.append(accuracy)
        #print("Accuracy: %.2f%%" % (accuracy))
        if acc[-1]>mx_v:
            mx_v = acc[-1]
            mx_e = est
    print("maxes were: ",(mx_e,mx_v))

    fig = plt.figure(figsize=(7,5))
    subPlot = fig.add_subplot(111)
    subPlot.plot(ests,acc,linewidth=3)

model = xgb.XGBClassifier(max_depth=5, n_estimators=300, learning_rate=0.05)
for i in...


I am working on the file "KaggelChallenges/titanic/explore.ipynb".
The first task is:
Feature Selection
Can you write Python code for it?

clf = ExtraTreesClassifier(n_estimators=200)
clf = clf.fit(X_train, Y_train)
features = pd.DataFrame()
features['feature'] = X_train.columns
features['importance'] = clf.feature_importances_
features.sort(['importance'],ascending=False)


I am working on the file "KaggelChallenges/titanic/explore.ipynb".
The first task is:
Not surprising, woman and children had higher survival rates.
Can you write Python code for it?

# Pclass

# sns.factorplot('Pclass',data=titanic_df,kind='count',order=[1,2,3])
if False:
    sns.factorplot('Pclass','Survived',order=[1,2,3], data=train_df,size=5)

# create dummy variables for Pclass column
pclass_dummies_titanic  = pd.get_dummies(train_df['Pclass'])
pclass_dummies_titanic.columns = ['Class_1','Class_2','Class_3']
#pclass_dummies_titanic.drop(['Class_3'], axis=1, inplace=True)

pclass_dummies_test  = pd.get_dummies(test_df['Pclass'])
pclass_dummies_test.columns = ['Class_1','Class_2','Class_3']
#pclass_dummies_test.drop(['Class_3'], axis=1, inplace=True)

train_df.drop(['Pclass'],axis=1,inplace=True)
test_df.drop(['Pclass'],axis=1,inplace=True)

train_df = train_df.join(pclass_dummies_titanic)
test_df    = test_df.join(pclass_dummies_test)

# Ticket
# Code based on that here: http://ahmedbesbes.com/how-to-score-08134-in-titanic-kaggle-challenge.html
# a function that extracts each prefix of the ticket, returns 'XXX' if no prefix (i.e the ticket is a digit)
def cleanTicket(ticket):
    ticket = ticket.replace('.','')
    ticket = ticket.replace('/','')
    ticket = ticket.split()
    ticket = map(lambda t : t.strip() , ticket)
    ticket = list(filter(lambda t : not t.isdigit(), ticket))
    if len(ticket) > 0:
        return ticket[0]
    else:
        return 'XXX'

train_df['Ticket'] = train_df['Ticket'].map(cleanTicket)
tickets_dummies = pd.get_dummies(train_df['Ticket'],prefix='Ticket')
train_df = pd.concat([train_df, tickets_dummies],axis=1)
train_df.drop('Ticket',inplace=True,axis=1)

test_df['Ticket'] = test_df['Ticket'].map(cleanTicket)
tickets_dummies = pd.get_dummies(test_df['Ticket'],prefix='Ticket')
test_df = pd.concat([test_df, tickets_dummies],axis=1)
test_df.drop('Ticket',inplace=True,axis=1)

train_df.head()

# Title
# a map of more aggregated titles
Title_Dictionary = {
                    "Capt":       "Officer",
                    "Col":        "Officer",
                    "Major":      "Officer",
                    "Jonkheer":   "Royalty",
                   ...


I am working on the file "KaggelChallenges/titanic/explore.ipynb".
The first task is:
OK, so clearly there were more people who got on at S, and it seems their survival is disproportional.  Let's check that.
Can you write Python code for it?

if False:
    embark_survive_perc = train_df[["Embarked", "Survived"]].groupby(['Embarked'],as_index=False).mean()
    sns.barplot(x='Embarked', y='Survived', data=embark_survive_perc,order=['S','C','Q'])


I am working on the file "KaggelChallenges/titanic/explore.ipynb".
The first task is:
The names are also pointless, so drop them too
Can you write Python code for it?

## Scale all features except passengerID
features = list(train_df.columns)
features.remove('PassengerId')
train_df[features] = train_df[features].apply(lambda x: x/x.max(), axis=0)

features = list(test_df.columns)
features.remove('PassengerId')
test_df[features] = test_df[features].apply(lambda x: x/x.max(), axis=0)

train_df.head()

test_df.head()