I am working on the file "Stanford DH Asia Stylometry.ipynb".
The first task is:
Save the information to use later.
info_list = []
for root, dirs, files in os.walk("corpus"):
    for filename in files:
        if filename[0] != ".":
            f = open(root + "/" + filename, "r")
            c = f.read()
            f.close()
            info_list.append(c)
for c in info_list:
    print(len(c))
Cleaning the texts
The texts must first be cleaned before they can do anything. The best way to do this is to write a function and will perform the cleaning. We will then call it when we need our texts to be cleaned. We will remove most unwanted characters with regular expressions. As we will have multiple characters not matching the regex, we will use a loop for the rest.
Can you write Python code for it?

import re

def clean(instring):
    # Remove mid-file markers
    instring = re.sub(r'~~~START\|.+?\|START~~~', "", instring)

    # This regex will remove all letters and numbers
    instring = re.sub(r'[a-zA-Z0-9]', "", instring)

    # A variety of characters to remove
    unwanted_chars = ['』','。', '！', '，', '：', '、', '（',
                      '）', '；', '？', '〉', '〈', '」', '「',
                      '『', '“', '”', '!', '"', '#', '$', '%',
                      '&', "'", '(', ')', '*', '+', ',', '-',
                      '.', '/', "《", "》", "·", "a", "b"]

    for char in unwanted_chars:
        # replace each character with nothing
        instring = instring.replace(char, "")

    # return the resulting string.
    return instring


I am working on the file "Stanford DH Asia Stylometry.ipynb".
The first task is:
Term Frequency and Term Frequency - Inverse Document Frequency
sci-kit learn also has a term frequency and tfidf vectorizer that you can use, depending on how you want to think about your texts.
Can you write Python code for it?

# Just use this instead when creating your vectorizer.
# To get TF, tell it to not use idf. otherwise, set to true
vectorizer = TfidfVectorizer(use_idf=False, analyzer="char",
                            ngram_range=(1,1), max_features=10)


I am working on the file "Stanford DH Asia Stylometry.ipynb".
The first task is:
This are not in order of the most common character. We can use a library called pandas to asscertain this easily.
Can you write Python code for it?

import pandas as pd
from pandas import Series

fullcorpus = ""
for text in info_list:
    fullcorpus += text

tokens = list(fullcorpus)

corpus_series = Series(tokens)

values = corpus_series.value_counts()

print(values[:10])


I am working on the file "Stanford DH Asia Stylometry.ipynb".
The first task is:
This can be made a bit more attractive
Can you write Python code for it?

dendrogram(linkage_matrix, labels=title_list)

# Add a Title
plt.title("Textual Relationships")

# Add x and y axis labels
plt.xlabel("Texts")
plt.ylabel("Distance")

# Set the angle of the labels so they are easier to read
plt.xticks(rotation=60)

# Show the plot
plt.show()


I am working on the file "Stanford DH Asia Stylometry.ipynb".
The first task is:
Tokenizing the text with sci-kit learn vectorizer
When you use sci-kit learn, you can give it a lot of options.
If you have a file with whitespace between items:
You can use whitespace to tokenize your documents. For example, if you have used the Stanford Word Parser, then you can use a vectorizer set up like this to take advantage of this:
Can you write Python code for it?

vectorizer = CountVectorizer(analyzer="word", ngram_range=(1,1),
                            token_pattern="\S+", max_features = 100)


I am working on the file "Stanford DH Asia Stylometry.ipynb".
The first task is:
Use the Metadata to store information about each file
I usually store the information in parallel lists. This way it is easy for the analysis part of the software to attach different labels.
The following code applies to using whole files (rather than breaking them apart).
Can you write Python code for it?

# Create empty lists to store the information
info_list = []
title_list = []
author_list = []
era_list = []
genre_list = []

# Create dictionaries store unique info:
title_author = {}
title_era = {}
title_genre = {}

for root, dirs, files in os.walk("corpus"):
    for filename in files:
        if filename[0] != ".":
            f = open(root + "/" + filename, "r", encoding="utf8")
            c = f.read()
            f.close()
            c = re.sub("\s+", "", c)
            c = clean(c)
            # Get metadata. the [:-4] removes the .txt from filename
            metainfo = metadata[filename[:-4]]

            info_list.append(c)
            title_list.append(metainfo[0])
            author_list.append(metainfo[1])
            era_list.append(metainfo[2])
            genre_list.append(metainfo[3])

            title_author[metainfo[0]] = metainfo[1]
            title_era[metainfo[0]] = metainfo[2]
            title_genre[metainfo[0]] = metainfo[3]

print(title_list)


I am working on the file "Stanford DH Asia Stylometry.ipynb".
The first task is:
Using Vocabulary
If you want to, you can give the vectorizer a set vocabulary to pay attention to, rather than just using the most common characters. This comes in handy when you have an idea which characters distinguish the texts most efficiently.
Can you write Python code for it?

my_vocab = ["的", "之", "曰", "说"]

vectorizer = CountVectorizer(analyzer="char",ngram_range=(1,1),
                             vocabulary = my_vocab)


I am working on the file "Stanford DH Asia Stylometry.ipynb".
The first task is:
Using this information
This function returns everything we will need to properly visualize our Principal component analysis.
Call the function and use the results.
Can you write Python code for it?

unique_labels, info_labels, unique_genres = info_for_graph(genre_list)

# Make a color list, the same length as unique labels
colors = ["red", "magenta", "blue"]

# Make the figure
plt.figure()

# Plot the points using color information.
# This code is partially adapted from brandonrose.org/clustering
for color, each_class, label in zip(colors, unique_labels, unique_genres):
    plt.scatter(my_pca[info_labels == each_class, 0],
               my_pca[info_labels == each_class, 1],
               label = label, color = color)

# You should title the plot label your axes
plt.title("Principal Component Analysis")
plt.xlabel("PC1: " + "{0:.2f}".format(pca.explained_variance_ratio_[0] * 100)+"%")
plt.ylabel("PC2: " + "{0:.2f}".format(pca.explained_variance_ratio_[1] * 100)+"%")

# Give it a legend
plt.legend()

plt.show()


I am working on the file "Stanford DH Asia Stylometry.ipynb".
The first task is:
You now have a similarity matrix
The similarity variable now contains the similarity measure between each document in the corpus. You can use this to create a linkage matrix which will allow you to visualize the relationships among these texts as a dendrogram.
Here we will use the "Ward" algorithm to cluster the texts together.
Can you write Python code for it?

from scipy.cluster.hierarchy import ward, dendrogram
linkage_matrix = ward(similarity)


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Change the second and third columns.
Can you write Python code for it?

star_wars['Do you consider yourself to be a fan of the Star Wars film franchise?'].isnull().value_counts()

star_wars['Have you seen any of the 6 films in the Star Wars franchise?'].value_counts()


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Cleaning the columns from index 3 to 9.
From the fourth column to ninth columns are checkbox questions:
 If values are the movie names: they have seen the movies.
 If values are NaN: they have not seen the movies.
We are going to convert the values of these columns to bool type.
Can you write Python code for it?

for col in star_wars.columns[3:9]:
    star_wars[col] = star_wars[col].apply(lambda x: False if pd.isnull(x) else True)


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Cleaning the columns from index 9 to 15.
Changing data type to float.
Can you write Python code for it?


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Convert fav to pivot table.
Can you write Python code for it?

fav_df_list = []
for col in fav.columns.tolist():
    row = fav[col].value_counts()
    d1 = pd.DataFrame(data={'favorably': row[0] + row[1], \
                   'neutral': row[2], \
                   'unfavorably': row[4] + row[5], \
                   'Unfamiliar': row[3]}, \
                  index=[col], \
                  columns=['favorably', 'neutral', 'unfavorably', 'Unfamiliar'])
    fav_df_list.append(d1)

fav_pivot = pd.concat(fav_df_list)

fav_pivot

fig = plt.figure()
ax = plt.subplot(111)

fav_pivot.plot(kind='barh', stacked=True, figsize=(10,10), ax=ax)

# Shrink current axis's height by 10% on the bottom
box = ax.get_position()
ax.set_position([box.x0, box.y0 + box.height * 0.1,
                 box.width, box.height * 0.9])

# Put a legend below current axis
ax.legend(loc='upper center', bbox_to_anchor=(0.5, -0.05),
          fancybox=True, shadow=True, ncol=5)

plt.show()


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Data Analysis
Finding The Most Seen Movie
Can you write Python code for it?

seen_sum = star_wars[['seen_1', 'seen_2', 'seen_3', 'seen_4', 'seen_5', 'seen_6']].sum()

seen_sum

seen_sum.idxmax()


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Data Cleaning
Remove invalid first column RespondentID which are NaN.
Can you write Python code for it?


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Finding The Highest Ranked Movie.
Can you write Python code for it?

ranking_mean = star_wars[['ranking_1', 'ranking_2', 'ranking_3', 'ranking_4', 'ranking_5', 'ranking_6']].mean()

ranking_mean

ranking_mean.idxmin()


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
From the data above, we can find that the most seen movie is the episode V.
Can you write Python code for it?

ax = seen_sum.plot(kind='bar')
for p in ax.patches:
    ax.annotate(str(p.get_height()), (p.get_x() * 1.005, p.get_height() * 1.01))
plt.show()


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Let's break down data by Gender.
Can you write Python code for it?

males = star_wars[star_wars['Gender'] == 'Male']
females = star_wars[star_wars['Gender'] == 'Female']


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Rename the columns from index 3 to 9 for better readibility.
seen_1 means Star Wars Episode I, and so on.
Can you write Python code for it?

star_wars.rename(columns={'Which of the following Star Wars films have you seen? Please select all that apply.': 'seen_1', \
                          'Unnamed: 4': 'seen_2', \
                          'Unnamed: 5': 'seen_3', \
                          'Unnamed: 6': 'seen_4', \
                          'Unnamed: 7': 'seen_5', \
                          'Unnamed: 8': 'seen_6'}, inplace=True)


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Renaming columns names.
Can you write Python code for it?

star_wars.rename(columns={'Please rank the Star Wars films in order of preference with 1 being your favorite film in the franchise and 6 being your least favorite film.': 'ranking_1', \
                          'Unnamed: 10': 'ranking_2', \
                          'Unnamed: 11': 'ranking_3', \
                          'Unnamed: 12': 'ranking_4', \
                          'Unnamed: 13': 'ranking_5', \
                          'Unnamed: 14': 'ranking_6'}, inplace=True)


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
The highest ranked movie is ranking_5 which is the episode V.
Can you write Python code for it?


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
The number of movies seen.
Can you write Python code for it?

males[males.columns[3:9]].sum().plot(kind='bar', title='male seen')
plt.show()

males[females.columns[3:9]].sum().plot(kind='bar', title='female seen')
plt.show()


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
The ranking of movies.
Can you write Python code for it?

males[males.columns[9:15]].mean().plot(kind='bar', title='Male Ranking')
plt.show()

females[males.columns[9:15]].mean().plot(kind='bar', title='Female Ranking')
plt.show()


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
The values for the second and third columns which are Have you seen any of the 6 films in the Star Wars franchise? and Do you consider yourself to be a fan of the Star Wars film franchise? respectively are Yes, No, NaN. We want to change them to True or False.
Can you write Python code for it?

star_wars['Have you seen any of the 6 films in the Star Wars franchise?'] = star_wars['Have you seen any of the 6 films in the Star Wars franchise?'].map({'Yes': True, 'No': False})

star_wars['Do you consider yourself to be a fan of the Star Wars film franchise?'] = star_wars['Do you consider yourself to be a fan of the Star Wars film franchise?'].map({'Yes': True, 'No': False})


I am working on the file "Star Wars survey/Star Wars survey.ipynb".
The first task is:
Who Shot First?
Can you write Python code for it?

shot_first = star_wars['Which character shot first?'].value_counts()

shot_first

shot_sum = shot_first.sum()

shot_first = shot_first.apply(lambda x: x / shot_sum * 100)

shot_first

ax = shot_first.plot(kind='barh')
for p in ax.patches:
    ax.annotate(str("{0:.2f}%".format(round(p.get_width(),2))), (p.get_width() * 1.005, p.get_y() + p.get_height() * 0.5))
plt.show()


I am working on the file "StarTrek.ipynb".
The first task is:
Build a Spiral Bubble Plot
The concept for this chart is borrowed from http://thestartrekproject.net/files/Star_Trek/ch4/miscellanea-chapter-mockup%2012.pdf
Can you write Python code for it?

from bokeh.plotting import figure, output_notebook, show, ColumnDataSource
from bokeh.models import HoverTool

output_notebook()

# spiral parameters
a = 0.45
b = 0.15

# bubble size and spacing
spacing = 0.01
size=np.log10(1.0+totalDeaths['TotalBodyCount'])

# convert bubble size and spacing to arclengths
arclength = np.cumsum(2*size+spacing)

# solve for polar angle using spiral arclength equation
theta = np.log(b*arclength/(a*np.sqrt(1+np.power(b,2))))/b

# solve for polar radius using logrithmic spiral equation
r = a*np.exp(b*theta)

# cartesian
x=r * np.cos(theta)
y=r * np.sin(theta)

# build column data source for bokeh
source = ColumnDataSource(
        data=dict(
            x=x,
            y=y,
            bodyCount=totalDeaths['TotalBodyCount'],
            size=size,
            color=["#%02x%02x%02x" % (int(red), int(green), 150) for red, green in zip(np.floor(100+2*x), np.floor(30+2*y))],
            desc=totalDeaths['DeathBy'].tolist(),
        )
    )

# setup hover tool for contextual labels
hover = HoverTool(
        tooltips=[
            ("Body Count", "@bodyCount"),
            ("Desc", "@desc"),
        ]
    )

# create the figure
p = figure(plot_width=800, plot_height=800, tools=[hover],
           title="Death By")

# create the bubble scatter plot
p.scatter('x', 'y', radius='size', fill_color='color',
          source=source, fill_alpha=0.8, line_color=None)

# display the figure
show(p)


I am working on the file "Stars_Turner.ipynb".
The first task is:
Part 1 Questions
4. The graph you produced shows right ascension on the x-axis and declination on the y-axis. What does the resulting graph actually show?
5. Your code also produced a chart- how are the chart and graph similar to each other? How are they different?
6. Are there any stars you can identify on both of them? What are their names? What aspect of the graph allowed you to do that? Is there

PART 2: Relationships Between Two Properties
Can you write Python code for it?

# format the points on the graph
transparency = 1
size = 1

# draws a scatter plot
fig = plt.figure(figsize=(20, 4.5))
plt.scatter(data.temp, data.lum, s=size, edgecolors='none', alpha=transparency)
plt.xlim(2000,15000)
plt.ylim(0,1000)
plt.title("Does hotter mean brighter?")
plt.ylabel("Luminosity")
plt.xlabel("Temperature (K)")


I am working on the file "Stars_Turner.ipynb".
The first task is:
Part 2 Questions
7. Each point on the scatter plot shows a star’s temperature and luminosity. What trends do you see in the data?
8. Around what temperature is the largest range of luminosities? Using the chart below the graph you produced determine the most likely color? If a star is blue, what temperature can you infer it is?

PART 3: Constellations and Star Properties
Can you write Python code for it?

# These are the abbreviations for all the constellations
data.sort_values('con').con.unique()


I am working on the file "Stars_Turner.ipynb".
The first task is:
Choose a Constellation from the list above and insert the 3 letter code below in the " ", for example "Vir".
Can you write Python code for it?

# This shows just one constellation
data_con = data.query('con == "Vir"')

# This plots where the brightest 10 stars are in the sky
data_con = data_con.sort_values('mag').head(10)
plt.scatter(data_con.ra,data_con.dec)
plt.gca().invert_xaxis()
plt.title("A constellation in the sky")
plt.xlabel('Right Ascension (degrees)')
plt.ylabel('Declination (Hours)')


I am working on the file "Stars_Turner.ipynb".
The first task is:
PART 1: All the Stars in Our Catalogue
Declination is the distance a star is North or South of the Celestial Equator, similar to lattitude on Earth. Right Ascension is how far east or west a star is, similar to longitude on Earth.
Can you write Python code for it?

fig = plt.figure(figsize=(15, 6))
plt.scatter(data.ra,data.dec, s=0.01)
plt.xlim(24, 0)
plt.title("All the Stars in the Catalogue")
plt.xlabel('Right Ascension (Hours)')
plt.ylabel('Declination (Degrees)')


I am working on the file "Stars_Turner.ipynb".
The first task is:
Run the Code below to create a chart containing all of the Stars with Exoplantes within 100 light years!
Can you write Python code for it?

from IPython.display import Image
from IPython.core.display import HTML

Image(url= 'http://www.hpcf.upr.edu/~abel/phl/nearby_stars_with_exoplanets.png')


I am working on the file "Stars_Turner.ipynb".
The first task is:
Run the code below to display a chart of Star color and temperatures.
Can you write Python code for it?


I am working on the file "State Overlay Tests.ipynb".
The first task is:
Tests overlaying inferred states on top of rule 18 spacetime diagram
Can you write Python code for it?

rule_18 = ECA(18, domain_18(500))
rule_18.evolve(500)

states_18 = epsilon_field(rule_18.get_spacetime())
states_18.estimate_states(3,3,1,alpha=0)
states_18.filter_data()

overlay_test(rule_18.get_spacetime(), states_18.get_causal_field(), t_min=200, t_max=240, x_min=200, x_max=240)

overlay_test(rule_18.get_spacetime(), states_18.get_causal_field(), t_min=200, t_max=220, x_min=200, x_max=220)

print states_18.get_causal_field()[200:220, 200:220]


I am working on the file "State Overlay Tests.ipynb".
The first task is:
Tests overlaying rule 18 values on top of spacetime diagram
Can you write Python code for it?

overlay_test(rule_18.get_spacetime(),rule_18.get_spacetime(),t_max=20, x_max=20, text_color='red')

overlay_test(rule_18.get_spacetime(),rule_18.get_spacetime(),t_max=20, x_max=20, colors=plt.cm.Set2, text_color='black')

overlay_test(rule_18.get_spacetime(),rule_18.get_spacetime(),t_max=20, x_max=20, colorbar=True)


I am working on the file "StateClustering.ipynb".
The first task is:
Load data
Here we consider lin2 data for gor01 on the first recording day (6-7-2006), since this session had the most units (91) of all the gor01 sessions, and lin2 has position data, whereas lin1 only has partial position data.
Can you write Python code for it?

datadirs = ['/home/etienne/Dropbox/neoReader/Data',
            'C:/etienne/Dropbox/neoReader/Data',
            '/Users/etienne/Dropbox/neoReader/Data']

fileroot = next( (dir for dir in datadirs if os.path.isdir(dir)), None)

animal = 'gor01'; month,day = (6,7); session = '16-40-19' # 91 units

spikes = load_data(fileroot=fileroot, datatype='spikes',animal=animal, session=session, month=month, day=day, fs=32552, verbose=False)
eeg = load_data(fileroot=fileroot, datatype='eeg', animal=animal, session=session, month=month, day=day,channels=[0,1,2], fs=1252, starttime=0, verbose=False)
posdf = load_data(fileroot=fileroot, datatype='pos',animal=animal, session=session, month=month, day=day, verbose=False)
speed = klab.get_smooth_speed(posdf,fs=60,th=8,cutoff=0.5,showfig=False,verbose=False)


I am working on the file "StateClustering.ipynb".
The first task is:
Part 2: Categorical state clustering
What questions am I trying to answer?

Can we distinguish (using log probability alone) RUN from NORUN point observations?
Can we distinguish (using log probability alone) RUN from NORUN sequence observations?
Can we distinguish (using log probability alone) FIRST from SECOND using point observations?
Can we distinguish (using log probability alone) FIRST from SECOND using sequence observations?
Can we see differences in state distributions between FIRST and SECOND, or RUN and NORUN? Of course, we expect that we must... is this really a useful question to ask? What conclusion will we be able to draw from this?

Bigger questions (above questions are not particularly useful):
* What is the underlying state of the animal? RUN, NORUN, SWR (MUA), REPLAY, REWARD, ...?
* What is the underlying context of the neural activity? Which environment/scenario?
*
Some other things
* train model at certain speed (timescale), evaluate data at different timescales in model—hopefully we can infer the proper timescale. Extend this to replay<--->behavioral situation
Note to self: If we have many states, then state clustering might work better for clustering states by first and second half. With fewer states, many of those (esp the center track states) will be shared.
train on ALL data with many states, and look at decoded state distribution for first and second half; as well as quiescent vs run
Can you write Python code for it?

## bin ALL spikes
ds = 0.0625 # bin spikes into 62.5 ms bins (theta-cycle inspired)
num_states = 65

binned_spikes = klab.bin_spikes(spikes.data, ds=ds, fs=spikes.samprate, verbose=True)

centerx = (np.array(posdf['x1']) + np.array(posdf['x2']))/2
centery = (np.array(posdf['y1']) + np.array(posdf['y2']))/2

tend = len(speed.data)/speed.samprate # end in seconds
time_axis = np.arange(0,len(speed.data))/speed.samprate
speed_0625, tvel_0625 = klab.resample_velocity(velocity=speed.data,t_bin=ds,tvel=time_axis,t0=0,tend=tend)
truepos_0625 = np.interp(np.arange(0,len(binned_spikes.data))*ds,time_axis,centerx)

# get bins where rat was running faster than thresh units per second
runidx_0625 = np.where(speed_0625>8)[0]
seq_stk_run_0625 = extract_subsequences_from_binned_spikes(binned_spikes,runidx_0625)

## split data into train, test, and validation sets:
tr_b,vl_b,ts_b = sq.data_split(seq_stk_run_0625, tr=50, vl=2, ts=50, randomseed = 0, verbose=True)

## train HMM on active behavioral data; training set (with a fixed, arbitrary number of states for now):
myhmm = sq.hmm_train(tr_b, num_states=num_states, n_iter=50, verbose=False)

###########################################################3
stacked_data = seq_stk_run_0625
###########################################################3

x0=0; xl=100; num_pos_bins=50
xx_left = np.linspace(x0,xl,num_pos_bins+1)
num_sequences = len(stacked_data.sequence_lengths)
num_states = myhmm.n_components
state_pos = np.zeros((num_states, num_pos_bins))
state_distr_1 = np.zeros((num_states,1))
state_distr_2 = np.zeros((num_states,1))

for seq_id in np.arange(0,num_sequences):
    tmpseqbdries = [0]; tmpseqbdries.extend(np.cumsum(stacked_data.sequence_lengths).tolist());
    obs = stacked_data.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
    ll, pp = myhmm.score_samples(obs)
    if stacked_data.boundaries[seq_id][0] < len(speed_0625)/2:
        #print('1st half')
        state_distr_1[:,0] = state_distr_1[:,0] + pp.sum(axis=0)
    else:
        #print('2nd half')
       ...


I am working on the file "StateClustering.ipynb".
The first task is:
Positional state clustering 5: RUN ($> 8$  u$\cdot$s$^{-1}$) in NORUN model @ 62.5 ms bins; $N_s$ = 50 spatial bins; MAP
Here I evaluate RUN > 8 data in the NORUN < 1 model.
Can you write Python code for it?

###########################################################3
stacked_data = seq_stk_run_0625
###########################################################3

x0=0; xl=100; num_pos_bins=50
xx_left = np.linspace(x0,xl,num_pos_bins+1)
num_sequences = len(stacked_data.sequence_lengths)
num_states = myhmmq.n_components
state_pos = np.zeros((num_states, num_pos_bins))

for seq_id in np.arange(0,num_sequences):
    tmpseqbdries = [0]; tmpseqbdries.extend(np.cumsum(stacked_data.sequence_lengths).tolist());
    obs = stacked_data.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
    ll, pp = myhmmq.score_samples(obs)
    xx = truepos_0625[stacked_data.boundaries[seq_id,0]:stacked_data.boundaries[seq_id,1]+1]
    digitized = np.digitize(xx, xx_left) - 1 # spatial bin numbers
    for ii, ppii in enumerate(pp):
        state_pos[:,digitized[ii]] += np.transpose(ppii)

## now order states by peak location on track
peaklocations = state_pos.argmax(axis=1)
peakorder = peaklocations.argsort()

fig, ax1 = plt.subplots(1, 1, figsize=(6, 4))
ax1.matshow(state_pos[peakorder,:], interpolation='none', cmap='OrRd')
ax1.set_xlabel('position bin')
ax1.set_ylabel('state')
ax1.set_title('positional state clustering after sorting; RUN > 8 in NORUN < 1')


I am working on the file "StateClustering.ipynb".
The first task is:
Positional state clustering 6: ALL data @ 62.5 ms bins; $N_s$ = 50 spatial bins; MAP
What if we don't know when the animal is running, and when it is resting? What if we just inject ALL of the data into the model to learn some underlying states? In this case, we should of course expect to learn states with place field like structures, but also very robust encoding of the reward sites. I expect that the model will do fine, but one potential problem is the fact that the RUN data constitutes a much smaller part of the data than the NORUN data, so that the quiescent color levels might mask the place field activity. I could be more careful in normalizing, or I could use a logarithmic color scale to reveal multi-level structures. Let's see what we get first:
Can you write Python code for it?

# get bins where rat was running slower than thresh units per second
allidx_0625 = np.where(speed_0625>=0)[0]
seq_stk_all_0625 = extract_subsequences_from_binned_spikes(binned_spikes,allidx_0625)

## train HMM on active behavioral data; training set (with a fixed, arbitrary number of states for now):
myhmmall = sq.hmm_train(seq_stk_all_0625, num_states=35, n_iter=50, verbose=False)

###########################################################3
stacked_data = seq_stk_all_0625
###########################################################3

x0=0; xl=100; num_pos_bins=50
xx_left = np.linspace(x0,xl,num_pos_bins+1)
num_sequences = len(stacked_data.sequence_lengths)
num_states = myhmmall.n_components
state_pos = np.zeros((num_states, num_pos_bins))

for seq_id in np.arange(0,num_sequences):
    tmpseqbdries = [0]; tmpseqbdries.extend(np.cumsum(stacked_data.sequence_lengths).tolist());
    obs = stacked_data.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
    ll, pp = myhmmall.score_samples(obs)
    xx = truepos_0625[stacked_data.boundaries[seq_id,0]:stacked_data.boundaries[seq_id,1]+1]
    digitized = np.digitize(xx, xx_left) - 1 # spatial bin numbers
    for ii, ppii in enumerate(pp):
        state_pos[:,digitized[ii]] += np.transpose(ppii)

def normalize_state_pos(state_pos):
    num_states = state_pos.shape[0]
    num_pos_bins = state_pos.shape[1]
    state_pos = state_pos / np.tile(np.reshape(state_pos.sum(axis=1),(num_states,1)),num_pos_bins)
    return state_pos

## now order states by peak location on track
state_pos = normalize_state_pos(state_pos)
peaklocations = state_pos.argmax(axis=1)
peakorder = peaklocations.argsort()

fig, ax1 = plt.subplots(1, 1, figsize=(6, 4))
im = ax1.matshow(state_pos[peakorder,:], interpolation='none', cmap='OrRd')
ax1.set_xlabel('position bin')
ax1.set_ylabel('state')
ax1.set_title('positional state clustering after sorting; ALL data')
cax = fig.add_axes([0.9, 0.1, 0.03, 0.8])
fig.colorbar(im, cax=cax)


I am working on the file "StateClustering.ipynb".
The first task is:
Remarks: Indeed, we again see a robust representation/encoding of the reward sites, and little else. What about stationary data decoded in the RUN model? And what about RUN data in the stationary model? Hopefully, both will reveal no significant place field structure outside of the reward sites, but let's confirm this next.
Positional state clustering 4: NORUN ($< 1$  u$\cdot$s$^{-1}$) in RUN model @ 62.5 ms bins; $N_s$ = 50 spatial bins; MAP
Here I evaluate NORUN < 1 data in the RUN > 8 model.
Can you write Python code for it?

###########################################################3
stacked_data = seq_stk_norun_0625
###########################################################3

x0=0; xl=100; num_pos_bins=50
xx_left = np.linspace(x0,xl,num_pos_bins+1)
num_sequences = len(stacked_data.sequence_lengths)
num_states = myhmm.n_components
state_pos = np.zeros((num_states, num_pos_bins))

for seq_id in np.arange(0,num_sequences):
    tmpseqbdries = [0]; tmpseqbdries.extend(np.cumsum(stacked_data.sequence_lengths).tolist());
    obs = stacked_data.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
    ll, pp = myhmm.score_samples(obs)
    xx = truepos_0625[stacked_data.boundaries[seq_id,0]:stacked_data.boundaries[seq_id,1]+1]
    digitized = np.digitize(xx, xx_left) - 1 # spatial bin numbers
    for ii, ppii in enumerate(pp):
        state_pos[:,digitized[ii]] += np.transpose(ppii)

## now order states by peak location on track
peaklocations = state_pos.argmax(axis=1)
peakorder = peaklocations.argsort()

fig, ax1 = plt.subplots(1, 1, figsize=(6, 4))
ax1.matshow(state_pos[peakorder,:], interpolation='none', cmap='OrRd')
ax1.set_xlabel('position bin')
ax1.set_ylabel('state')
ax1.set_title('positional state clustering after sorting; NORUN < 1 in RUN')


I am working on the file "StateClustering.ipynb".
The first task is:
Remarks: Indeed, we found a linear increase in the number of states representing the positions along the track. The proportion of states encoding this part of the behavior (running along the track) is also roughly consistent with the proportion of data: The animal ran faster than th = 8.0 units/s for a total of 237.7 seconds (out of a total of 2587.8 seconds) That is around 10 % of the time, but quiescent data are also expected to be more homogeneous, so that fewer states have to be recruited to capture the underlying dynamics.
Can you write Python code for it?

## now order states by peak location on track
state_pos = normalize_state_pos(state_pos)
peaklocations = state_pos.argmax(axis=1)
peakorder = peaklocations.argsort()

fig, ax1 = plt.subplots(1, 1, figsize=(6, 4))
ax1.matshow(state_pos[peakorder[6:17],:], interpolation='none', cmap='OrRd')
ax1.set_xlabel('position bin')
ax1.set_ylabel('state')
ax1.set_title('positional state clustering after sorting; ALL data', y=1.18)


I am working on the file "StateClustering.ipynb".
The first task is:
Remarks: Notice that the majority of the states have been recruited to encode the reward locations, with only a small number of states (6 to 11) representing other positions along the track. This is somewhat disappointing, but ultimately expected, since the majority of the data were spent at the reward locations, and comparatively very little data were collected when running along the track.
Can we possibly make the track representation better by considering more states?
Positional state clustering 7: RUN data in an ALL model @ 62.5 ms bins; $N_s$ = 50 spatial bins; MAP
Here we see what we can learn by using a model trained on ALL the data (mostly quiescent data, and a little bit of RUN data), so that most of the states are tuned to encode the reward locations... but this time, we look at the place fields by only evaluating RUN sequences in the ALL-trained model.
Can you write Python code for it?

###########################################################3
stacked_data = seq_stk_run_0625
###########################################################3

x0=0; xl=100; num_pos_bins=50
xx_left = np.linspace(x0,xl,num_pos_bins+1)
num_sequences = len(stacked_data.sequence_lengths)
num_states = myhmmall.n_components
state_pos = np.zeros((num_states, num_pos_bins))

for seq_id in np.arange(0,num_sequences):
    tmpseqbdries = [0]; tmpseqbdries.extend(np.cumsum(stacked_data.sequence_lengths).tolist());
    obs = stacked_data.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
    ll, pp = myhmmall.score_samples(obs)
    xx = truepos_0625[stacked_data.boundaries[seq_id,0]:stacked_data.boundaries[seq_id,1]+1]
    digitized = np.digitize(xx, xx_left) - 1 # spatial bin numbers
    for ii, ppii in enumerate(pp):
        state_pos[:,digitized[ii]] += np.transpose(ppii)

state_pos = normalize_state_pos(state_pos)
#peaklocations = state_pos.argmax(axis=1)
#peakorder = peaklocations.argsort()

fig, ax1 = plt.subplots(1, 1, figsize=(6, 4))
ax1.matshow(state_pos[peakorder,:], interpolation='none', cmap='OrRd')
ax1.set_xlabel('position bin')
ax1.set_ylabel('state')
ax1.set_title('positional state clustering after sorting; RUN > 8 in ALL; same sorting as ALL in ALL')


I am working on the file "StateClustering.ipynb".
The first task is:
Remarks: Perhaps unsurprisingly this time, we see results very similar to the NORUN results presented earlier. Indeed, the ALL model consists of mostly NORUN data, so evaluating the NORUN data in the ALL model should be very similar to evaluating NORUN data in a NORUN model. No surprises here, and things are as expected.
Discussion: In conclusion I think it is safe to say that the HMM effectively learned behaviorally relevant states, especially when presented with RUN data. However, even when given ALL the data, the model somehow still captured state representations for both (i) the reward locations, as well as (positions along the track).
Next, we see if we can represent the track better by using more model states.
Positional state clustering 9: ALL data @ 62.5 ms bins; $N_s$ = 50 spatial bins; MAP; 55 states
Previously we saw that around 5–6 states were recruited to represent positions along the track when we used $m=35$ model states and all the data. If we increase the model states to $m=55$, what should we expect? Perhaps we should expect a linear increase in the number of states representing the positions along the track, so that we should expect 10–11 states to represent positions between the reward locations.
Can you write Python code for it?

# get bins where rat was running slower than thresh units per second
allidx_0625 = np.where(speed_0625>=0)[0]
seq_stk_all_0625 = extract_subsequences_from_binned_spikes(binned_spikes,allidx_0625)

## train HMM on active behavioral data; training set (with a fixed, arbitrary number of states for now):
myhmmall = sq.hmm_train(seq_stk_all_0625, num_states=55, n_iter=50, verbose=False)

###########################################################3
stacked_data = seq_stk_all_0625
###########################################################3

x0=0; xl=100; num_pos_bins=50
xx_left = np.linspace(x0,xl,num_pos_bins+1)
num_sequences = len(stacked_data.sequence_lengths)
num_states = myhmmall.n_components
state_pos = np.zeros((num_states, num_pos_bins))

for seq_id in np.arange(0,num_sequences):
    tmpseqbdries = [0]; tmpseqbdries.extend(np.cumsum(stacked_data.sequence_lengths).tolist());
    obs = stacked_data.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
    ll, pp = myhmmall.score_samples(obs)
    xx = truepos_0625[stacked_data.boundaries[seq_id,0]:stacked_data.boundaries[seq_id,1]+1]
    digitized = np.digitize(xx, xx_left) - 1 # spatial bin numbers
    for ii, ppii in enumerate(pp):
        state_pos[:,digitized[ii]] += np.transpose(ppii)

## now order states by peak location on track
state_pos = normalize_state_pos(state_pos)
peaklocations = state_pos.argmax(axis=1)
peakorder = peaklocations.argsort()

fig, ax1 = plt.subplots(1, 1, figsize=(6, 4))
ax1.matshow(state_pos[peakorder,:], interpolation='none', cmap='OrRd')
ax1.set_xlabel('position bin')
ax1.set_ylabel('state')
ax1.set_title('positional state clustering after sorting; ALL data')


I am working on the file "StateClustering.ipynb".
The first task is:
Remarks: Somewhat surprisingly, we find that a wide range of states now encode the position along the track. Recall that we did not update the ALL model in any way. This is the SAME model as what was used for the ALL in ALL result above, where most of the states encoded reward locations, and only states 6 through 11 seemed to cover the remaining positions on the track. How do we reconcile these two different views of the underlying model? I have to think about it a little more carefully...
Here we have intentionally not resorted the states, so that we can compare it directly to the ALL in ALL result above. However, if we re-sort the states according to their peak locations along the track, we can better see the almost uniform coverage along the track that was nonetheless learned from the data:
Can you write Python code for it?

## now order states by peak location on track
state_pos = normalize_state_pos(state_pos)
peaklocations = state_pos.argmax(axis=1)
peakorder = peaklocations.argsort()

fig, ax1 = plt.subplots(1, 1, figsize=(6, 4))
ax1.matshow(state_pos[peakorder,:], interpolation='none', cmap='OrRd')
ax1.set_xlabel('position bin')
ax1.set_ylabel('state')
ax1.set_title('positional state clustering after sorting; RUN > 8 in ALL; re-sorted')


I am working on the file "StateClustering.ipynb".
The first task is:
Remarks: We can see that very few place field like patterns were found, but the reward locations are robustly encoded, both before and after shortening the track.
Positional state clustering 3: NORUN data only ($< 1$  u$\cdot$s$^{-1}$) @ 62.5 ms bins; $N_s$ = 50 spatial bins; MAP
Here I train and evaluate using NORUN < 1 data. We should see something very similar to when we were using NORUN < 4 data.
Can you write Python code for it?

# get bins where rat was running slower than thresh units per second
norunidx_0625 = np.where(speed_0625<1)[0]
seq_stk_norun_0625 = extract_subsequences_from_binned_spikes(binned_spikes,norunidx_0625)

## split data into train, test, and validation sets:
tr_q,vl_q,ts_q = sq.data_split(seq_stk_norun_0625, tr=50, vl=2, ts=50, randomseed = 0, verbose=True)

## train HMM on active behavioral data; training set (with a fixed, arbitrary number of states for now):
myhmmq = sq.hmm_train(tr_q, num_states=35, n_iter=50, verbose=False)

###########################################################3
stacked_data = seq_stk_norun_0625
###########################################################3

x0=0; xl=100; num_pos_bins=50
xx_left = np.linspace(x0,xl,num_pos_bins+1)
num_sequences = len(stacked_data.sequence_lengths)
num_states = myhmmq.n_components
state_pos = np.zeros((num_states, num_pos_bins))

for seq_id in np.arange(0,num_sequences):
    tmpseqbdries = [0]; tmpseqbdries.extend(np.cumsum(stacked_data.sequence_lengths).tolist());
    obs = stacked_data.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
    ll, pp = myhmmq.score_samples(obs)
    xx = truepos_0625[stacked_data.boundaries[seq_id,0]:stacked_data.boundaries[seq_id,1]+1]
    digitized = np.digitize(xx, xx_left) - 1 # spatial bin numbers
    for ii, ppii in enumerate(pp):
        state_pos[:,digitized[ii]] += np.transpose(ppii)

## now order states by peak location on track
peaklocations = state_pos.argmax(axis=1)
peakorder = peaklocations.argsort()

fig, ax1 = plt.subplots(1, 1, figsize=(6, 4))
ax1.matshow(state_pos[peakorder,:], interpolation='none', cmap='OrRd')
ax1.set_xlabel('position bin')
ax1.set_ylabel('state')
ax1.set_title('positional state clustering after sorting; NORUN < 1')


I am working on the file "StateFarmDeep.ipynb".
The first task is:
In Statefarm case. I construct Residual Network with Convolutional Neural Network. Keras have 2 ways to write model(Sequential and Functional) but Functional API is easier to implement ResNet than Sequential
Can you write Python code for it?

np.random.seed(123)
inputs = Input(shape=(40, 30, 3))
x = Convolution2D(64,3,3,activation='relu',border_mode='same')(inputs)
x = Dropout(0.2)(x)
x = MaxPooling2D(pool_size=(2,2))(x)
y = Convolution2D(64,3,3,activation='relu',border_mode='same')(x)
y = Dropout(0.2)(y)
y = Convolution2D(64,3,3,activation='relu',border_mode='same')(y)
y = Dropout(0.2)(y)
z = merge([x,y],mode = 'sum')
z = Flatten()(z)
z = Dense(1024,activation='relu')(z)
z = Dropout(0.5)(z)
predictions = Dense(10, activation='softmax')(z)
model = Model(input=inputs,output=predictions)
model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])
history = model.fit(train_x, train_y, validation_data=(test_x, test_y), nb_epoch=10, batch_size=32)
scores = model.evaluate(test_x, test_y, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))

from keras.utils.visualize_util import plot
plot(model, to_file='Desktop/model.png')

plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.title("Model Loss")
plt.ylabel("Loss")
plt.xlabel("epoch")
plt.legend(["train","test"],loc="upper right")
plt.show()


I am working on the file "StateFarmDeep.ipynb".
The first task is:
แปลงข้อมูลจากรูปแบบ PIL เป็น Numpy Array เพื่อให้สามารถทำงานกับตัวแบบที่เขียนด้วย Keras<br>
Convert image from PIL to Numpy Array to work with Keras
Can you write Python code for it?

train_x = np.array(X_train)
test_x = np.array(X_test)

train_y = np.array(Y_train)
test_y = np.array(Y_test)


I am working on the file "StateProxyExample.ipynb".
The first task is:
Expected -- no attribute foo in Implementation1.
Can you write Python code for it?

try:
    b.foo
except:
    print("no implementation")


I am working on the file "StateProxyExample.ipynb".
The first task is:
http://python-3-patterns-idioms-test.readthedocs.io/en/latest/Fronting.html
Can you write Python code for it?

def run(b):
    b.f()
    b.g()
    b.h()
    b.g()

b = State_d(Implementation1())
run(b)

b.changeImp(Implementation2())
run(b)

b.foo = 100

b.foo

b = State_d(Implementation1())
run(b)


I am working on the file "StateSpace/Bayesian state space estimation in Python via Metropolis-Hastings.ipynb".
The first task is:
It turns out it will be convenient to write the model in terms of the precision of εε, defined to be h−1≡σ2εh−1≡σε2, and the ratio of the variances: q≡σ2η/σ2εq≡ση2/σε2 so that qh−1=σ2ηqh−1=ση2.
Then our error terms can be written:
εt∼N(0,h−1)ηt∼N(0,qh−1)
εt∼N(0,h−1)ηt∼N(0,qh−1)
And the true values are:
h−10=1/3=0.33h0−1=1/3=0.33
q=10/3=3.33q=10/3=3.33
To take a Bayesian approach to this problem, we assume that ψψ is a random variable, and we want to learn about the values of ψψ based on the data YTYT; in fact we want a density p(ψ|YT)p(ψ|YT). To do this, we use Bayes rule to write:
p(ψ|YT)=p(YT|ψ)p(ψ)p(YT)
p(ψ|YT)=p(YT|ψ)p(ψ)p(YT)
or
p(ψ|YT)⏟posterior∝p(YT|ψ)⏟likelihoodp(ψ)⏟prior
p(ψ|YT)⏟posterior∝p(YT|ψ)⏟likelihoodp(ψ)⏟prior
The object of interest is the posterior; to achieve it we need to specify a prior density for the unknown parameters and the likelihood function of the model.
Prior
We will use the following priors:
Precision
Since the precision must be positive, but has no theoretical upper bound, we use a Gamma prior:
h∼Gamma(αh,βh)
h∼Gamma(αh,βh)
to be specific, the density is written:
p(h)=βαhhΓ(α)hαh−1e−βhh
p(h)=βhαhΓ(α)hαh−1e−βhh
and we set the hyperparameters as αh=2,βh=2αh=2,βh=2. In this case, we have E(h)=αh/βh=1E(h)=αh/βh=1 and also E(h−1)=E(σ2ε)=1E(h−1)=E(σε2)=1.
Ratio of variances
Similarly, the ratio of variances must be positive, but has no theoretical upper bound, so we again use an (independent) Gamma prior:
q∼Gamma(αq,βq)
q∼Gamma(αq,βq)
and we set the same hyperparameters, so αq=2,βq=2αq=2,βq=2. Since E(q)=1E(q)=1, our prior is of equal variances. We then have E(σ2η)=E(qh−1)=E(q)E(h−1)=1E(ση2)=E(qh−1)=E(q)E(h−1)=1.
Initial state prior
As noted above, the Kalman filter must be initialized with μ0∼N(m0,P0)μ0∼N(m0,P0). We will use the following approximately diffuse prior:
μ0∼N(0,106)
μ0∼N(0,106)
Likelihood
For given parameters, likelihood of this model can be calculated via prediction error decomposition using an application of the Kalman filter iterations.
Posterior Simulation:...
Can you write Python code for it?

# Priors
precision = mc.Gamma('precision', 2, 4)
ratio = mc.Gamma('ratio', 2, 1)

# Likelihood calculated using the state-space model
class LocalLevel(sm.tsa.statespace.MLEModel):
    def __init__(self, endog):
        # Initialize the state space model
        super(LocalLevel, self).__init__(endog, k_states=1,
                                         initialization='approximate_diffuse',
                                         loglikelihood_burn=1)

        # Initialize known components of the state space matrices
        self.ssm['design', :] = 1
        self.ssm['transition', :] = 1
        self.ssm['selection', :] = 1

    @property
    def start_params(self):
        return [1. / np.var(self.endog), 1.]

    @property
    def param_names(self):
        return ['h_inv', 'q']

    def update(self, params, transformed=True):
        params = super(LocalLevel, self).update(params, transformed)

        h, q = params
        sigma2_eps = 1. / h
        sigma2_eta = q * sigma2_eps

        self.ssm['obs_cov', 0, 0] = sigma2_eps
        self.ssm['state_cov', 0, 0] = sigma2_eta

# Instantiate the local level model with our simulated data
ll_mod = LocalLevel(y)

# Create the stochastic (observed) component
@mc.stochastic(dtype=LocalLevel, observed=True)
def local_level(value=ll_mod, h=precision, q=ratio):
    return value.loglike([h, q], transformed=True)

# Create the PyMC model
ll_mc = mc.Model((precision, ratio, local_level))

# Create a PyMC sample
ll_sampler = mc.MCMC(ll_mc)


I am working on the file "StateSpace/Bayesian state space estimation in Python via Metropolis-Hastings.ipynb".
The first task is:
Suppose we have a time series $YT≡{yt}Tt=0YT≡{yt}t=0T$ which we model as local level process:
ytμt+1=μt+εt,εt∼N(0,σ2ε)=μt+ηt,ηt∼N(0,σ2η)
yt=μt+εt,εt∼N(0,σε2)μt+1=μt+ηt,ηt∼N(0,ση2)
In this model, there are two unknown parameters, which we will collected in a vector ψψ, so that: ψ=(σ2ε,σ2η)ψ=(σε2,ση2); let's set their true values as follows (denoted with the subscript 0):
ψ0=(σ2ε,0,σ2η,0)=(3,10)
ψ0=(σε,02,ση,02)=(3,10)
Finally, we also must specify the prior μ0∼N(m0,P0)μ0∼N(m0,P0) to initialize the Kalman filter.
Set T=1000T=1000.
Can you write Python code for it?

# True values
T = 1000
sigma2_eps0 = 3
sigma2_eta0 = 10

# Simulate data
np.random.seed(1234)
eps = np.random.normal(scale=sigma2_eps0**0.5, size=T)
eta = np.random.normal(scale=sigma2_eta0**0.5, size=T)
mu = np.cumsum(eta)
y = mu + eps

# Plot the time series
fig, ax = plt.subplots(figsize=(13,2))
ax.plot(y);
ax.set(xlabel='$T$', title='Simulated series');


I am working on the file "Statefarm.ipynb".
The first task is:
1580 pictures are 'weird', which is not that much compared to our 18587 pictures (roughly 8.5%)
Can you write Python code for it?

to_remove_files = set([filenames[index] for index in to_remove])
list(to_remove_files)[:5]

out = open('weird_files.txt', 'w')
for f in to_remove_files:
    out.write('%s\n'%f)

print(path)
%pwd

to_remove_files = [x.rstrip() for x in open('/home/ubuntu/statefarm/train/weird_files.txt')]
len(to_remove_files)
to_remove_files[:5]

%%bash
mkdir weird_ones
mkdir weird_ones/train
for i in {0..9}; do mkdir /home/ubuntu/statefarm/weird_ones/train/c${i}; done

%cd /home/ubuntu/statefarm/train
for l in glob.glob('*/*jpg'):
    if l in to_remove_files:
        os.popen('mv %s ../weird_ones/train/%s'%(l, l))

%%bash
find . -type f | wc -l


I am working on the file "Statefarm.ipynb".
The first task is:
First, make the validation set with different drivers
Can you write Python code for it?

%%bash
cut -f 1 -d ',' driver_imgs_list.csv | grep -v subject | uniq -c
lines=$(expr `wc -l driver_imgs_list.csv | cut -f 1 -d ' '` - 1)
echo "Got ${lines} pics"


I am working on the file "Statefarm.ipynb".
The first task is:
Give me all training pictures that don't have a class 'probability' above 90%
Can you write Python code for it?

np.where(np.amax(bn_feat, axis=1) < 0.9)

def check_training_picture(bn_feat, filenames, number):
    print(bn_feat[number,:])
    print(filenames[number])
    plt.imshow(mpimg.imread('train/' + filenames[number]))

check_training_picture(bn_feat, filenames, 22)


I am working on the file "Statefarm.ipynb".
The first task is:
Let's predict on the test set
The following is mashed together from fast.ai
Can you write Python code for it?

test_batches = get_batches(path+'test', batch_size=batch_size, shuffle=False, class_mode=None)

conv_test_feat = conv_model.predict_generator(test_batches, test_batches.nb_sample)


I am working on the file "Statefarm.ipynb".
The first task is:
OK we removed the weird ones.
Can you write Python code for it?

path = "/home/ubuntu/statefarm/"

batch_size = 64

gen_t = image.ImageDataGenerator(rotation_range=15, height_shift_range=0.05,
                shear_range=0.1, channel_shift_range=20, width_shift_range=0.1)

trn_batches = get_batches(path+'train', gen_t, batch_size=batch_size)
val_batches = get_batches(path+'valid', batch_size=batch_size*2, shuffle=False)

from vgg16bn import Vgg16BN
model = vgg_ft_bn(10)

model.compile(optimizer=Adam(1e-3),
       loss='categorical_crossentropy', metrics=['accuracy'])

model.fit_generator(trn_batches, trn_batches.N, nb_epoch=3, validation_data=val_batches,
                 nb_val_samples=val_batches.N)

model.optimizer.lr = 1e-5
model.fit_generator(trn_batches, trn_batches.N, nb_epoch=3, validation_data=val_batches,
                 nb_val_samples=val_batches.N)

last_conv_idx = [i for i,l in enumerate(model.layers) if type(l) is Convolution2D][-1]
conv_layers = model.layers[:last_conv_idx+1]

conv_model = Sequential(conv_layers)

trn_batches = get_batches(path+'train', gen_t, batch_size=batch_size, shuffle=False)

conv_feat = conv_model.predict_generator(trn_batches, trn_batches.nb_sample)
conv_val_feat = conv_model.predict_generator(val_batches, val_batches.nb_sample)

#print(conv_layers[-1].output_shape)
def get_bn_layers(p):
    return [
        MaxPooling2D(input_shape=conv_layers[-1].output_shape[1:]),
        Flatten(),
        Dropout(p),
        Dense(512, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(512, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(10, activation='softmax')
        ]

p = 0.8

(val_classes, trn_classes, val_labels, trn_labels,
    val_filenames, filenames, test_filenames) = get_classes(path)

bn_model = Sequential(get_bn_layers(p))
bn_model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])

bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, nb_epoch=1,
             validation_data=(conv_val_feat,...


I am working on the file "Statefarm.ipynb".
The first task is:
Private score: 0.92506
Public score: 1.11814
RESULTS
Interestingly, the validation accuracy and validation loss is VERY similar, almost identical to the above. The training accuracy is slightly better.
TODO: Fix the validation problems too
TRYING OUT CUTTING FROM PICTURES
Can you write Python code for it?

hog = cv2.HOGDescriptor()
hog.setSVMDetector(cv2.HOGDescriptor_getDefaultPeopleDetector())
img = glob.glob('train/*/*jpg')[100]

img = cv2.imread(img)

(rects, weights) = hog.detectMultiScale(img, winStride=(4, 4), padding=(8, 8), scale=1.05)
for (x, y, w, h) in rects:
    cv2.rectangle(orig, (x, y), (x + w, y + h), (0, 0, 255), 2)
rects = np.array([[x, y, x + w, y + h] for (x, y, w, h) in rects])
pick = non_max_suppression(rects, probs=None, overlapThresh=0.5)
for (xA, yA, xB, yB) in pick:
    cv2.rectangle(img, (xA, yA), (xB, yB), (0, 255, 0), 2)
#plt.imshow(img)
#cv2.imshow('hi', img)
img.save('test.png')


I am working on the file "Statefarm.ipynb".
The first task is:
Private score: 0.94359
Public score: 1.18213
Can you write Python code for it?

bn_model.save_weights(path+'models/bn_model.h5')
bn_model.load_weights(path+'models/bn_model.h5')

bn_feat = bn_model.predict(conv_feat, batch_size=batch_size)
bn_val_feat = bn_model.predict(conv_val_feat, batch_size=batch_size)


I am working on the file "Statefarm.ipynb".
The first task is:
That took forever (one hour? didn't time it perfectly)
Can you write Python code for it?

preds = bn_model.predict(conv_test_feat, batch_size=batch_size*2)
subm = do_clip(preds,0.93)

subm_name = path+'results/subm.gz'

classes = sorted(trn_batches.class_indices, key=trn_batches.class_indices.get)

submission = pd.DataFrame(subm, columns=classes)
submission.insert(0, 'img', [a[4:] for a in test_filenames])
submission.head()
submission.to_csv(subm_name, index=False, compression='gzip')

from IPython.display import FileLink
FileLink(subm_name)


preds = bn_model.predict(conv_test_feat, batch_size=batch_size*2)
subm = do_clip(preds,0.93)

subm_name = path+'results/subm_woweird.gz'

classes = sorted(trn_batches.class_indices, key=trn_batches.class_indices.get)

submission = pd.DataFrame(subm, columns=classes)
submission.insert(0, 'img', [a[4:] for a in test_filenames])
submission.head()
submission.to_csv(subm_name, index=False, compression='gzip')

from IPython.display import FileLink
FileLink(subm_name)


I am working on the file "Statefarm.ipynb".
The first task is:
This doesn't have any 'good' class, everything is low, which is weird - could be the not-straight head angle, but who knows. I just realised that some pictures have a blue tape-like thing on the driver window (see above and below), some pictures don't have that sheet, which is probably confusing.
TODO: find a way to mask that window
Can you write Python code for it?


I am working on the file "Statefarm.ipynb".
The first task is:
This is marked as 'talking to passenger', but it may as well be c0, driving normally.
Kick out the 'bad' pictures
I believe that these low quality marks 'confuse' the network, so a network trained without those pictures should work slightly better.
Can you write Python code for it?

to_remove = np.where(np.amax(bn_feat, axis=1) < 0.9)[0]
print(len(to_remove))
print(1580./18587*100)


I am working on the file "Statefarm.ipynb".
The first task is:
This is marked as class0 -
c0: normal driving
c1: texting - right
c2: talking on the phone - right
c3: texting - left
c4: talking on the phone - left
c5: operating the radio
c6: drinking
c7: reaching behind
c8: hair and makeup
c9: talking to passenger
That hand is probably confusing, but it's mostly the correct class.
Can you write Python code for it?


I am working on the file "Statefarm.ipynb".
The first task is:
now starts the actual work
Can you write Python code for it?

batch_size = 64

gen_t = image.ImageDataGenerator(rotation_range=15, height_shift_range=0.05,
                shear_range=0.1, channel_shift_range=20, width_shift_range=0.1)

trn_batches = get_batches(path+'train', gen_t, batch_size=batch_size)
val_batches = get_batches(path+'valid', batch_size=batch_size*2, shuffle=False)

from vgg16bn import Vgg16BN
model = vgg_ft_bn(10)

model.compile(optimizer=Adam(1e-3),
       loss='categorical_crossentropy', metrics=['accuracy'])

model.fit_generator(trn_batches, trn_batches.N, nb_epoch=3, validation_data=val_batches,
                 nb_val_samples=val_batches.N)

model.optimizer.lr = 1e-5
model.fit_generator(trn_batches, trn_batches.N, nb_epoch=3, validation_data=val_batches,
                 nb_val_samples=val_batches.N)

last_conv_idx = [i for i,l in enumerate(model.layers) if type(l) is Convolution2D][-1]
conv_layers = model.layers[:last_conv_idx+1]

conv_model = Sequential(conv_layers)

trn_batches = get_batches(path+'train', gen_t, batch_size=batch_size, shuffle=False)

conv_feat = conv_model.predict_generator(trn_batches, trn_batches.nb_sample)
conv_val_feat = conv_model.predict_generator(val_batches, val_batches.nb_sample)

save_array(path+'results/conv_val_feat.dat', conv_val_feat)
save_array(path+'results/conv_feat.dat', conv_feat)
#print(type(conv_feat))

conv_feat = load_array(path+'results/conv_feat.dat')
conv_val_feat = load_array(path+'results/conv_val_feat.dat')
print(type(conv_feat))

#print(conv_layers[-1].output_shape)
def get_bn_layers(p):
    return [
        MaxPooling2D(input_shape=conv_layers[-1].output_shape[1:]),
        Flatten(),
        Dropout(p),
        Dense(512, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(512, activation='relu'),
        BatchNormalization(),
        Dropout(p),
        Dense(10, activation='softmax')
        ]

p = 0.8

(val_classes, trn_classes, val_labels, trn_labels,
    val_filenames, filenames, test_filenames) = get_classes(path)

bn_model =...


I am working on the file "Statements Assessment Test - Solutions.ipynb".
The first task is:
Go through the string below and if the length of a word is even print "even!"
Can you write Python code for it?

st = 'Print every word in this sentence that has an even number of letters'

for word in st.split():
    if len(word)%2 == 0:
        print word+" <-- has an even length!"


I am working on the file "Statements Assessment Test - Solutions.ipynb".
The first task is:
Use List Comprehension to create a list of the first letters of every word in the string below:
Can you write Python code for it?

st = 'Create a list of the first letters of every word in this string'

[word[0] for word in st.split()]


I am working on the file "Statements Assessment Test - Solutions.ipynb".
The first task is:
Use List comprehension to create a list of all numbers between 1 and 50 that are divisible by 3.
Can you write Python code for it?


I am working on the file "Statements Assessment Test - Solutions.ipynb".
The first task is:
Use range() to print all the even numbers from 0 to 10.
Can you write Python code for it?


I am working on the file "Statements Assessment Test - Solutions.ipynb".
The first task is:
Write a program that prints the integers from 1 to 100. But for multiples of three print "Fizz" instead of the number, and for the multiples of five print "Buzz". For numbers which are multiples of both three and five print "FizzBuzz".
Can you write Python code for it?

for num in xrange(1,101):
    if num % 5 == 0 and num % 3 == 0:
        print "FizzBuzz"
    elif num % 3 == 0:
        print "Fizz"
    elif num % 5 == 0:
        print "Buzz"
    else:
        print num


I am working on the file "Statements Assessment Test.ipynb".
The first task is:
Go through the string below and if the length of a word is even print "even!"
Can you write Python code for it?

st = 'Print every word in this sentence that has an even number of letters'

#Code in this cell
st = 'Print every word in this sentence that has an even number of letters'
for word in st.split():
    if len(word) % 2 == 0:
        print(word)


I am working on the file "Statements Assessment Test.ipynb".
The first task is:
Use List comprehension to create a list of all numbers between 1 and 50 that are divisible by 3.
Can you write Python code for it?

#Code in this cell
l = [number for number in range(1,51) if number % 3 == 0]
print(l)


#Code in this cell
[num for num in range(1, 50) if num % 3 == 0]


I am working on the file "Statements Assessment Test.ipynb".
The first task is:
Use range() to print all the even numbers from 0 to 10.
Can you write Python code for it?

#Code Here
for number in range(0,11):
    if number % 2 == 0:
        print(number)


I am working on the file "Statformulas.ipynb".
The first task is:
$$J_{a/H,b}=\begin{pmatrix} H &0 \0 & 1 \end{pmatrix}$$
$$J^T \begin{pmatrix}  H^{-2} A &  H^{-1} B \  H^{-1} B & C \end{pmatrix} J= \begin{pmatrix}  A &B \B & C\end{pmatrix}$$
$H=B/A$
$$J^T \begin{pmatrix}   A &   B \  B & C \end{pmatrix} J= \begin{pmatrix}  B^2/A &B^2/A \B^2/A & C\end{pmatrix}$$
$$J_{a+b,b}=\begin{pmatrix}  1 &-1 \0 & 1 \end{pmatrix}$$
$$J^T\begin{pmatrix}  A &A \A & B \end{pmatrix} J= \begin{pmatrix}  A &0 \0 & B-A\end{pmatrix}$$
$$J_{\log a,b}=\begin{pmatrix}  e^a &0 \0 & 1 \end{pmatrix}$$
$$J_{\log a,b}^T \begin{pmatrix}  e^{-2a} A & e^{-a} B \e^{-a} B & C \end{pmatrix} J_{\log a,b}= \begin{pmatrix}  A &B \B & C\end{pmatrix}$$
$$J_{e^a,b}=\begin{pmatrix}  1/a &0 \0 & 1 \end{pmatrix}$$
$$J^T \begin{pmatrix}  a^2 A & a B \ a B & C \end{pmatrix} J= \begin{pmatrix}  A &B \B & C\end{pmatrix}$$
$$J_{a^{-1},b}=\begin{pmatrix}  -a^{2} &0 \0 & 1 \end{pmatrix}$$
$$J^T \begin{pmatrix}  a^{-4} A & a^{-2} B \ a^{-2} B & C \end{pmatrix} J= \begin{pmatrix}  A &-B \-B & C\end{pmatrix}$$
Can you write Python code for it?


I am working on the file "Statformulas.ipynb".
The first task is:
Beta distribution
Parameters $\alpha$ and $\beta$ are orthogonal if
$$\operatorname{E}_X
\left[
 \frac{\partial \log f(X;\alpha,\beta)}{\partial\alpha \ \partial\beta}
 \right]=0$$
The probability density function of Beta distribution parametrized by shape parameters $\alpha$ and $\beta$ is
$$f(X=x;\alpha,\beta)=\frac{ x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}$$
Consider parametrization in terms of logarithm of geometric mean $E[\log X]=\gamma=\psi(\alpha)-\psi(\alpha+\beta)$ and the logarithm of geometric mean of $1-X$: $E[\log (1-X)]=\phi=\psi(\beta)-\psi(\alpha+\beta)$
Then the fisher information matrix of the distribution parametrized by shape parameters is
$$I_{\alpha,\beta}=\begin{pmatrix}\psi'(\alpha)-\psi'(\alpha+\beta) & -\psi'(\alpha+\beta)\
-\psi'(\alpha+\beta) & \psi'(\beta)-\psi'(\alpha+\beta)
\end{pmatrix}$$
Fisher information matrix when parametrized by $\gamma$ and $\phi$ is
$$I_{\gamma,\phi}=J^\mathrm{T} I_{\alpha,\beta} J$$
Where $J$ is the Jacobian matrix defined as
$$J=\begin{pmatrix}\frac{\partial \alpha}{\partial \gamma} & \frac{\partial \alpha}{\partial \phi}\
\frac{\partial \beta}{\partial \gamma} & \frac{\partial \beta}{\partial \phi}
\end{pmatrix}$$
Note that $I_{\alpha,\beta}$ can be written as:
$$I_{\alpha,\beta}=\begin{pmatrix}\frac{\partial \gamma}{\partial \alpha} & \frac{\partial \phi}{\partial \alpha} \ \frac{\partial \gamma}{\partial \beta}  & \frac{\partial \phi}{\partial \beta}
\end{pmatrix}$$
$$\mathrm{Cov}(\gamma,\phi)=J_{12}J_{11}\psi'(\alpha)+J_{21}J_{22}\psi'(\beta)-\psi'(\alpha+\beta)(J_{12}J_{11}+J_{21}J_{22}+J_{21}J_{12}+J_{11}J_{22})$$
$\gamma=\psi(\alpha)-\psi(\alpha+\beta)$
$\phi=\psi(\beta)-\psi(\alpha+\beta)$
$\gamma-\phi=\psi(\alpha)-\psi(\beta)$
$\alpha=\omega(\phi-\psi(\beta))-\beta$
$\beta=\omega(\gamma-\psi(\alpha))-\alpha$
$$\gamma=\frac{\partial \log \mathrm{B}(\alpha,\beta)}{\partial \alpha}=\frac{\partial \log \Gamma(\alpha)}{\partial \alpha}-\frac{\partial \log \Gamma(\alpha+\beta)}{\partial \alpha}$$
$$\phi=\frac{\partial \log...
Can you write Python code for it?

import pystan

model = """
data {
    int<lower=0> N; //nr subjects
    vector<lower=0>[N] yLT;
}parameters {
    real a;
    real b;
}model {
    for (n in 1:N)
        yLT[n]~beta(exp(a),exp(b));
}
"""
smBeta = pystan.StanModel(model_code=model)

from scipy import stats
x=np.linspace(0,1,101)[1:]
plt.plot(x,stats.beta.pdf(x,4,15,0,1))
temp={'yLT':stats.beta.rvs(4,15,0,1,size=100),'N':100}
fit=smBeta.sampling(data=temp,chains=6,n_jobs=6,
        seed=1,thin=4,iter=55000,warmup=5000)
print(fit)
w=fit.extract()
a=np.exp(w['a'])
b=np.exp(w['b'])

from scipy.special import gamma, digamma,polygamma,beta
plt.figure(figsize=(12,12))
gA=digamma(a)-digamma(a+b)
gB=digamma(b)-digamma(a+b)
tA=polygamma(1,a)-polygamma(1,a+b)

var=a*b/np.square(a+b)/(a+b+1)
ex=a/(a+b)
q1=ex
q2=var

#q2=g
#k=np.exp(a)
#l=np.exp(b)
#q1=np.log(np.square(k)*digamma(2)+digamma(1))/(2*digamma(2))-g/(polygamma(1,1)+1)

plt.plot(q1,q2,'.')
#plt.ylim([0,1])
#plt.xlim([0,1])
np.corrcoef(q1,q2)[0,1]


I am working on the file "Statformulas.ipynb".
The first task is:
Hierarchical parameter recovery
Can you write Python code for it?

model = """
data {
    int<lower=0> N; //nr subjects
    int<lower=0> M;
    real gm;
    real gs;
    real t;
}generated quantities{
    real g[N];
    real<lower=0> y[N,M];
    for (n in 1:N){
        g[n]=normal_rng(gm,gs);
        for (m in 1:M){
            y[n,m]=gamma_rng(invdigammaR(g[n]-t),exp(t));
}}}
"""
smGammaGen = pystan.StanModel(model_code=invgammafun+model)

N=10;M=20
fit=smGammaGen.sampling(data={'N':N,'M':M,'gm':5,'gs':2,'t':1},
    chains=4,n_jobs=4,seed=1,thin=1,iter=30,warmup=0,algorithm="Fixed_param")
w=fit.extract()
y=w['y'][0,:,:]
print(y.shape)

model = """
data {
    int<lower=0> N; //nr subjects
    int<lower=0> M;
    real<lower=0> y[N,M];
}parameters{
    real g[N];
    real gm;
    real<lower=0> gs;
    real t;
}model{
    for (n in 1:N){
        g[n]~normal(gm,gs);
        for (m in 1:M){
            y[n,m]~gamma(invdigammaR(g[n]-t),exp(t));
}}}
"""
smGamma = pystan.StanModel(model_code=invgammafun+model)

fit=smGamma.sampling(data={'N':N,'M':M,'y':y},
    chains=4,n_jobs=4,seed=2,thin=1,iter=1000,warmup=500)
print(fit)

%pylab inline
plt.plot(w['gm'])


I am working on the file "Statformulas.ipynb".
The first task is:
Hierarchical weibull
Can you write Python code for it?

N=20
M=50
e=np.random.randn(N)*1+2
t=np.random.randn(N)*1+1
#t=-1;e=1
k=4.313501020391736/(np.abs(e-t))

#print('k= ',k)
data={'y':stats.weibull_min.rvs(k,0,np.exp(t),size=(M,N)).T,'N':N,'M':M}
ms=[smWeibullH, smWeibullEH]
D,R=computeConvergence(ms,data,reps=50)

D


I am working on the file "Statformulas.ipynb".
The first task is:
Information matrix generalized gamma
$$f(y)=\frac{\kappa}{y \Gamma(\rho)}\left(\frac{y}{\theta}\right)^{\kappa \rho}e^{-\left(\frac{y}{\theta} \right)^\kappa}$$
$$\log f(y)=\log \kappa- \log y -\log \Gamma(\rho) +\kappa \rho \log y - \kappa \rho \log \theta -\left(\frac{y}{\theta} \right)^\kappa$$
$$I_{\rho \theta \kappa} = \begin{pmatrix} \psi'(\rho) & \frac{\kappa}{\theta} &- \frac{\psi(\rho)}{\kappa} \
. & \frac{\rho \kappa^2}{\theta^2} & -\frac{\rho}{\theta}\psi(\rho+1)\
. & . & \frac{\rho}{\kappa^2}\left(\psi'(\rho+1)+\psi(\rho+1)^2+\frac{1}{\rho}\right)\end{pmatrix} $$
$\rho (\psi'(\rho+1)+\psi(\rho+1)^2+\frac{1}{k})= \rho \psi'(\rho)+\rho \psi(\rho)^2 + 2\psi(\rho) +1$
$E[\log Y]= \log \theta + \psi(\rho)/\kappa$
$E[Y^s]=\theta^s \Gamma(\rho+s/\kappa)/\Gamma(\rho)$
$E[Y^\kappa]=\theta^\kappa \rho$
$E[Y^\kappa \log Y ]=\theta^\kappa \rho (\log \theta + \psi(\rho+1)/\kappa)= \theta^\kappa (\rho \log \theta + \rho \psi(\rho)/\kappa+1/\kappa)$
$E[\log^2 Y]= \log^2 \theta + 2 \log \theta \psi(\rho)/\kappa+(\psi'(\rho)+\psi(\rho)^2)/\kappa^2$
$E[Y^\kappa \log^2 Y]= \theta^\kappa \rho (\log^2 \theta + 2 \log \theta \psi(\rho+1)/\kappa+(\psi'(\rho+1)+\psi(\rho+1)^2)/\kappa^2)$
$E[Y^{2\kappa} \log^2 Y]= \theta^{2\kappa} (\rho+1) (\log^2 \theta + 2 \log \theta \psi(\rho+2)/\kappa+(\psi'(\rho+2)+\psi(\rho+2)^2)/\kappa^2)$
$\mathrm{Var}[\log Y]=\psi'(\rho)/\kappa^2$
$E[(Y/\theta)^\kappa]=\rho$
$\mathrm{Var}[(Y/\theta)^\kappa]=\rho$
$E[\log (Y/\theta)^\kappa]= \psi(\rho)$
$E[\log^2 (Y/\theta)^\kappa]= \psi'(\rho)+\psi(\rho)^2$
$E[(Y/\theta)^\kappa \log (Y/\theta)^\kappa ]= \rho \psi(\rho+1)= \rho \psi(\rho)+1$
$E[(Y/\theta)^\kappa \log^2(Y/\theta)^\kappa ]= \rho (\psi'(\rho+1)+\psi(\rho+1)^2)$
$$I_{\rho \tau \kappa} = \begin{pmatrix} \psi'(\rho) & \kappa &- \frac{\psi(\rho)}{\kappa} \
. & \rho \kappa^2 & -\rho\psi(\rho+1)\
. & . & \frac{\rho}{\kappa^2}\left(\psi'(\rho+1)+\psi(\rho+1)^2+\frac{1}{\rho}\right)\end{pmatrix} $$
$$I_{\rho, \tau, \log \kappa} = \begin{pmatrix} \psi'(\rho) & \kappa &- \psi(\rho) \
. & \rho...
Can you write Python code for it?

import pystan
ggcode='''functions{
//' Naive implementation of the generalized Gamma density.
//' @param x Value to evaluate density at.
//' @param alpha Shape parameter.
//' @param beta Scale parameter.
//' @param nu Tail parameter.
real gengamma_pdf(real x, real k, real b, real q) {
  real d;
  d = q/(b*tgamma(k))*(x/b)^(k*q-1) * exp(-(x/b)^q);
  return d;
}

real gengamma_lpdf(real x, real k, real b, real q) {
  real d;
  d = log(q) - log(b) - lgamma(k) +
    (k*q-1)*(log(x) - log(b)) - (x/b)^q;
  return d;
}

real generalized_gamma_cdf(real x, real k, real b, real q) {
  real d;
  d = gamma_p(k, (x/b)^q);
  return d;
}

real generalized_gamma_lcdf(real x, real k, real b, real q) {
  real d;
  d = log(generalized_gamma_cdf(x, k, b, q));
  return d;
}}'''

model = """
data {
    int<lower=0> N; //nr subjects
    vector<lower=0>[N] yLT;
}parameters {
    real k;
    //real b;
    real q;
}model {
    for (n in 1:N)
        yLT[n]~gengamma(exp(k),exp(0),exp(q));
}
"""
smGengamma = pystan.StanModel(model_code=ggcode+model)

from scipy import stats
x=np.linspace(0,10,101)[1:]
#k,q,0,b
k=2;b=1;q=3;
plt.plot(x,stats.gengamma.pdf(x,k,q,0,b))
temp={'yLT':stats.gengamma.rvs(k,q,0,b,size=100),'N':100}

fit=smGengamma.sampling(data=temp,chains=6,n_jobs=6,
        seed=1,thin=1,iter=10000,warmup=500)
print(fit)
w=fit.extract()
p=np.exp(w['k'])
#b=np.exp(w['b'])
H=(pg(1,p+1)+np.square(pg(0,p+1))+1/p)/pg(0,p+1)
e=H/np.exp(w['q'])+1
plt.figure()
plt.plot(p,e,'.')
np.corrcoef(p,e)[0,1]

from scipy.special import gamma, digamma,polygamma
plt.figure(figsize=(12,4))
g=np.log(b)+digamma(k)/q
c=(polygamma(1,k+1)+polygamma(0,k+1)**2+1/k)*q/polygamma(0,k+1)+np.log(b)
q1=g
q2=np.log(b)
q3=c
#*np.exp(-a)+q2
plt.subplot(1,3,1)
plt.plot(q1,q2,'.')
plt.title(np.corrcoef(q1,q2)[0,1])
plt.subplot(1,3,2)
plt.plot(q1,q3,'.')
plt.title(np.corrcoef(q1,q3)[0,1])
plt.ylim([-1000,1000])
plt.subplot(1,3,3)
plt.plot(q2,q3,'.')
plt.title(np.corrcoef(q2,q3)[0,1]);
plt.ylim([-50,50])


I am working on the file "Statformulas.ipynb".
The first task is:
Notation
$Y$ generic random variable
$U$ latent random variable
$V$ residual random variable
$X$ predictor
Parameters
$\eta$ and $\nu$ generic parameters
$\mu=E[Y]$ mean parameter
$\gamma=E[\log Y]$ geometric mean parameter
$\sigma^2=E[(Y-\mu)^2]$ standard deviation parameter
$Y=\alpha+U$ shift parameter
$Y= U/\theta$ scale parameter
$Y= U \lambda$ inverse-scale (rate) parameter
$Y=e^{-\tau} U$ log-scale parameter
$Y=U^\kappa$ exponent parameter
$Y=f(U,\rho)$ shape parameter
$Y=\alpha + \beta X$ linear predictor
$\psi$ digamma function
$\pi$ pi number
$\phi$ measurement scale
$\delta$ dirac function
$\zeta,\epsilon,\varepsilon,\vartheta,\iota,\xi,\varpi,\varrho,\varsigma,\varphi,\chi,\omega$
Gamma distribution
Paremeters $\eta$ and $\nu$ are orthogonal if
$$\operatorname{E}_Y
\left[
 \frac{\partial \log f(Y;\eta,\nu)}{\partial\eta \ \partial\nu}
 \right]=0$$
The probability density function of Gamma distribution parametrized by shape parameter $\rho$ and scale parameter $\theta$ is
$$f(Y=y;\rho,\theta)=\frac{1}{\Gamma(\rho) \theta^\rho} y^{\rho - 1} e^{-\frac{y}{\theta}}$$
with Fisher information
$$I_{\rho \theta} =  \begin{pmatrix}
\psi'(\rho) & \theta^{-1} \
\theta^{-1} & \rho \theta^{-2} \end{pmatrix} $$
Consider parametrization in terms of logarithm of geometric mean $\gamma=E[\log Y]=\psi(\rho)+\log \theta$ and log-scale $\tau=\log(\theta)$, where $\psi$ is the digamma function. Then the logarithm of density function parametrized by $\gamma$ and $\tau$ is
$$\log f(Y=y;\gamma,\tau)=-\log{\Gamma(\omega(\gamma-\tau)) -\tau \omega(\gamma-\tau) + (\omega(\gamma-\tau)-1)\log y- y e^{-\tau}}$$
where we use $\omega$ to label the inverse digamma function. By $\omega'(y)$ $\omega''(y)$ and  we denote the first and second derivative of inverse digamma function with respect to $y$. Next, we compute the first derivative of the log-density with respect to $\gamma$:
$$\begin{align} \frac{\partial}{\partial\gamma}\log f(Y;\gamma,\tau) &= -\psi(\omega(\gamma-\tau)) \omega'(\gamma-\tau)-\tau \omega'(\gamma-\tau) +...
Can you write Python code for it?

model = """
data {
    int<lower=0> N; //nr subjects
    real<lower=0> k;
    real<lower=0> t;
}generated quantities{
    real<lower=0> y;
    y=gamma_rng(k,1/t);
}
"""
smGammaGen = pystan.StanModel(model_code=model)

model = """
data {
    int<lower=0> N; //nr subjects
    real<lower=0> y[N];
}parameters{
    real<lower=0> k;
    real<lower=0> t;
}model{
    for (n in 1:N)
        y[n]~gamma(k,1/t);
}
"""
smGamma = pystan.StanModel(model_code=model)

N=1000
fit=smGammaGen.sampling(data={'N':N,'k':10,'t':np.exp(-10)},
    chains=1,n_jobs=1,seed=1,thin=1,iter=N,warmup=0,algorithm="Fixed_param")
w=fit.extract()
y=w['y']
print(y.shape)

fit=smGamma.sampling(data={'N':N,'y':y},
    chains=4,n_jobs=4,seed=1,thin=2,iter=2000,warmup=1000)
print(fit)
w=fit.extract()
t=np.log(w['t'])
g=pg(0,w['k'])+t
w=fit.extract()
plt.plot(g,t,'.')
np.corrcoef(g,t)[0,1]

invgammafun='''functions{
    vector invdigamma(vector x){
        vector[num_elements(x)] y; vector[num_elements(x)] L;
        for (i in 1:num_elements(x)){
            if (x[i]==digamma(1)){
                y[i]=1;
            }else{ if (x[i]>=-2.22){
                y[i]=(exp(x[i])+0.5);
            }else{
                y[i]=1/(x[i]-digamma(1));
        }}}
        L=digamma(y)-x;
        while (min(L)>10^-12){
            y=y-L ./trigamma(y);
            L=digamma(y)-x;
        }
        return y;}
    real invdigammaR(real x){
        real y; real L;
        if (x==digamma(1)){
            y=1;
        }else{ if (x>=-2.22){
            y=(exp(x)+0.5);
        }else{
            y=1/(x-digamma(1));
        }}
        L=digamma(y)-x;

        while (abs(L)>1e-5){
            y=y-L ./trigamma(y);
            L=digamma(y)-x;
        }
        return y;
    }} '''
model = """
data {
    int<lower=0> N; //nr subjects
    real<lower=0> y[N];
}parameters{
    real<lower=-100,upper=100> g;
    real<lower=-100,upper=100> t;
}transformed parameters{
    real k;
    k=invdigammaR(g-t);
}model{
    for (n in 1:N)
        y[n]~gamma(k,exp(-t));
}
"""
smGammaGeom...


I am working on the file "Statformulas.ipynb".
The first task is:
Wald distribution Fisher information
$$f(x)=\frac{\alpha}{\sigma \sqrt{2 \pi x^3}}\exp\left(-\frac{(\nu x-\alpha)^2}{2 \sigma^2 x}\right)$$
$E[X]=\alpha/\nu$
$E[1/X]=\nu/\alpha +\sigma^2/\alpha^2$
$$I_{\alpha \sigma \nu} = \begin{pmatrix} \frac{2}{\alpha^2}+\frac{\nu}{\sigma^2 \alpha} & \frac{2}{\sigma \alpha} & \frac{1}{\sigma}\
. & \frac{1}{\sigma^2} &0\
. & . & \frac{\alpha}{\sigma^2 \nu}\end{pmatrix} $$
$$I_{\log \alpha,\log \sigma \nu} = \begin{pmatrix} 2 \sigma+\frac{\nu \alpha}{\sigma} & 2 & \frac{1}{\sigma}\
. & 1 &0\
. & . & \frac{\alpha}{\sigma^2 \nu}\end{pmatrix} $$
Can you write Python code for it?

1/(1+pg(0,2)**2/pg(1,1))

pg(0,2)

from scipy.special import gamma

gamma(1)


I am working on the file "Static_tumbling_nn.ipynb".
The first task is:
Convert all static tumbling passes to vectors
Can you write Python code for it?

word_vectors = np.zeros((len(elements), len(vocab)), dtype=np.int_)
for ii, text in enumerate(elements):
    word_vectors[ii] = text_to_vector(text)

word_vectors


I am working on the file "Static_tumbling_nn.ipynb".
The first task is:
Create dictionary to map each element to an index
Can you write Python code for it?

word2idx = {word: i for i, word in enumerate(vocab)}

word2idx


I am working on the file "Static_tumbling_nn.ipynb".
The first task is:
Generate global vocabulary
Can you write Python code for it?

#Main vocabulary, based on the data set elements
main_vocab = set()
for line in elements:
    for element in line.split(" "):
        main_vocab.add(element)

main_vocab = list(main_vocab)

#Expanded vocabulary based on 49 permutations of the posible transitions

vocab = list(main_vocab)

for roll in product(main_vocab, repeat = 2 ):
     vocab.append("{} {}".format(roll[0],roll[1]))


I am working on the file "Static_tumbling_nn.ipynb".
The first task is:
I know total loss is still to high, but not that bad for the first round of hyper parameters, still room for total loss improvement
Saving de model
Can you write Python code for it?

# Load model
# model.load('Checkpoints/model-with-transitions-with-3-layers.tfl')

# Manually save model
model.save("Checkpoints/model-with-transitions-with-3-layers.tfl")


I am working on the file "Static_tumbling_nn.ipynb".
The first task is:
Load and prepare the data
Import data from static tumbling csv file
Can you write Python code for it?


I am working on the file "Static_tumbling_nn.ipynb".
The first task is:
Now test the model with data that wasn't in the data set
in this complex example the second element is a lot harder to execute
Can you write Python code for it?

test_element1 = "flick flick flick flick flick mortal giro giro giro2"
test_element2 = "mortal flick giro flick giro mortal giro2 giro2 giro2"

test_compare(test_element1,test_element2)


I am working on the file "Static_tumbling_nn.ipynb".
The first task is:
Now we check the accuracy of the mode, this test checks which static tumbling line is more difficult, the second one is not even in the data we trianed the neural network.
First we compare to static tumblin pass that has the same elements but different transition cost or effort,
acording to flick mortal and mortal flick it's harder to execute mortal flick
Can you write Python code for it?

element1 = "flick mortal"
element2 = "mortal flick"

test_compare(element1,element2)


I am working on the file "Static_tumbling_nn.ipynb".
The first task is:
Test data validation
Now the test values we separeted from the begining are going to be compared with the actual values to check model accuracy
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(15,6))
predictions = model.predict(testX)
ax.plot(predictions,label='Prediction')
ax.plot(testY, label='Data')
ax.set_xlim(right=len(predictions))
ax.legend()


I am working on the file "Static_tumbling_nn.ipynb".
The first task is:
Testing
Can you write Python code for it?

# Helper function that uses our model to get the score for the static tumbling pass
def test_score(sentence):
    score = model.predict([text_to_vector(sentence.lower())])
    print('Gym pass: {}'.format(sentence))
    print('Score: {}'.format(score))
    print()
    return score

# Helper function that uses our model to compare static tumbling passes
def test_compare(pass1, pass2):
    score1 = test_score(pass1)
    score2 = test_score(pass2)
    if score1 > score2:
        print('Gym pass 1: {}'.format(pass1))
    elif score2 > score1:
        print('Gym pass 2: {}'.format(pass2))
    else:
        print('same difficulty')


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
Distributions and The Central Limit Theorem
The Bernoulli Distribution
Bernoulli distributions are the result of a random variable with a binary outcome, like a coin flip or medical test giving a positive or negative result.  Typically we represent the outcomes of a Bernoulli Random variable $X$ of only taking values of 0 or 1, with probabilities $p$ and $1 - p$ respectively, mean $p$, variance $p(1 - p)$, and PMF given by
$$ P(X = x) = p^x (1 - p)^{1 - x} $$
Where $x$ is the outcome and $p$ is the probability of the positive outcome (1).
Bernoulli random variables crop up very often in statistical analysis — most often in the form of Binomial trials, or, as a sum of independent Bernoulli variables with PMF given by
$$ P(X = x) = {n \choose x} p^x (1 - p)^{n - x} $$
where
$$ {n \choose x} = \frac{n!}{x!(n - x)!} $$
In this exercise you'll take a look at the HR data and apply these concepts to gain some insight.
Using the HR data, answer the following.
1. Which variables in the HR data can be said to be Bernoulli random variables?
2. For the variables you identified in part 1, compute the probabilities $p_k$, of each having a positive $(x = 1)$ result, where $k$ is a placeholder for each variable.
3. Compute the variance of each of the variables in part 2 using $p_k$ as described above.
4. For each of the k variables, compute the probability of randomly selecting 3500 employees with a positive result.  Comment on your answer.
5. For each of the k variables, compute the probability of randomly selecting 3500 or less with a positive result.  Comment on your answer.
6. Now plot both the PMF and CDF as a function of the number of drawn samples for each of the k variables.  Comment on your results.
Can you write Python code for it?


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
From the plots we can see that the peaks of the normal are far apart, thus backing up our test results.
Can you write Python code for it?

# Test the hypothesis that the satisfaction level between each salary group, denoted k,
# differs signicantly from the mean. Namely
# H0:μ−μk=0H0:μ−μk=0
# Ha:μ−μk≠0Ha:μ−μk≠0

# print(df.satisfaction_level.mean())
# print(df_high.satisfaction_level.mean())
# print(df_medium.satisfaction_level.mean())
# print(df_low.satisfaction_level.mean())
# print(df.satisfaction_level.var())
# print(df_high.satisfaction_level.var())
# print(df_medium.satisfaction_level.var())
# print(df_low.satisfaction_level.var())

print('High level test: {}'.format(stats.ttest_1samp(df_high.satisfaction_level, df.satisfaction_level.mean())))
print('Medium level test: {}'.format(stats.ttest_1samp(df_medium.satisfaction_level, df.satisfaction_level.mean())))
print('Low level test: {}'.format(stats.ttest_1samp(df_low.satisfaction_level, df.satisfaction_level.mean())))


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
How would you interpret your results in part 4?
We can reject the null hypothesis for all three salary levels with $\alpha = 0.01$.
Can you write Python code for it?

# Generate plots for part 4 as you did in part 3. What conclusions can you draw from the plot?

fig = plt.figure(figsize=(15, 10))
ax = plt.axes()
# x1 = np.linspace(df.satisfaction_level.min(), df.satisfaction_level.max(), 1000)
# ax.plot(x1, stats.norm.pdf(x1, df.satisfaction_level.mean(), df.satisfaction_level.std() / np.sqrt(len(df))), label='all')
x2 = np.linspace(df_low.satisfaction_level.min(), df_low.satisfaction_level.max(), 1000)
ax.plot(x2, stats.norm.pdf(x2, df_low.satisfaction_level.mean(), df_low.satisfaction_level.std() / np.sqrt(len(df_low))), label='low')
x3 = np.linspace(df_medium.satisfaction_level.min(), df_medium.satisfaction_level.max(), 1000)
ax.plot(x3, stats.norm.pdf(x3, df_medium.satisfaction_level.mean(), df_medium.satisfaction_level.std() / np.sqrt(len(df_medium))), label='medium')
x4 = np.linspace(df_high.satisfaction_level.min(), df_high.satisfaction_level.max(), 1000)
ax.plot(x4, stats.norm.pdf(x4, df_high.satisfaction_level.mean(), df_high.satisfaction_level.std() / np.sqrt(len(df_high))), label='high')
ax.legend();


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
Hypothesis Testing
Hypothesis testing is essentially using the data to answer questions of interest.  For example, does a new medication provide any benefit over placebo?  Or is a subset of the population disproportionately more susceptible to a particular disease?  Or is the difference between two companies profits' significant or due to chance alone?
Before doing some hypothesis testing on the HR data, recall that hypothesis typically come in pairs of the form $H_0$, called the null hypothesis, versus $H_a$, called the alternative hypothesis.  The null hypothesis represents the "default" assumption -- that a medication has no effect for example, while the alternative hypothesis represents what exactly we are looking to discover, in the medication case, whether it provides a significant benefit.  Another common case is testing the difference between two means.  Here, the null hypothesis is that there is no difference between two population means, whereas the alternative hypothesis is that there is a difference.  Stated more precisely
$$H_0: \mu_1 - \mu_2 = 0$$
$$H_a: \mu_1 - \mu_2 \ne 0$$
Hypotheses are usually tested by constructing a confidence interval around the test statistic and selecting a "cut-off" significance level denoted $\alpha$.  A typical $\alpha$ significance is 0.05 and is often called a "p-value".  If a test produces a p-value of $\alpha$ or below, then the null hypothesis can be rejected, strengthening the case of the alternative hypothesis.  It is very important to remember that hypothesis testing can only tell you if your hypothesis is statistically significant -- this does not mean that your result may be scientifically significant which requires much more evidence.
In this exercise you'll explore the HR data more and test some hypothesis.
Using the HR data, answer the following.

Compute a confidence interval for satisfaction levels, at the 95% confidence level, of employees who left the company and those who didn't.  Do this using both a t distribution and a normal.  Comment on your...
Can you write Python code for it?

# Compute a confidence interval for satisfaction levels, at the 95% confidence level,
# of employees who left the company and those who didn't.
# Do this using both a t distribution and a normal. Comment on your results.

import statsmodels.stats.api as sm

# checking mean and variance
print('Employees who left mean and variance are {:.5f} and {:.5f}'.
      format(df_left.satisfaction_level.mean(),
             df_left.satisfaction_level.var()))
print('Employees who stayed mean and variance are {:.5f} and {:.5f}'.
      format(df_stayed.satisfaction_level.mean(),
             df_stayed.satisfaction_level.var()))

# using normal distribution
print('\nNormal Distribution\n')

# df_left_norm_confidence = df_left.satisfaction_level.mean() + df_left.satisfaction_level.std() / np.sqrt(len(df_left)) * np.array(stats.norm.ppf([0.025, 0.975]))
# print('Employees who left 95% confidence interval: {}'.format(df_left_norm_confidence))
# print('Employees who left 95% confidence interval: {}'.
#       format(stats.norm.interval(0.95,
#                                  df_left.satisfaction_level.mean(),
#                                  df_left.satisfaction_level.std()/np.sqrt(len(df_left)))))
print('Employees who left 95% confidence interval: {}'.format(sm.DescrStatsW(df_left.satisfaction_level).zconfint_mean(alpha=0.05)))

# df_stayed_norm_confidence = df_stayed.satisfaction_level.mean() + df_stayed.satisfaction_level.std() / np.sqrt(len(df_stayed)) * np.array(stats.norm.interval(0.95))
# print('Employees who stayed 95% confidence interval: {}'.format(df_stayed_norm_confidence))
# print('Employees who stayed 95% confidence interval: {}'.
#       format(stats.norm.interval(0.95,
#                                  df_stayed.satisfaction_level.mean(),
#                                  df_stayed.satisfaction_level.std()/np.sqrt(len(df_stayed)))))
print('Employees who stayed 95% confidence interval: {}'.format(sm.DescrStatsW(df_stayed.satisfaction_level).zconfint_mean(alpha=0.05)))

# using t distribution
print('\nT...


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
In this case the curves are not that far apart as in the previous case but they still seem reasonably distant.
Can you write Python code for it?

# Repeat parts 4-6 on a hypothesis of your choosing.

# Last evaluation mean differs between people who left and people who stayed

print('Last evaluation mean and variance for employees who left: {}, {}'.format(df_left.last_evaluation.mean(),
      df_left.last_evaluation.var()))
print('Last evaluation mean and variance for employees who stayed: {}, {}'.format(df_stayed.last_evaluation.mean(),
      df_stayed.last_evaluation.var()))

print('Test assuming different variance: {}'.format(stats.ttest_ind(df_left.last_evaluation,
                                                                    df_stayed.last_evaluation,
                                                                    equal_var=False)))


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
Probability, Expectation Values, and Variance
The concepts of probability, expectation values, and variance are the bedrock of statistical inference.  Let's begin by employing some of these concepts to see if we can find some interesting paths to go down which may provide some insight into the inner workings of this company.

What is the probability that a randomly selected employee left the company?  What about experienced a work accident?  Also compute the probability that a randomly selected employee left the company and experienced a work accident.
Compute the 25th, 50th, and 90th percentiles for the satisfaction level score for all employees that left the company.  Compare these results to the same percentiles for those that did not leave.  What can you say about the results?
Compute the variance and standard deviation of hours worked.
Compare the variance between the satisfaction levels of employees who left versus those who stayed.  Which is larger?  What does this mean?
Compute the mean satisfaction level for each salary category.  Comment on your results.
Given an employees salary level (low, medium, or high), calculate the probability that they worked more than two standard deviations of the average monthly hours across all groups.  In other words, compute
$$P(hours > 2\sigma \vert salary ) = \dfrac{P(salary \vert hours > 2\sigma) P(hours > 2\sigma)}{P(salary)}$$
What can you say about your results in part 6?
Repeat parts 6 and 7 for
$$P(left \vert salary ) = \dfrac{P(salary \vert left) P(left)}{P(salary)}$$
What is the odds ratio of an employee with a high salary getting a promotion within the past five years versus a low salary employee?  Comment on your results.
Suppose we were to pull a random sample of size 50 of employee satisfaction levels.  What would approximately be the mean of this sample?  What would be the mean of, say, 10 sets of random samples?  Demonstrate your assertions by writing some python code to do just that.
Can you write Python code for it?

n_employees = len(df)
left = df.left.sum()
accident = df.Work_accident.sum()
accident_left = len(df[(df['Work_accident'] == 1) & (df['left'] == 1)])

# probability that a randomly selected employee left the company
print(left/n_employees)
# probability that experienced a work accident
print(accident/n_employees)
# probability that a randomly selected employee left the company and experienced a work accident
print(accident_left/n_employees)

# Creating two dataframes, one for employees who left and one for those who stayed
df_left = df[df['left'] == 1]
df_stayed = df[df['left'] == 0]

# Compute the 25th, 50th, and 90th percentiles for the satisfaction level score for all employees that left the company.
print('Employees who left 25th, 50th and 90th percentile: {}, {}, {}'
      .format(df_left.satisfaction_level.quantile(0.25),
              df_left.satisfaction_level.quantile(0.5),
              df_left.satisfaction_level.quantile(0.9)))
# Compare these results to the same percentiles for those that did not leave. What can you say about the results?
print('Employees who stayed 25th, 50th and 90th percentile: {}, {}, {}'
      .format(df_stayed.satisfaction_level.quantile(0.25),
              df_stayed.satisfaction_level.quantile(0.5),
              df_stayed.satisfaction_level.quantile(0.9)))


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
The Central Limit Theorem
The Central Limit Theorem is perhaps one of the most remarkable results in statistics and mathematics in general.  In short, it says that the distribution of means of independent random variables, sampled from any distribution, tends to approach a normal distribution as the sample size increases.
An example of this would be taking a pair of dice, rolling them, and recording the mean of each result.  The Central Limit Theorem states, that after enough rolls, the distribution of the means will be approximately normal.  Stated formally, the result is
    $$ \bar{X_n} \sim N(\mu, \sigma^2/n) = \frac{\sqrt{n}}{\sigma \sqrt{2\pi}}e^{-n(\bar{X_n} - \mu)^2/\sigma^2}$$
In this exercise, you'll conduct some simulation experiments to explore this idea.
Using the HR data, answer the following.
1. Choose two variables which may be good candidates to test this theorem.
2. Using the variables chosen in part 1, randomly select a set of n = 10 samples and take the mean.  Repeat this 1000 times for each variable.
3. Plot a histogram for each variable used in part 2.  Comment on your results.
4. Repeat parts 2-3 for n = 100, n = 500, and n = 1000.  Comment on your results.
5. Overlay an normal curve on your n = 1000 plots, using the mean and variance computed from the data.  Comment on your results.
Can you write Python code for it?

# Choose two variables which may be good candidates to test this theorem.

central = ['average_montly_hours', 'last_evaluation']

# Using the variables chosen in part 1, randomly select a set of n = 10 samples and take the mean.
# Repeat this 1000 times for each variable.

def make_samples(n_samples, size, seed):
    res = {}

    for var in central:
        random.seed(seed)
        mean = []
        for i in range(n_samples):
            s = random.sample(range(0, n_employees), size)
            mean.append(df.iloc[s][var].mean())

        # from solution using a list comprehension:
        # mean = [df[var].sample(size).mean() for i in range(n_samples)]

        res[var] = mean

    return res

sampsize10 = make_samples(1000, 10, 7)

# Plot a histogram for each variable used in part 2. Comment on your results.

def plot_samples(samples):
    fig, ax = plt.subplots(1, 2, figsize=(16, 5))
    i = 0

    for var in central:
        ax[i].set_title(var)
        ax[i].hist(samples[var], bins=50)
        i += 1

plot_samples(sampsize10)

# Repeat parts 2-3 for n = 100, n = 500, and n = 1000. Comment on your results.

sampsize100 = make_samples(1000, 100, 7)
plot_samples(sampsize100)

sampsize500 = make_samples(1000, 500, 7)
plot_samples(sampsize500)

sampsize1000 = make_samples(1000, 1000, 7)
plot_samples(sampsize1000)

# Overlay an normal curve on your n = 1000 plots, using the mean and variance computed from the data.
# Comment on your results.

fig, ax = plt.subplots(1, 2, figsize=(16, 5))
i = 0

for var in central:
    x = np.linspace(min(sampsize1000[var]), max(sampsize1000[var]), 1000)
    ax[i].set_title(var)
    ax[i].hist(sampsize1000[var], bins=50, normed=True)
    # from solutions: divide by sqrt(1000) to find the std of a sampled distribution!
    # but if I do it results are not so good!
    ax[i].plot(x, stats.norm.pdf(x, loc=pd.Series(sampsize1000[var]).mean(), scale=pd.Series(sampsize1000[var]).std()), color='red')
    i += 1


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
The Poisson Distribution
The Poisson distribution is very versatile but is typically used to model counts per unit time or space, such as the number of ad clicks or arriving flights, each per unit time. It has a PDF given by
$$ P(X = x, \lambda) = \frac{\lambda^x e^{-\lambda}}{x!} $$
where the mean and variance are both equal to $\lambda$
Using the HR data, answer the following.

What variables would be good candidates for modeling with a Poisson distribution?
For each variable in part 1, divide each by salary and fit a Poisson distribution to each.
For each salary level, compute the probability of obtaining at least the mean of each variable — regardless of salary level — by using the Poisson distributions you constructed in part 2.  Comment on your results.

What variables would be good candidates for modeling with a Poisson distribution?

time_spend_company
Can you write Python code for it?

# For each variable in part 1, divide each by salary and fit a Poisson distribution to each.

poisson = ['time_spend_company']

for var in poisson:
    for level in ['low', 'medium', 'high']:
        mu = df[df['salary'] == level][var].mean()
        fit = stats.poisson(mu)
        print('{} {} mean and Poisson fit mean: {:.5f}, {:.5f}'.format(var, level, mu, fit.mean()))

x = np.arange(0, 20)
fig, ax = plt.subplots(3, 1, figsize=(16, 15))
i = 0

for var in poisson:
    for level in ['low', 'medium', 'high']:
        mu = df[df['salary'] == level][var].mean()
        ax[i].set_title(var + ' ' + level + ' salary')
        ax[i].plot(x, stats.poisson.pmf(x, mu))
        ax[i].hist(df[df['salary'] == level][var], bins=20, normed=True)
        i += 1

# For each salary level, compute the probability of obtaining at least the mean of each variable
# regardless of salary level by using the Poisson distributions you constructed in part 2.
# Comment on your results.

for var in poisson:
    global_mean = df[var].mean()
    # print(global_mean)
    for level in ['low', 'medium', 'high']:
        mu = df[df['salary'] == level][var].mean()
        # print(mu, df[df['salary'] == level][var].median())
        print('{} {} salary level probability is {:.5f}'.format(var, level, stats.poisson.sf(global_mean, mu)))


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
The results are almost the same because for $n \rightarrow \infty$ the t distribution tends to the normal.
Can you write Python code for it?

# Use a t-test to test the hypothesis that employees who left the company,
# had lower satisfaction levels than those who did not. If significant, is the mean difference?
# Comment on your results. (Hint: Do the two populations have equal variance?)

print('Test assuming equal variance: {}'.format(stats.ttest_ind(df_left.satisfaction_level,
                                                                df_stayed.satisfaction_level,
                                                                equal_var=True)))
print('Test assuming different variance: {}'.format(stats.ttest_ind(df_left.satisfaction_level,
                                                                   df_stayed.satisfaction_level,
                                                                   equal_var=False)))