I am working on the file ".ipynb_checkpoints/lv_notebook-checkpoint.ipynb".
The first task is:
Set up filters
TODO: Store selection filters as attributes or something that allows us to not have to call them before calculating indicators/qualityfactors
Can you write Python code for it?

    print('{}\nInitiating filters'.format('*'*nr_marks))
    first_filter = core.DataFilter('First filter', file_path = first_data_filter_file_path)
    winter_filter = core.DataFilter('winter_filter', file_path = winter_data_filter_file_path)
    winter_filter.save_filter_file(filter_directory + '/selection_filters/winter_data_filter_save.txt') # mothod available
    summer_filter = core.DataFilter('summer_filter', file_path = summer_data_filter_file_path)
    summer_filter.save_filter_file(filter_directory + '/selection_filters/summer_data_filter_save.txt') # mothod available
    tolerance_filter = core.ToleranceFilter('test_tolerance_filter', file_path = tolerance_filter_file_path)
    print('done\n{}.'.format('*'*nr_marks))


I am working on the file ".ipynb_checkpoints/lv_notebook_workspace_test-checkpoint.ipynb".
The first task is:
Apply filters to selected data
Can you write Python code for it?

    # Use first filter
    filtered_data = raw_data.filter_data(first_filter)

    # Save filtered data (first filter) as a test
    filtered_data.save_data(first_filter_data_directory)

    # Load filtered data (first filter) as a test
    loaded_filtered_data = core.DataHandler('first_filtered')
    loaded_filtered_data.load_data(first_filter_data_directory)


I am working on the file ".ipynb_checkpoints/lv_notebook_workspace_test-checkpoint.ipynb".
The first task is:
Calculate Quality elements
Create an instance of NP Qualityfactor class
Can you write Python code for it?

    qf_NP = core.QualityFactorNP()
    # use set_data_handler to load the selected data to the QualityFactor
    qf_NP.set_data_handler(data_handler = loaded_filtered_data)


I am working on the file ".ipynb_checkpoints/lv_notebook_workspace_test-checkpoint.ipynb".
The first task is:
Data
Select data and create DataHandler instance
Can you write Python code for it?

    # Handler (raw data)
    raw_data = core.DataHandler('raw')
    raw_data.add_txt_file(raw_data_file_path, data_type='column')


I am working on the file ".ipynb_checkpoints/lv_notebook_workspace_test-checkpoint.ipynb".
The first task is:
Set up filters
TODO: Store selection filters as attributes or something that allows us to not have to call them before calculating indicators/qualityfactors
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/matrici2-checkpoint.ipynb".
The first task is:
Composizione di due trasformazioni
Se ad esempio volessi applicare prima una simmetria rispetto all'asse x e poi una rotazione, di solito si indicano le due trasformazioni in questo modo
$$\begin{matrix}
x' = -x\
y' = y\
\end{matrix}
$$
$$\begin{matrix}
x'' = x' \cos \alpha + y' \sin \alpha\
y'' = -x' \sin \alpha + y' \cos \alpha\
\end{matrix}
$$
Sostituendo gli $x'$ e $y'$ nel secondo sistema si ottiene la trasformazione composta.
Un primo fatto notevole che possiamo osservare è che la composizione di trasformazioni non è sempre commutativa. Basta un semplice esempio per rendersene conto. In effetti le trasformazioni non sono altro che funzioni e come sappiamo la composizione di funzioni non è commutativa.
Can you write Python code for it?

from IPython.display import Image
Image(filename='noncomm.png')


I am working on the file ".ipynb_checkpoints/memoize-checkpoint.ipynb".
The first task is:
After the 2015-02-23 CohPy presentation of the above, it was suggested to use the .get() method of dictionaries to avoid the if statement inside helper(). The point is to make the code simpler and more elegant. Unfortunately, the .get() method would not change the dictionary, so the return value from the .get() method would have to be saved. That new memoize function would look like the following. It does succeed in getting rid the if statement, but the 'results[x] = results' looks ugly. Even worse, f(x) is always called, defeating the memoization.
Can you write Python code for it?

def memoize(f):
    results = {}
    def helper(x):
        results[x] = results.get(x, f(x))
        return results[x]
    return helper

@memoize
def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

%timeit fib(n)
fib(n)


I am working on the file ".ipynb_checkpoints/memoize-checkpoint.ipynb".
The first task is:
Another example of how to avoid the parentheses, without having to explicitly index each element of the tuple, is to use a leading '*' to unpack the tuple into separate arguments to the print function.
Can you write Python code for it?

for item in sorted(c.items(), key=(lambda x: x[0]), reverse=True):
    print(*item)
print()
print(sum(c.values()))


I am working on the file ".ipynb_checkpoints/memoize-checkpoint.ipynb".
The first task is:
It is correct. Check the output in the following cell.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/memoize-checkpoint.ipynb".
The first task is:
Taking out the diagnostic counting code made it faster yet.
The caching code makes the fib() function ugly and I don't
like needing to remember to initialize the cache,
so let's separate the caching into a separate function.
Note that memoize() doesn't know anything about the fib() function,
except that it takes a single argument.
memoize() can be applied to many single argument functions.
Can you write Python code for it?

def memoize(f):
    results = {}
    def helper(x):
        if x not in results:
            results[x] = f(x)
        return results[x]
    return helper

def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

fib = memoize(fib)

def foo(n):
    def fib(n):
        if n == 0:
            return 0
        elif n == 1:
            return 1
        else:
            return fib(n-1) + fib(n-2)
    fib = memoize(fib)
    %timeit -n 1 fib(n)

for i in range(20):
    foo(n)


I am working on the file ".ipynb_checkpoints/memoize-checkpoint.ipynb".
The first task is:
That's faster yet, but perhaps deceptively so.
%timeit did fib(n) many times, but initialized the results cache only once.
Next, let's use the syntactic sugar of a function decorator to eliminate the fib = memoize(fib) statement.
The @memoize decorator can be used for many single argument functions.
Can you write Python code for it?

@memoize
def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

%timeit fib(n)


I am working on the file ".ipynb_checkpoints/memoize-checkpoint.ipynb".
The first task is:
The problem is that the naïve function recalculates Fibonacci numbers.
Some code is added to keep track of how many times the function is called for each input value.
Can you write Python code for it?

from collections import Counter

def fib(n):
    global c

    c[n] += 1
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

c = Counter()
fib(n)

for i in sorted(c, reverse=True):
    print(i, c[i])
print()
print(sum(c.values()))


I am working on the file ".ipynb_checkpoints/memoize-checkpoint.ipynb".
The first task is:
Using the .setdefault() method instead of the .get() method looks better, but suffers the same flaw of defeating the memoization by always calling f(x).
Can you write Python code for it?

def memoize(f):
    results = {}
    def helper(x):
        return results.setdefault(x, f(x))
    return helper

@memoize
def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

%timeit fib(n)
fib(n)


I am working on the file ".ipynb_checkpoints/memoize-checkpoint.ipynb".
The first task is:
We can make the function quicker, by calculating each Fibonacci number only once.
When a previously calculated value is needed, we just return that previously calculated number.
Can you write Python code for it?

def fib(n):
    global c
    global cache

    c[n] += 1
    if n in cache:
        return cache[n]
    if n == 0:
        f = 0
    elif n == 1:
        f = 1
    else:
        f = fib(n-1) + fib(n-2)
    cache[n] = f
    return f

# just to make them global
c = None
cache = None

def foo(n):
    global c
    global cache

    c = Counter()
    cache = {}
    fib(n)
%timeit foo(n)


I am working on the file ".ipynb_checkpoints/memoize-checkpoint.ipynb".
The first task is:
We start with a function that calculates fibonacci numbers.
Its code is very much like the definition of Fibonacci numbers.
Can you write Python code for it?

def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)


I am working on the file ".ipynb_checkpoints/mention_detection-checkpoint.ipynb".
The first task is:
Load Wikidata and extract aliases
Can you write Python code for it?

# Unblock to load

# load_wikidata(wikidata_file, entity_alias_output_file)

entity_alias_file = entity_alias_output_file


I am working on the file ".ipynb_checkpoints/mention_detection-checkpoint.ipynb".
The first task is:
Lower the cases
Can you write Python code for it?

alias_to_entity_lower = {}
for als in alias_to_entity.keys():
    als_lower = als.lower()
    if als_lower in alias_to_entity_lower:
        alias_to_entity_lower[als_lower] |= alias_to_entity[als]
    else:
        alias_to_entity_lower[als_lower] = alias_to_entity[als]

experiment_alias(tweet_corpus, alias_to_entity_lower)

for s in alias_to_entity_lower:
    pass

15124./1981


I am working on the file ".ipynb_checkpoints/minuedudata-checkpoint.ipynb".
The first task is:
Cycles though every file found in folder.
Sort each file into folders/dict
Can you write Python code for it?

alfilext = []

allfiltype = []

dirtpy = dict()

for docf in docfiles:
    print docf

with open('/home/wcmckee/test.doc', 'rb') as docx_file:
    result = mammoth.convert_to_html(docx_file)
    print result.value # The generated HTML
        #messages = result.messages # Any messages, such as warnings during conversion

for f in files:
    #print f
    fileName, fileExtension = os.path.splitext(f)
    #print fileName
    npach = ntpath.basename(fileName)
    dirtpy.update({npach : f})
    #print fileExtension
    alfilext.append(fileExtension)

#x = {1: 2, 3: 4, 4: 3, 2: 1, 0: 0}
sorted_x = sorted(dirtpy.items(), key=operator.itemgetter(1))


I am working on the file ".ipynb_checkpoints/minuedudata-checkpoint.ipynb".
The first task is:
Filter html to display only certain extensions. For example - xls.
or doc.
Render the filtered extension
Can you write Python code for it?

print doc

mkind = open(indfil, 'w')

for exf in exlfile:
    #print exf
    opexf = open(exf, 'r')
    #rdex = xlrd.count_records(exf)
    #rdwk = xlrd.open_workbook(opexf)
    #print rdwk
    pandexl = pandas.read_excel(exf)
    #print pandexl.columns
    pandex = pandexl.to_html()
    yourstring = pandex.encode('ascii', 'ignore').decode('ascii')

    mkind.write(str(yourstring))

mkind.close()

document = Document('existing-document-file.docx')
document.save('new-file-name.docx')

for docf in docfiles:
    #print docf
    print docf
    docum = Document(docf)

document = Document()

document.


I am working on the file ".ipynb_checkpoints/mlnn-checkpoint.ipynb".
The first task is:

Feedforward Neural Network
Can you write Python code for it?

# import feedforward neural net
from mlnn import neural_net


I am working on the file ".ipynb_checkpoints/mlnn-checkpoint.ipynb".
The first task is:
<script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS_HTML"></script>
Let's build a 4-layer neural network. Our network has one input layer, two hidden layer and one output layer. Our model can be represented as a directed acyclic graph wherein each node in a layer is connected all other nodes in its succesive layer. The neural net is shown below-

Each node in the hidden layer uses a nonlinear activation function $f(x)$, which computes the outputs from its inputs and transfer these outputs to successive layers. Here we've used $f(x)= tanh(x)$, as our non-linear activation. Its derivative is given by- $f'(x)= 1-tanh(x)^2$.
Our network graph can be represented as-

| Layer No. | Notation |         Value                                |  Variable |
|----------:|-----------:|---------------------------------------------:|----------:|
|  1          |        X   |                                                       $X$|          X|
|   2         |   W1(~)+b1 |                                                 $W1X+b1$|     pre_act1|
|   2         |   tanh     |                                           $tanh(W1X+b1)$|         act1|
|   3         |   W2(~)+b2 |                                  $W2(tanh(W1X +b1))+b2$|     pre_act2|
|   3         |      tanh  |               $tanh(W2(tanhW1X+b1))+b2)$|         act2|
|   4         |   W3(~)+b3  |   $W3(tanh(W2(tanhW1X+b1))+b2)+b3$ |      pre_act3|
|   4         |   softmax ...
Can you write Python code for it?

# Visualize tanh and its derivative
x = np.linspace(-np.pi, np.pi, 120)
plt.figure(figsize=(8, 3))
plt.subplot(1, 2, 1)
plt.plot(x, np.tanh(x))
plt.title("tanh(x)")
plt.xlim(-3, 3)
plt.subplot(1, 2, 2)
plt.plot(x, 1 - np.square(np.tanh(x)))
plt.xlim(-3, 3)
plt.title("tanh\'(x)")
plt.show()


I am working on the file ".ipynb_checkpoints/mlnn-checkpoint.ipynb".
The first task is:
Animate Training:
Can you write Python code for it?

X_, y_ = sklearn.datasets.make_circles(n_samples=400, noise=0.18, factor=0.005, random_state=1)
plt.figure(figsize=(7, 5))
plt.scatter(X_[:, 0], X_[:, 1], s=15, c=y_, cmap=plt.cm.Spectral)
plt.show()

'''
Uncomment the code below to see classification process for above data.
To stop training early reduce no. of iterations.
'''

#new_nn = neural_net([2, 6, 2])
#new_nn.animate_preds(X_, y_, 0.001, 0.0001) # max iterations = 35000


I am working on the file ".ipynb_checkpoints/myFunctions-checkpoint.ipynb".
The first task is:
Check separations by histogram and scatter plot
Can you write Python code for it?

def plot_classcolor(df, x='y', y='x', hue=None):
    sns.lmplot(x, y, data=df, hue=hue, fit_reg=False)
    sns.plt.title("({} vs {})".format(y, x))
    plt.show()

plot_classcolor(df, 'sepal_length', 'sepal_width', hue='species')

def plot_histo_per_class(df, var, target):
    t_list = df[target].unique()
    for t in t_list:
        sns.distplot(
            df[df[target] == t][var], kde=False, norm_hist=True, label=str(t))
    sns.plt.legend()
    sns.plt.show()

plot_histo_per_class(df, 'sepal_length', "species_id")


I am working on the file ".ipynb_checkpoints/myFunctions-checkpoint.ipynb".
The first task is:
Control plots: ROC, Precision-Recall, ConfusionMatrix, top_k, classification report,
Can you write Python code for it?

### VERSIONE CORRETTA A LAVORO

def plot_ROC_PrecisionRecall(y_test, y_pred):
    """
    Plot ROC curve and Precision-Recall plot
    numpy arrays are required.
    """
    fpr_clf, tpr_clf, _ = roc_curve(y_test, y_pred)
    precision, recall, thresholds = precision_recall_curve(y_test, y_pred)
    f1 = np.array([2 * p * r / (p + r) for p, r in zip(precision, recall)])
    f1[np.isnan(f1)] = 0
    t_best_f1 = thresholds[np.argmax(f1)]
    roc_auc = auc(fpr_clf, tpr_clf)
    plt.figure(figsize=(25, 25))
    # plot_ROC
    plt.subplot(221)
    plt.plot(
        fpr_clf,
        tpr_clf,
        color='r',
        lw=2,
        label='ROC curve (area = %0.2f)' % roc_auc)
    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='-')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver operating characteristic')
    plt.legend(loc="lower right")
    # plot_PrecisionRecall
    plt.subplot(222)
    plt.plot(
        recall, precision, color='r', lw=2, label='Precision-Recall curve')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.title('Precison-Recall curve')
    plt.legend(loc="lower right")

    plt.show()
    return {"roc_auc": roc_auc, "t_best_f1": t_best_f1}

def plot_ROC_PR_test_train(y_train, y_test, y_test_pred, y_train_pred):
    """Plot ROC and Precision-Recall curve for test and train.
    Return the auc for test and train"""
    roc_auc_test = plot_ROC_PrecisionRecall(y_test, y_test_pred)
    roc_auc_train = plot_ROC_PrecisionRecall(y_train, y_train_pred)
    return roc_auc_test, roc_auc_train


I am working on the file ".ipynb_checkpoints/myFunctions-checkpoint.ipynb".
The first task is:
FastFourier Transformation
Can you write Python code for it?

from scipy.fftpack import fft, rfft, irfft, fftfreq
def rfourier_transformation(df, var, pass_high=-1, pass_low=-1, verbose=True, plot=True):
    """Return the signal after low and high filter applied.
    Use verbose and plot to see stats and plot the signal before and after the filter.
    """
    low = pass_high
    high = pass_low

    if (high < low) and (high>0):
        print("Cannot be pass_low < pass_high!!")
        return -1

    time = pd.Series(df.index.values[1:10] -
                     df.index.values[:10 - 1])  # using the first 10 data
    dt = time.describe()['50%']
    if (verbose):
        print("""
          sampling time: {0} s
          sampling frequency: {1} hz
          max freq in rfft: {2} hz
          """.format(dt, 1 / dt, 1 / (dt * 2), 1 / (dt)))

    signal = df[var]
    freq = fftfreq(signal.size, d=dt)
    f_signal = rfft(signal)

    m = {}
    if (low > 0):
        f_signal_lowcut = f_signal.copy()
        f_signal_lowcut[(freq < low)] = 0
        cutted_signal_low = irfft(f_signal_lowcut)
        m['low'] = 1

    if (high > 0):
        f_signal_highcut = f_signal.copy()
        f_signal_highcut[(freq > high)] = 0
        cutted_signal_high = irfft(f_signal_highcut)
        m['high'] = 1

    if (high > 0) & (low > 0):
        f_signal_bwcut = f_signal.copy()
        f_signal_bwcut[(freq < low) | (freq > high)] = 0
        cutted_signal_bw = irfft(f_signal_bwcut)
        m['bw'] = 1
        m['low'] = 2
        m['high'] = 3

    n = len(freq)

    if (plot):
        f, axarr = plt.subplots(len(m) + 1, 1, sharex=True, figsize=(18,15))
        f.canvas.set_window_title(var)

        # time plot
        axarr[0].plot(signal)
        axarr[0].set_title('Signal')
        if 'bw' in m:
            axarr[m['bw']].plot(df.index, cutted_signal_bw)
            axarr[m['bw']].set_title('Signal after low-high cut')
        if 'low' in m:
            axarr[m['low']].plot(df.index, cutted_signal_low)
            axarr[m['low']].set_title('Signal after high filter (low...


I am working on the file ".ipynb_checkpoints/myFunctions-checkpoint.ipynb".
The first task is:
PCA
Can you write Python code for it?

from sklearn import decomposition

def pca_decomposition(df):
    """Perform sklearn PCA. The returned components are already ordered by the explained variance"""
    pca = decomposition.PCA()
    pca.fit(df)
    return pca

def pca_stats(pca):
    print("variance explained:\n", pca.explained_variance_ratio_)
    print("pca components:\n", pca.components_)

def plot_classcolor(df, x='y', y='x', hue=None):
    sns.lmplot(x, y, data=df, hue=hue, fit_reg=False)
    sns.plt.title("({} vs {})".format(y, x))
    plt.show()

def add_pca_to_df(df, allvars, pca):
    df[["pca_" + str(i) for i, j in enumerate(pca.components_)
        ]] = pd.DataFrame(pca.fit_transform(df[allvars]))

pca = pca_decomposition( df[['sepal_length', 'sepal_width', 'petal_length', 'petal_width']] )

pca_stats(pca)

add_pca_to_df(df, ['sepal_length', 'sepal_width', 'petal_length', 'petal_width'], pca)

plot_classcolor(df, 'pca_0', 'pca_1', 'species_id')


I am working on the file ".ipynb_checkpoints/myFunctions-checkpoint.ipynb".
The first task is:
Progress Bar
There are many packages to create a progress bar in python, the one I use is tqdm
- tqdm: progress bar in a pandas DataFrame progress_apply (tqdm state that it will not noticeably slow pandas down)  https://pypi.python.org/pypi/tqdm
others are:
- progressbar: with each iterable (mainly for) https://pypi.python.org/pypi/progressbar2
- https://github.com/niltonvolpato/python-progressbar
Can you write Python code for it?

# Simple bar, the one to be used in a general python code
import tqdm
for i in tqdm.tqdm(range(0, 1000)):
    pass

# Bar to be used in a jupyter notebook
for i in tqdm.tqdm_notebook(range(0, 1000)):
    pass

# custom update bar
import time
tot = 4000
bar = tqdm.tqdm_notebook(desc='Status ', total=tot, mininterval=0.5, miniters=5, unit='cm', unit_scale=True)
# with the file options you can show the progress bar into a file
# mininterval: time in seconds to see an update on the progressbar
# miniters: Tweak this and `mininterval` to get very efficient loops, if 0 will only use mininterval
# unit_scale: use international scale for the units (k, M, m, etc...)
# bar_format: specify the bar format, default is '{l_bar}{bar}{r_bar}'. It can impact the performance if you ask for complicate bar format
# unit_divisor: [default: 1000], ignored unless `unit_scale` is True
# ncols: The width of the entire output message. If specified, dynamically resizes the progressbar to stay within this bound.

for l in range(0, tot):
    if ((l-1) % 10) == 0:
        bar.update(10)
    if l % 1000 == 0:
        bar.write('to print something without duplicate the progress bar (if you are using tqdm.tqdm instead of tqdm.tqdm_notebook)')
        print('or use the simple print if you are using tqdm.tqdm_notebook')
    time.sleep(0.001)

# with this extension you can use tqdm_notebook().pandas(...) instead of tqdm.pandas(...)
from tqdm import tqdm_notebook
!jupyter nbextension enable --py --sys-prefix widgetsnbextension

import pandas as pd
import numpy as np
import time
df = pd.DataFrame(np.random.randint(0, int(1e8), (100, 3)))

# Create and register a new `tqdm` instance with `pandas`
# (can use tqdm_gui, optional kwargs, etc.)
print('set tqdm_notebook for pandas, show the bar')
tqdm_notebook().pandas()
# Now you can use `progress_apply` instead of `apply`

print('example usage of progressbar in a groupby pandas statement')
df_g = df.groupby(0).progress_apply(lambda x: time.sleep(0.01))

print('example usage of progressbar in an...


I am working on the file ".ipynb_checkpoints/myFunctions-checkpoint.ipynb".
The first task is:
Save python object with pickle
Can you write Python code for it?

# save in pickle with gzip compression
import pickle
import gzip

def save(obj, filename, protocol=0):
    file = gzip.GzipFile(filename, 'wb')
    file.write(pickle.dumps(obj, protocol))
    file.close()

def load(filename):
    file = gzip.GzipFile(filename, 'rb')
    buffer = ""
    while True:
        data = file.read()
        if data == "":
            break
        buffer += data
    obj = pickle.loads(buffer)
    file.close()
    return obj


I am working on the file ".ipynb_checkpoints/myFunctions-checkpoint.ipynb".
The first task is:
Xgboost
To instal xgboost: https://github.com/dmlc/xgboost/tree/master/python-package
- conda install py-xgboost (pip install xgboost)
Can you write Python code for it?

import xgboost

#### VERSIONE GIUSTA A LAVORO

def plot_var_imp_xgboost(model, mode='gain', ntop=-1):
    """Plot the vars imp for xgboost model, where mode = ['weight','gain','cover']
    'weight' - the number of times a feature is used to split the data across all trees.
    'gain' - the average gain of the feature when it is used in trees
    'cover' - the average coverage of the feature when it is used in trees
    """
    importance = model.get_score(importance_type=mode)
    importance = sorted(
        importance.items(), key=operator.itemgetter(1), reverse=True)
    if ntop == -1: ntop = len(importance)
    importance = importance[0:ntop]
    my_ff = np.array([i[0] for i in importance])
    imp = np.array([i[1] for i in importance])
    indices = np.argsort(imp)
    pos = np.arange(len(my_ff[indices])) + .5
    plt.figure(figsize=(20, 0.75 * len(my_ff[indices])))
    plt.barh(pos, imp[indices], align='center')
    plt.yticks(pos, my_ff[indices], size=25)
    plt.xlabel('rank')
    plt.title('Feature importances (' + mode + ')', size=25)
    plt.grid(True)
    plt.show()
    return


I am working on the file ".ipynb_checkpoints/my_1st_trial-checkpoint.ipynb".
The first task is:
Apparantly, the smaller Pclass, the higher survive rate. Richer is better, right?
Sex vs. Survived
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/my_1st_trial-checkpoint.ipynb".
The first task is:
Female has way higher survive rate than male. Sex is another strong corelate feature to the survival.
Pclass and Sex vs. Survived
Now let's combine the Pclass and Sex to see what we got.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/my_1st_trial-checkpoint.ipynb".
The first task is:
Hmm... There are some data missing. Age could be an important feature. Cabin seems like a useless feature and I am going to discard it. Well, my 1st question, how do you decide which feature to be used and which not?
After i read other people's analysis, they show me this:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/my_1st_trial-checkpoint.ipynb".
The first task is:
Relationship between Features and Survival
Let's find out each feature's contribution to the survival. Well, maybe it is not correct. The feature might not contribute to the survival, just co-relate to it. :-)
Let's checkout the total survive rate
Can you write Python code for it?

sns.countplot(x='Survived', data=df_train)
plt.show()

df_train['Percentage'] = 1 # this is a helper colume
df_train[['Percentage','Survived']].groupby('Survived').count().apply(lambda x: (100 * x)/x.sum())


I am working on the file ".ipynb_checkpoints/my_1st_trial-checkpoint.ipynb".
The first task is:
Well, I don't really have any idea how to handle these data. So let's just take a look at them. Let's start from the trainning data.
Can you write Python code for it?

df_train.head()

df_train.describe()

df_train.info()

df_train.isnull().sum()


I am working on the file ".ipynb_checkpoints/network_basics-checkpoint.ipynb".
The first task is:
PLOT GRAPH
NOTE TYPE OF LAYOUTS
'circular_layout',
'random_layout',
'shell_layout',
'spring_layout',
'spectral_layout',
'fruchterman_reingold_layout'
Can you write Python code for it?

# choose layoutl pos=position for nodes
pos = nx.fruchterman_reingold_layout(g)

# draw network
nx.draw(g, pos, with_labels = True, node_size=800, node_color='pink', edge_color='grey')

# label edges
edge_labels = dict([((u,v,),d['distance']) for u,v,d in g.edges(data=True)])
nx.draw_networkx_edge_labels(graph, pos, edge_labels=edge_labels)


I am working on the file ".ipynb_checkpoints/network_basics-checkpoint.ipynb".
The first task is:
SHORTEST PATHS
Can you write Python code for it?

# shortest path
nx.shortest_path(graph, source=1, target=3, weight='distance')

# shortest path
paths = nx.all_shortest_paths(graph, source=1, target=7, weight='distance')
for path in paths:
    print path

# all simple paths without weight
path = nx.all_simple_paths(graph, source=1, target=7)
for i in path:
    print i


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:

nn.set_hidden_layer()は同時にシグモイド関数で変換する前の中間層も作る。

set_output_layer()は同時にシグモイド関数で変換する前の出力層、さらに教師データを入れる配列も作る。

nn.setup()で入力層ー中間層、中間層ー出力層間の重みを入れる配列を作成する。
nn.initialize()で重みを初期化する。重みは-1/√d ≤ w ≤ 1/√d (dは入力層及び中間層の数)の範囲で一様分布から決定される。
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
NNのクラスはすでにNN.pyからimportしてある。
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
nn.calculation()は学習せずに入力層から出力層の計算を行う。nn.simulate()内にも用いられている。
次に実際に学習を行う。サンプルデータは、
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)
print X, Y


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
nn.simulate(N, eta)は引数に更新回数と学習率をとる。普通はN=1で行うべきかもしれないが、工夫として作成してみた。N回学習した後に出力層を返す。
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
nn.supervised_function(f, idata)は教師データを作成する。引数は関数とサンプルデータをとる。
Can you write Python code for it?

idata = [1, 2]

nn.supervised_function(f, idata)


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
コブ・ダクラス型生産関数と課題文で例に出された関数を用いる。
いずれも定義域は0≤x≤1である。

コブ・ダグラス型生産関数は以下の通りである。

z = x_1**0.5*x_2*0.5

Can you write Python code for it?

def example1(x_1, x_2):
    z = x_1**0.5*x_2*0.5
    return z

fig = pl.figure()
ax = Axes3D(fig)

X = np.arange(0, 1, 0.1)
Y = np.arange(0, 1, 0.1)

X, Y = np.meshgrid(X, Y)
Z = example1(X, Y)

ax.plot_surface(X, Y, Z, rstride=1, cstride=1)

pl.show()


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
上手く近似できないので中間層の数を変えてみる。5層にしてみる。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

nn = NN()
nn.set_network(h=5)

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

for i in range(1000):
        l = np.random.choice([i for i in range(len(a))])
        m = nn.main(1, f, [a[l], b[l]], 0.5)

for x in X:
    for y in Y:
        idata = [x, y]
        c = np.append(c, nn.realize(f, idata))

fig = pl.figure()
ax = Axes3D(fig)

ax.scatter(a, b, c)

pl.show()


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
以下に使い方を説明する。
初めに、このコブ・ダグラス型生産関数を用いる。
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
例えば(0, 0)を入力すると0.52328635を返している（つまりa[0]とb[0]を入力して、c[0]の値を返している）。
ここでは交差検定は用いていない。
Can you write Python code for it?

fig = pl.figure()
ax = Axes3D(fig)

ax.scatter(a, b, c)

pl.show()


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
同じ方法でコブ・ダグラス型生産関数を学習させた様子をアニメーションにしてみた。この方法が何の意味を持つかは分からないが学習はまあまよくできていた。
mp4ファイルで添付した。
[考察]ここにはコードを書けなかったが、サンプル数が多くなった時は、ミニバッチ式の学習を試すべきだと思った。
最後に交差検定を行う。
初めに学習回数が極めて少ないNNである。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

evl = np.array([])

for i in range(len(a)):
    nn = NN()
    nn.set_network()
    for j in range(1):
        l = np.random.choice([i for i in range(len(a))])
        if l != i:
            m = nn.main(1, f, [a[l], b[l]], 0.5)
    idata = [a[i], b[i]]
    est = nn.realize(f, idata)
    evl = np.append(evl, math.fabs(est - nn.supervised_data))

np.average(evl)


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
目標と比べると大きく異なる。
サンプル数を、
Can you write Python code for it?

X = np.arange(-1, 1, 0.1)
Y = np.arange(-1, 1, 0.1)
print X, Y


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
確率的勾配降下法を100回繰り返したが見た感じから近づいている。回数を10000回に増やしてみる。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

nn = NN()
nn.set_network()

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

for i in range(10000):
        l = np.random.choice([i for i in range(len(a))])
        m = nn.main(1, f, [a[l], b[l]], 0.5)

for x in X:
    for y in Y:
        idata = [x, y]
        c = np.append(c, nn.realize(f, idata))

fig = pl.figure()
ax = Axes3D(fig)

ax.scatter(a, b, c)

pl.show()


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
見た感じ随分近づいているように見える。
同様のことを課題の例で使われた関数でも試してみる。
Can you write Python code for it?

x_1 = Symbol('x_1')
x_2 = Symbol('x_2')
f = (1+sin(4*math.pi*x_1))*x_2*1/2

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

nn = NN()
nn.set_network()

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

for i in range(1000):
        l = np.random.choice([i for i in range(len(a))])
        m = nn.main(1, f, [a[l], b[l]], 0.5)

for x in X:
    for y in Y:
        idata = [x, y]
        c = np.append(c, nn.realize(f, idata))

fig = pl.figure()
ax = Axes3D(fig)

ax.scatter(a, b, c)

pl.show()


I am working on the file ".ipynb_checkpoints/nn-checkpoint.ipynb".
The first task is:
誤差の平均であるので小さい方よい。
学習回数を大幅に増やした結果、精度が上がった。
次に回数を100回にして、中間層の数を2と5で精度を比較する。
Can you write Python code for it?

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

evl = np.array([])

for i in range(len(a)):
    for j in range(100):
        nn = NN()
        nn.set_network()
        l = np.random.choice([i for i in range(len(a))])
        if l != i:
            m = nn.main(1, f, [a[l], b[l]], 0.5)
    idata = [a[i], b[i]]
    est = nn.realize(f, idata)
    evl = np.append(evl, math.fabs(est - nn.supervised_data))

np.average(evl)

X = np.arange(0, 1, 0.2)
Y = np.arange(0, 1, 0.2)

a = np.array([])
b = np.array([])
c = np.array([])

for x in X:
    for y in Y:
        a = np.append(a, x)
        b = np.append(b, y)

evl = np.array([])

for i in range(len(a)):
    for j in range(100):
        nn = NN()
        nn.set_network(h=5)
        l = np.random.choice([i for i in range(len(a))])
        if l != i:
            m = nn.main(1, f, [a[l], b[l]], 0.5)
    idata = [a[i], b[i]]
    est = nn.realize(f, idata)
    evl = np.append(evl, math.fabs(est - nn.supervised_data))

np.average(evl)


I am working on the file ".ipynb_checkpoints/noise-checkpoint.ipynb".
The first task is:
And now we have to adjust our noise function to incorporate the smoothstep function.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/noise-checkpoint.ipynb".
The first task is:
And now we redefine our noise function so it will interpolate between the x and y value as well.
Can you write Python code for it?

def noise(x, y):
    xi, yi = int(x), int(y)
    tx, ty = x - xi, y - yi
    x0 = xi % len(r)
    x1 = 0 if x0 == (len(r) - 1) else x0 + 1
    y0 = yi % len(r[0])
    y1 = 0 if y0 == (len(r) - 1) else y0 + 1
    c00 = r[x0][y0]
    c01 = r[x1][y0]
    c10 = r[x0][y1]
    c11 = r[x1][y1]
    n0 = np.interp(tx, [0, 1], [c00, c01])
    n1 = np.interp(tx, [0, 1], [c10, c11])
    return np.interp(ty, [0, 1], [n0, n1])

img = np.array([[noise(x, y) for x in range(16)] for y in range(16)])
plt.imshow(img, plt.cm.ocean)


I am working on the file ".ipynb_checkpoints/noise-checkpoint.ipynb".
The first task is:
Note that instead of interpolating on tx and ty directly we are using the smoothstep-ed values of sx and sy instead.
Can you write Python code for it?

def gen_im(size=(64,64), c = lambda x: x):
    w, h = size
    pixels = [[noise(x, y, c) for x in range(w)] for y in range(h)]
    return np.array(pixels)

seed(1, (32, 32))
img = gen_im(c = smoothstep)
plt.imshow(img, cmap=plt.cm.ocean)

MAX_VERTICES = 32
MAX_VERTICES_MASK = MAX_VERTICES - 1


I am working on the file ".ipynb_checkpoints/noise-checkpoint.ipynb".
The first task is:
There's still a pattern but that is to be expected with our still reasonably small r vector. It does like a lot better already but another thing that is noticeable is that it appears to be a bit blocky. This is because of the linear interpolation we're using to calculate the final noise values. We can fix this by introducing the smoothstep function.
smoothstep
Can you write Python code for it?

def smoothstep(t):
    return 3 * t**2 - 2 * t**3

x = np.linspace(0, 1, 1000)
y = smoothstep(x)
plt.plot(x, y)


I am working on the file ".ipynb_checkpoints/noise-checkpoint.ipynb".
The first task is:
We can see clearly see a pattern here. That's because the dimensions of our r table are very small, only 4 in each axis. We'll get much better noise if we make it bigger.
Can you write Python code for it?

seed(0, (16,16))
img = np.array([[noise(x, y) for x in range(32)] for y in range(32)])
plt.imshow(img, cmap=plt.cm.ocean)


I am working on the file ".ipynb_checkpoints/noise-checkpoint.ipynb".
The first task is:
We can use this table to define a function noise that given an integer value x will return a value in the range of 0 to 1. The r vector only has a limited amount of values. In order to make it work for any value of x we can perform a modulo operation with the length of the r vector.
Can you write Python code for it?

def noise(x):
    x = int(x % len(r))
    return r[x]


I am working on the file ".ipynb_checkpoints/noise-checkpoint.ipynb".
The first task is:
value noise
The image above is an example of random noise. Although useful we need something a bit more controllable. For this we need a function that produces value noise instead of just random noise. Essentially this is just a function that given a vector of inputs, produces an float in the range of 0 to 1. For now we'll focus on one-dimensional noise but we'll deal with noise in more dimensions later.
There's various ways to produce value noise but we're gonna use a method that utilizes a seeded vector of random values. For no particular reason we're going to call this vector r and we'll use a numpy array to hold it. For convenience we will also write a function seed that we can use to initialize the table.
Can you write Python code for it?

def seed(n, shape=(8,)):
    global r
    np.random.seed(n)
    r = np.random.ranf(shape)


I am working on the file ".ipynb_checkpoints/noise_old-checkpoint.ipynb".
The first task is:
Now we setup a noise field and define a helper function noise1 in order to get our coherent noise.
Can you write Python code for it?

samples = 32
gen = noise.Field(d=(samples,))

def noise1(x, curve = lambda x: x):
    xi = int(x)
    xmin = xi % samples
    xmax = 0 if xmin == (samples - 1) else xmin + 1
    t = x - xi
    return noise.lerp(gen(xmin), gen(xmax), curve(t))

x = np.linspace(0, 10, 100)
y1 = [noise1(x) for x in x]
y2 = [noise1(x, noise.ss5) for x in x]
plt.plot(x, y1, '--')
plt.plot(x, y2)


I am working on the file ".ipynb_checkpoints/noise_old-checkpoint.ipynb".
The first task is:
We can do better though by using one of the smoothstep functions. Instead of calculating ${v_t}$ directly we can do some tricks on ${t}$ to modify the outcome.
For convience let's start with the ss3 function and plot it so we know what it looks like:
Can you write Python code for it?

x = np.linspace(0, 1.0, 100)
y = noise.ss3(x)
plt.plot(x, y)


I am working on the file ".ipynb_checkpoints/noise_old-checkpoint.ipynb".
The first task is:
linear interpolation
We need a function ${f}$ that, given values ${v_0}$ and ${v_1}$ and some interval ${t}$ where $0 \le {t} \le 1$, returns an interpolated value between ${v_0}$ and ${v_1}$.
The best way to start is with linear interpolation and that's what the lerp function does.
Let's assume we have to values ${v_0}$ and ${v_1}$:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/noise_old-checkpoint.ipynb".
The first task is:
noise field
For instance, to create a hypercube of noise we could do something like this:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/noise_old-checkpoint.ipynb".
The first task is:
perlin noise
Can you write Python code for it?

x = np.linspace(0, 4, 100)
y1 = [perlin.noise2d(x, 0) for x in x]
y2 = [0.5 * perlin.noise2d(x * 2, 0) for x in x]
y3 = [0.25 * perlin.noise2d(x * 4, 0) for x in x]
y4 = [0.125 * perlin.noise2d(x * 8, 0) for x in x]
plt.plot(x, y1)
plt.plot(x, y2)
plt.plot(x, y3)
plt.plot(x, y4)

x = np.linspace(0, 10, 100)
y = [perlin.fbm(x, 0) for x in x]
plt.plot(x, y)


I am working on the file ".ipynb_checkpoints/numero_de_datos_perdidos_slide-checkpoint.ipynb".
The first task is:
Generamos los datasets de todos los días

Notas:

Se leeran los archivos guardados en una carpeta por dia

El problema de la configuración actual de la telemetría del
    auto solar es que usa software hecho para registrar datos de
    un motor a la vez, lo que nos obliga a usar dos instancias
    cada una de las cuales escribe un archivo de datos diferente,
    sin embargo el protocolo tcp/ip si pierde momentáneamente
    la conexión intenta reconectar automáticamente y los programas
    se reconectan al primer motor que detectan.

Esto genera archivos mezclados, datos duplicados y pérdidas de
    datos

Cree un script que automatiza separar los motores, fusiona
    los datos duplicados y junta en una sola tabla ambos motores.

En primer lugar se extraen los datos de todos los archivos
    de cada día y se genera una lista de tablas separadas por motor
Can you write Python code for it?

dia1 = ext.extraer_data('dia1')

cd ..

dia2 = ext.extraer_data('dia2')

cd ..

dia3 = ext.extraer_data('dia3')

cd ..

dia4 = ext.extraer_data('dia4')


I am working on the file ".ipynb_checkpoints/numero_de_datos_perdidos_slide-checkpoint.ipynb".
The first task is:
Importamos las librerías creadas para trabajar
Can you write Python code for it?

import ext_datos as ext
import procesar as pro
import time_plot as tplt


I am working on the file ".ipynb_checkpoints/numero_de_datos_perdidos_slide-checkpoint.ipynb".
The first task is:
Medimos el número de filas de los archivos (con y sin valores perdidos)
Can you write Python code for it?

con1 , sin1 = len(motoresdia1), len(motoresdia1.dropna())

con2 , sin2 = len(motoresdia2), len(motoresdia2.dropna())

con3 , sin3 = len(motoresdia3), len(motoresdia3.dropna())

con4 , sin4 = len(motoresdia4), len(motoresdia4.dropna())

d = {'datos_con_valores_perdidos':[con1,con2,con3,con4],\
     'datos_sin_valores_perdidos':[sin1,sin2,sin3,sin4]}

tabla1= pd.DataFrame(d,index=[1,2,3,4])


I am working on the file ".ipynb_checkpoints/numero_de_datos_perdidos_slide-checkpoint.ipynb".
The first task is:
Se calcula la relación entre los datos efectivos por motor y
    el total que registaba cada archivo
Can you write Python code for it?

p1d1, p2d1 = round(m1d1/float(con1),4) , round(m2d1/float(con1),4)

p1d2, p2d2 = round(m1d2/float(con2),4) , round(m2d2/float(con2),4)

p1d3 , p2d3 = round(m1d3/float(con3),4) , round(m2d3/float(con3),4)

p1d4 , p2d4 = round(m1d4/float(con4),4) , round(m2d4/float(con4),4)

labels = {'motor1':[100*p1d1, 100*p1d2, 100*p1d3, 100*p1d4],'motor2':\
          [100*p2d1, 100*p2d2, 100*p2d3, 100*p2d4]}

tabla2 = pd.DataFrame(labels, index=[1,2,3,4])

tabla2


I am working on the file ".ipynb_checkpoints/numero_de_datos_perdidos_slide-checkpoint.ipynb".
The first task is:
Se procesan las listas anteriores, se concatenan por motor según
    la hora de los registros y se rellenan los espacios vacíos con
    datos NaN, luego se juntan de costado las tablas (join) y se
    le añade el sufijo _m1 y _m2 para diferenciar las columnas
Can you write Python code for it?

motoresdia1 = pro.procesar(dia1)

motoresdia2 = pro.procesar(dia2)

motoresdia3 = pro.procesar(dia3)

motoresdia4 = pro.procesar(dia4)


I am working on the file ".ipynb_checkpoints/numero_de_datos_perdidos_slide-checkpoint.ipynb".
The first task is:
Visualizamos los datos de la tabla
La primera columna corresponde al día de competencia
La segunda indica los datos obtenidos con valores perdidos de uno u otro motor
La tercera corresponde a los datos en donde ambos motores fueron registrados
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Analiza narodnosti
Zanima nas, koliko je smučarjev določene narodnosti. Najprej jih preštejmo, nato pa ponazorimo to z grafom:
Can you write Python code for it?

smucarji['drzava'].value_counts().head(10)

smucarji['drzava'].value_counts().plot(kind='bar', figsize = (12,6))


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Analiza smuči
Najprej si oglejmo, katere znamke smuči so najpogostejše v svetovnem pokalu:
Can you write Python code for it?

smucarji['smuci'].value_counts()

smucarji['smuci'].value_counts().plot(kind='pie', figsize=(6,6))


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Eva-Maria Brem je torej najpogosteje tekmuje v slalomu in veleslalomu. Ponazorimo to še z grafom:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Iz tabel je razvidno, da so njeni razultati v slalomu v vačini na repu trideseterice, med tem ko se v veleslalomu uvršča med 5 najboljših. To se še lepše vidi z grafov:
Can you write Python code for it?

brem[slalom]['mesto1'].value_counts().plot(kind='bar', title="Rezultati E. M. Brem v slalomu")

brem[veleslalom]['mesto1'].value_counts().plot(kind='bar', title='Rezultati E. M. Brem v veleslalomu')


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Kdo pa je osvojil največ točk? Poglejmo 10 najboljših:
Can you write Python code for it?

sezona.groupby(['id']).agg({'tocke':sum}).nlargest(columns = 'tocke', n = 10)


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Ker si z datumi ne moremo pomagati, glejmo le dosežena mesta:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Najprej sem spletne strani FIS pobrala podatke o smučarjih in njihovih id številkah na spletišču FIS. Id-je sem potrebovala za sestavljanje url naslovov posameznih športnikov. Zbrane podatke sem nato spravila v datoteko smucarji.csv.
Can you write Python code for it?

pot="csv-datoteke/smucarji.csv"
smucarji = pd.read_csv(pot, parse_dates=['rojstvo'], index_col='id')


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Nato sem za vsakega od tekmovalcev s strani z njegovimi rezultati (npr. Eva-Maria Brem) pobrala podatke o vsaki tekmi: datum, prizorišče, disciplino, uvrstitev, zaostanek.
Can you write Python code for it?

pot_brem = "csv-datoteke/BREM  Eva-Maria.csv"
brem = pd.read_csv(pot_brem, parse_dates=['datum'])


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Opazimo, da ena uvrstitev odstopa: to je njena prva uvrstitev v sezoni: "le" 8. mesto v Soeldnu 24. 10. 2015.
Poglejmo si še analizo po prizoriščih. Postavimo si vprašanje, kje dosega boljše rezultate.
Can you write Python code for it?

po_krajih = brem.groupby(['kraj'])['tocke'].sum() / brem['tocke'].sum()
po_krajih.nlargest(7)


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Podoben graf si poglejmo posebej še za veleslalom:
Can you write Python code for it?

tabela[tabela['disciplina'] == 'Giant Slalom']['mesto1'].value_counts().plot(kind='pie')


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Podobno si lahko pogledamo, katerim proizvajalcem smuči najbolj zaupajo smučarji iz avstrije:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Tabela izgleda tako:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Tabela za Evo-Mario Brem:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Tabelam zdaj dodamo stolpce:
Can you write Python code for it?

vse['mesto1'] = vse['mesto'].map(pretvori)
brem['mesto1'] = brem['mesto'].map(pretvori)
vse['tocke'] = vse['mesto1'].map(pretvori_2)
brem['tocke'] = brem['mesto1'].map(pretvori_2)
brem[:5]


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Tekmovalec ali tekmovalka z največ nastopi je nastopila 21-krat. Kdo pa je to?
Can you write Python code for it?

nova = sezona.groupby(['id']).size()
nova.nlargest(6)


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
V Are-ju je osvojila kar 20 % vseh svojih točk. Ali tam dosega tudi najvišja mesta?
Can you write Python code for it?

po_krajih1 = brem.groupby(['kraj'])['tocke'].mean()
po_krajih1.nlargest(7)


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
V kasnejši analizi se pojavi težava, da so podatki o uvrstitvi lahko številke ali besedilo (npr. DNQ1, DNF1, DSQ2 in DNS1), ki označuje odstope, diskvalifikacije in podobne anomalije.
Zato tabelam dodamo nov stolpec mesto1, kjer besedilne podatke identificiramo z 0. Tu nas ne zanima, zakaj tekmovalec ni osvojil točk.
Can you write Python code for it?

def pretvori(bes):
    if bes in ['DNQ1', 'DNF1', 'DSQ2', 'DNS1','DNF2','DSQ1','DNS2','DNQ2','DNF','DNC1','DSQ','DNS']:
        return 0
    else:
        return int(bes)


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Veleslalom je torej precej očitno disciplina, ki ji prinaša največ točk.
Poglejmo si še podatke za letošnjo sezono. Prva tekma je bila 24. 10. 2015 v Soeldnu.
Can you write Python code for it?

prvi_del = brem[brem['datum'].dt.year == 2016]
drugi_del = brem[(brem['datum'].dt.month > 9) & (brem['datum'].dt.year == 2015)]
tabela = prvi_del.append(drugi_del)
tabela


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Z analizo enega samega proizvajalca oz. ene same države ne dobimo širše slike, zato si oglejmo graf, ki za vsako državo prikaže, smuči katerega proizvajalca uporabljajo njeni reprezentanti.
Can you write Python code for it?

oboje = smucarji.groupby(['smuci','drzava']).size()
priprava = oboje.unstack().plot(kind='barh', stacked=True, figsize=(20, 14), fontsize=12)
priprava.legend(loc=(0.02,1), ncol=6)


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Zanima nas, koliko nastopov so zbrali tekmovalci v letošnji sezoni. Prikazujemo število nastopov za 10 tekmovalcev.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Zgornja tabela prikazuje predelano tabelo za Evo-Mario Brem. Enako smo naredili s tabelo z vsemi podatki:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Če bomo želeli delati analizo skupnega seštevka, moramo pretvoriti mesto tudi v točke. Definiramo seznam 'tocke', v katerega na i-to mesto (i teče od 0 do 30) zapišemo, koliko točk tekmovalec dobi za osvojeno i-to mesto. Nato napišemo še funkcijo, ki nam pretvori mesto v osvojene točke.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Čeprav najpogosteje tekmuje v slalomu in veleslalomu, pa to nista nujno disciplini, v katerih dosega najboljše rezultate. Najprej si poglejmo, kakšni so njeni rezultati v slalomu in nato še veleslalomu:
Can you write Python code for it?

slalom = brem['disciplina'] == 'Slalom'
brem[slalom][:10]

veleslalom = brem['disciplina'] == 'Giant Slalom'
brem[veleslalom][:10]


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-Copy1-checkpoint.ipynb".
The first task is:
Še njihova imena:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-checkpoint.ipynb".
The first task is:
Eva-Maria Brem je torej najpogosteje tekmuje v slalomu in veleslalomu. Ponazorimo to še z grafom:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-checkpoint.ipynb".
The first task is:
Iz tabel je razvidno, da so njeni razultati v slalomu v vačini na repu trideseterice, med tem ko se v veleslalomu uvršča med 5 najboljših. To se še lepše vidi z grafov:
Can you write Python code for it?

brem[slalom]['mesto1'].value_counts().plot(kind='bar')

urejen = brem[veleslalom].sort_values(['mesto1'], ascending=True)
#urejen['mesto1'].value_counts()
#urejen['mesto1'].value_counts().plot(kind='bar')
#ne more uredit, ker DNQ1, DNF1, DSQ2 in DNS1 niso legit uvrstitve.

brem[veleslalom]['mesto1'].value_counts().plot(kind='bar')


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-checkpoint.ipynb".
The first task is:
Nato sem za vsakega od tekmovalcev s strani z njegovimi rezultati (npr. Eva-Maria Brem) pobrala podatke o vsaki tekmi: datum, prizorišče, disciplino, uvrstitev, zaostanek.
Can you write Python code for it?

pot_brem = "C:/Users/Ziva/Documents/AlpineSkiing/csv-datoteke/BREM  Eva-Maria.csv"
brem = pd.read_csv(pot_brem, parse_dates=['datum'])


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-checkpoint.ipynb".
The first task is:
Opomba: mesto1 in tocke bi bilo bolj smiselno dodati v prvotni csv!!!!
Can you write Python code for it?

brem['tocke'] = brem['mesto'].map(pretvori_2)
brem[:15]


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-checkpoint.ipynb".
The first task is:
Pa si poglejmo, v katerih disciplinah najpogosteje tekmuje Eva-Maria Brem:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-checkpoint.ipynb".
The first task is:
Popravi graf, da najmanjše lepo prikaže!!!
Analiza smuči
Najprej si oglejmo, katere znamke smuči so najpogostejše v svetovnem pokalu:
Can you write Python code for it?

smucarji['smuci'].value_counts()

smucarji['smuci'].value_counts().plot(kind='pie', figsize=(6,6))


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-checkpoint.ipynb".
The first task is:
Tabela izgleda tako:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-checkpoint.ipynb".
The first task is:
Tabela za Evo-Mario Brem:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-checkpoint.ipynb".
The first task is:
V 2. in 13. vrstici je vidna razlika med stolpcema 'mesto' in 'mesto1'.
Če bomo želeli delati analizo skupnega seštevka, moramo pretvoriti mesto tudi v točke. Definiramo seznam 'tocke', v katerega na i-to mesto (i teče od 0 do 30) zapišemo, koliko točk tekmovalka dobi za osvojeno i-to mesto.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/obdelava_podatkov-checkpoint.ipynb".
The first task is:
Čeprav najpogosteje tekmuje v slalomu in veleslalomu, pa to nista nujno disciplini, v katerih dosega najboljše rezultate. Najprej si poglejmo, kakšni so njeni rezultati v slalomu in nato še veleslalomu:
Can you write Python code for it?

slalom = brem['disciplina'] == 'Slalom'
brem[slalom][:15]

veleslalom = brem['disciplina'] == 'Giant Slalom'
brem[veleslalom][:15]


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
All looks good, expect the endpoints. This was expected. This is why we have boundary conditions.
We can write our equation y''(x) = y(x) as y''(x) - y(x) = 0 and by introducing the double derivative operator L and the identity operator I, we could also write it as
(L - I) y = 0
The idenitity operator for vectors is of course just the idenitity matrix
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
As mentioned, this can be done much more efficiently (and without using loops).
But this requires that one understands how the reorganisation to vectors work.
Anyway, just to show how one could do it:
Can you write Python code for it?

L_quick = -4 * np.eye(n) + np.diag(np.ones(n-m), m) + np.diag(np.ones(n-m), -m)
a = np.ones(n-1); a[(m-1)::m] = 0
L_quick += np.diag(a,1) + np.diag(a,-1)
L_quick = L_quick/dx**2

print( (L == L_quick).all() )


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
Example problem
Let us solve the steady state heat (Laplace) equation with known temperature boundary conditions
Laplacian u(x, y) = 0
u(x, 0) = 1
u(0, y) = 1 - y
u(x, 1) = 0
y(1, y) = 0
We start by defining our domain:
Can you write Python code for it?

dx = 0.02
x = np.arange(0, 1 + dx, dx)
m = len(x)
print(x[0], x[-1])
X, Y = np.meshgrid(x, x)
shape = X.shape


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
Exercise
(Skip this exercise if you feel confident it's easy)
Change the above code to solve
x''(t) = - k sin(x(t))
If time permits: lookup documentation for scipy.integrate.odeint and use that to solve the above equation. This will be much more efficient. But do other exercises first.
Boundary Value Problems
Now we want to solve the equation
y''(x) = y(x)
y(0) = y0
y(1) = y1
We can no longer iterate forwards from a given start point as we could before, where both x(0) and x'(0) were known.
There are many methods to solve systems as the above (shootings methods, relaxation methods, etc), but we will use the simplest direct method here. Again we base it on finite differences.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
Extra exercise
What about Neumann boundary conditions? Try writing a solver for
y''(x) = y'(x) + sin(x)*exp(x)
y(-pi) = 1
y'(pi) = 0
As a check, the correct solution has y(0) = -0.956331.
Partial differential equations
Just as PDE's are harder to solve than ODE's, so are PDE's much harder to implement numerically.
Most courses tend to avoid these and say they "generalise from ODE's", which I would argue is not true.
But this being a maths department, PDE solving might relevant, so we will cover this area. But focus on simple methods.
3D functions
We start by showing how to plot
u(x,y) = exp(x) sin(y)
Can you write Python code for it?

dx = 0.01
x = np.arange(0, 7, dx)
y = np.arange(-3, 3, dx)
X, Y = np.meshgrid(x, y)
U = np.exp(X) * np.sin(Y)

plt.imshow(U, extent=(min(x), max(x), max(y), min(y)))
plt.xlabel('x')
plt.ylabel('y')
plt.colorbar()
plt.show()


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
How would we calculate the vector y''? In particular, can we find a matrix L such that y'' = L y?
Double derivatives can be estimated by finite differences as (see https://en.wikipedia.org/wiki/Finite_difference_coefficient)
y''(x) = (y(x + dx) + y(x - dx) - 2 y(x))/dx^2
Let's build a matrix based on that:
Can you write Python code for it?

L = np.diag(np.ones(n - 1), 1) + np.diag(np.ones(n - 1), -1) - 2 * np.diag(np.ones(n))
print(L)
L = L/dx**2

ypp = np.dot(L, y) # matrix product L y
plt.plot(x, ypp, lw=3, alpha=0.5)
plt.plot(x, 4*3*x**2, '--k')
plt.axis([0, 1, -3, 15])
plt.show()


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
Imagine that we had calculated y. It would be some array:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
In the above np.meshgrid has made the 2D arrays X,Y:
Can you write Python code for it?

print(X.shape, Y.shape, U.shape)
print(X)


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
It is great for testing one's code that we can compare to analytical solutions, but of course the strength of numerical methods is that we can solve equations that we do not have analytical solutions for.
Exercise
Solve
y''(x) = y'(x) + sin(x)*exp(x)
y(-pi) = 0
y(pi) = 1
The analytical solution is given below.
Can you write Python code for it?

dx = 0.01
x = np.arange(-np.pi, np.pi, dx)
y = 1/4*np.exp(-np.pi)/np.sinh(np.pi)*(-2+np.exp(x)*(1+2*np.exp(np.pi)+np.cos(x)+ \
        np.sin(x)-np.exp(2*(np.pi))*(1+np.cos(x)+np.sin(x))))
plt.plot(x, y, '--k')
plt.axis([-np.pi, np.pi, -6, 1.5])
plt.show()


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
Now let us see if we can calculate these derivatives numerically.
Can you write Python code for it?

def derivative(f, dx):
    return (f[1:] - f[:-1])/dx # returned function is one value shorter than input f
ndfs = [derivative(f, dx) for f in fs]

for i in range(3): plt.plot(x[:-1], ndfs[i], lw=3, alpha=0.5)
for i in range(3): plt.plot(x, dfs[i], '--k')
plt.show()


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
Now we implement the boundary conditions:
Can you write Python code for it?

b = np.zeros(n)
for i in range(n):
    if (x[i]==0 or x[i]==1 or y[i]==0 or y[i]==1): # For any boundary point
        L[i, :] = 0
        L[i, i] = 1

    # BC points that are not equal to zero:
    if x[i] == 0:
        b[i] = 1 - y[i]
    elif y[i] == 0:
        b[i] = 1


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
Numerical differential equations
In the simplest methods for solving differential equations numerically, one goes back to the definition of the differential operator: lim dx->0 (f(t + dx) - f(t))/dx and simply take dx small, but finite, which will approximate the true derivative.
We start by understanding this.
Numerical differentiation
We start by defining a few functions for which we know their analytical derivative.
Can you write Python code for it?

dx = 0.3
x = np.arange(0, 10, dx) # returns [0, dx, 2dx, 3dx, 4dx, 5dx, ...]
print(x)
f1 = np.sin(x)
f2 = x**2/100
f3 = np.log(1+x)-1
fs = [f1, f2, f3]

for i in range(3): plt.plot(x, fs[i])

df1 = np.cos(x)
df2 = x/50
df3 = 1/(1+x)
dfs = [df1, df2, df3]


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
That worked pretty well, but we can do even better by using central differences for estimating the deriavtive.
Can you write Python code for it?

def central_derivative(f, dx):
    return (f[2:] - f[:-2])/(2*dx) # returned function is two values shorter than input f
ndfs = [central_derivative(f, dx) for f in fs]
for i in range(3): plt.plot(x[1:-1], ndfs[i], lw=3, alpha=0.5)
for i in range(3): plt.plot(x, dfs[i], '--k')
plt.show()


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
The error is order dx for left/right derivatives, but only dx^2 for central derivatives.
Initial Value Problems
We now turn to simple ODE's. Consider the damped harmonic oscillator equation
x''(t) = - k x(t) - d x'(t)
which we can write as
x'(t) = v(t)
v'(t) = -k x(t) - d v(t).
We want to solve this with initial conditions x(0) = x0, v(0) = v0.
Can you write Python code for it?

def analytical_solution(t, k, d, x0, v0):
    d += 0j # Exploiting complex numbers for one general solution
    return np.real((np.exp(-((d*t)/2))*(np.sqrt(d**2-4*k)*x0*np.cosh(1/2*np.sqrt(d**2-4*k)*t)+(2*v0+d* \
           x0)*np.sinh(1/2*np.sqrt(d**2-4*k)*t)))/np.sqrt(d**2-4*k))

dt = 0.02
t = np.arange(0,25,dt)
k = 5; d = 0.3; x0 = 0; v0 = 1

x = np.zeros_like(t)
v = np.zeros_like(t)
x[0] = x0; v[0] = v0

for i in range(len(t)-1): # Step through time with step size dt
    v[i + 1] = v[i] + (-k * x[i] - d * v[i]) * dt
    x[i + 1] = x[i] + v[i+1] * dt # v[i+1] makes a big stability difference!

plt.plot(t, x, lw=3, alpha=0.5)
plt.plot(t, analytical_solution(t, k, d, x0, v0), '--k')
plt.show()


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
The first row extracts y[0] and the equation then sets it equal to y0(=1) and the last row extracts y[-1] and sets it equal to y1(=10).
All other lines sets double derivatives equal to zero.
Solving the above linear system of equations is easy.
Numpy linear algebra module can do it for us:
Can you write Python code for it?

from numpy.linalg import solve
y = solve(A, b)

analytical = (np.exp(-x)*(np.exp(1)*(np.exp(1)*y0-y1)+np.exp(2*x)*(-y0+np.exp(1)*y1)))/(-1+np.exp(2))

plt.plot(x, y, lw=3, alpha=0.5)
plt.plot(x, analytical, '--k')
plt.show()


I am working on the file ".ipynb_checkpoints/ode_pde-checkpoint.ipynb".
The first task is:
Where A is defining the full problem.
The problem is still not well-defined as we need to somehow implement our boundary conditions.
Let's do it, and explain afterwards:
Can you write Python code for it?

A[0, :] = 0
A[0, 0] = 1
A[-1, :] = 0
A[-1, -1] = 1
b = np.zeros(n)
b[0] = y0
b[-1] = y1

np.set_printoptions(suppress=True)
print(np.round(A))
print('* y == ')
print(b)


I am working on the file ".ipynb_checkpoints/original-nn-checkpoint.ipynb".
The first task is:
Check out your predictions
Here, use the test data to view how well your network is modeling the data. If something is completely wrong here, make sure each step in your network is implemented correctly.
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(8,4))

mean, std = scaled_features['cnt']
predictions = network.run(test_features).T*std + mean
ax.plot(predictions[0], label='Prediction')
ax.plot((test_targets['cnt']*std + mean).values, label='Data')
ax.set_xlim(right=len(predictions))
ax.legend()

dates = pd.to_datetime(rides.ix[test_data.index]['dteday'])
dates = dates.apply(lambda d: d.strftime('%b %d'))
ax.set_xticks(np.arange(len(dates))[12::24])
_ = ax.set_xticklabels(dates[12::24], rotation=45)


I am working on the file ".ipynb_checkpoints/original-nn-checkpoint.ipynb".
The first task is:
Load and prepare the data
A critical step in working with neural networks is preparing the data correctly. Variables on different scales make it difficult for the network to efficiently learn the correct weights. Below, we've written the code to load and prepare the data. You'll learn more about this soon!
Can you write Python code for it?

data_path = 'Bike-Sharing-Dataset/hour.csv'

rides = pd.read_csv(data_path)

rides.head()


I am working on the file ".ipynb_checkpoints/original-nn-checkpoint.ipynb".
The first task is:
Splitting the data into training, testing, and validation sets
We'll save the data for the last approximately 21 days to use as a test set after we've trained the network. We'll use this set to make predictions and compare them with the actual number of riders.
Can you write Python code for it?

# Save data for approximately the last 21 days
test_data = data[-21*24:]

# Now remove the test data from the data set
data = data[:-21*24]

# Separate the data into features and targets
target_fields = ['cnt', 'casual', 'registered']
features, targets = data.drop(target_fields, axis=1), data[target_fields]
test_features, test_targets = test_data.drop(target_fields, axis=1), test_data[target_fields]


I am working on the file ".ipynb_checkpoints/original-nn-checkpoint.ipynb".
The first task is:
Time to build the network
Below you'll build your network. We've built out the structure and the backwards pass. You'll implement the forward pass through the network. You'll also set the hyperparameters: the learning rate, the number of hidden units, and the number of training passes.
<img src="assets/neural_network.png" width=300px>
The network has two layers, a hidden layer and an output layer. The hidden layer will use the sigmoid function for activations. The output layer has only one node and is used for the regression, the output of the node is the same as the input of the node. That is, the activation function is $f(x)=x$. A function that takes the input signal and generates an output signal, but takes into account the threshold, is called an activation function. We work through each layer of our network calculating the outputs for each neuron. All of the outputs from one layer become inputs to the neurons on the next layer. This process is called forward propagation.
We use the weights to propagate signals forward from the input to the output layers in a neural network. We use the weights to also propagate error backwards from the output back into the network to update our weights. This is called backpropagation.

Hint: You'll need the derivative of the output activation function ($f(x) = x$) for the backpropagation implementation. If you aren't familiar with calculus, this function is equivalent to the equation $y = x$. What is the slope of that equation? That is the derivative of $f(x)$.

Below, you have these tasks:
1. Implement the sigmoid function to use as the activation function. Set self.activation_function in __init__ to your sigmoid function.
2. Implement the forward pass in the train method.
3. Implement the backpropagation algorithm in the train method, including calculating the output error.
4. Implement the forward pass in the run method.
Can you write Python code for it?

class NeuralNetwork(object):
    def __init__(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
        # Set number of nodes in input, hidden and output layers.
        self.input_nodes = input_nodes
        self.hidden_nodes = hidden_nodes
        self.output_nodes = output_nodes

        # Initialize weights
        self.weights_input_to_hidden = np.random.normal(0.0, self.input_nodes**-0.5,
                                       (self.input_nodes, self.hidden_nodes))

        self.weights_hidden_to_output = np.random.normal(0.0, self.hidden_nodes**-0.5,
                                       (self.hidden_nodes, self.output_nodes))
        self.lr = learning_rate

        #### TODO: Set self.activation_function to your implemented sigmoid function ####
        #
        # Note: in Python, you can define a function with a lambda expression,
        # as shown below.
        # self.activation_function = lambda x : 0  # Replace 0 with your sigmoid calculation.

        ### If the lambda code above is not something you're familiar with,
        # You can uncomment out the following three lines and put your
        # implementation there instead.
        #
        def sigmoid(x):
            return 1 / (1 + np.exp(-x))

        self.activation_function = sigmoid

    def train(self, features, targets):
        ''' Train the network on batch of features and targets.

            Arguments
            ---------

            features: 2D array, each row is one data record, each column is a feature
            targets: 1D array of target values

        '''
        n_records = features.shape[0]
        delta_weights_i_h = np.zeros(self.weights_input_to_hidden.shape)
        delta_weights_h_o = np.zeros(self.weights_hidden_to_output.shape)
        for X, y in zip(features, targets):

            #### Implement the forward pass here ####
            ### Forward pass ###
            # TODO: Hidden layer - Replace these...


I am working on the file ".ipynb_checkpoints/original-nn-checkpoint.ipynb".
The first task is:
Training the network
Here you'll set the hyperparameters for the network. The strategy here is to find hyperparameters such that the error on the training set is low, but you're not overfitting to the data. If you train the network too long or have too many hidden nodes, it can become overly specific to the training set and will fail to generalize to the validation set. That is, the loss on the validation set will start increasing as the training set loss drops.
You'll also be using a method know as Stochastic Gradient Descent (SGD) to train the network. The idea is that for each training pass, you grab a random sample of the data instead of using the whole data set. You use many more training passes than with normal gradient descent, but each pass is much faster. This ends up training the network more efficiently. You'll learn more about SGD later.
Choose the number of iterations
This is the number of batches of samples from the training data we'll use to train the network. The more iterations you use, the better the model will fit the data. However, if you use too many iterations, then the model with not generalize well to other data, this is called overfitting. You want to find a number here where the network has a low training loss, and the validation loss is at a minimum. As you start overfitting, you'll see the training loss continue to decrease while the validation loss starts to increase.
Choose the learning rate
This scales the size of weight updates. If this is too big, the weights tend to explode and the network fails to fit the data. A good choice to start at is 0.1. If the network has problems fitting the data, try reducing the learning rate. Note that the lower the learning rate, the smaller the steps are in the weight updates and the longer it takes for the neural network to converge.
Choose the number of hidden nodes
The more hidden nodes you have, the more accurate predictions the model will make. Try a few different numbers and see how it affects the performance. You can look at the losses...
Can you write Python code for it?

import sys

### Set the hyperparameters here ###
# 1- 10000 0.01 5  =   Training loss: 0.305 ... Validation loss: 0.485
# 2- 10000 0.45 2  = m Taining loss: 0.172 ... Validation loss: 0.328
# 4- 15000 0.45 2  =   Training loss: 0.156 ... Validation loss: 0.256
# 5- 12000 0.45 2  =   Training loss: 0.099 ... Validation loss: 0.179
# 3- 3000  0.45 2  =   Training loss: 0.217 ... Validation loss: 0.393

# Aumentar o numero de hidden nodes

# 6  12000 0.45 5  =   Training loss: 0.072 ... Validation loss: 0.180 (grafico ficou maior que dados)
# 7  12000 0.45 4  =   Training loss: 0.065 ... Validation loss: 0.201 ( validacao piorou)
# 8  12000 0.45 3  =   Training loss: 0.099 ... Validation loss: 0.173 ( validatin foi a melhor)

# mudar o learging rate
# 9  12000 0.44 3  =   Training loss: 0.065 ... Validation loss: 0.140 ( validacao melhorou )
# 10 12000 0.43 3  =   Training loss: 0.078 ... Validation loss: 0.152 ( validacao e train piorou)
# 11 12000 0.435 3  =  Training loss: 0.074 ... Validation loss: 0.165 ( piorou )
# 12 12000 0.445 3  =  Training loss: 0.080 ... Validation loss: 0.164 (piorou)
# 13 12000 0.439 3  =  Training loss: 0.066 ... Validation loss: 0.138  (melhor)
# 14 6000  0.439 3  =  Training loss: 0.074 ... Validation loss: 0.186 (piorou)

# 15 12000 0.439 3  =  Training loss: 0.099 ... Validation loss: 0.185
# 16 12000 0.439 3  =  Training loss: 0.081 ... Validation loss: 0.178

# As the reviewer suggests lets add hidden nodes

# 4000 0.1 15 Training loss: 0.247 ... Validation loss: 0.434 (loss curve shows high learning rate shape )
# 4000 0.01 15  Training loss: 0.499 ... Validation loss: 0.835 (loss curve shows low learning rate shape )
# 4000 0.05 15  Training loss: 0.283 ... Validation loss: 0.452 (loss curve shows high learning rate shape)

# 40000 0.027 15 Training loss: 0.143 ... Validation loss: 0.291
# 40000 0.015 15 Training loss: 0.229 ... Validation loss: 0.390 ficou pior
# 40000 0.03 15  Training loss: 0.112 ... Validation loss: 0.241
# 40000 0.035 15 Training loss: 0.101 ......


I am working on the file ".ipynb_checkpoints/original-nn-checkpoint.ipynb".
The first task is:
We'll split the data into two sets, one for training and one for validating as the network is being trained. Since this is time series data, we'll train on historical data, then try to predict on future data (the validation set).
Can you write Python code for it?

# Hold out the last 60 days or so of the remaining data as a validation set
train_features, train_targets = features[:-60*24], targets[:-60*24]
val_features, val_targets = features[-60*24:], targets[-60*24:]


I am working on the file ".ipynb_checkpoints/pairplotr_demo-checkpoint.ipynb".
The first task is:
Here is an example interpretation using pairplotr:
Row/column 1/1 indicates that survival (1) and death (0) are indicated by cyan and gray, respectively.
Row/column 3/1 indicates that most women survived (I'd guess about ~80%).
Row/column 3/2 indicates that more than half of all women were from Pclasses 1 and 2. This makes me curious about what characteristics women from Pclass 3 might have.
We can slice the data using normal Pandas notation and use it with pairplotr. Here's an example that investigates women from Pclass 3:
Can you write Python code for it?

%%time

where = (visualize_df['Sex']=='female')&(visualize_df['Pclass']==3) # Women from Pclass 3

ppr.compare_data(visualize_df[where],scatter_plot_filter='Survived')


I am working on the file ".ipynb_checkpoints/pairplotr_demo-checkpoint.ipynb".
The first task is:
Inspect data
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pairplotr_demo-checkpoint.ipynb".
The first task is:
Note, how the data has no missing values. This is required for the current version of pairplotr.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pairplotr_demo-checkpoint.ipynb".
The first task is:
Row/column 1/1 automatically shows that only about half of Pclass 3 women survived.
Row/column 8/1 is interesting. It seems to indicate that most women from Embarked values Q and C survived, while the bulk of Pclass 3 women from Embarked S died.
Row/column pairs 8/5 and 8/6 seem to indicate that Embarked S had a higher concentration of larger amounts of Siblings/Spouses and Parents/Childen.
Additionally, row/colum pairs 5/1 and 6/1 seem to indicate that women with less family had a better chance to survive. Here I zoom in on these two figures to check:
Can you write Python code for it?

%%time

where = (visualize_df['Sex']=='female')&(visualize_df['Pclass']==3) # Women from Pclass 3

ppr.compare_data(visualize_df[where],scatter_plot_filter='Survived',zoom=['SibSp','Survived'])

%%time

where = (visualize_df['Sex']=='female')&(visualize_df['Pclass']==3) # Women from Pclass 3

ppr.compare_data(visualize_df[where],scatter_plot_filter='Survived',zoom=['Parch','Survived'])


I am working on the file ".ipynb_checkpoints/pairplotr_demo-checkpoint.ipynb".
The first task is:
Unpickle data
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pairplotr_demo-checkpoint.ipynb".
The first task is:
We can zoom in on individual plots by using the zoom keyword argument:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:

Can you write Python code for it?

y = np.rec.array([(1,2.,'Hello'),(2,3.,"World")],
                          dtype=[('foo', 'i4'),('bar', 'f4'), ('baz', 'S10')]) ## record array

y[0]

y['foo']

y.dtype.names

y[['foo','bar']]  # access multiple fields simultaneously

y[1].baz  # access one element


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2. Let's plot the dataset!
2.1 Scatter_matrix
Using pandas' scatter_matrix function.
This is similar to R's pairs and makes a quick scatter plot map for the quantitative
features of the dataframe
First we'll make a list named colors as a book-keeper.
Then to visualise the different target categories (i.e. the Species) we'll use the target numerical categorisation (0,1,2) to differently colour each row of the dataframe based on the target variable
Can you write Python code for it?

colors = list()

palette = { 0: "red", 1: "green", 2:"blue"}

for c in np.nditer(iris.target):
    colors.append(palette[int(c)])  # c is 0,1 or 2 and we append red, green blue

scatterplot = pd.scatter_matrix(dataframe,
                               alpha    = 0.3,
                               figsize  = (10,10),
                               diagonal = 'hist', #'kde'
                               color    = colors,
                               marker   = 'o',
                               grid     = True)


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.11 Andrews Curves
They are used to plot multivariate data as a large number of curves that are created using the atributes of samples as coefficients for Fourier series. By coloring these curves per class it is possible to visualize data clustering.
It is a good way to visualize structure in high-dimensional data. Each data point $x={x_1 , x_2, ..., x_d}$ defines a finite Fourrier series:
$f_{x}(t) = \frac{x_1}{\sqrt{2}} + x_2 sin(t) + x_3 cos(t) + x_4 sin(2t) + x_5 cos(2t)+...$
This function is plotted for $-\pi < t < \pi$. Therefore, each data point may be viewed as a line between $-\pi$ and $\pi$. The formula can be thought of as the projection of the data point onto the vector
$(\frac{1}{\sqrt{2}}, sin(t), cos(t), sin(2t), cos(2t),...)$
Each df row is a line per category. On the horizontal axis is the $-\pi < t < \pi$ and on the vertical the $f_x(t)$.
Curves belonging to samples of the same class will usually be closer together and form larger structures!
Can you write Python code for it?

from pandas.tools.plotting import andrews_curves

new_df = dataframe;
new_df['Species']=target_df;
new_df.head(5)

andrews_curves(new_df, "Species")


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.12 Parallel Coordinates
Parallel coordinates is a way of visualising high-dimensional geometry and analysing multivariate data.
To show a set of points in a n-dimensional space, the horizontal axis is split consisting n parallel vertical (typically) lines. A point in the n-dimensional space is represented as a polyline with vertices on the parallel axes; the position of the vertex on the i-th axis corresponds to the i-th coordinate of the point.
This type of visualisation is closely related to time series visualisation, with the difference that it is applied on data that have no time dependance.
Three points must be taken into account when the plot is used for statistical inference:
1. The order of the axis is critical for finding features (many reorderings are done in practice).
2. The rotation of the axis is a translation in the parallel coordinates; if the lines intersected outside the parallel axes, it can be translated between them by rotations (i.e. 180$\circ$).
3. The scaling is necessary since the plot is based on interpolation of consecutive pairs of variables. Thus the variables must be on a common scale and orthogonal to each parallel axis.
Points that tend to cluster will appear closer together.
Can you write Python code for it?

from pandas.tools.plotting import parallel_coordinates
parallel_coordinates(new_df, "Species")


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.13 Lag plot
Lag plots are used to identify randomness in data. Random data should not exhibit any structure in the lag plot
Non-random structure implies that data are not random.
A lag is a fixed time displacement. For example, given a dataset $Y1,~Y2~...,~Yn$, $Y2$ and $Y7$ have lag $5$ since $7 - 2 = 5$. Lag plots can be generated for any arbitrary lag, although the most commonly used lag is $1$.
Can you write Python code for it?

from pandas.tools.plotting import lag_plot
lag_plot(new_df)


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.15 Bootstrap plot
Bootstrap plots are made to assess the uncertainty of a statistic metric (mean, median, midrange etc)
To make a bootstrap uncertainty estimate for one metric given a dataset, a subset of the sample of size les or equal to the size of the dataset is generated from the data and the statistic is calculated. This subset is generated with replacement, and thus each data point can be resampled multiple times or not at all. The process is repeated, usually 500 to 1000 times. The computed values for the statistic form an estimate of the sampling distribution of the statistics.
For example, in a sample of 50 values you want to bootstrap the median. You generate a subset of the sample with 50 elements and calculate the median. Repeat this 500 times, so that you have at least 500 values for the median. To calculate the 90% confidence interval for the median, the sample of medians are sorted into ascending order and the value of the 25th median is the lower confidence limit, while the value of 475th median is the upper confidence limit (first and third quartiles).
The plots generated are the series and the histograms for mean, median and mid-range.  For uniformly distributed values, mid-range has the smallest variance.
Can you write Python code for it?

from pandas.tools.plotting import bootstrap_plot
data = pd.Series(np.random.rand(1000));

bootstrap_plot(data, size=50, samples=500, color='grey')


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.16 RadViz Plot
RadVis is a way to visualise multivariate data; to visualise n-dimensional points into a two dimensional space. In this case, the mapping is not linear. The technique is based on a simple spring tension minimization algorithm.
Imagine $n$ points, $S_1,S_2,...S_n$ arranged to be equally spaced around the cirumference of the unit circle. Now suppose a set of $n$ springs being fixed at one end to each of these points and the other ends to a puck. Finally, assume the stiffness constant (as in Hooke's law) of the j$th$ string in the $x_{ij}$ for one of the data points $i$.
If the puck is released and allowed to reach equilibrium position, the coordinates of this position, $(u_i, v_j)^{T}$, are the projection in the two dimensional space of the point $(x_{i1}, x_{i2}, ... x_{in})^{T}$ in the $n$-dimensional space. If the $(u_i, v_j)^{T}$ is computed for $i=1,2...n$ and the points plotted, a visualisation of the $n$-dimensional dataset in the two dimensions is achieved.
To understand more about the projection of the $n$-dimensional space into the two dimensional one, consider the forces acting on the puck. When the puck is in equilibrium, there are no resultant forces acting on it (their sum is 0). Denoting the position vectors of $S_1$ to $S_n$ by $\mathbf{S_1}$ to $\mathbf{S_n}$ and putting $\mathbf{u_{i}}=(u_i, v_i)^{T}$ we have:
$$\sum_{j=1,n}=(\mathbf{S}{j}-\mathbf{u}{i})x_{ij} = 0 $$
which when solved for $\mathbf{u}_{i}$ :
$$\mathbf{u}{i} = \sum{j=1,n}w_{ij}\mathbf{S}_j$$
where
$$ w_{ij} = \left( \sum_{j=1,n} x_{ij} \right)^{-1} x_{ij}$$
This means that for each $i$ (i.e. dataframe row), $\mathbf{u_i}$ is the weighted mean of the $\mathbf{S_j}$'s whose weights are the $n$ variables for case $i$ normalised to unity.
N.B. This normalisation makes the mapping $\mathcal{R}^{n}\rightarrow \mathcal{R}^{2}$ non-linear.
Can you write Python code for it?

from pandas.tools.plotting import radviz

radviz(new_df, 'Species')


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.3 Histogram (for one column, for all, or split by diff )
Can you write Python code for it?

dataframe['sepal length (cm)'].plot(kind='hist', color='red', alpha=0.3, figsize=(10,8))

dataframe.plot(kind='hist', alpha=0.3, orientation='horizontal',
               cumulative=True, bins=15, figsize=(10,8))

dataframe.diff().hist(color='k', alpha=0.5, bins=50, figsize=(10,8))
                    # diff() gets the difference of all
                    # rows with row-0


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.4 Box plots
Let's remind here that box plot is a convenient way of depicting groups of numerical data and their quartiles.
The first quartile is the 25th percentile (splits off the lowest 25% of data from the highest 75%), the second quartile (median) is the 50th percentile (cuts the dataset in half) and the third quartile is the upper quartile or the 75th percentile (splits off the highest 25% from the lowest 75%). IRQ (interquantile range) is the difference between upper and lower quartiles IRQ=Q3-Q1).
The line is the median (Q2 or 50%).
The box represents the IRQ, the from median to highest values is the Q2 to Q3 region, while the lowest is the Q1 to Q2 region.
```
      _   -> maximum
        |
        |
       ---    -> third quartile
      |   |
      |_|   -> median (Q2)
      |   |
      |   |
       ---    -> first quartile
        |
        |
      _____   -> minimum
```
 Outliers  --> 3 x IRQ
 Suspected Outliers  1.5 x IRQ
Can you write Python code for it?

dataframe.plot(kind='box', vert=False, by='X', figsize=(10,8))


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.5 Area plots!
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.6 Scatter Plots

Defining the x and y axis
plotting two sets with the ax=ax
palette height
Can you write Python code for it?

dataframe.plot(kind='scatter', x='sepal length (cm)', y='sepal width (cm)', figsize=(10,8))

ax = dataframe.plot(kind='scatter', x='sepal length (cm)', y='sepal width (cm)',
                    color='DarkBlue', label='Group 1');
dataframe.plot(kind='scatter', x='petal length (cm)', y='petal width (cm)',
                    color='DarkGreen', label='Group 2' , figsize=(10,8),
                    ax=ax); # ax = ax to overlay
                            # if not set 2 plots made

dataframe.plot(kind='scatter', x='petal length (cm)', y='petal width (cm)',
               c='sepal width (cm)', s=50, figsize=(10,8))


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.7 Hexbin
Hexbin are alternative to scatter plots if the data are too dense to plot individually
Can you write Python code for it?

dataframe.plot(kind='hexbin', x='petal length (cm)', y='petal width (cm)',
               gridsize=25, figsize=(10,8))


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
2.9 Missing data
Depending on the plot type pandas handle by default the NAs
Plot type | Handling
---|---
Line  | Leave gaps at NA
Line stacked | Fill 0's
Bar | Fill 0's
Scatter | Drop NA
Histogram | Drop NA
Box | Drop NA
Area | Fill 0's
KDE (density) | Fill 0's
Hexbin | Drop NA
Pie | Fill 0
If this is not the wanted behaviour use: fillna() or dropna() before plotting

2.10  Density/KDE Plot
Can you write Python code for it?

dataframe.ix[:,0:2].plot(kind='kde', figsize=(10,8))
# this is to take all rows for the first two columns of the dataframe (or df.iloc[:,0:2])


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
3.1.2. Series from dictionary
Generating from a dictionary, the keys are taken as index.
If extra index argument is given  this the keys values are overwritten for the matching cases, while if the index argument is larger than the len(dict.keys()) then NaN's are created.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
3.2.2 DataFrame from dict of ndarrays/lists
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
3.2.4 DataFrame from list of dicts
Now make a list of dictionaries
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
3.2.5 DataFrame from dict of tuples
Now make a dict containing tuples:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
3.2.6 DataFrame from Series
This is including various series for the creation of a DF

4. Dealing with dates and problematic data
Now let's make one dataframe and see what we can do when we have dates and NaN values in it.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
4.1 Automatic Formating of "date strings"
The read_csv() function has an option that specifyies which column contains dates. Then these dates can be parsed and formated accordingly.
Just include the argiment parse_dates=[list of columns with dates]
Can you write Python code for it?

# read it in pandas and process dates
df_csv2 = pd.read_csv("file.csv", sep=',', parse_dates=[0]); df_csv2


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
4.2 Handle NaNs
There are various ways of dealing with NaN values within pandas. But first one must find them!
4.2.1 Find NaN Values
There are various ways of finding NaN values. Mainly these are revolting around the isnull() and notnull() functions.
Can you write Python code for it?

df_nulls = pd.read_csv("file.csv", sep=',', parse_dates=[0]); df_nulls

pd.isnull(df_nulls['Temperature_city_1'])  # looking for nulls in one column

df_nulls['Temperature_city_1'].isnull()   # same thing different format

df_nulls.notnull()   # look for actual values (NON NaN's) in the whole df


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
4.2.1.1 NaN and None
It is very important to remember that NaN is not None :
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
4.2.5 Calculations with NaNs
Missing values do propagate naturally in operations with Pandas Objects
Can you write Python code for it?

## Let's make some dfs with NaNs
a1 = pd.Series(np.random.randn(3), index = ['e','f','h']);
a2 = pd.Series(np.random.randn(5), index = ['a','c','e','f','h']);
a = {'one': a1, 'two': a2};
dfa = pd.DataFrame(a);
print dfa

b1 = pd.Series(np.random.randn(2), index = ['e','f']);
b2 = pd.Series(np.random.randn(5), index = ['a','c','e','f','h']);
b3 = pd.Series(np.random.randn(5), index = ['a','c','e','f','h']);
b = {'one':b1, 'two':b2, 'three':b3};
dfb = pd.DataFrame(b);
print dfb

dfa+dfb ## NaN are propagated properly

(dfa+dfb).mean(axis=0) #mean for rows


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
4.2.6 Interpolate NaN
Series and DataFrames have the interpolate method, which performs linear interpolation (by default) for the missing datapoints
Can you write Python code for it?

df_nulls

df_nulls.count()

df_nulls.interpolate().count() # first interpolate the NaN

df_nulls.interpolate()


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
Also NaN are not comparable, this means that
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
Also vectorised operations can be done. The difference with ndarray is that the result is aligned automatically based on the labels.
Can you write Python code for it?

s+s2; s2*2; np.exp(s)

s5 = pd.Series(np.random.randn(5), name='something'); s5.name


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
Functions like cumsum() (cumulative sum over requested axis) and cumprod() (cumulative product over requested axis) ignore the NaNs. Similarly the groupby()
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
Let's use pandas' to convert the ndarray into dataframe, keeping the column names!
Can you write Python code for it?

dataframe = pd.DataFrame(iris.data, columns=iris.feature_names)

dataframe.head()

target_df = pd.DataFrame(iris.target, columns=['Species'])

target_df.tail(2)


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
N.B. The argument axis=0 implies the calculation of the means that span the rows so the obtained means that they extend column-wise. The axis=1 spans columns and the row-wise results are obtained.

4.2.5 Drop NaN rows
One might choose to drop the NaN values this can be done by
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
Notice the CParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 4
Now trying with the option:
Can you write Python code for it?

bdf = pd.read_csv('bad_dataset.csv', error_bad_lines=False) #i.e. "I don't want a parser check"

bdf


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
Pandas/Numpy use this to separate None and NaN, since
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
Series can behave as ndarrays or as dictionaries including all the operations!
For example
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
So indeed the series are filled with NaN's and the plots with 0's!.

3.1.3. Series from scalar
Now introducing series with scalar input, the values will all be identical.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
The index and columns can be accessed directly:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
df_csv3 is the original DF and has not replaced its NaN values.

4.2.2 Fill NaN with mean or Median
One can replace the NaN values with the column mean or median so to minimise the guessing error. This can easily be done by
Can you write Python code for it?

print type(df_csv3.mean(axis=0));
df_csv3.mean(axis=0)

df_csv5 = df_csv3.fillna(df_csv3.mean(axis=0)); df_csv5


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
keep in mind that this does not replace the values in the original DataFrame, but it must be stored in a copy.
So let's replace and see the values:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
the df_csv4 has indeed replaced NaN with -1, but what happened of df_csv3 ?
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
this does not work for a 2D ndarray
Can you write Python code for it?

#s = pd.Series(np.random.randn(5,5)) # this returns an exception : Data must be 1-dimensional


I am working on the file ".ipynb_checkpoints/pandas_data_munging-checkpoint.ipynb".
The first task is:
to see how this interpolation is done we can plot it
Can you write Python code for it?

df_nulls.plot(title='no linear interpolation')
df_nulls.interpolate().plot(title='with linear interpolation')


I am working on the file ".ipynb_checkpoints/peer_examples-checkpoint.ipynb".
The first task is:
A node's degree is the number of connections it has.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/peer_examples-checkpoint.ipynb".
The first task is:
The local clustering coefficient is the fraction of a node's connections that are also connected.
Can you write Python code for it?

nx.clustering(g, 0)

nx.clustering(g, 4)

nx.clustering(g, 1)