I am working on the file "03-python2-errors-instructors.ipynb".
The first task is:

Identifying Item Errors

Read the code below, and (without running it) try to identify what the errors are.

Run the code, and read the error message. What type of error is it?

Fix the error.

seasons = ['Spring', 'Summer', 'Fall', 'Winter']
print('My favorite season is ', seasons[4])

Can you write Python code for it?


I am working on the file "03-python2-errors-instructors.ipynb".
The first task is:

Identifying Syntax Errors

Read the code below, and (without running it) try to identify what the errors are.

Run the code, and read the error message. Is it a SyntaxError or an IndentationError?

Fix the error.

Repeat steps 2 and 3, until you have fixed all the errors.

def another_function
  print("Syntax errors are annoying.")
   print("But at least python tells us about them!")
  print("So they are usually not too hard to fix.")

Can you write Python code for it?

# def another_function - SyntaxError
# print("But at least...") - IndentationError


I am working on the file "03-python2-errors-instructors.ipynb".
The first task is:

Identifying Variable Name Errors

Read the code below, and (without running it) try to identify what the errors are.

Run the code, and read the error message. What type of NameError do you think this is? In other words, is it a string with no quotes, a misspelled variable, or a variable that should have been defined but was not?

Fix the error.

Repeat steps 2 and 3, until you have fixed all the errors.

for number in range(10):
    # use a if the number is a multiple of 3, otherwise use b
    if (Number % 3) == 0:
        message = message + a
    else:
        message = message + "b"
print(message)

Can you write Python code for it?

#1.
# if (Number % 3) == 0: - NameError
# message = message + a - NameError (twice)

#2.
# misspelled variable
# variable not defined
# string with no quotes


I am working on the file "03-python2-errors-instructors.ipynb".
The first task is:
Both SyntaxError and IndentationError indicate a problem with the syntax of your program, but an IndentationError is more specific: it always means that there is a problem with how your code is indented.

Tabs and Spaces

A quick note on indentation errors: they can sometimes be insidious, especially if you are mixing spaces and tabs. Because they are both whitespace, it is difficult to visually tell the difference. The IPython notebook actually gives us a bit of a hint, but not all Python editors will do that. In the following example, the first two lines are using a tab for indentation, while the third line uses four spaces:

def some_function():
    msg = "hello, world!"
    print(msg)
    return msg

  File "<ipython-input-5-653b36fbcd41>", line 4
    return msg
              ^
IndentationError: unindent does not match any outer indentation level

By default, one tab is equivalent to eight spaces, so the only way to mix tabs and spaces is to make it look like this. In general, is is better to just never use tabs and always use spaces, because it can make things very confusing.

Variable Name Errors
Another very common type of error is called a NameError, and occurs when you try to use a variable that does not exist. For example:
Can you write Python code for it?


I am working on the file "03-python2-errors-instructors.ipynb".
The first task is:
Finally, the third possibility is that you made a typo when you were writing your code. Let’s say we fixed the error above by adding the line Count = 0 before the for loop. Frustratingly, this actually does not fix the error. Remember that variables are case-sensitive, so the variable count is different from Count. We still get the same error, because we still have not defined count:
Can you write Python code for it?

Count = 0
for number in range(10):
    count = count + number
print("The count is: " + str(count))


I am working on the file "03-python2-errors-instructors.ipynb".
The first task is:
Here, Python tells us that there is a SyntaxError on line 1, and even puts a little arrow in the place where there is an issue. In this case the problem is that the function definition is missing a colon at the end.
Actually, the function above has two issues with syntax. If we fix the problem with the colon, we see that there is also an IndentationError, which means that the lines in the function definition do not all have the same indentation:
Can you write Python code for it?

def some_function():
    msg = "hello, world!"
    print(msg)
     return msg


I am working on the file "03-python2-errors-instructors.ipynb".
The first task is:
One reason for receiving this error is that you specified an incorrect path to the file. For example, if I am currently in a folder called myproject, and I have a file in myproject/writing/myfile.txt, but I try to just open myfile.txt, this will fail. The correct path would be writing/myfile.txt. It is also possible (like with NameError) that you just made a typo.
Another issue could be that you used the "read" flag instead of the "write" flag. Python will not give you an error if you try to open a file for writing when the file does not exist. However, if you meant to open a file for reading, but accidentally opened it for writing, and then try to read from it, you will get an IOError error telling you that the file was not opened for reading:
Can you write Python code for it?

file_handle = open('myfile.txt', 'w')
file_handle.read()


I am working on the file "03-python2-errors-instructors.ipynb".
The first task is:
The second is that you just forgot to create the variable before using it. In the following example, count should have been defined (e.g., with count = 0) before the for loop:
Can you write Python code for it?

for number in range(10):
    count = count + number
print("The count is: " + str(count))


I am working on the file "03-python2-errors-instructors.ipynb".
The first task is:
These are the most common errors with files, though many others exist. If you get an error that you’ve never seen before, searching the Internet for that error type often reveals common reasons why you might get that error.

Reading Error Messages

Run the code below and read the traceback. Identify the following pieces of information about it:

How many levels does the traceback have?

What is the file name where the error occurred?

What is the function name where the error occurred?

On which line number in this function did the error occurr?

What is the type of error?

What is the error message?

Can you write Python code for it?

import errors_02
errors_02.print_friday_message()

#1. 3
#2. ~/workshop_sunpy_astropy/errors_02.py
#3. print_message()
#4. 11
#5. KeyError
#6. 'Friday'


I am working on the file "03-python2-errors-instructors.ipynb".
The first task is:
This particular traceback has two levels. You can determine the number of levels by looking for the number of arrows on the left hand side. In this case:

The first shows code from the cell above, with an arrow pointing to Line 2 (which is favorite_ice_cream()).

The second shows some code in another function (favorite_ice_cream, located in the file errors_01.py), with an arrow pointing to Line 7 (which is print(ice_creams[3])).

The last level is the actual place where the error occurred. The other level(s) show what function the program executed to get to the next level down. So, in this case, the program first performed a function call to the function favorite_ice_cream. Inside this function, the program encountered an error on Line 7, when it tried to run the code print(ice_creams[3]).

Long Tracebacks

Sometimes, you might see a traceback that is very long – sometimes they might even be 20 levels deep! This can make it seem like something horrible happened, but really it just means that your program called many functions before it ran into the error. Most of the time, you can just pay attention to the bottom-most level, which is the actual place where the error occurred.

So what error did the program actually encounter? In the last line of the traceback, Python helpfully tells us the category or type of error (in this case, it is an IndexError) and a more detailed error message (in this case, it says “list index out of range”).
If you encounter an error and don’t know what it means, it is still important to read the traceback closely. That way, if you fix the error, but encounter a new one, you can tell that the error changed. Additionally, sometimes just knowing where the error occurred is enough to fix it, even if you don’t entirely understand the message.
If you do encounter an error you don’t recognize, try looking at the official...
Can you write Python code for it?

def some_function()
    msg = "hello, world!"
    print(msg)
     return msg


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
And if you want an integer result from division in Python 3, use a double-slash:
Can you write Python code for it?


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
As this example shows, parameters are matched up from left to right, and any that haven’t been given a value explicitly get their default value. We can override this behavior by naming the value as we pass it in:
Can you write Python code for it?

print('only setting the value of c')
display(c=77)


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
Composing Functions
Now that we’ve seen how to turn Kelvin into Celsius, let's try converting Celsius to Fahrenheit:
Can you write Python code for it?

def celsius_to_fahr(temp):
    return temp * (9/5) + 32

print('freezing point of water:', celsius_to_fahr(0))
print('boiling point of water:', celsius_to_fahr(100))


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
If you are using Python 2 and want to keep the fractional part of division you need to convert one or the other number to floating point:
Can you write Python code for it?


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
That looks right, so let’s try center on our real data:
Can you write Python code for it?

data = np.loadtxt(fname='data/inflammation-01.csv', delimiter=',')
print(centre(data, 0))


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
That seems almost right: the original mean was about 6.1, so the lower bound from zero is how about -6.1. The mean of the centered data isn’t quite zero — we’ll explore why not in the challenges — but it’s pretty close. We can even go further and check that the standard deviation hasn’t changed:
Can you write Python code for it?

print('std dev before and after:', data.std(), centered.std())


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
The function definition opens with the word def, which is followed by the name of the function and a parenthesized list of parameter names. The body of the function — the statements that are executed when it runs — is indented below the definition line, typically by four spaces.
When we call the function, the values we pass to it are assigned to those variables so that we can use them inside the function. Inside the function, we use a return statement to send a result back to whoever asked for it.
Let’s try running our function. Calling our own function is no different from calling any other function:
Can you write Python code for it?

print('absolute zero in Celsius:', kelvin_to_celsius(0.0))


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
The key change is that the second parameter is now written desired=0.0 instead of just desired. If we call the function with two arguments, it works as it did before:
Can you write Python code for it?

test_data = np.zeros((2, 2))
print(centre(test_data, 3))


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
There’s a better way, though. If the first thing in a function is a string that isn’t assigned to a variable, that string is attached to the function as its documentation:
Can you write Python code for it?

def centre(data, desired):
    '''Return a new array containing the original data centered around the desired value.'''
    return (data - data.mean()) + desired


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
There’s a lot of information here, but the most important part is the first couple of lines:

loadtxt(fname, dtype=<type 'float'>, comments='#', delimiter=None, converters=None, skiprows=0, usecols=None,
        unpack=False, ndmin=0)

This tells us that loadtxt has one parameter called fname that doesn’t have a default value, and eight others that do. If we call the function like this:
Can you write Python code for it?


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
This is better because we can now ask Python’s built-in help system to show us the documentation for the function:
Can you write Python code for it?


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
This is handy: if we usually want a function to work one way, but occasionally need it to do something else, we can allow people to pass a parameter when they need to but provide a default to make the normal case easier. The example below shows how Python matches values to parameters:
Can you write Python code for it?

def display(a=1, b=2, c=3):
    print('a:', a, 'b:', b, 'c:', c)

print('no parameters:')
display()
print('one parameter:')
display(55)
print('two parameters:')
display(55, 66)


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
To understand what’s going on, and make our own functions easier to use, let’s re-define our center function like this:
Can you write Python code for it?

def centre(data, desired=0.0):
    '''Return a new array containing the original data centered around the desired value (0 by default).
    Example: center([1, 2, 3], 0) => [-1, 0, 1]'''
    return (data - data.mean()) + desired


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
Unfortunately, this wasn’t the case in Python 2:
Can you write Python code for it?


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
We could test this on our actual data, but since we don’t know what the values ought to be, it will be hard to tell if the result was correct. Instead, let’s use NumPy to create a matrix of 0’s and then center that around 3:
Can you write Python code for it?


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
We’ve successfully called the function that we defined, and we have access to the value that we returned.
Integer division
We are using Python 3 division, which always returns a floating point number:
Can you write Python code for it?


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
With that in hand, let’s look at the help for numpy.loadtxt:
Can you write Python code for it?


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
and another function called detect_problems that checks for those systematics we noticed:
Can you write Python code for it?

def detect_problems(filename):

    data = np.loadtxt(fname=filename, delimiter=',')

    if data.max(axis=0)[0] == 0 and data.max(axis=0)[20] == 20:
        print('Suspicious looking maxima!')
    elif data.min(axis=0).sum() == 0:
        print('Minima add up to zero!')
    else:
        print('Seems OK!')


I am working on the file "03-python2-functions-instructors.ipynb".
The first task is:
then the filename is assigned to fname (which is what we want), but the delimiter string ',' is assigned to dtype rather than delimiter, because dtype is the second parameter in the list. However ',' isn’t a known dtype so our code produced an error message when we tried to run it. When we call loadtxt we don’t have to provide fname= for the filename because it’s the first item in the list, but if we want the ',' to be assigned to the variable delimiter, we do have to provide delimiter= for the second parameter since delimiter is not the second parameter in the list.

Combining strings

“Adding” two strings produces their concatenation: 'a' + 'b' is 'ab'. Write a function called fence that takes two parameters called original and wrapper and returns a new string that has the wrapper character at the beginning and end of the original. A call to your function should look like this:

print(fence('name', '*'))

*name*

Can you write Python code for it?

def fence(original, wrapper='#'):
    """Return a new string which consists of the original string with the wrapper character before and after"""
    return wrapper + original + wrapper

print(fence('name', '*'))


I am working on the file "03-python2-matplotlib-animation_Instructor.ipynb".
The first task is:
Animations
Matplotlib supports animation of all of its figure types. We will show you an example here.
First things first, we need to import Animation.
Can you write Python code for it?

%matplotlib nbagg
import matplotlib.pyplot as plt
import matplotlib.animation as ani

import numpy as np


I am working on the file "03-python2-matplotlib-animation_Instructor.ipynb".
The first task is:
Bringing it all Together - Animations in Matplotlib
In this final exercise we are going to make use of all the skills you have developed in Python today, and approximate pi with a nice matplotlib animation.
In this example we are going to approximate pi by calculating the difference in the area of a unit square and a unit circle. We shall do this by randomly picking a point inside the unit square and determining if that point is inside the unit circle contained within the square. The ratio of points inside both the square and the circle to the ratio of points only in the square will allow us to approximate pi.
First we will introduce you to the matplotlib animation module.
Challenge 1
Approximate pi by the Monte Carlo method in a for loop. For each iteration of the loop you will need to generate a random x and y coordinate of your point (between 0 and 1), and then calculate if this is inside the circule of radius 1. You can then approximate $\pi$ using the following equation:
$$
\pi = \frac{\text{number of points in circle}}{\text{total number of points}}
$$
You will probably find numpy.random.rand() and numpy.sqrt() useful.
Print the approximation at each step to see it getting more accurate.
Can you write Python code for it?

import numpy as np

points_in  = 0

for i in range(100):
    x = np.random.rand()
    y = np.random.rand()

    r = np.sqrt(x**2 + y**2)

    if r <= 1:
        points_in += 1

    pi_4 = points_in/(i+1)

    print("pi = {}".format(pi_4 * 4.))


I am working on the file "03-python2-matplotlib-animation_Instructor.ipynb".
The first task is:
So animation plotting is based off creating a function. So in this case, we are animating a line plot.
Can you write Python code for it?

fig, ax = plt.subplots()

x = np.linspace(0, 6*np.pi, 1000)
y = np.sin(x)

line, = ax.plot(x, y)

def update(i):
    shift = np.pi/50
    x = np.linspace(0, 6*np.pi, 1000)
    y = np.sin(x + i*shift)
    return line.set_data(x, y)

anim = ani.FuncAnimation(fig, update, frames=100)

plt.show()


I am working on the file "03-simulate-models.ipynb".
The first task is:
Basically, fba calls model.optimize() and wraps the optimization solution in a FluxDistributionResult object. The maximum objective values (corresponding to a maximum growth rate) can be obtained from result.objective_value.
Can you write Python code for it?


I am working on the file "03-simulate-models.ipynb".
The first task is:
In cameo, flux balance analysis can be performed with the function fba.
Can you write Python code for it?

from cameo import fba
%time fba_result = fba(model)


I am working on the file "03-simulate-models.ipynb".
The first task is:
Parsimonious Flux Balance Analysis
Parsimonious flux balance analysis (Lewis et al., 2010), a variant of FBA, performs FBA in in a first step to determine the maximum objective value $Z_{obj}$, fixes it in form of an additional model constraint ($\mathbf{c}^{T} \mathbf{v} \ge Z_{obj}$), and then minimizes the $L_1$ norm of $\mathbf{v}$. The assumption behind pFBA is that cells try to minimize flux magnitude as well in order to keep protein costs low.
$$
\begin{align}
 Max ~ & ~ \lvert \mathbf{v} \rvert\
 \text{s.t.}~ & ~ \mathbf{S} \mathbf{v} = 0 \
 & ~ \mathbf{c}^{T} \mathbf{v} \ge Z_{obj} \
 ~ & ~ \mathbf{v}{lb} \leq \mathbf{v} \leq \mathbf{v}{ub} \,.
\end{align}
$$
In cameo, pFBA can be performed with the function pfba.
Can you write Python code for it?

from cameo import pfba
%time pfba_result = pfba(model)


I am working on the file "03-simulate-models.ipynb".
The first task is:
ROOM is a difficult computational problem. If the bounds of the system are not large enough, it can take many hours to simulate. To improve the speed of the simulation and the chances of finding a solution, we increase the bounds.
Can you write Python code for it?

for reaction in model.reactions:
    if reaction.upper_bound == 1000:
        reaction.upper_bound = 99999999
    if reaction.lower_bound == -1000:
        reaction.lower_bound = -99999999

%time room_result = room(model, reference=pfba_result.fluxes)
room_result[model.reactions.BIOMASS_Ec_iJO1366_core_53p95M]


I am working on the file "03-simulate-models.ipynb".
The first task is:
Simulate using different methods:
Can you write Python code for it?

%time fba_knockout_result = fba(model)
fba_knockout_result[model.reactions.BIOMASS_Ec_iJO1366_core_53p95M]

%time pfba_knockout_result = pfba(model)
pfba_knockout_result[model.reactions.BIOMASS_Ec_iJO1366_core_53p95M]


I am working on the file "03-simulate-models.ipynb".
The first task is:
The objective_function value is $\lvert \mathbf{v} \rvert$ ...
Can you write Python code for it?


I am working on the file "03-sklearn-sentiment-classifier.ipynb".
The first task is:
Feeling Good? - Let's Update Kaggle Submission
Steps:

Load Test Dataset
Vectorize the Features (Review)
Predict the sentiment
Create the CSV file and update the submission
Can you write Python code for it?

# Read in the Test Dataset
# Note that it's missing the Sentiment Column.  That's what we need to Predict
#
test_df = pd.read_csv("data/test.tsv", sep="\t")
test_df.head()

# Vectorize the Review Text

X_test = test_df.review
X_test_vect = vect.transform(X_test)

y_test_pred = model.predict(X_test_vect)

df = pd.DataFrame({
    "document_id": test_df.document_id,
    "sentiment": y_test_pred
})

df.to_csv("data/logistic_reg_submission1.csv", index=False)

!head data/logistic_reg_submission1.csv


I am working on the file "03-sklearn-sentiment-classifier.ipynb".
The first task is:
Model - Logistic Regression
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()

model.fit(X_train_vect, y_train)

# Training Accuracy
print("Training Accuracy: {:.3f}".format(model.score(X_train_vect, y_train)))

## Validation Accuracy
print("Validation Accuracy: {:.3f}".format(model.score(X_valid_vect, y_valid)))


I am working on the file "03-sklearn-sentiment-classifier.ipynb".
The first task is:
Model Tuning
Model seems to be Overfitting.  Try, Regularization to bring Training Accuracy closer to Validation Accuracy

What options are available in Logisitc Regression
Can you write Python code for it?

model = LogisticRegression(C=0.1)
model.fit(X_train_vect, y_train)

# Training Accuracy
print("Training Accuracy: {:.3f}".format(model.score(X_train_vect, y_train)))
print("Validation Accuracy: {:.3f}".format(model.score(X_valid_vect, y_valid)))


I am working on the file "03-sklearn-sentiment-classifier.ipynb".
The first task is:
Training process

Split the Overall Training examples into Training and Validation
Build the Models on Training Data
Score on Validation data
Choose the best model and submit to Kaggle

Caution:  If you do this enough times, you will be overfitting to the Validation data.  To avoid that it might be advisable to split into three ways like Train-Validation-Test and generate the final score on Test Data.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_valid, y_train, y_valid = train_test_split(train_df["review"], train_df["sentiment"], test_size=0.2)

print("Training Data: {}, Validation: {}".format(len(X_train), len(X_valid)))


I am working on the file "03-sklearn-sentiment-classifier.ipynb".
The first task is:
Vectorize Data (a.k.a. covert text to numbers)
Computers don't understand Texts, so we need to convert texts to numbers before we could do any math on it and see if we can build a system to classify a review as Positive or Negative.
Ways to vectorize data:

Bag of Words
TF-IDF
Word Embeddings (Word2Vec)

Scikit-Learn has nice APIs for preprocessing and feature extraction modules.  In fact, these can be used even if you build your own models or use another libriary for model building process.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer

# The API is very similar to model building process.
# Step 1: Instantiate the Vectorizer or more generally called Transformer

vect = CountVectorizer(max_features=5000, binary=True, stop_words="english")

# Fit your Training Data
vect.fit(X_train)

# Transform your training and validation data
X_train_vect = vect.transform(X_train)
X_valid_vect = vect.transform(X_valid)

X_train.head()

# Creates a Sparse Matrix
X_train_vect

# Understand the Vectorizer
vect

# Does similar things to what we did manually in our bag of words model

# vect.vocabulary_

# Does similar things to what we did manually in our bag of words model
from itertools import islice

list(islice(vect.vocabulary_.items(), 10))

pd.DataFrame(X_train_vect.todense(), columns=vect.vocabulary_.keys()).head()


I am working on the file "03-statistical-inference.ipynb".
The first task is:
Next, we'll generate some data from a normal distribution. In the first example, the treatment, t, and the control, c, will be centered at 5. We'll combine the treatment and control arrays.
Can you write Python code for it?

np.random.seed(42)
t = np.random.normal(loc=5, size=m)
c = np.random.normal(loc=5, size=N-m)
outcomes = np.append(t, c)


I am working on the file "03-statistical-inference.ipynb".
The first task is:
This can be expressed in terms of potential outcomes (Gerber and Green, 2012):
$$ SE(\widehat{ATE}) = \sqrt{ \frac{1}{N-1} \left( \frac{ m\sigma^2_{y_{0i}} }{ N-m } + \frac{ (N-m) \sigma^2_{y_{1i}} }{ m } + 2 cov(y_{0i}, y_{1i}) \right) }\text{,}$$
where $m$ is the number of units in the treatment group, $N$ is the total number of units (treatment and control), and $\sigma^2$ is the variance. The last term represents the covariance of $y_{0i}$ and $y_{1i}$ and measures the association between the potential outcomes.
Using this formulation, Gerber and Green discuss the ways that the experimental design can produce more precise estimates of the ATE. They do this by changing single inputs at a time while holding others fixed. These are summarized below.

A larger $N$ leads to a smaller standard error. This is true even if $m$ is held constant, which implies that only the control group increases. Similarly, increasing the number of units in the control group, $m$, also decreases the standard error.
The smaller that $\sigma^2_{y_{0i}}$ or $\sigma^2_{y_{1i}}$ are, the smaller the standard error. This implies that the treatment and control group units should be as similar as possible in terms of potential outcomes.
The smaller the covariance between $y_{0i}$ and $y_{1i}$, the smaller the standard error.

Gerber and Green provide an additional suggestion:

if $\sigma^2_{y_{0i}}$ and $\sigma^2_{y_{1i}}$ are similar, about half of the units should be assigned to treatment and the rest to control
if the variances of the potential outcomes are different, assign more units to the corresponding group
in practice, however, this variance is rarely known
thus, try to assign an equal number of units to each condition

Standard errors measure uncertainty, indicating how estimates vary across all possible random assignments (Gerber and Green, 2012). One goal of an experiment is to reduce variability. There are several ways to do this, which we'll discuss further in a subsequent chapter, but, in general, one can:

make more...
Can you write Python code for it?

from code.permutation import permutation_test


I am working on the file "03-statistical-inference.ipynb".
The first task is:
This distribution was generated using 100,000 random data points with zero mean ($\mu = 0$) and unit variance ($\sigma^2 = 1$). Thus, the true ATE is zero, meaning there is no treatment effect, and we see that the histogram is centered around that value. We can also see the variation around zero. We'll discuss this next.
Standard Error
While experiments provide unbiased estimates of the ATE, they are not necessarily precise—there is variability around the true ATE. Sampling variability tells us about the uncertainty associated with our estimate. The statistic of choice for characterizing this variability is the standard error.
The standard error is the standard deviation of the sampling distribution. The formula for the population standard deviation is:
$$\sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - \bar{x})^2 } \text{.}$$
In other words, we take the square root of the average squared deviations from the mean. (Note: when the data is a random sample from the population, the sample standard deviation divides the squared deviations by $N-1$ rather than $N$.)
For our hypothetical sampling distribution, we can calculate the standard deviation as follows.
Can you write Python code for it?


I am working on the file "03-statistical-inference.ipynb".
The first task is:
We could also use NumPy's built-in standard deviation method.
Can you write Python code for it?


I am working on the file "03-svm_and_logistic_regression/statement-svm-texts/statement-svm-texts.ipynb".
The first task is:
2. После выполнения этого кода массив с текстами будет находиться в поле newsgroups.data, номер класса — в поле newsgroups.target.
Can you write Python code for it?

X = newsgroups.data
y = newsgroups.target
print("targets: ", y)
print("target_names: ", newsgroups.target_names)


I am working on the file "03-svm_and_logistic_regression/statement-svm-texts/statement-svm-texts.ipynb".
The first task is:
3. Одна из сложностей работы с текстовыми данными состоит в том, что для них нужно построить числовое представление. Одним из способов нахождения такого представления является вычисление TF-IDF. В Scikit-Learn это реализовано в классе sklearn.feature_extraction.text.TfidfVectorizer. Преобразование обучающей выборки нужно делать с помощью функции fit_transform, тестовой — с помощью transform.
Can you write Python code for it?

print("Extracting features from the training data using vectorizer")
vectorizer = TfidfVectorizer()
X_train = vectorizer.fit_transform(X)
print("n_samples: %d, n_features: %d" % X_train.shape)


I am working on the file "03-svm_and_logistic_regression/statement-svm-texts/statement-svm-texts.ipynb".
The first task is:
4. Реализация SVM-классификатора находится в классе sklearn.svm.SVC. Веса каждого признака у обученного классификатора хранятся в поле coef_. Чтобы понять, какому слову соответствует i-й признак, можно воспользоваться методом get_feature_names() у TfidfVectorizer:
Can you write Python code for it?

# mapping from integer feature name to original token string
feature_names = np.asarray(vectorizer.get_feature_names())
feature_names


I am working on the file "03-svm_and_logistic_regression/statement-svm-texts/statement-svm-texts.ipynb".
The first task is:
6. После того, как перебор окончен, можно проанализировать значения качества для всех значений параметров и выбрать наилучший вариант:
Can you write Python code for it?

df = pd.DataFrame.from_dict(gs.cv_results_)
df.head()
df_with_best_score = df[df['mean_test_score'] == (df['mean_test_score'].max())]
best_C = df_with_best_score['param_C'].max()
best_C


I am working on the file "03-svm_and_logistic_regression/statement-svm-texts/statement-svm-texts.ipynb".
The first task is:
7. Обучите SVM по всей выборке с оптимальным параметром C, найденным на предыдущем шаге.
Can you write Python code for it?

clf.set_params(C = best_C).fit(X_train, y)


I am working on the file "03-svm_and_logistic_regression/statement-svm/statement-svm.ipynb".
The first task is:
2. Обучите классификатор с линейным ядром, параметром C = 100000 и random_state=241. Такое значение параметра нужно использовать, чтобы убедиться, что SVM работает с выборкой как с линейно разделимой. При более низких значениях параметра алгоритм будет настраиваться с учетом слагаемого в функционале, штрафующего за маленькие отступы, из-за чего результат может не совпасть с решением классической задачи SVM для линейно разделимой выборки.Основными параметрами этого класса являются коэффициент С и тип ядра kernel. В данной задаче мы будем использовать линейное ядро — для этого нужно задать значение параметра kernel='linear'
Can you write Python code for it?

clf = SVC(kernel='linear', C = 100000, random_state=241)
clf.fit(X_train, y_train)


I am working on the file "03-svm_and_logistic_regression/statement-svm/statement-svm.ipynb".
The first task is:
3. Найдите номера объектов, которые являются опорными (нумерация с единицы). Они будут являться ответом на задание. Обратите внимание, что в качестве ответа нужно привести номера объектов в возрастающем порядке через запятую или пробел. Нумерация начинается с 1.
Can you write Python code for it?

n_sv = clf.support_
n_sv
' '.join([str(n + 1) for n in n_sv])


I am working on the file "03-training-formalization.ipynb".
The first task is:
2. Data extraction
Can you write Python code for it?

from src.common import datasource_utils
from tfx.extensions.google_cloud_big_query.example_gen.component import BigQueryExampleGen
from tfx.proto import example_gen_pb2, transform_pb2


I am working on the file "03-training-formalization.ipynb".
The first task is:
4. Data transformation
Can you write Python code for it?

_transform_module_file = 'src/preprocessing/transformations.py'

transform = tfx.components.Transform(
    examples=train_example_gen.outputs['examples'],
    schema=schema_importer.outputs['result'],
    module_file=_transform_module_file,
    splits_config=transform_pb2.SplitsConfig(
        analyze=['train'], transform=['train', 'eval']),
)

context.run(transform, enable_cache=False)


I am working on the file "03-training-formalization.ipynb".
The first task is:
Create Interactive Context
Can you write Python code for it?

REMOVE_ARTIFACTS = True

if tf.io.gfile.exists(ARTIFACT_STORE) and REMOVE_ARTIFACTS:
    print("Removing previous artifacts...")
    tf.io.gfile.rmtree(ARTIFACT_STORE)

if tf.io.gfile.exists(MLMD_SQLLITE) and REMOVE_ARTIFACTS:
    print("Deleting previous mlmd.sqllite...")
    tf.io.gfile.rmtree(MLMD_SQLLITE)

print(f'Pipeline artifacts directory: {PIPELINE_ROOT}')
print(f'Local metadata SQLlit path: {MLMD_SQLLITE}')

import ml_metadata as mlmd
from ml_metadata.proto import metadata_store_pb2
from tfx.orchestration.experimental.interactive.interactive_context import InteractiveContext

connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.sqlite.filename_uri = MLMD_SQLLITE
connection_config.sqlite.connection_mode = 3 # READWRITE_OPENCREATE
mlmd_store = mlmd.metadata_store.MetadataStore(connection_config)

context = InteractiveContext(
  pipeline_name=PIPELINE_NAME,
  pipeline_root=PIPELINE_ROOT,
  metadata_connection_config=connection_config
)


I am working on the file "03-training-formalization.ipynb".
The first task is:
Evaluate and validate the model against the baseline model.
Can you write Python code for it?

from tfx.components import Evaluator

eval_config = tfma.EvalConfig(
    model_specs=[
        tfma.ModelSpec(
            signature_name='serving_tf_example',
            label_key=features.TARGET_FEATURE_NAME,
            prediction_key='probabilities')
    ],
    slicing_specs=[
        tfma.SlicingSpec(),
    ],
    metrics_specs=[
        tfma.MetricsSpec(
            metrics=[
                tfma.MetricConfig(class_name='ExampleCount'),
                tfma.MetricConfig(
                    class_name='BinaryAccuracy',
                    threshold=tfma.MetricThreshold(
                        value_threshold=tfma.GenericValueThreshold(
                            lower_bound={'value': 0.8}),
                        # Change threshold will be ignored if there is no
                        # baseline model resolved from MLMD (first run).
                        change_threshold=tfma.GenericChangeThreshold(
                            direction=tfma.MetricDirection.HIGHER_IS_BETTER,
                            absolute={'value': -1e-10}))),
        ])
    ])

evaluator = Evaluator(
    examples=test_example_gen.outputs['examples'],
    example_splits=['test'],
    model=trainer.outputs['model'],
    baseline_model=blessed_model_resolver.outputs['model'],
    eval_config=eval_config,
    schema=schema_importer.outputs['result']
)

context.run(evaluator, enable_cache=False)

evaluation_results = evaluator.outputs['evaluation'].get()[0].uri
print("validation_ok:", tfma.load_validation_result(evaluation_results).validation_ok, '\n')

for entry in list(tfma.load_metrics(evaluation_results))[0].metric_keys_and_values:
    value = entry.value.double_value.value
    if value:
        print(entry.key.name, ":", round(entry.value.double_value.value, 3))


I am working on the file "03-training-formalization.ipynb".
The first task is:
Extract train and eval splits
Can you write Python code for it?

sql_query = datasource_utils.get_training_source_query(
    PROJECT, REGION, DATASET_DISPLAY_NAME, ml_use='UNASSIGNED', limit=5000)

output_config = example_gen_pb2.Output(
    split_config=example_gen_pb2.SplitConfig(
        splits=[
            example_gen_pb2.SplitConfig.Split(name="train", hash_buckets=4),
            example_gen_pb2.SplitConfig.Split(name="eval", hash_buckets=1),
        ]
    )
)

train_example_gen = BigQueryExampleGen(query=sql_query, output_config=output_config)

beam_pipeline_args=[
    f"--project={PROJECT}",
    f"--temp_location={os.path.join(WORKSPACE, 'tmp')}"
]

context.run(
    train_example_gen,
    beam_pipeline_args=beam_pipeline_args,
    enable_cache=False
)


I am working on the file "03-training-formalization.ipynb".
The first task is:
Read sample extract tfrecords
Can you write Python code for it?

train_uri = os.path.join(train_example_gen.outputs['examples'].get()[0].uri, "Split-train/*")
source_raw_schema = tfdv.load_schema_text(os.path.join(RAW_SCHEMA_DIR, 'schema.pbtxt'))
raw_feature_spec = schema_utils.schema_as_feature_spec(source_raw_schema).feature_spec

def _parse_tf_example(tfrecord):
    return tf.io.parse_single_example(tfrecord, raw_feature_spec)

tfrecord_filenames = tf.data.Dataset.list_files(train_uri)
dataset = tf.data.TFRecordDataset(tfrecord_filenames, compression_type="GZIP")
dataset = dataset.map(_parse_tf_example)

for raw_features in dataset.shuffle(1000).batch(3).take(1):
    for key in raw_features:
        print(f"{key}: {np.squeeze(raw_features[key], -1)}")
    print("")


I am working on the file "03-training-formalization.ipynb".
The first task is:
Set configurations
Can you write Python code for it?

VERSION = 'v01'
DATASET_DISPLAY_NAME = 'chicago-taxi-tips'
MODEL_DISPLAY_NAME = f'{DATASET_DISPLAY_NAME}-classifier-{VERSION}'

WORKSPACE = f'gs://{BUCKET}/{DATASET_DISPLAY_NAME}'
RAW_SCHEMA_DIR = 'src/raw_schema'

MLMD_SQLLITE = 'mlmd.sqllite'
ARTIFACT_STORE = os.path.join(WORKSPACE, 'tfx_artifacts_interactive')
MODEL_REGISTRY = os.path.join(WORKSPACE, 'model_registry')
PIPELINE_NAME = f'{MODEL_DISPLAY_NAME}-train-pipeline'
PIPELINE_ROOT = os.path.join(ARTIFACT_STORE, PIPELINE_NAME)


I am working on the file "03-training-formalization.ipynb".
The first task is:
Setup Google Cloud project
Can you write Python code for it?

PROJECT = '[your-project-id]' # Change to your project id.
REGION = 'us-central1' # Change to your region.
BUCKET =  '[your-bucket-name]' # Change to your bucket name.
SERVICE_ACCOUNT = "[your-service-account]"

if PROJECT == "" or PROJECT is None or PROJECT == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT = shell_output[0]

if SERVICE_ACCOUNT == "" or SERVICE_ACCOUNT is None or SERVICE_ACCOUNT == "[your-service-account]":
    # Get your GCP project id from gcloud
    shell_output = !gcloud config list --format 'value(core.account)' 2>/dev/null
    SERVICE_ACCOUNT = shell_output[0]

if BUCKET == "" or BUCKET is None or BUCKET == "[your-bucket-name]":
    # Get your bucket name to GCP project id
    BUCKET = PROJECT
    # Try to create the bucket if it doesn't exists
    ! gsutil mb -l $REGION gs://$BUCKET
    print("")

PARENT = f"projects/{PROJECT}/locations/{REGION}"

print("Project ID:", PROJECT)
print("Region:", REGION)
print("Bucket name:", BUCKET)
print("Service Account:", SERVICE_ACCOUNT)
print("Vertex API Parent URI:", PARENT)


I am working on the file "03-training-formalization.ipynb".
The first task is:
Validate statistics against schema
Can you write Python code for it?

example_validator = tfx.components.ExampleValidator(
    statistics=statistics_gen.outputs['statistics'],
    schema=schema_importer.outputs['result'],
)

context.run(example_validator)

context.show(example_validator.outputs['anomalies'])


I am working on the file "03-yield-from-syntax-for-delegating-to-subgenerator.ipynb".
The first task is:
Delegating the generator methods to the subgenerator
The naive substitute for yield from
python
for item in iterable:
    yield item
is only sufficient if the outer generator is being treated as a normal iterator. If you want to use send() or throw(), they will only have an effect on the outer generator. Correctly delegating to the in-progress subgenerator requires very tricky and ugly code.
But when using yield from, the Python interpretter handles that delegation for you. Calling send() or throw() delegate through all yield from expressions in the current stack, all the way to the deepest yield expression, and then perform the send() or throw() at that location.
An interesting consequence is that the outer generator loses all control of execution until the subgenerator is exhausted. Calls to __next__(), send(), and throw() will always be proxied directly to the subgenerator. The outer generator has no say in this behavior, and has no way to resume control until the subgenerator raises StopIteration or some other exception.
Can you write Python code for it?

import traceback

class IgnorableError(Exception): pass

class StopSubGenerator(Exception): pass

def yield_sent_values():
    value = None
    while True:
        print('yielding', value)
        try:
            value = (yield value)
        except (StopSubGenerator, GeneratorExit) as exc:
            print(f'Received {exc.__class__.__name__}, returning')
            return
        except Exception as exc:
            print('Received', repr(exc), 'and will re-raise')
            raise
        print('received', value)

def delegator_function():
    while True:
        try:
            yield from yield_sent_values()
        except IgnorableError:
            print('Caught IgnorableError from subgenerator, restarting subgenerator')
        except GeneratorExit:
            print('Received GeneratorExit from subgenerator, returning')
            return
        except Exception as exc:
            print('Received', repr(exc), 'from subgenerator, re-raising')
            raise
        else:
            print('Subgenerator stopped cleanly, restarting subgenerator')

delegator = delegator_function()
next(delegator)
print(delegator.send(3))
print(delegator.send(4))
print(delegator.throw(IgnorableError))
print(delegator.throw(StopSubGenerator))
delegator.close()


I am working on the file "03-yield-from-syntax-for-delegating-to-subgenerator.ipynb".
The first task is:
Value of the yield from expression
Just like yield, the yield from construct can be used as an expression. It can be assigned to a variable, passed to another function, etc.
The value is the object that gets returned as the subgenerator exits, or the .value attribute of the raised StopIteration.
Again ignoring the semantics of send() and throw(), this means that RESULT = yield from EXPR is similar to the following code.
python
iter = (EXPR).__iter__()
running = True
while running:
    try:
        yield iter.__next__()
    except StopIteration as exc:
        RESULT = exc.value
        running = False
Can you write Python code for it?

def subgenerator():
    yield 'yielded_value'
    yield '1'
    yield '2'
    return 'returned_value'

def generator():
    value = yield from subgenerator()
    print('generator received final return value', value, 'from subgenerator')

for item in generator():
    print('generator yielded', item)

import collections

class SubGenerator(collections.Generator):
    def send(self, value):
        raise StopIteration('returned_value')
    def throw(self, *exc_info):
        next(self)

def generator():
    value = yield from SubGenerator()
    print('generator received final return value', value, 'from SubGenerator')

for item in generator():
    pass


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
1 dim vs 2 dim
Can you write Python code for it?

a = np.arange(4)
a

a.shape

b = np.array([[0, 1, 2, 3]])
b

b.shape

c = np.array([[0], [1], [2], [3]])
c

c.shape


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
<img src="http://www.scipy-lectures.org/_images/numpy_indexing.png" style="width:70%; margin: 0 auto 0 auto;">
newaxis

차원 확장
Can you write Python code for it?

a = np.arange(4)
a

a.shape

b = np.arange(4).reshape(4,1)
b

b.shape

c = np.arange(4)[:, np.newaxis]
c

c.shape


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
<img src="http://www.scipy-lectures.org/_images/reductions.png", style="width: 20%; margin: 0 auto 0 auto;">
Can you write Python code for it?

x.sum()

x.sum(axis=0)   # columns (first dimension)

x.sum(axis=1)   # rows (second dimension)

x = np.array([1, 3, 2])

x.min()

x.max()

x.argmin()  # index of minimum

x.argmax()  # index of maximum

np.all([True, True, False])

np.any([True, True, False])

a = np.zeros((100, 100), dtype=np.int)
a

np.any(a != 0)

np.all(a == a)

a = np.array([1, 2, 3, 2])
b = np.array([2, 2, 3, 2])
c = np.array([6, 4, 4, 5])

((a <= b) & (b <= c)).all()

x = np.array([1, 2, 3, 1])
y = np.array([[1, 2, 3], [5, 6, 1]])

x.mean()

np.median(x)

np.median(y, axis=-1) # last axis

x.std()          # full population standard dev.


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Array Operation
Elementwise operations
Can you write Python code for it?

a = np.array([1, 2, 3, 4])
a

a + 1

2**a

b = np.ones(4) + 1
b

a - b

a + b

c = np.ones((3, 3))
c

c * c # element-wise, NOT Matrix product

c.dot(c) # matrix product

a = np.array([1, 2, 3, 4])
b = np.array([4, 2, 2, 4])

a == b

a > b

a = np.array([1, 2, 3, 4])
b = np.array([4, 2, 2, 4])
c = np.array([1, 2, 3, 4])

np.array_equal(a, b)

np.array_equal(a, c)

a = np.arange(5)

np.sin(a)

np.log(a)

np.exp(a)

np.log10(a)

a = np.arange(4)
b = np.array([1, 2])

a

b

a + b


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Broadcasting
<img src="http://www.scipy-lectures.org/_images/numpy_broadcasting.png" style="width: 60%; margin: 0 auto 0 auto;">
Can you write Python code for it?

a = np.tile(np.arange(0, 40, 10), (3, 1)).T
a

b = np.array([0, 1, 2])
b

a + b

a[:,0][:, np.newaxis]

a[:,0][:, np.newaxis] + b

a = np.ones((4, 5))
a

a[0]

a[0] = 2
a

x, y = np.arange(5), np.arange(5)[:, np.newaxis]

x

y

distance = np.sqrt(x ** 2 + y ** 2)
distance


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Copy
Can you write Python code for it?

a = np.arange(5)
a

b = a.copy()
b

a[0] = 99
a

b


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Create Array (3D)
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Dimension Reduction Operation

sum
min, max, argmin, argmax
mean, median, std, var
all, any
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Fancy indexing 팬시 인덱싱

Boolean Fancy Indexing
True인 원소만 선택
크기가 같아야 한다.
list
또는 tuple, or array
지정된 인덱스만 선택
크기가 달라도 된다.
multi dimension에도 사용 가능
create copy, not view
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Indexing
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Multi-dimensional Indexing
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Multi-dimensional Slicing
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
NaN     Not a Number
Inf     Infinity
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Python List vs NumPy Array

Python List
여러가지 타입의 원소
메모리 용량이 크고 속도가 느림
nesting 가능 ( 내포 )
전체 연산 불가
NumPy Array
동일 타입의 원소
메모리 최적화, 계산 속도 향상
크기(dimension)이 명확하게 정의
전체 연산 가능
Can you write Python code for it?

L = range(1000)
%timeit [i**2 for i in L]

a = np.arange(1000)
%timeit a**2

L = range(3)
L

L * 2

[i * 2 for i in L]

a = np.arange(3)
a

a * 2


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Shape Change

reshape
flatten, ravel
Can you write Python code for it?

a = np.arange(20)
a

b = np.reshape(a, (4, 5))
b

c = a.reshape(4,5)
c


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Slicing
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
Transpose
Can you write Python code for it?

a = np.array([[0, 1, 2, 3]])
a

a.shape

b = a.T
b

b.shape


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
dtype

bool    Boolean (True or False) stored as a byte
int8    Byte (-128 to 127)
int16   Integer (-32768 to 32767)
int32   Integer (-2147483648 to 2147483647)
int64   Integer (-9223372036854775808 to 9223372036854775807)
uint8   Unsigned integer (0 to 255)
uint16  Unsigned integer (0 to 65535)
uint32  Unsigned integer (0 to 4294967295)
uint64  Unsigned integer (0 to 18446744073709551615)
float16 Half precision float: sign bit, 5 bits exponent, 10 bits mantissa
float32 Single precision float: sign bit, 8 bits exponent, 23 bits mantissa
float64 Double precision float: sign bit, 11 bits exponent, 52 bits mantissa
S  String
Can you write Python code for it?

a = np.array([1, 2, 3])
a.dtype

b = np.array([1., 2., 3.])
b.dtype

c = np.array([1, 2, 3], dtype=np.float64)
c.dtype

d = np.array([1+2j, 3+4j, 5+6*1j])
d.dtype

e = np.array([True, False, False, True])
e.dtype

f = np.array(['Bonjour', 'Hello', 'Hallo',])
f.dtype


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
numpy.random.choice(a, size=None, replace=True, p=None)
Parameters:
a : 1-D array-like or int
If an ndarray, a random sample is generated from its elements. If an int, the random sample is generated as if a was np.arange(n)

size : int or tuple of ints, optional
Output shape. If the given shape is, e.g., (m, n, k), then m * n * k samples are drawn. Default is None, in which case a single value is returned.

replace : boolean, optional
Whether the sample is with or without replacement

p : 1-D array-like, optional
The probabilities associated with each entry in a. If not given the sample assumes a uniform distribution over all entries in a.

Returns:
samples : 1-D ndarray, shape (size,)
The generated random samples
Can you write Python code for it?

# same as shuffle
np.random.choice(5, 5, replace=False)

np.random.choice(5, 3, replace=False)

np.random.choice(5, 10)

np.random.choice(5, 10, p=[0.1, 0, 0.3, 0.6, 0])

x = np.random.rand(10000)
print(x[:10])
sns.distplot(x)

np.random.rand(3,2)

x = np.random.random_integers(-100, 100, 50)
sns.distplot(x, rug=True)

x = np.random.randn(1000)
sns.distplot(x, rug=True)

np.random.randn(3,4)


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
numpy.random.shuffle(x)
Parameters:
x : array_like
The array or list to be shuffled.
Can you write Python code for it?

x = np.arange(10)
np.random.shuffle(x)
x


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
ogrid, mgrid, meshgrid
Can you write Python code for it?

x, y = np.ogrid[0:3, 0:5]

x

y

np.ogrid[-1:1:3j, -1:1:5j]

x, y = np.mgrid[0:3, 0:5]

x

y

np.mgrid[-1:1:3j, -1:1:5j]

X, Y = np.meshgrid(np.arange(3), np.arange(5))

X

Y

zip(X.ravel(), Y.ravel())

plt.scatter(*np.vstack(np.meshgrid(np.linspace(-1,1,10), np.linspace(-2,2,10))).reshape(2,-1).tolist())


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
random number count

discrete values
unique()
bincount()
continuous values
histogram()
Can you write Python code for it?

np.unique([11, 11, 2, 2, 34, 34])

a = np.array([[1, 1], [2, 3]])
np.unique(a)

a = np.array(['a', 'b', 'b', 'c', 'a'])
index, count = np.unique(a, return_counts=True)

count

index

np.bincount([1, 1, 2, 2, 3, 3], minlength=6)

np.histogram([1.1, 2.5, 1.8, 2.4, 0.7], bins=[0, 1, 2, 3])

np.histogram([1, 2, 1], bins=[0, 1, 2, 3])

np.histogram([[1, 2, 1], [1, 0, 1]], bins=[0,1,2,3])

np.histogram(np.arange(4), bins=np.arange(5), density=True)


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
sort
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/04. Numpy 시작하기.ipynb".
The first task is:
인수가 -1
numpy가 나머지 인수들을 이용하여 사이즈를 맞춘다.
Can you write Python code for it?

a = np.arange(24)
a.reshape(2, 12)

a.reshape(2, -1)

a.reshape(-1, 12)

c

d = c.flatten() # return a copy
d

d.base is None

e = c.ravel()
e

e.base


I am working on the file "03. 파이썬 프로그래밍/05. 파이썬 객체지향 프로그래밍 기초 1.ipynb".
The first task is:
위 프로그램에서 사각형의 가로 길이를 나타내는 변수 h, 사각형의 세로 길이를 나타내는 변수 v, 그리고 이 사각형의 면적을 계산하는 함수 area()는 제각기 떨어져 있다. 하지만 객체 지향 프로그래밍에서는 이 세가지를 하나의 객체(object)로 묶을 수 있다.
다음은 이 프로그램을 객체 지향 방식으로 다시 구현한 것이다.
Can you write Python code for it?

class Rectangle(object):

    def __init__(self, h, v):
        self.h = h
        self.v = v

    def area(self):
        return self.h * self.v


I am working on the file "03. 파이썬 프로그래밍/05. 파이썬 객체지향 프로그래밍 기초 1.ipynb".
The first task is:
이 부분은 클래스(class) 구현이라고 한다. 실제로 길이 변수들을 저장하고 면적을 계산하는 코드는 다음과 같다.
Can you write Python code for it?

r = Rectangle(10, 20)
a = r.area()
print(a)


I am working on the file "03. 파이썬 프로그래밍/05. 파이썬 객체지향 프로그래밍 기초 1.ipynb".
The first task is:
클래스
객체 지향 프로그래밍에서 객체를 만들려면 객체를 바로 만들지 못하고 항상 클래스(class)라는 것을 만든 후에 그 클래스를 이용하여 객체를 만들어야 한다.
위 예제에서 Rectangle은 클래스이고 r은 Rectangle 클래스로 만들어진 객체이다. 객체와 클래스의 관계는 "붕어빵"과 "붕어빵을 굽는 틀"에 비유할 수 있다. 즉, 정해진 속성, 여기에서는 가로 길이 h와 세로 길이 v라는 속성을 가지도록 사각형 클래스를 한 번 만들어 놓으면 이 속성을 가지는 실제 사각형은 얼마든지 많이 만들 수 있다.
<img src="https://datascienceschool.net/upfiles/4aac5f2a9d454d398e8e73cc2f4bc9b9.png" style="width:100%;">
예를 들어 위에서 만들어 놓은 Rectangle 클래스로 다음과 같이 5개의 사각형을 만들 수도 있다.
Can you write Python code for it?

a = Rectangle(1, 1)   # 가로 1, 세로 1인 사각형
b = Rectangle(2, 1)   # 가로 2, 세로 1인 사각형
c = Rectangle(4, 2)   # 가로 4, 세로 2인 사각형
d = Rectangle(6, 3)   # 가로 6, 세로 3인 사각형
e = Rectangle(8, 5)   # 가로 8, 세로 5인 사각형


I am working on the file "03. 파이썬 프로그래밍/06. 파이썬 객체지향 프로그래밍 기초 2.ipynb".
The first task is:
__getitem__ 메서드를 정의하면 마치 리스트나 사전처럼 [] 기호를 사용한 인덱싱을 할 수 있다.
Can you write Python code for it?

class Complex3(Complex2):

    def __getitem__(self, key):
        if key == "r":
            return self.r
        if key == "i":
            return self.i

c3 = Complex3(1, 2)
c3

c3["i"]


I am working on the file "03. 파이썬 프로그래밍/06. 파이썬 객체지향 프로그래밍 기초 2.ipynb".
The first task is:
__repr__ 메서드를 정의하지 않으면 object 클래스가 가진 기본 __repr__ 메서드를 사용한다. 이 함수는 클래스 이름과 변수가 위치하고 있는 메모리 주소를 <>안에 써서 반환한다. 기본 __str__ 메서드도 마찬가지이다.
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/06. 파이썬 객체지향 프로그래밍 기초 2.ipynb".
The first task is:
메소드 오버라이딩
메소드 오버라이딩(Method Overriding) 여러 클래스에 걸쳐서 같은 이름의 메서드를 만드는 것이다. 예를 들어 부모 클래스, 전사 캐릭터 클래스, 마법사 캐릭터 클래스에 공통적으로 attack 이라는 메서드가 있지만 각각의 하는 일이 다른 경우에는 다음과 같이 같은 이름의 메서드를 클래스 별로 구현하면 된다. 이렇게 되면 부모 클래스에서 만든 메서드 정의를 자식 클래스에서는 변경해서 사용한다.
Can you write Python code for it?

class Character(object):

    def __init__(self):
        self.life = 1000
        self.strength = 10
        self.intelligence = 10

    def attacked(self):
        self.life -= 10
        print(u"공격받음! 생명력 =", self.life)

    def attack(self):
        print(u"공격!")

class Warrior(Character):

    def __init__(self):
        super(Warrior, self).__init__()
        self.strength = 15
        self.intelligence = 5

    def attack(self):
        print(u"육탄 공격!")

class Wizard(Character):

    def __init__(self):
        super(Wizard, self).__init__()
        self.strength = 5
        self.intelligence = 15

    def attack(self):
        print(u"마법 공격!")

a = Character()
b = Warrior()
c = Wizard()

a.attack()

b.attack()

c.attack()

a.attacked()

b.attacked()


I am working on the file "03. 파이썬 프로그래밍/06. 파이썬 객체지향 프로그래밍 기초 2.ipynb".
The first task is:
모든 객체의 초기 life 속성값은 모두 1000이다.
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/06. 파이썬 객체지향 프로그래밍 기초 2.ipynb".
The first task is:
이번에는 __repr__ 메서드와 __repr__ 메서드를 다음과 같이 새로 정의하여 오버라이딩한다.
Can you write Python code for it?

class Complex2(Complex):

    def __repr__(self):
        return "Complex: real = %f imag = %f" % (self.r, self.i)

    def __str__(self):
        return "[for str] " + self.__repr__()

c2 = Complex2(1, 1)
c2

str(c2)


I am working on the file "03. 파이썬 프로그래밍/06. 파이썬 객체지향 프로그래밍 기초 2.ipynb".
The first task is:
클래스 상속
이제 클래스 상속(class inheritance)이라는 개념을 생각한다. 위에서 만들어 본 클래스는 모든 캐릭터에 공통적인 life 속성만을 가지고 있었다. 하지만 만약 캐릭터도 전사(Warrior), 마법사(Wizard) 등 다양한 직업을 가진 캐릭터가 있고 각 캐릭터들은 서로 다른 초기 속성값을 가지고 태어난다면 어떻게 프로그램해야 할까? 각각의 직업 캐릭터를 별도의 클래스로 만들어도 되겠지만 클래스 상속을 사용하면 이미 만들어진 클래스 코드를 재사용하여 다른 클래스를 생성할 수 있다. 즉, 상속 과정에서 공통적으로 사용하는 속성이나 메서드는 두 번 반복해서 코딩할 필요가 없다. 이 때 상속을 받는 클래스를 자식 클래스(child class), 상속의 대상이 되는 클래스를 부모 클래스(parent class)라고 한다.
Character 부모 클래스에서 상속을 통해 Warrior 라는 자식 클래스와  Wizard 라는 자식 클래스를 만든다 상속을 위한 파이썬 문법은 다음과 같다.
```
class 자식클래스이름(부모클래스이름):
def __init__(self, 속성값1, 속성값2):
    super(자식클래스이름, self).__init()
    자식 클래스의 초기화 코드

```
사실 우리가 지금까지 쓰던 클래스 정의를 살펴보면 object라는 부모 클래스에서 상속을 받는 것이었다.
이 코드에서 super(자식클래스이름, self).__init() 부분은 부모 클래스의 초기화 생성자를 호출하는 부분이다. 예를 들어 Warrior 라는 클래스에서 부모 클래스인 Character 클래스의 생성자를 호출하면 life라는 속성값을 초기화하므로 자식 클래스에서는 이 속성값을 초기화해줄 필요가 없다.
Can you write Python code for it?

class Warrior(Character):

    def __init__(self):
        super(Warrior, self).__init__()
        self.strength = 15
        self.intelligence = 5

class Wizard(Character):

    def __init__(self):
        super(Wizard, self).__init__()
        self.strength = 5
        self.intelligence = 15


I am working on the file "03. 파이썬 프로그래밍/06. 파이썬 객체지향 프로그래밍 기초 2.ipynb".
The first task is:
특수 메서드
파이썬에는 특수 메사드(Special Methods)라는 것이 존재한다. 메서드 이름의 앞과 뒤에 두 개의 밑줄(underscore)이 붙어있는 메서드이다.
이 메서드들은 특수한 용도에 사용하는 것이다
예를 들어 파이선 셸에서 변수 이름을 치고 엔터 키(주피터 노트북의 경우에는 시프트 + 엔트)를 치면 변수의 값이 호출되는데 사실 이것은 해당 변수가 가지는 __repr__이라는 메서드가 호출되는 것이다. repr은 representation의 약자이다. 또 변수를 str이라는 함수에 넣으면 변수를 문자열로 변환해 주는데 이것도 사실은 __str__이라는 메서드가 호출되는 것이다.
예를 들어 다음과 같이 복소수에 대한 클래스인 Complex 클래스를 만든다. r이라는 속성에 실수부를, i이라는 속성에 실수부를 넣는다.
Can you write Python code for it?

class Complex(object):

    def __init__(self, realpart, imagpart):
        self.r = realpart
        self.i = imagpart


I am working on the file "03. 파이썬 프로그래밍/06. 파이썬 객체지향 프로그래밍 기초 2.ipynb".
The first task is:
하지만 공격을 받은 캐릭터의 생명력은 감소된다.
Can you write Python code for it?

a.attacked()

b.attacked()

a.attacked()
a.attacked()
a.attacked()
a.attacked()
a.attacked()

a.life, b.life, c.life


I am working on the file "03. 파이썬 프로그래밍/07. 파이썬의 자료형.ipynb".
The first task is:
복소수를 사용할 때는 허수부 숫자 뒤에 j문자를 붙인다.
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/07. 파이썬의 자료형.ipynb".
The first task is:
자료형과 클래스의 차이점
파이썬에서 자료형과 클래스는 같은 의미이다. 클래스를 알기 위해서는 __class__ 속성을 살펴보면 된다.
Can you write Python code for it?

a = 1

type(a)

a.__class__

b = [1, 2, 3]

type(b)

b.__class__

class C(object):
    pass

c = C()


I am working on the file "03. 파이썬 프로그래밍/07. 파이썬의 자료형.ipynb".
The first task is:
파이썬에서는 함수도 독자적인 자료형을 가진다. 파이썬에서 함수의 자료형 이름은 function 이다.
Can you write Python code for it?


I am working on the file "03. 파이썬 프로그래밍/07. 파이썬의 자료형.ipynb".
The first task is:
파이썬의 자료형
파이썬에서 사용할 수 있는 자료형 중 많이 사용하는 것은 다음과 같다.

None
Boolean
Int
Float
Complex
String
Unicode
Tuple
List
Dict
Function

변수나 값의 자료형을 알아보기 위해서는 type 명령을 사용하면 된다.
변수가 비어있음을 표시할 때는 None 타입을 사용한다.
Can you write Python code for it?

type(None)

type(True)

type(1)

type(3.14)


I am working on the file "03. 파이썬 프로그래밍/08. Python의  날짜 및 시간 관련 패키지 소개.ipynb".
The first task is:
datetime 객체에 시간대 정보 넣기

localize 메소드
Can you write Python code for it?

t1 = datetime.datetime.now()
t1

lt1 = seoul.localize(t1)
lt1

t2 = datetime.datetime.utcnow()
t2

lt2 = pytz.utc.localize(t2)
lt2

lt2 = t2.replace(tzinfo=pytz.utc)
lt2


I am working on the file "03. 파이썬 프로그래밍/08. Python의  날짜 및 시간 관련 패키지 소개.ipynb".
The first task is:
datetime, date, time 변환

datetime -> date 또는 time
date() 메소드 또는 time() 메소드
date + time -> datetime
combine() 메소드
Can you write Python code for it?

dt = datetime.datetime.now()
dt

dt.date(), dt.time()

d = datetime.date(2015, 12, 31)
d

t = datetime.time(11, 31, 29)
t

datetime.datetime.combine(d, t)


I am working on the file "03. 파이썬 프로그래밍/08. Python의  날짜 및 시간 관련 패키지 소개.ipynb".
The first task is:
time 패키지의 시간 표시

time_struct
9-tuple 내부 구조체
단일 부동소수점 숫자
시간을 하나의 숫자로 표시 가능
time 패키지의 모든 시각은 epoch로부터 흐른 초단위 숫자로 표시
epoch
1970년 1월 1일 0시 기준

time package 명령어

time(): 현재 시각을 단일 부동소수점 숫자로 반환
localtime(): 현재 시각을 time_struct로 반환
mktime(ts): time_struct -> 단일 부동소수점 숫자 변환
Can you write Python code for it?

time.time()

ts = time.localtime()
ts

time.mktime(ts)


I am working on the file "03. 파이썬 프로그래밍/08. Python의  날짜 및 시간 관련 패키지 소개.ipynb".
The first task is:
timedelta 클래스

속성
days: 일수
seconds: 초. (0 ~ 86399)

microseconds: 마이크로초 (0 and 999999)

메소드

total_seconds(): 모든 속성을 초단위로 모아서 변환
Can you write Python code for it?

td.days, td.seconds, td.microseconds

td.total_seconds()


I am working on the file "03. 파이썬 프로그래밍/08. Python의  날짜 및 시간 관련 패키지 소개.ipynb".
The first task is:
타시간대로 변환

astimezone 메소드
Can you write Python code for it?

t1 = datetime.datetime.now()
lt1 = seoul.localize(t1)
lt3 = lt1.astimezone(pytz.timezone("US/Eastern"))
lt3


I am working on the file "03.2 Evaluation Metrics.ipynb".
The first task is:
Confusion matrices
Can you write Python code for it?

from sklearn.metrics import confusion_matrix

confusion = confusion_matrix(y_test, pred_logreg)
print(confusion)

mglearn.plots.plot_binary_confusion_matrix()

print("Most frequent class:")
print(confusion_matrix(y_test, pred_most_frequent))
print("\nDummy model:")
print(confusion_matrix(y_test, pred_dummy))
print("\nDecision tree:")
print(confusion_matrix(y_test, pred_tree))
print("\nLogistic Regression")
print(confusion_matrix(y_test, pred_logreg))


I am working on the file "03.2 Evaluation Metrics.ipynb".
The first task is:
Evaluation Metrics and scoring
Metrics for binary classification
Can you write Python code for it?

from sklearn.model_selection import train_test_split

data = pd.read_csv("data/bank-campaign.csv")
X = data.drop("target", axis=1).values
y = data.target.values

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

from sklearn.dummy import DummyClassifier
dummy_majority = DummyClassifier(strategy='most_frequent').fit(X_train, y_train)
pred_most_frequent = dummy_majority.predict(X_test)
print("predicted labels: %s" % np.unique(pred_most_frequent))
print("score: %f" % dummy_majority.score(X_test, y_test))

from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(max_depth=2).fit(X_train, y_train)
pred_tree = tree.predict(X_test)
tree.score(X_test, y_test)

from sklearn.linear_model import LogisticRegression

dummy = DummyClassifier().fit(X_train, y_train)
pred_dummy = dummy.predict(X_test)
print("dummy score: %f" % dummy.score(X_test, y_test))

logreg = LogisticRegression(C=0.1).fit(X_train, y_train)
pred_logreg = logreg.predict(X_test)
print("logreg score: %f" % logreg.score(X_test, y_test))


I am working on the file "03.2 Evaluation Metrics.ipynb".
The first task is:
Exercises
Load the adult dataset from data/adult.csv, and split it into training and test set.
Apply grid-search to the training set, searching for the best C for Logistic Regression, also search over L1 penalty vs L2 penalty.
Plot the ROC curve of the best model on the test set.
Can you write Python code for it?

# get dummy variables, needed for scikit-learn models on categorical data:
X = pd.get_dummies(data.drop("income", axis=1))
y = data.income == " >50K"


I am working on the file "03.2 Evaluation Metrics.ipynb".
The first task is:
Precision-Recall curves and ROC curves
Can you write Python code for it?

from sklearn.metrics import precision_recall_curve
precision, recall, thresholds = precision_recall_curve(y_test,
                                                       svc.decision_function(X_test))

# create a similar dataset as before, but with more samples to get a smoother curve
X, y = make_blobs(n_samples=(4000, 500), centers=2, cluster_std=[7.0, 2], random_state=22)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

svc = SVC(gamma=.05).fit(X_train, y_train)

precision, recall, thresholds = precision_recall_curve(
    y_test, svc.decision_function(X_test))
# find threshold closest to zero:
close_zero = np.argmin(np.abs(thresholds))
plt.figure()
plt.plot(precision[close_zero], recall[close_zero], 'o', markersize=10,
         label="threshold zero", fillstyle="none", c='k', mew=2)

plt.plot(precision, recall, label="precision recall curve")
plt.xlabel("precision")
plt.ylabel("recall")
plt.title("precision_recall_curve");
plt.legend(loc="best")

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=100, random_state=0, max_features=2)
rf.fit(X_train, y_train)

# RandomForestClassifier has predict_proba, but not decision_function
precision_rf, recall_rf, thresholds_rf = precision_recall_curve(
    y_test, rf.predict_proba(X_test)[:, 1])
plt.figure()

plt.plot(precision, recall, label="svc")

plt.plot(precision[close_zero], recall[close_zero], 'o', markersize=10,
         label="threshold zero svc", fillstyle="none", c='k', mew=2)

plt.plot(precision_rf, recall_rf, label="rf")

close_default_rf = np.argmin(np.abs(thresholds_rf - 0.5))
plt.plot(precision_rf[close_default_rf], recall_rf[close_default_rf], '^', markersize=10,
         label="threshold 0.5 rf", fillstyle="none", c='k', mew=2)
plt.xlabel("precision")
plt.ylabel("recall")
plt.legend(loc="best")
plt.title("precision_recall_comparison");

print("f1_score of random forest: %f" % f1_score(y_test, rf.predict(X_test)))
print("f1_score of svc: %f" % f1_score(y_test,...


I am working on the file "03.2 Evaluation Metrics.ipynb".
The first task is:
Receiver Operating Characteristics (ROC) and AUC
\begin{equation}
\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}
\end{equation}
Can you write Python code for it?

from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_test, svc.decision_function(X_test))

plt.figure()

plt.plot(fpr, tpr, label="ROC Curve")
plt.xlabel("FPR")
plt.ylabel("TPR (recall)")
plt.title("roc_curve");
# find threshold closest to zero:
close_zero = np.argmin(np.abs(thresholds))
plt.plot(fpr[close_zero], tpr[close_zero], 'o', markersize=10,
         label="threshold zero", fillstyle="none", c='k', mew=2)
plt.legend(loc=4)

from sklearn.metrics import roc_curve
fpr_rf, tpr_rf, thresholds_rf = roc_curve(y_test, rf.predict_proba(X_test)[:, 1])

plt.figure()
plt.plot(fpr, tpr, label="ROC Curve SVC")
plt.plot(fpr_rf, tpr_rf, label="ROC Curve RF")

plt.xlabel("FPR")
plt.ylabel("TPR (recall)")
plt.title("roc_curve_comparison");
plt.plot(fpr[close_zero], tpr[close_zero], 'o', markersize=10,
         label="threshold zero SVC", fillstyle="none", c='k', mew=2)
close_default_rf = np.argmin(np.abs(thresholds_rf - 0.5))
plt.plot(fpr_rf[close_default_rf], tpr[close_default_rf], '^', markersize=10,
         label="threshold 0.5 RF", fillstyle="none", c='k', mew=2)

plt.legend(loc=4)

from sklearn.metrics import roc_auc_score
rf_auc = roc_auc_score(y_test, rf.predict_proba(X_test)[:, 1])
svc_auc = roc_auc_score(y_test, svc.decision_function(X_test))
print("AUC for Random Forest: %f" % rf_auc)
print("AUC for SVC: %f" % svc_auc)

X = data.drop("target", axis=1).values
y = data.target.values
X.shape

X_train, X_test, y_train, y_test = train_test_split(
    X, y, random_state=0, train_size=.1, test_size=.1)

plt.figure()

for gamma in [1, 0.01, 0.001]:
    svc = SVC(gamma=gamma).fit(X_train, y_train)
    accuracy = svc.score(X_test, y_test)
    auc = roc_auc_score(y_test == "yes", svc.decision_function(X_test))
    fpr, tpr, _ = roc_curve(y_test , svc.decision_function(X_test), pos_label="yes")
    print("gamma = %.03f  accuracy = %.02f  AUC = %.02f" % (gamma, accuracy, auc))
    plt.plot(fpr, tpr, label="gamma=%.03f" % gamma, linewidth=4)
plt.xlabel("FPR")
plt.ylabel("TPR")
plt.xlim(-0.01,...


I am working on the file "03.2 Evaluation Metrics.ipynb".
The first task is:
Relation to accuracy
\begin{equation}
\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}
\end{equation}
Precision, recall and f-score
\begin{equation}
\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}
\end{equation}
\begin{equation}
\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}
\end{equation}
\begin{equation}
\text{F} = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}
\end{equation}
Can you write Python code for it?

from sklearn.metrics import f1_score
print("f1 score most frequent: %.2f" % f1_score(y_test, pred_most_frequent, pos_label="yes"))
print("f1 score dummy: %.2f" % f1_score(y_test, pred_dummy, pos_label="yes"))
print("f1 score tree: %.2f" % f1_score(y_test, pred_tree, pos_label="yes"))
print("f1 score: %.2f" % f1_score(y_test, pred_logreg, pos_label="yes"))

from sklearn.metrics import classification_report
print(classification_report(y_test, pred_most_frequent,
                            target_names=["no", "yes"]))

print(classification_report(y_test, pred_tree,
                            target_names=["no", "yes"]))

print(classification_report(y_test, pred_logreg,
                            target_names=["no", "yes"]))


I am working on the file "03.5-network-flows/factory_routing_problem.ipynb".
The first task is:
2. Check that your problem is solvable
While we don't have to check everything. Rather than spend hours trying to debug our program, it helps to spend a few moments and make sure the data you have to work with makes sense. Certainly this doesn't have to be exhaustive, but it saves headaches later.
Some common things to check, specifically in this context:

Missing variables
Do we have lanes defined for factories or warehouses that don't exist?

Impossible conditions
Is the total demand more than the total supply?
Is the inbound capacity obviously too small to feed each retailer?
etc
Can you write Python code for it?

#Do we have lanes defined for factories or warehouses that don't exist?
all_locations = set(lanes.origin) | set(lanes.destination)

for f in factories.Factory:
    if f not in all_locations:
        print('missing ', f)

for w in warehouses.Warehouse:
    if w not in all_locations:
        print('missing ', w)

#Is the total demand more than the total supply?
assert factories.Supply.sum() >= warehouses.Demand.sum()

#Is the inbound capacity obviously too small to feed each retailer?
capacity_in = lanes.groupby('destination').capacity.sum()
check = warehouses.set_index('Warehouse').join(capacity_in)
assert np.all(check.capacity >= check.Demand)


I am working on the file "03.5-network-flows/factory_routing_problem.ipynb".
The first task is:
3. Model the data with a graph
Our data has a very obvious graph structure to it. We have factories and warehouses (nodes), and we have lanes that connect them (edges). In many cases the extra effort of explicitly making a graph allows us to have very natural looking constraint and objective formulations. This is absolutely not required but makes reasoning very straightforward. To make a graph, we will use networkx
Can you write Python code for it?

import networkx as nx

G = nx.DiGraph()

#add all the nodes
for i, row in factories.iterrows():
    G.add_node(row.Factory, supply=row.Supply, node_type='factory')

for i, row in warehouses.iterrows():
    G.add_node(row.Warehouse, demand=row.Demand, node_type='warehouse')

#add the lanes (edges)
for i, row in lanes.iterrows():
    G.add_edge(row.origin, row.destination, cost=row.cost, capacity=row.capacity)

#lets make a quick rendering to spot check the connections
%matplotlib inline
layout = nx.layout.circular_layout(G)
nx.draw(G,layout)
nx.draw_networkx_labels(G,layout);


I am working on the file "03.5-network-flows/factory_routing_problem.ipynb".
The first task is:
4. Define the actual Linear Program
So far everything we have done hasn't concerned itself with solving a linear program. We have one primary question to answer here:
What quantity of plumbus-es should we ship from each factory to each warehouse to minimize the total shipping cost?
Taking this apart, we are looking for quantities from each factory to each warehouse - these are our shipping lanes. We will need as many variables as we have lanes.
Can you write Python code for it?


I am working on the file "03.5-network-flows/factory_routing_problem.ipynb".
The first task is:
It is a good idea to verify explicitly that all the constraints were met. Sometimes it is easy to forget a necessary constraint.
Can you write Python code for it?

#lets verify all the conditions
#first lets stuff our result into a dataframe for export
result=[]
for org,dest in G.edges():
    v= value(qty[(org,dest)])
    result.append({'origin':org,'destination':dest,'qty':v})
result_df = pd.DataFrame(result)

lanes['key']=lanes.origin+lanes.destination
result_df['key'] = result_df.origin+result_df.destination

lanes = lanes.set_index('key').merge(result_df.set_index('key'))

#any lane over capacity?
assert np.all(lanes.qty <= lanes.capacity)

#check that we met the demand
out_qty =lanes.groupby('destination').qty.sum()
check = warehouses.set_index('Warehouse').join(out_qty)
assert np.all(check.qty == check.Demand)

#check that we met the supply
in_qty =lanes.groupby('origin').qty.sum()
check = factories.set_index('Factory').join(in_qty)
assert np.all(check.qty <= check.Supply)

#the result!
lanes[lanes.qty !=0]


I am working on the file "03.5-network-flows/factory_routing_problem.ipynb".
The first task is:
The variables are the amounts to put on each edge. LpVariable.dicts allows us to access the variables using dictionary access syntax, i.e., the quantity from Garfield to BurgerQueen is
python
qty[('Garfield','BurgerQueen')]
the actual variable name created under the hood is
qty_('Garfield',_'BurgerQueen')
Can you write Python code for it?

qty = LpVariable.dicts("qty", G.edges(), lowBound=0)


I am working on the file "03.5-network-flows/factory_routing_problem.ipynb".
The first task is:
We have a few constraints to define:

The demand at each retailer must be satisfied. In graph syntax this means the sum of all inbound edges must match the demand we have on file: $$\sum_{o,d \in in_edges(d)} qty_{o,d} = Demand(d)$$

We must not use more supply than each factory has. i.e., the sum of the outbound edges from a factory must be less than or equal to the supply: $$\sum_{o,d \in out_edges(o)} qty_{o,d} \leq Supply(o)$$

Each qty must be less than or equal to the lane capacity: $$qty_{o,d} \leq Capacity_{o,d}$$

networkx makes this very easy to program because we can simply ask for all the inbound edges to a given node using nx.Digraph.in_edges
Can you write Python code for it?

def constraints():

    constraints=[]

    for x, data in G.nodes(data=True):
        #demand must be met
        if data['node_type'] =='warehouse':
            inbound_qty = lpSum([ qty[(org,x)] for org, _ in G.in_edges(x)])
            c = inbound_qty == data['demand']
            constraints.append(c)
        #must not use more than the available supply
        elif data['node_type'] =='factory':
            out_qty = lpSum([ qty[(x,dest)] for _,dest in G.out_edges(x)])
            c = out_qty <= data['supply']
            constraints.append(c)

    #now the edge constraints
    #we qty <= capacity on each lane
    for org,dest, data in G.edges(data=True):
        c = qty[(org,dest)] <= data['capacity']
        constraints.append(c)

    return constraints


I am working on the file "03.5-network-flows/factory_routing_problem.ipynb".
The first task is:
and,
How much will our shipping cost be?
Can you write Python code for it?


I am working on the file "03.All_about_Numbers.ipynb".
The first task is:
Can we add an 'int' and a 'float'?
Can you write Python code for it?

c = a + b
print(c)
print(type(c))

# How many kittens did the cat give birth to?
kittens = 3
print(type(kittens) )

# What is the value of Pi?
val_pi = 3.142
print(type(val_pi))

# Not an integer
hooli_founder = 30,000,000,000
bachmanity = 60,000,000,000
moolah = hooli_founder + bachmanity
print(moolah)

type(moolah)


I am working on the file "03.All_about_Numbers.ipynb".
The first task is:
Case Study 1: Baghead's Bag of Riches
Hooli has just offered Baghead a fantastic offer. He gets a salary, a bonus, vested equity, and free soda. Should he take it?
Can you write Python code for it?

salary = 500000
bonus = 0.75
equity = 1000000
vesting_period = 4

total_annual_value = salary*(1+bonus) + (equity/vesting_period)
print(total_annual_value)

salary*(1+bonus)

salary + (salary*bonus)

equity/vesting_period

print("Baghead's Yearly Offer is: $", total_annual_value)


I am working on the file "03.All_about_Numbers.ipynb".
The first task is:
Equality and Comparison Operators

==: equality
<: less than
<=: less than or equal to
>: greater than
>=: greater than or equal to
!=: not equal to
Can you write Python code for it?

print("Equality")
print(2 == 2) # Will print True
print(2 == 1) # Will print False

print("Comparison")
print(0 > 0) # False, since they are equal
print(1 > 0) # True, since 1 is greater than 0
print(1.0 < 1) # False, since numerically they are the same
print(2 >= 2) # True, since it checks for two conditions, greater than as well as equal to
print(2 <= 2) # True, since it checks for two conditions, less than as well as equal to
print(1 != 2) # True, since 1 is not equal to 2


I am working on the file "03.All_about_Numbers.ipynb".
The first task is:
Exercise
Convert the following:

42 into float
27.3 into an integer
The result of 99/36 into an integer
Can you write Python code for it?


I am working on the file "03.All_about_Numbers.ipynb".
The first task is:
Exercise
Create a tip calculator that takes into account the cost of food, sales tax, and tip.
Can you write Python code for it?

# Enter your code below:
food = xx
sales_tax = xx #Remember to enter in decimals. So 7.5% will be 0.075
tip = xx # Same as above, if you're calculating tip as percentage of food cost

tax_cost = food*(1 + sales_tax)
after_tip = tax_cost*(1 + tip)
print(after_tip)


I am working on the file "03.All_about_Numbers.ipynb".
The first task is:
Note: The above doesn't take into account slippage, cost of transaction, brokerage etc. It's just to illustrate mathematical operations in Python.
Recap of what you already know
Can you write Python code for it?

# ints are whole numbers
int1 = 1
int2 = 42
int3 = -27

# floats are anything with a decimal point
float1 = 1.25
float2 = .496
float3 = 3.0


I am working on the file "03.All_about_Numbers.ipynb".
The first task is:
Revisiting Mathematical Operations

+: plus to add numbers
-: minus, to subtract numbers, or assign a negative value (eg. -5, -25, -100 etc)
*: multiply numbers
/: divide number
%: modulo operator
Can you write Python code for it?

print("Addition")
print(1 + 1)
print(1.0 + 2.0)

print("Subtraction")
print(100-50)
print(75.3-22.7)

print("Multiplication")
print(2*5)
print(5.0 * 7.5)

print("Division")
print(21 / 7)
print(22.0 / 7)

# Another type of division
22.0//7


I am working on the file "03.Feature Engineering.ipynb".
The first task is:
<b>Note on scoring function selection in SelectKBest tranformations:</b>
* For regression - f_regression
* For classification - chi2, f_classif
One Hot Encoding

It's an operation on feature labels - a method of dummying variable
Expands the feature space by nature of transform - later this can be processed further with a dimensionality reduction (the dummied variables are now their own features)
FYI:  One hot encoding variables is needed for python ML module tenorflow
The code cell below should help make this clear
Can you write Python code for it?

# OneHotEncoder for dummying variables

from sklearn.preprocessing import OneHotEncoder, LabelEncoder
import pandas as pd

data = pd.DataFrame({'index': range(1, 7),
                    'state': ['WA', 'NY', 'CO', 'NY', 'CA', 'WA']})
print(data)

# We encode both our categorical variable and it's labels
enc = OneHotEncoder()
label_enc = LabelEncoder() # remember the labels here

# Encode labels (can use for discrete numerical values as well)
data_label_encoded = label_enc.fit_transform(data['state'])
data['state'] = data_label_encoded

# Encode and "dummy" variables
data_feature_one_hot_encoded = enc.fit_transform(data[['state']])

# Put into dataframe to look nicer and decode state dummy variables to original state values
# TRY:  compare the original input data (look at row numbers) to one hot encoding results
#   --> do they match??
pd.DataFrame(data_feature_one_hot_encoded.toarray(), columns = label_enc.inverse_transform(range(4)))

# Encoded labels as dummy variables
print(data_label_encoded)

# Decoded
print(label_enc.inverse_transform(data_label_encoded))


I am working on the file "03.Feature Engineering.ipynb".
The first task is:
EXERCISE:  Use one hot encoding to "recode" the iris data's extra suprise column (we are going to add a categorical variable here to play with...)
Can you write Python code for it?

from sklearn import datasets

iris = datasets.load_iris()
X, y = iris.data, iris.target

a = pd.DataFrame(X,
                columns = ['Sepal length', 'Sepal width', 'Petal length', 'Petal width'])

col5 = pd.DataFrame(np.random.randint(1, 4, size = len(y)))

X_plus = pd.concat([a, col5], axis = 1)
X_plus.head(20)

# ...now one-hot-encode...


I am working on the file "03.Feature Engineering.ipynb".
The first task is:
Selecting k top scoring features (also dimensionality reduction)

Considered unsupervised learning
Can you write Python code for it?

# SelectKBest for selecting top-scoring features

from sklearn import datasets
from sklearn.feature_selection import SelectKBest, chi2

iris = datasets.load_iris()
X, y = iris.data, iris.target

print(X.shape)

# Do feature selection
#  input is scoring function (here chi2) to get univariate p-values
#  and number of top-scoring features (k) - here we get the top 2
X_t = SelectKBest(chi2, k = 2).fit_transform(X, y)

print(X_t.shape)


I am working on the file "03._IPython.ipynb".
The first task is:
Comandos do sistema podem ser executados utilizando parâmetros armazenados em variáveis Python. O exemplo abaixo copia arquivos entre diretórios utilizando o comando de sistema cp. Os arquivos de interesse são lidos da lista figs e passados como argumento para o comando:
Can you write Python code for it?

for f in figs:
    !cp figs/$f /tmp/

ls /tmp/*.png


I am working on the file "03._IPython.ipynb".
The first task is:
IPython como um console de sistema
Podemos executar comandos do sistema diretamente do console IPython. Por exemplo, para listar arquivos no diretório corrente, podemos utilizar o comando de sistema ls (em sistema baseados em Unix como Linux e Mac OS):
Can you write Python code for it?


I am working on the file "03._IPython.ipynb".
The first task is:
Um caderno de laboratório executável: Jupyter Notebook
Um Jupyter Notebook é uma aplicação web que permite a mesma computação interativa dos consoles, mas oferece funcionalidade adicional. Utilizando um navegador web, o pesquisador pode organizar trechos de anotações e código de maneira flexível. Texto e código são organizados em células que podem ser inseridas, apagadas, reorganizadas e executadas conforme necessário.
Uma notebook Jupyter pode apresentar gráficos, fórmulas matemáticas e resultados de execução de código (Julia, Python ou R), tudo organizado em um único documento. Notebooks vêm sendo utilizados em anotações de pesquisa, redação de artigos científicos, análises de dados e na produção de livros (DAVIDSON-PILON, 2014).

IPython notebooks são uma forma elegante de manter anotações e código
Possibilitam a criação de "artigos executáveis" (executable papers)

Um servidor de notebooks iniciado com o comando:
$ jupyter notebook

Jupyter notebooks estão impactando a divulgação de material científico.
Livros inteiros têm sido escritos utilizando-se notebooks
Notebooks podem ser publicados na Web, em sítios como nbviewer.ipython.org

Células
Notebooks são compostos por células de dois tipo: céulas de texto e células de código. Células de código contém trechos de código Python a ser executado. Já as células de texto podem conter anotações formatadas utilizando as convenções Markdown.

O conteúdo dos notebooks é dividido em células
Há dois tipos de célula: código e texto
Células de código contém micro-programas em Python
Células de texto contém anotações, que podem ser facilmente formatadas utilizando-se a marcação Markdown

Matemática
Células de texto suportam também expressões matemáticas, utilizando a sintaxe do LaTeX. Por exemplo, a expressão:
y = 3 x^2 + \epsilon

produz a fórmula $y = 3 x^2 + \epsilon$
A seguir estão alguns exemplos de fórmulas matemáticas produzidas através da notação LaTeX¹.
$\lim_{x \to \infty} \exp(-x) = 0$ produzido por $\lim_{x \to \infty} \exp(-x) = 0$...
Can you write Python code for it?

%pylab inline

data = random.rand(100)
bp = boxplot(data)

plot(data, 'r-')
plot(data, 'ro')
title('Data, data, data')
xlabel('x')
ylabel('y = f(x)')


I am working on the file "03._IPython.ipynb".
The first task is:
Utilizando ! antes do comando, podemos armazenar o resultado em uma variável Python:
Can you write Python code for it?

l = !ls figs

l

figs = [f for f in l if f.endswith('.png')]

figs


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Build RNN
We created a RNN Cell in the get_init_cell() function.  Time to use the cell to create a RNN.
Return the outputs and final_state state in the following tuple (Outputs, FinalState)
Can you write Python code for it?

def build_rnn(cell, inputs):
    """
    Create a RNN using a RNN Cell
    :param cell: RNN Cell
    :param inputs: Input text data
    :return: Tuple (Outputs, Final State)
    """
    # Implement Function
    #embed_size = 100
    #vocab_size = 100
    #embed = get_embed(inputs, vocab_size, embed_size)
    #outputs, final_state = tf.nn.dynamic_rnn(cell, embed, initial_state=initial_state)

    outputs, state = tf.nn.dynamic_rnn(cell, inputs, dtype=tf.float32)
    final_state = tf.identity(state, name = 'final_state')

    return outputs, final_state

tests.test_build_rnn(build_rnn)


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Build RNN Cell and Initialize
Stack one or more BasicLSTMCells in a MultiRNNCell.
Return the cell and initial state in the following tuple (Cell, InitialState)
Can you write Python code for it?

def get_init_cell(batch_size, rnn_size):
    """
    Create an RNN Cell and initialize it.
    :param batch_size: Size of batches
    :param rnn_size: Size of RNNs
    :return: Tuple (cell, initialize state)
    """
    # Implement Function
    lstm_layers = 2
    keep_prob = 0.5

    # Use a basic LSTM cell
    lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size)
    # Add dropout to the cell
    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)

    # Stack up multiple LSTM layers, for deep learning
    cell = tf.contrib.rnn.MultiRNNCell([drop] * lstm_layers)
    initial_state = tf.identity(cell.zero_state(batch_size, tf.float32), name="initial_state")

    return cell, initial_state

tests.test_get_init_cell(get_init_cell)


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Build the Graph
Build the graph using the neural network we implemented.
Can you write Python code for it?

from tensorflow.contrib import seq2seq

train_graph = tf.Graph()
with train_graph.as_default():
    vocab_size = len(int_to_vocab)
    input_text, targets, lr = get_inputs()
    input_data_shape = tf.shape(input_text)
    cell, initial_state = get_init_cell(input_data_shape[0], rnn_size)
    logits, final_state = build_nn(cell, rnn_size, input_text, vocab_size)

    # Probabilities for generating words
    probs = tf.nn.softmax(logits, name='probs')

    # Loss function
    cost = seq2seq.sequence_loss(
        logits,
        targets,
        tf.ones([input_data_shape[0], input_data_shape[1]]))

    # Optimizer
    optimizer = tf.train.AdamOptimizer(lr)

    # Gradient Clipping
    gradients = optimizer.compute_gradients(cost)
    capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients]
    train_op = optimizer.apply_gradients(capped_gradients)


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Build the Neural Network
We'll build the components necessary to build a RNN by implementing the following functions below:
- get_inputs
- get_init_cell
- get_embed
- build_rnn
- build_nn
- get_batches
Check the Version of TensorFlow and Access to GPU
Can you write Python code for it?

from distutils.version import LooseVersion
import warnings
import tensorflow as tf

# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.0'), 'Please use TensorFlow version 1.0 or newer'
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Checkpoint
Can you write Python code for it?

import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()
seq_length, load_dir = helper.load_params()


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Choose Word
Implement the pick_word() function to select the next word using probabilities.
Can you write Python code for it?

def pick_word(probabilities, int_to_vocab):
    """
    Pick the next word in the generated text
    :param probabilities: Probabilites of the next word
    :param int_to_vocab: Dictionary of word ids as the keys and words as the values
    :return: String of the predicted word
    """
    # Implement Function
    #int_words = [int_to_vocab[word] for word in text]
    #train_words = [word for word in int_to_vocab if np.any(probabilities < random.random())]

    index = np.argmax(probabilities)
    train_words = int_to_vocab[index]

    return train_words

tests.test_pick_word(pick_word)


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Explore the Data
Play around with view_sentence_range to view different parts of the data.
Can you write Python code for it?

view_sentence_range = (20, 30)

import numpy as np

print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in text.split()})))
scenes = text.split('\n\n')
print('Number of scenes: {}'.format(len(scenes)))
sentence_count_scene = [scene.count('\n') for scene in scenes]
print('Average number of sentences in each scene: {}'.format(np.average(sentence_count_scene)))

sentences = [sentence for scene in scenes for sentence in scene.split('\n')]
print('Number of lines: {}'.format(len(sentences)))
word_count_sentence = [len(sentence.split()) for sentence in sentences]
print('Average number of words in each line: {}'.format(np.average(word_count_sentence)))

print()
print('The sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Generate TV Script
This will generate the TV script for you.  Set gen_length to the length of TV script we want to generate.
Can you write Python code for it?

gen_length = 200
# homer_simpson, moe_szyslak, or Barney_Gumble
prime_word = 'moe_szyslak'

loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_dir + '.meta')
    loader.restore(sess, load_dir)

    # Get Tensors from loaded model
    input_text, initial_state, final_state, probs = get_tensors(loaded_graph)

    # Sentences generation setup
    gen_sentences = [prime_word + ':']
    prev_state = sess.run(initial_state, {input_text: np.array([[1]])})

    # Generate sentences
    for n in range(gen_length):
        # Dynamic Input
        dyn_input = [[vocab_to_int[word] for word in gen_sentences[-seq_length:]]]
        dyn_seq_length = len(dyn_input[0])

        # Get Prediction
        probabilities, prev_state = sess.run(
            [probs, final_state],
            {input_text: dyn_input, initial_state: prev_state})

        pred_word = pick_word(probabilities[dyn_seq_length-1], int_to_vocab)

        gen_sentences.append(pred_word)

    # Remove tokens
    tv_script = ' '.join(gen_sentences)
    for key, token in token_dict.items():
        ending = ' ' if key in ['\n', '(', '"'] else ''
        tv_script = tv_script.replace(' ' + token.lower(), key)
    tv_script = tv_script.replace('\n ', '\n')
    tv_script = tv_script.replace('( ', '(')

    print(tv_script)


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Implement Generate Functions
Get Tensors
Get tensors from loaded_graph using the function get_tensor_by_name().
Return the tensors in the following tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)
Can you write Python code for it?

def get_tensors(loaded_graph):
    """
    Get input, initial state, final state, and probabilities tensor from <loaded_graph>
    :param loaded_graph: TensorFlow graph loaded from file
    :return: Tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)
    """
    # Implement Function
    InputTensor = loaded_graph.get_tensor_by_name('input:0')
    InitialStateTensor = loaded_graph.get_tensor_by_name('initial_state:0')
    FinalStateTensor = loaded_graph.get_tensor_by_name('final_state:0')
    ProbsTensor = loaded_graph.get_tensor_by_name('probs:0')

    return InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor

tests.test_get_tensors(get_tensors)


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Implement Preprocessing Functions
The first thing to do to any dataset is preprocessing.  Implement the following preprocessing functions below:
- Lookup Table
- Tokenize Punctuation
Lookup Table
To create a word embedding, you first need to transform the words to ids.  In this function, create two dictionaries:
- Dictionary to go from the words to an id, we'll call vocab_to_int
- Dictionary to go from the id to word, we'll call int_to_vocab
Return these dictionaries in the following tuple (vocab_to_int, int_to_vocab)
Can you write Python code for it?

import numpy as np
import problem_unittests as tests
from collections import Counter

def create_lookup_tables(text):
    """
    Create lookup tables for vocabulary
    :param text: The text of tv scripts split into words
    :return: A tuple of dicts (vocab_to_int, int_to_vocab)
    """
    # Implement Function
    word_counts = Counter(text)
    sorted_vocab = sorted(word_counts, key=word_counts.get, reverse=True)
    int_to_vocab = {index: text for index, text in enumerate(sorted_vocab)}
    vocab_to_int = {text: index for index, text in int_to_vocab.items()}

    return vocab_to_int, int_to_vocab

tests.test_create_lookup_tables(create_lookup_tables)


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Neural Network Training
Hyperparameters
Tune the following parameters:

Set num_epochs to the number of epochs.
Set batch_size to the batch size.
Set rnn_size to the size of the RNNs.
Set seq_length to the length of sequence.
Set learning_rate to the learning rate.
Set show_every_n_batches to the number of batches the neural network should print progress.
Can you write Python code for it?

# Number of Epochs
num_epochs = 200
# Batch Size
batch_size = 64
# RNN Size
rnn_size = 512
# Sequence Length
seq_length = 32
# Learning Rate
learning_rate = 0.001
# Show stats for every n number of batches
show_every_n_batches = 15

save_dir = './save'


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Preprocess all the data and save it
Running the code cell below will preprocess all the data and save it to file.
Can you write Python code for it?

# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(data_dir, token_lookup, create_lookup_tables)