I am working on the file "25--logreg.ipynb".
The first task is:
Exercise. Verify the following properties of the logistic function, $G(y)$.
$$
\begin{array}{rcll}
  G(y)
    & = & \frac{e^y}{e^y + 1}
    & \mathrm{(P1)} \
  G(-y)
    & = & 1 - G(y)
    & \mathrm{(P2)} \
  \dfrac{dG}{dy}
    & = & G(y) G(-y)
    & \mathrm{(P3)} \
  {\dfrac{d}{dy}} {\left[ \ln G(y) \right]}
    & = & G(-y)
    & \mathrm{(P4)} \
  {\dfrac{d}{dy}} {\ln \left[ 1 - G(y) \right]}
    & = & -G(y)
    & \mathrm{(P5)}
\end{array}
$$
Determining $\theta$ via Maximum Likelihood Estimation
Previously, you determined $\theta$ for our synthetic dataset experimentally. Can you compute a good $\theta$ automatically? One of the standard techniques in statistics is to perform a maximum likelihood estimation (MLE) of a model's parameters, $\theta$.
Indeed, MLE is basis for the "statistical" way to derive the normal equations in the case of linear regression, though that is of course not how we encountered it in this class.
"Likelihood" as an objective function
MLE derives from the following idea. Consider the joint probability of observing all of the labels, given the points and the parameters, $\theta$:
$$
  \mathrm{Pr}[l\,|\,X, \theta].
$$
Suppose these observations are independent and identically distributed (i.i.d.). Then the joint probability can be factored as the product of individual probabilities,
$$
\begin{array}{rcl}
  \mathrm{Pr}[l\,|\,X,\theta] = \mathrm{Pr}[l_0, \ldots, l_{m-1}\,|\,x_0, \ldots, x_{m-1}, \theta]
  & = & \mathrm{Pr}[l_0\,|\,x_0, \theta] \cdots \mathrm{Pr}[l_{m-1}\,|\,x_{m-1}, \theta] \
  & = & \displaystyle \prod_{i=0}^{m-1} \mathrm{Pr}[l_i\,|\,x_i,\theta].
\end{array}
$$
The maximum likelihood principle says that you should try to choose a parameter $\theta$ that maximizes the chances ("likelihood") of seeing these particular observations. Thus, we can simply reinterpret the preceding probability as an objective function to optimize. Mathematically, it is equivalent and convenient to consider the logarithm of the likelihood, or log-likelihood, as the objective function, defining...
Can you write Python code for it?

MAX_STEP = 100
PHI = 0.1

# Get the data coordinate matrix, X, and labels vector, l
X = points
l = labels.astype (dtype=float)

# Store *all* guesses, for subsequent analysis
thetas = np.zeros ((3, MAX_STEP+1))

for t in range (MAX_STEP):
    # @YOUSE: Fill in this code
    pass

print "Your (hand) solution:", theta.T.flatten ()
print "Computed solution:", thetas[:, MAX_STEP]

theta_mle = thetas[:, MAX_STEP:]

# Generate 0/1 labels for computed discriminant:
is_correct_mle = gen_labels_part3 (points, labels, theta_mle)

print "Number of misclassified points using MLE:", (len (points) - sum (is_correct_mle))[0]
print "\n(Run the code cell below to visualize the results.)"

# Visually inspect the above results
traces_mle = make_2d_scatter_traces (points, is_correct_mle)
traces_mle.append (gen_lin_discr_trace (points, theta_mle))

# Plot it!
layout_mle = Layout (xaxis=dict (range=[-1.25, 2.25]),
                     yaxis=dict (range=[-3.25, 2.25]))
fig_mle = Figure (data=traces_mle, layout=layout_mle)
py.iplot (fig_mle)


I am working on the file "25--logreg.ipynb".
The first task is:
However, as the lobsters example suggests, real data are not likely to be cleanly separable, especially when the number of features we have at our disposal is relatively small.
Since the labels are binary, a natural idea is to give the classification problem a probabilistic interpretation. The logistic function provides at least one way to do so:
$$
\begin{array}{rcl}
  G(y) & \equiv & \frac{1}{1 + e^{-y}}
\end{array}
$$

This function is also sometimes called the logit or sigmoid function.

The logistic function takes any value in the range $(-\infty, +\infty)$ and produces a value in the range $(0, 1)$. Thus, given a value $x$, we can interpret it as a conditional probability that the label is 1.
Exercise. Consider a set of 1-D points generated by a mixture of Gaussians. That is, suppose that there are two Gaussian distributions over the 1-dimensional variable, $x \in (-\infty, +\infty)$, that have the same variance ($\sigma^2$) but different means ($\mu_0$ and $\mu_1$). Show that the conditional probability of observing a point labeled "1" given $x$ may be written as,
$$\mathrm{Pr}\left[l=1\,|\,x\right]
    \propto \displaystyle \frac{1}{1 + e^{-(\theta_0 + \theta_1 x)}},$$
for a suitable definition of $\theta_0$ and $\theta_1$. To carry out this computation, recall Bayes's rule (also: Bayes's theorem):
$$
\begin{array}{rcl}
  \mathrm{Pr}[l=1\,|\,x]
    & = &
      \dfrac{\mathrm{Pr}[x\,|\,l=1] \, \mathrm{Pr}[l=1]}
            {\mathrm{Pr}[x\,|\,l=0] \, \mathrm{Pr}[l=0]
             + \mathrm{Pr}[x\,|\,l=1] \, \mathrm{Pr}[l=1]
            }.
\end{array}
$$
You may assume the prior probabilities of observing a 0 or 1 are given by $\mathrm{Pr}[l=0] \equiv p_0$ and $\mathrm{Pr}[l=1] \equiv p_1$.

Time and interest permitting, we'll solve this exercise on the whiteboard.

Exercise. Implement the logistic function. Inspect the resulting plot of $G(y)$ in 1-D and then the contour plot of $G(\theta^T{x})$. Your function should accept a Numpy matrix of values, Y, and apply the sigmoid elementwise.
Can you write Python code for it?

def logistic (Y):
    # @YOUSE: Implement the logistic function G(y) here
    pass


I am working on the file "25--logreg.ipynb".
The first task is:
Next, let's extract the coordinates as a Numpy matrix of points and the labels as a Numpy column vector labels. Mathematically, the points matrix corresponds to $X$ and the labels vector corresponds to $l$.
Can you write Python code for it?

points = np.insert (df.as_matrix (['x_1', 'x_2']), 0, 1.0, axis=1)
labels = df.as_matrix (['label'])

print "First and last 5 points:\n", '='*23, '\n', points[:5], '\n...\n', points[-5:], '\n'
print "First and last 5 labels:\n", '='*23, '\n', labels[:5], '\n...\n', labels[-5:], '\n'


I am working on the file "25--logreg.ipynb".
The first task is:
Note about slicing columns from a Numpy matrix
If you want to extract a column i from a Numpy matrix A and keep it as a column vector, you need to use the slicing notation, A[:, i:i+1]. Not doing so can lead to subtle bugs. To see why, compare the following slices.
Can you write Python code for it?


I am working on the file "25--logreg.ipynb".
The first task is:
Sample data: Rock lobsters!
As a concrete example of a classification task, consider the results of this experiment. Some marine biologists took a bunch of lobsters of varying sizes (size being a proxy for stage of development), and then tethered and exposed these lobsters to a variety of predators. The outcome that they measured is whether the lobsters survived or not.
In this case, the data consists of a set of points, one point per lobster, where there is a single predictor (size) and the response is whether the lobsters survived (label "1") or died (label "0").

For the original paper, see this link. I can only imagine that this image is what marine biologists look like when experimenting with lobsters.

Here is a plot of the raw data.
Can you write Python code for it?

# http://www.stat.ufl.edu/~winner/data/lobster_survive.txt
df_lobsters = pd.read_table ('http://www.stat.ufl.edu/~winner/data/lobster_survive.dat',
                             sep=r'\s+', names=['CarapaceLen', 'Survived'])
display (df_lobsters.head ())
print "..."
display (df_lobsters.tail ())

sns.violinplot (x="Survived", y="CarapaceLen",
                data=df_lobsters, inner="quart")


I am working on the file "25--logreg.ipynb".
The first task is:
The following is the code to generate the plot; look for the place to try different values of $\theta$ a couple of code cells below.
Can you write Python code for it?

def heaviside_int (Y):
    """Evaluates the heaviside function, but returns integer values."""
    return heaviside (Y).astype (dtype=int)

def assert_discriminant (theta, d=2):
    """
    Verifies that the given coefficients correspond to a
    d-dimensional linear discriminant ($\theta$).
    """
    assert len (theta) == (d+1)

def gen_lin_discr_labels (points, theta, fun=heaviside_int):
    """
    Given a set of points and the coefficients of a linear
    discriminant, this function returns a set of labels for
    the points with respect to this discriminant.
    """
    assert_points_2d (points)
    assert_discriminant (theta)

    score = lin_discr (points, theta)
    labels = fun (score)
    return labels

def gen_lin_discr_trace (points, theta, name='Discriminant'):
    """
    Given a set of points and the coefficients of a linear
    discriminant, this function returns a set of Plotly
    traces that show how the points are classified as well
    as the location of the discriminant boundary.
    """
    assert_points_2d (points)
    assert_discriminant (theta)

    x1 = [min (points[:, 1]), max (points[:, 1])]
    m = -theta[1] / theta[2]
    b = -theta[0] / theta[2]
    x2 = [(b + m*x) for x in x1]

    return Scatter (x=x1, y=x2, mode='lines', name=name)

def np_row_vec (init_list):
    """Generates a Numpy-compatible row vector."""
    return np.array (init_list, order='F', ndmin=2)

def np_col_vec (init_list):
    """Generates a Numpy-compatible column vector."""
    return np_row_vec (init_list).T

def gen_labels_part3 (points, labels, theta):
    your_labels = gen_lin_discr_labels (points, theta)
    return (labels == your_labels)

# @YOUSE: Part 3 -- Select parameters for theta!
theta = np_col_vec ([0., -1., 3.])

# Generate 0/1 labels for your discriminant:
is_correct = gen_labels_part3 (points, labels, theta)

print "Number of misclassified points:", (len (points) - sum (is_correct))[0]
print "\n(Run the code cell below to visualize the results.)"

# Visually inspect...


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
5. File writing

File writing follows a similar pattern to reading. The steps are:

Open the output file (automatically created if it doesn't exist)
Write to the file
Close the output file

Here is a simple example of file writing:
Can you write Python code for it?

# print some text to a new file
fileName = "output.txt"
outFile = open(fileName, 'w')

outFile.write("This is me,")
outFile.write("printing to \n a file.")

outFile.close()


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
6. Test your understanding: practice set 3

For the following blocks of code, first try to guess what the output will be, and then run the code yourself. These examples may introduce some ideas and common pitfalls that were not explicitly covered in the text above, so be sure to complete this section.
Can you write Python code for it?

for i in range(1, 10, 2):
    print i

for i in range (5, 1, -1):
    print i

count = 0
while (count < 5):
    print count
    count = count + 1

total = 0
for i in range(4):
    total = total + i
print total

name = "Mits"
for letter in name:
    print letter

name = "Wilfred"
newName = ""
for letter in name:
    newName = newName + letter
print newName

name = "Wilfred"
newName = ""
for letter in name:
    newName = letter + newName
print newName


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
But as you'll see today, we can accomplish the same thing like this:
Can you write Python code for it?

import random

count = 0
while count < 10:
    print random.randint(0,1)
    count = count + 1


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
Here, we just ignored the value of char in the loop, and just made use of the fact that the loop would loop a pre-defined number of times (equal to the number of characters in the string).
There are a lot of cases where doing this sort of thing could be quite useful. For example, at the very beginning of the lesson, we wanted to generate a certain number of random numbers. Luckily, there's an easier way to do this that doesn't involve generating strings of different lengths!
Iterating a certain number of times
Now it's time to introduce a new function called range(). You've already seen it in action a few times in this lesson, but now we'll look at it a little more closely. On the most superficial level, all you need to know is that "range(x)" will cause the for loop to repeat x times. For example:
Can you write Python code for it?

for i in range(5):
    print "This will print 5 times."


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
If you run this code, it should print a new file called output.txt to your current directory (probably the directory where this notebook is stored). Go and take a look at it now. It should look like this:
This is me,printing to
 a file.

Let's go over what this code is doing, line by line.
fileName = "output.txt"
outFile = open(fileName, 'w')

Here, we opened output file called "output.txt". Note that this looks just like file reading, except we used 'w' instead of 'r' in the open() function. This tells the function that you want this to be an output file. Note that if this file does not yet exist, Python will create it for you. More importantly, though, if the file does already exist, Python will completely overwrite it! So be careful.
outFile.write("This is me,")
outFile.write("printing to \n a file.")

The .write() function is what allows us to specify what text should be written. We can call this function as many times as we want while the output file is open to print things. Important note: unlike print, .write() does not automatically add a newline \n to the end of each line you print! You have to manually add these yourself. If you look at the output of this code, you'll see that the line break occurs only where we specified the \n, and nowhere else!
outFile.close()

Finally, as with file reading, it's good practice to always close your file connection when you are finished. Strange things can happen if you don't, such as incomplete printing.
You can only print strings to a file!
If you want to print numerical data (ints or floats), you will need to convert them to strings before using .write(). For example:
Can you write Python code for it?

fileName = "output2.txt"
outFile = open(fileName, 'w')

outFile.write(25)

outFile.close()

# The simple fix: use str()
fileName = "output2.txt"
outFile = open(fileName, 'w')

outFile.write(str(25))

outFile.close()


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
Let's break down what's happening here:

notGuessed is initially True, so we enter the loop.
We ask the user to input a number, which we convert to an int and compare against our "secret" number.
If the user guessed correctly, we simply set notGuessed to False. This makes the loop's conditional False, so we will stop looping once the conditional is checked again (this will happen once the rest of the code in the block is completed).
If the user guessed wrong, we leave notGuessed as True, and therefore repeat the loop again.

3. The for loop

The for loop works fairly differently. The main purpose of the for loop is to provide an easier way to (A) loop a specific number of times, and (B) loop over some iterable object and run the same code on each item in the iterable (e.g. every character in a string, every element in a list, or every line in a file). In fact, we can do both of these things with a while loop, too, but the for loop makes it a little more convenient.

[ Definition ] The for loop
Purpose: execute a block of code a specific number of times or loop over an iterable object.
Syntax:
for var in iterable:
    this indented code will execute until we run out of things in the iterable

Notes:

iterable - Anything that you can iterate over, including lists, strings, files, and dictionaries. The actual unit of iteration (i.e. the subunits of the iterable) is pre-defined by Python and depends on the type of the iterable. For strings, the unit is characters. For lists, the unit is elements of the list. We'll talk about others as they come up.
var - A variable. You can name this anything you want, and you do not need to declare/define it beforehand.
What happens here is that var takes on each value in the iterable, one at a time. We can then use that value in the code of the loop by using var. Each time the loop loops, the previous value of var is overwritten with the next value in the iterable. When there are no more values in the iterable, the loop ends.

Examples:
Can you write Python code for it?

for i in [1, "A", 45, True]:
    print i

for i in "Hello!":
    print i

for i in range(5):
    print i


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
Now that we understand \n... what was going on in the previous example?
When we read each line of the file, there is a \n on the end of each line. Like a text editor, Python uses the location of each \n to figure out where one line ends and the next begins. Importantly, though, when Python reads in a given line, it includes the \n (or \r\n) with it as part of the string. Then, when we go to print out the line using print, the print statement adds another \n on the end! This is what causes the double spacing – we technically have \n\n on the end of each string.
As we'll see later, it can be very problematic to have these pesky \n characters at the end of every line we read in. Therefore, we'll almost always want to remove them before do any computations or processing of the line. To do this, we'll add the following line to our code:
line = line.rstrip('\r\n')

What rstrip() does is strip the indicated characters from the right side of the string, if and only if they are present (so if for some reason there was no \r or \n, this would do nothing). It will strip all the characters indicated, even if they are not in the indicated order. We'll strip off both \r and \n, just to be safe.
Here's the same code, but with \n removal. You can use this as a general template for almost all file reading!
Can you write Python code for it?

# Read and print genes.txt
fileName = "genes.txt"

inFile = open(fileName, 'r')
for line in inFile:
    line = line.rstrip('\r\n')
    print line
inFile.close()


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
Or, to give a more useful example, we may want to do something like count the number of times a certain nucleotide appears:
Can you write Python code for it?

sequence = "ATGGTCGATCGGTCGGGCTCGGGATATTACCGCGCGCGCGATGGCTAGGGGGG"
count = 0

for nt in sequence:
    if nt == "G":
        count = count + 1

print "Found", count, "G's"


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
These two code blocks are called the while loop and for loop, respectively. Much like the if / else statement, these blocks allow our code to make decisions for us based on some condition. The reason we have two different types of loops is because they are designed with different situations in mind. The for loop is specialized for cases where you know exactly how many times you want to loop in advance, and makes it very easy to "loop over" the elements of almost any iterable object (more on that in a minute). while loops, on the other hand, are more general and are better for when you want to loop indefinitely until some condition is met.
2. The while loop

The while loop allows us to repetitively execute the same code until some condition is met. The syntax is similar to the if / else statement, in that we're going to test some conditional statement to see if it is True or False. As long as the conditional is True, we keep looping!

[ Definition ] The while loop
Purpose: execute a block of code until the conditional statement becomes False.
Syntax:
while conditional:
    this indented code will execute until the conditional becomes False

Example:
Can you write Python code for it?

x = 0
while x < 4:
    print "Looped!"
    x = x + 1


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
This is similar to the counter example above, but instead of incrementing by 1 every time, we're summing up various numbers.
This is sometimes called an accumulator. Like the counter, it's useful in many different contexts, so keep it in mind.
Can you write Python code for it?

count = 0
for nt in "CTCCAGGG":
    if nt == "C":
        count = count + 1
print count

oldSeq = "ATG"
newSeq = ""
for nt in oldSeq:
    newSeq = newSeq + nt + "*"
print newSeq


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
This is sort of like an accumulator for strings. We can build up a string in a loop by repeatedly concatenating characters to an existing string.
Important: Don't concatenate onto the original string as you iterate over it. This is bad form and could cause weird results. Just create a new string. (So in this example, we should not modify oldSeq while inside the loop. Modifying newSeq is fine, though.)

Loop wrap-up: So which kind of loop should I use???
You can usually use either type of loop, but one will feel a lot more natural and be easier to code. That's the one you should go with. In general:
Use a for loop when:

You know exactly how many times you need to loop (use range())
You want to process each character of a string, item in a list, or line of a file (as we'll see next!)

Use a while loop when:

You need to loop until some condition is fulfilled, but you don't know when that will happen

4. File reading

Reading from and writing to files is something you'll probably be doing a lot in your work. Luckily, Python makes it super easy using the for loop.
There are 3 basic steps of file reading:

Open the input file
Read in data line by line, do some processing
Close the input file

Let's start with an example.
In the same directory as this notebook, you should have a file called "genes.txt". This file contains a simple list of 7 gene names, as follows:
uc007afd.1
uc007aln.1
uc007afr.1
uc007atn.1
uc007bcd.1
uc007bmh.1
uc007byr.1

Often, we will want to read a file like this into our code so that we can use the data. This can be done with a simple for loop, like so:
Can you write Python code for it?

# Read and print genes.txt
fileName = "genes.txt"

inFile = open(fileName, 'r')
for line in inFile:
    print line
inFile.close()


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
We haven't talked about lists yet, so don't worry about this too much yet. All you need to know is that a list is an iterable, so we can use it in a for loop. The unit of iteration in a list is the elements of the list. So this means that in addition to just using range() to loop a certain number of times, we can also use it to generate numbers to use in our loop:
Can you write Python code for it?


I am working on the file "25_Prelab_Python-III/Lesson3.ipynb".
The first task is:
for loops in action
Below are several examples of code using for loops. For each code block, first try to guess what the output will be, and then run the block to see the answer.
Can you write Python code for it?

for i in range(4):
    print i

for i in range(4):
    print i * 2

count = 0
for i in range(4):
    count = count + 1
print count


I am working on the file "25_Python-III/Lesson3_extra_material.ipynb".
The first task is:
As you'll see in the next lesson and saw in the Unix lessons earlier in the course, we often work with tab-delimited text files, so it's useful to know how to make a tab in your output files.
The += shortcut

In this lesson, we saw a lot of situations like this:
Can you write Python code for it?

count = 0

while count < 10:
    count = count + 1

print count

string = ""

for i in range(5):
    string = string + "a"

print string


I am working on the file "25_Python-III/Lesson3_extra_material.ipynb".
The first task is:
More on reading and writing files

Other ways to read files
You don't have to read a file using a loop. You can also read it manually line by line, or dump all the lines into a list. We won't go over those methods much here, but here is a list of some of the more useful functions that are available:

.read() - reads in the entire file at once as a single string
.readline() - reads one line at a time
.readlines() - reads all lines in file into a list
.close() - close the file

Example:
Can you write Python code for it?

inFile = open("genes.txt", 'r')

header = inFile.readline()       #read first line of file (often we do this to get rid of a header)
line = inFile.readline()         #read second line of file (repeatedly calling .readline() progresses through the file)
restOfLines = inFile.readlines() #read rest into list (each line will be an element)

inFile.close()

print header
print line
print restOfLines


I am working on the file "26--logreg-mle-numopt.ipynb".
The first task is:
Definition 3. The heaviside function maps strictly positive values to the value 1 and non-positive values to 0:
$$
\begin{array}{rcl}
  H(y) & \equiv & \left{\begin{array}{ll}
      1 & \mathrm{if}\ y > 0
      \
      0 & \mathrm{if}\ y \leq 0
    \end{array}\right..
\end{array}
$$
In the cse6040utils module, use heaviside (Y) to apply $H(y)$ elementwise to any Numpy multidimensional array object.
Can you write Python code for it?


I am working on the file "26--logreg-mle-numopt.ipynb".
The first task is:
Definition 4. The logistic function maps a real number continuously into the interval (0, 1):
$$
\begin{array}{rcl}
  G(y) & \equiv & \frac{1}{1 + e^{-y}}.
\end{array}
$$
Recall that $G(y) \rightarrow 0$ as $y \rightarrow -\infty$, and $G(y) \rightarrow 1$ as $y \rightarrow +\infty$.
In the cse6040utils module, use logistic (Y) to apply $G(y)$ elementwise to any Numpy multidimensional array object.
Can you write Python code for it?


I am working on the file "26--logreg-mle-numopt.ipynb".
The first task is:
Demo: Manual classification
Recall that in the last class you generated label predictions using $H(\theta^T x)$, that is, by using the heaviside function with a linear discriminant. You determined the discriminant coefficients, $\theta$, manually.
Can you write Python code for it?

from cse6040utils import check_labels
from cse6040utils import np_col_vec
from cse6040utils import gen_lin_discr_trace

#theta = np_col_vec ([0., -1., 3.])
#theta = np_col_vec ([-0.55, -2., -0.5])
theta = np_col_vec ([-1.35, -6.5, -1.])

# Generate 0/1 labels for your discriminant:
is_correct = check_labels (points, labels,
                           fun=lambda X: heaviside (lin_discr (X, theta)))

print "Number of misclassified points:", (len (points) - sum (is_correct))[0]
print "\n(Run the code cell below to visualize the results.)"

# Visually inspect the above results
traces = make_2d_scatter_traces (points, is_correct)
traces.append (gen_lin_discr_trace (points, theta))

# Plot it!
layout = Layout (xaxis=dict (range=[-1.25, 2.25]),
                 yaxis=dict (range=[-3.25, 2.25]))
fig = Figure (data=traces, layout=layout)
py.iplot (fig)


I am working on the file "26--logreg-mle-numopt.ipynb".
The first task is:
Determining $\theta$ via Maximum Likelihood Estimation
Previously, you determined $\theta$ for our synthetic dataset experimentally. Can you compute a good $\theta$ automatically? One of the standard techniques in statistics is to perform a maximum likelihood estimation (MLE) of a model's parameters, $\theta$.
Indeed, MLE is basis for the "statistical" way to derive the normal equations in the case of linear regression, though that is of course not how we encountered it in this class.
"Likelihood" as an objective function
MLE derives from the following idea. Consider the joint probability of observing all of the labels, given the points and the parameters, $\theta$:
$$
  \mathrm{Pr}[l\,|\,X, \theta].
$$
Suppose these observations are independent and identically distributed (i.i.d.). Then the joint probability can be factored as the product of individual probabilities,
$$
\begin{array}{rcl}
  \mathrm{Pr}[l\,|\,X,\theta] = \mathrm{Pr}[l_0, \ldots, l_{m-1}\,|\,x_0, \ldots, x_{m-1}, \theta]
  & = & \mathrm{Pr}[l_0\,|\,x_0, \theta] \cdots \mathrm{Pr}[l_{m-1}\,|\,x_{m-1}, \theta] \
  & = & \displaystyle \prod_{i=0}^{m-1} \mathrm{Pr}[l_i\,|\,x_i,\theta].
\end{array}
$$
The maximum likelihood principle says that you should try to choose a parameter $\theta$ that maximizes the chances ("likelihood") of seeing these particular observations. Thus, we can simply reinterpret the preceding probability as an objective function to optimize. Mathematically, it is equivalent and convenient to consider the logarithm of the likelihood, or log-likelihood, as the objective function, defining it by,
$$
\begin{array}{rcl}
  \mathcal{L}(\theta; l, X)
    & \equiv &
      \log \left{ \displaystyle \prod_{i=0}^{m-1} \mathrm{Pr}[l_i\,|\,x_i,\theta] \right} \
    & = &
      \displaystyle \sum_{i=0}^{m-1} \log \mathrm{Pr}[l_i\,|\,x_i,\theta].
\end{array}
$$

We are using the symbol $\log$, which could be taken in any convenient base, such as the natural logarithm ($\ln y$) or the information theoretic base-two logarithm ($\log_2...
Can you write Python code for it?

def log_likelihood (theta, l, X):
    # @YOUSE: Complete this function to evaluate the log-likelihood
    pass


I am working on the file "26--logreg-mle-numopt.ipynb".
The first task is:
Exercise. Complete the code below, which implements Newton's method.
Can you write Python code for it?

MAX_STEP = 10

# Get the data coordinate matrix, X, and labels vector, l
X = points
l = labels.astype (dtype=float)

# Store *all* guesses, for subsequent analysis
thetas_newt = np.zeros ((3, MAX_STEP+1))

for t in range (MAX_STEP):
    theta_t = thetas_newt[:, t:t+1]
    # @YOUSE: Fill in this code
    pass
    thetas_newt[:, t+1:t+2] = theta_t + delta_t

theta_newt = thetas_newt[:, MAX_STEP:]
print "Your (hand) solution:", theta.T.flatten ()
print "Computed solution:", theta_newt

num_correct_newt = count_correct_logreg (l, X, thetas_newt)
is_correct_newt = check_correct_logreg (l, X, thetas_newt[:, -1:])

print "\nNumber of misclassified points using MLE:", (len (points) - num_correct_newt[-1])
print "\n(Run the code cell below to visualize the results.)"

print "\n=== Comparisons ==="
print "\n\\theta_0/\\theta_2:", \
      "manual =", theta[0]/theta[2], \
      ", vs. Newton =", theta_newt[0]/theta_newt[2]
print "\n\\theta_1/\\theta_2:", \
      "manual =", theta[1]/theta[2], \
      ", vs. Newton =", theta_newt[1]/theta_newt[2]

# Visually inspect the above results
traces_newt = make_2d_scatter_traces (points, is_correct_newt)
traces_newt.append (gen_lin_discr_trace (points, theta_newt))

# Plot it!
layout_newt = Layout (xaxis=dict (range=[-1.25, 2.25]),
                      yaxis=dict (range=[-3.25, 2.25]))
fig_newt = Figure (data=traces_newt, layout=layout_newt)
py.iplot (fig_newt)

trace_thetas_newt = Scatter (x=thetas_newt[1, :], y=thetas_newt[2, :], mode='markers+lines')
py.iplot ([trace_ll_grid, trace_thetas_newt])

I_gd = range (len (num_correct_gd))
trace_gd_mistakes = Scatter (x=I_gd, y=len (points) - num_correct_gd,
                              mode='markers+lines', name='Gradient descent'
                             )

I_newt = range (len (num_correct_newt))
trace_newt_mistakes = Scatter (x=I_newt, y=len (points) - num_correct_newt,
                               mode='markers+lines', name='Newton'
                              )

layout_mistakes = Layout (xaxis=dict (type='log'),...


I am working on the file "26--logreg-mle-numopt.ipynb".
The first task is:
Exercise. Make a contour plot of the log-likelihood and draw the trajectory taken by the $\theta(t)$ values laid on top of it.
In particular, your function, log_likelihood (theta, l, X), should return the value of the log-likelihood given a column vector of discriminant coefficients, theta; a column vector of labels, l; and a coordinate matrix, X.
The gradient ascent trajectory
Let's take a look at how the gradient ascent algorithm's progress looks. Try changing the $\phi$ parameter and see how it affects the results.
Can you write Python code for it?

n1_ll = 100
x1_ll = np.linspace (-20., 0., n1_ll)
n2_ll = 100
x2_ll = np.linspace (-20., 0., n2_ll)
x1_ll_grid, x2_ll_grid = np.meshgrid (x1_ll, x2_ll)

ll_grid = np.zeros ((n1_ll, n2_ll))
for i1 in range (n1_ll):
    for i2 in range (n2_ll):
        theta_i1_i2 = np.array ([[thetas[0, MAX_STEP]],
                                 [x1_ll_grid[i1][i2]],
                                 [x2_ll_grid[i1][i2]]
                                ])
        ll_grid[i1][i2] = log_likelihood (theta_i1_i2, l, X)

trace_ll_grid = Contour (x=x1_ll, y=x2_ll, z=ll_grid)
trace_thetas = Scatter (x=thetas[1, :], y=thetas[2, :], mode='markers+lines')
py.iplot ([trace_ll_grid, trace_thetas])


I am working on the file "26--logreg-mle-numopt.ipynb".
The first task is:
Exercise. Show that the Hessian of the log-likelihood for logistic regression is,
$$
\begin{array}{rcl}
  H_{\theta} \left( \mathcal{L}(\theta; l, X) \right)
    & = & \left( X \odot G(X \theta) \right)^T \left( X \odot G(-X \theta) \right).
\end{array}
$$
Exercise. Implement a function to compute the Hessian of the log-likelihood.
Can you write Python code for it?

def hessian_log_likelihood (theta, l, X):
    """Returns the Hessian of the log-likelihood."""
    # @YOUSE: Implement the Hessian
    pass


I am working on the file "26--logreg-mle-numopt.ipynb".
The first task is:
Heaviside vs. logistic functions
Where we ended in the last class was comparing the use of the heaviside function against use of the logistic function, both assuming a linear discriminant scoring function.
Here is that comparison visually.
Can you write Python code for it?

# Use Numpy's handy meshgrid() to create a regularly-spaced grid of values.
# http://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html

x1 = np.linspace (-2., +2., 100)
x2 = np.linspace (-2., +2., 100)
x1_grid, x2_grid = np.meshgrid (x1, x2)
h_grid = heaviside (theta[0] + theta[1]*x1_grid + theta[2]*x2_grid)

trace_grid = Contour (x=x1, y=x2, z=h_grid)
py.iplot ([trace_grid])

x_logit_1d = np.linspace (-6.0, +6.0, 101)
y_logit_1d = logistic (x_logit_1d)
trace_logit_1d = Scatter (x=x_logit_1d, y=y_logit_1d)
py.iplot ([trace_logit_1d])

g_grid = logistic (theta[0] + theta[1]*x1_grid + theta[2]*x2_grid)

trace_logit_grid = Contour (x=x1, y=x2, z=g_grid)
py.iplot ([trace_logit_grid])


I am working on the file "26--logreg-mle-numopt.ipynb".
The first task is:
Notation
Recall the basic notation used in Lab 25.
Definition 1. There are $m$ labeled data points in a $d$-dimensional vector space. The $i$-th observation is represented by its label, $l_i$, and its augmented coordinates ("point"), $x_i \equiv (1.0, x_{i,1}, x_{i,2}, \ldots, x_{i,d})^T$; that is, the point $x_i$ is a column vector of length $d+1$ with a dummy value of 1.0 in the very first (0-th) entry. The matrix of data coordinates, $X$, stacks these points rowwise into a $m \times (d+1)$ matrix, i.e.,
$$
\begin{array}{rcl}
  X \equiv
    \left(\begin{array}{c}
      x_0^T \
      x_1^T \
      \vdots \
      x_{m-1}^T
    \end{array}\right)
  & = &
    \left(\begin{array}{ccccc}
      1 & x_{0,1} & x_{0,2} & \cdots & x_{0,d} \
      1 & x_{1,1} & x_{1,2} & \cdots & x_{1,d} \
        &         &         & \vdots & \
      1 & x_{m-1,1} & x_{m-1,2} & \cdots & x_{m-1,d} \
    \end{array}\right).
\end{array}
$$
We will take the labels to be a binary column vector, $l \equiv \left(l_0, l_1, \ldots, l_{m-1}\right)^T$.
Definition 2. A linear discriminant is a function that assigns a "score" to a given point $x$ by linearly transforming its coordinates. It may be specified by a $d+1$-dimensional column vector of coefficients, $\theta \equiv (\theta_0, \theta_1, \ldots, \theta_d)^T$; the score is then taken to be $\theta^T x$.
In the cse6040utils module, use lin_discr (X, theta) to evaluate a linear discriminant given by theta on an augmented coordinate matrix X.
Can you write Python code for it?


I am working on the file "26--logreg-mle-numopt.ipynb".
The first task is:
Numerical optimization via Newton's method
The fudge factor in gradient ascent might be troubling. Can you choose the step size or direction in a better or more principled way?
One idea is Newton's method, summarized below.
The basic idea, in 1-D
Suppose you start at a point $x$ and, assuming you are not yet at the optimum, you wish to take a step $x + p$ so that $f(x + p)$ is the maximum.
However, instead of trying to maximize $f(x + p)$ directly, let's replace $f(x + p)$ with some approximation $q(p)$, and then choose a $p$ to maximize $q(p)$. A simple choice for $q(p)$ is a quadratic function in $p$. This choice is motivated by two factors: (a) since it's quadratic, it should have some sort of extreme point (and hopefully an actual maximum), and (b) it is a higher-order approximation than a linear one, and so hopefully more accurate than a linear one as well.
$$
\begin{array}{rcl}
  f(x + p)
    & \approx & f(x) + p \dfrac{df}{dx} + \frac{1}{2} p^2 \dfrac{d^2 f}{dx^2}
    & \equiv  & q(p).
\end{array}
$$
To maximize $q(p)$, take its derivative and then solve for the $p_$ such that $q(p_) = 0$:
$$
\begin{array}{rcl}
  \left.\dfrac{dq}{dp}\right|{p=p}
    & = & \dfrac{df}{dx} + p_ \dfrac{d^2 f}{dx^2} = 0 \
  \implies p_*
    & = & -\left(\dfrac{d^2 f}{dx^2}\right)^{-1} \dfrac{df}{dx}.
\end{array}
$$
Generalizing to higher dimensions
To see how this procedure works in higher dimensions, you will need not only the gradient of $f(x)$, but also its Hessian, which is the moral equivalent of a second derivative.
Definition: the Hessian. Let $f(v)$ be a function that takes a vector $v$ of length $n$ as input and returns a scalar. The Hessian of $f(v)$ is an $n \times n$ matrix, $H_v(f)$, whose entries are all $n^2$ possible second derivatives with respect to the components of $v$. That is, the $(i, j)$ element of $H_v(f)$ is given by $h_{ij}$ such that
$$
\begin{array}{rcl}
  h_{ij}
    & \equiv & \dfrac{\partial^2}{\partial v_i \partial v_j} f(v).
\end{array}
$$
Armed with a Hessian, the Newton step is defined as...
Can you write Python code for it?

A = np.array ([[1, 2, 3],
               [4, 5, 6]])
B = np.array ([[-1, 2, -3],
               [4, -5, 6]])

print np.multiply (A, B) # elementwise product
print np.multiply (A, B[:, 0:1]) # "auto-extend" version


I am working on the file "26. 앙상블 방법론/01. 모형 결합(배깅, 랜덤포레스트).ipynb".
The first task is:
다수결 모형이 개별 모형보다 더 나은 성능을 보이는 이유는 다음 실험에서도 확인 할 수 있다.
만약 어떤 개별 모형이 오차를 출력할 확률이 $p$인 경우에 이러한 모형을 $N$ 개 모아서 다수결 모형을 만들면 오차를 출력할 확률이 다음과 같아진다.
$$ \sum_{k>\frac{N}{2}}^N \binom N k p^k (1-p)^{N-k} $$
Can you write Python code for it?

def total_error(p, N):
    te = 0.0
    for k in range(int(np.ceil(N/2)), N + 1):
        te += sp.misc.comb(N, k) * p**k * (1-p)**(N-k)
    return te

x = np.linspace(0, 1, 100)
plt.plot(x, x, 'g:', lw=3, label="individual model")
plt.plot(x, total_error(x, 10), 'b-', label="voting model (N=10)")
plt.plot(x, total_error(x, 100), 'r-', label="voting model (N=100)")
plt.xlabel("performance of individual model")
plt.ylabel("performance of voting model")
plt.legend(loc=0)
plt.show()


I am working on the file "26. 앙상블 방법론/01. 모형 결합(배깅, 랜덤포레스트).ipynb".
The first task is:
랜덤 포레스트
랜덤 포레스트(Random Forest)는 의사 결정 나무(Decision Tree)를 개별 모형으로 사용하는 모형 결합 방법을 말한다.
배깅과 마찬가지로 데이터 샘플의 일부만 선택하여 사용한다. 하지만 노드 분리시 모든 독립 변수들을 비교하여 최선의 독립 변수를 선택하는 것이 아니라 독립 변수 차원을 랜덤하게 감소시킨 다음 그 중에서 독립 변수를 선택한다. 이렇게 하면 개별 모형들 사이의 상관관계가 줄어들기 때문에 모형 성능의 변동이 감소하는 효과가 있다.
이러한 방법을 극단적으로 적용한 것이 Extremely Randomized Trees 모형으로 이 경우에는 각 노드에서 랜덤하게 독립 변수를 선택한다.
랜덤 포레스트와 Extremely Randomized Trees 모형은 각각 RandomForestClassifier클래스와 ExtraTreesClassifier 클래스로 구현되어 있다.
Can you write Python code for it?

from sklearn import clone
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.tree import DecisionTreeClassifier

iris = load_iris()

n_classes = 3
n_estimators = 30
plot_colors = "ryb"
cmap = plt.cm.RdYlBu
plot_step = 0.02
RANDOM_SEED = 13

models = [DecisionTreeClassifier(max_depth=4),
          RandomForestClassifier(max_depth=4, n_estimators=n_estimators),
          ExtraTreesClassifier(max_depth=4, n_estimators=n_estimators)]

plot_idx = 1
plt.figure(figsize=(12, 12))
for pair in ([0, 1], [0, 2], [2, 3]):
    for model in models:
        X = iris.data[:, pair]
        y = iris.target

        idx = np.arange(X.shape[0])
        np.random.seed(RANDOM_SEED)
        np.random.shuffle(idx)
        X = X[idx]
        y = y[idx]

        mean = X.mean(axis=0)
        std = X.std(axis=0)
        X = (X - mean) / std

        clf = clone(model)
        clf = model.fit(X, y)

        plt.subplot(3, 3, plot_idx)
        model_title = str(type(model)).split(".")[-1][:-2][:-len("Classifier")]
        if plot_idx <= len(models):
            plt.title(model_title)

        x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
        y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
        xx, yy = np.meshgrid(np.arange(x_min, x_max, plot_step),
                             np.arange(y_min, y_max, plot_step))

        if isinstance(model, DecisionTreeClassifier):
            Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
            Z = Z.reshape(xx.shape)
            plt.contourf(xx, yy, Z, cmap=cmap)
        else:
            estimator_alpha = 1.0 / len(model.estimators_)
            for tree in model.estimators_:
                Z = tree.predict(np.c_[xx.ravel(), yy.ravel()])
                Z = Z.reshape(xx.shape)
                plt.contourf(xx, yy, Z, alpha=estimator_alpha, cmap=cmap)

        for i, c in zip(range(n_classes), plot_colors):
            idx = np.where(y == i)
            plt.scatter(X[idx, 0], X[idx, 1], c=c,...


I am working on the file "26. 앙상블 방법론/01. 모형 결합(배깅, 랜덤포레스트).ipynb".
The first task is:
랜덤 포레스트의 장점 중 하나는 각 독립 변수의 중요도(feature importance)를 계산할 수 있다는 점이다.
포레스트 안에서 사용된 모든 노드에 대해 어떤 독립 변수를 사용하였고 그 노드에서 얻은 information gain을 구할 수 있으므로 각각의 독립 변수들이 얻어낸 information gain의 평균을 비교하면 어떤 독립 변수가 중요한지를 비교할 수 있다.
Can you write Python code for it?

from sklearn.datasets import make_classification
from sklearn.ensemble import ExtraTreesClassifier

X, y = make_classification(n_samples=1000, n_features=10, n_informative=3, n_redundant=0, n_repeated=0,
                           n_classes=2, random_state=0, shuffle=False)

forest = ExtraTreesClassifier(n_estimators=250, random_state=0)
forest.fit(X, y)

importances = forest.feature_importances_

std = np.std([tree.feature_importances_ for tree in forest.estimators_], axis=0)
indices = np.argsort(importances)[::-1]

print("Feature ranking:")
for f in range(X.shape[1]):
    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))

plt.title("Feature importances")
plt.bar(range(X.shape[1]), importances[indices], color="r", yerr=std[indices], align="center")
plt.xticks(range(X.shape[1]), indices)
plt.xlim([-1, X.shape[1]])
plt.show()

from sklearn.datasets import fetch_olivetti_faces
from sklearn.ensemble import ExtraTreesClassifier

data = fetch_olivetti_faces()
X = data.images.reshape((len(data.images), -1))
y = data.target

mask = y < 5  # Limit to 5 classes
X = X[mask]
y = y[mask]

forest = ExtraTreesClassifier(n_estimators=1000, max_features=128, random_state=0)
forest.fit(X, y)

importances = forest.feature_importances_
importances = importances.reshape(data.images[0].shape)

plt.figure(figsize=(8, 8))
plt.imshow(importances, cmap=plt.cm.bone_r)
plt.grid(False)
plt.title("Pixel importances with forests of trees")
plt.show()


I am working on the file "26. 앙상블 방법론/01. 모형 결합(배깅, 랜덤포레스트).ipynb".
The first task is:
모형 결합
모형 결합(model combining) 방법은 앙상블 방법론(ensemble methods)이라고도 한다. 이는 특정한 하나의 예측 방법이 아니라 복수의 예측 모형을 결합하여 더 나은 성능의 예측을 하려는 시도이다.
모형 결합 방법을 사용하면 일반적으로 계산량은 증가하지만 다음과 같은 효과가 있다.

단일 모형을 사용할 때 보다 성능 분산이 감소하고
과최적화를 방지한다.

모형 결합 방법은 크게 나누어 평균(averaging, aggregation) 방법론과 부스팅(boosting) 방법론으로 나눌 수 있다.

평균 방법론은 사용할 모형의 집합이 이미 결정되어 있지만
부스팅 방법론은 사용할 모형을 점진적으로 늘려간다.

각 방법론의 대표적인 방법들은 아래와 같다.

평균 방법론
다수결 (Majority Voting)
배깅 (Bagging)

랜덤 포레스트 (Random Forests)

부스팅 방법론

에이다부스트 (AdaBoost)
경사 부스트 (Gradient Boost)

다수결 방법
다수결 방법은 가장 단순한 모형 결합 방법으로 전혀 다른 모형도 결합할 수 있다. 다수결 방법은 Hard Voting 과 Soft Voting 두 가지로 나뉘어진다.

hard voting: 단순 투표. 개별 모형의 결과 기준
soft voting: 가중치 투표. 개별 모형의 조건부 확률의 합 기준

Scikit-Learn 의 ensemble 서브패키지는 다수결 방법을 위한 VotingClassifier 클래스를 제공한다.
sklearn.ensemble.VotingClassifier(estimators, voting='hard', weights=None)

입력 인수:
estimators :
개별 모형 목록, 리스트나 named parameter 형식으로 입력

voting : 문자열 {‘hard’, ‘soft’} (디폴트 ’hard’)
hard voting 과 soft voting 선택

weights : 리스트
사용자 가중치
Can you write Python code for it?

X = np.array([[-1.0, -1.0], [-1.2, -1.4], [1, -0.5], [-3.4, -2.2], [1.1, 1.2], [-2.1, -0.2]])
y = np.array([1, 1, 1, 2, 2, 2])
x_new = [0, 0]
plt.scatter(X[y==1,0], X[y==1,1], s=100, c='r')
plt.scatter(X[y==2,0], X[y==2,1], s=100, c='b')
plt.scatter(x_new[0], x_new[1], s=100, c='g')

from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.ensemble import VotingClassifier

clf1 = LogisticRegression(random_state=1)
clf2 = SVC(random_state=1, probability=True)
clf3 = GaussianNB()
eclf = VotingClassifier(estimators=[('lr', clf1), ('ksvc', clf2), ('gnb', clf3)], voting='soft', weights=[2, 1, 1])

probas = [c.fit(X, y).predict_proba([x_new]) for c in (clf1, clf2, clf3, eclf)]
class1_1 = [pr[0, 0] for pr in probas]
class2_1 = [pr[0, 1] for pr in probas]

ind = np.arange(4)
width = 0.35  # bar width
p1 = plt.bar(ind, np.hstack(([class1_1[:-1], [0]])), width, align="center", color='green')
p2 = plt.bar(ind + width, np.hstack(([class2_1[:-1], [0]])), width, align="center", color='lightgreen')
p3 = plt.bar(ind, [0, 0, 0, class1_1[-1]], width, align="center", color='blue')
p4 = plt.bar(ind + width, [0, 0, 0, class2_1[-1]], width, align="center", color='steelblue')

plt.xticks(ind + 0.5 * width, ['LogisticRegression\nweight 2',
                               'Kernel SVC\nweight 1',
                               'GaussianNB\nweight 1',
                               'VotingClassifier'])
plt.ylim([0, 1.1])
plt.title('Class probabilities for sample 1 by different classifiers')
plt.legend([p1[0], p2[0]], ['class 1', 'class 2'], loc='upper left')
plt.show()

from itertools import product

x_min, x_max = -4, 2
y_min, y_max = -3, 2
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.025), np.arange(y_min, y_max, 0.025))
f, axarr = plt.subplots(2, 2)
for idx, clf, tt in zip(product([0, 1], [0, 1]),
                        [clf1, clf2, clf3, eclf],
                        ['LogisticRegression', 'Kernel SVC', 'GaussianNB', 'VotingClassifier']):
    Z =...


I am working on the file "26. 앙상블 방법론/01. 모형 결합(배깅, 랜덤포레스트).ipynb".
The first task is:
배깅
배깅(bagging)은 동일한 모형과 모형 모수를 사용하는 대신 부트스트래핑(bootstrapping)과 유사하게 트레이닝 데이터를 랜덤하게 선택해서 다수결 모형을 적용한다.
트레이닝 데이터를 선택하는 방법에 따라 다음과 같이 부르기도 한다.

같은 데이터 샘플을 중복사용(replacement)하지 않으면: Pasting
같은 데이터 샘플을 중복사용(replacement)하면 Bagging
데이터가 아니라 다차원 독립 변수 중 일부 차원을 선택하는 경우에는: Random Subspaces
데이터 샘플과 독립 변수 차원 모두 일부만 랜덤하게 사용하면: Random Patches

성능 평가시에는 트레이닝용으로 선택한 데이터가 아닌 다른 데이터를 사용할 수도 있다. 이런 데이터를 OOB(out-of-bag) 데이터라고 한다.
Scikit-Learn 의  ensemble 서브패키지는 배깅 모형 결합을 위한 BaggingClassifier 클래스를 제공한다. 사용법은 다음과 같다.
sklearn.ensemble.BaggingClassifier(base_estimator=None, n_estimators=10, max_samples=1.0, max_features=1.0, bootstrap=True, bootstrap_features=False, oob_score=False, warm_start=False, n_jobs=1, random_state=None, verbose=0)

인수:
base_estimator:
기본 모형
n_estimators: 정수. 디폴트 10
모형 갯수
max_samples: 정수 혹은 실수. 디폴트 1.0
데이터 샘플 중 선택할 샘플의 수 혹은 비율
max_features: 정수 혹은 실수. 디폴트 1.0
다차원 독립 변수 중 선택할 차원의 수 혹은 비율
bootstrap: 불리언, 디폴트 True
데이터 중복 사용 여부
bootstrap_features: 불리언, 디폴트 False
차원 중복 사용 여부
oob_score: 불리언 디폴트 False
성능 평가시 OOB(out-of-bag) 샘플 사용 여부
Can you write Python code for it?

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import BaggingClassifier

iris = load_iris()
X, y = iris.data[:, [0, 2]], iris.target

model1 = DecisionTreeClassifier().fit(X, y)
model2 = BaggingClassifier(DecisionTreeClassifier(), bootstrap_features=True, random_state=0).fit(X, y)

x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1), np.arange(y_min, y_max, 0.1))

plt.figure(figsize=(8,12))

plt.subplot(211)
Z1 = model1.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
plt.contourf(xx, yy, Z1, alpha=0.6, cmap=mpl.cm.jet)
plt.scatter(X[:, 0], X[:, 1], c=y, alpha=1, s=50, cmap=mpl.cm.jet)

plt.subplot(212)
Z2 = model2.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
plt.contourf(xx, yy, Z2, alpha=0.6, cmap=mpl.cm.jet)
plt.scatter(X[:, 0], X[:, 1], c=y, alpha=1, s=50, cmap=mpl.cm.jet)

plt.tight_layout()
plt.show()


I am working on the file "26. 앙상블 방법론/01. 모형 결합(배깅, 랜덤포레스트).ipynb".
The first task is:
에이다 부스트
에이다 부스트와 같은 부스트(boost) 방법은 미리 정해진 모형 집합을 사용하는 것이 아니라 단계적으로 모형 집합에 포함할 개별 모형을 선택한다. 부스트 방법에서 성능이 떨어지는 개별 모형을 weak classifier라고 한다.
또한 다수결 방법을 사용하지 않고 각 weak classifier $k$개에 대한 가중치를 주고 선형 결합하여 최종 모형인 boosted classifier $C$를 생성한다.
$$ C_{(m-1)}(x_i) = \alpha_1k_1(x_i) + \cdots + \alpha_{m-1}k_{m-1}(x_i) $$
$$ C_{m}(x_i) = C_{(m-1)}(x_i) + \alpha_m k_m(x_i) $$

$k_m$ 선택 방법

가중치 오차가 가장 적은 $k_m$
$$ E = \sum_{y_i \neq k_m(x_i)} w_i^{(m)} = \sum_{y_i \neq k_m(x_i)} e^{-y_i C_{m-1}(x_i)}$$

$\alpha_m$ 결정 방법

$$ \epsilon_m = \dfrac{\sum_{y_i \neq k_m(x_i)} w_i^{(m)} }{ \sum_{i=1}^N w_i^{(m)}} $$
$$ \alpha_m = \frac{1}{2}\ln\left( \frac{1 - \epsilon_m}{\epsilon_m}\right) $$

속도 조절을 위해 learning rate $\nu$를 추가

$$ C_{m}(x_i) = C_{(m-1)}(x_i) + \nu\alpha_m k_m(x_i) $$
에이다 부스트 클래스는 AdaBoostClassifier 이다.
Can you write Python code for it?

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_gaussian_quantiles

# Construct dataset
X1, y1 = make_gaussian_quantiles(cov=2.,
                                 n_samples=200, n_features=2,
                                 n_classes=2, random_state=1)
X2, y2 = make_gaussian_quantiles(mean=(3, 3), cov=1.5,
                                 n_samples=300, n_features=2,
                                 n_classes=2, random_state=1)
X = np.concatenate((X1, X2))
y = np.concatenate((y1, - y2 + 1))

# Create and fit an AdaBoosted decision tree
bdt = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1),
                         algorithm="SAMME",
                         n_estimators=200)

bdt.fit(X, y)

plot_colors = "br"
plot_step = 0.02
class_names = "AB"

plt.figure(figsize=(12,6))

plt.subplot(121)
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, plot_step),
                     np.arange(y_min, y_max, plot_step))

Z = bdt.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
cs = plt.contourf(xx, yy, Z, cmap=plt.cm.Paired)
plt.axis("tight")

for i, n, c in zip(range(2), class_names, plot_colors):
    idx = np.where(y == i)
    plt.scatter(X[idx, 0], X[idx, 1],
                c=c, cmap=plt.cm.Paired,
                label="Class %s" % n)
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.legend(loc='upper right')
plt.xlabel('x')
plt.ylabel('y')
plt.title('Decision Boundary')

twoclass_output = bdt.decision_function(X)
plot_range = (twoclass_output.min(), twoclass_output.max())
plt.subplot(122)
for i, n, c in zip(range(2), class_names, plot_colors):
    plt.hist(twoclass_output[y == i],
             bins=10,
             range=plot_range,
             facecolor=c,
             label='Class %s' % n,
             alpha=.5)
x1, x2, y1, y2 = plt.axis()
plt.axis((x1, x2, y1, y2 * 1.2))
plt.legend(loc='upper...


I am working on the file "26. 앙상블 방법론/01. 모형 결합(배깅, 랜덤포레스트).ipynb".
The first task is:
예: 이미지 완성
Can you write Python code for it?

from sklearn.datasets import fetch_olivetti_faces
from sklearn.utils.validation import check_random_state

from sklearn.ensemble import ExtraTreesRegressor
from sklearn.linear_model import LinearRegression

data = fetch_olivetti_faces()
targets = data.target

data = data.images.reshape((len(data.images), -1))
train = data[targets < 30]
test = data[targets >= 30]

n_faces = 5
rng = check_random_state(4)
face_ids = rng.randint(test.shape[0], size=(n_faces, ))
test = test[face_ids, :]

n_pixels = data.shape[1]
X_train = train[:, :int(np.ceil(0.5 * n_pixels))]  # Upper half of the faces
y_train = train[:, int(np.floor(0.5 * n_pixels)):]  # Lower half of the faces
X_test = test[:, :int(np.ceil(0.5 * n_pixels))]
y_test = test[:, int(np.floor(0.5 * n_pixels)):]

ESTIMATORS = {
    "Linear regression": LinearRegression(),
    "Extra trees": ExtraTreesRegressor(n_estimators=10, max_features=32, random_state=0),
}

y_test_predict = dict()
for name, estimator in ESTIMATORS.items():
    estimator.fit(X_train, y_train)
    y_test_predict[name] = estimator.predict(X_test)

image_shape = (64, 64)
n_cols = 1 + len(ESTIMATORS)
plt.figure(figsize=(3*n_cols, 3*n_faces))
plt.suptitle("Face completion with multi-output estimators", size=16)

for i in range(n_faces):
    true_face = np.hstack((X_test[i], y_test[i]))
    if i:
        sub = plt.subplot(n_faces, n_cols, i * n_cols + 1)
    else:
        sub = plt.subplot(n_faces, n_cols, i * n_cols + 1, title="true faces")
    sub.axis("off")
    sub.imshow(true_face.reshape(image_shape), cmap=plt.cm.gray, interpolation="nearest")

    for j, est in enumerate(ESTIMATORS):
        completed_face = np.hstack((X_test[i], y_test_predict[est][i]))
        if i:
            sub = plt.subplot(n_faces, n_cols, i * n_cols + 2 + j)
        else:
            sub = plt.subplot(n_faces, n_cols, i * n_cols + 2 + j, title=est)

        sub.axis("off")
        sub.imshow(completed_face.reshape(image_shape), cmap=plt.cm.gray, interpolation="nearest");


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
<br>
Accessing elements in a list
As we saw above, an element of a list can be accessed using its index. Do not to try to access an index that's not yet in the list -- this will give an error.
someData = myList[index]

You can also index backwards using negative indices:
lastElement = myList[-1]

Examples:
Can you write Python code for it?

contestRanking = ["Sally", "Billy", "Tommy", "Wilfred"]

print "First place goes to", contestRanking[0], "!"
print "Congrats also to our second and third place winners,", contestRanking[1], "and", contestRanking[2]
print "And in last place...", contestRanking[-1]

contestRanking = ["Sally", "Billy", "Tommy", "Wilfred"]

print "6th place goes to", contestRanking[5]


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
<br>
Adding to a dictionary
Add a new key-value pair to an existing dictionary:
myDict[newKey] = newVal

Note that if the specified key is already in the dictionary, the associated value will be overwritten by the new value we assign here!
Can you write Python code for it?

myDict = {"Joe": 25, "Sally": 35}
myDict["Bobby"] = 65
myDict["Joe"] = 104
print myDict


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
<br>
Adding to a list
After creating a list, you can add additional elements to the end using .append(). This is an in-place function, meaning that it directly modifies the list.
myList.append(element)

Insert an element at the specified index. Elements that come after that index will shift up one index. (in-place function)
myList.insert(index, value)

Example:
Can you write Python code for it?

shoppingList = ["pizza", "ice cream", "cat food"]
shoppingList.append("english muffins")

print shoppingList

shoppingList.insert(2, "lembas")
print shoppingList


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
<br>
Iterating through a list
This should be pretty familiar by now. Since a list is an iterable, we can loop through it using a for loop:
for element in myList:
    ... do something

Example:
Can you write Python code for it?

currentCats = ["Mittens", "Tatertot", "Meatball", "Star Destroyer"]

for cat in currentCats:
    print cat


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:

Other list operations
listLen = len(myList)   # Get the length of a list

myList.sort()           # Sort (in-place function)

myList.reverse()        # Reverse (in-place function)

Related data structure: Tuples
There is another data structure in Python that is similar to lists, but not exactly the same, called tuples. We won't focus too much on these for this class, but essentially a tuple is, like a list, a sequence of items, but unlike a list, these items are immutable, meaning that once you have defined your tuple, you cannot change the items in it. These structures are useful in cases when you know exactly how many items you want to use, and if you aren't going to be changing these items.
Tuples are defined similarly to lists, except that they use rounded parentheses instead of square brackets, and can also be defined without any parentheses. For example:
myTuple = ()                   # create an empty tuple
myTuple = "a", "b", "c", "d"   # create a tuple of strings
myTuple = ("a", "b", "c", "d") # this makes the same exact tuple
myTuple = ("a", "b", 3, 4)     # tuples can also include different data types
myTuple = ("a")                # to make a tuple with a single value, you need a comma

For the purposes of this class, we won't go through all the different methods for tuples, except that they can be accessed similarly to lists, i.e.
myTuple[index]

For more info on tuples, see http://www.tutorialspoint.com/python/python_tuples.htm. Just note that you cannot change individual elements in a tuple!
2. Data structures II: Dictionaries

The next data structure we'll talk about is the dictionary. There are two key differences between dictionaries and lists:

In a dictionary, you retrieve elements using a key rather than an index
Dictionaries are unordered

Difference 1: Dictionaries are indexed by keys
With a dictionary (sometimes called a hash table in other programming languages), you access elements by a name ("key") that you pick:

Keys can be...
Can you write Python code for it?

myDict = {'age':3, 'animal':'cat', 'num':56.9, 203:4, 'count':10, 'flag':True}   # creates the dictionary shown above

print myDict['animal']
print myDict[203]


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
<br>
Removing from a list
After creating a list, you can remove elements from it using .pop(). Elements that come after the removed element will be moved up one index so that there are no empty spaces in the list. .pop() also returns the element that was "popped". (in-place function)
myList.pop(index)  # removes (and returns) the element at the specified index
myList.pop()       # removes (and returns) the last element

You can also remove the first occurrence of a specified element using .remove(). Elements that come after will shift down one index. (in-place function)
myList.remove(element)

Examples:
Can you write Python code for it?

toDoList = ["Water plants", "feed cat", "do dishes", "make python lesson"]
doNext = toDoList.pop()

print doNext
print toDoList

pizzaToppings = ["peppers", "sausage", "bananas", "pepperoni"]
pizzaToppings.remove("bananas")

print pizzaToppings


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
<br>
Special dictionary functions
Often, we'll want to quickly get a list of all the keys or values. Python provides the following functions to do this.
Get a list of the keys only:
keyList = myDict.keys()

Get a list of the values only:
valueList = myDict.values()

When are dictionaries useful?
One of the most natural applications of the dictionary is to create a "lookup table". There are many examples of lookup tables in our everyday lives -- things like a phonebook, the index at the back of a textbook, or (surprise surprise) a regular old dictionary. What these examples have in common is that they allow you to take one piece of information that you already know (a friend's name, a topic, a word) and use it to quickly look up some information that you don't know, but need (a phone number, a page number, a definition).
Here's a simple toy example of creating a phonebook using a dictionary:
Can you write Python code for it?

phonebook = {}
phonebook["Joe Shmo"] = "958-273-7324"
phonebook["Sally Shmo"] = "958-273-9594"
phonebook["George Smith"] = "253-586-9933"

name = raw_input("Lookup number for: ")
print phonebook[name]


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
<img align="left" src="list_diagram2.PNG" />
This ability to access a potentially huge amount of data using just one variable name is part of what makes data structures so useful. Imagine if you had 20,000 gene IDs you wanted to use in your code -- it wouldn't be feasible to create a separate variable name for each one. Instead, you can just dump all the gene IDs into a single list, and access them by index. We'll see how to actually do this sort of thing later in this lesson!

Important side note: "in-place" functions
Before we talk more about lists, we need to briefly introduce the idea of "in-place" functions.
The functions we've seen so far do not modify variables directly -- they simply "return" a value. For example, line.rstrip('\n') does nothing to the original string line, it just returns a modified version. To actually change line, you need to say line = line.rstrip('\n'), which overwrites line with the new value. Here's a similar example in code:
Can you write Python code for it?

line = "ATGCGTA***********"

line.rstrip("*")
print line

line = line.rstrip("*")
print line


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
Difference 2: Dictionaries are unordered
Lists are all about keeping elements in some order. Though you may change the ordering from time to time, it's still in some predictable order at all times.
You should think of dictionaries more like magic grab bags. You mark each piece of data with a key, then throw it in the bag. When you want that data back, you just tell the bag the key and it spits out the data assigned to that key. There's no intrinsic order to the things in the bag, they're all just kind of jumbled around in there. Because of this, dictionaries aren't great for situations where we need to keep data in a specific order -- but they're very convenient in other situations, as we'll see in a minute.

Technical side note:
Ok, so in reality, there is an order to your dictionary. But it is an order that Python picks that obeys complex rules and is essentially unpredictable by us. So as far as we're concerned, it may as well be unordered.

For example, here's what happens when we print a dictionary -- you can see that the elements are not maintained in the same order they were added:
Can you write Python code for it?

myDict = {'age':3, 'animal':'cat', 'num':56.9, 203:4, 'count':10, 'flag':True}
print myDict


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
Example: parsing a file with multiple columns
In the same folder as this notebook, you should have a file called init_sites.txt. This file contains some real data on translation initiation sites from a ribosome profiling study in mouse (Ingolia et al., Cell 2011). Here's what the first few lines look like:
knownGene   GeneName    InitCodon[nt]   DisttoCDS[codons]   FramevsCDS  InitContext[-3to+4] CDSLength[codons]   HarrPeakStart   HarrPeakWidth   #HarrReads  PeakScore   Codon   Product
uc007zzs.1  Cbr3    36  -23 -1  GCCACGG 22  35  3   379 4.75    nearcog uorf
uc009akk.1  Rac1    196 0   0   CAGATGC 192 195 3   3371    4.70    aug canonical
uc009eyb.1  Saps1   204 -91 1   GCCACGG 23  203 3   560 4.68    nearcog uorf
uc008wzq.1  Ppp1cb  96  0   0   AAGATGG 327 94  4   3218    4.56    aug canonical
uc007hnl.1  Pa2g4   38  -23 0   AGCCTGT 14  37  4   6236    4.54    nearcog uorf
uc007hnl.1  Pa2g4   40  -22 -1  CCTGTGG 17  37  4   6236    4.54    nearcog uorf
...
...
(Note: the header looks like it's on two lines here due to text wrapping, but it's actually just one line!)
Let's say the only info I'm want from this file is the initiation context of each translation initiation site. This is contained in column 6 of each row (under the header label "InitContext[-3to+4]"). How can I extract this information?
Think for a moment how you might do this, then take a look at the solution:
Can you write Python code for it?

input = open("init_sites.txt", 'r')
input.readline()  #skip header line

for line in input:
    line = line.rstrip('\r\n')
    data = line.split()  #splits line on whitespace (includes tabs), returns a list
    print data[5]  #remember, list indexing starts at 0, so the 6th column = index 5 in the list!

input.close()


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
If instead we were interested in some other column of the file, we just need to switch data[5] to whichever index holds our information of interest, e.g. data[0] to get the "knownGene" column or data[2] to get the "InitCodon[nt]". Remember, though, that the lines of a file are always read in as strings, so you will need to convert numbers using int() or float() as appropriate. So for the InitCodon[nt], we will probably want to say int(data[2]) before doing any computations on those numbers.
4. Test your understanding: practice set 4

For the following blocks of code, first try to guess what the output will be, and then run the code yourself. These examples may introduce some ideas and common pitfalls that were not explicitly covered in the text above, so be sure to complete this section.
Can you write Python code for it?

# RUN THIS BLOCK FIRST TO SET UP VARIABLES! (and re-run it if the lists/dictionary are changed in subsequent code blocks)
fruits = {"apple":"red", "banana":"yellow", "grape":"purple"}
names = ["Wilfred", "Manfred", "Wadsworth", "Jeeves"]
ages = [65, 34, 96, 47]
str1 = "Good morning, Mr. Mitsworth."

print len(ages)

print len(ages) == len(names)

print names[-1]

for age in ages:
    print age

for i in range(len(names)):
    print names[i],"is",ages[i]

if "Willard" not in names:
    names.append("Willard")
print names

ages.sort()
print ages

ages = ages.sort()
print ages

parts = str1.split()
print parts
print str1

parts = str1.split(",")
print parts

oldList = [2, 2, 6, 1, 2, 6]
newList = []

for item in oldList:
    if item not in newList:
        newList.append(item)
print newList

print fruits["banana"]

query = "apple"
print fruits[query]

print fruits[0]

print fruits.keys()

print fruits.values()

for key in fruits:
    print fruits[key]

del fruits["banana"]
print fruits

print fruits["pear"]

fruits["apple"] = fruits["apple"] + " or green"
print fruits["apple"]

fruits["pear"] = "green"
print fruits["pear"]


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
Now we'll go over more formally what a list is and how it can be used.

Side note for people who have used other programming languages:
Lists are similar to what other programming languages call arrays. There are actually some subtle (but important) differences between lists and arrays (Python lists are closer to what people usually call a "linked list"), but for most purposes they perform the same role. The most obvious difference you might notice is that you don't need to specify ahead of time how large your list will be. This is because the size of the list grows dynamically as you add things to it (it also shrinks automatically as you take things out).

How lists work
A Python list looks something like this when we print it out:
[3, "cat", 56.9, 4, 10, True]

However, it may be more helpful to think of a list as looking something like this:
<img align="left" src="list_diagram.PNG" />
Here, each thing in the list (element) is stored in its own cell, and each cell is given a sequential integer index, starting at 0.
We use only one variable name to refer to the whole list. To access a specific element in the list, we use the index of the element with following syntax: listName[index]. For example:
Can you write Python code for it?

myList = [3, "cat", 56.9, 4, 10, True]  # recreating the list above
print myList[0]
print myList[1]


I am working on the file "26_Prelab_Python-IV/Lesson4.ipynb".
The first task is:
[ Definition ] .split()
Purpose: Splits a string into parts based on a specified delimiter. If no delimiter is given, splits on whitespace (spaces, tabs, and newlines). Returns a list.
Syntax:
result = string.split()
result = string.split(delimiter)

Notes:

The delimiter itself is not included in the output

Using .split() to parse text files
Most data files come in a tabular format, where the data is arranged in rows and columns in some consistent way. For example, you might have a file where each row is a gene, and each column is some type of information about the gene. If you open up this file in a plain text editor, you'll see something like this:
ucscID  geneName    numReads    proteinProduct
uc007afd.1  Mrpl15  368 internal-out-of-frame
uc007afh.1  Lypla1  783 n-term-trunc
uc007afi.1  Tcea1   3852    canonical
uc007afn.1  Atp6v1h 1407    n-term-trunc
uc007agb.1  Pcmtd1  65  uorf
This might look a bit messy to read by eye, but in fact this is a perfect format for reading into our code. The important point is that on each line, the data belonging to each "column" is separated by a consistent delimiter. In this case, the delimiter is a single tab (other common delimiters are commas and spaces). Using the .split() function, we can split up each line from this file into its separate "column" components so that each piece of information can be used separately. This is often what we mean when we say we're "parsing" a data file -- we're breaking it up into meaningful parts.
Here's an example of splitting up on of the lines above:
Can you write Python code for it?

line = "uc007afd.1    Mrpl15    368    internal-out-of-frame"
print line.split()


I am working on the file "27--kmeans-part1.ipynb".
The first task is:
Check if k-means has already converged
Can you write Python code for it?

def has_converged(old_centers, centers):
    # @YOUSE: return true if the k center points remain the same
    # note that the ordering may be different
    pass


I am working on the file "27--kmeans-part1.ipynb".
The first task is:
Exercise: Can you make it even faster? We will discuss about it next time.
Clustering (grouping) based on the distance matrix
Can you write Python code for it?

def cluster_points(D):
    # @YOUSE: return an (n x 1) numpy array which shows the assigned cluster for each point
    # For example, D = [[0.3, 0.2],
    #                   [0.1, 0.5],
    #                   [0.4, 0.2]]
    # should return np.array([1,0,1])
    pass


I am working on the file "27--kmeans-part1.ipynb".
The first task is:
Putting it all together
Can you write Python code for it?

def kmeans(X, k):
    # @YOUSE: implement the k-means algorithm
    # print the objective function value (WCSS) at each iteration until it converges
    # return the final clustering
    pass

clustering = kmeans(points, k)


I am working on the file "27. 모형 최적화/01. 모형 하이퍼 파라미터 튜닝.ipynb".
The first task is:
GridSearchCV 사용예
GridSearchCV  클래스는 validation_curve 함수와 달리 모형 래퍼(Wrapper) 성격의 클래스이다.  클래스 객체에 fit 메서드를 호출하면 grid search를 사용하여 자동으로 복수개의 내부 모형을 생성하고 이를 모두 실행시켜서 최적 파라미터를 찾아준다. 생성된 복수개와 내부 모형과 실행 결과는 다음 속성에 저장된다.

grid_scores_
param_grid 의 모든 파리미터 조합에 대한 성능 결과. 각각의 원소는 다음 요소로 이루어진 튜플이다.
parameters: 사용된 파라미터
mean_validation_score: 교차 검증(cross-validation) 결과의 평균값
cv_validation_scores: 모든 교차 검증(cross-validation) 결과
best_score_
최고 점수
best_params_
최고 점수를 낸 파라미터
best_estimator_
최고 점수를 낸 파라미터를 가진 모형
Can you write Python code for it?

from sklearn.grid_search import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

pipe_svc = Pipeline([('scl', StandardScaler()), ('clf', SVC(random_state=1))])

param_range = [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0]
param_grid = [
    {'clf__C': param_range, 'clf__kernel': ['linear']},
    {'clf__C': param_range, 'clf__gamma': param_range, 'clf__kernel': ['rbf']}]

gs = GridSearchCV(estimator=pipe_svc, param_grid=param_grid, scoring='accuracy', cv=10, n_jobs=1)
%time gs = gs.fit(X, y)

print(gs.best_score_)
print(gs.best_params_)

gs.grid_scores_


I am working on the file "27. 모형 최적화/01. 모형 하이퍼 파라미터 튜닝.ipynb".
The first task is:
ParameterGrid 사용예
때로는 scikit-learn 이 제공하는 GridSearchCV 이외의 방법으로 그리드 탐색을 해야하는 경우도 있다. 이 경우 파라미터를 조합하여 탐색 그리드를 생성해 주는 명령어가 ParameterGrid 이다. ParameterGrid 는 탐색을 위한 iterator 역할을 한다.
Can you write Python code for it?

from sklearn.grid_search import ParameterGrid

param_grid = {'a': [1, 2], 'b': [True, False]}
list(ParameterGrid(param_grid))

param_grid = [{'kernel': ['linear']}, {'kernel': ['rbf'], 'gamma': [1, 10]}]
list(ParameterGrid(param_grid))


I am working on the file "27. 모형 최적화/02. 모형 최적화 분산 처리.ipynb".
The first task is:
Parallel Map

map/reduce 연산을 engine process들에게 맏겨서 동시 실행
Can you write Python code for it?

def pyprimes(kmax):
    p = np.zeros(1000)
    result = []
    if kmax > 1000:
        kmax = 1000
    k = 0
    n = 2
    while k < kmax:
        i = 0
        while i < k and n % p[i] != 0:
            i = i + 1
        if i == k:
            p[k] = n
            k = k + 1
            result.append(n)
        n = n + 1
    return result

%time result = map(pyprimes, range(700, 1000))

%time parallel_result = dview.map_sync(pyprimes, range(700, 1000))

parallel_result == result

async_result = dview.map_async(pyprimes, range(700, 1000))

async_result.progress

async_result.get()[0][-10:]


I am working on the file "27. 모형 최적화/02. 모형 최적화 분산 처리.ipynb".
The first task is:
모형 저장
모형을 분산처리하기 위해서는 sklearn.externals 서브패키지의 joblib.dump 명령과 joblib.load 명령 사용
ipyparalle 을 사용한 분산 모형 최적화
Can you write Python code for it?

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline

news = fetch_20newsgroups(subset="all")
n_samples = 3000
X_train = news.data[:n_samples]
y_train = news.target[:n_samples]

model = Pipeline([
        ('vect', TfidfVectorizer(stop_words="english", token_pattern=ur"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b")),
        ('svc', SVC()),
    ])

from sklearn.externals import joblib
import os
from sklearn.cross_validation import KFold, cross_val_score

def persist_cv_splits(X, y, K=3, name="data", suffix="_cv_%03d.pkl"):
    cv_split_filenames = []
    cv = KFold(n_samples, K, shuffle=True, random_state=0)
    for i, (train, test) in enumerate(cv):
        cv_fold = ([X[k] for k in train], y[train],
                   [X[k] for k in test], y[test])
        cv_split_filename = name + suffix % i
        cv_split_filename = os.path.abspath(cv_split_filename)
        joblib.dump(cv_fold, cv_split_filename)
        cv_split_filenames.append(cv_split_filename)
    return cv_split_filenames

cv_filenames = persist_cv_splits(X_train, y_train, name="news")
cv_filenames

def compute_evaluation(cv_split_filename, model, params):
    from sklearn.externals import joblib
    X_train_, y_train_, X_test_, y_test_ = joblib.load(cv_split_filename, mmap_mode="c")
    model.set_params(**params)
    model.fit(X_train_, y_train_)
    test_scores = model.score(X_test_, y_test_)
    return test_scores

from sklearn.grid_search import ParameterGrid
def parallel_grid_search(lb_view, model, cv_split_filenames, param_grid):
    all_tasks = []
    all_parameters = list(ParameterGrid(param_grid))
    for i, params in enumerate(all_parameters):
        task_for_params = []
        for j, cv_split_filename in enumerate(cv_split_filenames):
            t = lb_view.apply(compute_evaluation, cv_split_filename, model, params)
            task_for_params.append(t)
        all_tasks.append(task_for_params)
    return...


I am working on the file "27_Prelab_Python-V/Python_V_Prelab.ipynb".
The first task is:
6) Test your understanding.
Can you write Python code for it?

import random

#Removes a random letter from the given word
def removeLetter(word):
    wSize = len(word)
    posToRemove = random.randint(0,wSize-1) #choose which character to remove
    return word[:posToRemove] + word[(posToRemove+1):] #adds the string before that character to the string after

print removeLetter("abc")


I am working on the file "27_Prelab_Python-V/Python_V_Prelab.ipynb".
The first task is:
Functions don't always require input. In the following example, the function outputs a string which we can store to a variable. Try running the following code:
Can you write Python code for it?

def noInput():
    answer = "abcd"
    return answer

output = noInput()
print output
print answer


I am working on the file "27_Prelab_Python-V/Python_V_Prelab.ipynb".
The first task is:
Why are we able to print the "output" variable, but not the "answer" variable? When a variable is defined within a function, it is not stored outside the function. Only the values passed through return() are accessible outside the function.
4) How do you call a function?
I can then call this function with the syntax: squareMax(num1, num2). Now run the following functions, so that the result is printed below.
Can you write Python code for it?

print squareMax(5,2)

print squareMax(3,3)


I am working on the file "28--kmeans-part2.ipynb".
The first task is:
Elbow method to determine a good k
Elbow method is a general rule of thumb when selecting parameters.
The idea is to that one should choose a number of clusters so that adding another cluster doesn't give much better modeling of the data
Can you write Python code for it?

df_kcurve = pd.DataFrame(columns = ['k', 'distortion'])
for i in range(1,10):
    _, distortion = kmeans(points, i)
    df_kcurve.loc[i] = [i, distortion]

df_kcurve.plot(x="k", y="distortion")


I am working on the file "28--kmeans-part2.ipynb".
The first task is:
Fast implementation of the distance matrix computation
The idea is that $$||(x - c)||^2 = ||x||^2 -  2\langle x, c \rangle + ||c||^2 $$
This has many advantages.
1. The centers are fixed (during a single iteration), so only needs to compute once
2. Data points are usually sparse, but centers are not
3. If implement cleverly, we don't need to use for loops
Can you write Python code for it?

def compute_d2(X, centers):
    D = np.empty((n, k))
    for i in range(n):
        D[i, :] = np.linalg.norm(X[i,:] - centers, axis=1) ** 2

    return D

def compute_d2_fast(X, centers):

    # @YOUSE: compute a length-n array, where each entry is the square of norm of a point
    first_term =

    # @YOUSE: compute a (n * k) matrix, where entry (i,j) is the two times of inner product of row i of X and row j of centers
    second_term =

    # @YOUSE: compute a length-k array, where each entry is the square of norm of a center
    third_term =

    D = np.tile(first_term, (centers.shape[0], 1)).T - second_term + np.tile(third_term, (n,1))
    D[D < 0] = 0

    return D


I am working on the file "28--kmeans-part2.ipynb".
The first task is:
K-means implementation in Scipy
Actually, Python has a builtin K-means implementation in Scipy.
Scipy is a superset of Numpy, and is installed by default in many Python distributions.
Can you write Python code for it?

from scipy.cluster.vq import kmeans,vq

# distortion is the same as WCSS.
# It is called distortion in the Scipy document, since clustering can be used in compression.
centers, distortion = kmeans(points, k)

# vq return the clustering (assignment of group for each point)
# based on the centers obtained by the kmeans function.
# _ here means ignore the second return value
clustering, _ = vq(points, centers)

plot_clustering_k2(centers, clustering)


I am working on the file "28--kmeans-part2.ipynb".
The first task is:
Let's see the different in running time of the two implementations.
Can you write Python code for it?

centers = init_centers(points, k)
%timeit D = compute_d2(points, centers)
%timeit D = compute_d2_fast(points, centers)

def cluster_points(D):
    return np.argmin(D, axis=1)

def update_centers(X, clustering):
    centers = np.empty((k, d))
    for i in range(k):
        members = (clustering == i)
        if any(members):
            centers[i, :] = np.mean(X[members, :], axis=0)
    return centers

def WCSS(D):
    min_val = np.amin(D, axis=1)
    return np.sum(min_val)

def has_converged(old_centers, centers):
    return set([tuple(x) for x in old_centers]) == set([tuple(x) for x in centers])

def kmeans_basic(X, k):
    old_centers = init_centers(X, k)
    centers = init_centers(X, k)
    i = 1
    while not has_converged(old_centers, centers):
        old_centers = centers
        D = compute_d2_fast(X, centers)
        clustering = cluster_points(D)
        centers = update_centers(X, clustering)
        print "iteration", i, "WCSS = ", WCSS(D)
        i += 1
    return centers, clustering

centers, clustering = kmeans_basic(points, k)

def plot_clustering_k2(centers, clustering):
    df['clustering'] = clustering
    sns.lmplot(data=df, x="x_1", y="x_2", hue="clustering", fit_reg=False,)
    if df['clustering'][0] == 0:
        colors = ['b', 'g']
    else:
        colors = ['g', 'b']
    plt.scatter(centers[:,0], centers[:,1], s=500, c=colors, marker=u'*' )

plot_clustering_k2(centers, clustering)


I am working on the file "28_Prelab_ML-I/ML1-prelab.ipynb".
The first task is:
Now we can use the k-means function to cluster! In this case, we're saying we want to find three clusters.
Can you write Python code for it?


I am working on the file "28_Prelab_ML-I/ML1-prelab.ipynb".
The first task is:
The next function actually performs k-means clustering. You need to understand how the algorithm works at the level of the video lecture. You don't need to understand every line of this, but you should feel free to dive in if you're interested!
Can you write Python code for it?

def kmeans(data, k):
    """
    performs k-means clustering for two-dimensional data.

    params:
    data - A numpy array of shape N, 2
    k - The number of clusters.

    returns:
    a dictionary with three elements
        - ['centroids']: a list of the final centroid positions.
        - ['members']: a list [one per centroid] of the points assigned to
          that centroid at the conclusion of clustering.
        - ['paths']: a list [one per centroid] of lists [one per iteration]
          containing the points occupied by each centroid.
    """

    # http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.ndarray.shape.html#numpy.ndarray.shape
    # .shape returns the size of the input numpy array in each dimension
    # if there are not 2 dimensions, we can't handle it here.
    if data.shape[1] != 2:
        return 'This implementation only supports two dimensional data.'
    if data.shape[0] < k:
        return 'This implementation requires at least as many points as clusters.'

    # pick random points as initial centroids
    centroids = []
    for x in random.sample(data, k):
        # note the use of tuples here
        centroids.append(tuple(x.tolist()))

    paths = []
    for i in range(k):
        paths.append([centroids[i],])

    # we'll store all previous states
    # so if we ever hit the same point again we know to stop
    previous_states = set()

    # continue until we repeat the same centroid positions
    assignments = None
    while not tuple(centroids) in previous_states:
        previous_states.add(tuple(centroids))
        assignments = []
        for point in data:
            assignments.append(assign_nearest(centroids, point))

        centroids_sum = [] # Make a list for each centroid to store position sum
        centroids_n = [] # Make a list for each centroid to store counts
        for i in range(k):
            centroids_sum.append((0,0))
            centroids_n.append(0)

        for i in...


I am working on the file "28_Prelab_ML-I/ML1-prelab.ipynb".
The first task is:
The next function is used to assign an observation to the centroid that is nearest to it.
Can you write Python code for it?

def assign_nearest(centroids, point):
    """
    assigns the point to its nearest centroid

    params:
    centroids - a list of centroids, each of which has 2 dimensions
    point - a point, which has two dimensions

    returns:
    the index of the centroid the point is closest to.
    """
    nearest_idx = 0
    nearest_dist = sys.float_info.max  # largest float on your computer
    for i in range(len(centroids)):
        # sqrt((x1-x2)^2 + (y1-y2)^2)
        dist = sqrt((centroids[i][0]-point[0])**2 + (centroids[i][1]-point[1])**2)
        if dist < nearest_dist:  # smallest distance thus far
            nearest_idx = i
            nearest_dist = dist

    return nearest_idx


I am working on the file "28_Prelab_ML-I/ML1-prelab.ipynb".
The first task is:
The next line will load a file of data using the numpy function loadtxt. We've created a population of points.
Can you write Python code for it?

pop = np.loadtxt('kmeans-population.csv', delimiter=',')


I am working on the file "28_Prelab_ML-I/ML1-prelab.ipynb".
The first task is:
This next cell is full of plotting code. It uses something called matplotlib
to show kmeans clustering. Specifically it shows the path centroids took,
where they ended up, and which points were assigned to them. Feel free
to take a look at this, but understanding it goes beyond the scope of the
class.
Can you write Python code for it?

def plot_km(km, points):
    """
    Plots the results of a kmeans run.

    params:
    km - a kmeans result object that contains centroids, paths, and members

    returns:
    a matplotlib figure object
    """

    (xmin, ymin) = np.amin(points, axis=0)
    (xmax, ymax) = np.amax(points, axis=0)

    plt.figure(1)
    plt.clf()
    plt.plot(points[:, 0], points[:, 1], 'k.', markersize=2)

    for path in km['paths']:
        nppath = np.asarray(path)
        plt.plot(nppath[:, 0], nppath[:, 1])

    # Plot the calculated centroids as a red X
    centroids = np.asarray(km['centroids'])
    plt.scatter(centroids[:, 0], centroids[:, 1],
                marker='x', s=169, linewidths=3,
                color='r', zorder=10)

    plt.title('K-means clustering of simulated data.\n'
              'estimated (red), path (lines)')
    plt.xlim(xmin, xmax)
    plt.ylim(ymin, ymax)
    plt.xticks(())
    plt.yticks(())
    plt.yticks(())
    plt.show()


I am working on the file "29. 추천 시스템/01. 추천 시스템의 기초.ipynb".
The first task is:
Collaborative Filter
CF(Collaborative Filter) 방법은 모든 사용자의 데이터를 균일하게 사용하는 것이 아니라 평점 행렬이 가진 특정한 패턴을 찾아서 이를 평점 예측에 사용하는 방법이다. CF 방법도 사용자나 상품 기준으로 평점의 유사성을 살피는 Neighborhood 모형과 행렬의 수치적 특징을 이용하는 Latent Factor 모형이 있다.
Neighborhood 모형
Neighborhood 모형은 Memory-based CF라고도 한다. 이 방법은 특정 사용자의 평점을 예측하기 위해 사용하는 것이아니라 해당 사용자와 유사한(similar) 사용자에 대해 가중치를 준다.
특히 해당 사용자와 유사한 사용자를 찾는 방법 즉, 평점 행렬에서 유사한 사용자 행 벡터를 찾아서 이를 기반으로 빈 데이터를 계산하는 방법을 사용자 기반 (User-based) CF라고 한다.
이와 달리 특정한 상품에 대해 사용자가 준 점수 즉, 평점 행렬의 상품 열 벡터의 유사성을 찾고 특정 상품과 유사한 평점 정보를 가지는 상품들로 해당 상품의 빈 데이터를 예측하는 방법을 상품 기반 (Item-based) CF라고 한다.
유사도 계산
사용자 특성 벡터(평점 행렬의 행 벡터)이나 상품 특성 벡터(평점 행렬의 열 벡터)의 유사도(similarity)을 비교하기 위한 기준도 여러가지가 있을 수 있다.
surprise 패키지에서는 다음과 같은 유사도 기준을 제공한다.

평균제곱차이 유사도 (Mean Squared Difference Similarity)
코사인 유사도 (Cosine Similarity)
피어슨 유사도 (Pearson Similarity)
피어슨-베이스라인 유사도 (Pearson-Baseline Similarity)

평균제곱차이 유사도 (Mean Squared Difference Similarity)
일단 다음과 같이 msd(Mean Squared Difference)값을 계산한다. msd는 유클리드 공간에서의 거리 제곱에 비례하는 값이다.

사용자 $u$와 사용자 $v$간의 msd
$$ \text{msd}(u, v) = \frac{1}{|I_{uv}|} \cdot \sum\limits_{i \in I_{uv}} (r_{ui} - r_{vi})^2 $$
위 식에서 $I_{uv}$는 사용자 $u$와 사용자 $v$ 모두에 의해 평가된 상품의 집합이고 $|I_{uv}|$는 사용자 $u$와 사용자 $v$ 모두에 의해 평가된 상품의 수

상품 $i$와 상품 $j$간의 msd
$$ \text{msd}(i, j) = \frac{1}{|U_{ij}|} \cdot \sum\limits_{u \in U_{ij}} (r_{ui} - r_{uj})^2 $$
위 식에서 $U_{ij}$는 상품 $i$와 상품 $j$ 모두를 평가한 사용자의 집합이고 $|U_{ij}|$는 상품 $i$와 상품 $j$ 모두를 평가한 사용자의 수

유사도는 msd 값의 역수로 계산한다. 즉 거리가 멀수록 유사도는 떨어진다. msd값이 0이 되는 경우를 대비하여 1을 더해준다.
$$ \begin{split}\text{msd_sim}(u, v) &= \frac{1}{\text{msd}(u, v) + 1}\
\text{msd_sim}(i, j) &= \frac{1}{\text{msd}(i, j) + 1}\end{split} $$
코사인 유사도 (Cosine Similarity)
코사인 유사도(Cosine Similarity)는 두 특성 벡터의 각도에 대한 코사인 값을 말한다. 벡터 $x$와 벡터 $y$ 사이의 각도 $\theta$ 는 두 벡터의 내적 $x \cdot y$와 다음과 같은 관계가 있다. 각도 $\theta$가 0도이면 코사인 유사도는 1이다. 반대로 각도 $\theta$가 90도이면 코사인 유사도는 0이다.
$$ x \cdot y = |x| |y| \cos\theta $$
$$ \cos\theta = \dfrac{x \cdot y}{|x| |y|}$$

사용자 $u$와 사용자 $v$간의 msd

$$ \text{cosine_sim}(u, v) = \frac{
\sum\limits_{i \in I_{uv}}...
Can you write Python code for it?

sim_options = {'name': 'msd'}
algo = surprise.KNNBasic(sim_options=sim_options)
surprise.evaluate(algo, data)

sim_options = {'name': 'cosine'}
algo = surprise.KNNBasic(sim_options=sim_options)
surprise.evaluate(algo, data)

sim_options = {'name': 'pearson'}
algo = surprise.KNNBasic(sim_options=sim_options)
surprise.evaluate(algo, data)

sim_options = {'name': 'pearson_baseline'}
algo = surprise.KNNBasic(sim_options=sim_options)
surprise.evaluate(algo, data)

sim_options = {'name': 'pearson_baseline'}
algo = surprise.KNNWithMeans(sim_options=sim_options)
surprise.evaluate(algo, data)

sim_options = {'name': 'pearson_baseline'}
algo = surprise.KNNBaseline(sim_options=sim_options)
surprise.evaluate(algo, data)


I am working on the file "29. 추천 시스템/01. 추천 시스템의 기초.ipynb".
The first task is:
여기에서 user 열은 사용자 아이디, item 열은 상품 아이디, rate 열은 평점이다. 즉, 196번 사용자는 242번 영화에 대해 평점 3점을 주었음을 알 수 있다.
이 데이터프레임에서 볼 수 있듯이

추천 시스템은 사용자 아이디와 상품 아이디라는 두 개의 카테고리 입력과 평점 출력을 가지는 예측 시스템

이다.
이 데이터를 다음과 같이 피봇테이블(pivot table) 형태로 만들면 x축이 상품, y축이 사용자 아이디인 평점 행렬(rate matrix) $R$ 이 된다.
평점 행렬 $R$의 행은 특정 사용자의 평점이고 평점 행렬 $R$의 열은 특정 상품의 평점이다.
Can you write Python code for it?

df_table = df.set_index(["user", "item"]).unstack()


I am working on the file "29. 추천 시스템/01. 추천 시스템의 기초.ipynb".
The first task is:
이 데이터는 다음과 같이 데이터프레임으로 변환할 수 있다.
Can you write Python code for it?

df = pd.DataFrame(data.raw_ratings, columns=["user", "item", "rate", "id"])
del df["id"]
df.head(10)


I am working on the file "29. 추천 시스템/01. 추천 시스템의 기초.ipynb".
The first task is:
추천 시스템 알고리즘
추천 시스템은 두 개의 카테고리 값 입력에서 하나의 실수 값 출력을 예측하는 회귀 모형이지만 여러가지 방법으로 예측 성능을 향상시키고 있다. 추천 시스템에서 사용되는 알고리즘은 다음과 같다.

베이스라인 모형
Collaborative Filtering
2-1. Neighborhood Models
User-based CF
Item-based CF

2-2. Latent Factor Models
Matrix Factorization
SVD

Content-Based Recommendation

베이스라인 모형
베이스라인 모형(baseline model)은 사용자 아이디 $u$, 상품 아이디 $i$, 두 개의 카테고리 값 입력에서 평점 $r_{ui}$의 예측치 $\hat{r}_{ui}$  을 예측하는 가장 단순한 모형으로 다음과 같이 사용자와 상품 특성에 의한 평균 평점의 합으로 나타난다.
$$ \hat{r}_{ui} = \mu + b_u + b_i $$
이 식에서 $\mu$는 전체 평점의 평균이고, $b_u$는 동일한 사용자에 의한 평점 조정값,  $b_i$는 동일한 상품에 대한 평점 조정값이다.
베이스라인 모형은 오차 함수를 최소화하도록 구해진다.
$$ \sum_{r_{ui} \in R_{train}} \left(r_{ui} - (\mu + b_u + b_i)\right)^2 $$
여기에서 $R_{train}$는 트레이닝을 위한 데이터셋을 말한다.
과최적화를 피하기 위해 다음과 같이 정규화(regulaization) 항을 추가할 수 있다.
$$ \sum_{r_{ui} \in R_{train}} \left(r_{ui} - (\mu + b_u + b_i)\right)^2 + \lambda \left(b_u^2 + b_i^2 \right) $$
최적화 알고리즘
surprise 패키지는 오차 함수를 최소화하기 위해 다음과 같은 두 가지 최적화 알고리즘을 제공한다. 알고리즘의 선택은 method 인수를 사용한다. 최적화 알고리즘에 따라 나머지 최적화 인수가 달라진다. 최적화 인수에 대한 자세한 설명은 Koren2010 논문을 참조한다.

ALS (Alternating Least Squares)의 인수
reg_i: 상품에 대한 정규화 가중치. 디폴트는 10.
reg_u: 사용자에 대한 정규화 가중치. 디폴트는 15.

n_epochs: 최적화 반복 횟수. 디폴트는 10.

SGD (Stochastic Gradient Descent)의 인수

reg: 정규화 가중치. 디폴트는 0.02.
learning_rate: 최적화 스텝 사이즈. 디폴트는 0.005.
n_epochs: 최적화 반복 횟수. 디폴트는  20.

모형 사용법
베이스라인 모형을 비롯한 surprise 패키지 모형을 사용하기 위해서는 다음과 같은 순서를 거친다.

데이터세트의 split,  folds 메소드를 사용하여 K-Fold 트레이닝 데이터셋과 테스트 데이터셋을 만든다.
모형 알고리즘 객체를 생성한다.
모형 알고리즘 객체의 train 메서드와 트레이닝 데이터셋으로 모수를 추정한 후, test 메서드로 테스트 데이터셋에 대한 예측을 실시한다.
accuracy 서브패키지의 성능평가 함수를 사용하여 예측 성능을 계산한다.

이 과정은 evaluate 명령으로 단축할 수도 있다.
surprise 패키지는 베이스라인 모형을 위한 BaselineOnly 클래스를 제공한다.
우선 베이스라인 모형으로 다음과 같이 MovieLens 데이터를 처리해 보자. FCP(Fraction of Concordant Pairs)로 계산한 평가 점수는 약 0.70점이다.
Can you write Python code for it?

data.split(n_folds=3)

bsl_options = {'method': 'als',
               'n_epochs': 5,
               'reg_u': 12,
               'reg_i': 5
               }
algo = surprise.BaselineOnly(bsl_options=bsl_options)

np.random.seed(0)
acc = np.zeros(3)
for i, (trainset, testset) in enumerate(data.folds()):
    algo.train(trainset)
    predictions = algo.test(testset)
    acc[i] = surprise.accuracy.fcp(predictions, verbose=True)
acc.mean()


I am working on the file "29. 추천 시스템/01. 추천 시스템의 기초.ipynb".
The first task is:
추천성능 평가기준
accuracy 서브패키지에서는 다음과 같은 추천성능 평가기준을 제공한다.

RMSE (Root Mean Squared Error)
$$ \text{RMSE} = \sqrt{\frac{1}{|\hat{R}|} \sum_{\hat{r}{ui} \in \hat{R}}(r{ui} - \hat{r}_{ui})^2} $$

이 식에서 $\hat{R}$은 테스트 데이터셋을 뜻한다.

MAE (Mean Absolute Error)
$$ \text{MAE} = \frac{1}{|\hat{R}|} \sum_{\hat{r}{ui} \in \hat{R}}|r{ui} - \hat{r}_{ui}| $$

FCP (Fraction of Concordant Pairs)

위 평가기준 중에서 FCP는 Karen2011 에서 제안된 것으로 평점 자체가 아닌 평점에 의한 순위를 기준으로 하는 방법이다.
evaluate 명령을 사용하면 위 코드를 다음과 같이 짧게 만들면서 여러가지 평가기준을 사용할 수 있다.
Can you write Python code for it?


I am working on the file "29. 추천 시스템/01. 추천 시스템의 기초.ipynb".
The first task is:
평점 행렬의 빈칸을 흰색, 점수를 검은색으로 시각화 하면 다음과 같다.
Can you write Python code for it?

plt.imshow(df_table)
plt.grid(False)
plt.xlabel("item")
plt.ylabel("user")
plt.title("Rate Matrix")
plt.show()


I am working on the file "29_ML-II/ML2_svms_and_overfitting.ipynb".
The first task is:
Congratulations! You've built your first SVM, and on training data it separates tumor data from normal data with over 90% accuracy! Now that we've done this with some biomedical data, let's take a step back and talk about things we should consider as we build a model.
Q1: What are our labels?
Q2: What are our features?
Q3: What are our examples?
Overfitting in machine learning
When you train a computer to build a model that describes data that you've seen, a challenge known as "overfitting" can arise. When fitting the model, we want to find a model that fits the data as well as possible. However, real data is noisy. The model that fits data we have with the least error may capture the main features of the data, but may also capture noise in the data that we don't intend to model.  When a model fits noise in training data, we call this problem overfitting.
For example, imagine that a professor wants to test a group of students' knowledge of calculus. She gives the students previous exam questions and answers to study. However, in the final exam, she uses the same questions to test the students. Some of the students could do very well because they memorized answers to the questions even though they don't understand calculus. The professor realizes this problem and then gives the students a new set of questions to test them. The students who memorized all the answers to previous exam questions may fail the new exam because they have no idea how to solve the new problems. We would say that those students have "overfit" to training data.
How can overfitting be a problem with machine learning? Don't we want the model to fit the data as well as possible? The reason is we want a model that captures the features that will also exist in some new data. If the model fits the noise in the data, the model will perform poorly on new data sets!
Let's use simulations to illustrate the overfitting problem. We are going to simulate two variables x and y and we let y = x + e, where e is some noise. That is, y is a linear...
Can you write Python code for it?

## Load necessary Python packages
import numpy as np # numpy makes it convenient to load/modify matrices of data
import sklearn.linear_model as lm # this scikit learn module has code to fit a line
import matplotlib.pyplot as plt # this lets us plot our results
from sklearn.metrics import mean_squared_error # we use this to see how well our model fits data
%matplotlib inline

# This code will make our data by adding random noise to a linear relationship
# Simulate two variables x and y
# y=x+e, e is some noise
x = np.linspace(0., 2, 10)
y = x + 0.5*np.random.randn(len(x))


I am working on the file "29_ML-II/ML2_svms_and_overfitting.ipynb".
The first task is:
Let's plot the old models applied to the new data.
Can you write Python code for it?

plt.figure(figsize=(8,6))
plt.scatter(x_new, y_new)
plt.plot(x, y_lr, 'g',label='Simple regression')
plt.title("Linear regression")
plt.plot(x_plot, y_plot,label='Multiple regression')
plt.legend(loc=2)


I am working on the file "29_ML-II/ML2_svms_and_overfitting.ipynb".
The first task is:
MSE for multiple regression on new data:
Can you write Python code for it?


I am working on the file "29_ML-II/ML2_svms_and_overfitting.ipynb".
The first task is:
MSE for simple regression on new data:
Can you write Python code for it?


I am working on the file "29_ML-II/ML2_svms_and_overfitting.ipynb".
The first task is:
Next, we want to train linear regression models on x and use the models to predict y. The models we are going to use are:
1. A simple linear regression model: Y~X
2. A complex multiple regression model: Y ~ X + X^2 + X^3 + X^4 ... + X^10
We want to choose the model that will most accurately predict y.
Let's use ski-learn to train these two models:
Can you write Python code for it?

# You don't need to know how this code works. We're not going to focus on regression
# during this course. You may want to have it to refer to in the future.

### simple regression
lr = lm.LinearRegression()
lr.fit(x[:,np.newaxis], y);
y_lr = lr.predict(x[:, np.newaxis])
### multiple regression
lrp = lm.LinearRegression()
lrp.fit(np.vander(x, N=10, increasing=True), y)
y_lrp = lrp.predict(np.vander(x, N=10, increasing=True))
x_plot = np.linspace(0., 2, 1000)
y_plot = lrp.predict(np.vander(x_plot, N=10, increasing=True))


I am working on the file "29_ML-II/ML2_svms_and_overfitting.ipynb".
The first task is:
The multiple regression model fits the data perferlly (MSE is almost 0). The predicted values are the exact the same as the observed values since the prediction curve goes through every point. However, the simple regression model captures the linear relation between x and y but it didn't predict perfectlly well with the observed values. Then, shoud we choose multiple regression model rather than simple regression model since the former fitts the data much better than the latter?
Q4: Which model do you think is the better model? Why?
Remember that we want to find a model that fits the data well and, most importantly, can predict well on some new data. Let's simulate some new data and see the prediction performance of each model on the new data.
Can you write Python code for it?

x_new = np.linspace(0., 2, 10)
y_new = x + 0.5*np.random.randn(len(x_new))
y_lr_new = lr.predict(x_new[:, np.newaxis])
y_lrp_new = lrp.predict(np.vander(x_new, N=10, increasing=True))