I am working on the file "MainCodeDoc.ipynb".
The first task is:
We also calculate the delay column, which is one if a flight was cancelled, diverted, or has delayed over 30 min
Can you write Python code for it?

sampledf["Delayed"] = sampledf.apply(lambda row: 1 if(row.CANCELLED or row.DIVERTED or row.ARR_DELAY > 30) else 0 , axis=1)
sampledf.columns


I am working on the file "MainFile.ipynb".
The first task is:
First, define some useful functions. These are used in the printing of data later on. Also used for rendering images to the HTML page.
Can you write Python code for it?

def printMatrix(data):  #used to print matricies to HTML
    display(HTML(
        '

{}

'.format(
            '

'.join(
                '{}	'
                .format('	'
                .join(str(_) for _ in row)) for row in data)
        )
    ))

def printText(text):
    display(HTML('
' + text + '
'))

def displayHTML(html):
    display(HTML(html))

def drawImg(img):
    b = BytesIO()
    img.save(b, format='png')
    displayHTML(""
                .format(b64encode(b.getvalue()).decode('utf-8')))


I am working on the file "MainFile.ipynb".
The first task is:
Question 5
The autocorrelation function for each year’s data, where τ ranges from 0 to 364 is now generated.
Confidence intervals are drawn as a cone. By default, this is set to a 95% confidence interval, suggesting that correlation values outside of this code are very likely a correlation and not a statistical fluke.
First, each Autocorrelation is drawn on its own graph. After, they are overlayed.
Autocorrelation is found through the use of the plot_acf function from the tsaplots statistics library. From a signal processing point of view, this can be seen as:
$$R_{ff}(\tau) = (f * g_{-1}(\overline{f}))(\tau) = \int_{-\infty}^\infty f(u+\tau)\overline{f}(u)\, {\rm d}u = \int_{-\infty}^\infty f(u)\overline{f}(u-\tau)\, {\rm d}u$$
Comment on findings
This autocorrelation shows how each year is related to a time-shifted version of itself. This can be seen as an effective convolution process. From this, one can see how each part of the year is related to the rest of the year for a given shift amount τ. For example, at τ=0, the magnitude is 1. At this time, no shifting has occurred. At approximately τ=150, there is the highest negative correlation seen. This is due to the correlation between the middle of summer and winter of the two data sets. At this negative region, the corilation is comparing the increase in summer against the decrease in winter showing a negative value.
Can you write Python code for it?

counter = 0;
for year in weatherData:

    if (counter == 0):  #on the first loop, generate the subplots
        fig, axs = plt.subplots(1, 2)

    if counter == 4:  # if we are at the last figure, we want it to be on its own line
        plot_acf(year[:, 2],
                 title = "Autocorrelation for {}"
                 .format(1995 + 5 * counter))
        plt.savefig("Figures/Autocorrelation for 2015.pdf")
        plt.show()
    else:
        plot_acf(year[:, 2],
                 title = "Autocorrelation for {}"
                 .format(1995 + 5 * counter),
                 ax=axs[counter % 2])
    if counter % 2 == 1:  # every two figures, we need to generate a new row

        plt.savefig("Figures/Autocorrelation for {} and {}.pdf"
                    .format(1995 + 5 * counter-5,1995 + 5 * counter))
        plt.show()
        if (counter < 2):  #a new sub plot is needed on second row
            fig, axs = plt.subplots(1, 2)
    counter = counter + 1

counter = 0;
fig, axs = plt.subplots(1, 1)
for year in weatherData:

    plot_acf(year[:, 2],
             title = "Autocorrelation All 5 years together"
             .format(1995 + 5 * counter),
            ax=axs)
plt.savefig("Figures/All years Autocorrelation.pdf")
plt.show()


I am working on the file "MainFile.ipynb".
The first task is:
Question 6.a &6.b
Next, each year, temp is broken down into subdivisions in the range:
$$[minimum-0.1=t_0,maximum+0.1=t_10]$$
into ten equal intervals. This is then used to generate 10 intervals, as:
$$[[t_0,t_1],...,[t_9,t_10]]$$

binCorours are predefined RGB values, given in the question
The linspace function below is used to generate a set of numbers with equal distributions between the min and max data for each year.
Digitisation is used to put the data into the respective buckets. This can then be used later on when the graphs are drawn
generate a matrix to store the image colour values
iterate over each day of the year*2. double as we need to have two rows for each day(high and low)
set the current iteration-1 row of pixels to the low digitised index value from colour matrix provided.
same as above, but for the current iterator for the high values provided
convert matrix to image
draw image to screen

The output below this code shows first the bins for each year, for question 6.a. Then, the blankets are drawn for question 6.b
Can you write Python code for it?

binColours = [[0.139681, 0.311666, 0.550652], [0.276518, 0.539432, 0.720771],
              [0.475102, 0.695344, 0.802081], [0.670448, 0.803486, 0.824645],
              [0.809791, 0.848259, 0.777550], [0.861927, 0.803423, 0.673050],
              [0.830690, 0.667645, 0.546349], [0.742023, 0.475176, 0.424114],
              [0.613033, 0.281826, 0.306352], [0.450385, 0.157961, 0.217975]]
counter = 0;
for year in weatherData:
    num = 10
    binsLow = np.linspace(year[:, 3].min() - 0.1, year[:, 3].max() + 0.1, num)
    binsHigh = np.linspace(year[:, 1].min() - 0.1, year[:, 1].max() + 0.1, num)

    digitizedLow = np.digitize(year[:, 3], binsLow)  #put the data into the bins
    digitizedHigh = np.digitize(year[:, 3], binsHigh)

    rows = 2 * len(year[:, 3])
    image = np.zeros((rows, 451, 3))  #make a matrix to store the image
#itterate over each year's values from the above values and set pixels colours
    for x in range(0, rows, 2):
        image[x - 1, 0:451] = binColours[int(digitizedLow[int(x / 2)])]
        image[x, 0:451] = binColours[int(digitizedHigh[int(x / 2)])]
    printText("Tempreture blanket for year: {}".format(1995 + 5 * counter))

    displayHTML(pd.DataFrame(np.column_stack((binsLow.reshape(10, 1),
                                              (binsHigh.reshape(10, 1)))),
                             columns=["Low bin", "High bin"]).to_html())

    outputImage = smp.toimage(image)
    outputImage.save('WeatherBlancketsOutput/{}.jpg'.format(1995 + 5 * counter))
    drawImg(outputImage)  #Draw image to screen, using custom draw function to put output in window
    counter = counter + 1;


I am working on the file "MainFile.ipynb".
The first task is:
Question 7
Next, a surface plot of a stochastic process is generated. For this, avarages are generated over all 5 sets of data for each year, as well as a standard diviation for each day. Next, a linear space is generated representing the posible tempreture ranges. finally, the PDF is found then plotted in 3D.
Can you write Python code for it?

plt.rcParams['figure.figsize'] = (30, 20)  #Make the figure for this question bigger

#define the variables to store the mean, std deviation,
# range for each day(temp), pdfs and a vector for number of days
meanDay = np.zeros((365, 1))
stdDay = np.zeros((365, 1))
dayArray = np.zeros((5, 1))
pdf = np.zeros((365, 365))
dayRange = range(0, 365, 1)

for day in dayRange:
    for inx, year in enumerate(weatherData):
        dayArray[inx, 0] = year[:, 2][day]
    meanDay[day] = dayArray.mean()  #calculate the mean
    stdDay[day] = dayArray.std()  # calculate the standard deviation

#generate a linear space of all days in the region, for the min to max temp
tempretureSpace = np.linspace(meanDay.min(),
                              meanDay.max(),
                              len(weatherData[0][:, 2]))

#itterate over the days again, now generating the pdf
for day in dayRange:
    pdf[day, :] = norm.pdf(tempretureSpace,
                           meanDay[day],
                           stdDay[day])

#convert the values to a meshgrid
# (return coordinate matrices from coordinate vectors)
tempretureSpace, dayRange = np.meshgrid(tempretureSpace, dayRange)

#finally, plot it as a 3d surf
fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(tempretureSpace,
                       dayRange,
                       pdf,
                       cmap=cm.coolwarm,
                       linewidth=0,
                       antialiased=False)

ax.set_xlabel('Temperature (C)')
ax.set_ylabel('Days')
ax.set_zlabel('f(x)')

#add a colour bar on the side to see magnitudes
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.title('Surface plot of stochastic process')
plt.savefig("Figures/surface plot of stochastic process.pdf")
plt.show()


I am working on the file "Making better plots.ipynb".
The first task is:
It almost looks like we took two steps forward and one step back: now we have a box again. But seaborn provides a useful function for removing axis lines: despine.
Can you write Python code for it?


I am working on the file "Making better plots.ipynb".
The first task is:
Now let's do some refining. Figures for exploratory work can be any size that's convenient, but when making figures for a publication, you must consider the real size of the figure in the final printed form. Considering a page in the U.S. is typically 8.5 x 11 inches, a typical figure should be no more than 4 inches wide to fit in a single column of the page. We can adjust figure sizes by giving matplotlib a bit more detail:
Can you write Python code for it?

fig = plt.figure(figsize=(4, 2))
ax = fig.add_subplot(1,1,1)

ax.plot(x, y)
sns.despine(offset=10, ax=ax)

# let's add some axes labels to boot
ax.set_ylabel(r'displacement ($\AA$)')
ax.set_xlabel('time (ns)')


I am working on the file "Making better plots.ipynb".
The first task is:
The default plots created with matplotlib aren't bad, but they do have elements that are, at best, unnecessary. At worst, these elements detract from the display of quantitative information. We want to change this. First, let's change matplotlib's style with a built-in style sheet.
Can you write Python code for it?

# this gives us a style and color palette similar to ggplot2
plt.style.use('ggplot')

plt.plot(x, y)


I am working on the file "Making better plots.ipynb".
The first task is:
This produces something a bit more pleasing to the eye, with what probably amounts to better coloration. It replaced the box with a grid, however. Although this is useful for some plots, in particular panels of plots in which one needs to compare across different plots, it is often just unnecessary noise.
Using seaborn can help with this. We'll import seaborn's helper functions without importing its style:
Can you write Python code for it?

# import seaborn's helpful functions without applying its style
import seaborn.apionly as sns

# importing seaborn can sometimes reset matplotlib's style to default
plt.style.use('ggplot')

# this will remove the noisy grids the 'ggplot' style gives
sns.set_style('ticks')

plt.plot(x, y)


I am working on the file "Making better plots.ipynb".
The first task is:
We added some axes labels, too. Because this is a timeseries, we deliberately made the height of the figure less than the width. This is because timeseries are difficult to interpret when the variations with time are smashed together. Tufte's general rule is that no line in the timeseries be greater than 45$^\circ$; we would have a hard time doing that here with such noisy data, but going wider than tall is a step in the right direction.
Plotting to files
We can save figures in a variety of formats. It's useful to save a version as a PDF so that it can be postprocessed using vector graphics tools like Inkscape and Adobe Illustrator, but because vector graphics must be rendered by the viewer on load, it's useful to also write out a PNG.
PNGs are raster graphics: they are just a matrix of pixels with four components (red, green, blue, and alpha (transparency)). This means they are quick to render with your favorite viewer, even if the plot originally had hundreds of thousands of points. However, they are not so great for making posters and final publication-quality figures, since they cannot be scaled to any size like vector graphics.
Can you write Python code for it?

fig = plt.figure(figsize=(4, 2))
ax = fig.add_subplot(1,1,1)

ax.plot(x, y)
sns.despine(offset=10, ax=ax)

# let's add some axes labels to boot
ax.set_ylabel(r'displacement ($\AA$)')
ax.set_xlabel('time (ns)')

fig.savefig('testfigure.pdf')
fig.savefig('testfigure.png', dpi=300)


I am working on the file "Making better plots.ipynb".
The first task is:
We can also go further, moving the axes a bit so they distract even less from the data, which should be front-and-center.
Can you write Python code for it?


I am working on the file "Making better plots.ipynb".
The first task is:
We can view the resulting PNG directly:
Can you write Python code for it?

from IPython.display import Image

Image(filename='testfigure.png')


I am working on the file "Making better plots.ipynb".
The first task is:
Woah...something's wrong. The figure doesn't fit in the frame! This is because the figure elements were adjusted after the figure object was created, and so some of these elements, including the axis labels, are beyond the figure's edges. We can usually fix this with a call to plt.tight_layout to ensure everything fits in the plots we write out.
Can you write Python code for it?

fig = plt.figure(figsize=(4, 2))
ax = fig.add_subplot(1,1,1)

ax.plot(x, y)
sns.despine(offset=10, ax=ax)

# let's add some axes labels to boot
ax.set_ylabel(r'displacement ($\AA$)')
ax.set_xlabel('time (ns)')

# and now we'll also refine the y-axis ticks a bit
ax.set_ylim(-4.5, 4.5)
ax.set_yticks(np.linspace(-4, 4, 5))
ax.set_yticks(np.linspace(-3, 3, 4), minor=True)

plt.tight_layout()

fig.savefig('testfigure.pdf')
fig.savefig('testfigure.png', dpi=300)

Image(filename='testfigure.png')


I am working on the file "Making_Data_Management.ipynb".
The first task is:
General information on the Gapminder data
Can you write Python code for it?

display(Markdown("Number of countries: {}".format(len(data))))
display(Markdown("Number of variables: {}".format(len(data.columns))))

# Convert interesting variables in numeric format
for variable in ('internetuserate', 'suicideper100th', 'employrate'):
    data[variable] = pd.to_numeric(data[variable], errors='coerce')


I am working on the file "Making_Data_Management.ipynb".
The first task is:
Suicide per 100,000 people frequencies
Can you write Python code for it?

display(Markdown("Suicide per 100,000 people (min, max) = ({:.2f}, {:.2f})".format(subdata['suicideper100th'].min(), subdata['suicideper100th'].max())))

suiciderate_bins = pd.cut(subdata['suicideper100th'],
                          bins=np.linspace(0, 40., num=21))

counts2 = suiciderate_bins.value_counts(sort=False, dropna=False)
percentage2 = suiciderate_bins.value_counts(sort=False, normalize=True, dropna=False)
data_struct = {
    'Counts' : counts2,
    'Cumulative counts' : counts2.cumsum(),
    'Percentages' : percentage2,
    'Cumulative percentages' : percentage2.cumsum()
}

suiciderate_summary = pd.DataFrame(data_struct)
suiciderate_summary.index.name = 'Suicide (per 100 000 people)'
(suiciderate_summary[['Counts', 'Cumulative counts', 'Percentages', 'Cumulative percentages']]
                     .style.set_precision(3)
                           .set_properties(**{'text-align':'right'}))


I am working on the file "Making_Data_Management.ipynb".
The first task is:
Unemployment rate frequencies
Can you write Python code for it?

display(Markdown("Unemployment rate (min, max) = ({0:.2f}, {1:.2f})".format(subdata['unemployrate'].min(), subdata['unemployrate'].max())))

unemployment_bins = pd.cut(subdata['unemployrate'],
                         bins=np.linspace(0, 100., num=21))

counts3 = unemployment_bins.value_counts(sort=False, dropna=False)
percentage3 = unemployment_bins.value_counts(sort=False, normalize=True, dropna=False)
data_struct = {
    'Counts' : counts3,
    'Cumulative counts' : counts3.cumsum(),
    'Percentages' : percentage3,
    'Cumulative percentages' : percentage3.cumsum()
}

unemployment_summary = pd.DataFrame(data_struct)
unemployment_summary.index.name = 'Unemployement rate (% population age 15+)'
(unemployment_summary[['Counts', 'Cumulative counts', 'Percentages', 'Cumulative percentages']]
                     .style.set_precision(3)
                           .set_properties(**{'text-align':'right'}))


I am working on the file "Making_HTTP_requests.ipynb".
The first task is:
Custom headers
Headers of the response are easy to check, example follows.
Can you write Python code for it?

r = requests.get("http://www.python.org/")
print(r.headers)


I am working on the file "Making_HTTP_requests.ipynb".
The first task is:
The Requests function json() convert the json response to Python dictionary. In next code block is demonstrated how to get data from obtained response.
Persistent session with Requests
Session with Requests are handy for cases where you need to use same cookies (session cookies for example) or authentication for multiple requests.
Can you write Python code for it?

s = requests.Session()
print("No cookies on start: ")
print(dict(s.cookies))
r = s.get('http://google.cz/')
print("\nA cookie from google: ")
print(dict(s.cookies))
r = s.get('http://google.cz/?q=cat')
print("\nThe cookie is perstent:")
print(dict(s.cookies))


I am working on the file "Making_HTTP_requests.ipynb".
The first task is:
The request headers can be modified in simple way as follows.
Can you write Python code for it?

headers = {
    "Accept": "text/plain",
}

r = requests.get("http://www.python.org/", headers=headers)
print(r.status_code)


I am working on the file "Making_HTTP_requests.ipynb".
The first task is:
The variable data contains returned HTML code (full page) as string. You can process it, save it, or do anything else you need.
Requests
Example how to get static content of web page with Requests follows.
Can you write Python code for it?

import requests

r = requests.get("http://www.python.org/")
data = r.text

print("Status code:", r.status_code)


I am working on the file "Manage Complexity.ipynb".
The first task is:
Analysing performance
Can you write Python code for it?

def average(numbers):
    """Return the average (arithmetic mean) of a sequence of numbers."""
    return sum(numbers) / float(len(numbers))

def timedcalls(n, fn, *args):
    """Call fn(*args) repeatedly: n times if n is an int, or up to
    n seconds if n is a float; return the min, avg, and max time"""
    if isinstance(n, int):
        times = [timedcall(fn, *args)[0] for _ in xrange(n)]
    else:
        times = []
        while sum(times) <= n:
            times.append(timedcall(fn, *args)[0])
    return min(times), average(times), max(times)

print "It takes around {1:.3f} seconds to find the answer.".format(*timedcalls(10, zebra_puzzle))

def c(sequence):
    """Generate items in sequence. c.start is the number of sequences start.
    c.count is the number of items generated."""
    c.start += 1
    for item in sequence:
        c.count += 1
        yield item
def zebra_puzzle():
    """Return a tuple (WATER, ZEBRA) indicating their houses numbers."""
    houses = first, _, middle, _, _ = range(1,6)
    orderings = list(it.permutations(houses))
    return next((WATER, ZEBRA)
            for (red, green, ivory, yellow, blue) in c(orderings)
            if imright(green, ivory)
            for (Englishman, Spaniard, Ukranian, Japanese, Norwegian) in c(orderings)
            if Englishman is red
            if Norwegian is first
            if nextto(Norwegian, blue)
            for (dog, snails, fox, horse, ZEBRA) in c(orderings)
            if Spaniard is dog
            for (coffee, tea, milk, oj, WATER) in c(orderings)
            if coffee is green
            if Ukranian is tea
            if milk is middle
            for (OldGold, Kools, Chesterfields, LuckyStrike, Parliaments) in c(orderings)
            if OldGold is snails
            if Kools is yellow
            if nextto(Chesterfields, fox)
            if nextto(Kools, horse)
            if LuckyStrike is oj
            if Japanese is Parliaments
            )

def instrument_fn(fn, *args):
    c.start, c.count = 0, 0
    result =...


I am working on the file "Manage Complexity.ipynb".
The first task is:
Cryptarithmetic
Can you write Python code for it?

import string, re, itertools

def solve(formula):
    """Given a formula like 'ODD + ODD == EVEN', fill in digits to solve it.
    Input formula is a string; output is a digit-filled-in string or None."""
    for f in fill_in(formula):
        if valid(f):
            return f

def fill_in(formula):
    "Generate all possible fillings-in of letters in formula with digits."
    letters = ''.join(set(re.findall('[A-Z]', formula)))
    for digits in itertools.permutations('1234567890', len(letters)):
        table = string.maketrans(letters, ''.join(digits))
        yield formula.translate(table)

def valid(f):
    """Formula f is valid if and only if it has no
    numbers with leading zero, and evals true."""
    try:
        return not re.search(r'\b0[0-9]', f) and eval(f) is True
    except ArithmeticError:
        return False

def test():
    examples = """ODD + ODD == EVEN
                  A**2 + B**2 == C**2
                  APPLE + PEN == APPLEPEN"""
    for example in examples.strip().split("\n"):
        print solve(example.strip())

test()

import cProfile
cProfile.run("test()")


I am working on the file "Manage Complexity.ipynb".
The first task is:
From the profiling, we could see that 'valid' takes far more time than 'fill_in'. In order to speed the funciton of 'valid', we should compile formula into function instead of using 'eval' too many times. For example, compiling 'YOU == ME*2' into 'lambda Y,M,E,U,O: (U+10*O+100*Y) == (E+10M)**2'.
Can you write Python code for it?

def faster_solve(formula):
    """Given a formula like 'ODD + ODD == EVEN', fill in digits to solve it.
    Input formula is a string; output is a digit-filled-in string or None.
    This version precompiles the formula; only one eval per formula."""
    f, letters = compile_formula(formula)
    for digits in itertools.permutations(range(0,10), len(letters)):
        try:
            if f(*digits):
                table = string.maketrans(letters, ''.join(map(str, digits)))
                return formula.translate(table)
        except ArithmeticError:
            pass

def compile_word(word):
    if word.isupper():
        terms = [("{}*{}".format(10**i, d))
                 for (i, d) in enumerate(word[::-1])]
        return "({})".format("+".join(terms))
    else:
        return word

def compile_formula(formula, verbose=False):
    """Compile formula into a function. Also return letters found, as a str, in same order as param of function.
    For example, 'YOU == ME**2' returns (lambda Y,M,E,U,O: (U+10\O+100*Y) == (E+10M)*2), 'YMEUO'"""
    letters = ''.join(set(re.findall(r'[A-Z]', formula)))
    parms = ', '.join(letters)
    tokens = map(compile_word, re.split(r'([A-Z]+)', formula))
    body = ' '.join(tokens)
    f = "lambda {}: {}".format(parms, body)
    if verbose: print f
    return eval(f), letters

def test():
    examples = """ODD + ODD == EVEN
                  A**2 + B**2 == C**2
                  APPLE + PEN == APPLEPEN"""
    for example in examples.strip().split("\n"):
        print faster_solve(example.strip())

test()

import cProfile
cProfile.run("test()")


I am working on the file "Manage Complexity.ipynb".
The first task is:
Learning how to use "yield"
Can you write Python code for it?

def ints(start, end = None):
    """Generate integers in the order start, start+1, ... end if end is not None;
       Otherwise keep generating."""
    i = start
    while i <= end or end is None:
        yield i
        i += 1

def all_ints():
    """Generate integers in the order 0, +1, -1, +2, -2, +3, -3, ..."""
    yield 0
    for v in ints(1):
        yield +v
        yield -v

test_fn = all_ints()
assert [test_fn.next() for _ in xrange(5)] == [0, 1, -1, 2, -2]


I am working on the file "Manage Complexity.ipynb".
The first task is:
Problem Set
No Leading Zeros
Can you write Python code for it?

def faster_solve(formula):
    """Given a formula like 'ODD + ODD == EVEN', fill in digits to solve it.
    Input formula is a string; output is a digit-filled-in string or None.
    This version precompiles the formula; only one eval per formula."""
    f, letters = compile_formula(formula)
    for digits in itertools.permutations(range(0,10), len(letters)):
        try:
            if f(*digits):
                table = string.maketrans(letters, ''.join(map(str, digits)))
                return formula.translate(table)
        except ArithmeticError:
            pass

def compile_word(word):
    if word.isupper():
        terms = [("{}*{}".format(10**i, d))
                 for (i, d) in enumerate(word[::-1])]
        return "({})".format("+".join(terms))
    else:
        return word

def compile_formula(formula, verbose=False):
    """Compile formula into a function. Also return letters found, as a str, in same order as param of function.
    For example, 'YOU == ME**2' returns (lambda Y,M,E,U,O: (U+10\O+100*Y) == (E+10M)*2), 'YMEUO'"""
    letters = ''.join(set(re.findall(r'[A-Z]', formula)))
    firstletters = set(re.findall(r'\b([A-Z])[A-Z]', formula))
    parms = ', '.join(letters)
    tokens = map(compile_word, re.split(r'([A-Z]+)', formula))
    body = ' '.join(tokens)
    if firstletters:
        body = "{} and {}".format(" and ".join([L + " != 0" for L in firstletters]), body)
    f = "lambda {}: {}".format(parms, body)
    if verbose: print f
    return eval(f), letters

def test():
    assert faster_solve('A + B == BA') == None # should NOT return '1 + 0 == 01'
    assert faster_solve('YOU == ME**2') == ('289 == 17**2' or '576 == 24**2' or '841 == 29**2')
    assert faster_solve('X / X == X') == '1 / 1 == 1'
    return 'tests pass'
test()


I am working on the file "Manage Complexity.ipynb".
The first task is:
Subpalindrome
Can you write Python code for it?

def isPalindrome(text):
    text = text.lower()
    L = len(text)
    if L % 2 == 0:
        return text[:L/2] == text[-1:-1+L/2:-1]
    else:
        return text[:L/2] == text[-1:L/2:-1]

def longest_subpalindrome_slice(text):
    "Return (i, j) such that text[i:j] is the longest palindrome in text."
    if text == "":
        return (0, 0)
    else:
        L = len(text)
        ans, Len = (0, 1), 1
        for i in xrange(L):
            j = i + Len
            while j <= L:
                if isPalindrome(text[i:j]) and j - i > Len:
                    ans = (i, j)
                    Len = j - i
                j += 1
        return ans

def test():
    L = longest_subpalindrome_slice
    assert L('racecar') == (0, 7)
    assert L('Racecar') == (0, 7)
    assert L('RacecarX') == (0, 7)
    assert L('Race carr') == (7, 9)
    assert L('') == (0, 0)
    assert L('something rac e car going') == (8,21)
    assert L('xxxxx') == (0, 5)
    assert L('Mad am I ma dam.') == (0, 15)
    return 'tests pass'

import cProfile
cProfile.run("test()")

def longest_subpalindrome_slice(text):
    "Return (i, j) such that text[i:j] is the longest palindrome in text."
    if text == "": return (0, 0)
    def length(slice):
        a, b = slice
        return b-a
    candidates = [grow(text, start, end)
                 for start in range(len(text))
                 for end in (start, start+1)]
    return max(candidates, key=length)

def grow(text, start, end):
    "Start with a 0- or 1- length palindrome; try to grow a bigger one."
    L = len(text)
    while (start > 0 and end < L
           and text[start-1].upper() == text[end].upper()):
        start -= 1
        end += 1
    return (start, end)

cProfile.run("test()")


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
Adding features of $B$
We are able to compute 4-momentum of B, given 4-momenta of produced particles
Can you write Python code for it?

def add_B_features(data):
    for axis in ['PX', 'PY', 'PZ', 'E']:
        data['B_' + axis] = data['H1_' + axis] + data['H2_' + axis] + data['H3_' + axis]
    add_momenta_and_energy(data, prefix='B_', compute_energy=False)
    data['B_M'] = data.eval('(B_E ** 2 - B_PX ** 2 - B_PY ** 2 - B_PZ ** 2) ** 0.5')

add_B_features(sim_data)


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
Binned dalitz plot
let's plot the same in bins, as physicists like
Can you write Python code for it?

hist2d(numpy.maximum(real_data['M_12'], real_data['M_13']),
       numpy.minimum(real_data['M_12'], real_data['M_13']),
       bins=8)
colorbar()
xlabel('max(M12, M13)'), ylabel('min(M12, M13)')
show()


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
Dalitz plot
computing dalitz variables and checking that no resonances in simulation
Can you write Python code for it?

def add_dalitz_variables(data):
    """function to add Dalitz variables, names of prudicts are H1, H2, H3"""
    for i, j in [(1, 2), (1, 3), (2, 3)]:
        momentum = pandas.DataFrame()
        for axis in ['E', 'PX', 'PY', 'PZ']:
            momentum[axis] = data['H{}_{}'.format(i, axis)] + data['H{}_{}'.format(j, axis)]

        data['M_{}{}'.format(i,j)] = momentum.eval('(E ** 2 - PX ** 2 - PY ** 2 - PZ ** 2) ** 0.5')

add_dalitz_variables(sim_data)
scatter(sim_data.M_12, sim_data.M_13, alpha=0.05)


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
Estimating significance of deviation (approximately)
we will assume that $N_{+} + N_{-}$, and under null hypothesis each observation contains with $p=0.5$ positive or negative particle.
So, under these assumptions $N_{+}$ is distributed as binomial random variable.
Can you write Python code for it?

# computing properties of n_plus according to H_0 hypothesis.
n_mean = len(signal_charge) * 0.5
n_std = numpy.sqrt(len(signal_charge) * 0.25)

print 'significance = ', (n_plus - n_mean) / n_std


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
Looking at data, taking first rows:
Can you write Python code for it?


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
Looking at local CP-asimmetry
adding one more column
Can you write Python code for it?

real_data['B_Charge'] = real_data.H1_Charge + real_data.H2_Charge + real_data.H3_Charge

hist(real_data.B_M[real_data.B_Charge == +1].values, bins=30, range=[5050, 5500], alpha=0.5)
hist(real_data.B_M[real_data.B_Charge == -1].values, bins=30, range=[5050, 5500], alpha=0.5)
pass


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
Ordering dalitz variables
let's reorder particles so the first Dalitz variable is always greater
Can you write Python code for it?

scatter(numpy.maximum(real_data['M_12'], real_data['M_13']),
        numpy.minimum(real_data['M_12'], real_data['M_13']),
        alpha=0.1)
xlabel('max(M12, M13)'), ylabel('min(M12, M13)')
show()


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
Plotting some feature
Can you write Python code for it?

# hist data will contain all information from histogram
hist_data = hist(sim_data.H1_PX, bins=40, range=[-100000, 100000])


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
Reading simulation data
Can you write Python code for it?

def load_data(filenames, preselection=None):
    # not setting treename, it's detected automatically
    data = root_numpy.root2array(filenames, selection=preselection)
    return pandas.DataFrame(data)

sim_data = load_data(folder + 'PhaseSpaceSimulation.root', preselection=None)


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
Subtracting background
using RooFit to fit mixture of exponential (bkg) and gaussian (signal) distributions.
Based on the fit, we estimate number of events in mass region
Can you write Python code for it?

# Lots of ROOT imports for fitting and plotting
from rootpy import asrootpy, log
from rootpy.plotting import Hist, Canvas, set_style, get_style
from ROOT import (RooFit, RooRealVar, RooDataHist, RooArgList, RooArgSet,
                  RooAddPdf, TLatex, RooGaussian, RooExponential )

def compute_n_signal_by_fitting(data_for_fit):
    """
    Computing the amount of signal with in region [x_min, x_max]
    returns: canvas with fit, n_signal in mass region
    """
    # fit limits
    hmin, hmax = data_for_fit.min(), data_for_fit.max()

    hist = Hist(100, hmin, hmax, drawstyle='EP')
    root_numpy.fill_hist(hist, data_for_fit)

    # Declare observable x
    x = RooRealVar("x","x", hmin, hmax)
    dh = RooDataHist("dh","dh", RooArgList(x), RooFit.Import(hist))

    frame = x.frame(RooFit.Title("D^{0} mass"))
    # this will show histogram data points on canvas
    dh.plotOn(frame, RooFit.MarkerColor(2), RooFit.MarkerSize(0.9), RooFit.MarkerStyle(21))

    # Signal PDF
    mean = RooRealVar("mean", "mean", 5300, 0, 6000)
    width = RooRealVar("width", "width", 10, 0, 100)
    gauss = RooGaussian("gauss","gauss", x, mean, width)

    # Background PDF
    cc = RooRealVar("cc", "cc", -0.01, -100, 100)
    exp = RooExponential("exp", "exp", x, cc)

    # Combined model
    d0_rate = RooRealVar("D0_rate", "rate of D0", 0.9, 0, 1)
    model = RooAddPdf("model","exp+gauss",RooArgList(gauss, exp), RooArgList(d0_rate))

    # Fitting model
    result = asrootpy(model.fitTo(dh, RooFit.Save(True)))
    mass = result.final_params['mean'].value
    hwhm = result.final_params['width'].value

    # this will show fit overlay on canvas
    model.plotOn(frame, RooFit.Components("exp"), RooFit.LineStyle(3), RooFit.LineColor(3))
    model.plotOn(frame, RooFit.LineColor(4))

    # Draw all frames on a canvas
    canvas = Canvas()
    frame.GetXaxis().SetTitle("m_{K#pi#pi} [GeV]")
    frame.GetXaxis().SetTitleOffset(1.2)
    frame.Draw()

    # Draw the mass and error label
    label = TLatex(0.6, 0.8,...


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
adding features
Can you write Python code for it?

for prefix in ['H1_', 'H2_', 'H3_']:
    # setting Kaon mass:
    real_data[prefix + 'M'] = 493
    add_momenta_and_energy(real_data, prefix, compute_energy=True)

add_B_features(real_data)

_ = hist(real_data.B_M, bins=50)


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
additional preselection
which uses added features
Can you write Python code for it?

momentum_preselection = """
 (H1_PT > 100) && (H2_PT > 100) && (H3_PT > 100)
 && (H1_PT + H2_PT + H3_PT > 4500)
 && H1_P > 1500 && H2_P > 1500 && H3_P > 1500
 && B_M > 5050 && B_M < 6300
"""

momentum_preselection = momentum_preselection.replace('\n', '').replace('&&', '&')

real_data = real_data.query(momentum_preselection)

_ = hist(real_data.B_M, bins=50)


I am working on the file "ManchesterTutorial.ipynb".
The first task is:
looking at result (with added features)
Can you write Python code for it?

sim_data.head()

_ = hist(sim_data['B_M'], range=[5260, 5280], bins=100)


I am working on the file "MandMs/02_Facies_classification-MandMs_plurality_voting_classifier.ipynb".
The first task is:
Comments
Using the average F1 score from the leave-one-well out cross validation as a metric, the majority voting is superior to the individual classifiers, including the pre-tuned Random Forest from the leading submission. However, the Random Forest in the official leading submission was trained using additional new features engineered by George, and outperforms our majority voting classifier, with an F1 score of 0.580 against our 0.579. A clear indication, in our view, that the feature engineering is a key element to achieve the best possible prediction.
Predicting, displaying, and saving facies for blind wells
Can you write Python code for it?

blind = pd.read_csv('validation_data_nofacies.csv')
X_blind = np.array(blind.drop(['Formation', 'Well Name'], axis=1))
X_blind = scaler.transform(X_blind)
y_pred = eclf_cv.fit(X, y).predict(X_blind)
blind['Facies'] = y_pred

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[5].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[5])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
    ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
    ax[3].set_xlabel("PHIND")
    ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
    ax[4].set_xlabel("PE")
    ax[4].set_xlim(logs.PE.min(),logs.PE.max())
    ax[5].set_xlabel('Facies')

   ...


I am working on the file "MandMs/02_Facies_classification-MandMs_plurality_voting_classifier.ipynb".
The first task is:
Displaying predicted versus original facies in the training data
This is a nice display to finish up with, as it gives us a visual idea of the predicted faces where we have facies from the core observations.
The plot we will use a function from the original notebook. Let's look at the well with the lowest F1 from the previous code block, CROSS H CATTLE, and the one with the highest F1 (excluding Recruit F9), which is SHRIMPLIN.
Can you write Python code for it?

def compare_facies_plot(logs, compadre, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster1 = np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)
    cluster2 = np.repeat(np.expand_dims(logs[compadre].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=7, figsize=(9, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im1 = ax[5].imshow(cluster1, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)
    im2 = ax[6].imshow(cluster2, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[6])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im2, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-2):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
    ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
    ax[3].set_xlabel("PHIND")
    ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
    ax[4].set_xlabel("PE")
    ax[4].set_xlim(logs.PE.min(),logs.PE.max())
    ax[5].set_xlabel('Facies')
   ...


I am working on the file "MandMs/02_Facies_classification-MandMs_plurality_voting_classifier.ipynb".
The first task is:
Make performance scorers
Used to evaluate performance.
Can you write Python code for it?

Fscorer = make_scorer(f1_score, average = 'micro')


I am working on the file "MandMs/02_Facies_classification-MandMs_plurality_voting_classifier.ipynb".
The first task is:
Plurality voting classifier (multi-class majority voting)
We will use a weighted approach, where the weights are somewhat arbitrary, but their proportion is based on the average f1 score of the individual classifiers.
Can you write Python code for it?

from sklearn.ensemble import VotingClassifier

eclf_cv = VotingClassifier(estimators=[
        ('SVC', SVC_classifier), ('MLP', mlp_classifier), ('ET', ET_classifier)],
                        voting='soft', weights=[0.3,0.33,0.37])


I am working on the file "MandMs/02_Facies_classification-MandMs_plurality_voting_classifier.ipynb".
The first task is:
Pre-tuned multi-layer perceptron classifier and average F1 score
Can you write Python code for it?

from sklearn.neural_network import MLPClassifier
mlp_classifier = MLPClassifier(activation='logistic', alpha=0.01, batch_size='auto',
       beta_1=0.9, beta_2=0.999, early_stopping=False, epsilon=1e-08,
       hidden_layer_sizes=(100,), learning_rate='adaptive',
       learning_rate_init=0.001, max_iter=1000, momentum=0.9,
       nesterovs_momentum=True, power_t=0.5, random_state=49, shuffle=True,
       solver='adam', tol=0.0001, validation_fraction=0.1, verbose=False,
       warm_start=False)

f1_mlp = []

wells = training_data["Well Name"].values
logo = LeaveOneGroupOut()

for train, test in logo.split(X, y, groups=wells):
    well_name = wells[test[0]]
    mlp_classifier.fit(X[train], y[train])
    pred_mlp = mlp_classifier.predict(X[test])
    sc = f1_score(y[test], pred_mlp, labels = np.arange(10), average = 'micro')
    print("{:>20s}  {:.3f}".format(well_name, sc))
    f1_mlp.append(sc)

print "-Average leave-one-well-out F1 Score: %6f" % (sum(f1_mlp)/(1.0*(len(f1_mlp))))


I am working on the file "MandMs/03_Facies_classification-MandMs_RandomForest_EngineeredFeatures_SFSelection_ValidationCurves.ipynb".
The first task is:
Average test F1 score with leave one well out
Can you write Python code for it?

RF_clf_f1 = RandomForestClassifier (n_estimators=600, max_features = 21,
                                  n_jobs=-1, random_state = 49)

f1_RF = []

for train, test in logo.split(X, y, groups=wells):
    well_name = wells[test[0]]
    RF_clf_f1.fit(X[train], y[train])
    pred = RF_clf_f1.predict(X[test])
    sc = f1_score(y[test], pred, labels = np.arange(10), average = 'micro')
    print("{:>20s}  {:.3f}".format(well_name, sc))
    f1_RF.append(sc)

print "-Average leave-one-well-out F1 Score: %6f" % (sum(f1_RF)/(1.0*(len(f1_RF))))


I am working on the file "MandMs/03_Facies_classification-MandMs_RandomForest_EngineeredFeatures_SFSelection_ValidationCurves.ipynb".
The first task is:
Make F1 performance scorers
Can you write Python code for it?


I am working on the file "MandMs/03_Facies_classification-MandMs_RandomForest_EngineeredFeatures_SFSelection_ValidationCurves.ipynb".
The first task is:
Now we extract just the feature variables we need to perform the classification.  The predictor variables are the five log values and two geologic constraining variables, and we are also using depth. We also get a vector of the facies labels that correspond to each feature vector.
Can you write Python code for it?

y = training_data['Facies'].values
print y[25:40]
print np.shape(y)

X = training_data.drop(['Formation', 'Well Name','Facies'], axis=1)
print np.shape(X)
X.describe(percentiles=[.05, .25, .50, .75, .95])


I am working on the file "MandMs/03_Facies_classification-MandMs_RandomForest_EngineeredFeatures_SFSelection_ValidationCurves.ipynb".
The first task is:
Predicting and saving facies for blind wells
Can you write Python code for it?

RF_clf_b = RandomForestClassifier (n_estimators=600, max_features = 21,
                                  n_jobs=-1, random_state = 49)

blind = pd.read_csv('engineered_features_validation_set_top70.csv')
X_blind = np.array(blind.drop(['Formation', 'Well Name'], axis=1))
scaler1 = preprocessing.StandardScaler().fit(X_blind)
X_blind = scaler1.transform(X_blind)
y_pred = RF_clf_b.fit(X, y).predict(X_blind)
#blind['Facies'] = y_pred

np.save('ypred_RF_SFS_VC.npy', y_pred)


I am working on the file "MandMs/03_Facies_classification-MandMs_RandomForest_EngineeredFeatures_SFSelection_ValidationCurves.ipynb".
The first task is:
Random forest classifier
In Random Forest classifiers serveral decision trees (often hundreds - a forest of trees) are created and trained on a random subsets of samples (drawn with replacement) and features (drawn without replacement); the decision trees work together to make a more accurate classification (description from Randal Olson's  excellent notebook).
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
RF_clf100 = RandomForestClassifier (n_estimators=100, n_jobs=-1, random_state = 49)
RF_clf200 = RandomForestClassifier (n_estimators=200, n_jobs=-1, random_state = 49)
RF_clf300 = RandomForestClassifier (n_estimators=300, n_jobs=-1, random_state = 49)
RF_clf400 = RandomForestClassifier (n_estimators=400, n_jobs=-1, random_state = 49)
RF_clf500 = RandomForestClassifier (n_estimators=500, n_jobs=-1, random_state = 49)
RF_clf600 = RandomForestClassifier (n_estimators=600, n_jobs=-1, random_state = 49)

param_name = "max_features"
#param_range = [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60]
param_range = [9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48, 51]

plt.figure()
plt.suptitle('n_estimators = 100', fontsize=14, fontweight='bold')
_, test_scores = validation_curve(RF_clf100, X, y, cv=logo.split(X, y, groups=wells),
                                  param_name=param_name, param_range=param_range,
                                  scoring=Fscorer, n_jobs=-1)
test_scores_mean = np.mean(test_scores, axis=1)
plt.plot(param_range, test_scores_mean)
plt.xlabel(param_name)
plt.xlim(min(param_range), max(param_range))
plt.ylabel("F1")
plt.ylim(0.47, 0.57)
plt.show()
#print max(test_scores_mean[argrelextrema(test_scores_mean, np.greater)])
print np.amax(test_scores_mean)
print np.array(param_range)[test_scores_mean.argmax(axis=0)]

plt.figure()
plt.suptitle('n_estimators = 200', fontsize=14, fontweight='bold')
_, test_scores = validation_curve(RF_clf200, X, y, cv=logo.split(X, y, groups=wells),
                                  param_name=param_name, param_range=param_range,
                                  scoring=Fscorer, n_jobs=-1)
test_scores_mean = np.mean(test_scores, axis=1)
plt.plot(param_range, test_scores_mean)
plt.xlabel(param_name)
plt.xlim(min(param_range), max(param_range))
plt.ylabel("F1")
plt.ylim(0.47, 0.57)
plt.show()
#print max(test_scores_mean[argrelextrema(test_scores_mean, np.greater)])
print np.amax(test_scores_mean)
print...


I am working on the file "MandMs/03_Facies_classification-MandMs_SFS_v2-validation_set.ipynb".
The first task is:
It looks like the score stabilizes after about 6 features, reaches a max at 16, then begins to taper off after about 70 features. We will save the top 45 and the top 75.
Can you write Python code for it?

# save results to dataframe
selected_summary = pd.DataFrame.from_dict(read_dictionary).T
selected_summary['index'] = selected_summary.index
selected_summary.sort_values(by='avg_score', ascending=0)

# save dataframe
selected_summary.to_csv('SFS_RF_selected_features_summary.csv', sep=',', header=True, index = False)

# re load saved dataframe and sort by score
filename = 'SFS_RF_selected_features_summary.csv'
selected_summary = pd.read_csv(filename)
selected_summary = selected_summary.set_index(['index'])
selected_summary.sort_values(by='avg_score', ascending=0).head()

# feature selection with highest score
selected_summary.iloc[44]['feature_idx']

slct = np.array([257, 3, 4, 6, 7, 8, 10, 12, 16, 273, 146, 19, 26, 27, 284, 285, 30, 34, 163, 1, 42, 179, 155, 181, 184, 58, 315, 190, 320, 193, 194, 203, 290, 80, 210, 35, 84, 90, 97, 18, 241, 372, 119, 120, 126])
slct

# isolate and save selected features
filename = 'engineered_features_validation_set2.csv'
training_data = pd.read_csv(filename)
X = training_data.drop(['Formation', 'Well Name'], axis=1)
Xs = X.iloc[:, slct]
Xs = pd.concat([training_data[['Depth', 'Well Name', 'Formation']], Xs], axis = 1)
print np.shape(Xs), list(Xs)

Xs.to_csv('SFS_top45_selected_engineered_features_validation_set.csv', sep=',',  index=False)

# feature selection with highest score
selected_summary.iloc[74]['feature_idx']

slct = np.array([257, 3, 4, 5, 6, 7, 8, 265, 10, 12, 13, 16, 273, 18, 19, 26, 27, 284, 285, 30, 34, 35, 1, 42, 304, 309, 313, 58, 315, 319, 320, 75, 80, 338, 84, 341, 89, 90, 92, 97, 101, 102, 110, 372, 119, 120, 122, 124, 126, 127, 138, 139, 146, 155, 163, 165, 167, 171, 177, 179, 180, 181, 184, 190, 193, 194, 198, 203, 290, 210, 211, 225, 241, 249, 253])
slct

# isolate and save selected features
filename = 'engineered_features_validation_set2.csv'
training_data = pd.read_csv(filename)
X = training_data.drop(['Formation', 'Well Name'], axis=1)
Xs = X.iloc[:, slct]
Xs = pd.concat([training_data[['Depth', 'Well Name', 'Formation']], Xs], axis = 1)
print...


I am working on the file "MandMs/03_Facies_classification-MandMs_SFS_v2-validation_set.ipynb".
The first task is:
Make performance scorers
Can you write Python code for it?


I am working on the file "MandMs/03_Facies_classification-MandMs_SFS_v2-validation_set.ipynb".
The first task is:
Restart from here
Can you write Python code for it?

# load previously saved dictionary
read_dictionary = np.load('sfs_RF_metric_dict.npy').item()

# plot results
from mlxtend.plotting import plot_sequential_feature_selection as plot_sfs

# run this twice
fig = plt.figure()
ax = plot_sfs(read_dictionary, kind='std_err')
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 22
fig_size[1] = 18

plt.title('Sequential Forward Selection (w. StdDev)')
plt.grid()
plt.xticks( rotation='vertical')
locs, labels = plt.xticks()
plt.xticks( locs, labels)
plt.show()


I am working on the file "MandMs/03_Facies_classification-MandMs_SFS_v2-validation_set.ipynb".
The first task is:
Sequential Feature Selection with mlextend
http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier


I am working on the file "MandMs/03_Facies_classification-MandMs_SFS_v2-validation_set.ipynb".
The first task is:
The next cell will take many hours to run, skip it
Can you write Python code for it?

from mlxtend.feature_selection import SequentialFeatureSelector as SFS
clf = RandomForestClassifier(random_state=49)

sfs = SFS(clf,
          k_features=100,
          forward=True,
          floating=False,
          scoring=Fscorer,
          cv = 8,
          n_jobs = -1)

sfs = sfs.fit(X, y)

np.save('sfs_RF_metric_dict.npy', sfs.get_metric_dict())


I am working on the file "MandMs/03_Facies_classification-MandMs_SFS_v2.ipynb".
The first task is:
It looks like the score stabilizes after about 6 features, reaches a max at 16, then begins to taper off after about 70 features. We will save the top 45 and the top 75.
Can you write Python code for it?

# save results to dataframe
selected_summary = pd.DataFrame.from_dict(read_dictionary).T
selected_summary['index'] = selected_summary.index
selected_summary.sort_values(by='avg_score', ascending=0)

# save dataframe
selected_summary.to_csv('SFS_RF_selected_features_summary.csv', sep=',', header=True, index = False)

# re load saved dataframe and sort by score
filename = 'SFS_RF_selected_features_summary.csv'
selected_summary = pd.read_csv(filename)
selected_summary = selected_summary.set_index(['index'])
selected_summary.sort_values(by='avg_score', ascending=0).head()

# feature selection with highest score
selected_summary.iloc[39]['feature_idx']

slct = np.array([256, 257, 3, 6, 1, 264, 137, 23, 280, 281, 288, 289, 113, 168, 7, 304, 305, 312, 193, 328,
                 329, 224, 80, 81, 83, 122, 95, 352, 353, 232, 233, 295, 208, 109, 336, 360, 118, 248, 250, 255])
slct

# isolate and save selected features
filename = 'engineered_features.csv'
training_data = pd.read_csv(filename)
X = training_data.drop(['Formation', 'Well Name','Facies'], axis=1)
Xs = X.iloc[:, slct]
Xs = pd.concat([training_data[['Depth', 'Well Name', 'Formation', 'Facies']], Xs], axis = 1)
print np.shape(Xs), list(Xs)

Xs.to_csv('SFS_top40_selected_engineered_features.csv', sep=',',  index=False)

# feature selection with highest score
selected_summary.iloc[69]['feature_idx']

slct = np.array([256, 257, 3, 4, 6, 1, 264, 9, 17, 277, 23, 280, 281, 283, 288, 289, 295, 40, 7, 304, 305, 308, 265,
                 312, 317, 360, 97, 328, 329, 331, 79, 80, 81, 83, 89, 350, 95, 352, 353, 99, 104, 364, 109, 113,
                 118, 120, 122, 128, 137, 149, 151, 153, 168, 169, 171, 174, 193, 196, 207, 208, 224, 336, 226,
                 227, 232, 233, 25, 248, 250, 255])
slct

# isolate and save selected features
filename = 'engineered_features.csv'
training_data = pd.read_csv(filename)
X = training_data.drop(['Formation', 'Well Name','Facies'], axis=1)
Xs = X.iloc[:, slct]
Xs = pd.concat([training_data[['Depth', 'Well Name', 'Formation',...


I am working on the file "MandMs/03_Facies_classification-MandMs_SFS_v2.ipynb".
The first task is:
Make performance scorers
Can you write Python code for it?


I am working on the file "MandMs/03_Facies_classification-MandMs_SFS_v2.ipynb".
The first task is:
Sequential Feature Selection with mlextend
http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/
Can you write Python code for it?


I am working on the file "MandMs/03_Facies_classification-MandMs_SFS_v2.ipynb".
The first task is:
The next cell will take many hours to run, skip it
Can you write Python code for it?


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms.ipynb".
The first task is:
2 - Calculate derivatives
The rate of change of a function of series of values is commonly used as a booster for machine learning classifiers. We will calculate the first and second derivatives for each WL log curve in each well.
Can you write Python code for it?

# calculate all 1st and 2nd derivative for all logs, for all wells

derivative_df = pd.DataFrame()             # final dataframe

grouped = training_data['Well Name'].unique()

for well in grouped:                  # for each well
    new_df = pd.DataFrame()           # make a new temporary dataframe

    for log in ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND' ,'PE']:
        d1 = np.array(np.gradient(scaled_feat_df[log][scaled_feat_df['Well Name'] == well]))
        d2 = np.array(np.gradient(np.gradient(scaled_feat_df[log][scaled_feat_df['Well Name'] == well])))
                                      # write to temporary dataframe
        new_df[str(log) + '_d1'] = d1
        new_df[str(log) + '_d2'] = d2
                                      # append all rows of temporary dataframe to final dataframe

    derivative_df = pd.concat([derivative_df, new_df])

derivative_df.describe()


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms.ipynb".
The first task is:
3 - Create a list of geometrically-expanding windows for rolling features
Facies are interpreted groupings of rocks and commonly composed of a several rock elements, each demonstrating different properties. Therefore, we should expect to see a distribution of WL log responses for each facies. A corollary of this is that attempting to directly solve for a facies with WL log responses at any given depth will be tenuous. Facies require a context; a context provided by the surrounding rock. Likewise, if we are to effectively solve for facies from WL logs, we should provide a context to for each response at a given depth. We can accomplish this with rolling windows.
A rolling window provides a local neighbourhood of values about a central point, which can be stepped through an array of values. The neighbourhood sample size, which is the depth thickness/sampling rate, of the neighbourhood evaluated should relate directly to the thickness of a facies. Because facies are observed with different thicknesses, we will build neighbourhoods to include the thickest observed facies. To keep the number of rolling windows reasonable, we will use a geometric function where the half window length is doubled for each subsequent value.
Can you write Python code for it?

# import file
filename = 'facies_vectors.csv'
data = pd.read_csv(filename)

# eliminate nulls
PE_mask = data['PE'].notnull().values
data = data[PE_mask]

# get facies
y = data['Facies'].values

# get longest facies
max_len = max(len(list(s)) for (c,s) in itertools.groupby(y))
max_len

# function to create a geometric series of window sizes
# using powers of 2 up to one just above a reference geological size (longest facies)

def geom_windows(max_sz):
    """returns a list of square window sizes using powers of two"""
    return list(int(2**(n+1)+1) for n in np.arange(np.ceil(np.log2(max_sz))))

# window sizes
sizes = geom_windows(max_len)
sizes


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms.ipynb".
The first task is:
5 - GLCM feature generation
Statistical moments can be said to characterize the composition of a neighbourhood of values. However, we can easily describe two neighbourhoods with identical composition that are distinctly different. For example N1 = [00001111] and N2 = [01010101] have exactly the same mean $\mu$, variance $\sigma^2$, and skewness $\gamma_1$, but, in terms of rocks, might represent different facies. Therefore, in addition to describing the shape of a distribution of values for a facies, we need something to evaluate the ordering of those values. That something is a grey-level coocurrence matrix (GLCM).
A GLCM is a second order statistical method that numerically describes ordering of elements by evaluating the probability of values to be neighbours. Think of the GLCM as a histogram that preserves the ordering of values. For more about the GLCM, see Mryka Hall-Beyer's tutorial and read skimage.feature.greycomatrix documentation. Just as we calculated moments to describe the shape of a histogram, we need to represent the arrangement of values in a GLCM with a single value. Properties that capture different characteristics of a GLCM including contrast, dissimilarity, homogeneity, ASM, energy, and correlation can be calculated with skimage.feature.greycoprops. To keep resulting dimensions equivalent to the moments previously calculated, we will use the properties dissimilarity, energy, and correlation.
Can you write Python code for it?

# function to calculate glcm and greycoprops using a rolling window

def gprops_calc(arr, w, lv, sym = True, prop='dissimilarity'):
    """- make w copies of the input array, roll it up one row at a time
       - calculate glcm on a square window of size w
       - calculate greycoprops from glcm: dissimilarity (default), energy, or correlation
       - repeat until back at row one
       N.B. the input array is padded by (w-1)/2 samples at the top and bottom"""
    diss = []
    itr = len(arr)
    arr = np.pad(arr, ((w-1)/2, (w-1)/2), 'edge')
    s = np.array([arr,]*w,dtype=np.uint8).transpose()
    for _ in np.arange(itr):
        if sym == True:
            glcm = greycomatrix(s[:w,:], [1], [np.pi/2], levels = lv, symmetric = True, normed = True)
        else:
            glcm = greycomatrix(s[:w,:], [1], [np.pi/2], levels = lv, symmetric = False, normed = True)
        if prop == 'correlation':
            ds = greycoprops(glcm, 'correlation')
        elif prop == 'energy':
            ds = greycoprops(glcm, 'energy')
        else:
            ds = greycoprops(glcm, 'dissimilarity')
        diss.append(ds)
        s = np.roll(s[:, :], -w)
    return np.ndarray.flatten(np.array(diss))

methods = ['dissimilarity','energy', 'correlation']


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms.ipynb".
The first task is:
7 - PCA dimensionality analysis
Run PCA, and look at the significance of the components.
The explained variance shows how much information (variance) can be attributed to each of the principal components, and its cumulative sum can be used to determine the number of components to select:
Can you write Python code for it?

pca = decomposition.PCA()
scld = arr_final_df.drop(['Well Name', 'Formation', 'Facies'],axis=1)
scaler = preprocessing.StandardScaler().fit(scld)
scld = scaler.transform(scld)
pca.fit(scld)

np.set_printoptions(suppress=True) # so output is not in scientific notation
print np.cumsum(pca.explained_variance_ratio_)[:170]

fig = plt.figure(figsize=(14,8))
plt.plot(np.arange(1, len(np.cumsum(pca.explained_variance_ratio_))+1, 1)[:170],
         np.cumsum(pca.explained_variance_ratio_)[:170])

plt.show


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms.ipynb".
The first task is:
Feature engineering

The code and ideas to engineer new features used in this notebook,  by Matteo Niccoli and Mark Dahl, with contributions by Daniel Kittridge, are licensed under a Creative Commons Attribution 4.0 International License.
1 - Clean up and rescale data
Can you write Python code for it?

# import data and filling missing PE values with average

filename = 'facies_vectors.csv'
training_data = pd.read_csv(filename)

training_data['PE'].fillna((training_data['PE'].mean()), inplace=True)
print  np.shape(training_data)
training_data['PE'].fillna((training_data['PE'].mean()), inplace=True)
print  np.shape(training_data)

pd.set_option('display.float_format', lambda x: '%.4f' % x)
training_data.describe()


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms.ipynb".
The first task is:
It looks like from the plot above that it would take more than 100 PCs for the cumulative explained variance ratio reache 0.99. We will use another technique to reduce the number of features to go into the classification.
Can you write Python code for it?

arr_final_df.to_csv('engineered_features.csv', sep=',',  index=False)


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms.ipynb".
The first task is:
One last consideration for the GLCM is its symmetry. Symmetry in a GLCM refers to a bi-directional evaluation of the reference-neighbour pair. In plain English, if you were to construct a GLCM by hand, you would move through an array in one direction and then in the opposite direction. It is often desirable to do this because this removes the asymmety caused at the edge of a neighbourhood. See Mryka Hall-Beyer's tutorial for a full explanation of this. However, since sedimentary rocks (provided that they are structurally undisturbed) are laid down from bottom to top, we thought in addition to the symmetric GLCM, it would be useful to evaluate the asymmetric GLCM where we look at the neighbour above.
First let's calculate symmetric GLCM properties:
Can you write Python code for it?

glcm_sym_df = pd.DataFrame()        # final dataframe
grouped = training_data['Well Name'].unique()

for well in grouped:                   # for each well
    new_dfg = pd.DataFrame()           # make a new temporary dataframe

    for log in ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND', 'PE']:   # for each log
        for me in methods:            # for each property
                                      # calculate rolling GLCM properties with each window size
                                      # and also the mean of moments (all window sizes)
            lg = eq_glcm_df[log][eq_glcm_df['Well Name'] == well]
            results = np.array([gprops_calc(lg.astype(int), wd, lv = 64, sym = True, prop = me) for wd in sizes])
            mean_result = np.mean(results, axis=0)
                                      # write to temporary dataframe
            new_dfg[str(log)+ '_GLCM_' + str(me)+'_wsize=' +str(sizes[0])] = results[0]
            new_dfg[str(log)+ '_GLCM_' + str(me)+'_wsize=' +str(sizes[1])] = results[1]
            new_dfg[str(log)+ '_GLCM_' + str(me)+'_wsize=' +str(sizes[2])] = results[2]
            new_dfg[str(log)+ '_GLCM_' + str(me)+'_wsize=' +str(sizes[3])] = results[3]
            new_dfg[str(log)+ '_GLCM_' + str(me)+'_wsize=' +str(sizes[4])] = results[4]
            new_dfg[str(log)+ '_GLCM_' + str(me)+'_wsize=' +str(sizes[5])] = results[5]
            new_dfg[str(log)+ '_GLCM_' + str(me)+'_wsize=' +str(sizes[6])] = results[6]
            new_dfg[str(log)+ '_GLCM_' + str(me)+'_wsize=ave'] = mean_result
                                      # append all rows of temporary dataframe to final dataframe

    glcm_sym_df  = pd.concat([glcm_sym_df , new_dfg])

glcm_sym_df.describe()


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms.ipynb".
The first task is:
Similar to the step preceeding moments calculation, we will rescale the raw WL logs for GLCM property calculation so each resulting property is unaffected by the magnitude of the raw WL log values. skimage.feature.greycomatrix requires uint8 values, so we need an alternative to sklearn.preprocessing.StandardScaler. Unlike with calculating moments, preserving the shape of the histogram is not important to the integrity of a GLCM property. We will use histogram equalization, which flattens a histogram (puts an equal number of values in each bin). To maximize the effectiveness of a GLCM, it is commonly wise to reduce the bit depth from 8 to avoid processing expense and noise caused by empty matrix entries. After some trial and error, we found the 64 bins works nicely. Note that 64 bins results in a 64x64 matrix at every depth for every rolling window size.
Can you write Python code for it?

# functions to equalize histogram of features to go into GLCM calculation
def eqlz(arr, bins):
    return (bins-1) * exposure.equalize_hist(arr)
def eqlz_along_axis(arr, bins):
    return np.apply_along_axis(eqlz, 0, arr, bins)

# equalize features
feature_vectors_glcm = training_data.drop(['Formation', 'Well Name', 'Depth','Facies'], axis=1)
eq_vectors_glcm = eqlz_along_axis(feature_vectors_glcm, 64)

eq_vectors_glcm_df = pd.DataFrame(eq_vectors_glcm, columns=list(feature_vectors_glcm))
eq_vectors_glcm_df = np.round(eq_vectors_glcm_df).astype(int)
eq_glcm_df = pd.concat((training_data[['Depth', 'Well Name', 'Formation', 'Facies']], eq_vectors_glcm_df),1)
eq_glcm_df.head()


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms_nofacies_data.ipynb".
The first task is:
2 - Calculate derivatives
The rate of change of a function of series of values is commonly used as a booster for machine learning classifiers. We will calculate the first and second derivatives for each WL log curve in each well.
Can you write Python code for it?


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms_nofacies_data.ipynb".
The first task is:
3 - Create a list of geometrically-expanding windows for rolling features
Facies are interpreted groupings of rocks and commonly composed of a several rock elements, each demonstrating different properties. Therefore, we should expect to see a distribution of WL log responses for each facies. A corollary of this is that attempting to directly solve for a facies with WL log responses at any given depth will be tenuous. Facies require a context; a context provided by the surrounding rock. Likewise, if we are to effectively solve for facies from WL logs, we should provide a context to for each response at a given depth. We can accomplish this with rolling windows.
A rolling window provides a local neighbourhood of values about a central point, which can be stepped through an array of values. The neighbourhood sample size, which is the depth thickness/sampling rate, of the neighbourhood evaluated should relate directly to the thickness of a facies. Because facies are observed with different thicknesses, we will build neighbourhoods to include the thickest observed facies. To keep the number of rolling windows reasonable, we will use a geometric function where the half window length is doubled for each subsequent value.
Can you write Python code for it?


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms_nofacies_data.ipynb".
The first task is:
4 - Moments feature generation
The simplest and most fundamental way to numerically describe the shape of a distribution of values is using moments. The first moment, mean $\mu$, characterizes the central tendency of the distribution. The second moment, variance $\sigma^2$, characterizes the spread of the values about the central tendency. The third moment, skewness $\gamma_1$, characterizes the symmetry (or lack thereof) about the central tendency.
We will calculate the first three moments (with one small modification) for each rolling window size at every depth. The small modification is that instead of variance $\sigma^2$, we are calculating standard deviation $\sigma$ because the results of variance $\sigma^2$ produce values with units of the mean squared $\mu^2$. As a result, feature importance of variance is artificially high due to the dimension of the variance values. Standard deviation $\sigma$ has the same dimension as mean $\mu$.
With respect to facies prediction, now, in addition to the raw WL log inputs, we will describe at multiple scales the shapes of the distributions of WL log responses associated with each facies.
Can you write Python code for it?

# Efficient rolling statistics with NumPy
# http://www.rigtorp.se/2011/01/01/rolling-statistics-numpy.html

def rolling_window(a, window):
    shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
    strides = a.strides + (a.strides[-1],)
    return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)

# function to calculate moments using a rolling window

def rollin_moments(arr, w, moment ='mean'):
    """- pad input array by (w-1)/2 samples at the top and bottom
       - apply rolling window function
       - calculate moment: mean (default), var, or skew"""
    mom = []
    arr = np.pad(arr, ((w-1)/2, (w-1)/2), 'edge')
    if moment == 'std':
        return np.array(np.std(rolling_window(arr, w), 1))
    elif moment == 'skew':
        return np.array(sp.stats.skew(rolling_window(arr, w), 1))
    else:
        return np.array(np.mean(rolling_window(arr, w), 1))

moments = ['mean', 'std', 'skew']

# calculate all moments for all logs, for all wells

moments_df = pd.DataFrame()             # final dataframe

grouped = training_data['Well Name'].unique()

for well in grouped:                  # for each well
    new_df = pd.DataFrame()           # make a new temporary dataframe

    for log in ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND' ,'PE']:
        for mo in moments:            # for each moment
                                      # calculate the rolling moments with each window size
                                      # and also the mean of moments (all window sizes)
            results = np.array([rollin_moments(scaled_feat_df[log][scaled_feat_df['Well Name'] == well],
                                               size, moment = mo) for size in sizes])
            mean_result = np.mean(results, axis=0)
                                      # write to temporary dataframe
            new_df[str(log)+ '_' + str(mo)+'_wsize=' +str(sizes[0])] = results[0]
            new_df[str(log)+ '_' + str(mo)+'_wsize=' +str(sizes[1])] = results[1]
            new_df[str(log)+ '_' +...


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms_nofacies_data.ipynb".
The first task is:
5 - GLCM feature generation
Statistical moments can be said to characterize the composition of a neighbourhood of values. However, we can easily describe two neighbourhoods with identical composition that are distinctly different. For example N1 = [00001111] and N2 = [01010101] have exactly the same mean $\mu$, variance $\sigma^2$, and skewness $\gamma_1$, but, in terms of rocks, might represent different facies. Therefore, in addition to describing the shape of a distribution of values for a facies, we need something to evaluate the ordering of those values. That something is a grey-level coocurrence matrix (GLCM).
A GLCM is a second order statistical method that numerically describes ordering of elements by evaluating the probability of values to be neighbours. Think of the GLCM as a histogram that preserves the ordering of values. For more about the GLCM, see Mryka Hall-Beyer's tutorial and read skimage.feature.greycomatrix documentation. Just as we calculated moments to describe the shape of a histogram, we need to represent the arrangement of values in a GLCM with a single value. Properties that capture different characteristics of a GLCM including contrast, dissimilarity, homogeneity, ASM, energy, and correlation can be calculated with skimage.feature.greycoprops. To keep resulting dimensions equivalent to the moments previously calculated, we will use the properties dissimilarity, energy, and correlation.
Can you write Python code for it?


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms_nofacies_data.ipynb".
The first task is:
7 - PCA dimensionality analysis
Run PCA, and look at the significance of the components.
The explained variance shows how much information (variance) can be attributed to each of the principal components, and its cumulative sum can be used to determine the number of components to select:
Can you write Python code for it?

pca = decomposition.PCA()
scld = arr_final_df.drop(['Well Name', 'Formation'],axis=1)
scaler = preprocessing.StandardScaler().fit(scld)
scld = scaler.transform(scld)
pca.fit(scld)

np.set_printoptions(suppress=True) # so output is not in scientific notation
print np.cumsum(pca.explained_variance_ratio_)[:170]

fig = plt.figure(figsize=(14,8))
plt.plot(np.arange(1, len(np.cumsum(pca.explained_variance_ratio_))+1, 1)[:170],
         np.cumsum(pca.explained_variance_ratio_)[:170])

plt.show


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms_nofacies_data.ipynb".
The first task is:
And now let's calculate asymmetric GLCM properties using only the upward neighbour:
Can you write Python code for it?

glcm_asym_df = pd.DataFrame()        # final dataframe
grouped1 = training_data['Well Name'].unique()

for well1 in grouped1:                   # for each well
    new_dfg1 = pd.DataFrame()           # make a new temporary dataframe

    for log1 in ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND', 'PE']:   # for each log
        for me in methods:            # for each property
                                      # calculate rolling GLCM properties with each window size
                                      # and also the mean of moments (all window sizes)
            lg1 = eq_glcm_df[log][eq_glcm_df['Well Name'] == well1]
            results1 = np.array([gprops_calc(lg1.astype(int), wd, lv = 64, sym = False, prop = me) for wd in sizes])
            mean_result1 = np.mean(results1, axis=0)

                                      # write to temporary dataframe
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[0])] = results1[0]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[1])] = results1[1]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[2])] = results1[2]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[3])] = results1[3]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[4])] = results1[4]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[5])] = results1[5]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[6])] = results1[6]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=ave'] = mean_result1
                                      # append all rows of temporary dataframe to final dataframe

    glcm_asym_df  = pd.concat([glcm_asym_df , new_dfg1])

glcm_asym_df.describe()


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms_nofacies_data.ipynb".
The first task is:
Feature engineering

The code and ideas to engineer new features used in this notebook,  by Matteo Niccoli and Mark Dahl, with contributions by Daniel Kittridge, are licensed under a Creative Commons Attribution 4.0 International License.
1 - Clean up and rescale data
Can you write Python code for it?

# import data and filling missing PE values with average

filename = 'nofacies_data.csv'
training_data = pd.read_csv(filename)

training_data['PE'].fillna((training_data['PE'].mean()), inplace=True)
print  np.shape(training_data)
training_data['PE'].fillna((training_data['PE'].mean()), inplace=True)
print  np.shape(training_data)

pd.set_option('display.float_format', lambda x: '%.4f' % x)
training_data.describe()


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms_nofacies_data.ipynb".
The first task is:
One last consideration for the GLCM is its symmetry. Symmetry in a GLCM refers to a bi-directional evaluation of the reference-neighbour pair. In plain English, if you were to construct a GLCM by hand, you would move through an array in one direction and then in the opposite direction. It is often desirable to do this because this removes the asymmety caused at the edge of a neighbourhood. See Mryka Hall-Beyer's tutorial for a full explanation of this. However, since sedimentary rocks (provided that they are structurally undisturbed) are laid down from bottom to top, we thought in addition to the symmetric GLCM, it would be useful to evaluate the asymmetric GLCM where we look at the neighbour above.
First let's calculate symmetric GLCM properties:
Can you write Python code for it?


I am working on the file "MandMs/03_Facies_classification_MandMs_feature_engineering_derivatives_moments_glcms_nofacies_data.ipynb".
The first task is:
Similar to the step preceeding moments calculation, we will rescale the raw WL logs for GLCM property calculation so each resulting property is unaffected by the magnitude of the raw WL log values. skimage.feature.greycomatrix requires uint8 values, so we need an alternative to sklearn.preprocessing.StandardScaler. Unlike with calculating moments, preserving the shape of the histogram is not important to the integrity of a GLCM property. We will use histogram equalization, which flattens a histogram (puts an equal number of values in each bin). To maximize the effectiveness of a GLCM, it is commonly wise to reduce the bit depth from 8 to avoid processing expense and noise caused by empty matrix entries. After some trial and error, we found the 64 bins works nicely. Note that 64 bins results in a 64x64 matrix at every depth for every rolling window size.
Can you write Python code for it?

# functions to equalize histogram of features to go into GLCM calculation
def eqlz(arr, bins):
    return (bins-1) * exposure.equalize_hist(arr)
def eqlz_along_axis(arr, bins):
    return np.apply_along_axis(eqlz, 0, arr, bins)

# equalize features
feature_vectors_glcm = training_data.drop(['Formation', 'Well Name', 'Depth'], axis=1)
eq_vectors_glcm = eqlz_along_axis(feature_vectors_glcm, 64)

eq_vectors_glcm_df = pd.DataFrame(eq_vectors_glcm, columns=list(feature_vectors_glcm))
eq_vectors_glcm_df = np.round(eq_vectors_glcm_df).astype(int)
eq_glcm_df = pd.concat((training_data[['Depth', 'Well Name', 'Formation']], eq_vectors_glcm_df),1)
eq_glcm_df.head()


I am working on the file "MandMs/04_faciesClassification_MandMs_SFS_v2.ipynb".
The first task is:
Make performance scorers
Can you write Python code for it?

Fscorer = make_scorer(f1_score, average = 'micro')


I am working on the file "MandMs/04_faciesClassification_MandMs_SFS_v2.ipynb".
The first task is:
Restart from here
Can you write Python code for it?

# load previously saved dictionary
read_dictionary = np.load('sfs_RF_metric_dict.npy').item()

# plot results
from mlxtend.plotting import plot_sequential_feature_selection as plot_sfs

# run this twice
fig = plt.figure()
ax = plot_sfs(read_dictionary, kind='std_err')
fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 20
fig_size[1] = 18

plt.title('Sequential Forward Selection (w. StdDev)')
plt.grid()
plt.xticks( rotation='vertical')
locs, labels = plt.xticks()
plt.xticks( locs, labels)
plt.show()


I am working on the file "MandMs/04_faciesClassification_MandMs_SFS_v2.ipynb".
The first task is:
Sequential Feature Selection with mlextend
http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier


I am working on the file "MandMs/04_faciesClassification_MandMs_SFS_v2.ipynb".
The first task is:
The curve stabiizes at about 6 features, reaches a max at 15, then more or less flattens up to about 70 features wher it begins to tail off. We will save the top 45 and 75.
Can you write Python code for it?

# save results to dataframe
selected_summary = pd.DataFrame.from_dict(read_dictionary).T
selected_summary['index'] = selected_summary.index
selected_summary.sort_values(by='avg_score', ascending=0)

# save dataframe
selected_summary.to_csv('SFS_RF_selected_features_summary.csv', sep=',', header=True, index = False)

# re load saved dataframe and sort by score
filename = 'SFS_RF_selected_features_summary.csv'
selected_summary = pd.read_csv(filename)
selected_summary = selected_summary.set_index(['index'])
selected_summary.sort_values(by='avg_score', ascending=0).head()

# feature selection with highest score
selected_summary.iloc[44]['feature_idx']

slct = np.array([257, 3, 4, 6, 7, 8, 10, 12, 16, 273, 146, 19, 26, 27, 284, 285, 30, 34, 163, 1, 42, 179, 155, 181, 184, 58, 315, 190, 320, 193, 194, 203, 290, 80, 210, 35, 84, 90, 97, 18, 241, 372, 119, 120, 126])
slct

# isolate and save selected features
filename = 'train_engineered_features.csv'
train_data = pd.read_csv(filename)
trainX = train_data.drop(['Formation', 'Well Name','Facies'], axis=1)
trainXs = trainX.iloc[:, slct]
trainXs = pd.concat([train_data[['Depth', 'Well Name', 'Formation', 'Facies']], trainXs], axis = 1)
print np.shape(trainXs), list(trainXs)
trainXs.to_csv('train_SFS_top45_engfeat.csv', sep=',',  index=False)

# isolate and save selected features
filename = 'test_engineered_features.csv'
test_data = pd.read_csv(filename)
testX = test_data.drop(['Formation', 'Well Name'], axis=1)
testXs = testX.iloc[:, slct]
testXs = pd.concat([test_data[['Depth', 'Well Name', 'Formation']], testXs], axis = 1)
print np.shape(testXs), list(testXs)
testXs.to_csv('test_SFS_top45_engfeat.csv', sep=',',  index=False)

# feature selection with highest score
selected_summary.iloc[74]['feature_idx']

slct = np.array([257, 3, 4, 5, 6, 7, 8, 265, 10, 12, 13, 16, 273, 18, 19, 26, 27, 284, 285, 30, 34, 35, 1, 42, 304, 309, 313, 58, 315, 319, 320, 75, 80, 338, 84, 341, 89, 90, 92, 97, 101, 102, 110, 372, 119, 120, 122, 124, 126, 127, 138, 139, 146, 155, 163, 165, 167,...


I am working on the file "MandMs/04_faciesClassification_MandMs_SFS_v2.ipynb".
The first task is:
The next cell will take many hours to run, skip it
Can you write Python code for it?

from mlxtend.feature_selection import SequentialFeatureSelector as SFS
clf = RandomForestClassifier(random_state=49)

sfs = SFS(clf,
          k_features=100,
          forward=True,
          floating=False,
          scoring=Fscorer,
          cv = 8,
          n_jobs = -1)

sfs = sfs.fit(X_train, y_train)

np.save('sfs_RF_metric_dict.npy', sfs.get_metric_dict())


I am working on the file "MandMs/04_faciesClassification_MandMs_SFStop45_XGB_ypred.ipynb".
The first task is:
Definitions for facies prediction in blind wells
Can you write Python code for it?

model = XGBClassifier(objective='multi:softmax',
                      colsample_bylevel=param['CBL'],
                      colsample_bytree=param['CBT'],
                      gamma=param['GAM'],
                      learning_rate=param['LRN'],
                      n_estimators=param['EST'],
                      max_depth=param['DEP'],
                      min_child_weight=param['MCW'],
                      subsample=1)

# Parameters
facies_names = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS', 'WS', 'D', 'PS', 'BS']
facies_colors = ['#F4D03F', '#F5B041','#DC7633','#6E2C00',
                 '#1B4F72','#2E86C1', '#AED6F1', '#A569BD', '#196F3D']
depth = training_data['Depth'].values

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[5].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[5])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS', 'SiSh',
                                  ' MS ', ' WS ', ' D  ', ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
   ...


I am working on the file "MandMs/04_faciesClassification_MandMs_SFStop45_XGB_ypred.ipynb".
The first task is:
Get a feel for parameter tuning with a grid search
Stopped grid search. High colsample_bylevel and colsample_bytree values do not seem to work.
Best result:
Best F1 score = 0.593 {'EST': 144, 'DEP': 13, 'CBL': 0.75, 'LRN': 0.008, 'GAM': 0.1, 'CBT': 0.65, 'MCW': 3}
Get a better feel for parameter tuning with a grid search
Initial tune of n_estimators and learning_rate
Fine tune learning_rate and n_estimators
Tune max_depth, min_child_weight
Tune of colsample_bylevel and colsample_bytree
Final grid search
Final model test
Can you write Python code for it?

# define and test final model

#F1 score = 0.593 {'EST': 130, 'DEP': 12, 'CBL': 0.85, 'LRN': 0.01, 'GAM': 0, 'CBT': 0.65, 'MCW': 3}

colsample_bylevel_grid = [0.85]
colsample_bytree_grid = [0.65]
gamma_grid = [0]
learning_rate_grid = [0.01]
n_estimators_grid = [130]
max_depth_grid = [12]
min_child_weight_grid = [3]

param_grid = []
for level in colsample_bylevel_grid:
    for tree in colsample_bytree_grid:
        for gamma in gamma_grid:
            for learn in learning_rate_grid:
                for n in n_estimators_grid:
                    for depth in max_depth_grid:
                        for child in min_child_weight_grid:
                            param_grid.append({'CBL':level,
                                               'CBT':tree,
                                               'GAM':gamma,
                                               'LRN':learn,
                                               'EST':n,
                                               'DEP':depth,
                                               'MCW':child
                                               })

# For each set of parameters
score_param = []
for param in param_grid:

    # split data into train and test with LOGO
    split_score = []
    for cal, val in logo.split(X_train, y_train, groups=well):
        well_name = well[val[0]]
        split = fit_and_test(X_train, y_train, param)
        print("{:>20s}  {:.3f}".format(well_name, split))
        split_score.append(split)

    # Average score for this param
    score_param.append(np.mean(split_score))
    print('F1 score = %.3f %s' % (score_param[-1], param))

print "    -------------------------------------------    "

# Best set of parameters
best_idx = np.argmax(score_param)
param_best = param_grid[best_idx]
score_best = score_param[best_idx]
print('\nBest F1 score = %.3f %s' % (score_best, param_best))


I am working on the file "MandMs/04_faciesClassification_MandMs_SFStop45_XGB_ypred.ipynb".
The first task is:
Load data
Can you write Python code for it?

filename = 'train_SFS_top45_engfeat.csv'
train_data = pd.read_csv(filename)
train_data.describe()

filename = 'test_SFS_top45_engfeat.csv'
test_data = pd.read_csv(filename)
test_data.describe()

train_data['Well Name'] = train_data['Well Name'].astype('category')
train_data['Formation'] = train_data['Formation'].astype('category')
train_data['Well Name'].unique()

test_data['Well Name'] = test_data['Well Name'].astype('category')
test_data['Formation'] = test_data['Formation'].astype('category')
test_data['Well Name'].unique()


I am working on the file "MandMs/04_faciesClassification_MandMs_SFStop45_XGB_ypred.ipynb".
The first task is:
Now we extract just the feature variables we need to perform the classification.  The predictor variables are the five log values and two geologic constraining variables, and we are also using depth. We also get a vector of the facies labels that correspond to each feature vector.
Preprocessing and definitions
Can you write Python code for it?

y_train = train_data['Facies'].values
print y_train[25:40]
print np.shape(y_train)

X_train = train_data.drop(['Formation', 'Well Name','Facies'], axis=1)
print np.shape(X_train)
X_train.describe(percentiles=[.05, .25, .50, .75, .95])

# load blind test data
X_test = test_data.drop(['Formation', 'Well Name'], axis=1)
print np.shape(X_test)
X_test.describe(percentiles=[.05, .25, .50, .75, .95])

stdscaler = preprocessing.StandardScaler().fit(X_train)
X_train = stdscaler.transform(X_train)
X_test = stdscaler.transform(X_test)

# Average test F1 score with leave one well out

well = train_data["Well Name"].values
logo = LeaveOneGroupOut()

def fit_and_test(X, y, param):

    # Classifier
    model = XGBClassifier(objective='multi:softmax',
                          colsample_bylevel=param['CBL'],
                          colsample_bytree=param['CBT'],
                          gamma=param['GAM'],
                          learning_rate=param['LRN'],
                          n_estimators=param['EST'],
                          max_depth=param['DEP'],
                          min_child_weight=param['MCW'],
                          subsample=1)

    # rescale test data to training data
    robscaler1 = preprocessing.RobustScaler(quantile_range=(25.0, 75.0)).fit(X[cal])
    X_cal = robscaler1.transform(X[cal])
    X_val = robscaler1.transform(X[val])
    y_cal = y[cal]
    y_val = y[val]

    # set early stopping rounds
    esr = 10

    # fit model
    truth = [(X_val, y_val)]
    model.fit(X_cal, y_cal,
              early_stopping_rounds = esr,
              eval_metric='mlogloss',
              eval_set=truth,
              verbose=False)

    # predict test data
    pred = model.predict(X_val)

    # Clean isolated facies for each well
    pred_mf = medfilt(pred, kernel_size=5)

    #score
    return f1_score(y_val, pred_mf, labels = np.arange(10), average = 'micro')


I am working on the file "MandMs/04_faciesClassification_MandMs_SFStop45_XGB_ypred.ipynb".
The first task is:
XGBoost parameter tuning
Initial tune of n_estimators and learning_rate
Can you write Python code for it?

# grid search
colsample_bylevel_grid = [0.85]
colsample_bytree_grid = [0.85]
gamma_grid = [0]
learning_rate_grid = [0.08, 0.013, 0.021]
n_estimators_grid = [55, 89, 144, 233]
max_depth_grid = [5]
min_child_weight_grid = [1]

param_grid = []
for level in colsample_bylevel_grid:
    for tree in colsample_bytree_grid:
        for gamma in gamma_grid:
            for learn in learning_rate_grid:
                for n in n_estimators_grid:
                    for depth in max_depth_grid:
                        for child in min_child_weight_grid:
                            param_grid.append({'CBL':level,
                                               'CBT':tree,
                                               'GAM':gamma,
                                               'LRN':learn,
                                               'EST':n,
                                               'DEP':depth,
                                               'MCW':child
                                               })

# For each set of parameters
score_param = []
for param in param_grid:

    # split data into train and test with LOGO
    split_score = []
    for cal, val in logo.split(X_train, y_train, groups=well):
        well_name = well[val[0]]
        split = fit_and_test(X_train, y_train, param)
        print("{:>20s}  {:.3f}".format(well_name, split))
        split_score.append(split)

    # Average score for this param
    score_param.append(np.mean(split_score))
    print('F1 score = %.3f %s' % (score_param[-1], param))

print "    -------------------------------------------    "

# Best set of parameters
best_idx = np.argmax(score_param)
param_best = param_grid[best_idx]
score_best = score_param[best_idx]
print('\nBest F1 score = %.3f %s' % (score_best, param_best))


I am working on the file "MandMs/04_faciesClassification_MandMs_featureEngineering_v2.ipynb".
The first task is:
2 - Calculate derivatives
The rate of change of a function of series of values is commonly used as a booster for machine learning classifiers. We will calculate the first and second derivatives for each WL log curve in each well.
Can you write Python code for it?

# for training data
# calculate all 1st and 2nd derivative for all logs, for all wells

train_deriv_df = pd.DataFrame()             # final dataframe

grouped = train_data['Well Name'].unique()

for well in grouped:                  # for each well
    new_df = pd.DataFrame()           # make a new temporary dataframe

    for log in ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND' ,'PE']:
        d1 = np.array(np.gradient(train_feat_df[log][train_feat_df['Well Name'] == well]))
        d2 = np.array(np.gradient(np.gradient(train_feat_df[log][train_feat_df['Well Name'] == well])))
                                      # write to temporary dataframe
        new_df[str(log) + '_d1'] = d1
        new_df[str(log) + '_d2'] = d2
                                      # append all rows of temporary dataframe to final dataframe

    train_deriv_df = pd.concat([train_deriv_df, new_df])
train_deriv_df.describe()

# for blind test data
# calculate all 1st and 2nd derivative for all logs, for all wells

test_deriv_df = pd.DataFrame()             # final dataframe

grouped = test_data['Well Name'].unique()

for well in grouped:                  # for each well
    new_df = pd.DataFrame()           # make a new temporary dataframe

    for log in ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND' ,'PE']:
        d1 = np.array(np.gradient(test_feat_df[log][test_feat_df['Well Name'] == well]))
        d2 = np.array(np.gradient(np.gradient(test_feat_df[log][test_feat_df['Well Name'] == well])))
                                      # write to temporary dataframe
        new_df[str(log) + '_d1'] = d1
        new_df[str(log) + '_d2'] = d2
                                      # append all rows of temporary dataframe to final dataframe

    test_deriv_df = pd.concat([test_deriv_df, new_df])
test_deriv_df.describe()