I am working on the file "Beer_User_Analyze.ipynb".
The first task is:
Explore Binary Classifier
Even though no clear linear correlation is apparent to us based on our analysis so far of the select features, we thought it'd be worthwhile to test different linear classification methods with our selected feature (especially considering the potential non-linear correlation we've noticed). By doing so, we would be able to evaluate how predictive our set of features really is (based on an actual model evaluation accuracy score).
Specifically, we were interested in whether our identified list of features could be used to predict whether a user's review contribution would increase or decrease followiing the change in the website. As we were attempting to create a positive/negative classifier, we first created a balanced pool of data to work from -- where there were an equal number of positive and negative samples (where positive/negative is determined by the user's review count plus-minus).  Following this, we split the data pool 80/20 into a training and test set. We chose to perform cross_validation (n=4) to evaluate different models and used a standard mean/standard deviation scaling to normalize our feature values.
For the purposes of this assignment, we looked at 4 different sklearn classifiers: rbf SVM, linear SVM, dual-l2 logistic regression, non-dual-l2 logistic regression. From a cross-validation score, the model which performed best was the rbf SVM algorithm.
Overall, this resulted in a 62% accuracy (consistent with the output from the other classifiers) -- a score that is not very good and not that much better than random. This leads us to believe that the features we selected are not strongly predictive of future review contributions to the RateBeer website.
Can you write Python code for it?


I am working on the file "Beer_User_Analyze.ipynb".
The first task is:
While performing the linear correlation analysis, we noticed several interesting visually-identifiable, non-linear trends when plotting the following variables against the # of post-split review count:

Number of Pre-Split Styles Reviewed
Average Pre-Split Rating
Average Review Length

Specifically we see the following:

There does seem to be a positive correlation between reviewed number of styles (before) and number of reviews (after)
Most users in our dataset give ratings around 0.7 (dense) - there is also a peak number of post-split reviews at this average
Most users in our dataset wrote reviews around 50 words in length (dense) - there is also a peak number of post-split reviews at this average
Users who wrote extremely long/short/positive/negative reviews have few post-split reviews (both graphs taper at the ends)
Can you write Python code for it?

_ = plt.plot(np_x[:,1], np_y_after, 'b.')
plt.ylim(0)
plt.xlabel('# of Pre-Split Styles Reviewed')
plt.ylabel('# of Post-Split Reviews')
plt.title('Correlation btwn Pre Styles Types and Post Review Count')

_= plt.plot(np_x[:,2], np_y_after, 'b.')
plt.ylim(0)
plt.xlabel('Average Pre-Split Overall Rating of Reviews')
plt.ylabel('# of Post-Split Reviews')
plt.title('Correlation btwn Pre Average Rating and Post Review Count')

_= plt.plot(np_x[:,4], np_y_after, 'b.')
plt.ylim(0)
plt.xlabel('Average Pre-Split Review Length (Word Count)')
plt.ylabel('# of Post-Split Reviews')
plt.title('Correlation btwn Average Review Length and Post Review Count')


I am working on the file "Beginners/notebook/Objects_Classes_Constructors_and_Functions.ipynb".
The first task is:
Accessing an object function that also takes a keyword argument:
Can you write Python code for it?

my_dog.dog_color(color = 'Pink')

my_dog.dog_color(color='Blue')


I am working on the file "Beginners/notebook/Objects_Classes_Constructors_and_Functions.ipynb".
The first task is:
Accessing object functions:

To access a function inside of an object in python, the notation similar to accessing a variable.
Can you write Python code for it?


I am working on the file "Beginners/notebook/Objects_Classes_Constructors_and_Functions.ipynb".
The first task is:
Another basic python function:

This is a function to add two numbers.
Uses keyword arguments instead of positional arguments.
The function takes two decimal values instead of two integer values in the function above.
Explicit error messages.
Can you write Python code for it?

def add_function(input_1=None, input_2=None):
  if input_1 and input_2 !=None:
    print ('Calculating the sum of: ' + str(input_1) + " , " + str(input_2))
    return (a+b)
  else:
    print ('Nothing to do here ...')
    print ('Ensure add_function passes two keyword variables: input_1 and input_2 ...')

a = float(input())
b = float(input())

result = add_function(input_2=b, input_1=a)

print (result)


I am working on the file "Beginners/notebook/Objects_Classes_Constructors_and_Functions.ipynb".
The first task is:
Assign an object to the class:

To assign the above class(template) to an object(variable), do the following:
Can you write Python code for it?


I am working on the file "Beginners/notebook/Objects_Classes_Constructors_and_Functions.ipynb".
The first task is:
Change the values of a variable in an object:
Can you write Python code for it?

my_dog.name = "Cute Puppy"

my_dog.my_dog()

my_dog.dog_color(color='Black')


I am working on the file "Beginners/notebook/Objects_Classes_Constructors_and_Functions.ipynb".
The first task is:
Explicit error message looks like this:
Can you write Python code for it?


I am working on the file "Beginners/notebook/Objects_Classes_Constructors_and_Functions.ipynb".
The first task is:
Explicitly calling garbage collection to free-up working memory:
Can you write Python code for it?


I am working on the file "Beginners/notebook/Objects_Classes_Constructors_and_Functions.ipynb".
The first task is:
Part 02 -- Classes in python:
Python is an “object-oriented programming language.” This means almost all the code is implemented using a special construct called classes. Programmers use classes to keep related things together. This is done using the keyword “class,” which is a grouping of object-oriented constructs.
Define a very basic class in python:

Contains a variable called "some_variable".
Contains a function called "some_function".
Can you write Python code for it?

class some_class:
    some_variable = "Some string inside python class."
    def some_function(self):
        return ("This is a message inside the class.")


I am working on the file "Beginners/notebook/Objects_Classes_Constructors_and_Functions.ipynb".
The first task is:
The input() helps user pass an input value.
Can you write Python code for it?

input_1 = int(input())
input_2 = int(input())

result = add_function(input_1, input_2)

print (result)


I am working on the file "Beginners/notebook/Pickle_tutorial.ipynb".
The first task is:
Create Pickle dump
Can you write Python code for it?


I am working on the file "Beginners/notebook/Pickle_tutorial.ipynb".
The first task is:
Create a dataframe with random integers between 0 and 1000
Can you write Python code for it?


I am working on the file "Beginners/notebook/Pickle_tutorial.ipynb".
The first task is:
Function to track memory utilization
Can you write Python code for it?

def memory_utilization():
  print('Current memory utilization: {}% ...'.format(psutil.virtual_memory().percent))


I am working on the file "Beginners/notebook/Pickle_tutorial.ipynb".
The first task is:
Remove the variable from memory
Can you write Python code for it?

del var
_=gc.collect()
memory_utilization()


I am working on the file "Beginners/notebook/Pickle_tutorial.ipynb".
The first task is:
Restore the variable from disk
Can you write Python code for it?


I am working on the file "Behavior Cloning/notebook/Behavior Cloning.ipynb".
The first task is:
Dataset
I used training dataset provided by Udacity. I use all 3 positions of camera with correction of 0.25 , i.e addition of 0.25 to steering angle for left-positioned camera and substraction of 0.25 for right-positioned camera.
I could have self-produced ore data but due to time constraint, I only used Udacity dataset
Moreover, after unable to complete a whole lap, I follow advice from forum and decided to randomly choose camera to select from
The dataset is split into 20% of test set. Also, the training set is shuffled before training
Loading Dataset
Can you write Python code for it?

'''Read data'''
image_path = '../../../data'
# row in log path is IMG/<name>
driving_log_path = '../../../data/driving_log.csv'

rows = []
with open(driving_log_path) as csvfile:
    reader = csv.reader(csvfile)
    for row in reader:
        rows.append(row)


I am working on the file "Behavior Cloning/notebook/Behavior Cloning.ipynb".
The first task is:
Model Architecture Definition
Can you write Python code for it?

model = Sequential()
#The cameras in the simulator capture 160 pixel by 320 pixel images., after cropping, it is 66x200
model.add(Cropping2D(cropping = ((74,20), (60,60)),input_shape=(160, 320, 3)))

model.add(Lambda(lambda x: x / 255.0 - 0.5, input_shape=(66, 200, 3)))
model.add(Convolution2D(24, 5, 5, subsample=(2,2), activation='relu'))
model.add(Dropout(.5))
model.add(Convolution2D(36, 5, 5, subsample=(2,2), activation='relu'))
model.add(Dropout(.5))
model.add(Convolution2D(48, 5, 5, subsample=(2,2), activation='relu'))
model.add(Dropout(.5))
model.add(Convolution2D(64, 3, 3, activation='relu'))
model.add(Convolution2D(64, 3, 3, activation='relu'))
model.add(Flatten())
model.add(Dense(100))
model.add(Dense(50))
model.add(Dense(10))
model.add(Dense(1))
'''Training: using MSE for regression'''
model.compile(loss='mse', optimizer='adam')


I am working on the file "Behavior Cloning/notebook/Behavior Cloning.ipynb".
The first task is:
Model Training
For every time of training and parameter tunning, the model was tested by running it through the simulator and ensuring that the vehicle could stay on the track.
At epoch of 10, the training and validation loss both went down fast and I think they would converge if I'd have increased number of epoch (graph is plotted in the notebook. However, for this kind of regression problem, both trainning loss and accuracy do not seem to be useful, it is more important to test it on the simulator provided by Udacity. Therefore, I simply tune number of epoch until the vehicle run well on the track.
Final Model parameters:
- Optimizer: Adam optimizer, so the learning rate was not tuned manually
- Epoch: 5
- Batch size: 32
Can you write Python code for it?

print('Training model')
samples = list(zip(X_total, y_total))
train_samples, validation_samples = train_test_split(samples, test_size = 0.2)
train_generator = generator(train_samples, batch_size = 32)
validation_generator = generator(validation_samples, batch_size = 32)

history_object = model.fit_generator(train_generator,
                                    samples_per_epoch = len(train_samples),
                                    validation_data = validation_generator,
                                    nb_val_samples = len(validation_samples),
                                    nb_epoch = 5,
                                    verbose = 1)
print('Endding training, starting to save model')
model.save('../model.h5')

print(history_object.history.keys())
###plot the training and validation loss for each epoch
plt.plot(history_object.history['loss'])
plt.plot(history_object.history['val_loss'])
plt.title('model mean squared error loss')
plt.ylabel('mean squared error loss')
plt.xlabel('epoch')
plt.legend(['training set', 'validation set'], loc='upper right')
plt.show()


I am working on the file "BehaviorDrivenDevWithLettuceAndFlask.ipynb".
The first task is:
Automated tests using lettuce
Can you write Python code for it?

%%bash
lettuce test/features/user.features
# Excuse the double printing. This does not happen on the CLI.


I am working on the file "BehaviorDrivenDevWithLettuceAndFlask.ipynb".
The first task is:
Behavior test implementation
User features define behaviors, but Python code is required to test the behaviors.  These are implemented in Python in file by convention named steps.py.
If one runs lettuce without implementing the step(s), lettuce will quite helpfully provide stubs that will execute but, at this point correctly, fail. Here is an example for the step And the user 'david01' is removed.
You can implement step definitions for undefined steps with these snippets:

# -*- coding: utf-8 -*-
from lettuce import step

@step(u'And the user \'([^\']*)\' is removed')
def and_the_user_group1_is_removed(step, group1):
    assert False, 'This step must be implemented'

Here are the behavior tests for user queries and deletes.
Can you write Python code for it?


I am working on the file "BehaviorDrivenDevWithLettuceAndFlask.ipynb".
The first task is:
Running the REST server manually with curl
GET
Can you write Python code for it?

%%bash
curl --silent localhost:5000/

%%bash
curl --silent localhost:5000/users

%%bash
curl --silent localhost:5000/users/zero00


I am working on the file "BellCurve.ipynb".
The first task is:
I know mu sounds like "mew", the sound a kitten makes, and that's sometimes insisted upon by sticklers, for when we have a continuous function, versus one that's discrete.  Statisticians make a big deal about the difference between digital and analog, where the former is seen as a "sampling" of the latter.  Complete data may be an impossibility.  We're always stuck with something digital trying to approximate something analog, or so it seems.  Turn that around in your head sometimes:  we smooth it over as an approximation, because a discrete treatment would require too high a level of precision.
The sticklers say "mu" for continuous, but "x-bar" (an x with a bar over it) for plain old "average" of discrete sets.  I don't see this conventions holding water necessarily, for one thing because it's inconvenient to always reach for the most fancy typography.  Python does have full access to Unicode, and to LaTex, but do we have to bother?  Lets leave that question for another day and move on to...
The Guassian (Binomial if Discrete)
Can you write Python code for it?

mu = 0  # might be x-bar if discrete
sigma = 1  # standard deviation, more below


I am working on the file "BellCurve.ipynb".
The first task is:
Integrating the Gaussian
Apparently there's no closed form, however sympy is able to do an integration somehow.
Can you write Python code for it?

from sympy import var, Lambda, integrate, sqrt, pi, exp, latex

fig = plt.gcf()
fig.set_size_inches(8,5)
var('a b x sigma mu')
pdf = Lambda((x,mu,sigma),
  (1/(sigma * sqrt(2*pi)) * exp(-(mu-x)**2 / (2*sigma**2)))
)
cdf = Lambda((a,b,mu,sigma),
  integrate(
    pdf(x,mu,sigma),(x,a,b)
  )
)
display(Latex('$ cdf(a,b,\mu,\sigma) = ' + latex(cdf(a,b,mu,sigma)) + '$'))


I am working on the file "BellCurve.ipynb".
The first task is:
Lets plot the integral of the Bell Curve.  This curve somewhat describes the temporal pattern whereby a new technology is adopted, first by early adopters, then comes the bandwagon effect, then come the stragglers.  Not the every technology gets adopted in this way.  Only some do.
Can you write Python code for it?


I am working on the file "BellCurve.ipynb".
The first task is:
Lets stop right here and note the pdf and cdf have been defined, using sympy's Lambda and integrate, and the cdf will be fed a lot of data, one hundred points, along with mu and sigma.  Then it's simply a matter of plotting.
What's amazing is our ability to get something from sympy that works to give a cdf, independently of scipy.stats.norm.
Can you write Python code for it?

x = np.linspace(50,159,100)
y = np.array([cdf(-1e99,v,100,15) for v in x],dtype='float')
plt.grid(True)
plt.title('Cumulative Distribution Function')
plt.xlabel('IQ')
print(type(plt.xlabel))
plt.ylabel('Y')
plt.text(65,.75,'$\mu = 100$',fontsize=16)
plt.text(65,.65,'$\sigma = 15$',fontsize=16)
plt.plot(x,y,color='gray')
plt.fill_between(x,y,0,color='#c0f0c0')
plt.show()


I am working on the file "BellCurve.ipynb".
The first task is:
The above is truly a testament to Python's power, or the Python ecosystem's power.  We've brought in sympy, able to do symbolic integration, and talk LaTeX at the same time.  That's impressive.  Here's the high IQ source for the original version of the above code.
There's no indefinite integral of the Gaussian, but there's a definite one.  sympy comes with its own generic sympy.stats.cdf function which produces Lambdas (symbolic expressions) when used to integrate different types of probability spaces, such as Normal (a continuous PDF).  It also accepts discrete PMFs as well.
<pre>
Examples
========

>>> from sympy.stats import density, Die, Normal, cdf
>>> from sympy import Symbol

>>> D = Die('D', 6)
>>> X = Normal('X', 0, 1)

>>> density(D).dict
{1: 1/6, 2: 1/6, 3: 1/6, 4: 1/6, 5: 1/6, 6: 1/6}
>>> cdf(D)
{1: 1/6, 2: 1/3, 3: 1/2, 4: 2/3, 5: 5/6, 6: 1}
>>> cdf(3*D, D > 2)
{9: 1/4, 12: 1/2, 15: 3/4, 18: 1}

>>> cdf(X)
Lambda(_z, -erfc(sqrt(2)*_z/2)/2 + 1)
</pre>

LAB:  convert the Normal Distribution Below to IQ Curve...
That means domain is 0-200, standard deviation 15, mean = 100.
Can you write Python code for it?

domain = np.linspace(0, 200, 3000)
IQ = st.norm.pdf(domain, 100, 15)
plt.plot(domain, IQ, color = "red")

domain = np.linspace(0, 200, 3000)
mu = 100
sigma = 15
IQ = vg(domain)
plt.plot(domain, IQ, color = "green")


I am working on the file "BellCurve.ipynb".
The first task is:
What I do below is semi-mysterious, and something I'd like to get to in numpy in more detail.  The whole idea behind numpy is every function, or at least the unary ones, are vectorized, meaning the work element-wise through every cell, with no need for any for loops.
My Gaussian formula above won't natively understand how to have relations with a numpy array, unless we store it in vectorized form.  I'm not claiming this will make it run any faster than under the control of for loops, we can test that.  Even without a speedup, here we have a recipe for shortening our code.
As many have proclaimed around numpy: one of its primary benefits is it allows one to "lose the loops".
Can you write Python code for it?


I am working on the file "BellCurve.ipynb".
The first task is:
What we have here (below) is a typical Python numeric function, although it does get its pi from numpy instead of math.  That won't matter.  The sigma and mu in this function are globals and set above.  Some LaTex would be in order here, I realize.  Let me scavange the internet for something appropriate...
$pdf(x,\mu,\sigma) = \frac{1}{ \sigma \sqrt{2 \pi}} e^{\left(-\frac{{\left(\mu - x\right)}^{2}}{2 \, \sigma^{2}}\right)}$
Use of dollar signs is key.
Here's another way, in a code cell instead of a Markdown cell.
Can you write Python code for it?

from IPython.display import display, Latex

ltx = '$ pdf(x,\\mu,\\sigma) = \\frac{1}{ \\sigma' + \
 '\\sqrt{2 \\pi}} e^{\\left(-\\frac{{\\left(\\mu - ' + \
 'x\\right)}^{2}}{2 \\, \\sigma^{2}}\\right)} $'
display(Latex(ltx))


I am working on the file "BellCurve.ipynb".
The first task is:
see Wikipedia figure
These are Gaussian PDFs or Probability Density Functions.
68.26% of values happen within -1 and 1.
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo("xgQhefFOXrM")

a = st.norm.cdf(-1)  #  Cumulative distribution function
b = st.norm.cdf(1)
b - a

a = st.norm.cdf(-2)
b = st.norm.cdf(2)
b - a

# 99.73% is more correct than 99.72%
a = st.norm.cdf(-3)
b = st.norm.cdf(3)
b - a

# 95%
a = st.norm.cdf(-1.96)
b = st.norm.cdf(1.96)
b - a

# 99%
a = st.norm.cdf(-2.58)
b = st.norm.cdf(2.58)
b - a

from IPython.display import YouTubeVideo
YouTubeVideo("zZWd56VlN7w")


I am working on the file "Benchmark Data.ipynb".
The first task is:
Benchmark Report (per Project)
Can you write Python code for it?

from source_code_analysis.models import SoftwareProject

projects = SoftwareProject.objects.all()


I am working on the file "Benchmark Data.ipynb".
The first task is:
Raw Data Report Structure
All the report files containing the raw data of the methods share exactly the same multiline structure.
That is (for each method):
method_id, method_name, class_name, software_system
filepath, start_line, end_line,
Length of the Head Comments
Head Comment
Length of the Implementation
Method Implementation
###
Can you write Python code for it?

# Write Raw Data Report
def write_raw_data_report(raw_report_filepath, target_methods):
    with open(raw_report_filepath, 'w') as raw_report:
        for method in target_methods:
            software_system_name = method.project.name + method.project.version.replace('.', '')
            raw_report.write('{mid}, {method_name}, {class_name}, {software_system}\n'.format(
                                    mid=method.id, method_name=method.method_name, class_name=method.code_class.class_name,
                                    software_system=software_system_name))

            method_fp = method.file_path
            relative_filepath = method_fp[method_fp.find('extracted')+len('extracted')+1:]
            raw_report.write('{filepath}, {start_line}, {end_line}\n'.format(filepath=relative_filepath,
                                                                             start_line=method.start_line,
                                                                             end_line=method.end_line))

            raw_report.write('{comment_len}\n'.format(comment_len=len(method.comment.splitlines())))
            raw_report.write('{comment}'.format(comment=method.comment))
            if not method.comment.endswith('\n'):
                raw_report.write('\n')

            raw_report.write('{code_len}\n'.format(code_len=len(method.code_fragment.splitlines())))
            raw_report.write('{code}'.format(code=method.code_fragment))
            if not method.code_fragment.endswith('\n'):
                raw_report.write('\n')

            # Last Line of this method
            raw_report.write('###\n')

RAW_DATA_SUFFIX = 'Raw_Data.txt'
COHERENCE_DATA_SUFFIX = 'Coherence_Data.txt'

import os

# Create Report Folder
report_folderpath = os.path.join(os.path.abspath(os.path.curdir), 'report_files')
if not os.path.exists(report_folderpath):
    os.makedirs(report_folderpath)

all_methods_list = list()

# Project-Specific Reports

for project in projects:
   ...


I am working on the file "Benchmark Data.ipynb".
The first task is:
The Replication Dataset contains the following report files:

(Benchmark_Coherence_Data.txt, Benchmark_Raw_Data.txt):
    Report files containing information about the Coherence and the raw data of methods from all the 4 considered
    Software Systems.
(CoffeeMaker_Coherence_Data.txt, CoffeeMaker_Raw_Data.txt):
    Report files providing the Coherence and the raw data of methods gathered from the CoffeeMaker Software System.
(JFreeChart060_Coherence_Data.txt, JFreeChart060_Raw_Data.txt):
    Report files providing the Coherence and the raw data of methods gathered from the JFreeChart 0.6.0 Software System.
(JFreeChart071_Coherence_Data.txt, JFreeChart071_Raw_Data.txt):
    Report files providing the Coherence and the raw data of methods gathered from the JFreeChart 0.7.1 Software System.
(JHotDraw741_Coherence_Data.txt, JHotDraw741_Raw_Data.txt):
    Report files providing the Coherence and the raw data of methods gathered from the JHotDraw 7.4.1 Software System.

Coherence Data Report Structure
Report files providing information about the Coherence of methods are structured according to the CSV
(i.e., Comma Separated Values) format.
Each line of the file contains the following information:
method_id, coherence

method_id: the unique identifier of the corresponding method
coherence : the coherence value associated to the comment and the implementation of the referred method.
    Allowed Coherence Values are: NOT_COHERENT and COHERENT.
    In case, it would be more than straightforward to translate these values into 0, 1 values, respectively.
Can you write Python code for it?

# Write Coherence Report
def write_coherence_report(coherence_report_filepath, target_methods):
    with open(coherence_report_filepath, 'w') as coherence_report:
        for method in target_methods:
            evaluation = method.agreement_evaluations.all()[0]
            coherence_value = 'COHERENT' if evaluation.agreement_vote in [3, 4] else 'NOT_COHERENT'
            coherence_report.write('{0}, {1}\n'.format(method.pk, coherence_value))


I am working on the file "Benchmark.ipynb".
The first task is:
As we can see, for this particular problem, PyMCEF is orders of magnitude faster than GLPK. And the solutions obtained by these two methods agree up to machine epsilon.
Can you write Python code for it?

%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
diffs = np.mean(np.abs(weights_pymcef - weights_glpk), axis=1)
plt.plot(diffs)
plt.yscale('log')
plt.xlabel('portfolio indices')
plt.ylabel('L1 norm of weight error')


I am working on the file "Benchmark.ipynb".
The first task is:
Then we solve it with GLPK:
Can you write Python code for it?

tic = time()
weights_glpk = Solve_With_GLPK(returns, lbds, t)
cost_glpk = time() - tic
print("Total time cost for GLPK: " + str(cost_glpk) + " s")


I am working on the file "Benchmark.ipynb".
The first task is:
Use all three packages to solve for more and more costy problem:
Can you write Python code for it?

Sims_all = [100*2**i for i in range(6)]
costs_clp = np.zeros(len(Sims_all))
costs_glkp = np.zeros(len(Sims_all))
costs_pymcef = np.zeros(len(Sims_all))

for j in range(len(Sims_all)):
    numSims = Sims_all[j]
    returns = mn(means, covs, numSims)

    # using PyMCEF
    tic = time()
    simple_ef = SimpleEF(target_return=t, training_set=returns.T,
                         risk_measure=RiskMeasure.FixTargetUnderPerformance)
    ports = simple_ef.frontier_in_sample
    weights_pymcef = np.zeros((len(ports), 3))
    lbds = []
    for i in range(len(ports)):
        ws = ports[i]['weight']
        lbd_l = ports[i]['lambda_l']
        lbd_u = ports[i]['lambda_u']
        lbds.append((lbd_l + lbd_u)/2.0 if i > 0 else 2.0*lbd_l)
        for k, v in ws.items():
            weights_pymcef[i][k] = v
    costs_pymcef[j] = time() - tic

    # using GlPK
    tic = time()
    weights_glpk = Solve_With_GLPK(returns, lbds, t)
    costs_glkp[j] = time() - tic

    #using ClP
    tic = time()
    weights_clp = Solve_With_CLP(returns, lbds, t)
    costs_clp[j] = time() - tic

numTotalReturns = len(means) * np.array(Sims_all)
plt.xscale('log')
plt.yscale('log')
plt.plot(numTotalReturns, costs_pymcef, 'k-o',label='PyMCEF')
plt.plot(numTotalReturns, costs_clp, 'b-o',label='CLP')
plt.plot(numTotalReturns, costs_glkp, 'g-o',label='GLPK')
plt.xlabel('Number of total simulated returns')
plt.ylabel('Total time in seconds')
plt.tick_params(axis='y', right='on', labelright='on')
plt.legend(loc='upper left')
plt.savefig('output/performance.png')


I am working on the file "Benchmark.ipynb".
The first task is:
We can conclude that the correctness and accuracy of PyMCEF is successfully validated by GLPK.
More on speed
CLP is another open source linear programming library, which is faster than GLPK.
We would like to compare the time costs of all the three packages, on the above problem with increasing number of simulated returns.
CyLP can use inequality constraints, let's reformulate the problem:
\begin{eqnarray}
\min_{u_{k},w_{n},v_{k}} & \frac{1}{K}\sum_{k=1}^{K}u_{k}-\lambda\sum_{n=1}^{N}w_{n}\overline{Y}{n}\
\mathrm{subject\ to} & \sum{n=1}^{N}w_{n} & =1\
 & u_{k}+\sum_{n=1}^{N}w_{n}Y_{n}^{k} & \ge t\
 & w_{n},u_{k} & \ge0.
\end{eqnarray}
Let's set up CLP to solve this problem.
Can you write Python code for it?

# prepare CLP solver via CyLP
import warnings
warnings.filterwarnings('ignore')
from cylp.cy import CyClpSimplex
from cylp.py.modeling.CyLPModel import CyLPArray
def Solve_With_CLP(returns, lbds, t):
    numSims = returns.shape[0]
    numAssets = returns.shape[1]
    smplx = CyClpSimplex()

    # Add variables
    w = smplx.addVariable('w', numAssets)
    u = smplx.addVariable('u', numSims)
    A = np.matrix(returns)
    B = np.matrix(np.eye(numSims, dtype='float64'))
    b = CyLPArray([t] * numSims)

    # Add constraints
    smplx += B * u + A * w >= b
    smplx += w.sum() == 1.0
    smplx += w >= 0.0
    smplx += u >= 0.0
    # Set the objective function
    c = CyLPArray(returns.mean(axis=0))
    c2 = CyLPArray([1.0/numSims] * numSims)

    # Solve the LP problems
    weights_clp = np.zeros((len(lbds), numAssets))
    for i in range(len(lbds)):
        lbd = lbds[i]
        c1 = lbd * c
        smplx.objective = c2 * u -  c1 * w
        smplx.primal()
        weights_clp[i,:]=smplx.primalVariableSolution['w']
    return weights_clp


I am working on the file "Benchmark_between_Python_and_Julia.ipynb".
The first task is:
And now the same code executed by an external Pypy interpreter (Python 2.7.13 and PyPy 5.8.0 with GCC 5.4.0)
Can you write Python code for it?

%%time
%%pypy

import math
import random

f = lambda x: (2.0 / math.sqrt(math.pi)) * math.exp(-x**2)

# Same code
def romberg_pypy(f, xmin, xmax, n=8, m=None):
    assert xmin <= xmax
    if m is None:
        m = n
    assert n >= m >= 0
    # First value:
    r = [[0 for _ in range(n+1)] for _ in range(m+1)]
    r[0][0] = (xmax - xmin) * (f(xmax) + f(xmin)) / 2.

    # One side of the triangle:
    for i in range(1, n + 1):
        h_i = (xmax - xmin) / 2.**i
        r[i][0] = (0.5 * r[i - 1][0]) + h_i * math.fsum(
            f(xmin + ((2 * k - 1) * h_i))
            for k in range(1, 1 + 2**(i - 1))
        )

    # All the other values:
    for j in range(1, m + 1):
        for i in range(j, n + 1):
            r[i][j] = (((4.**j) * r[i][j - 1]) - r[i - 1][j - 1]) / ((4.**j) - 1.)

    return r[n][m]

for _ in range(100000):
    a = 0
    b = 1
    romberg_pypy(f, a, b)
print(romberg_pypy(f, a, b))


%%time
%%pypy

import math
import random

f = lambda x: (12*x+1)/(1+math.cos(x)**2)

# Same code
def romberg_pypy(f, xmin, xmax, n=8, m=None):
    assert xmin <= xmax
    if m is None:
        m = n
    assert n >= m >= 0
    # First value:
    r = [[0 for _ in range(n+1)] for _ in range(m+1)]
    r[0][0] = (xmax - xmin) * (f(xmax) + f(xmin)) / 2.

    # One side of the triangle:
    for i in range(1, n + 1):
        h_i = (xmax - xmin) / 2.**i
        r[i][0] = (0.5 * r[i - 1][0]) + h_i * math.fsum(
            f(xmin + ((2 * k - 1) * h_i))
            for k in range(1, 1 + 2**(i - 1))
        )

    # All the other values:
    for j in range(1, m + 1):
        for i in range(j, n + 1):
            r[i][j] = (((4.**j) * r[i][j - 1]) - r[i - 1][j - 1]) / ((4.**j) - 1.)

    return r[n][m]

for _ in range(100000):
    a = random.randint(-2000, 2000)
    b = a + random.randint(0, 100)
    romberg_pypy(f, a, b)


I am working on the file "Benchmark_between_Python_and_Julia.ipynb".
The first task is:
It converges quite quickly to the "true" value as given by scipy.integrate.quad.

The Romberg method, better dynamic programming version in Python
Instead of using a dictionary, which gets filled up dynamically (and so, slowly), let us use a numpy arrays, as we already know the size of the array we need ($n+1 \times m+1$).
Note that only half of the array is used, so we could try to use sparse matrices maybe, for triangular matrices? From what I know, it is not worth it, both in term of memory information (if the sparsity measure is $\simeq 1/2$, you don't win anything from LIL or other sparse matrices representation...
We could use numpy.tri but this uses a dense array so nope.
Can you write Python code for it?

import numpy as np

def romberg_better(f, xmin, xmax, n=8, m=None):
    assert xmin <= xmax
    if m is None:
        m = n
    assert n >= m >= 0
    # First value:
    r = np.zeros((n+1, m+1))
    r[0, 0] = (xmax - xmin) * (f(xmax) + f(xmin)) / 2.

    # One side of the triangle:
    for i in range(1, n + 1):
        h_i = (xmax - xmin) / 2.**i
        r[i, 0] = (0.5 * r[i - 1, 0]) + h_i * math.fsum(
            f(xmin + ((2 * k - 1) * h_i))
            for k in range(1, 1 + 2**(i - 1))
        )

    # All the other values:
    for j in range(1, m + 1):
        for i in range(j, n + 1):
            r[i, j] = (((4.**j) * r[i, j - 1]) - r[i - 1, j - 1]) / ((4.**j) - 1.)

    return r[n, m]

romberg_better(f, a, b, n=0)  # really not accurate!
romberg_better(f, a, b, n=1)  # alreay pretty good!
romberg_better(f, a, b, n=2)
romberg_better(f, a, b, n=3)
romberg_better(f, a, b, n=8)  # Almost the exact value.
romberg_better(f, a, b, n=10)  # Almost the exact value.
romberg_better(f, a, b, n=12)  # Almost the exact value.


I am working on the file "Benchmark_between_Python_and_Julia.ipynb".
The first task is:
It converges quite quickly to the "true" value as given by scipy.integrate.quad.

The Romberg method, dynamic programming version in Python
See https://mec-cs101-integrals.readthedocs.io/en/latest/_modules/integrals.html#romberg for the code and https://mec-cs101-integrals.readthedocs.io/en/latest/integrals.html#integrals.romberg for the doc.
It is not hard to make this function non-recursive, by storing the intermediate results.
Can you write Python code for it?

def romberg(f, xmin, xmax, n=8, m=None):
    assert xmin <= xmax
    if m is None:
        m = n
    assert n >= m >= 0
    # First value:
    r = {(0, 0): 0.5 * (xmax - xmin) * (f(xmax) + f(xmin))}

    # One side of the triangle:
    for i in range(1, n + 1):
        h_i = (xmax - xmin) / float(2**i)
        xsamples = [xmin + ((2 * k - 1) * h_i) for k in range(1, 1 + 2**(i - 1))]
        r[(i, 0)] = (0.5 * r[(i - 1, 0)]) + h_i * math.fsum(f(x) for x in xsamples)

    # All the other values:
    for j in range(1, m + 1):
        for i in range(j, n + 1):
            try:
                r[(i, j)] = (((4**j) * r[(i, j - 1)]) - r[(i - 1, j - 1)]) / float((4**j) - 1)
            except:
                raise ValueError("romberg() with n = {}, m = {} and i = {}, j = {} was an error.".format(n, m, i, j))

    return r[(n, m)]

romberg(f, a, b, n=0)  # really not accurate!
romberg(f, a, b, n=1)  # alreay pretty good!
romberg(f, a, b, n=2)
romberg(f, a, b, n=3)
romberg(f, a, b, n=8)  # Almost the exact value.
romberg(f, a, b, n=10)  # Almost the exact value.
romberg(f, a, b, n=12)  # Almost the exact value.


I am working on the file "Benchmark_between_Python_and_Julia.ipynb".
The first task is:
It fails! Almost as always when trying Numba, it fails cryptically, too bad. I don't want to spend time debugging this.

Naive Julia version

Thanks to this page for a nice and short introduction to Julia.
Can you write Python code for it?

%%script julia

function f(x)
    (12*x + 1) / (1 + cos(x)^2)
end

a = 1993
b = 2017

function romberg_julia(f, xmin, xmax; n=8)
    m = n
    # First value:
    r = Dict()
    r[(0, 0)] = (xmax - xmin) * (f(xmax) + f(xmin)) / 2.

    # One side of the triangle:
    for i in 1 : n
        h_i = (xmax - xmin) / (2^i)
        sum_f_x = 0
        for k in 1 : 2^(i - 1)
            sum_f_x += f(xmin + ((2 * k - 1) * h_i))
        end
        r[(i, 0)] = (r[(i - 1, 0)] / 2.) + (h_i * sum_f_x)
    end

    # All the other values:
    for j in 1 : m
        for i in j : n
            r[(i, j)] = (((4^j) * r[(i, j - 1)]) - r[(i - 1, j - 1)]) / (4^j - 1.)
        end
    end

    r[(n, m)]
end

println(romberg_julia(f, a, b, n=0))  # really not accurate!
println(romberg_julia(f, a, b, n=1))  # alreay pretty good!
println(romberg_julia(f, a, b, n=2))
println(romberg_julia(f, a, b, n=3))
println(romberg_julia(f, a, b, n=8))  # Almost the exact value.
println(romberg_julia(f, a, b, n=10))  # Almost the exact value.
println(romberg_julia(f, a, b, n=12))  # Almost the exact value.


I am working on the file "Benchmark_between_Python_and_Julia.ipynb".
The first task is:
Let try it with this function $f(x)$ on $[a,b]=[1993,2015]$:
$$ f(x) := \frac{12x+1}{1+\cos(x)^2} $$
Can you write Python code for it?

import math

f = lambda x: (12*x+1)/(1+math.cos(x)**2)
a, b = 1993, 2017

quad(f, a, b)


I am working on the file "Benchmark_between_Python_and_Julia.ipynb".
The first task is:
Oh, this time it finally seems faster. Really faster? Yes, about 3 to 4 time faster than Pypy.
Remark also that this last cells compared by using the magic %%pypy and %%script julia, so they both need a warm-up time (opening the pipe, the sub-process, initializing the JIT compiler etc).
But it is fair to compare Pypy to Julia this way.

Second benchmark
Let try the same numerical algorithm but with a different integrand function.
$$\int_{0}^{1} \exp(-x^2) \mathrm{d}x \approx 0.842700792949715$$
First with Python:
Can you write Python code for it?

%%time

import numpy as np
import math
import random

f = lambda x: (2.0 / math.sqrt(math.pi)) * math.exp(-x**2)

# Same code
def romberg(f, xmin, xmax, n=8, m=None):
    assert xmin <= xmax
    if m is None:
        m = n
    assert n >= m >= 0
    # First value:
    r = np.zeros((n+1, m+1))
    r[0, 0] = (xmax - xmin) * (f(xmax) + f(xmin)) / 2.

    # One side of the triangle:
    for i in range(1, n + 1):
        h_i = (xmax - xmin) / 2.**i
        r[i, 0] = (0.5 * r[i - 1, 0]) + h_i * math.fsum(
            f(xmin + ((2 * k - 1) * h_i))
            for k in range(1, 1 + 2**(i - 1))
        )

    # All the other values:
    for j in range(1, m + 1):
        for i in range(j, n + 1):
            r[i, j] = (((4.**j) * r[i, j - 1]) - r[i - 1, j - 1]) / ((4.**j) - 1.)

    return r[n, m]

for _ in range(100000):
    a = 0
    b = 1
    romberg(f, a, b)
print(romberg(f, a, b))


I am working on the file "Benchmark_between_Python_and_Julia.ipynb".
The first task is:
On this first test, it doesn't look faster than Pypy...
But what if we use the improved version, with an array instead of dictionary?
Can you write Python code for it?

%%time
%%script julia

function f(x)
    (12*x + 1) / (1 + cos(x)^2)
end

function romberg_julia_better(f, xmin, xmax; n=8)
    m = n
    # First value:
    r = zeros((n+1, m+1))  # https://docs.julialang.org/en/latest/stdlib/arrays/#Base.zeros
    r[1, 1] = (xmax - xmin) * (f(xmax) + f(xmin)) / 2.

    # One side of the triangle:
    for i in 1 : n
        h_i = (xmax - xmin) / (2^i)
        sum_f_x = 0
        for k in 1 : 2^(i - 1)
            sum_f_x += f(xmin + ((2 * k - 1) * h_i))
        end
        r[i + 1, 1] = (r[i, 1] / 2.) + (h_i * sum_f_x)
    end

    # All the other values:
    for j in 1 : m
        for i in j : n
            r[i + 1, j + 1] = (((4.^j) * r[i + 1, j]) - r[i, j]) / (4.^j - 1.)
        end
    end

    r[n + 1, m + 1]
end

for _ in 1:100000
    a = rand(-2000:2000)
    b = a + rand(0:100)
    romberg_julia_better(f, a, b)
end


I am working on the file "Benchmark_between_Python_and_Julia.ipynb".
The first task is:
Still not faster than Pypy... So what is the goal of Julia?
Can you write Python code for it?

%%time
%%script julia

function f(x)
    (2.0 / sqrt(pi)) * exp(-x^2)
end

function romberg_julia_better(f, xmin, xmax; n=8)
    m = n
    # First value:
    r = zeros((n+1, m+1))  # https://docs.julialang.org/en/latest/stdlib/arrays/#Base.zeros
    r[1, 1] = (xmax - xmin) * (f(xmax) + f(xmin)) / 2.

    # One side of the triangle:
    for i in 1 : n
        h_i = (xmax - xmin) / (2^i)
        sum_f_x = 0
        for k in 1 : 2^(i - 1)
            sum_f_x += f(xmin + ((2 * k - 1) * h_i))
        end
        r[i + 1, 1] = (r[i, 1] / 2.) + (h_i * sum_f_x)
    end

    # All the other values:
    for j in 1 : m
        for i in j : n
            r[i + 1, j + 1] = (((4.^j) * r[i + 1, j]) - r[i, j]) / (4.^j - 1.)
        end
    end

    r[n + 1, m + 1]
end

for _ in 1:100000
    a = 0
    b = 1
    romberg_julia_better(f, a, b)
end
println(romberg_julia_better(f, 0, 1))


I am working on the file "Benchmark_between_Python_and_Julia.ipynb".
The first task is:
This version uses the improved memoization trick (no dictionary), but uses nested lists and not numpy arrays, I didn't bother to install numpy on my Pypy installation (even thought it should be possible).

Numba version for Python
Can you write Python code for it?

from numba import jit

@jit
def romberg_numba(f, xmin, xmax, n=8):
    assert xmin <= xmax
    m = n
    # First value:
    r = {(0, 0): 0.5 * (xmax - xmin) * (f(xmax) + f(xmin))}

    # One side of the triangle:
    for i in range(1, n + 1):
        h_i = (xmax - xmin) / float(2**i)
        xsamples = [xmin + ((2 * k - 1) * h_i) for k in range(1, 1 + 2**(i - 1))]
        r[(i, 0)] = (0.5 * r[(i - 1, 0)]) + h_i * math.fsum(f(x) for x in xsamples)

    # All the other values:
    for j in range(1, m + 1):
        for i in range(j, n + 1):
            try:
                r[(i, j)] = (((4**j) * r[(i, j - 1)]) - r[(i - 1, j - 1)]) / float((4**j) - 1)
            except:
                raise ValueError("romberg() with n = {}, m = {} and i = {}, j = {} was an error.".format(n, m, i, j))

    return r[(n, m)]

romberg_numba(f, a, b, n=8)  # Almost the exact value.


I am working on the file "Benchmark_between_Python_and_Julia.ipynb".
The first task is:
We already see that the recursive version is much slower than the dynamic programming one!
But there is not much difference between the one using dictionary (romberg()) and the one using a numpy array of a known size (romberg_better()).

Using Pypy for speedup
Can you write Python code for it?

%%time

import numpy as np
import math
import random

f = lambda x: (12*x+1)/(1+math.cos(x)**2)

# Same code
def romberg(f, xmin, xmax, n=8, m=None):
    assert xmin <= xmax
    if m is None:
        m = n
    assert n >= m >= 0
    # First value:
    r = np.zeros((n+1, m+1))
    r[0, 0] = (xmax - xmin) * (f(xmax) + f(xmin)) / 2.

    # One side of the triangle:
    for i in range(1, n + 1):
        h_i = (xmax - xmin) / 2.**i
        r[i, 0] = (0.5 * r[i - 1, 0]) + h_i * math.fsum(
            f(xmin + ((2 * k - 1) * h_i))
            for k in range(1, 1 + 2**(i - 1))
        )

    # All the other values:
    for j in range(1, m + 1):
        for i in range(j, n + 1):
            r[i, j] = (((4.**j) * r[i, j - 1]) - r[i - 1, j - 1]) / ((4.**j) - 1.)

    return r[n, m]

for _ in range(100000):
    a = random.randint(-2000, 2000)
    b = a + random.randint(0, 100)
    romberg(f, a, b)


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
$\implies$ We obtained the same result, OK our function works!
Trying 1000 random examples
On a small sentence:
Can you write Python code for it?

hash_SHA2("My name is Zorro !")

h = hashlib.sha256()
h.update(b"My name is Zorro !")
h.hexdigest()


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
As SHA-2 plays with big-endian and little-endian integers, and at the end it requires a leftshift to combine the 5 hash pieces into one.
Can you write Python code for it?

def leftshift(x, c):
    """ Left shift the number x by c bytes."""
    return x << c

def rightshift(x, c):
    """ Right shift the number x by c bytes."""
    return x >> c


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
Check on SHA-2
Let try the example from SHA-2 Wikipedia page :
Can you write Python code for it?

hash_SHA2_Numba("The quick brown fox jumps over the lazy dog")
assert hash_SHA2_Numba("The quick brown fox jumps over the lazy dog") == 'd7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592'


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
Even a small change in the message will (with overwhelming probability) result in a mostly different hash, due to the avalanche effect. For example, adding a period at the end of the sentence:
Can you write Python code for it?

hash_SHA2("The quick brown fox jumps over the lazy dog.")
assert hash_SHA2("The quick brown fox jumps over the lazy dog.") == 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c'


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
I failed to make numba.jit work on that function :-(

Cython-power code for the SHA-2 hashing function
Requirements
You need cython and the cython Jupyter extension to be installed.
Can you write Python code for it?


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
I will need at least this one:
Can you write Python code for it?

assert 'SHA256' in hashlib.algorithms_available


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
It starts to look good.
Can you write Python code for it?

def true_hash_SHA2(data):
    h = hashlib.sha256()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    h.update(data)
    return h.hexdigest()


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
Lets check that they have the block size and digest size announced:
Can you write Python code for it?

name = 'SHA256'
s = hashlib.sha256()
print("For {:<8} : the block size is {:<3} and the digest size is {:<2}.".format(name, s.block_size, s.digest_size))


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
Numba-powered code for the SHA-2 hashing function
Requirements
You need numba to be installed.
Can you write Python code for it?


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
On basic functions like this, of course we don't get any speedup with Cython:
Can you write Python code for it?

from numpy.random import randint

%timeit leftrotate(randint(0, 100000), 5)
%timeit leftrotate_cython(randint(0, 100000), 5)

%timeit rightrotate(randint(0, 100000), 5)
%timeit rightrotate_cython(randint(0, 100000), 5)

%%cython

cpdef int leftshift_cython(int x, int c):
    """ Left shift the number x by c bytes."""
    return x << c

cpdef int rightshift_cython(int x, int c):
    """ Right shift the number x by c bytes."""
    return x >> c

leftshift_cython?
rightshift_cython?


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
Pure Python code for the SHA-2 hashing function
Let now study and implement a last hashing function, again slightly harder to write but more secure: SHA-2, "Secure Hash Algorithm, version 2".
See the SHA-2 hashing function on Wikipedia, if needed.
Remark: it is not (yet) considered broken, and it is the military standard for security and cryptographic hashing. SHA-3 is preferred for security purposes.
Useful functions the SHA-2 algorithm
This is exactly like for MD5. But SHA-2 requires right-rotate as well.
Can you write Python code for it?

def leftrotate(x, c):
    """ Left rotate the number x by c bytes."""
    x &= 0xFFFFFFFF
    return ((x << c) | (x >> (32 - c))) & 0xFFFFFFFF

def rightrotate(x, c):
    """ Right rotate the number x by c bytes."""
    x &= 0xFFFFFFFF
    return ((x >> c) | (x << (32 - c))) & 0xFFFFFFFF


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
The SHA2_Cython class
And similarly for the SHA2 class, we write it in a %%cython cell, and we type everything.
Can you write Python code for it?

%%cython
# cython: c_string_type=unicode, c_string_encoding=utf8

cdef int rightrotate_cython(int x, int c):
    """ Right rotate the number x by c bytes."""
    return (x >> c) | (x << (32 - c))

cdef int rightshift_cython(int x, int c):
    """ Right shift the number x by c bytes."""
    return x >> c

# See http://cython.readthedocs.io/en/latest/src/tutorial/array.html
from cpython cimport array
import array

cdef array.array empty_64 = array.array('i', [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
cdef int[:] view_empty_64 = empty_64

cpdef void update_cython(int[:] hash_pieces, int[:] k, bytearray arg):
    """ One pass of the SHA-256 algorithm, update hash_pieces on place. """
    # Extract the 8 variables
    cdef int h0 = hash_pieces[0], h1 = hash_pieces[1], h2 = hash_pieces[2], h3 = hash_pieces[3], h4 = hash_pieces[4], h5 = hash_pieces[5], h6 = hash_pieces[6], h7 = hash_pieces[7]
    # 1. Pre-processing, exactly like MD5
    cdef bytearray data = arg
    cdef long orig_len_in_bits = 8 * len(data)
    # 1.a. Add a single '1' bit at the end of the input bits
    data.append(0x80)
    # 1.b. Padding with zeros as long as the input bits length ≡ 448 (mod 512)
    while len(data) % 64 != 56:
        data.append(0x0)
    # 1.c. append original length in bits mod (2 pow 64) to message
    data += orig_len_in_bits.to_bytes(8, byteorder='big')
    assert len(data) % 64 == 0, "Error in padding"

    # Declare loop indexes and variables
    cdef int offset, i
    cdef int a, b, c, d, e, f, g, h
    cdef int temp1, temp2

    # 2. Computations
    # Process the message in successive 512-bit = 64-bytes chunks:
    cdef int[:] w = view_empty_64

    for offset in range(0, len(data), 64):
        # 2.a. 512-bits = 64-bytes chunks
        # 2.b. Break chunk into sixteen 32-bit = 4-bytes words w[i], 0 ≤ i ≤ 15
        for i in range(16):
            w[i] =...


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
The SHA2_Numba class
And similarly for the SHA2 class, with the numba.jit decorator to the update function.
Can you write Python code for it?

class SHA2_Numba(Hash):
    """SHA256 hashing, speed-up with Numba.jit, see https://en.wikipedia.org/wiki/SHA-2#Pseudocode."""

    def __init__(self):
        self.name        = "SHA256"
        self.byteorder   = 'big'
        self.block_size  = 64
        self.digest_size = 32
        # Note 2: For each round, there is one round constant k[i] and one entry in the message schedule array w[i], 0 ≤ i ≤ 63
        # Note 3: The compression function uses 8 working variables, a through h
        # Note 4: Big-endian convention is used when expressing the constants in this pseudocode,
        #         and when parsing message block data from bytes to words, for example,
        #         the first word of the input message "abc" after padding is 0x61626380

        # Initialize hash values:
        # (first 32 bits of the fractional parts of the square roots of the first 8 primes 2..19):
        h0 = 0x6a09e667
        h1 = 0xbb67ae85
        h2 = 0x3c6ef372
        h3 = 0xa54ff53a
        h4 = 0x510e527f
        h5 = 0x9b05688c
        h6 = 0x1f83d9ab
        h7 = 0x5be0cd19

        # Initialize array of round constants:
        # (first 32 bits of the fractional parts of the cube roots of the first 64 primes 2..311):
        self.k = [
            0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
            0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
            0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
            0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
            0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
            0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
            0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
           ...


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
The hash of the zero-length string is:
Can you write Python code for it?


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
Useful functions the SHA-2 algorithm
For the functions defined before, we rewrite them with type annotations in %%cython cells.
All variables are int, i.e., 32-bits integer (64-bits are long).
Can you write Python code for it?

%%cython

cpdef int leftrotate_cython(int x, int c):
    """ Left rotate the number x by c bytes."""
    return (x << c) | (x >> (32 - c))

cpdef int rightrotate_cython(int x, int c):
    """ Right rotate the number x by c bytes."""
    return (x >> c) | (x << (32 - c))

leftrotate_cython?
rightrotate_cython?


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
Useful functions the SHA-2 algorithm
Let just add the numba.jit decorator to every function we defined before:
Can you write Python code for it?

@jit
def leftrotate_numba(x, c):
    """ Left rotate the number x by c bytes."""
    x &= 0xFFFFFFFF
    return ((x << c) | (x >> (32 - c))) & 0xFFFFFFFF

@jit
def rightrotate_numba(x, c):
    """ Right rotate the number x by c bytes."""
    x &= 0xFFFFFFFF
    return ((x >> c) | (x << (32 - c))) & 0xFFFFFFFF

@jit
def leftshift_numba(x, c):
    """ Left shift the number x by c bytes."""
    return x << c

@jit
def rightshift_numba(x, c):
    """ Right shift the number x by c bytes."""
    return x >> c


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
We can also write a function to directly compute the hex digest from some bytes data.
Can you write Python code for it?

def hash_SHA2_Cython(data):
    """ Shortcut function to directly receive the hex digest from SHA2_Cython(data)."""
    h = SHA2_Cython()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    print("type(data) =", type(data))
    h.update(data)
    return h.hexdigest()

data = bytes("", encoding='utf8')
h = SHA2_Cython()

h.hash_pieces[:1]
type(h.hash_pieces)
h.k[:1]
type(h.k)
data
type(data)

update_cython(h.hash_pieces, h.k, bytearray(data))


def hash_SHA2_Numba(data):
    """ Shortcut function to directly receive the hex digest from SHA2_Numba(data)."""
    h = SHA2_Numba()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    h.update(data)
    return h.hexdigest()


def hash_SHA2(data):
    """ Shortcut function to directly receive the hex digest from SHA2(data)."""
    h = SHA2()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    h.update(data)
    return h.hexdigest()


I am working on the file "Benchmark_of_the_SHA256_hash_function__Python_Cython_Numba.ipynb".
The first task is:
We can check the available algorithms, some of them being guaranteed to be on any platform, some are not.
Can you write Python code for it?


I am working on the file "Benchmarks.ipynb".
The first task is:
Basic principles of electrolyte chemistry for microfluidic electrokinetics. Part I: Acid–base equilibria and pH buffers
Persat,  2009
Can you write Python code for it?

pKa = [4.75, 7, 9.25]
c = np.logspace(-6, 0)

for valence, extra_pKa, style in [(-1, -2, 'k'), (1, 16, '--k')]:
    for k in pKa + [extra_pKa]:
        ion = ionize.Ion('dummy', [valence], [k], [valence])
        pH = [ionize.Solution(ion, cp).pH for cp in c]
        plot.semilogy(pH, c, style)

plot.xlim(2, 12)
plot.xlabel('pH')
plot.ylabel('Concentration (M)')
plot.show()


I am working on the file "Benchmarks.ipynb".
The first task is:
Fig 1  Mobility dependence of several binary electrolytes on ionic strength. Plot shows ionic mobility of a cation for binary electrolytes with various anion:cation valence values. (a) Varia- tion of electrophoretic mobility of cations with ionic strength using Onsager–Fuoss model with finite ionic radius, Eq. (3). Univalent cations have lesser reduction in mobility than divalent ions for increasing ionic strength. Thus, cations in  1:1 and  2:1 electrolytes have lesser mobility, respectively, than cations in  1:2 and  2:2 electrolytes. In addition, greater charge on counter-ions (anions) causes greater reduction in mobility. Thus, cations in  2:1 and  1:2 electrolytes have lesser mobility, respectively, than cations in  1:1 and –2:2 electrolytes. (b) The effect of finite radius correction to Onsager–Fuoss model. Dotted line corresponds to Onsager–Fuoss model assuming ions as point charges, Eq. (2), and solid line corresponds to Onsager–- Fuoss model using finite ionic radius, Eq. (3). Neglecting ionic radius overpredicts the reduction in mobility at finite ionic strengths. For these sample calculations, the mobilities of all ions is taken as m0 1⁄4 50   10 9 m2/V/s.
Can you write Python code for it?

fl = db['fluorescein']
fl= ionize.Ion(reference_mobility=[-3.6000000000000004e-08, -1.9e-08], valence=[-2, -1], alias=None, reference_pKa=[6.8, 4.4], reference_temperature=25.0, name='fluorescein')
I = np.linspace(0, .1)
pH = [9.35, 7.15]

for p in pH:
    mob = np.array([fl.mobility(p, Ip) for Ip in I])
    plot.plot(I, -mob, 'k')

plot.xlim(0, .1)
plot.ylim(1e-8, 4e-8)
plot.xlabel('Ionic Strength (M)')
plot.ylabel('mobility (m^2/V/s)')
plot.show()


I am working on the file "Benchmarks.ipynb".
The first task is:
Fig 1 Flood’s diagram for strong and weak acids (solid lines), and strong and weak bases (dashed lines) in solution. The left half of the diagram corresponds to the logarithm of total concentration of a single acid as a function of pH. The pH of weak acids deviates from the strong acid relation pH 1⁄4  log10cA. pH always decreases with addition of weak acid, but more and more weakly as their pKaincreases. Similarly, weak bases deviate from the effect of strong base on pH (pH 1⁄4 14 + log10cB). For the same base concentration, the pH of a stronger base (higher pKa) is higher than for a weak base.
Can you write Python code for it?

pKa = [4.75, 7, 9.25]
c = np.logspace(-8, 0)
valence = -1

for k in pKa:
        ion = ionize.Ion('dummy', [valence], [k], [valence])
        pH = [ionize.Solution(ion, cp).pH for cp in c]
        plot.semilogy(pH, c, 'k')

plot.semilogy(-np.log10(c), c, 'k')
plot.semilogy(-np.log10(c), 10*c, '--k')
# plot.Rectangle((0, 1e-3), 3, 100, color='b') # fix area plot
plot.xlim(2, 7)
plot.ylim(1e-8, 1)
plot.xlabel('pH')
plot.ylabel('Concentration (M)')
plot.show()


I am working on the file "Benchmarks.ipynb".
The first task is:
Fig 1 Ionic strength and pH effects on electrophoretic mobility of fluorescein. We use the Pitts equation (equation (12)) and Davies equa- tions (eq. (40) of Part I of this two-paper series) to correct electrophoretic mobility and pKa for ionic strength effects, respectively. We plot the absolute value of fluorescein mobility (all values are actually negative). The values of concentrations correspond to ionic strength of the solution, and the concentration of fluorescein is small compared to ionic strength. We used sodium as counterion. The dissociation coefficients of fluores- cein yield the strongest influence of pH on mobility according to equa- tions (6). The shaded zone corresponds to pH values where fluorescein fluorescence is below 10% its maximum value.25,26 We used pK 1 1⁄4 4.45, pK 2 1⁄4 6.8,26 and m 2 1⁄4  39.5   10 9 m2 V 1 s 1and m 1 1⁄4  25   10 9 m2 V 1 s 1,23,24 and a 1⁄4 4 A ̊ . In the electrophoretic mobility calculations, we neglected the effect of the cationic form of fluorescein (pK1 1⁄4 2.1).
Fig 3 Effect of electrode reaction on anode pH and cathode pH for Tris and acetate buffers titrated with respectively hydrochloric acid and sodium hydroxide. (a) and (b): pH variation for 10 mM Tris initially titrated with hydrochloric acid to pH0, which is varied from pKa   1 to pKa + 1. We consider 2.5 kV applied to a 5 cm long, 50 mm wide (20 mm deep) channel. (a) pH at the 100 ml anode reservoir. If initial pH is below pKa, the anode pH drops dramatically, as the anode generates H+. (b) pH at the 100 mL cathode reservoir. The pH change is slow as the cathode reservoir fills with weak base. Inset: anode (black solid lines) and cathode (grey solid lines) pH for 10 mL, 25 mL and 100 mL reservoirs and pH0 1⁄4 pKa. (c) and (d): pH variation for 10 mM acetate buffer (titrated with sodium hydroxide) in otherwise the same conditions as the previous case. (c) pH at the 100 mL anode reservoir. The pH change is slow as the anode reservoir fills with weak acid. Inset: anode (black solid lines) and cathode...
Can you write Python code for it?

concentrations = np.linspace(0, 0.14)
ref_mob = 50.e-9
z = [1, 2]
for zp, zm in itertools.product(z, repeat=2):
    positive_ion = ionize.Ion('positive', [zp], [14], [ref_mob])
    negative_ion = ionize.Ion('negative', [-zm], [0], [-ref_mob])
    mob = []
    i = []
    for c in concentrations:
        sol = ionize.Solution([positive_ion, negative_ion], [c/zp, c/zm])
        mob.append(sol.ions[0].actual_mobility() / ref_mob )
        i.append(sol.ionic_strength)
    plot.plot(i, mob, label='-{}:{}'.format(zm, zp))
plot.ylim(0, 1)
plot.xlim(0, .14)
plot.legend(loc='lower left')
plot.xlabel('Concentration (M)')
plot.ylabel('$\mu$/$\mu_o$')
plot.show()


I am working on the file "Benchmarks.ipynb".
The first task is:
Fig 3 Titration of a weak acid (or base) by strong base (or acid). The difference between the buffer’s pH and the weak electrolyte’s pKa (pKweak) is plotted as a function of the concentration ratio of strong electrolyte (titrant) to weak electrolyte ctitrant/cweak. The solid portion of the curves assume moderate pH. The buffer’s sensitivity to titrant is lowest when ctitrant/cweak 1⁄4 0.5, at which point pH 1⁄4 pKweak, also called the half titration point. Buffer pH is most sensitive to addition of titrant when ctitrant z cweak or when ctitrant z 0. In these regions (marked by dashed lines), the moderate pH assumption breaks down, and the exact shape of the curve will depend strongly on the species’ pKa and the weak acid’s concentration. The inflection point which occurs at ctitrant 1⁄4 cweak (marked by ‘x’) is known as the equivalence point.
Can you write Python code for it?

tris = db['tris']
acetic = db['acetic acid']
conc = np.linspace(0, .1)
pH = [ionize.Solution([tris, acetic], [.05, c]).pH for c in conc]
plot.plot(conc*1000, pH, 'k')
plot.xlabel('C_a (mM)')
plot.ylabel('Buffer pH')
plot.show()


I am working on the file "Benchmarks.ipynb".
The first task is:
Fig 4 Titration of 50 mM Tris with up to 100 mM acetic acid (acetic acid denoted as ‘‘a’’). We here use concentration values which describe the total (analytical) concentration of acetic acid in solution. This is unlike the typical approach of plotting titration curves with respect to added volume (or number of moles) of titrant, but has the advantage of making the curve universal for all added volumes. At ca 1⁄4 0, the pH is elevated so that the concentration of Tris+ ions is approximately equal to that of OH  (and ‘‘moderate pH’’ assumption is violated). The buffer pH is dominated by the pKa of Tris ( 8.1) at low ( 25 mM) acetic acid concentrations, and by the pKa of acetic acid ( 4.8) at high ( 75 to 100 mM) acetic acid concentrations. The optimal buffer regions (i.e., the lowest slopes in the curve) occur, as expected, near pH 1⁄4 4.8 and 8.1. The inset shows the overprediction of eqn (28) due to its moderate pH assumption. Note that the error, while substantial at low titrant concentrations (over 1 pH unit for 10 mM acetic acid), is negligible even for 1 mM acetic acid.
Can you write Python code for it?

tris = db['tris']
acetic = db['acetic acid']
strong_acid = ionize.Ion('strong acid', [-1], [-2], [-1])
strong_base = ionize.Ion('strong base', [1], [16], [1])
pH = np.linspace(3, 11)
conc = [.01, .03, .05]

for buff, titrant, style in [(tris, strong_acid, 'k'),
                             (acetic, strong_base, '--k')]:
    for c in conc:
        cap = []
        for p in pH:
            try:
                sol = ionize.Solution(buff, c).titrate(titrant, p)
                cap.append(sol.buffering_capacity())
            except:
                cap.append(None)
        plot.plot(pH, cap, style)
plot.xlabel('Buffer pH')
plot.ylabel('Buffer Capacity (M)')
plot.show()


I am working on the file "Benchmarks.ipynb".
The first task is:
Fig 5 Buffering capacity for a weak acid buffer (acetic acid, pKa 1⁄4 4.75, dashed lines) and a weak base buffer (Tris, pKa 1⁄4 8.1, solid lines), at
several concentrations. As evident from equations (32) and (33), buffers
are most effective at high concentrations and near their pK ’s. At both a
high pH (pH > 10) and low pH (pH < 4) the self-ionization of water provides an additional buffering effect.
Can you write Python code for it?

H = ionize.Aqueous.henry_CO2(25)
c = .0004 * H
pH = np.linspace(3, 10)
CO2 = db['carbonic acid']

CT = []
C1 = []
C2 = []
I = []
for p in pH:
    f1, f2 = CO2.ionization_fraction(p)
    f0 = 1 - f1 - f2
    CT.append(c / f0)
    C1.append(CT[-1] * f1)
    C2.append(CT[-1] * f2)
    I.append(CT[-1] * CO2.charge(p, moment=2) / 2)

plot.semilogy(pH, CT, 'k')
plot.semilogy(pH, C1, 'k')
plot.semilogy(pH, C2, 'k')
plot.semilogy(pH, I, '--k')

plot.xlabel('pH')
plot.ylabel('c (M)')
plot.ylim(3e-7, 3e-2)
plot.show()


I am working on the file "Benchmarks.ipynb".
The first task is:
Fig 7 Flood’s diagram for three different Tris hydrochloride (pKa 1⁄4 8.2at 20  C) buffers under the influence of dissolved CO2 and carbonate species. The dashed vertical lines correspond to three idealized cases where the atmospheric carbon dioxide concentration is neglected, and moderate pH. The solid curves correspond to a realistic buffer subject to atmospheric effects. The curve labels indicate the pH value in idealized conditions (free of CO2). The middle curve (labelled pKa) corresponds to a buffer prepared with cbuffer of Tris and 0.5 cbuffer of HCl, so that, ideally, pH 1⁄4 pKa 1⁄4 8.2 (from Henderson–Hasselbalch equation (21)). For cbuffer > 100 mM, the buffer pH has values reasonably close to those predicted by neglecting CO2. However, we see strong deviation from ideal pH for buffer concentrations smaller than 10 mM. For example, a 1 mM Tris, 0.5 mM HCl solution has an actual pH about 7.7; while a 0.1 mM, Tris 0.05 mM HCl solution has an actual pH of about 6.9 (vs. 8.2). The left- (labelled pKa   1) and right-hand (labelled pKa + 1) curves correspond to buffers prepared with cbuffer of Tris and respectively 0.91 cbuffer and 11 cbuffer of HCl; and so, in the absence of CO2, would yield respectively pH1⁄4pKa  11⁄47.2andpH1⁄4pKa +11⁄49.2.
Basic principles of electrolyte chemistry for microfluidic electrokinetics. Part II: Coupling between ion mobility, electrolysis, and acid–base equilibria
Persat,  2009
Can you write Python code for it?

fl = db['fluorescein']
pH = np.linspace(4, 9)
I = [0, 1e-3, 1e-2, 1e-1, .5]

for Ip in I:
    mob = np.array([fl.mobility(pp, Ip)
                    for pp in pH])
    plot.plot(pH, -mob, 'k')

plot.show()


I am working on the file "Benchmarks/Age_Screen_Nested_Upstream.ipynb".
The first task is:
Here I am doing a drop-1 test for an interaction term of age X HIV.  In the second act of this analysis we filter a few more patients with data that doesn't agree between aging models.  In general I think these are bad samples, and removing them does improve this result and make the lines more parallel than shown here.  I'm not using these filters here because it seemed a little ad-hoc to go back and apply them before we even looked at aging models and it is generally hard to report an iterative analysis in a journal paper.
Can you write Python code for it?

import statsmodels.api as sm

y = vH[0]
intercept = pd.Series(1, index=y.index)

X = pd.concat([intercept, age, hiv], axis=1, keys=['Intercept', 'age', 'HIV'])
X = X.dropna().ix[y.index]
m1 = sm.OLS(y, X).fit()

X = pd.concat([intercept, age, hiv, hiv*age], axis=1, keys=['Intercept', 'age', 'HIV', 'int'])
X = X.dropna().ix[y.index]
m2 = sm.OLS(y, X).fit()

m2.compare_lr_test(m1)

m2.pvalues

m1.pvalues


I am working on the file "Benchmarks/Age_Screen_Nested_Upstream.ipynb".
The first task is:
The first PC captures about 25% of the variance in the data
Can you write Python code for it?

p = S ** 2 / sum(S ** 2)
p[:5]

fig, ax = subplots(1,1, figsize=(4,3))
rr = 1*vH[0]
k = fx.pred_c.index
hiv = fx.duration.ix[rr.index].dropna() != 'Control'
age = fx.age

sns.regplot(*match_series(age.ix[k], rr.ix[ti(hiv==0)]),
            ax=ax, label='HIV+', ci=None)
sns.regplot(*match_series(age.ix[k], rr.ix[ti(hiv>0)]),
            ax=ax, label='Control', ci=None)
ax.set_ylabel('First PC', size=12)
ax.set_xlabel('Chronological age (years)', size=14)

ax.set_yticks([0])
ax.axhline(0, ls='--', lw=2.5, color='grey', zorder=-1)
ax.set_xbound(23,70)
ax.set_ybound(-.25,.25)
prettify_ax(ax)
fig.tight_layout()
fig.savefig(FIGDIR + 'PC1_validated.pdf')


I am working on the file "Benchmarks/Cell_Composition_Bechmark.ipynb".
The first task is:
Cell Composition from estimateCellCounts
Looking for changes in cell composition with HIV
Can you write Python code for it?

duration = duration.ix[pred_c.index]

screen_feature(duration=='Control', kruskal_pandas, cell_counts.T, align=False)

fig, axs = subplots(2,1, figsize=(6,9), sharex=True)
o = ['Control','HIV Short','HIV Long']
violin_plot_pandas(duration, cell_counts.CD4T, order=o, ax=axs[0])
violin_plot_pandas(duration, cell_counts.CD8T, order=o, ax=axs[1])
for ax in axs:
    prettify_ax(ax)
    ax.set_xlabel('')
axs[0].set_ylabel('CD4T Cell Percentage')
axs[1].set_ylabel('CD8T Cell Percentage')
fig.tight_layout()
fig.savefig(FIGDIR + 'tcells.png', dpi=300)

((cell_counts.CD8T + .01) / (cell_counts.CD4T + .01)).groupby(duration=='Control').mean()

violin_plot_pandas(duration, np.log2((cell_counts.CD4T + .01) / (cell_counts.CD8T + .01)),
                   order=['Control','HIV Short','HIV Long'])


I am working on the file "Benchmarks/Cell_Composition_Bechmark.ipynb".
The first task is:
Mockup of Cell-Composition Adjustment
Real adjustment is done in BMIQ_Normalization notebook.
Can you write Python code for it?

betas = pd.read_hdf(HDFS_DIR + 'dx_methylation.h5', 'betas')
betas = betas['s2'].ix[:, duration.index]
betas = betas.groupby(level=0).first()

flow_sorted_data = pd.read_hdf(HDFS_DIR + 'methylation_annotation.h5','flow_sorted_data')
cell_type = pd.read_hdf(HDFS_DIR + 'methylation_annotation.h5', 'label_map')

n2 = flow_sorted_data.groupby(cell_type, axis=1).median()
avg = n2[cell_counts.columns].dot(cell_counts.T)

b1 = logit_adj(betas).T.corrwith(cell_counts.CD8T)
b2 = logit_adj(df_hiv).ix[:, betas.columns].T.corrwith(cell_counts.CD8T)

ss = avg.std(1)

b1 = b1.groupby(level=0).first()
b2 = b2.groupby(level=0).first()

idx = b1.abs() > .01
(b1[idx].abs() < b2[idx].abs()).value_counts()

from scipy.stats import binom_test

binom_test((b1.abs() < b2.abs()).value_counts())

fig, axs = subplots(1,2, figsize=(9,3))

ss.clip_upper(.05).hist(bins=30, ax=axs[0])
axs[0].set_xticks([0,.01,.02,.03,.04,.05])
axs[0].set_xticklabels([0,.01,.02,.03,.04,'0.05+'])
axs[0].set_xlabel('adjustment magnitude')
axs[0].set_ylabel('# of patients')

axs[1].hexbin(*match_series(b1.abs(), b2.abs()), gridsize=50,
              bins='log')
axs[1].set_aspect(1)
axs[1].plot([0,.9],[0,.9], color='grey', lw=4, ls='--', dash_capstyle='round')
axs[1].set_xlabel('CD8T coorelation \n(pre-adjustment)')
axs[1].set_ylabel('CD8T coorelation \n(post-adjustment)')

for ax in axs:
    prettify_ax(ax)

fig.tight_layout()

flow_sorted_data.T.groupby(cell_type == 'CD8T').mean().diff().ix[1].order().ix[500:].head(10)

p = 'cg19163395'

diff = avg.ix[p].ix[duration.index] - avg.ix[p].median()
diff.hist()

series_scatter(diff, cell_counts.CD8T)

fig, axs = subplots(1,2, figsize=(9,3))
cell_counts.CD8T.ix[duration.index].hist(color='grey', ax=axs[0])
axs[0].set_xlabel('% CD8T (estimated)')
axs[0].set_xticks([0,.1,.2,.3])
axs[0].set_ylabel('# of patients')
box_plot_pandas(cell_type[cell_type.isin(cell_counts.columns)],
                flow_sorted_data.ix[p], ax=axs[1],
               ...


I am working on the file "BenzDatasetAnalysis.ipynb".
The first task is:
Binary variables which shows a good color difference in the above graphs between 0 and 1 are likely to be more predictive given the the count distribution is also good between both the classes (can be seen from the previous graph). We will dive more into the important variables in the later part of the notebook.
ID variable:
One more important thing we need to look at it is ID variable. This will give an idea of how the splits are done across train and test (random or id based) and also to help see if ID has some potential prediction capability (probably not so useful for business)
Let us first see how the 'y' variable changes with ID variable.
Can you write Python code for it?

var_name = "ID"
plt.figure(figsize=(12,6))
sns.regplot(x=var_name, y='y', data=train_df, scatter_kws={'alpha':0.5, 's':30})
plt.xlabel(var_name, fontsize=12)
plt.ylabel('y', fontsize=12)
plt.title("Distribution of y variable with "+var_name, fontsize=15)
plt.show()


I am working on the file "BenzDatasetAnalysis.ipynb".
The first task is:
Good to see that there are no missing values in the dataset :)
Integer Columns Analysis:
Can you write Python code for it?

unique_values_dict = {}
for col in train_df.columns:
    if col not in ["ID", "y", "X0", "X1", "X2", "X3", "X4", "X5", "X6", "X8"]:
        unique_value = str(np.sort(train_df[col].unique()).tolist())
        tlist = unique_values_dict.get(unique_value, [])
        tlist.append(col)
        unique_values_dict[unique_value] = tlist[:]
for unique_val, columns in unique_values_dict.items():
    print("Columns containing the unique values : ",unique_val)
    print(columns)
    print("--------------------------------------------------")


I am working on the file "BenzDatasetAnalysis.ipynb".
The first task is:
Now let us have a look at the data type of all the variables present in the dataset.
Can you write Python code for it?

dtype_df = train_df.dtypes.reset_index()
dtype_df.columns = ["Count", "Column Type"]
dtype_df.groupby("Column Type").aggregate('count').reset_index()


I am working on the file "BenzDatasetAnalysis.ipynb".
The first task is:
Seems like a random split of ID variable between train and test samples.
Important Variables:
Now let us run and xgboost model to get the important variables.
Can you write Python code for it?

for f in ["X0", "X1", "X2", "X3", "X4", "X5", "X6", "X8"]:
        lbl = preprocessing.LabelEncoder()
        lbl.fit(list(train_df[f].values))
        train_df[f] = lbl.transform(list(train_df[f].values))

train_y = train_df['y'].values
train_X = train_df.drop(["ID", "y", "eval_set"], axis=1)

# Thanks to anokas for this #
def xgb_r2_score(preds, dtrain):
    labels = dtrain.get_label()
    return 'r2', r2_score(labels, preds)

xgb_params = {
    'eta': 0.05,
    'max_depth': 6,
    'subsample': 0.7,
    'colsample_bytree': 0.7,
    'objective': 'reg:linear',
    'silent': 1
}
dtrain = xgb.DMatrix(train_X, train_y, feature_names=train_X.columns.values)
model = xgb.train(dict(xgb_params, silent=0), dtrain, num_boost_round=100, feval=xgb_r2_score, maximize=True)

# plot the important features #
fig, ax = plt.subplots(figsize=(12,18))
xgb.plot_importance(model, max_num_features=50, height=0.8, ax=ax)
plt.show()


I am working on the file "BenzDatasetAnalysis.ipynb".
The first task is:
Seems like a single data point is well above the rest.
Now let us plot the distribution graph.
Can you write Python code for it?

ulimit = 180
train_df['y'].ix[train_df['y']>ulimit] = ulimit

plt.figure(figsize=(12,8))
sns.distplot(train_df.y.values, bins=50, kde=False)
plt.xlabel('y value', fontsize=12)
plt.show()


I am working on the file "BenzDatasetAnalysis.ipynb".
The first task is:
So all the integer columns are binary with some columns have only one unique value 0. Possibly we could exclude those columns in our modeling activity.
Now let us explore the categorical columns present in the dataset.
Can you write Python code for it?

var_name = "X0"
col_order = np.sort(train_df[var_name].unique()).tolist()
plt.figure(figsize=(12,6))
sns.stripplot(x=var_name, y='y', data=train_df, order=col_order)
plt.xlabel(var_name, fontsize=12)
plt.ylabel('y', fontsize=12)
plt.title("Distribution of y variable with "+var_name, fontsize=15)
plt.show()

var_name = "X1"
col_order = np.sort(train_df[var_name].unique()).tolist()
plt.figure(figsize=(12,6))
sns.stripplot(x=var_name, y='y', data=train_df, order=col_order)
plt.xlabel(var_name, fontsize=12)
plt.ylabel('y', fontsize=12)
plt.title("Distribution of y variable with "+var_name, fontsize=15)
plt.show()

var_name = "X2"
col_order = np.sort(train_df[var_name].unique()).tolist()
plt.figure(figsize=(12,6))
sns.boxplot(x=var_name, y='y', data=train_df, order=col_order)
plt.xlabel(var_name, fontsize=12)
plt.ylabel('y', fontsize=12)
plt.title("Distribution of y variable with "+var_name, fontsize=15)
plt.show()

var_name = "X3"
col_order = np.sort(train_df[var_name].unique()).tolist()
plt.figure(figsize=(12,6))
sns.violinplot(x=var_name, y='y', data=train_df, order=col_order)
plt.xlabel(var_name, fontsize=12)
plt.ylabel('y', fontsize=12)
plt.title("Distribution of y variable with "+var_name, fontsize=15)
plt.show()

var_name = "X4"
col_order = np.sort(train_df[var_name].unique()).tolist()
plt.figure(figsize=(12,6))
sns.violinplot(x=var_name, y='y', data=train_df, order=col_order)
plt.xlabel(var_name, fontsize=12)
plt.ylabel('y', fontsize=12)
plt.title("Distribution of y variable with "+var_name, fontsize=15)
plt.show()

var_name = "X5"
col_order = np.sort(train_df[var_name].unique()).tolist()
plt.figure(figsize=(12,6))
sns.boxplot(x=var_name, y='y', data=train_df, order=col_order)
plt.xlabel(var_name, fontsize=12)
plt.ylabel('y', fontsize=12)
plt.title("Distribution of y variable with "+var_name, fontsize=15)
plt.show()

var_name = "X6"
col_order = np.sort(train_df[var_name].unique()).tolist()
plt.figure(figsize=(12,6))
sns.boxplot(x=var_name, y='y', data=train_df, order=col_order)
plt.xlabel(var_name,...


I am working on the file "BenzDatasetAnalysis.ipynb".
The first task is:
So majority of the columns are integers with 8 categorical columns and 1 float column (target variable)
Can you write Python code for it?


I am working on the file "BenzDatasetAnalysis.ipynb".
The first task is:
Target Variable:
"y" is the variable we need to predict. So let us do some analysis on this variable first.
Can you write Python code for it?

plt.figure(figsize=(8,6))
plt.scatter(range(train_df.shape[0]), np.sort(train_df.y.values))
plt.xlabel('index', fontsize=12)
plt.ylabel('y', fontsize=12)
plt.show()


I am working on the file "BenzDatasetAnalysis.ipynb".
The first task is:
There seems to be a slight decreasing trend with respect to ID variable. Now let us see how the IDs are distributed across train and test.
Can you write Python code for it?

plt.figure(figsize=(6,10))
train_df['eval_set'] = "train"
test_df['eval_set'] = "test"
full_df = pd.concat([train_df[["ID","eval_set"]], test_df[["ID","eval_set"]]], axis=0)

plt.figure(figsize=(12,6))
sns.violinplot(x="eval_set", y='ID', data=full_df)
plt.xlabel("eval_set", fontsize=12)
plt.ylabel('y', fontsize=12)
plt.title("Distribution of ID variable with evaluation set", fontsize=15)
plt.show()


I am working on the file "BenzDatasetAnalysis.ipynb".
The first task is:
Wow the number of rows are small with 388 columns. We should try not to overfit :)
Let us look at the top few rows.
Can you write Python code for it?


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Accessing data in a Table
A column of Table data is accessed by indexing it by name.  This returns the column as a numpy array.
Can you write Python code for it?


I am working on the file "BerkeleySalary.ipynb".
The first task is:
As is often the case in the real world, the categorization used for operations is not directly useful for analysis.  We often need to build categories in order to get a handle on what's going on.
What do all those job titles look like
Can you write Python code for it?

ordered_by_title['Job Title']

categories = {
    'Police': ["POLICE"],
    'Fire': ["FIRE"],
    'Animal Control':["ANIMAL"],
    'Health': ["HEALTH", "PSYCH", "HLTH"],
    'Library': ['LIBRARY','LIBRARIAN'],
    'Offical' : ['MAYOR','COUNCIL', 'COMMISSIONER', 'CITY MANAGER'],
    'Trades' :["ELECTRICIAN","MECHANIC", "ENGINEER"],
    'Parking' : ["PARKING"],
    'Recreation' : ["RECREATION", "AQUATICS"],
    'Gardener' : ["GARDEN"],
    "Labor" : ["LABOR", "JANITOR"],
    'Community': ["COMMUNITY"],
    'Admin' : ["ADMIN"],
    'Traffic' : ["TRAFFIC"],
    'Accounting' : ["ACCOUNT"],
    'Dispatch' : ["DISPATCH"],
    'Waste' : ["WASTE", "SEWER"],
    'Analyst' : ["ANALYS"],
    'Office' : ["OFFICE "],
    'Legal' : ['LEGISLAT', 'ATTORN', 'ATTY'],
    'IT' : ["PROG", "INFORMATION SYSTEMS"],
    'School' : ["SCHOOL"],
    'Architect' : ["ARCHITECT"],
    'Planner' : ["PLANNER", "PERMIT"]
    }

categories


I am working on the file "BerkeleySalary.ipynb".
The first task is:
As is often the case working with real data, we often need to iterate a bit to get what we want out of it.  With all those titles, a lot of stuff is likely to end up as other.
Here we have a little iterative process to get enough of the job titles categorized
Can you write Python code for it?

job_categories.where('Category', 'Other')

job_categories.where('Category', 'Other').group('Job Title',sum).sort('Cat sum', descending=True)

job_categories.where('Category', 'Other').group('Job Title',sum).sort('Total Pay Benefits sum', descending=True)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Base pay seems to chunk into categories, perhaps by job title.
Most members of the force do little overtime, but a few do a lot!
How many are in each Job Title?
Can you write Python code for it?

police.group('Job Title')

# We can actually get all the data by title
police.select(['Job Title','Base Pay', 'Overtime Pay']).group('Job Title', collect=lambda x:x)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Clean derivatives of raw tables
It is good hygene to keep the raw data raw and produce distinct, clean derivatives.  Let's start by making a copy of the raw table.  A new name and a new table.
Can you write Python code for it?


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Descriptive Statistics Summary
Now we can try to get a summary of the data with some descriptive statistics.
The stats method on Tables computes a list of statistics over each column and creates a new Table containing these statistics.  The default is tailored to the Berkeley Data8 course. Here we provide what you expect from the summary operation in R
Can you write Python code for it?

def firstQtile(x) : return np.percentile(x,25)
def thirdQtile(x) : return np.percentile(x,25)
summary_ops = (min, firstQtile, np.median, np.mean, thirdQtile, max)

berkeley_sal_2011.select(paylabels).stats(ops=summary_ops)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Grouping and Sorting Table data

Table.drop creates a new Table without some columns.  It is like select, but you don't have to name everythng you want.
Table.group aggregates data by grouping all the rows that contain a common value in one (or more) columns.  Here we group in "Job Title" summing the entries in all other columns for each group.  We placed a column full of 1 to get a count, while summing salaries and such.
Table.sort sorts the rows in a Table by a column - just like sort in Excel.
Can you write Python code for it?

# lose the individual names
job_titles = berkeley_sal_2011.drop("Employee Name")
# Build a handy column full of 1s
job_titles["Title"] = 1
# Group by title summing the number of rows per
by_title = job_titles.group("Job Title", sum)
# Sort by the number of employees per title
ordered_by_title = by_title.sort('Title sum', descending = True)
# let's see what we get
ordered_by_title

ordered_by_title.num_rows


I am working on the file "BerkeleySalary.ipynb".
The first task is:
How about that, a few job categories have most of the employes and most of the spend, but it is far from uniform.  Let's look a little deeper.  Which categories consume most of the budget?
Can you write Python code for it?

by_title.sort('Total Pay Benefits sum', descending = True)

by_title.sort('Total Pay Benefits sum', descending = True).row(0)

by_title.select(('Job Title', 'Total Pay Benefits sum')).sort('Total Pay Benefits sum', descending=True)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
How do the pay labels spread across the force?
We can look at histograms by pay label.  First all toegether and then broken apart.
Can you write Python code for it?

police.select(paylabels).hist(bins=30,normed=False)

police.select(paylabels).hist(bins=30,normed=False, overlay=False)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Interesting.  Base pay is bimodal.  Most employees get no overtime, but there is a looong tale.  Let's look at the individual level.  Who's at the top?
Can you write Python code for it?

berkeley_sal_2011.sort('Total Pay Benefits', descending=True)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Let's compare the 2011 data with more recent 2013 data.
Can you write Python code for it?

raw_berkeley_sal_2013 = Table.read_table("./data/City_of_Berkeley_Employee_Salaries_-_2013.csv")
raw_berkeley_sal_2013


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Let's see who got promoted or demoted
Can you write Python code for it?

b11_13.where(b11_13['Job Title'] != b11_13['Title 2013']).sort('Total 2013', descending=True).show()

b11_13.where(b11_13['Job Title'] == b11_13['Title 2013']).sort('Increase', descending=True).show()


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Now we can put the two tables together to see what happened with employees who were around in both years.  Here we get to use another powerful operations on tables.

Table.join: joins two tables together using a column of each that contains common values.

Here we have the employee names in each table.  The join will give us the title and salary in both years for those employees in both tables, i.e., working for the city in both years
Can you write Python code for it?


I am working on the file "BerkeleySalary.ipynb".
The first task is:
OK, so it looks like the average salary is about 86k, and it ranges up to 300k with some hefty overtime pay.  Let's see if we can understand what is going on.
Visualizing data

Table.select creates a new table consisting of the specified list of columns.
Table.hist plots a histogram of each of the columns in a table.  It can either overlay the histograms or show them separately.  Here we have specified the number of bins
Can you write Python code for it?

berkeley_sal_2011.select(["Base Pay",
                          "Overtime Pay",
                          "Total Pay Benefits"]).hist(overlay=False,normed=False,
                                                      bins=40)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
On the tails we have people who joined part way through 2011 or left part way through 2013.
Can you write Python code for it?


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Perhaps we might want to look at the relationship of these two variables.  That leads to another useful operator

Table.scatter: does a scatter plot of columns against one columns
Can you write Python code for it?


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Plotting data

Table.plot plots each of the columns in a table, either on separate charts or overlayed on a single chart.  Optionally one of the columns can be specified as the horizontal axis and all others plotted against this.
Can you write Python code for it?

ordered_by_title.select(['Title sum','Total Pay Benefits sum']).sort('Title sum').plot(overlay=False)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Reading raw data into a Table
Let's pull a big wad of City Employee salary data from the Berkeley Open Data portal.
Since this was a trial till June 30, 2015 and you have to mouse around to get csv files, we happen to have pulled a local copy.
Can you write Python code for it?

raw_berkeley_sal_2011 = Table.read_table("./data/City_of_Berkeley_Employee_Salaries_-_2011.csv")

# Tables are rendered to work in a notebook, even if they are large.  Only part shows.
# You can adjust how much of it you see, but here we'd have ~1500 rows!
raw_berkeley_sal_2011


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Snap! They decided that case was a good idea for proper nouns.  Let's go back to the old way.
Can you write Python code for it?

b2013['Employee Name'] = b2013.apply(str.upper, 'Employee Name')
b2013


I am working on the file "BerkeleySalary.ipynb".
The first task is:
So let's try to understand the police category a bit more.
Can you write Python code for it?

police = job_categories.where('Category', 'Police')
police


I am working on the file "BerkeleySalary.ipynb".
The first task is:
So no job title left has more than 10 employees in it, but some have quite a bit of cost.  We could go back and add more entries to our category table and iterate a bit.  The important thing is that we create new tables, we don't clobber old ones.
Well this shows the challenge in managing budget pretty nicely.  Most of the money is spent in a few job categories.  But then there are still over 200 employees in a zillion other categories that are stile the #2 spend.
Can you write Python code for it?

by_categories.sort('Total Pay Benefits sum', descending=True).barh('Category', select=['Total Pay Benefits sum', 'Cat sum'], overlay=False)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
So where does the $alary go? First, how many employees?
Table.num_rows returns just what you'd think. The number of rows.  Which in this case is the number of employees on the city payroll.
Can you write Python code for it?


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Tables allow columns to have customized formatters
In Excel you do this by formatting the cells.  We want to have the data as numbers, keep track of the type, and have it look nice.
Can you write Python code for it?


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Well that's interesting.  Total compensation seems to have dropped.  Did the budget actually go down?
Can you write Python code for it?

sum2011 = np.sum(berkeley_sal_2011['Total Pay Benefits'])
"${:,}".format(int(sum2011))

sum2013 = np.sum(berkeley_sal_2013['Total Pay & Benefits'])
"${:,}".format(int(sum2013))

"${:,}".format(int(sum2013-sum2011))

"{:.1%}".format((sum2013-sum2011)/sum2011)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Well, the data base changed.  It picked up a few columns over the years. And we need to convert the salary strings to numbers so we can do analysis on them.  All in one go...
Can you write Python code for it?

berkeley_sal_2013 = raw_berkeley_sal_2013.drop(['Year','Notes','Agency'])
berkeley_sal_2013

berkeley_sal_2013.set_format(berkeley_sal_2013.labels[2:], CurrencyFormatter)

berkeley_sal_2013["Total Pay & Benefits"]

berkeley_sal_2013.sort('Total Pay & Benefits',descending=True)


I am working on the file "BerkeleySalary.ipynb".
The first task is:
Wow, 305 Job Titles for 1437 employees!
Can you write Python code for it?

"{0:.3} employees per Job Code".format(berkeley_sal_2011.num_rows/ordered_by_title.num_rows)


I am working on the file "Bernoulli/Bernoulli hierarchical.ipynb".
The first task is:
Now a 4-sided die and pull out hierarchial info
Get the right answer from the data plain
Can you write Python code for it?

# fair fwould be all the same, we want different
# 1/4 = 0.25, so we want [10%, 20%, 30%, 40%]
observations = np.array([50*.1, 50*.2, 50*.3 ,50*.4])
with pm.Model():
    probs = pm.Dirichlet('probs', a=np.ones(4))  # flat prior
    rolls = pm.Multinomial('rolls', n=observations.sum(), p=probs, observed=observations)
    trace = pm.sample(5000)
# pm.traceplot(trace)
pm.plot_posterior(trace);


I am working on the file "Better4Industry/Feature_Selection_Collection/try_shap_xgboost.ipynb".
The first task is:
Check Individual Cases

It can only show 2 records here, the red one means higher feature value, the blue one means lower feature value. But none of them means it's high or low in the whole population.
When a record is at the top left, it means it tends to contribute to the negative class/lower value prediction; top right means it tends to contribute to the positive class/higher value prediction.
But just today, in the real work, we saw opposite cases, where real high probability positive class has feature values all indicate to negative class..
Can you write Python code for it?

shap.summary_plot(shap_values[7:9, :], X.iloc[7:9, :])

# comparing with xgboost feature importance (by default it's using gain to rank fearure importance)
print(X.columns)
model.feature_importances_

# using absolute mean value of SHAP values to rank the features
shap.summary_plot(shap_values, X, plot_type="bar")


I am working on the file "Better4Industry/Feature_Selection_Collection/try_shap_xgboost.ipynb".
The first task is:
Summarize Feature Importance
Summary Plot

Higher feature value (pink) with longer length means higher value has larger impact to the prediction. Towards left means impact more on lower value/negative class, towards right means impoact more on higher value/positive class.
From upper to lower, features are ranked by importance decreasing order
Can you write Python code for it?

shap.summary_plot(shap_values, X)

X.iloc[7:9, :]

shap_values[7:9, :]


I am working on the file "Better4Industry/explore_features_fastai.ipynb".
The first task is:
Confidence of tree variance

average of predicion value
variance
Can you write Python code for it?

%time
preds = np.stack([t.predict(X_test) for t in m.estimators_])  # each estimator is a tre
print(preds)
np.mean(preds[:,0]), np.std(preds[:,0])


I am working on the file "Better4Industry/explore_features_fastai.ipynb".
The first task is:
Out Of Bag Score (OOB Score)

The idea for OOB score is to calculate error on the training set, but only include the trees in the calculation of a row's error where that row was not included in training that tree. This allows us to see whether the model is over-fitting, without needing a separate validation set.
The result here can also indicate that Item_MRP is very important to the prediction, because if we remove this feature, the OOB score dropped dramatically
Can you write Python code for it?

def split_vals(a,n):
    return a[:n], a[n:]

def get_oob(df, n_trn, y_train):
    m = RandomForestRegressor(n_estimators=30, min_samples_leaf=5, max_features=0.6, n_jobs=-1, oob_score=True)
    x, _ = split_vals(df, n_trn)
    m.fit(x, y_train)
    return m.oob_score_

get_oob(X_train, len(X_train), y_train)

# remove each feature and check oob score

for col in X_train.columns:
    print(col, get_oob(X_train.drop(col, axis=1), len(X_train), y_train))


I am working on the file "Better4Industry/explore_features_fastai.ipynb".
The first task is:
Plot Feature Confidence

It's in fact using pandas plot, you just need the feature importance is a dataframe.
Sklearn randomforest can generate feature importance and do the same plot if it's dataframe
Can you write Python code for it?

fi = rf_feat_importance(m, X_train)
fi

def plot_fi(fi):
    return fi.plot('cols', 'imp', 'barh', figsize=(12,7), legend=False, color='g')

plot_fi(fi)


I am working on the file "Better4Industry/explore_features_fastai.ipynb".
The first task is:
Tree Interpreter

Predictions are the values predicted by the random forest model
Bias is the average value of the target variable for the dataset
Contributions are the amount by which the predicted value was changed by each column. Negative value means negative impact on the prediction.
Can you write Python code for it?

from treeinterpreter import treeinterpreter as ti

X_test.reset_index(drop=True, inplace=True)
y_test.reset_index(drop=True, inplace=True)

check_idx = 0
for i in range(len(X_test)):
    if i == check_idx:
        r = X_test.values[None,i]
        print('True Value:', y_test.values[i])
        prediction, bias, contributions = ti.predict(m, r)
        print('Prediction', prediction)
        print('Average of the Prediction', bias)
        print('Contribution:', [elem for elem in zip(X_test.columns, X_test.iloc[0], contributions[0])])
        break

contribution_dct = {}
for i in range(len(X_test)):
    if i == 10:
        break
    r = X_test.values[None,i]
    prediction, bias, contributions = ti.predict(m, r)
    contribution_dct[i] = dict(zip(X_test.columns, contributions[0]))

contribution_df = pd.DataFrame(contribution_dct).T
contribution_df.head()

avg_contribution = contribution_df.apply(np.mean)
avg_contribution


I am working on the file "Bezier-triangular-patch-in-plotly.ipynb".
The first task is:
Data for a cubic (degree  $n=3$) triangular Bézier patch and its control polyhedron:
Can you write Python code for it?


I am working on the file "Bezier-triangular-patch-in-plotly.ipynb".
The first task is:
Finally we generate our first Bézier patch of degree $n=4$, over a grid corresponding to  $m=80$:
Can you write Python code for it?


I am working on the file "Bezier-triangular-patch-in-plotly.ipynb".
The first task is:
Hence to generate a Bézier triangular patch of total degree $3$ we start with a net of points:
$$\begin{array}{lllllll}
   & &  &          {\bf b}{300} &        &  &  \
   & & & & & & \
   &  & {\bf b}{210} &           & {\bf b}{201} &  &  \
    & & & & & & \
   & {\bf b}{120} &  &      {\bf b}{111}    &  & {\bf b}{102} &  \
    & & & & & & \
  {\bf b}{030} &  & {\bf b}{021}&          & {\bf b}{012} &  & {\bf b}{003}
\end{array}$$
The point indices in a triangular net or the indices that define points in  a triangular grid  of $\Delta$,  as above,
 are returned by the following function::
Can you write Python code for it?

def indexes(n):
    return [(i,j, n-(i+j)) for i in range(n,-1, -1) for j in range(n-i, -1, -1)]

print indexes(3)


I am working on the file "Bezier-triangular-patch-in-plotly.ipynb".
The first task is:
In order to plot the discretized patch we need to construct three arays of shape (2m+1,2m+1) (or a list of lists) that contain in an appropriate position
the x, y, respectively z-coordinates of the computed points.
The  values in a list of coordinates are   placed  succesively in the red positions, while outside   the triangle, similar to the triangle $A_1A_1A_2$, we store None values:
Can you write Python code for it?


I am working on the file "Bezier-triangular-patch-in-plotly.ipynb".
The first task is:
Let us define a Bézier triangular graph of degree $n=4$, over the triangle $T$:
Can you write Python code for it?