I am working on the file "Applied Plotting in Python/Week 2/Week 2 - course.ipynb".
The first task is:
Scatterplots
Can you write Python code for it?

import numpy as np

x = np.array([1,2,3,4,5,6,7,8])
y = x

plt.figure()
plt.scatter(x, y) # similar to plt.plot(x, y, '.'), but the underlying child objects in the axes are not Line2D

import numpy as np

x = np.array([1,2,3,4,5,6,7,8])
y = x

# create a list of colors for each point to have
# ['green', 'green', 'green', 'green', 'green', 'green', 'green', 'red']
colors = ['green']*(len(x)-1)
colors.append('red')

plt.figure()

# plot the point with size 100 and chosen colors
plt.scatter(x, y, s=100, c=colors)

# convert the two lists into a list of pairwise tuples
zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])

print(list(zip_generator))
# the above prints:
# [(1, 6), (2, 7), (3, 8), (4, 9), (5, 10)]

zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])
# The single star * unpacks a collection into positional arguments
print(*zip_generator)
# the above prints:
# (1, 6) (2, 7) (3, 8) (4, 9) (5, 10)

# use zip to convert 5 tuples with 2 elements each to 2 tuples with 5 elements each
print(list(zip((1, 6), (2, 7), (3, 8), (4, 9), (5, 10))))
# the above prints:
# [(1, 2, 3, 4, 5), (6, 7, 8, 9, 10)]

zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])
# let's turn the data back into 2 lists
x, y = zip(*zip_generator) # This is like calling zip((1, 6), (2, 7), (3, 8), (4, 9), (5, 10))
print(x)
print(y)
# the above prints:
# (1, 2, 3, 4, 5)
# (6, 7, 8, 9, 10)

plt.figure()
# plot a data series 'Tall students' in red using the first two elements of x and y
plt.scatter(x[:2], y[:2], s=100, c='red', label='Tall students')
# plot a second data series 'Short students' in blue using the last three elements of x and y
plt.scatter(x[2:], y[2:], s=100, c='blue', label='Short students')

# add a label to the x axis
plt.xlabel('The number of times the child kicked a ball')
# add a label to the y axis
plt.ylabel('The grade of the student')
# add a title
plt.title('Relationship between ball kicking and grades')

# add a legend (uses the labels from plt.scatter)
plt.legend()

# add the legend to loc=4 (the lower right hand corner), also...


I am working on the file "Applied Plotting in Python/Week 2/Week 2 - course.ipynb".
The first task is:
We can use html cell magic to display the image.
Can you write Python code for it?

%%html
<img src='test.png' />

# create a new figure
plt.figure()

# plot the point (3,2) using the circle marker
plt.plot(3, 2, 'o')

# get the current axes
ax = plt.gca()

# Set axis properties [xmin, xmax, ymin, ymax]
ax.axis([0,6,0,10])

# create a new figure
plt.figure()

# plot the point (1.5, 1.5) using the circle marker
plt.plot(1.5, 1.5, 'o')
# plot the point (2, 2) using the circle marker
plt.plot(2, 2, 'o')
# plot the point (2.5, 2.5) using the circle marker
plt.plot(2.5, 2.5, 'o')

# get current axes
ax = plt.gca()
# get all the child objects the axes contains
ax.get_children()


I am working on the file "Applied Plotting in Python/Week 4/.ipynb_checkpoints/Week4-checkpoint.ipynb".
The first task is:
DataFrame.plot
Can you write Python code for it?

np.random.seed(123)

df = pd.DataFrame({'A': np.random.randn(365).cumsum(0),
                   'B': np.random.randn(365).cumsum(0) + 20,
                   'C': np.random.randn(365).cumsum(0) - 20},
                  index=pd.date_range('1/1/2017', periods=365))
df.head()

df.plot(); # add a semi-colon to the end of the plotting call to suppress unwanted output


I am working on the file "Applied Plotting in Python/Week 4/.ipynb_checkpoints/Week4-checkpoint.ipynb".
The first task is:
Kernel density estimation plots are useful for deriving a smooth continuous function from a given sample.
Can you write Python code for it?


I am working on the file "Applied Plotting in Python/Week 4/.ipynb_checkpoints/Week4-checkpoint.ipynb".
The first task is:
We can select which plot we want to use by passing it into the 'kind' parameter.
Can you write Python code for it?


I am working on the file "Applied Plotting in Python/Week 4/.ipynb_checkpoints/Week4-checkpoint.ipynb".
The first task is:
You can also choose the plot kind by using the DataFrame.plot.kind methods instead of providing the kind keyword argument.
kind :
- 'line' : line plot (default)
- 'bar' : vertical bar plot
- 'barh' : horizontal bar plot
- 'hist' : histogram
- 'box' : boxplot
- 'kde' : Kernel Density Estimation plot
- 'density' : same as 'kde'
- 'area' : area plot
- 'pie' : pie plot
- 'scatter' : scatter plot
- 'hexbin' : hexbin plot
Can you write Python code for it?

# create a scatter plot of columns 'A' and 'C', with changing color (c) and size (s) based on column 'B'
df.plot.scatter('A', 'C', c='B', s=df['B'], colormap='viridis')

ax = df.plot.scatter('A', 'C', c='B', s=df['B'], colormap='viridis')
ax.set_aspect('equal')

df.plot.box();

df.plot.hist(alpha=0.7);


I am working on the file "Applied Plotting in Python/Week 4/.ipynb_checkpoints/Week4-checkpoint.ipynb".
The first task is:
pandas.tools.plotting
Iris flower data set
Can you write Python code for it?

iris = pd.read_csv('iris.csv')
iris.head()

pd.tools.plotting.scatter_matrix(iris);

plt.figure()
pd.tools.plotting.parallel_coordinates(iris, 'Name');


I am working on the file "Applied Voxelization Computations.ipynb".
The first task is:
If we take a D = two centimeter edge tetrahedron and call that our unit, six of which then form a rhombic dodecahedron, and then apply 1/S3, a conversion constant, we'll have a corresponding cubic volume measure.
Dividing our total volume by the voxel volume gives the number of voronoi cells, or voxels.  That's assuming a two centimeter long face diagonal.
However, that was not our initial assumption, as a baseball has a diameter of 7.4 cm.  Lets set R = 1 cm (a reference sphere radius) and D = 2 cm (the same sphere's diameter), but have our larger baseball diameter in terms of D (2 cm intervals).
The tetrahedron of edges 7.4/2 or 3.7 D, will be a 1/6th of the RD in question.  That's thinking in tetravolumes, units smaller than R-edged cubes.
The R-cube to D-tetrahedron ratio is what we call S3 or $\sqrt{9/8}$.  We switch back and forth between these two units of volume by means of this constant.  This practice derives from Buckminster Fuller's Synergetics: Explorations in the Geometry of Thinking.
Tetravolumes times 1/S3 then gives the equivalent volume in plain old cubic centimeters (cm3).
A tetrahedron of edges 7.4 cm has a tetravolume of $(7.4/2)^3$ meaning the RD-shaped voronoi cell surrounding the baseball would have a conventional cubic centimeter volume of...
Can you write Python code for it?

S3 = (9/8)**(1/2)
RD_cm3 = (1/S3) * 6 * (7.4/2) ** 3  # 2 cm unit i.e. 2R
print("Voronoi Cell volume in cubic centimeters (cm3): ", RD_cm3)

from math import ceil
conversion = 100**3 # cubic centimeters
david_cm3 = 2.16 * conversion
N = ceil(david_cm3 / RD_cm3) # round up
print("Number of Voxels per Original David = ~{:,}".format(N))


I am working on the file "Applied Voxelization Computations.ipynb".
The first task is:
What is the volume of the model David?  TBD
What is the volume of the original David?  2.16 m3 i.e. 2,160,000 cm3.

Michelangelo's David for 3D Printer
Our technique will be to start with the number of cubic centimeters in the original David, a figure we might obtain directly, by searching, thereby short circuiting the 3D printing and displacement step.
However for generic objects, a 3D printed approximation with water displacement and then a scale factor applied, may be the most practical way to get a measure.
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo("0v86Yk14rf8")


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Compare CTP (Click Through Probability)

Because CTP is a proportion in the population, so we need to use pooled standard deviation to calculate the margin of error.
p_pool = (experiment_clicks + control_clicks)/(experiment_pageviews + control_pageviews)
std_pool = sqrt(p_pool*(1-p_pool)*(1/experiment_pageviews + 1/control_pageviews))
Can you write Python code for it?

control_ctp = control_clicks/control_pageviews
experiment_ctp = experiment_clicks/experiment_pageviews
p_pool = (control_clicks + experiment_clicks)/(control_pageviews + experiment_pageviews)
std_pool = math.sqrt(p_pool*(1-p_pool)*(1/experiment_pageviews + 1/control_pageviews))
me = get_marginOferror(std_pool, alpha)

diff = round(experiment_ctp - control_ctp, 4)

print('If ' + str(diff) +' is within [' + str(round(0 - me, 4)) + ', ' + str(round(0 + me, 4)) + '], then the difference is expected.')


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Compare Net Conversion

<b>Observation</b>
As we can see, it's not statistically significant but it's practically significant.
From control group to experiment group, there is 0.49% drop. Practically it's significant means there will be less payment for free trail option, comparing with access to materials option. This drop will affect the business.
Can you write Python code for it?

control_clicks = control_df['Clicks'].loc[control_df['Payments'].notnull()].sum()
experiment_clicks = experiment_df['Clicks'].loc[experiment_df['Payments'].notnull()].sum()
print('Clicks', control_clicks, experiment_clicks)

control_paid = control_df['Payments'].sum()
experiment_paid = experiment_df['Payments'].sum()
print('Payments', control_paid, experiment_paid)

control_NC = control_paid/control_clicks
experiment_NC = experiment_paid/experiment_clicks
print('Net Conversion', control_NC, experiment_NC)

p_pool = (control_paid + experiment_paid)/(control_clicks + experiment_clicks)
std_pool = math.sqrt(p_pool*(1-p_pool)*(1/control_clicks + 1/experiment_clicks))
me = get_marginOferror(std_pool, alpha)

print(p_pool, std_pool, me)

# Statistical significance
NC_diff = round(experiment_NC - control_NC, 4)

print('If ' + str(NC_diff) +' is within [' + str(round(0 - me, 4)) + ', ' + str(round(0 + me, 4)) + '], then the difference is expected, and the change is not significant.')

# Practically significance
d_min = net_conversion['d_min']

print('If ' + str(NC_diff) +' is within [' + str(round(d_min - me, 4)) + ', ' + str(round(d_min + me, 4)) + '], then the change is not practically significant.')


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Compare pageviews

We want to verify that the difference between pageview counts in the 2 groups are not significant.
When sample size n is large enough, it can be approximated as normal distribution. We want to test that pbserved p_hat = control group pageviews/both groups pageviews is not significantly different from p=0.5.
Margin of Error ME = Z1-α/2 * std
We need to calculate ME at 95% confidence interval

Then we will get Confidence Interval CI = [p_hat - ME, p_hat + ME]
If p=0.5 is within CI, then the difference between the 2 groups are expected
Can you write Python code for it?

control_pageviews = control_df['Pageviews'].sum()
experiment_pageviews = experiment_df['Pageviews'].sum()

print(control_pageviews, experiment_pageviews)

total_pageviews = control_pageviews + experiment_pageviews
p_hat = control_pageviews/(total_pageviews)
std = get_std(p, total_pageviews)
me = get_marginOferror(std, alpha)

print('If ' + str(p) +' is within [' + str(round(p_hat - me, 4)) + ', ' + str(round(p_hat + me, 4)) + '], then the difference is expected.')


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Gross Conversion Sample Size

Calculated sample_size means estimated number of enrolled in each group
In order to get estimated page_views (unique cookies), needs to use sample_size/(400/5000) * 2 for both control and experiments groups
multuple 2 means for both groups
Can you write Python code for it?

gross_conversion['sample_size'] = round(get_sample_size(get_stds(gross_conversion['p_hat'],
                                                          gross_conversion['d_min']), alpha, beta,
                                                 gross_conversion['d_min']))
gross_conversion['page_views'] = 2*round(gross_conversion['sample_size']/(gross_conversion['n']/5000))

gross_conversion


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Gross Conversion std

p_hat = enrolled/Cl
probability of daily enrolled among daily clicked free trial button
Can you write Python code for it?

gross_conversion = {}

gross_conversion['d_min'] = 0.01
gross_conversion['p_hat'] = sample_baseline['GConversion']
gross_conversion['n'] = sample_baseline['Clicks']
gross_conversion['std'] = get_binomial_std(gross_conversion['p_hat'],
                                          gross_conversion['n'])

gross_conversion


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Net Conversion Sample Size

Calculated sample_size means estimated number of paid in each group
In order to get estimated page_views (unique cookies), needs to use sample_size/(400/5000) * 2 for both control and experiments groups
multuple 2 means for both groups
Assume 40000 page_views per day, in order to get such amount page views, it takes about 3 weeks.
Can you write Python code for it?

net_conversion['sample_size'] = round(get_sample_size(get_stds(net_conversion['p_hat'],
                                                         net_conversion['d_min']), alpha, beta,
                                                         net_conversion['d_min']))
net_conversion['page_views'] = 2*round(net_conversion['sample_size']/(net_conversion['n']/5000))

net_conversion


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Net Conversion std
Can you write Python code for it?

net_conversion = {}

net_conversion['d_min'] = 0.0075
net_conversion['p_hat'] = sample_baseline['NConversion']
net_conversion['n'] = sample_baseline['Clicks']
net_conversion['std'] = get_binomial_std(net_conversion['p_hat'],
                                        net_conversion['n'])

net_conversion


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Retention Sample Size

Calculated sample_size means estimated number of paid in each group
In order to get estimated page_views (unique cookies), needs to use sample_size/(82.5/5000) * 2 for both control and experiments groups
multuple 2 means for both groups
The required page_views is too large, if 40000 views a day, it will take 100 days to get the data. So this metric Retention might be given up.
Can you write Python code for it?

retention['sample_size'] = round(get_sample_size(get_stds(retention['p_hat'],
                                                          retention['d_min']), alpha, beta,
                                                 retention['d_min']))
retention['page_views'] = 2*round(retention['sample_size']/(retention['n']/5000))
retention


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Revention std
Can you write Python code for it?

retention = {}

retention['d_min'] = 0.01
retention['p_hat'] = sample_baseline['Retention']
retention['n'] = sample_baseline['Enrollments']
retention['std'] = get_binomial_std(retention['p_hat'],
                                   retention['n'])

retention


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Step 1 - Choose Metrics
Invariate Metrics

Ck = unique daily cookies (pageviews) on a page
Dmin = 3000
Cl = unique daily clicks on the free trial button
Dmin = 240
CTP = Cl/Ck, free trial button click through probability
Dmin = 0.01

Evaluation Metrics

GConversion = enrolled/Cl
It's gross conversion
Dmin = 0.01
Probability of daily enrolled among daily clicked free trial button
Retention = paid/enrolled
Dmin = 0.01
Probability of daily paid among daily enrolled
NConversion = paid/Cl
It's net conversion
Dmin = 0.01
Probability of daily paid among daily clicked free trial button

Step 2.1 - Estimate Metrics Baseline Values

The baseline values are the values of these metrics before the change.
These values are given by the data provider Udacity, it's their rough estimation.
Can you write Python code for it?

baseline = {'Cookies': 40000, 'Clicks': 3200, 'Enrollments': 660, 'CTP': 0.08,
           'GConversion': 0.20625, 'Retention': 0.53, 'NConversion': 0.109313}


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Step 2.2 Estimate Standard Deviation & Sample Size
Estimate Standard Deviation of Metrics

This is for later estimating sample size, confidence interval.
The more variant of a metric, the more difficult to reach to a significant result.
In order to estimate variance, assume metrics probabilities p_hat are binomially distributed, (probability density function PDF is binomially distribution), then standard deviation is:
std = sqrt(p_hat*(1-p_hat)/n)
p_hat: baseline probability of the event to occur
n: sample size

The reason we assume it's binomial distribution for probability density function is because, the logic here is if not option A then option B.
The assumption is only valid when the unit of diversion of the experiment is equal to the unit of analysis (the denominator of the metric formula). If the assumption is not valid, the calculated std can be different and better to estimate empirically.
Can you write Python code for it?

sample_baseline = baseline.copy()

sample_baseline['Cookies'] = 5000  # assume sample size is 5000
sample_baseline['Clicks'] = baseline['Clicks'] * 5000/baseline['Cookies']
sample_baseline['Enrollments'] = baseline['Enrollments'] * 5000/baseline['Cookies']

sample_baseline

def get_binomial_std(p_hat, n):
    """
    p_hat: baseline probability of the event to occur
    n: sample size
    return: the standard deviation
    """
    std = round(math.sqrt(p_hat*(1-p_hat)/n),4)

    return std


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Step 3 - Control Group vs. Experiment Group
Can you write Python code for it?


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Step 3.1 - Differences in Invariant Metrics (Sanity Check)

The goal is to verify the experiment is conducted as expected, and won't be affected by other factors. Also to make sure the data collection is correct.
Invariant Metrics
Ck = unique daily cookies (pageviews) on a page
Cl = unique daily clicks on the free trial button
CTP = Cl/Ck, free trial button click through probability
<b>We need to compare the invariant metrics in both groups, to make sure the differences are not significant.</b>
Can you write Python code for it?

p=0.5
alpha=0.05

def get_std(p, total_size):
    std = math.sqrt(p*(1-p)/total_size)
    return std

def get_marginOferror(std, alpha):
    me = round(get_z_score(1-alpha/2)*std, 4)
    return me


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Step 3.3 - Differences in Trending Sign Test

The purpose of this test is to check whether the decrease/increase trend is evident in daily data.
prob(success) = (n!/(x! * (n-x)!)) * pow(p, x) * pow(1-p, n-x)
x - number of "success", success means when experiment group increased from control group in the record
n - total records
p - the probability of being success, this is binomial distribution, so p=0.5
p-value is the sum of prob(success) from each success record. When p-value is smaller than alpha, then the success is significant.
This is the online calculator to get p-value: https://www.graphpad.com/quickcalcs/binomial1/
Given x, n, p
It provides both single & double sided p-value
Observation
Same as what found in Step 3.2, the changes in Gross Conversion is significant, while in Net Conversion is not statistically significant.
One thing to note here is, no matter to check prob(success) or prob(failure) here, the significance results are the same, even though p_value can be different.
Can you write Python code for it?

control_experiment_df = control_df.join(experiment_df, lsuffix='_control', rsuffix='_experiment')
print(control_experiment_df.shape)
control_experiment_df.head()

control_experiment_df.isnull().sum()

control_experiment_df.dropna(inplace=True)
print(control_experiment_df.shape)
control_experiment_df.isnull().sum()

# If it's "success", assign 1, otherwise 0

control_experiment_df['GC_increase'] = np.where(
    control_experiment_df['Enrollments_experiment']/control_experiment_df['Clicks_experiment'] \
    > control_experiment_df['Enrollments_control']/control_experiment_df['Clicks_control'], 1, 0)

control_experiment_df['NC_increase'] = np.where(
    control_experiment_df['Payments_experiment']/control_experiment_df['Clicks_experiment'] \
    > control_experiment_df['Payments_control']/control_experiment_df['Clicks_control'], 1, 0)

control_experiment_df[['GC_increase', 'NC_increase']].head()

print(control_experiment_df['GC_increase'].value_counts())
print(control_experiment_df['NC_increase'].value_counts())

GC_success_ct = control_experiment_df['GC_increase'].value_counts()[1]
NC_success_ct = control_experiment_df['NC_increase'].value_counts()[1]

print(GC_success_ct, NC_success_ct)

p = 0.5
alpha = 0.05
n = control_experiment_df.shape[0]

print(n)

def get_probability(x, n):
    prob = round(math.factorial(n)/(math.factorial(x)*math.factorial(n-x))*pow(p,x)*pow(1-p, n-x), 4)
    return prob

def get_p_value(x, n):
    p_value = 0

    for i in range(0, x+1):
        p_value += get_probability(i, n)

    return round(p_value*2, 4)  # 2 side p_value

print ("GC Change is significant if", get_p_value(GC_success_ct,n), "is smaller than", alpha)
print ("NC Change is significant if", get_p_value(NC_success_ct,n), "is smaller than", alpha)


I am working on the file "Applied_Statistics/ABTest_Experiments/detailed_ABTest_Experiment.ipynb".
The first task is:
Summary for Sanity Check

We have checked all the 3 invariant metrics between the 2 groups, all showing the differences are not significant, so the experiments of using these 2 groups should be less likely affected by other factors.

Step 3.2 - Differences in Evaluation Metrics

Similar to the sanity check above, here is to check the differences in evaluation metrics between the 2 groups,to see:
Whether the differences are statistically significant.

Whether the differences are practically significant.

So that the changes are big enough to be beneficial to the business.
Difference is not included in the confidence interval [Dmin - ME, Dmin + ME]

As Step 2 has found, Gross Conversion and Net Conversion can be the evaluation metrics, while Retention is not. All because of the limitation in data collection in reality.

Evaluation Metrics

GConversion = enrolled/Cl
It's gross conversion
Dmin = 0.01
Probability of daily enrolled among daily clicked free trial button

NConversion = paid/Cl
It's net conversion
Dmin = 0.01
Probability of daily paid among daily clicked free trial button
Can you write Python code for it?


I am working on the file "Applied_Statistics/Learning_Notes/ANOVA_practice.ipynb".
The first task is:
One Way ANOVA

H0: Item_Outlet_Sales has no difference among Outlet_Types

H1: Item_Outlet_Sales has a significant difference among Outlet_Types

The idea is similar to feature importance, to check test whether this 1 independent variable (Outlet_Types) is important to the dependent variable (Item_Outlet_Sales).
Can you write Python code for it?


I am working on the file "Applied_Statistics/Learning_Notes/ANOVA_practice.ipynb".
The first task is:
💡 BTW, with q-q plot here, it's close to the diagnoal line, so the data should be close to normal distribution.
Post Hoc Test

Since we have found there is a significant different among groups (factor levels), now we want to find where the group differences lay.
The Post hoc test method used here is "Tukey HSD test".
Can you write Python code for it?

mc = MultiComparison(selected_df['transformed_sales'], selected_df['Outlet_Type'])
post_hoc_results = mc.tukeyhsd()
print(post_hoc_results)


I am working on the file "Applied_Statistics/Learning_Notes/ANOVA_practice.ipynb".
The first task is:
💡 Both method rejected the null-hypothesis with very low p-value. So there is a significant difference between group variance. Homogeneity doesn't stand. However, since all the groups have equal size, the results of ANOVA can still be trustworthy.
ANOVA test
Can you write Python code for it?

F, p = stats.f_oneway(selected_df.loc[selected_df['Outlet_Type']==0]['transformed_sales'],
                     selected_df.loc[selected_df['Outlet_Type']==1]['transformed_sales'],
                     selected_df.loc[selected_df['Outlet_Type']==2]['transformed_sales'],
                     selected_df.loc[selected_df['Outlet_Type']==3]['transformed_sales'])
print(F, p)


I am working on the file "Applied_Statistics/Learning_Notes/ANOVA_practice.ipynb".
The first task is:
💡 NOTE

From OLS we can check both model p-value and the p-value from residual output.
Both p-value is low (< 0.05, even less than 0.01 significance level), indicating H0 should be rejected, sales is significant between outlet types, and therefore outlet types is important.
Can you write Python code for it?

import statsmodels.api as sm
res = model.resid
fig = sm.qqplot(res, line='s', color='green')
plt.show()


I am working on the file "Applied_Statistics/Learning_Notes/bayesian4fun/chapter3_expt2.ipynb".
The first task is:
Assign μ， σ of the Cluster to each Observation
Can you write Python code for it?

with model:
    sds = pm.Uniform("sds", 0, 100, shape=2)
    centers = pm.Normal("centers",
                        mu=np.array([120, 200]),
                        sd=np.array([10, 10]),
                        shape=2)

    center_i = pm.Deterministic('center_i', centers[assignment])
    sd_i = pm.Deterministic('sd_i', sds[assignment])

    # combine with the real observations:
    observations = pm.Normal("obs", mu=center_i, sd=sd_i, observed=data)

print("Random assignments: ", assignment.tag.test_value[:10], "...")
print("Assigned center: ", center_i.tag.test_value[:10], "...")
print("Assigned standard deviation: ", sd_i.tag.test_value[:10], "...")
print("Observations: ", observations.tag.test_value[:10], "...")
print(data[:10], "...")

print(sds.tag.test_value)
print(centers.tag.test_value)


I am working on the file "Applied_Statistics/Learning_Notes/bayesian4fun/chapter3_expt2.ipynb".
The first task is:
Define Sampling Methods to Explore Define Space

Metropolis() for our continuous variables
CategoricalGibbsMetropolis() for categorical variable
Sample 25000 iterations

The results are saved in trace, which records the value of specificed vars for each observation

These vars are, p, sds, centers and assignment

NOTE:

I tried find_map() before sample(), but it simply doesn't work
Can you write Python code for it?

with model:
    step1 = pm.Metropolis(vars=[p, sds, centers])
    step2 = pm.CategoricalGibbsMetropolis(vars=[assignment])
    trace = pm.sample(25000, step=[step1, step2])

print(trace['centers'].shape)
print(trace['p'].shape)
print(trace['sds'].shape)
print(trace['assignment'].shape)

plt.figure(figsize=(12,3))
lw = 1
center_trace = trace["centers"]

plt.plot(center_trace[:, 0], label="cluster 0", c='g', lw=lw)
plt.plot(center_trace[:, 1], label="cluster 1", c='purple', lw=lw)
plt.xlabel('Steps')
plt.title("Traces of Centers")
leg = plt.legend(loc="upper right")
leg.get_frame().set_alpha(0.7)

plt.figure(figsize=(12,3))
lw = 1
center_trace = trace["sds"]

plt.plot(std_trace[:, 0], label="cluster 0", c='g', lw=lw)
plt.plot(std_trace[:, 1], label="cluster 1", c='purple', lw=lw)
plt.xlabel('Steps')
plt.title('Traces of Standard Deviations')
leg = plt.legend(loc="upper right")
leg.get_frame().set_alpha(0.7)

plt.figure(figsize=(12,3))
lw = 1
p_trace = trace["p"]

plt.plot(p_trace, label="cluster 0", c='g', lw=lw)
plt.xlabel('Steps')
plt.title('Traces of Cluster 0 Assignment Frequency')
leg = plt.legend(loc="upper right")
leg.get_frame().set_alpha(0.7)

# Sample 50,000 more to achieve more convergence
with model:
    trace = pm.sample(50000, step=[step1, step2], trace=trace)

figsize(12, 3)
prev_center_trace = trace["centers"][:25000]
center_trace = trace["centers"][25000:]

x = np.arange(25000)
plt.plot(x, prev_center_trace[:, 0], label="previous center 0",
     lw=lw, alpha=0.4, c='green')
plt.plot(x, prev_center_trace[:, 1], label="previous center 1",
     lw=lw, alpha=0.4, c='purple')

x = np.arange(25000, 300000)
plt.plot(x, center_trace[:, 0], label="new center 0", lw=lw, c="green")
plt.plot(x, center_trace[:, 1], label="new center 1", lw=lw, c="purple")

plt.title("Previous & New traces of Centers")
leg = plt.legend(loc="upper right")
leg.get_frame().set_alpha(0.8)
plt.xlabel("Steps");

# Plot posterior clusters
figsize(12, 3)
plt.title("Posterior of Centers")
plt.hist(center_trace[:, 0], bins=30,...


I am working on the file "Applying Machine Learning Techniques-Regression.ipynb".
The first task is:
Classification accuracy: percentage of correct predictions
Can you write Python code for it?

#After being fitted, the model can then be used to predict the class of samples:
y_pred_class = clf.predict(x_test);

#Alternatively, the probability of each class can be predicted,
#which is the fraction of training samples of the same class in a leaf:
clf.predict_proba(x_test);

# calculate accuracy
from sklearn import metrics

print(metrics.accuracy_score(y_test, y_pred_class))


I am working on the file "Applying Machine Learning Techniques-Regression.ipynb".
The first task is:
Classifier comparison
http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html
A comparison of a several classifiers in scikit-learn on synthetic datasets. The point of this example is to illustrate the nature of decision boundaries of different classifiers. This should be taken with a grain of salt, as the intuition conveyed by these examples does not necessarily carry over to real datasets.
Particularly in high-dimensional spaces, data can more easily be separated linearly and the simplicity of classifiers such as naive Bayes and linear SVMs might lead to better generalization than is achieved by other classifiers.
The plots show training points in solid colors and testing points semi-transparent. The lower right shows the classification accuracy on the test set.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from matplotlib.colors import ListedColormap

#classifiers

#x_train

#sns.pairplot(x_train)

x_train_scaled = StandardScaler().fit_transform(x_train)

x_test_scaled = StandardScaler().fit_transform(x_test)

x_train_scaled[0]

len(x_train_scaled[0])

df_x_train_scaled = pd.DataFrame(columns=x_train.columns, data=x_train_scaled)

df_x_train_scaled.head()

#sns.pairplot(df_x_train_scaled)

names = ["Nearest Neighbors", "Linear SVM", "RBF SVM",
         "Decision Tree", "Random Forest", "Neural Net", "AdaBoost",
         "Naive Bayes", "QDA", "Gaussian Process"]

classifiers = [
    KNeighborsClassifier(3),
    SVC(kernel="linear", C=0.025),
    SVC(gamma=2, C=1),
    DecisionTreeClassifier(max_depth=5),
    RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1),
    MLPClassifier(alpha=1),
    AdaBoostClassifier(),
    GaussianNB(),
    QuadraticDiscriminantAnalysis()
    #, GaussianProcessClassifier(1.0 * RBF(1.0), warm_start=True), # Take too long...
    ]

# iterate over classifiers
for name, model in zip(names, classifiers):
    model.fit(x_train_scaled, y_train)
    acc_score = model.score(x_test_scaled, y_test)
    print(name, " - accuracy score: ", acc_score)
#end for


I am working on the file "Applying Machine Learning Techniques-Regression.ipynb".
The first task is:
Conclusion: ???

Classification accuracy is the easiest classification metric to understand
But, it does not tell you the underlying distribution of response values
And, it does not tell you what "types" of errors your classifier is making

Confusion matrix
Table that describes the performance of a classification model
Can you write Python code for it?

# IMPORTANT: first argument is true values, second argument is predicted values
print(metrics.confusion_matrix(y_test, y_pred_class))


I am working on the file "Applying Machine Learning Techniques-Regression.ipynb".
The first task is:
Null accuracy: accuracy that could be achieved by always predicting the most frequent class
Can you write Python code for it?

# examine the class distribution of the testing set (using a Pandas Series method)
y_test.value_counts()

# calculate the percentage of ones
y_test.mean()

# calculate the percentage of zeros
1 - y_test.mean()

# calculate null accuracy (for binary classification problems coded as 0/1)
max(y_test.mean(), 1 - y_test.mean())

# calculate null accuracy (for multi-class classification problems)
y_test.value_counts().head(1) / len(y_test)


I am working on the file "Applying Machine Learning Techniques-Regression.ipynb".
The first task is:
Polynomialy Regression
Quadratic Regression, regession with a second order polynomial, is given by the following formula:
$$y = \alpha +\beta_1x^1+\beta_2x^2$$
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

model = LinearRegression()

model.fit(x_train, y_train)

xx = np.linspace(0, 26, 100)
#yy = np.linspace(0, 26, 100)

#yy = model.predict(xx.reshape(xx.shape[0],1))

#plt.plot(xx, yy)

quadratic_featurizer = PolynomialFeatures(degree=2)

x_train_quadratic = quadratic_featurizer.fit_transform(x_train)
x_test_quadratic = quadratic_featurizer.fit(x_test)

x_train.head()

model_quadratic = LinearRegression()

model_quadratic.fit(x_train_quadratic, y_train)

#predictions = model_quadratic.predict(x_test_quadratic)

#r_squared = model_quadratic.score(x_test_quadratic, y_test)

#r_squared

#print("R-squared: %.4f" %r_squared)


I am working on the file "Applying Machine Learning Techniques-Regression.ipynb".
The first task is:
Random Forests
Naive Bayes
Simple Linear Regression
Recall that Simple Linear Regression is given by the following equation:
$y = \alpha + \beta x$
Our goal is to solve the values $\alpha$ and $\beta$ that minimize the cost function.
$$\beta = \frac{cov(x,y)}{var(x)}$$
where $cov(x,y)$ denotes a measure of how far a set of values is spread out.
Note that:
* Variance is zero if all of the features are spread out equally.
* A SMALL variance indicates that the numbers are NEAR the mean of the set
* A LARGE variance when the numbers are FAR the mean of the set
$$var(x) = \frac{\sum\limits_{i=1}^{n}{\left( x_i - \overline{x} \right)}}{n-1}$$
$$cov(x,y) = \frac{\sum\limits_{i=1}^{n}{\left( x_i - \overline{x} \right)\left( y_i - \overline{y} \right)}}{n-1}$$
Having solved $\beta$, we can estimate $\alpha$ using the following formula:
$$\alpha = \overline{y} - \beta \overline{x}$$
Evaluating the Model
Using r-squared - that measures how well the observed values of the response variables are predicted by the model. In the case of simple linear regression, r-squared is equal to Pearson's r. In this method, r-squared must be  a positive number between zero and one. In others, r-squared can return a negative number if the model performs extremely poorly.
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

model = LinearRegression()

model.fit(x_train, y_train)

r_squared = model.score(x_test, y_test)

print("R-squared: %.4f" %r_squared)


I am working on the file "Applying Machine Learning Techniques-Regression.ipynb".
The first task is:
Specificity: When the actual value is negative, how often is the prediction correct?

How "specific" (or "selective") is the classifier in predicting positive instances?
Can you write Python code for it?


I am working on the file "Applying SentiWordNet to News Data.ipynb".
The first task is:
And from there we can aggregate up to the monthly range, to match our CCI indicator:
Can you write Python code for it?

monthly_sentiment = data_subset[["date", "score", "count"]].set_index("date").resample("MS").sum()
monthly_sentiment["avg_score"] = monthly_sentiment["score"] / monthly_sentiment["count"]
monthly_sentiment.head()

daily_sentiment.to_csv("./monthly_sentiment_1990_2016.csv")


I am working on the file "Applying SentiWordNet to News Data.ipynb".
The first task is:
As we can see, the scores are very nicely distributed around 0. Some examples of positive sentiment articles:
Can you write Python code for it?


I am working on the file "Applying SentiWordNet to News Data.ipynb".
The first task is:
Generating sentiment scores with SentiWordNet
First, we need to load the entire NYTimes article corpus and load the sentiment analysis routine we created (from sentiwordnet.py).
Can you write Python code for it?

DATA_DIR = "./data/"

all_data_list = []
for year in range(1990,2017):
    data = pd.read_csv(DATA_DIR + '{}_Output.csv'.format(year), header=None, encoding="utf-8")
    all_data_list.append(data) # list of dataframes
data = pd.concat(all_data_list, axis=0)
data.columns = ['id','date','headline', 'lead']

# Drop dupes and fix dates
data = data.drop_duplicates("id").reset_index(drop=True)
data.date = pd.to_datetime(data.date)

# Fill null articles
data.loc[data.lead.isnull(), "lead"] = ""

# Can be found in sentiwordnet.py, or also the SentiWordNet notebook.

# If it complains about missing corpuses, try:
# nltk.download("sentiwordnet")
# nltk.download("stopwords")

import sentiwordnet


I am working on the file "Applying SentiWordNet to News Data.ipynb".
The first task is:
Now, we can aggregate sentiment data up to days:
Can you write Python code for it?

daily_sentiment = data_subset[["date", "score", "count"]].set_index("date").resample("D").sum()
daily_sentiment["avg_score"] = daily_sentiment["score"] / daily_sentiment["count"]
daily_sentiment.head()


I am working on the file "Applying SentiWordNet to News Data.ipynb".
The first task is:
Now, we can also reduce the data to only the first two weeks of each month, to see if we can later use this as a predictor for the CCI, which gets released at the beginning of every month:
Can you write Python code for it?

first_two = daily_sentiment[daily_sentiment.index.day < 15]

first_two.head()

plt.figure(figsize=(15,5))
plt.title("Sentiment scores from 1990-2016")
first_two.avg_score.resample("MS").mean().plot(label="First two weeks of the month")
daily_sentiment.avg_score.resample("MS").mean().plot(label="Monthly score")
plt.legend()


I am working on the file "Applying SentiWordNet to News Data.ipynb".
The first task is:
Some examples of negative snetiment articles:
Can you write Python code for it?


I am working on the file "Applying SentiWordNet to News Data.ipynb".
The first task is:
We can then load these up again later, and take a look at the format:
Can you write Python code for it?

years = range(1990, 2017)

sentiments = []
for year in years:
    #print(year)
    sentiment = pd.read_csv("./sentiments_{}.csv".format(year))
    sentiments.append(sentiment)

sentiments = pd.concat(sentiments)
sentiments.head()

data_subset = data[data.date.dt.year.isin(years)].set_index("id")
data_subset = data_subset.join(sentiments.set_index("id"))


I am working on the file "Approximate Sparse Filtrations.ipynb".
The first task is:
Now let's run an approximate version and plot H1 for both next to each other
Questions

What happens to the approximation when you change $\epsilon$?  What happens to the runtime?
Can you write Python code for it?

eps = 0.05

#Cmpute the sparse filtration
tic = time.time()
#First compute all pairwise distances and do furthest point sampling
D = pairwise_distances(X, metric='euclidean')
lambdas = getGreedyPerm(D)
#Now compute the sparse distance matrix
DSparse = getApproxSparseDM(lambdas, eps, D)
#Finally, compute the filtration
resultsparse = ripser(DSparse, distance_matrix=True)
toc = time.time()
timesparse = toc-tic
percent_added = 100.0*float(resultsparse['num_edges'])/resultfull['num_edges']
print("Elapsed Time: %.3g seconds, %i Edges added"%(timesparse, resultsparse['num_edges']))

#Plot the sparse distance matrix and edges that were added
plt.figure(figsize=(10, 5))
plt.subplot(121)
D = pairwise_distances(X, metric='euclidean')
plt.imshow(D)
plt.title("Original Distance Matrix: %i Edges"%resultfull['num_edges'])
plt.subplot(122)
DSparse = DSparse.toarray()
DSparse = DSparse + DSparse.T
plt.imshow(DSparse)
plt.title("Sparse Distance Matrix: %i Edges"%resultsparse['num_edges'])

#And plot the persistence diagrams on top of each other
plt.figure(figsize=(10, 5))
plt.subplot(121)
plot_diagrams(resultfull['dgms'], show=False)
plt.title("Full Filtration: Elapsed Time %g Seconds"%timefull)
plt.subplot(122)
plt.title("Sparse Filtration (%.3g%% Added)\nElapsed Time %g Seconds"%(percent_added, timesparse))
plot_diagrams(resultsparse['dgms'], show=False)


I am working on the file "Approximate Sparse Filtrations.ipynb".
The first task is:
Now, we define a "greedy permutation," or a function which performs furthest points sampling, a key step in the algorithm used to determine "insertion radii" $\lambda_i$ for each point (this is similar to a cover tree).  For an animation of how this works, please visit:
https://gist.github.com/ctralie/128cc07da67f1d2e10ea470ee2d23fe8
Can you write Python code for it?

def getGreedyPerm(D):
    """
    A Naive O(N^2) algorithm to do furthest points sampling

    Parameters
    ----------
    D : ndarray (N, N)
        An NxN distance matrix for points

    Return
    ------
    lamdas: list
        Insertion radii of all points
    """

    N = D.shape[0]
    #By default, takes the first point in the permutation to be the
    #first point in the point cloud, but could be random
    perm = np.zeros(N, dtype=np.int64)
    lambdas = np.zeros(N)
    ds = D[0, :]
    for i in range(1, N):
        idx = np.argmax(ds)
        perm[i] = idx
        lambdas[i] = ds[idx]
        ds = np.minimum(ds, D[idx, :])
    return lambdas[perm]


I am working on the file "Approximate Sparse Filtrations.ipynb".
The first task is:
Now, we define a function which, given a distance matrix representing a point cloud, a set of insertion radii, and an approximation factor $\epsilon$, returns a sparse distance matrix with re-weighted edges, whose persistence diagrams are each guaranteed to be a $(1+\epsilon)$ multiplicative approximation of the true persistence diagrams (see [1])
Can you write Python code for it?

def getApproxSparseDM(lambdas, eps, D):
    """
    Purpose: To return the sparse edge list with the warped distances, sorted by weight

    Parameters
    ----------
    lambdas: list
        insertion radii for points
    eps: float
        epsilon approximation constant
    D: ndarray
        NxN distance matrix, okay to modify because last time it's used

    Return
    ------
    DSparse: scipy.sparse
        A sparse NxN matrix with the reweighted edges
    """
    N = D.shape[0]
    E0 = (1+eps)/eps
    E1 = (1+eps)**2/eps

    #Create initial sparse list candidates (Lemma 6)
    nBounds = ((eps**2+3*eps+2)/eps)*lambdas #Search neighborhoods
    D[D > nBounds[:, None]] = np.inf #Set all distances outside of search neighborhood to infinity
    [I, J] = np.meshgrid(np.arange(N), np.arange(N))
    idx = I < J
    I = I[(D < np.inf)*(idx == 1)]
    J = J[(D < np.inf)*(idx == 1)]
    D = D[(D < np.inf)*(idx == 1)]

    #Prune sparse list and update warped edge lengths (Algorithm 3 pg. 14)
    minlam = np.minimum(lambdas[I], lambdas[J])
    maxlam = np.maximum(lambdas[I], lambdas[J])
    #Rule out edges between vertices whose balls stop growing before they touch
    #or where one of them would have been deleted.  M stores which of these
    #happens first
    M = np.minimum((E0 + E1)*minlam, E0*(minlam + maxlam))
    #M = E0*(minlam+maxlam)

    t = np.arange(len(I))
    t = t[D <= M]
    (I, J, D) = (I[t], J[t], D[t])
    minlam = minlam[t]
    maxlam = maxlam[t]

    #Now figure out the metric of the edges that are actually added
    t = np.ones(len(I))
    t[D <= 2*minlam*E0] = 0 #If cones haven't turned into cylinders, metric is unchanged
    #Otherwise, if they meet before the M condition above, the metric is warped
    D[t == 1] = 2.0*(D[t == 1] - minlam[t == 1]*E0) #Multiply by 2 convention
    return sparse.coo_matrix((D, (I, J)), shape=(N, N)).tocsr()


I am working on the file "Approximate Sparse Filtrations.ipynb".
The first task is:
We can also compute the Wasserstein distance between the two diagrams to see how close they are.  Try this with different approximations above
Can you write Python code for it?

#Show the Wasserstein distance for H1
I1 = resultfull['dgms'][1]
I2 = resultsparse['dgms'][1]
matchdist, (matchidx, D) = wasserstein(I1, I2, matching=True)
plt.figure(figsize=(5, 5))
#Plot wasserstein matching
wasserstein_matching(I1, I2, matchidx, labels = ['exact', '$\epsilon = %.3g$'%eps])
plt.title("Wasserstein Dist: %.3g"%matchdist)


I am working on the file "Appunti vari.ipynb".
The first task is:
Divisione tra numeri interi
In python esistono due operatori per effettuare la divisione tra due numeri $a$ e $b$:

\ effettua la divisione come una "normale" calcolatrice (a meno della precisione numerica)
\\ viene chiamato il floor division operator ed effettua la divisione "intera", ovvero $\lfloor \frac{a}{b} \rfloor$
Can you write Python code for it?

a = 2.41
b = 2.42
print(a/b)
print(a//b)
print(b//a)


I am working on the file "Appunti vari.ipynb".
The first task is:
Se viene richiesta una precionse più alta si possono utilizzare due librerie diverse:

decimal — Decimal fixed point and floating point arithmetic
fractions — Rational numbers

Vediamo un esempio per la seconda libreria.
Can you write Python code for it?

from fractions import Fraction
a = Fraction(1,10)
b = Fraction(1,10)
c = Fraction(1,10)
d = Fraction(3,10)
a + b + c - d == 0


I am working on the file "Appunti vari.ipynb".
The first task is:
Sympy: Matematica simbolica
Sympy è una libreria Python per effettuare dei calcoli matematici simbolici. Per un approfondimento su come è stata sviluppata questa libreria si consiglia di leggere l'articolo SymPy: symbolic computing in Python o direttamente la documentazione della libreria.
Seguono un paio di esempi per rendere l'idea di cosa si può calcolare con Sympy.
Can you write Python code for it?

from sympy import *
x,y = symbols('x y')
init_printing(use_unicode=True)

diff(sin(x)*exp(-x**2), x)

integrate(cos(x), (x,0,3))


I am working on the file "Appunti vari.ipynb".
The first task is:
ovvero la differenza non è nulla. Questo va considerato soprattutto quando si effetuano dei confronti tra due numeri float:
Can you write Python code for it?


I am working on the file "Arahanov-Bohm Oscillations/Cylindrical Shell and Arahanov-Bohm Oscillations.ipynb".
The first task is:
I am studying the variation of T near the Fermi level.
Transmission function map
Transmission function map is T calculated as a function of the energy and the flux.
Can you write Python code for it?

def calc_T(flux):
    print("Flux : ",flux)
    params = Parameters()
    params.N_z = 2
    params.N_phi = 30
    params.flux = flux
    params.t_z = 1
    params.t_phi = 1*np.exp(1j*2*np.pi*params.flux/params.N_phi)
    params.mu = 0
    params.kT = 1e-3

    E_linspace = np.linspace(0,8,100)
    T = np.array([calc_current(E,params) for E in E_linspace])
    return T

T_map = np.array([calc_T(flux) for flux in np.linspace(0,6,100)])

XX,YY = np.meshgrid(np.linspace(0,6,100),np.linspace(0,8,100))
plt.figure(1)
plt.pcolor(XX,YY,T_map.T,cmap="coolwarm")
plt.xlabel(r"$\frac{\Phi}{\Phi_0}$",fontsize=16)
plt.ylabel("Energy/t",fontsize=16)
cbar = plt.colorbar()
cbar.set_label('Transmission (T) [a.u.]')

plt.plot(T_map.T[70,:])


I am working on the file "Arahanov-Bohm Oscillations/Cylindrical Shell and Arahanov-Bohm Oscillations.ipynb".
The first task is:
I now want to introduce a magnetic field in the simulation. I think one way to do it is to add a phase to t_phi parameter.
Can you write Python code for it?

for i in range(10):
    params = Parameters()
    params.N_z = 1
    params.N_phi = 50
    params.flux = 5*i
    params.t_z = 1
    params.t_phi = 1*np.exp(1j*2*np.pi*params.flux/N_phi)
    params.mu = 2

    H = calc_H(params)
    E_linspace = np.linspace(0,4,1000)
    dos = np.array([calc_DOS(E,H) for E in E_linspace])

    #plt.figure(i)
    plt.plot(E_linspace,dos)
    plt.xlabel('Energy (E)')
    plt.ylabel('Density of States(E) [a.u.]')
    #plt.title("flux" + str(params.flux))

# code for transport

def calc_current(E,params):
    if params.N_z < 2:
        raise Exception("Calculation of current requires atleast two logitudianl points in the device.")
    eta = 1e-10
    # calcualtion of the surface functions
    def surface_g(E,parmas):
        dummy_contact_params = Parameters()
        dummy_contact_params.N_z = 2
        dummy_contact_params.N_phi = params.N_phi
        dummy_contact_params.flux = params.flux
        dummy_contact_params.t_z = params.t_z
        dummy_contact_params.t_phi = params.t_phi
        dummy_contact_params.mu = params.mu

        H_mat = calc_H(dummy_contact_params)
        N_dof_lat = params.N_phi
        alpha = H_mat[:N_dof_lat,:N_dof_lat]
        beta = H_mat[:N_dof_lat,N_dof_lat:2*N_dof_lat]

        err = 1.0
        iter_count = 0
        iter_limit = 100000
        err_limit = 1e-6

        g = np.linalg.inv((E + 1j*eta)*np.eye(alpha.shape[0]) - alpha)
        g_old = np.linalg.inv((E + 1j*eta)*np.eye(alpha.shape[0]) - alpha)
        # iterate over iter_limit iterations or until err < err_limit
        for i in range(iter_limit):
            g = np.linalg.inv((E + 1j*eta)*np.eye(alpha.shape[0]) - alpha - np.dot(np.dot(np.conj(beta.T),g),beta))
            g = 0.5*(g + g_old)

            err = np.linalg.norm(g-g_old)/np.sqrt(np.linalg.norm(g)*np.linalg.norm(g_old))
            g_old = g
            if(err < err_limit):
                #print("Finished at",i,"Error :",err)
                break;
            if(i == (iter_limit - 1)):
               ...


I am working on the file "Arahanov-Bohm Oscillations/Cylindrical Shell and Arahanov-Bohm Oscillations.ipynb".
The first task is:
I will now simulate a thick nanowire with less points in the length and more points in the circumference. The different l sub-bands are observed.
Can you write Python code for it?

params = Parameters()
params.t_z = 1
params.flux = 0
params.t_phi = 1*np.exp(1j*2*np.pi*params.flux)
params.N_z = 1
params.N_phi = 50
params.mu = 2

H = calc_H(params)
E_linspace = np.linspace(-4,6,10000)
dos = np.array([calc_DOS(E,H) for E in E_linspace])

plt.plot(E_linspace,dos)
plt.xlabel('Energy (E)')
plt.ylabel('Density of States(E) [a.u.]')


I am working on the file "Archimedes and Pi.ipynb".
The first task is:
And the result is:
Can you write Python code for it?


I am working on the file "Archimedes and Pi.ipynb".
The first task is:
Our function aprox_pi() will compute the aproximation of $\pi$ for a $2^n$ polygon:
Can you write Python code for it?

def aprox_pi(n = 10):
    s = 2.0
    for i in range(1, n):
        s = side_next(s)
    return 2.0**(n-1) * s


I am working on the file "Archimedes and Pi.ipynb".
The first task is:
So, what's going on? We should expect better approximations as the number of sides increases, right? But, the best result we get is with 14 iterations and a polygon of 16384 sides. After that the approximations of $\pi$ get worse.
The problem is that our algorithm is not very good in terms of producing the end result. If you look at the expression $P_n = 2^n \times s_n$ what we are doing is multiplying a very large number (the number of sides) by a very small number (the length of a side). After a certain point, because we are using floating point, this is a recipe for disaster. In particular, for a 16384-sided polygon, the length of a side is approximatly:
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Exercises/05_APS Captcha.ipynb".
The first task is:
Preprocessing
Can you write Python code for it?

import os
from skimage import io
from skimage.color import rgb2gray
from skimage import transform
from math import ceil

IMGSIZE = (100, 100)

def load_images(folder, scalefactor=(2, 2), labeldict=None):
    images = []
    labels = []
    files = os.listdir(folder)

    for file in (fname for fname in files if fname.endswith('.png')):

        img = io.imread(folder + file).astype(float)
        img = rgb2gray(img)
        # Crop since some of the real world pictures are other shape
        img = img[:IMGSIZE[0], :IMGSIZE[1]]
        # Possibly downscale to speed up processing
        img = transform.downscale_local_mean(img, scalefactor)
        # normalize image range
        img -= np.min(img)
        img /= np.max(img)
        images.append(img)

        if labeldict is not None:
            # lookup label for real world data in dict generated from labels.txt
            key, _ = os.path.splitext(file)
            labels.append(labeldict[key])
        else:
            # infere label from filename
            if file.find("einstein") > -1 or file.find("curie") > -1:
                labels.append(1)
            else:
                labels.append(0)

    return np.asarray(images)[:, None], np.asarray(labels)

x_train, y_train = load_images('data/aps/train/')
# Artifically pad Einstein's and Curie't to have balanced training set
# ok, since we use data augmentation later anyway
sel = y_train == 1
repeats = len(sel) // sum(sel) - 1
x_train = np.concatenate((x_train[~sel], np.repeat(x_train[sel], repeats, axis=0)),
                         axis=0)
y_train = np.concatenate((y_train[~sel], np.repeat(y_train[sel], repeats, axis=0)),
                         axis=0)

x_test, y_test = load_images('data/aps/test/')

rw_labels = {str(key): 0 if label == 0 else 1
             for key, label in np.loadtxt('data/aps/real_world/labels.txt', dtype=int)}
x_rw, y_rw = load_images('data/aps/real_world/', labeldict=rw_labels)

from mpl_toolkits.axes_grid import ImageGrid
from math...


I am working on the file "Archiv_Session_Spring_2017/Exercises/05_APS Captcha.ipynb".
The first task is:
So training on raw pixel values might not be a good idea. Let's build a feature extractor based on the trained LeNet (or any other pretrained image classifier)
Can you write Python code for it?

model = load_model('aps_lenet.h5')
enc_layers = it.takewhile(lambda l: not isinstance(l, keras.layers.Flatten),
                          model.layers)
encoder_model = keras.models.Sequential(enc_layers)
encoder_model.add(keras.layers.Flatten())
x_train_sampled_enc = encoder_model.predict(x_train_sampled, verbose=True)

rfe = RandomForestClassifier(n_estimators=64, criterion='entropy', n_jobs=-1,
                             verbose=True)
rfe = rfe.fit(x_train_sampled_enc, y_train_sampled)

y_pred_rw = rfe.predict(encoder_model.predict(x_rw, verbose=False))
plot_cm(confusion_matrix(y_rw, y_pred_rw), normalize=True,
        classes=["Not Einstein", "Einstein"])
pl.show()


I am working on the file "Archiv_Session_Spring_2017/Exercises/05_APS Captcha.ipynb".
The first task is:
Training LeNet
First, we will train a simple CNN with a single hidden fully connected layer as a classifier.
Can you write Python code for it?

from keras.layers import Conv2D, Dense, Flatten, MaxPooling2D
from keras.models import Sequential
from keras.backend import image_data_format

def generate(figsize, nr_classes, cunits=[20, 50], fcunits=[500]):
    model = Sequential()
    cunits = list(cunits)
    input_shape = figsize + (1,) if image_data_format == 'channels_last' \
        else (1,) + figsize

    model.add(Conv2D(cunits[0], (5, 5), padding='same',
                     activation='relu', input_shape=input_shape))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Convolutional layers
    for nr_units in cunits[1:]:
        model.add(Conv2D(nr_units, (5, 5), padding='same',
                         activation='relu'))
        model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Fully connected layers
    model.add(Flatten())
    for nr_units in fcunits:
        model.add(Dense(nr_units, activation='relu'))

    # Output layer
    activation = 'softmax' if nr_classes > 1 else 'sigmoid'
    model.add(Dense(nr_classes, activation=activation))

    return model

from keras.optimizers import Adam
from keras.models import load_model

try:
    model = load_model('aps_lenet.h5')
    print("Model succesfully loaded...")
except OSError:
    print("Saved model not found, traing...")
    model = generate(figsize=x_train.shape[-2:], nr_classes=1,
                     cunits=[24, 48], fcunits=[100])
    optimizer = Adam()
    model.compile(loss='binary_crossentropy', optimizer=optimizer,
                  metrics=['accuracy'])

    model.fit_generator(imggen.flow(x_train, y_train, batch_size=len(x_train)),
                        validation_data=imggen.flow(x_test, y_test),
                        steps_per_epoch=100,  epochs=5,
                        verbose=1, validation_steps=256)
    model.save('aps_lenet.h5')

from sklearn.metrics import confusion_matrix

def plot_cm(cm, classes, normalize=False,
            title='Confusion matrix', cmap=pl.cm.viridis):
    """
    This function prints and plots the confusion...


I am working on the file "Archiv_Session_Spring_2017/Exercises/05_APS Captcha.ipynb".
The first task is:
Training Random Forests
Preprocessing to a fixed size training set since sklearn doesn't suppport streaming training sets?
Can you write Python code for it?

# Same size training set as LeNet
TRAININGSET_SIZE = len(x_train) * 5 * 100

batch_size = len(x_train)
nr_batches = TRAININGSET_SIZE // batch_size + 1
imgit = imggen.flow(x_train, y=y_train, batch_size=batch_size)
x_train_sampled = np.empty((TRAININGSET_SIZE, 1,) + x_train.shape[-2:])
y_train_sampled = np.empty(TRAININGSET_SIZE)

for batch, (x_batch, y_batch) in enumerate(it.islice(imgit, nr_batches)):
    buflen = len(x_train_sampled[batch * batch_size:(batch + 1) * batch_size])
    x_train_sampled[batch * batch_size:(batch + 1) * batch_size] = x_batch[:buflen]
    y_train_sampled[batch * batch_size:(batch + 1) * batch_size] = y_batch[:buflen]

from sklearn.ensemble import RandomForestClassifier

rfe = RandomForestClassifier(n_estimators=64, criterion='entropy', n_jobs=-1,
                             verbose=True)
rfe = rfe.fit(x_train_sampled.reshape((TRAININGSET_SIZE, -1)), y_train_sampled)

y_pred_rw = rfe.predict(x_rw.reshape((len(x_rw), -1)))
plot_cm(confusion_matrix(y_rw, y_pred_rw), normalize=True,
        classes=["Not Einstein", "Einstein"])
pl.show()

print("Rightly classified Einsteins:")
imsshow(x_rw[((y_rw - y_pred_rw) == 0) * (y_rw == 1)])
pl.show()

print("Wrongly classified images:")
imsshow(x_rw[(y_rw - y_pred_rw) != 0])
pl.show()


I am working on the file "Archiv_Session_Spring_2017/Exercises/05_aps_capcha.ipynb".
The first task is:
Image before/after compression
Can you write Python code for it?

i=11
plt.subplot(1,2,1)
plt.imshow(training_set[i],cmap='gray')
plt.subplot(1,2,2)
plt.imshow(X_train[i][0],cmap='gray')


I am working on the file "Archiv_Session_Spring_2017/Exercises/05_aps_capcha.ipynb".
The first task is:
Lenet neural network
Can you write Python code for it?

# import the necessary packages
from keras.models import Sequential
from keras.layers.convolutional import Convolution2D
from keras.layers.convolutional import MaxPooling2D
from keras.layers.core import Activation
from keras.layers.core import Flatten
from keras.layers.core import Dense
from keras.optimizers import SGD

# this code comes from http://www.pyimagesearch.com/2016/08/01/lenet-convolutional-neural-network-in-python/

class LeNet:
	@staticmethod
	def build(width, height, depth, classes, weightsPath=None):
		# initialize the model
		model = Sequential()

		# first set of CONV => RELU => POOL
		model.add(Convolution2D(20, 5, 5, border_mode="same",input_shape=(depth, height, width)))
		model.add(Activation("relu"))
		model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

		# second set of CONV => RELU => POOL
		model.add(Convolution2D(50, 5, 5, border_mode="same"))
		model.add(Activation("relu"))
		model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

		# set of FC => RELU layers
		model.add(Flatten())
		model.add(Dense(500))
		model.add(Activation("relu"))

		# softmax classifier
		model.add(Dense(classes))
		model.add(Activation("softmax"))

		# return the constructed network architecture
		return model


I am working on the file "Archiv_Session_Spring_2017/Exercises/05_aps_capcha.ipynb".
The first task is:
We build the neural network and fit it on the training set
Can you write Python code for it?

model = LeNet.build(width=32, height=32, depth=1, classes=2)
opt = SGD(lr=0.01)#Sochastic gradient descent with learning rate 0.01
model.compile(loss="categorical_crossentropy", optimizer=opt,metrics=["accuracy"])
model.fit(X_train, Y_train, batch_size=10, nb_epoch=300,verbose=1)

y_pred = model.predict_classes(X_test)
print(y_pred)
print(test_labels)


I am working on the file "Archiv_Session_Spring_2017/Exercises/05_aps_capcha.ipynb".
The first task is:
We now compare with the real world images (with the deshear method)
Can you write Python code for it?

real_world_set=[]
for i in np.arange(1,73):
    filename=path+'images/real_world/'+str(i)+'.png'
    real_world_set.append(im.deshear(filename))
fake_label=np.ones(len(real_world_set),dtype='int32')
X_real,Y_real=prep_datas(real_world_set,fake_label)
y_pred = model.predict_classes(X_real)


I am working on the file "Archiv_Session_Spring_2017/Exercises/06_aps_with_classifiers.ipynb".
The first task is:
Now we load the real set of images and test it. This part of the program has been taken from Alexandre's program from last week. First we load the 'real world' images
Can you write Python code for it?

real_world_set=[]
for i in np.arange(1,73):
    filename=path+'images/real_world/'+str(i)+'.png'
    real_world_set.append(im.deshear(filename))
fake_label=np.ones(len(real_world_set),dtype='int32')
X_real,Y_real=prep_datas(real_world_set,fake_label)


I am working on the file "Archiv_Session_Spring_2017/Exercises/06_aps_with_classifiers.ipynb".
The first task is:
Then we load the training and the test set:
Can you write Python code for it?

training_set, training_labels = im.load_images(path_train)
X_train, Y_train = prep_datas(training_set,training_labels)

test_set, test_labels = im.load_images(path_test)
X_test,Y_test=prep_datas(test_set,test_labels)


I am working on the file "Archiv_Session_Spring_2017/Exercises/06_aps_with_classifiers.ipynb".
The first task is:
We define the classifiers Random Forest Classifier and Support Machine Classifier and we train them throught the fit function. Taking a linear kernel for SVC gives the best results for this classifier.
Can you write Python code for it?

classifierForest = RandomForestClassifier(n_estimators=1000)
classifierSVC=svm.SVC(kernel='linear')

classifierForest.fit(X_train, Y_train)
classifierSVC.fit(X_train,Y_train)


I am working on the file "Archiv_Session_Spring_2017/Exercises/06_aps_with_classifiers.ipynb".
The first task is:
We define the function prep_datas (props to Alexandre), already used the previous week. However now we reshape the images from a 32x32 matrix (this value seems unnecessary, however the bigger the image the worst the classifiers will work) to a flat 1024 vector, a constraint given by the Random Forest classifier.
Can you write Python code for it?

def prep_datas(xset,xlabels):
    X=list(xset)
    for i in range(len(X)):
        X[i]=resize(X[i],(32,32,1)) # reduce the size of the image from 100X100 to 32X32. Also flattens the color levels
        X[i]=np.reshape(X[i],1024) # reshape from 32x32 to a flat 1024 vector
    X=np.array(X) # transforms it into an array
    Y=np.asarray(xlabels) # transforms from list to array
    return X,Y


I am working on the file "Archiv_Session_Spring_2017/Exercises/10_CIFAR with sklearn.ipynb".
The first task is:
A little function for plotting
Can you write Python code for it?

def plotting(X, Y):
    fig = plt.figure()
    ax1 = fig.add_subplot(111)

    ax1.scatter(X[:, 0],X[:, 1], c='b', label='Cats')
    ax1.scatter(Y[:, 0], Y[:, 1], c='r', label='Dogs')
    plt.legend(loc='upper left');

    return plt.show()


I am working on the file "Archiv_Session_Spring_2017/Exercises/10_CIFAR with sklearn.ipynb".
The first task is:
Finally t-SNE
Can you write Python code for it?

print("Computing t-SNE")
tSNE = manifold.TSNE(n_components=n_components, init='pca', random_state=0).fit_transform(cats_and_dogs)
print("Done.")
plotting(tSNE[:n_cats], tSNE[n_cats:])


I am working on the file "Archiv_Session_Spring_2017/Exercises/10_CIFAR with sklearn.ipynb".
The first task is:
Let's first try the  Isomap embedding
Can you write Python code for it?

print("Computing Isomap embedding")
iso = manifold.Isomap(n_neighbors, n_components).fit_transform(cats_and_dogs)
print("Done.")
plotting(iso[:n_cats], iso[n_cats:])


I am working on the file "Archiv_Session_Spring_2017/Exercises/10_CIFAR with sklearn.ipynb".
The first task is:
Let's use Patrick's program to load the CIFAR images. We load the images + reshape them as numpy arrays, needed for the following steps.
Can you write Python code for it?

catsdogs = CifarLoader(path = 'data_batch_1')
cats_and_dogs = catsdogs.cats + catsdogs.dogs
n_cats, n_dogs = len(catsdogs.cats), len(catsdogs.dogs)
dims = cats_and_dogs[0].shape
cats_and_dogs = np.array(cats_and_dogs).reshape((n_cats+n_dogs, dims[0]*dims[1]*dims[1]))


I am working on the file "Archiv_Session_Spring_2017/Exercises/10_CIFAR with sklearn.ipynb".
The first task is:
Now let's do Spectral embedding. There are different method for this routine. However this 'standard' is the one giving less problems!
Can you write Python code for it?

print("Computing Spectral embedding")
emb = manifold.LocallyLinearEmbedding(n_neighbors, n_components, eigen_solver='auto',
                                        method = 'standard').fit_transform(cats_and_dogs)
print("Done.")
plotting(emb[:n_cats], emb[n_cats:])


I am working on the file "Archiv_Session_Spring_2017/Exercises/11 Markov random field.ipynb".
The first task is:
First we need to 'binarize' the picture we want to embed in the markow field. This function does it
Can you write Python code for it?

def prep_datas(pic, size):
    X=resize(pic,(size,size)) # reduce the size of the image from 100X100 to 32X32. Also flattens the color levels
    X=np.reshape(X,size**2) # reshape from 32x32 to a flat 1024 vector

    X=np.array(X) # transforms it into an array

    for j in range(len(X)): # let's binarized the image
        if X[j] < 0.5:
            X[j] = -1
        else:
            X[j] = 1;

    X_pic = np.reshape(X,(size,size))

    return X, X_pic


I am working on the file "Archiv_Session_Spring_2017/Exercises/11 Markov random field.ipynb".
The first task is:
We load and prepare the picture
Can you write Python code for it?

size_lattice = 100 # Size of the lattice in which we want to embed the picture

einstein_pic = io.imread("einstein.png")
training_chain, ein_pic = prep_datas(einstein_pic, size_lattice)

plt.subplot(1,2,1)
plt.imshow(einstein_pic,cmap='gray')
plt.subplot(1,2,2)
plt.imshow(ein_pic,cmap='gray')


I am working on the file "Archiv_Session_Spring_2017/Exercises/11 Markov random field.ipynb".
The first task is:
We now define the spin-spin correlation function for the lattice
Can you write Python code for it?

def corr(mat, d): # ensemble average of lattice (in form of 1D vector) for d-nearest neigbors spin

    item = deque(mat)
    item.rotate(d)

    mean = np.correlate(mat, np.asanyarray(item))/len(mat)

    return mean


I am working on the file "Archiv_Session_Spring_2017/Exercises/11_Markov_random_field.ipynb".
The first task is:
NOTE: The structure of J is funny. The first two labels $(i,j)$ denote each spin. The two second ones $(j,k)$ denote the neighbors of the corresponding spin, i.e., spin $(i+k,j+l)$. Self-energies $J_{i,j,0,0}$ are not taken into account. Only neighbors to the right and down need to be stored due to the symmetry of the interaction.
Functions needed for computing the negative phase terms
Can you write Python code for it?

# Initialize parameters

htrained = 2 * np.random.rand(size, size) - np.ones(training_set[0].shape)
Jtrained = np.zeros(Jpos.shape)
Jtrained[abs(Jpos) > 0] = 2 * np.random.rand() - 1    # Funny way to initialize only relevant cells in J

def potential(z, h):    # Potential energy of a configuration
    return np.sum(np.multiply(h, z))
def interactions(z, J):    # Interaction energy of a configuration
    return np.sum(np.array([J[i][j][k][l] * z[i][j] * z[i + k][j + l] for i in range(size)
                            for j in range(size) for k in range(size - i) for l in range(size - j)]))
def P(z, h, J, norm):    # Thermal distribution of configurations with temperature=1
    return np.exp(-np.sum(potential(z, h)) - np.sum(interactions(z, J))) / norm
def mse(a, b):    # Mean squared error (used during training to assess convergence)
    return ((a - b) ** 2).mean()

# Generate all possible configurations of spins for a given size
m = size ** 2
d = np.array(range(2 ** m))
allconfs = (((d[:,None] & (1 << np.arange(m)))) > 0).astype(int).reshape(2 ** m, size, size)
allconfs = 2 * allconfs - np.ones(allconfs.shape)    # Change 0s by -1s


I am working on the file "Archiv_Session_Spring_2017/Exercises/11_Markov_random_field.ipynb".
The first task is:
Sampling
Can you write Python code for it?

## Given some partial image, we are going to choose the most probable configuration

keep = [0]    # Rows that we want to keep from the original image

norm = np.sum([np.exp(-np.sum(potential(conf, htrained)) - np.sum(interactions(conf, Jtrained))) for conf in allconfs])
possible = np.array([[z, P(z, htrained, Jtrained, norm)]
            for z in allconfs[reduce(np.multiply,[np.prod(allconfs[:, i]==einst[i], axis=1) for i in keep]).astype(bool)]])

tru = possible[:, 0][np.argmax(possible[:,1])]

plt.imshow(tru)
plt.show()


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
2. Scraping
The amount of insight you can gain by scraping free information from the net is astonishing, especially given how easy it is to scrape. Whatever you want to learn about, there is free information out there (and probably a Python package to help).
We will work on the simplest scenario: pulling off files that do not need authentication or setting cookies. We will ignore robot rules and pretend that we are a Firefox browser. For instance, this will get you BIST's main site:
Can you write Python code for it?

url = "http://bist.eu/"
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
content = urlopen(req).read()


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
3. Cleaning structured data
We will integrate several data sources, as this is the most common pattern you encounter in day to day data science. We will have two structured sources of data, and one semistructured:

A table of journal titles and matching abbreviations. This will help us standardize journal titles in the data, as some will be given as full titles, others as abbreviations.

A table of journals and matching Impact Factors.

Search results in HTML format that contain the metadata of the papers we want to study

We start with the easier part, obtaining and cleaning the structured data. Let us get the journal abbrevations:
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
Actually, Toni has another Nature, but that is not on arXiv yet. Not all of Maciej's paper are on arXiv either, especially not the old ones.
We can do the same plots with subjects:
Can you write Python code for it?

subjects = db["Primary Subject"].drop_duplicates()
fig, ax = plt.subplots(ncols=1)
for subject in subjects:
    data = db[db["Primary Subject"] == subject]["Journal Impact Factor"]
    sns.distplot(data, kde=False, label=subject)
ax.legend()
ax.set_yscale("log")
plt.show()


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
But not perfect:
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
Convert it to UTF-8 to see what it contains in a nicer way:
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
Exercise 1. Copy an image location from the BIST website. Scrape it and display it. For the display, you can import Image from IPython.display, and then use Image(data=content). Technically you could use imshow from Matplotlib, but then you would have decompress the image first.
We put everything together in a function that only attempts to download a file if it is not present locally. If you do not specify a filename, it tries to extract it from the URL by taking everything after the last "/" character as the filename.
Can you write Python code for it?

def get_file(url, filename=None):
    if filename is None:
        slash = url.rindex("/")
        filename = url[slash+1:]
    if os.path.isfile(filename):
        with open(filename, 'rb') as file:
            content = file.read()
    else:
        req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
        content = urlopen(req).read()
        with open(filename, 'wb') as file:
            file.write(content)
            file.close()
    return content


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
Exercise 3. Cut the number of rotten apples in half by defining a replacement dictionary and doing another round of standardization.
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
His topics are somewhat predictable.
Let's add one more column to indicate the number of authors:
Can you write Python code for it?

db["#Authors"] = db["Authors"].apply(lambda x: len(x))
sns.stripplot(x="#Authors", y="Journal Impact Factor", data=db)


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
How about the length of title? Number of words in title?
Can you write Python code for it?

db["Length of Title"] = db["Title"].apply(lambda x: len(x))
db["Number of Words in Title"] = db["Title"].apply(lambda x: len(x.split()))
fig, axes = plt.subplots(ncols=2, figsize=(12, 5))
sns.stripplot(x="Length of Title", y="Journal Impact Factor", data=db, ax=axes[0])
sns.stripplot(x="Number of Words in Title", y="Journal Impact Factor", data=db, ax=axes[1])
plt.show()


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
If we inspect what it is, it looks like HTML:
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
If we take a closer look at a particular element, we will notice that something is off:
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
It is the same drill with our other contender, except that the short version of his name is uniquely his. On the other hand, that single accent in the surname introduces N+1 spelling variants, which we should standardize.
Can you write Python code for it?

acin = get_search_result("Acin_A")
for dd, dt in zip(acin.find_all("dd"), acin.find_all("dt")):
    id_, year = extract_id_and_year(dt.find("a", attrs={"title": "Abstract"}).text)
    div = dd.find("div", class_="list-authors")
    subject = extract_subject(dd.find("span", class_ = "primary-subject").text)
    journal = dd.find("div", class_ = "list-journal-ref")
    if journal:
        names = [drop_punctuation(a.text) for a in div.find_all("a")]
        journal = extract_journal(journal.text)
        long_name = abbs[abbs["Abbreviation"] == journal]
        if len(long_name) > 0:
            journal = long_name["Full Journal Title"].values[0]
        papers.append([id_, extract_title(dd.find("div", class_ = "list-title mathjax").text),
                       names, subject, year, journal])
for paper in papers:
    names = paper[2]
    for i, name in enumerate(names):
        if re.search("A.* Ac.n", name):
            names[i] = "Antonio Acín"

rotten_apples = find_rotten_apples(papers)
rotten_apples


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
Let's see what it contains:
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
Let's start plotting distributions:
Can you write Python code for it?

groups = ["AA", "ML", "AAML"]
fig, ax = plt.subplots(ncols=1)
for group in groups:
    data = db[db["Group"] == group]["Journal Impact Factor"]
    sns.distplot(data, kde=False, label=group)
ax.legend()
ax.set_yscale("log")
plt.show()


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
Notice that we lost a few papers:
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
Now you start to feel the pain of being a data scientist. The sloppiness of manual data entering is unbounded. Your duty is to clean up this mess. A quick fix is to tinker with the drop_punctuation function to replace the retarded encoding of JRC. Then go through the creation of the papers array and the standardization again.
Can you write Python code for it?

def drop_punctuation(string):
    result = string.replace(".", " ")
    result = result.replace(",", " ")
    result = result.replace("(", " ")
    result = result.replace(": ", "-")
    return " ".join(result.split())


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
Skipping the boring header part, the source code of the search result looks like this:
```html
<h3>Showing results 1 through 389 (of 389 total) for
<a href="/find/all/1/au:+Lewenstein_M/0/1/0/all/0/1?skip=0&query_id=ff0631708b5d0dd5">au:Lewenstein_M</a></h3>
<dl>
<dt>1.  <span class="list-identifier"><a href="/abs/1703.09814" title="Abstract">arXiv:1703.09814</a> [<a href="/pdf/1703.09814" title="Download PDF">pdf</a>, <a href="/ps/1703.09814" title="Download PostScript">ps</a>, <a href="/format/1703.09814" title="Other formats">other</a>]</span></dt>
<dd>
<div class="meta">
<div class="list-title mathjax">
<span class="descriptor">Title:</span> Efficient Determination of Ground States of Infinite Quantum Lattice  Models in Three Dimensions
</div>
<div class="list-authors">
<span class="descriptor">Authors:</span>
<a href="/find/cond-mat/1/au:+Ran_S/0/1/0/all/0/1">Shi-Ju Ran</a>,
<a href="/find/cond-mat/1/au:+Piga_A/0/1/0/all/0/1">Angelo Piga</a>,
<a href="/find/cond-mat/1/au:+Peng_C/0/1/0/all/0/1">Cheng Peng</a>,
<a href="/find/cond-mat/1/au:+Su_G/0/1/0/all/0/1">Gang Su</a>,
<a href="/find/cond-mat/1/au:+Lewenstein_M/0/1/0/all/0/1">Maciej Lewenstein</a>
</div>
<div class="list-comments">
<span class="descriptor">Comments:</span> 11 pages, 9 figures
</div>
<div class="list-subjects">
<span class="descriptor">Subjects:</span> <span class="primary-subject">Strongly Correlated Electrons (cond-mat.str-el)</span>; Computational Physics (physics.comp-ph)

</div>
</div>
</dd>
```
This is the entire first result. It might look intimidating, but as long as you know that in HTML a mark-up starts with `<whatever>` and ends with `</whatever>`, you will find regular and hierarchical patterns. If you stare hard enough, you will see that the `<dd>` tag contains most of the information we want: it has the authors, the title, the journal reference if there is one (not in the example shown), and the primary subject. It does not actually matter what `<dd>` is: we are not writing a browser, we are scraping data.

As a...
Can you write Python code for it?

titles = []
for dd in lewenstein.find_all("dd"):
    titles.append(dd.find("div", class_="list-title mathjax"))
len(titles)


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
So it is only one person, and we can be reasonably confident that Maciej Lewenstein is unlikely to publish in these subjects. The other good news is that all the short forms of the name belong to physics papers, and not computer science. Armed with this knowledge, we can filter out the correct manuscripts.
We need to filter one more thing: we are only interested in papers for which the journal reference is given. Further digging in the HTML code lets us find the correct tag. While we are putting together the correct records, we also normalize his name.
Annoyingly, the <dd> tag we have been focusing on does not contain the arXiv ID or the year. We zip the main loop's iterator with the <dt> tag, because only this one contains the arXiv ID, from which the year can be extracted. This always goes in pair with the <dd> tag, completing the metadata of the manuscripts.
We define yet another set of auxiliary functions to extract everything we need. The routine for extracting the journal title already performs stripping and converting to upper case. We assume that the name of the journal is the stuff on the matching line before the first digit (which is probably the volume or some other bibliographic information).
Can you write Python code for it?

def extract_journal(journal):
    start = journal.index(" ")
    raw = journal[start+1:-1]
    m = re.search("\d", raw)
    return drop_punctuation(raw[:m.start()]).strip().upper()

def extract_title(title):
    start = title.index(" ")
    return title[start+1:-1]

def extract_id_and_year(arXiv):
    start = arXiv.index(":")
    if "/" in arXiv:
        year_index = arXiv.index("/")
    else:
        year_index = start
    year = arXiv[year_index+1:year_index+3]
    if year[0] == "9":
        year = int("19" + year)
    else:
        year = int("20" + year)
    return arXiv[start+1:], year

papers = []
for dd, dt in zip(lewenstein.find_all("dd"), lewenstein.find_all("dt")):
    id_, year = extract_id_and_year(dt.find("a", attrs={"title": "Abstract"}).text)
    div = dd.find("div", class_="list-authors")
    subject = extract_subject(dd.find("span", class_ = "primary-subject").text)
    journal = dd.find("div", class_ = "list-journal-ref")
    if journal:
        names = [drop_punctuation(a.text) for a in div.find_all("a")]
        for i, name in enumerate(names):
            if re.search("M.* Lewenstein", name):
                if name not in true_lewenstein:
                    break
                else:
                    names[i] = "Maciej Lewenstein"
        else:
            papers.append([id_, extract_title(dd.find("div", class_ = "list-title mathjax").text),
                           names, subject, year, extract_journal(journal.text)])


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
Some stupid multi-column copyright notice at the bottom distorts the entire collection. This is why spreadsheets are the most hated data format for structured data, as the uncontrolled blend of content and formatting lets ignorant people give you a headache.
The journals are listed according to their ranking, in decreasing order. We know that we do not care about journals that do not have an Impact Factor. We take this as a cut-off for rows, and re-read only the rows and columns we are interested in:
Can you write Python code for it?

skip = len(ifs) - ifs["Journal Impact Factor"].last_valid_index() + 1
ifs = pd.read_excel("2014_SCI_IF.xlsx", skiprows=2, index_col=0,
                    skip_footer=skip, parse_cols="A,B,E")


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
The logarithmic scale makes the raw number of papers appear more balanced, which is fair given the difference in age between the two authors. A single Nature paper makes a great outlier:
Can you write Python code for it?

db[db["Journal Impact Factor"] == db["Journal Impact Factor"].max()]


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
The next problem we face is that not all of these papers belong to Maciej Lewenstein: some impostors have the same abbreviated name M. Lewenstein. They are easy to detect if they uses the non-abbreviated name. Let us run through the page again, noting which subject the the impostors publish in. For this, let us introduce another auxiliary function that extract the short name of the the subject. We also note the primary subject when the abbreviated form of the name appears. We use another function to drop "." and merge multiple white space to a single one. We will use a simple regular expression to find the candidate Lewensteins.
Can you write Python code for it?

def extract_subject(long_subject):
    start = long_subject.index("(")
    return long_subject[start+1:-1]

def drop_punctuation(string):
    result = string.replace(".", " ")
    return " ".join(result.split())

true_lewenstein = ["Maciej Lewenstein", "M Lewenstein"]
impostors = set()
primary_subjects = set()
for dd in lewenstein.find_all("dd"):
    div = dd.find("div", class_="list-authors")
    subject = extract_subject(dd.find("span", class_ = "primary-subject").text)
    names = [drop_punctuation(a.text) for a in div.find_all("a")]
    for name in names:
        if re.search("M.* Lewenstein", name):
            if name not in true_lewenstein:
                impostors.add(name + " " + subject)
            elif "Maciej" not in name:
                primary_subjects.add(subject)
print(impostors)
print(primary_subjects)


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
The string starts with an unnecessary white space. We also do not want to miss any possible match because of differences in the use of capitalization. So we strip starting and trailing white space and convert every entry to upper case:
Can you write Python code for it?

abbs = abbs.applymap(lambda x: x.upper().strip())


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
The way we read it, the content is a byte array:
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
This is fairly straightforward: the file starts with a bunch of comments, then two columns containing the full title and the abbreviation, separated by the "=" character. We can load it with Pandas without thinking much:
Can you write Python code for it?

abbs = pd.read_csv("journal_abbreviations_webofscience.txt", sep='=',
                   comment='#', names=["Full Journal Title", "Abbreviation"])


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
This looks much better:
Can you write Python code for it?


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
We would be almost done if journal names were all entered same way. Of course they were not. Let us try to standardize them: this is where we first combine two sources of data. We use the journal abbreviations table to default to the long title of the journal in our paper collection.
Can you write Python code for it?

for i, paper in enumerate(papers):
    journal = paper[-1]
    long_name = abbs[abbs["Abbreviation"] == journal]
    if len(long_name) > 0:
        papers[i][-1] = long_name["Full Journal Title"].values[0]


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
You are safe with quant-ph and quantum gases, but stay clear of atom physics. It is amusing to restrict the histogram to Professor Acín's subset:
Can you write Python code for it?

fig, ax = plt.subplots(ncols=1)
for subject in subjects:
    data = db[(db["Primary Subject"] == subject) & (db["Group"] != "ML")]
    if len(data) > 1:
        sns.distplot(data["Journal Impact Factor"], kde=False, label=subject)
ax.legend()
ax.set_yscale("log")
plt.show()


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
You can save it to a file so that you do not have to scrape it again. This is important when you scrape millions of files, and the scraping has to be restarted. Have mercy on the webserver, or you risk getting blacklisted.
Can you write Python code for it?

with open("bist.html", 'wb') as file:
    file.write(content)
    file.close()


I am working on the file "Archiv_Session_Spring_2017/Tutorials/Advanced_Data_Science.ipynb".
The first task is:
You can technically open this in a browser, but it is only the HTML part of the site: the style files, images, and a whole lot of other things that make a page working are missing. Lazy people can launch a browser from Python to see what the page looks like:
Can you write Python code for it?

from subprocess import run
run(["firefox", "-new-tab", "bist.html"])


I am working on the file "Archiv_Session_Spring_2018/Coding_Exercises/autoencoders.ipynb".
The first task is:
1. Shallow Autoencoder
Can you write Python code for it?

# this is our input placeholder
input_img = Input(shape=(784,))
n_feature=50
encoded = Dense(n_feature, activation='relu')(input_img)
decoded = Dense(784, activation='sigmoid')(encoded)
# this model maps an input to its reconstruction
autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

# We import the training data (MNIST)
(x_train, _), (x_test, _) = mnist.load_data()
# We normalize the data between 0 and 1
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
# We reshape the images 28X28 to vectors of 784
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

# We train the model
autoencoder.fit(x_train, x_train,epochs=50,batch_size=256,shuffle=True,validation_data=(x_test, x_test))


I am working on the file "Archiv_Session_Spring_2018/Coding_Exercises/autoencoders.ipynb".
The first task is:
2. Shallow Autoencoder for dimensionality reduction
Can you write Python code for it?

# this is our input placeholder
input_img = Input(shape=(784,))
n_feature=2
encoded = Dense(n_feature, activation='relu')(input_img)
decoded = Dense(784, activation='sigmoid')(encoded)
# this model maps an input to its reconstruction
autoencoder = Model(input_img, decoded)
# this model maps an input to its encoded representation
encoder = Model(input_img, encoded)

autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
# We import the training data (MNIST)
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# We normalize the data between 0 and 1
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
# We reshape the images 28X28 to vectors of 784
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
# We train the model
autoencoder.fit(x_train, x_train,epochs=50,batch_size=256,shuffle=True,validation_data=(x_test, x_test))


I am working on the file "Archiv_Session_Spring_2018/Coding_Exercises/autoencoders.ipynb".
The first task is:
3. Shallow Autoencoder for denoising
Can you write Python code for it?

(x_train, _), (x_test, _) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

noise_factor = 0.5
x_train_noisy = x_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train.shape)
x_test_noisy = x_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test.shape)

x_train_noisy = np.clip(x_train_noisy, 0., 1.)
x_test_noisy = np.clip(x_test_noisy, 0., 1.)

n = 10
plt.figure(figsize=(20, 2))
for i in range(n):
    ax = plt.subplot(1, n, i+1)
    plt.imshow(x_test_noisy[i].reshape(28, 28),cmap='gray')
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()

# Shallow autoencoder
input_img = Input(shape=(784,))
n_feature=50
encoded = Dense(n_feature, activation='relu')(input_img)
decoded = Dense(784, activation='sigmoid')(encoded)
# this model maps an input to its reconstruction
autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
# We train the model
autoencoder.fit(x_train_noisy, x_train,epochs=50,batch_size=256,shuffle=True,validation_data=(x_test, x_test))


I am working on the file "Archiv_Session_Spring_2018/Coding_Exercises/autoencoders.ipynb".
The first task is:
4. Deep Autoencoder for dimensionality reduction
Can you write Python code for it?

# this is our input placeholder
input_img = Input(shape=(784,))
encoded = Dense(1000, activation='relu')(input_img)
encoded = Dense(500, activation='relu')(encoded)
encoded = Dense(250, activation='relu')(encoded)
encoded = Dense(125, activation='relu')(encoded)
encoded = Dense(2, activation='relu')(encoded)

decoded = Dense(125, activation='relu')(encoded)
decoded = Dense(250, activation='relu')(decoded)
decoded = Dense(500, activation='relu')(decoded)
decoded = Dense(1000, activation='relu')(decoded)
decoded = Dense(784, activation='sigmoid')(decoded)

# this model maps an input to its reconstruction
autoencoder = Model(input_img, decoded)

# this model maps an input to its encoded representation
encoder = Model(input_img, encoded)

autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

autoencoder.fit(x_train, x_train,epochs=50,batch_size=256,\
                shuffle=True,validation_data=(x_test, x_test))

encoded_imgs = encoder.predict(x_test)
recon_imgs=autoencoder.predict(x_test)


I am working on the file "Archiv_Session_Spring_2018/Coding_Exercises/autoencoders.ipynb".
The first task is:
Epoch 50/50
60000/60000 [==============================] - 31s - loss: 0.1860 - val_loss: 0.1863
Can you write Python code for it?

pca = decomposition.PCA(n_components=2)
pca.fit(x_test)
x_red = pca.transform(x_test)
x_pca = pca.inverse_transform(x_red)

plt.figure()
plt.subplot(1,2,1)
plt.scatter(encoded_imgs[:,0],encoded_imgs[:,1],c=y_test,cmap='jet')
plt.title('Deep autoencoder')
plt.subplot(1,2,2)
plt.scatter(x_red[:,0],x_red[:,1],c=y_test,cmap='jet')
plt.title('PCA')
plt.colorbar()

plt.figure()

n = 10  # how many digits we will display
plt.figure(figsize=(20, 4))
for i in range(n):
    # display original
    ax = plt.subplot(3, n, i + 1)
    plt.imshow(x_test[i].reshape(28, 28),cmap='gray')
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # display reconstruction
    ax = plt.subplot(3, n, i + 1 + n)
    plt.imshow(recon_imgs[i,:].reshape(28, 28),cmap='gray')
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # display reconstruction
    ax = plt.subplot(3, n, i + 1 + 2*n)
    plt.imshow(x_pca[i,:].reshape(28, 28),cmap='gray')
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)


I am working on the file "Archiv_Session_Spring_2018/Coding_Exercises/autoencoders.ipynb".
The first task is:
Epoch 50/50
60000/60000 [==============================] - 4s - loss: 0.0958 - val_loss: 0.0942
Can you write Python code for it?

decoded_imgs = autoencoder.predict(x_test)

n = 10  # how many digits we will display
plt.figure(figsize=(20, 4))
for i in range(n):
    # display original
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(x_test[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # display reconstruction
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(decoded_imgs[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


I am working on the file "Archive/Actors.ipynb".
The first task is:
To actually create an actor, we can instantiate this class by calling Counter.remote ( ).
Can you write Python code for it?

a1 = Counter.remote()
a2 = Counter.remote()