I am working on the file "Chapters/Two/Simple Linear Regression.ipynb".
The first task is:
Multiple linear regression
Add topings to our model of pizza price prediction
Can you write Python code for it?

from numpy.linalg import inv
from numpy import dot, transpose

X = [[1, 6, 2], [1, 8, 1], [1, 10, 0], [1, 14, 2], [1, 18, 0]]
X

y = [[7], [9], [13], [17.5], [18]]

#Solve using linear algebra
dot(inv(dot(transpose(X),X)), dot(transpose(X),y))

#Solve using numpy least squares procedure
from numpy.linalg import lstsq
lstsq(X,y)[0]

#Compare simple vs multinomial
X = [[6, 2], [8, 1], [10, 0], [14, 2], [18, 0]]
Y = [[7],    [9],    [13],    [17.5],  [18]]

model = LinearRegression()
model.fit(X,Y)

X_test = [[8, 2], [9, 0], [11, 2], [16, 2], [12, 0]]
Y_test = [[11],   [8.5],  [15],    [18],    [11]]

predictions = model.predict(X_test)

for i, prediction in enumerate(predictions):
    print("Predicted: %s, Target: %s" %(prediction, Y_test[i]))

print("R square:", model.score(X_test, Y_test))


I am working on the file "Chapters/Two/Simple Linear Regression.ipynb".
The first task is:
Polynomial regression
Use PolynomialFeatures to transform the data
Can you write Python code for it?

from sklearn.preprocessing import PolynomialFeatures

X_train = [[6], [8], [10], [14],   [18]]
y_train = [[7], [9], [13], [17.5], [18]]
X_test = [[6],  [8],   [11], [16]]
y_test = [[8], [12], [15], [18]]

regressor = LinearRegression()
regressor.fit(X_train, y_train)

xx = np.linspace(0, 26, 100)
xx

yy = regressor.predict(xx.reshape(xx.shape[0],1))
plt.plot(xx,yy)

quadratic_featureziser = PolynomialFeatures(degree=2)
X_train_quadratic = quadratic_featureziser.fit_transform(X_train)
X_train_quadratic

X_test_quadratic = quadratic_featureziser.transform(X_test)

regressor_quadratic = LinearRegression()

regressor_quadratic.fit(X_train_quadratic,y_train)

xx_quadratic = quadratic_featureziser.transform(xx.reshape(xx.shape[0],1))

plt.plot(xx, regressor_quadratic.predict(xx_quadratic), c='r',linestyle = '--')
plt.title("Pizza price regressed on diameter")
plt.xlabel("Diameter in inches")
plt.ylabel("Price in dollars")
plt.axis([0,25,0,25])
plt.grid(True)
plt.scatter(X_train, y_train)
plt.show()


I am working on the file "Chapters/Two/Simple Linear Regression.ipynb".
The first task is:
Solving OLS for simple regression
Goal is to calculate vector of coefficients beta that minimizes cost function.
Can you write Python code for it?

from __future__ import division

xbar = np.mean(X)
ybar = np.mean(Y)
print("Mean of X is:", xbar)
print("Mean of Y is:", ybar)

#Make own function for variance and covariance to better understand how it works
def variance(X):
    return np.sum((X - np.mean(X))**2 / (len(X)-1))

def covariance(X,Y):
    return np.sum((X - np.mean(X)) * (Y - np.mean(Y)) / (len(X)-1))

print("Variance of X: ", variance(X))
print("Covariance of X, Y is: ", covariance(X,Y))

#For simple linear regression, beta is cov/var.
#Following calculation of beta, I can also get alpha a = y - bx

beta = covariance(X,Y) / variance(X)
beta


I am working on the file "ChartHomework/ChartsHomework.ipynb".
The first task is:
Chart 3 features a dark grid style. Although the grid helps assign values to the bars, the color is distracting and tends to chart junk
Can you write Python code for it?

sns.set_style("darkgrid")
sns.barplot(x="count", y="COUNTY", data=df, color="#7fe5ba")
plt.ylabel('')
plt.xlabel('Mountain Lion Sightings')
sns.despine(bottom=True, left=True)


I am working on the file "ChartHomework/ChartsHomework.ipynb".
The first task is:
Read the csv file into a panda
Can you write Python code for it?


I am working on the file "Chart_Samples.ipynb".
The first task is:
Coin tosses: Binomial-Beta
Can you write Python code for it?

plt.figure(figsize=(11, 9))

import scipy.stats as stats

beta = stats.beta
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
x = np.linspace(0, 1, 100)

# For the already prepared, I'm using Binomial's conj. prior.
for k, N in enumerate(n_trials):
    sx = plt.subplot(len(n_trials) / 2, 2, k + 1)
    plt.xlabel("$p$, probability of heads") \
        if k in [0, len(n_trials) - 1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    heads = data[:N].sum()
    #posterior distribution.
    y = beta.pdf(x, 1 + heads, 1 + N - heads)
    plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
    plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)

    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities",
             y=1.02,
             fontsize=14)

plt.tight_layout()


I am working on the file "Chart_Samples.ipynb".
The first task is:
Compute Common Support
The common support is an important concept, as for each pair of restaurants, its the number of people who reviewed both. It will be used to modify similarity between restaurants. If the common support is low, the similarity is less believable.
Can you write Python code for it?

restaurants=smalldf.business_id.unique()  #get all the unique restaurant ids
supports=[]
for i,rest1 in enumerate(restaurants):  # first restaurant (rest1) in the pair
    for j,rest2 in enumerate(restaurants): # second restaurant(rest2) in the pair
        if  i < j: #skip pairing same restaurants and forming duplicate pairs
            rest1_reviewers = smalldf[smalldf.business_id==rest1].user_id.unique() #find all unique users who reviewed restaurant 1
            rest2_reviewers = smalldf[smalldf.business_id==rest2].user_id.unique() #find all unique users who reviewed restaurant 2
            common_reviewers = set(rest1_reviewers).intersection(rest2_reviewers) # find common reviewers by taking intersection
            supports.append(len(common_reviewers)) # add the no of common reviewers to list
print "Mean support is:",np.mean(supports)
plt.hist(supports) #plot hist of list
histogram_style()


I am working on the file "Chart_Samples.ipynb".
The first task is:
Gaussian with known $\sigma$
Can you write Python code for it?

Y = [16.4, 17.0, 17.2, 17.4, 18.2, 18.2, 18.2, 19.9, 20.8]
# Prior mean
mu_prior = 19.5
# prior std
tau = 10
N = 15000

#Data Quantities
sig = np.std(Y) # assume that is the value of KNOWN sigma (in the likelihood)
mu_data = np.mean(Y)
n = len(Y)
kappa = sig**2 / tau**2
sig_post =np.sqrt(1./( 1./tau**2 + n/sig**2));
# posterior mean
mu_post = kappa / (kappa + n) *mu_prior + n/(kappa+n)* mu_data

#samples
theta_prior = np.random.normal(loc=mu_prior, scale=tau, size=N);
theta_post = np.random.normal(loc=mu_post, scale=sig_post, size=N);

plt.hist(theta_post, bins=30, alpha=0.9, label="posterior");
plt.hist(theta_prior, bins=30, alpha=0.2, label="prior");
plt.xlim([10, 30])
plt.xlabel("wing length (mm)")
plt.ylabel("Number of samples")
plt.legend();


I am working on the file "Chart_Samples.ipynb".
The first task is:
Get data from a url using beautiful soup
Can you write Python code for it?

#Your code here
def get_all_votes():
    url = 'http://www.govtrack.us/data/congress/113/votes/2013'
    response = requests.get(url)
    soup = bs(response.content, "lxml")
    s_votes = [a['href'][1:-1] for a in soup.find_all('a')
     if a['href'].startswith('s')]
    return [get_senate_vote(vote) for vote in s_votes]


I am working on the file "Chart_Samples.ipynb".
The first task is:
Get json data from url
Can you write Python code for it?

#your code here
def get_senate_vote(vote):
    url = 'https://www.govtrack.us/data/congress/113/votes/2013/s'+str(vote)+'/data.json'
    vote_data = requests.get(url)
    return json.loads(vote_data.text)


I am working on the file "Chart_Samples.ipynb".
The first task is:
Let us plot the probabilities obtained from predict_proba, overlayed on the samples with their true labels:
Can you write Python code for it?

plt.figure(figsize =(12,8))
plt.grid(True)
ax=plt.gca()
points_plot_prob(ax, Xtrain_l, Xtest_l, ytrain_l, ytest_l, clf_l, psize=20, alpha=0.1);


I am working on the file "Chart_Samples.ipynb".
The first task is:
Of the critics with > 100 reviews, plot the distribution of average "freshness" rating per critic
Can you write Python code for it?

df = critics.copy()
df['fresh'] = df.fresh == 'fresh'

grp = df.groupby('critic')
counts = grp.rtid.count()
means = grp.fresh.mean()
means[counts > 100].hist(bins=10, edgecolor='w', lw=1)
plt.xlabel("Average rating per critic")
plt.ylabel("N")
plt.yticks([0, 2, 4, 6, 8, 10])
histogram_style()


I am working on the file "Chart_Samples.ipynb".
The first task is:
Plotting the data
Histograms
Simulating Elections
Can you write Python code for it?

predictwise = pd.read_csv('data/predictwise.csv').set_index('States')
predictwise.head(10)

#Your code here
def simulate_election(model, n_sim):
    simulations = np.random.uniform(size= (51,n_sim))
    obama_votes = (simulations < model.Obama.values.reshape(-1,1))* model.Votes.values.reshape(-1,1)
    results  = obama_votes.sum(axis = 0)
    return results

result = simulate_election(predictwise, 10000)
result

#your code here
def plot_simulation(simulation):
    plt.hist(simulation, bins=np.arange(200, 538, 1),
             label='simulations', align='left', normed=True)
    plt.axvline(332, 0, .5, color='r', label='Actual Outcome')
    plt.axvline(269, 0, .5, color='k', label='Victory Threshold')
    p05 = np.percentile(simulation, 5.)
    p95 = np.percentile(simulation, 95.)
    iq = int(p95 - p05)
    pwin = ((simulation >= 269).mean() * 100)
    plt.title("Chance of Obama Victory: %0.2f%%, Spread: %d votes" % (pwin, iq))
    plt.legend(frameon=False, loc='upper left')
    plt.xlabel("Obama Electoral College Votes")
    plt.ylabel("Probability")
    remove_border()

plot_simulation(result)
plt.xlim(240,380)


I am working on the file "Chart_Samples.ipynb".
The first task is:
This graph shows a trend towards a lower average score, as well as a greater abundance of low scores, with time. This is probably at least partially a selection effect -- Rotten Tomatoes probably doesn't archive reviews for all movies, especially ones that came out before the website existed. Thus, reviews of old movies are more often "the classics". Mediocre old movies have been partially forgotten, and are underrepresented in the data. Other questions worth exploring:
Does the down trend mean:
* the quality of movies has dropped ?
* the top critics became more stringent in giving higher ratings?
* more top critics with stringent ratings entered the scene?
Can you write Python code for it?

df=pd.read_csv("01_heights_weights_genders.csv")
df.head()

from sklearn.linear_model import LogisticRegression
clf_l, Xtrain_l, ytrain_l, Xtest_l, ytest_l  = do_classify(LogisticRegression(), {"C": [0.01, 0.1, 1, 10, 100]}, df, ['Weight', 'Height'], 'Gender','Male')

plt.figure(figsize =(12,8))
plt.xlabel("Weight")
plt.ylabel("Height")
ax=plt.gca()
points_plot(ax, Xtrain_l, Xtest_l, ytrain_l, ytest_l, clf_l, alpha=0.2);


I am working on the file "Chart_Samples.ipynb".
The first task is:
Useful functions and tricks
Can you write Python code for it?


I am working on the file "Chart_Samples.ipynb".
The first task is:
Using the original movies dataframe, plot the rotten tomatoes Top Critics Rating as a function of year. Overplot the average for each year, ignoring the score=0 examples (some of these are missing data).
Can you write Python code for it?

# Get the data
from io import StringIO
movie_txt = requests.get('https://raw.github.com/cs109/cs109_data/master/movies.dat').text
movie_file = StringIO(movie_txt) # treat a string like a file
movies = pd.read_csv(movie_file, delimiter='\t')
movies = movies.dropna()
movies = movies.reset_index(drop = True)

# process data
data = movies[['year','rtTopCriticsRating']]
#data = data.convert_objects(convert_numeric=True) #deprecated
data = data.apply(pd.to_numeric)
data = data[(data.rtTopCriticsRating > 0.00)]
means  = data.groupby('year').mean()

# plotting
plt.plot(data['year'], data['rtTopCriticsRating'], 'o', mec = 'none', alpha = .5, label = 'Data' )
plt.plot(means.index, means['rtTopCriticsRating'], '-',  label = 'Yearly Average' )
plt.legend(loc ='lower left', frameon = False)
plt.xlabel("Year")
plt.ylabel("Average Score")
remove_border()


I am working on the file "Chart_Samples.ipynb".
The first task is:
What does the distribution of number of reviews per reviewer look like?
Can you write Python code for it?

plt.hist(critics.groupby('critic').rtid.count(),log = True, bins=range(20), edgecolor='white')
plt.xlabel("Number of reviews per critic")
plt.ylabel("N")
histogram_style()


I am working on the file "Check Database Tables.ipynb".
The first task is:
For each table, I want to
* assert that it actually exists
* get a rowcount
Can you write Python code for it?

table_count(table_names[0])

%%time
table_counts={
    table_name:table_count(table_name)
    for table_name in table_names
}

pandas.Series(table_counts)

len(table_counts)


I am working on the file "Check installs.ipynb".
The first task is:
Linear regreession
Can you write Python code for it?

X = df[["X"]].copy()
y = df["Y"].copy()
print "X.shape: ", X.shape
print "Y.shape: ", y.shape

model_linear = LinearRegression()
model_linear.fit(X, y)

y_pred = model_linear.predict(X)
print "Y_pred.shape: ", y_pred.shape

X["X^2"] = X["X"]**2

X.columns

model_sqr = LinearRegression()
model_sqr.fit(X, y)
y_pred_sqr = model_sqr.predict(X)
print "Y_pred_sqr.shape: ", y_pred_sqr.shape

plt.scatter(X["X"], y, marker="o", label="data", alpha=0.5, s=30)
plt.plot(X["X"], y_pred, linestyle="--", linewidth=1.5, color="k", label="fit [linear]")
plt.plot(X["X"], y_pred_sqr, linestyle="--", linewidth=1.5, color="r", label="fit [square]")
plt.xlabel("X")
plt.ylabel("Y")
plt.legend()

model_linear.coef_

model_sqr.coef_


I am working on the file "Check installs.ipynb".
The first task is:
Matplotlib checks
More details at: http://matplotlib.org/users/pyplot_tutorial.html
Can you write Python code for it?

plt.plot(x,y, marker="o", color="r", label="demo")
plt.xlabel("X axis")
plt.ylabel("Y axis")
plt.title("Demo plot")
plt.legend()


I am working on the file "Check installs.ipynb".
The first task is:
Pandas checks
More details at: http://pandas.pydata.org/pandas-docs/stable/tutorials.html
Can you write Python code for it?

df = pd.DataFrame()
df["X"] = x
df["Y"] = y
df["G"] = np.random.randint(1,10,size=x.shape)
df["E"] = np.random.randint(1,5,size=x.shape)
df.shape

df.head()

df.describe()

df.G = df.G.astype("category")
df.E = df.E.astype("category")


I am working on the file "Check installs.ipynb".
The first task is:
Seaborn checks
More details at: https://stanford.edu/~mwaskom/software/seaborn/index.html
Can you write Python code for it?

sns.barplot(x="G", y="Y", data=df, estimator=np.mean, color="dodgerblue")

g = sns.jointplot("X", "Y", data=df, kind="reg",
                  color="r", size=7)

sns.pairplot(df, hue="E")

# Initialize a grid of plots with an Axes for each walk
grid = sns.FacetGrid(df, col="G", hue="E", col_wrap=4, size=3, legend_out=True)

# Draw a horizontal line to show the starting point
grid.map(plt.axhline, y=30, ls=":", c=".5")

# Draw a line plot to show the trajectory of each random walk
t = grid.map(plt.plot, "X", "Y", marker="o", ms=4).add_legend(title="E values")
#grid.fig.tight_layout(w_pad=1)


I am working on the file "Check me.ipynb".
The first task is:
Start a server, That check inbox in every 15 sec and give notification if new mail recive
Can you write Python code for it?


I am working on the file "CheckLabelVsFit.ipynb".
The first task is:
Load data
Extract labels
Remove labels and quote number
    Convert date to datetime
Can you write Python code for it?

execfile('preprocessing.py')

# keep only good rows
chosen_indices = range(0, 53) + range(68, 109) + range(113, 169) + range(183, 234) + range(248, 294)
train_chosen = train.iloc[:, chosen_indices]
print train_chosen.shape, train.shape


I am working on the file "CheckLabelVsFit.ipynb".
The first task is:
label encoder vs one hot encoder
median calculation
data normalization (without median)
Can you write Python code for it?

Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.1, random_state=888)
#Xvalidation, Xtest, yvalidation, ytest = train_test_split(Xtest, ytest, test_size = 0.7, random_state = 8)
print Xtrain.shape, Xvalidation.shape, Xtest.shape

print xgb.cv?

def check_xboost(Xtrain, ytrain, Xvalidation, yvalidation, Xtest, ytest, num_boost_root=30):
    dtrain = xgb.DMatrix(Xtrain, ytrain)
    clf = xgb.XGBClassifier(learning_rate=0.02, max_depth = 10,
                            subsample =0.85, colsample_bytree = 0.66)
    booster_params = clf.get_params()
    booster_params['eval_metric'] = "auc"
    booster_params['eval_set'] = ((Xvalidation, yvalidation), (Xtrain, ytrain))
    v = xgb.cv(booster_params, dtrain, num_boost_round = num_boost_round,
              early_stopping_rounds = 150
              )
    print v

def check_classifier(Xtrain, ytrain, Xvalidation, yvalidation, Xtest, ytest, clf):
    print Xtrain.shape, ytrain.shape, Xvalidation.shape, yvalidation.shape
    clf.fit(Xtrain, ytrain,
           #eval_metric = "auc",
            #early_stopping_rounds = 150,
            #eval_set=((Xtrain, ytrain), (Xvalidation, yvalidation), ),
            #verbose = False
           )
    print clf.score(Xtrain, ytrain)
    print clf.score(Xtest, ytest)

clf.fit?

%%timeit -n 1 -r 1
dc = lambda: xgb.XGBClassifier(n_estimators=100,learning_rate=0.02, max_depth = 10, subsample =0.85,
                               colsample_bytree = 0.66)
clf = dc()
check_classifier(Xtrain, ytrain, Xvalidation, yvalidation, Xtest, ytest,clf )
#check_xboost(Xtrain, ytrain, Xvalidation, yvalidation, Xtest, ytest)

clf = dc()
clfbag = BaggingClassifier(clf, n_estimators=5)
check_classifier(Xtrain, ytrain,  Xvalidation, yvalidation, Xtest, ytest,clfbag )

clf = dc()
clf_isotonic = CalibratedClassifierCV(clf, cv=5, method='isotonic')
check_classifier(Xtrain, ytrain,  Xvalidation, yvalidation, Xtest, ytest,clf_isotonic )

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf_isotonic =...


I am working on the file "Check_bug_correction.ipynb".
The first task is:
Check the shape of the data
Can you write Python code for it?


I am working on the file "Check_bug_correction.ipynb".
The first task is:
Compute the difference between the LR
Can you write Python code for it?

diff_lr = merged['lr_5_old']-merged['lr_5_new']

len(diff_lr)


I am working on the file "Check_bug_correction.ipynb".
The first task is:
Direct comparison of the LR of the sources that changed. Most of them are close to the threshold change but in 4 cases there are big differences.
Can you write Python code for it?

cond_change = ((merged['lr_5_old'] >= told) & (merged['lr_5_new'] < tnew))
plt.subplot(1,2,1)
plt.semilogx(merged['lr_5_new'][cond_change], merged['lr_5_old'][cond_change], marker="x", ls="")
plt.xlabel("log LR (new)")
plt.ylabel("LR (old)")
plt.subplot(1,2,2)
plt.plot(merged['lr_5_new'][cond_change], merged['lr_5_old'][cond_change], marker="x", ls="")
plt.xlabel("LR (new)")
plt.ylabel("LR (old)")


I am working on the file "Check_bug_correction.ipynb".
The first task is:
Explore the differences
Plot of the difference in LR with respect to the new LR
Can you write Python code for it?

plt.plot(merged['lr_5_new'], diff_lr, marker=",", ls="")
plt.xlabel("LR (new)")
plt.ylabel("LR diff")


I am working on the file "Check_bug_correction.ipynb".
The first task is:
Plot of the absolute difference in LR with respect to the new LR in log space.
Can you write Python code for it?

plt.loglog(merged['lr_5_new'], np.abs(diff_lr), marker=",", ls="")
plt.xlabel("log LR (new)")
plt.ylabel("log abs. LR diff")


I am working on the file "Check_bug_correction.ipynb".
The first task is:
Plot of the percentage of absolute difference in LR with respect to the new LR in log space. Sources always below the threshold are plotted in orange and the galaxies that are not matched now are plotted as green crosses.
Can you write Python code for it?

cond_sel = ((merged['lr_5_old'] >= told) | (merged['lr_5_new'] >= tnew)) & (~np.isnan(merged['category_old']))
cond_nosel = ((merged['lr_5_old'] < told) & (merged['lr_5_new'] < tnew)) & (~np.isnan(merged['category_old']))
cond_change = ((merged['lr_5_old'] >= told) & (merged['lr_5_new'] < tnew)) & (~np.isnan(merged['category_old']))
plt.loglog(merged['lr_5_new'][cond_sel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_sel], marker=",", ls="", label="Selected")
plt.loglog(merged['lr_5_new'][cond_nosel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_nosel], marker=",", ls="", label="Rejected")
plt.loglog(merged['lr_5_new'][cond_change], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_change], marker="x", ls="", label="Changes")
plt.xlabel("log LR (new)")
plt.ylabel("log percent abs. LR diff")
plt.legend()


I am working on the file "Check_bug_correction.ipynb".
The first task is:
Plot of the percentage of absolute difference in LR with respect to the new LR with the y axis in log space. Sources always below the threshold are plotted in orange.
Can you write Python code for it?

cond_sel = ((merged['lr_5_old'] >= told) | (merged['lr_5_new'] >= tnew))
cond_nosel = ((merged['lr_5_old'] < told) & (merged['lr_5_new'] < tnew))
plt.semilogy(merged['lr_5_new'][cond_sel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_sel], marker=",", ls="")
plt.semilogy(merged['lr_5_new'][cond_nosel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_nosel], marker=",", ls="")
plt.xlabel("LR (new)")
plt.ylabel("log percent abs. LR diff")


I am working on the file "Check_bug_correction.ipynb".
The first task is:
The percentage of changes due to the bug
Can you write Python code for it?

4/91332*100

cond_change = ((merged['lr_5_old'] >= told) & (merged['lr_5_new'] < tnew) & (np.abs(diff_lr)/merged['lr_5_new']*100 > 10))

merged[cond_change]

cond = (np.abs(diff_lr)/merged['lr_5_new']*100 > 10)

merged[cond]
# ['Source_Name','ML_LR_new','AllWISE_new',
#  'objID_new','category_new','W1mag_new','i_new','lr_index_5_new',
#  'lr_dist_5_new','lr_5_new','lr_index_sel_5_new','ML_LR_old','AllWISE_old',
#  'objID_old','category_old','W1mag_old','i_old','lr_index_5_old',
#  'lr_dist_5_old','lr_5_old','lr_index_sel_5_old']

cond = (np.abs(diff_lr)/merged['lr_5_new']*100 > 1)
merged[cond & (merged['lr_5_old'] >= told)]['Source_Name','ML_LR_new','AllWISE_new',
  'objID_new','category_new','W1mag_new','i_new','lr_index_5_new',
  'lr_dist_5_new','lr_5_new','lr_index_sel_5_new','ML_LR_old','AllWISE_old',
  'objID_old','category_old','W1mag_old','i_old','lr_index_5_old',
  'lr_dist_5_old','lr_5_old','lr_index_sel_5_old']

cond = (np.abs(diff_lr)/merged['lr_5_new']*100 > 1)
merged[cond & (merged['lr_5_old'] >= told) & (merged['lr_index_5_old'] != merged['lr_index_5_new'])]['Source_Name','ML_LR_new','AllWISE_new',
  'objID_new','category_new','W1mag_new','i_new','lr_index_5_new',
  'lr_dist_5_new','lr_5_new','lr_index_sel_5_new','ML_LR_old','AllWISE_old',
  'objID_old','category_old','W1mag_old','i_old','lr_index_5_old',
  'lr_dist_5_old','lr_5_old','lr_index_sel_5_old']


I am working on the file "Check_bug_correction.ipynb".
The first task is:
The same plot than before but using the new threshol
Can you write Python code for it?

# Use new threshold for old data as well
cond_sel = ((merged['lr_5_old'] >= tnew) | (merged['lr_5_new'] >= tnew))
cond_nosel = ((merged['lr_5_old'] < tnew) & (merged['lr_5_new'] < tnew))
cond_change = ((merged['lr_5_old'] >= tnew) & (merged['lr_5_new'] < tnew))
plt.loglog(merged['lr_5_new'][cond_sel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_sel], marker=",", ls="", label="Selected")
plt.loglog(merged['lr_5_new'][cond_nosel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_nosel], marker=",", ls="", label="Rejected")
plt.loglog(merged['lr_5_new'][cond_change], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_change], marker="x", ls="", label="Changes")
plt.xlabel("log LR (new)")
plt.ylabel("log percent abs. LR diff")
plt.legend()


I am working on the file "Chesson Lesson.ipynb".
The first task is:

2. Coexistência em equilíbrio: nichos

Para existir coexistência em equilíbrio é necessário que haja diferença de nicho entre as espécies, o que requer a introdução de novos ingredientes (e.g. recursos). Em geral, para que $n$ espécies coexistam (novamente, em equilíbrio!) são necessários no mínimo $n$ recursos.
Aqui vamos tomar um exemplo simples e assumir que exista um segundo recurso implícito (ou seja, não modelado diretamente) que limita o crescimento da melhor competidora.

TODO: explicar sobreposição de nicho baseado em modelo com 2 recursos, com vetores de consumo, estilo Chesson 1990, e relacionar isso a fatores estabilizadores/equalizadores.
Can you write Python code for it?


I am working on the file "Chi-Square_Test.ipynb".
The first task is:
Data management
Can you write Python code for it?

nesarc = pd.read_csv('nesarc_pds.csv', low_memory=False)

races = {1 : 'White',
         2 : 'Black',
         3 : 'American India \n Alaska',
         4 : 'Asian \n Native Hawaiian \n Pacific',
         5 : 'Hispanic or Latino'}
subnesarc = (nesarc[['S3BQ1A5', 'ETHRACE2A']]
             .assign(S3BQ1A5=lambda x: pd.to_numeric(x['S3BQ1A5'].replace((2, 9), (0, np.nan)), errors='coerce'))
             .assign(ethnicity=lambda x: pd.Categorical(x['ETHRACE2A'].map(races)),
                     use_cannabis=lambda x: pd.Categorical(x['S3BQ1A5']))
             .dropna())
subnesarc.use_cannabis.cat.rename_categories(('No', 'Yes'), inplace=True)


I am working on the file "Chi-Square_Test.ipynb".
The first task is:
The p-value of 3.7e-91 confirm that the null hypothesis can be safetly rejected.
The next obvious questions is which ethnic groups have a statistically significant difference regarding the use of cannabis. For that, the Chi-Square test will be performed on each pair of group thanks to the following code.
Can you write Python code for it?

list_races = list(races.keys())
p_values = dict()
for i in range(len(list_races)):
    for j in range(i+1, len(list_races)):
        race1 = races[list_races[i]]
        race2 = races[list_races[j]]
        subethnicity = subnesarc.ETHRACE2A.map(dict(((list_races[i], race1),(list_races[j], race2))))
        comparison = pd.crosstab(subnesarc.use_cannabis, subethnicity)
        display(Markdown("Crosstable to compare {} and {}".format(race1, race2)))
        display(comparison)
        display(comparison/comparison.sum(axis=0))

        chi_square, p, _, expected_counts = stats.chi2_contingency(comparison)
        p_values[(race1, race2)] = p


I am working on the file "Chi-Square_Test.ipynb".
The first task is:
Variance analysis
Now that the univariate distribution as be plotted and described, the bivariate graphics will be plotted in order to test our research hypothesis.
From the bivariate graphic below, it seems that there are some differences. For example American Indian versus Asian seems quite different.
Can you write Python code for it?

g = sns.factorplot(x='ethnicity', y='S3BQ1A5', data=subnesarc,
                   kind="bar", ci=None)
g.set_xticklabels(rotation=90)
plt.ylabel('Ever use cannabis')
_ = plt.title('Average number of cannabis user depending on the ethnicity')

ct1 = pd.crosstab(subnesarc.use_cannabis, subnesarc.ethnicity)
display(Markdown("Contingency table of observed counts"))
ct1

# Note: normalize keyword is available starting from pandas version 0.18.1
ct2 = ct1/ct1.sum(axis=0)
display(Markdown("Contingency table of observed counts normalized over each columns"))
ct2


I am working on the file "ChicagoEmployeeGenderSalary.ipynb".
The first task is:
Gender prediction
To estimate the gender of an employee based on his or her first name, I used a data set of
baby names. For each unique name, I counted
how many times, from years 1940 to 2016, that name had been given to a boy versus a girl. If
the name was more frequently given to boys, then I predicted the gender associated with the
name to be male, and vice-versa for female.
Can you write Python code for it?

# Data are in seperate CSV files per year, and are concatenated here
name_data = []
for yob in range(1940, 2017):
    df = pd.read_csv('names/yob' + str(yob) + '.txt',
                     header=0, names=['Name', 'Gender', 'Count'])
    name_data.append(df)
names = pd.concat(name_data, axis=0)

# Lower-case first name so that it can be joined with the workers dataframe
names['Name'] = names['Name'].str.lower()

names.head(5)

# Count how often a name is given to boys and girls
gender_frequency = names.groupby(['Name', 'Gender']).sum().reset_index()

gender_frequency.sample(5)

def predict_gender(df):
    max_idx = df['Count'].idxmax()
    return df.loc[max_idx]

# Select the more frequent gender for each name
gender_guess = gender_frequency.groupby('Name').agg(predict_gender).reset_index()

gender_guess.sample(10)


I am working on the file "ChicagoEmployeeGenderSalary.ipynb".
The first task is:
The first ten rows of the data set look like this now:
Can you write Python code for it?


I am working on the file "ChicagoEmployeeGenderSalary.ipynb".
The first task is:
To make grouping and matching first names easier, I extracted the first name from each
employee record and lower-cased it:
Can you write Python code for it?

workers['Name'] = workers['Name'].apply(lambda s: s.split(',')[1].split()[0].lower())


I am working on the file "Child shooting victims in the 11th district.ipynb".
The first task is:
Let's just use the "District" column
After doing all the spatial stuff, I realized there is a "District" column I could have used to filter.  Note that in the above data, there is one row with a "District" column of "10".  Perhaps this is because it was mislabeled.
Can you write Python code for it?

def is_11th_district(district):
    try:
        return int(district) == 11
    except ValueError:
        return False

child_shooting_victims_since_2012_in_11th_dist = child_shooting_victims_since_2012[child_shooting_victims_since_2012['District'].apply(is_11th_district)]
print("There have been {} victims, under age 18, who have been shot in the 11th district since 2012".format(
    len(child_shooting_victims_since_2012_in_11th_dist)))
child_shooting_victims_16_and_under_since_2012_in_11th_dist = child_shooting_victims_16_and_under_since_2012[child_shooting_victims_16_and_under_since_2012['District'].apply(is_11th_district)]
print("There have been {} victims, under age 18, who have been shot in the 11th district since 2012".format(
    len(child_shooting_victims_since_2012_in_11th_dist)))

child_shooting_victims_since_2012_in_11th_dist_by_year = pd.DataFrame(
    child_shooting_victims_since_2012_in_11th_dist.groupby('year').size(),
    columns=['num_victims']
)
child_shooting_victims_since_2012_in_11th_dist_by_year


I am working on the file "Child shooting victims in the 11th district.ipynb".
The first task is:
Load police district boundaries
Can you write Python code for it?

import requests
from shapely.geometry import shape

# The City of Chicago's Socrata-based Data Portal provides a GeoJSON export of its spatial datasets.
# We'll use this so we don't have to save spatial data to the repo.
POLICE_DISTRICT_BOUNDARIES_GEOJSON_URL = "https://data.cityofchicago.org/api/geospatial/fthy-xz3r?method=export&format=GeoJSON"

r = requests.get(POLICE_DISTRICT_BOUNDARIES_GEOJSON_URL)
police_districts = r.json()

# Get the District 11 GeoJSON feature
district_11_feature = next(f for f in police_districts['features']
                           if f['properties']['dist_num'] == "11")

# Convert it to a Shapely shape so we can detect our
district_11_boundary = shape(district_11_feature['geometry'])


I am working on the file "Child shooting victims in the 11th district.ipynb".
The first task is:
So, it looks like using this method there are 3 more victims in 2014.  This could be due to bad or missing coordinates in some of the rows that cause them to not be categorized when we use the coordinates to detect district.
Can you write Python code for it?

child_shooting_victims_16_and_under_since_2012_in_11th_dist_by_year = pd.DataFrame(
    child_shooting_victims_16_and_under_since_2012_in_11th_dist.groupby('year').size(),
    columns=['num_victims']
)
child_shooting_victims_16_and_under_since_2012_in_11th_dist_by_year


I am working on the file "Choice-models-demo.ipynb".
The first task is:
Multinomial Logit
Can you write Python code for it?

# Import or generate estimation data

def rand(len, min, max):
    """ Generate `len` random floats uniformly distributed from `min` to `max` """
    return (max - min) * np.random.rand(len) + min

numalts = 50  # alternatives
X = np.concatenate((rand(numalts/2, 0, 10), rand(numalts/2, 100, 110)))

numobs = 1000  # agents/observations
beta = np.zeros(1000) + 1.5

U = [[beta[n]*x + np.random.gumbel() for x in X] for n in range(numobs)]  # utility matrix

choices = [np.argmax(a) for a in U]


I am working on the file "Choice-models-demo.ipynb".
The first task is:
This is what the data looks like:

N = 1000 agents/observations with true taste coefficients in array "beta"
J = 50 alternatives with single attributes in array "X"
N choice outcomes in array "choices"
Can you write Python code for it?

# Set up the estimation dataset in long format

d = [[n, i, int(choices[n]==i), X[i]] for n in range(numobs) for i in range(numalts)]
df = pd.DataFrame(d, columns=['obs_id', 'alt_id', 'chosen', 'x'])

print(df.describe())

# Set up model spec

spec = OrderedDict([('x', 'all_same')])
labels = OrderedDict([('x', 'beta_x')])

def estimate_model(init_val):
    """
    Initialize and fit a model, returning it as an object. Will use the
    current values of `df`, `spec`, and `labels`.
    """
    m = choicemodels.MNLogit(data = df,
                             alt_id_col = 'alt_id',
                             obs_id_col = 'obs_id',
                             choice_col = 'chosen',
                             specification = spec,
                             names = labels)

    results = m.fit_mle(init_vals = np.array([init_val]))
    return results

%%time
results = estimate_model(init_val = 1.2)
print(results.summary())


I am working on the file "Chord Diagrams for Bokeh.ipynb".
The first task is:
The interactions between the groups can now be visualised by calling the plot method of the ChordDiagram class. The index of the group corresponds to the row of the input data.
Can you write Python code for it?

fig = cd.plot(group=0)
t = show(row(fig, ), notebook_handle=True)


I am working on the file "Chord-diagram.ipynb".
The first task is:
A chord diagram encodes information in two graphical objects:
  - ideograms, represented by   distinctly colored arcs of circles;
  - ribbons, that are planar shapes bounded by two quadratic Bezier curves and two arcs of circle, that can degenerate to a point;
Ideograms
Summing up the entries on each matrix row, one gets a value (in our example this value is equal to the number of posts by a community member).
Let us denote by total_comments the total number of posts recorded in this community.
Theoretically the interval [0, total_comments) is mapped linearly onto the unit circle, identified with the interval $[0,2\pi)$.
For a better looking plot one proceeds as follows: starting from the angular position $0$, in counter-clockwise direction, one draws succesively, around the unit circle,  two parallel arcs of length equal to a mapped row sum value, minus a fixed gap. Click the image below:
Can you write Python code for it?

from IPython.display import IFrame

IFrame('https://plot.ly/~empet/12234/', width=377, height=420)


I am working on the file "Chord-diagram.ipynb".
The first task is:
Compute the row sums and the lengths of corresponding ideograms:
Can you write Python code for it?

row_sum = [matrix[k,:].sum() for k in range(L)]

#set the gap between two consecutive ideograms
gap = 2*pi*0.005
ideogram_length = 2*pi * np.asarray(row_sum) / sum(row_sum) - gap*np.ones(L)


I am working on the file "Chord-diagram.ipynb".
The first task is:
Define the array of data:
Can you write Python code for it?

matrix = np.array([[16,  3, 28,  0, 18],
                   [18,  0, 12,  5, 29],
                   [ 9, 11, 17, 27,  0],
                   [19,  0, 31, 11, 12],
                   [23, 17, 10,  0, 34]], dtype=int)

def check_data(data_matrix):
    L, M = data_matrix.shape
    if L != M:
        raise ValueError('Data array must have a (n,n) shape')
    return L

L = check_data(matrix)


I am working on the file "Chord-diagram.ipynb".
The first task is:
Finally we are ready to define data and layout for the Plotly plot of the chord diagram.
Can you write Python code for it?

import plotly.graph_objects as go

def plot_layout(title, plot_size):

    return dict(title=title,
                xaxis=dict(visible=False),
                yaxis=dict(visible=False),
                showlegend=False,
                width=plot_size,
                height=plot_size,
                margin=dict(t=25, b=25, l=25, r=25),
                hovermode='closest',
                 )


I am working on the file "Chord-diagram.ipynb".
The first task is:
For a better looking chord diagram,
Circos documentation recommends to sort increasingly each row of the mapped_data.

The  array idx_sort, defined below, has on each row the indices that sort the corresponding row in mapped_data:
Can you write Python code for it?

idx_sort = np.argsort(mapped_data, axis=1)
idx_sort


I am working on the file "Chord-diagram.ipynb".
The first task is:
Now let us explain the key point of associating  ribbons  to right data:
From the definition of ribbon_ends we notice that ribbon_ends[k][j] corresponds to data stored in
matrix[k][sigma[j]], where sigma is the permutation of indices $0, 1, \ldots L-1$, that sort the row k in mapped_data.
If sigma_inv is the inverse permutation of sigma, we get that to matrix[k][j] corresponds the
ribbon_ends[k][sigma_inv[j]].
ribbon_info is a list of dicts setting the information that is displayed when hovering the mouse over the  ribbon ends.
Set the radius of Bézier control point, $b_1$, for each ribbon associated to a diagonal data entry:
Can you write Python code for it?

radii_sribb = [0.4, 0.30, 0.35, 0.39, 0.12]# these value are set after a few trials

ribbon_info = []
shapes = []
for k in range(L):

    sigma = idx_sort[k]
    sigma_inv = invPerm(sigma)
    for j in range(k, L):
        if matrix[k][j] == 0 and matrix[j][k]==0: continue
        eta = idx_sort[j]
        eta_inv = invPerm(eta)
        l = ribbon_ends[k][sigma_inv[j]]

        if j == k:
            shapes.append(make_self_rel(l, 'rgb(175,175,175)' ,
                                    ideo_colors[k], radius=radii_sribb[k]))
            z = 0.9*np.exp(1j*(l[0]+l[1])/2)

            #the text below will be displayed when hovering the mouse over the ribbon
            text = f'{labels[k]} commented on {int(matrix[k][k])} of herself Fb posts'

            ribbon_info.append(go.Scatter(x=[z.real],
                                       y=[z.imag],
                                       mode='markers',
                                       marker=dict(size=0.5, color=ideo_colors[k]),
                                       text=text,
                                       hoverinfo='text'
                                       )
                              )
        else:
            r = ribbon_ends[j][eta_inv[k]]
            zi = 0.9 * np.exp(1j*(l[0]+l[1])/2)
            zf = 0.9 * np.exp(1j*(r[0]+r[1])/2)
            #texti and textf are the strings that will be displayed when hovering the mouse
            #over the two ribbon ends
            texti = f'{labels[k]} commented on {int(matrix[k][j])} of {labels[j]} Fb posts'
            textf = f'{labels[j]} commented on {int(matrix[j][k])} of {labels[k]} Fb posts'

            ribbon_info.append(go.Scatter(x=[zi.real],
                                          y=[zi.imag],
                                          mode='markers',
                                          marker=dict(size=0.5, color=ribbon_color[k][j]),
                                          text=texti,
                                         ...


I am working on the file "Chord-diagram.ipynb".
The first task is:
Ribbons in a chord diagram
While ideograms illustrate how many comments posted each member of the Facebook community,  ribbons
give a comparative information on the flows of comments from one friend to another.
To illustrate this flow we map data onto the unit circle. More precisely, for each matrix row, $k$, the application:
t$\mapsto$ t*ideogram_length[k]/row_sum[k]
maps the interval [0, row_sum[k]] onto
the interval [0, ideogram_length[k]]. Hence each entrymatrix[k][j]in the $k^{th}$  row is mapped tomatrix[k][j] * ideogram_length[k] / row_value[k]`.
The function map_data maps all matrix entries to the corresponding values in the intervals associated to ideograms:
Can you write Python code for it?

def map_data(data_matrix, row_value, ideogram_length):
    mapped = np.zeros(data_matrix.shape)
    for j  in range(L):
        mapped[:, j] = ideogram_length * data_matrix[:,j] / row_value
    return mapped

mapped_data = map_data(matrix, row_sum, ideogram_length)
mapped_data


I am working on the file "Chord-diagram.ipynb".
The first task is:
Set ideograms labels and colors:
Can you write Python code for it?


I am working on the file "Chord-diagram.ipynb".
The first task is:
The aim of our visualization is to illustrate the total number of posts by each community member, and the
flows of posts between pairs of friends.
Can you write Python code for it?

import platform, plotly
import numpy as np
from numpy import pi

print(f'Python version: {platform.python_version()}')
print(f'Plotly version: {plotly.__version__}')


I am working on the file "Chord-diagram.ipynb".
The first task is:
The function make_ideogram_arc returns equally spaced points on an ideogram arc, expressed as complex
numbers in polar form:
Can you write Python code for it?

def make_ideogram_arc(R, phi, a=50):
    # R is the circle radius
    # phi is the list of  angle coordinates of an arc ends
    # a is a parameter that controls the number of points to be evaluated on an arc
    if not test_2PI(phi[0]) or not test_2PI(phi[1]):
        phi = [moduloAB(t, 0, 2*pi) for t in phi]
    length = (phi[1]-phi[0]) % 2*pi
    nr = 5 if length <= pi/4 else int(a*length/pi)

    if phi[0] < phi[1]:
        theta = np.linspace(phi[0], phi[1], nr)
    else:
        phi = [moduloAB(t, -pi, pi) for t in phi]
        theta = np.linspace(phi[0], phi[1], nr)
    return R * np.exp(1j*theta)


I am working on the file "Chord-diagram.ipynb".
The first task is:
The next function returns the list of  end angular coordinates for each ideogram arc:
Can you write Python code for it?

def get_ideogram_ends(ideogram_len, gap):
    ideo_ends = []
    left = 0
    for k in range(len(ideogram_len)):
        right = left + ideogram_len[k]
        ideo_ends.append([left, right])
        left = right + gap
    return ideo_ends

ideo_ends = get_ideogram_ends(ideogram_length, gap)


I am working on the file "Chord-diagram.ipynb".
The first task is:
The symmetric locations are not modified, because  we do not access
ribbon_color[k][j], $k>j$, when drawing the ribbons.
Functions that return the Plotly SVG paths that are  ribbon boundaries:
Can you write Python code for it?

def make_q_bezier(b):# defines the Plotly SVG path for a quadratic Bezier curve defined by the
                     #list of its control points
    if len(b) != 3:
        raise valueError('control poligon must have 3 points')
    A, B, C = b
    return f'M {A[0]}, {A[1]} Q {B[0]}, {B[1]} {C[0]}, {C[1]}'

b=[(1,4), (-0.5, 2.35), (3.745, 1.47)]

make_q_bezier(b)


I am working on the file "Chord-diagram.ipynb".
The first task is:
To each pair of values (mapped_data[k][j], mapped_data[j][k]), $k<=j$,  one associates a ribbon, that is a curvilinear filled rectangle (that can be degenerate), having as opposite sides two subarcs of the $k^{th}$ ideogram, respectively $j^{th}$ ideogram, and two arcs of quadratic Bézier curves.

Here we illustrate the ribbons associated to pairs (mapped_data[0][j], mapped_data[j][0]), $j=\overline{0,4}$,
that illustrate the flow of comments between Emma and all other friends, and  herself:
Can you write Python code for it?


I am working on the file "Chord-diagram.ipynb".
The first task is:
We note that ribbon_ends[k][j] corresponds to mapped_data[i][idx_sort[k][j]], i.e. the length of the arc of ends in ribbon_ends[k][j] is equal to mapped_data[i][idx_sort[k][j]].
Now we define a few functions that compute the  control points for Bézier ribbon sides.
The function control_pts returns the cartesian coordinates of  the  control points, $b_0, b_1, b_2$, supposed as being  initially located on the unit circle, and thus defined only by their angular coordinate.  The angular coordinate
of the point $b_1$ is the mean of angular coordinates of the points $b_0, b_2$.
Since for a  Bézier ribbon side only $b_0, b_2$ are placed on the unit circle, one gives radius as a parameter that controls position of $b_1$.  radius is the distance  of $b_1$ to the circle center.
Can you write Python code for it?

def control_pts(angle, radius):
    #angle is a  3-list containing angular coordinates of the control points b0, b1, b2
    #radius is the distance from b1 to the  origin O(0,0)

    if len(angle) != 3:
        raise InvalidInputError('angle must have len =3')
    b_cplx = np.array([np.exp(1j*angle[k]) for k in range(3)])
    b_cplx[1] = radius * b_cplx[1]
    return list(zip(b_cplx.real, b_cplx.imag))

def ctrl_rib_chords(l, r, radius):
    # this function returns a 2-list containing control poligons of the two quadratic Bezier
    #curves that are opposite sides in a ribbon
    #l (r) the list of angular variables of the ribbon arc ends defining
    #the ribbon starting (ending) arc
    # radius is a common parameter for both control polygons
    if len(l) != 2 or len(r) != 2:
        raise ValueError('the arc ends must be elements in a list of len 2')
    return [control_pts([l[j], (l[j]+r[j])/2, r[j]], radius) for j in range(2)]


I am working on the file "Chord-diagram.ipynb".
The first task is:
and then eventually we are changing  the color in a few positions.
For our example we are perfotming the following color change:
Can you write Python code for it?


I am working on the file "Chord-diagram.ipynb".
The first task is:
make_ribbon_arc  returns the Plotly SVG path corresponding to an arc represented by its end angular coordinates, theta0, theta1.
Can you write Python code for it?

def make_ribbon_arc(theta0, theta1):

    if test_2PI(theta0) and test_2PI(theta1):
        if theta0 < theta1:
            theta0 = moduloAB(theta0, -pi, pi)
            theta1 = moduloAB(theta1, -pi, pi)
            if theta0  *theta1 > 0:
                raise ValueError('incorrect angle coordinates for ribbon')

        nr = int(40 * (theta0 - theta1) / pi)
        if nr <= 2: nr = 3
        theta = np.linspace(theta0, theta1, nr)
        pts=np.exp(1j*theta)# points in polar complex form, on the given arc

        string_arc = ''
        for k in range(len(theta)):
            string_arc += f'L {pts.real[k]}, {pts.imag[k]} '
        return   string_arc
    else:
        raise ValueError('the angle coordinates for an arc side of a ribbon must be in [0, 2*pi]')

make_ribbon_arc(np.pi/3, np.pi/6)


I am working on the file "Chris Tiu Project/Test2.ipynb".
The first task is:
As you can see, our stability criteria is still met, since $\Delta t \leq (\Delta x_{0})^{2}$. Infact, with a 40% larger N we see that we have less error as well! But what if we set N=15?
(The cell below is in Raw-NB Convert since we didn't want it to slow you down the first time you ran this program. Notice the N is now 15, go ahead and run this below cell and see what happens THIS time)
Can you write Python code for it?

dt=2.0e-6
nt=500000

N = 15

U_VGM15, s_VGM15, dx_VGM15 = VGM(nt, N, U0, dx, s, dt)

XX = numpy.linspace(0, 1, N)
Max_tempVGM15=max(U_VGM15)
pyplot.plot(XX,U_VGM15, color = '#654322', ls = '--', lw =4, label = 'VGM')
pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1, label='Analytical')
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(-.5, 2)
pyplot.xlim(0, 1.0)
pyplot.legend();
print('Time Step dt is:', dt)
print('Initial spatial step, dx[0]^2 is:', dx_VGM15[0]**2 )
print('Max error (@x=0) is:',abs(Max_tempVGM15-Max_tempA)*(100/Max_tempA),'Percent')


I am working on the file "Chris Tiu Project/Test2.ipynb".
The first task is:
As you can see, the ice SHOULD have melted about 0.07 meters  (or 2.75 inches) into our rod. In reality, our boundary interface has moved from x=0 to the right as time elapsed.  Not only should our temperature distribution profile change due to the differences in properties ($\rho, k, c_{p}$), but also the feedback from the moving boundary condition.
Solutions to the Dimensionless Stefan problem:
Analytical Solution:
Before we continue we need to make some simplfications:
1) No convection, heat transfer is limited to conduction,
2) Pressure is constant,
3) Density does not change between the solid and liquid phase (Ice/Water), ie $\rho_{ice}=\rho_{water}\approx 1000 \frac{kg}{m^3}$,
4) The phase change interface at $s(t)$ has no thickness
Looking closer at the problem at hand, we see that our temperature distribution must conform to the below figure:

 #### Phase Change Domain
As you can see, our temperature distribution experiences a discontinuity at our solid-liquid interface at x=s(t). Furthermore, this boundary moves to the right as time elapses. If we wish to analyze the distribution in one region or the other, we need to take into account a growing or shrinking domain, and therefore, the boundary of the domain has to be found as part of the solution [2].  In the Melting Problem depicted above, a moving interface separates the liquid and solid phases. The displacement of the boundary interface (denoted as $\dot{s}$) is driven by the heat transport through it [2]. The relationship  between the moving interface, s(t), and the temperature distribution through it was developed by and is known as the Stefan Condition (or Stefan Equation) and takes the form of the following boundary conditions:
when x =s(t),
[1, 3, 2]
$$\dot{s} = \frac{ds}{dt} = -\frac{\partial U}{\partial x}$$
$$U=0$$
There are many solution methods for the Stefan problem, for this lesson we will focus on finding the temperature distribution in the liquid region only (ie 0<x<s(t)). Critical parameters to track are both temperature, U(x,t),...
Can you write Python code for it?

def Exact_Stefan( nt, dt, x):
    U=numpy.ones(nx)
    for n in range(nt):
        U = numpy.exp(n*dt-x)-1
    return U

dt_A=0.002
nt_A=500
ExactS=Exact_Stefan(nt_A, dt_A, x)
Max_tempA=max(ExactS)

pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1)
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
print('Analytically, this is our temperature profile after:', nt_A*dt_A,'seconds')
print('The temperature at our LHS boundary is:', Max_tempA, 'degrees')


I am working on the file "Chris Tiu Project/Test2.ipynb".
The first task is:
Does the solution look right? YES! Remember, this temperature profile is "Dimensionless" so we can't compare it to our previous example (not only is the diffusivity constant vastly different, but we are looking 1 second into the diffusion vs 1 hr!). Also it accounts ONLY for the temperature distribution in the liquid, assuming an ever increasing domain due to an ever expanding RHS boundary. Not only did we expect temperature to be highest at the input side, but our moving boundary interface, s(t), moves to the right with time and always hits our melting temperature ($0^{\circ} c$) at x=1, just as one would expect.
Numerical Solution: The Variable Grid Method
What makes this problem unique is that we must now track the time-dependant moving boundary. Now, for previous numerical analysis in a 1D domain, you had a constant number of spatial intervals nx (or N) between your two boundaries x=0, and x=L. Thus your spatial grid steps dx were constant and defined as $dx = L/(nx-1)$. But now you have one fixed bounary and one moving boundary, and your domain increases as time passes. [1]

One of the key tenements of the Variable Grid Method for solving the Stefan probem is that you keep the NUMBER of grid points (nx) fixed, thus your grid SIZE (dx) will increase as the domain increases. Your grid size now varies dependant on the location of the interaface front, dx = s(t)/N.
$$dx = \frac{L}{N-1}     \longrightarrow      dx = \frac{s(t)}{N}$$

Now, while one might be tempted to view the figure above as the new "Variable Grid" stencil, one must remember that the "Y"-axis is time, and therefore as you move up you are moving forward in time, therefore a more accurate depiction of the FTCS method in stencil form for the Variable Grid method would be:

and so it becomes clear - that the our spatial step ($dx$) will be depentant on our time step ($m$)!!
Derivation of the new Governing Equation
Now let's set up the new governing equations to be discretized for our code. We know that the 1D Diffusion equation must be valid...
Can you write Python code for it?

def VGM(nt, N, U0, dx, s, dt):

    Uo=0                      #Initial Temperature input
    U0 = numpy.ones(N)*(Uo)
    s0=0.02                   #Initial Interface Position
                              #(cannot chose zero or our expressions would blow up!)
    s=numpy.ones(nt)*(s0)
    dx=numpy.ones(nt)*(s[1]/N)
    sdot=(s[1]-s[0])/dt

    s[1]= s[0] - (dt/(2*dx[0]))*(3*U0[N-1]-4*U0[N-2]+U0[N-3])
    dx[0]=(s[1]/N)

    for m in range(0, nt-1):

        for i in range(0, N-1):

         #LHS BC (x=0, i=0)
            if(i==0):U0[i]= (1-2*(dt/dx[m]**2))*U0[i]+2*(dt/dx[m]**2)*U0[i+1] + (2*(dt/dx[m])-dt*(dx[m]*i)*sdot/s[m])*numpy.exp(m*dt)

        #Governing Equation (B)
            else: U0[i]= U0[i]+ ((dt*(dx[m]*i)*sdot)/(2*dx[m]*s[m]))*(U0[i+1]-U0[i-1]) + (dt/(dx[m]**2))*(U0[i+1]-2*U0[i]+U0[i-1])

        #RHS BC (x=L, L=dx{m*N})
            U0[N-1]=0

        s[m+1]= s[m] - (dt/(2*dx[m]))*(3*U0[N-1]-4*U0[N-2]+U0[N-3])         #Heat Balance Equation, (E)
        sdot=(s[m+1]-s[m])/dt                                                                                #Updating Speed of Interface
        dx[m+1]=(s[m+1]/N)                                                                                #Updating dx for each time step m

        if (U0[i]>0):
            Location=s[m]

    return U0, s, dx

# First we set up some new initial parameters.These values differ from our first example since we are now dealing with
# a new non-dimensionalized heat equation, and new governing equations.

#1.0 Numerical Solution Parameters:
dt=2.0e-6        # Size of our Time-step, Set constant, Chosen in accordance with Caldwell and Savovic [5]
nt=500000      # Number of time steps, chosen so that elapsed solution will determine heat diffusion after
                       # (nt*dt= 1.0 seconds) Just like in our Analytical Solution.
N = 10            #  Number of spatial grid points, Chosen, for now, in accordance with Caldwell...


I am working on the file "Chris Tiu Project/Test2.ipynb".
The first task is:
Hey now! That looks pretty good! Let's compare this result to our earlier Analytical (exact) solution to the 1D Dimensionless problem, for 1 second into the diffusion:
Can you write Python code for it?

pyplot.figure(figsize=(12,10))
pyplot.plot(XX,U_VGM, color = '#654322', ls = '--',  lw =4, label = 'Variable Grid Method')
pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1, label='Analytical Solution')
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(0, 1.8)
pyplot.xlim(0, 1.0)
pyplot.legend();

print('Max error (@x=0) is:',abs(Max_tempVGM - Max_tempA)*(100/Max_tempA),'percent')


I am working on the file "Chris Tiu Project/Test2.ipynb".
The first task is:
Now let's define a function to run the governing equations using the Forward-Time/Central-Difference Method discretization
Can you write Python code for it?

def FTCS(U, nt, dt, dx, alpha, beta):

    for n in range(nt):

        Un=U.copy()
        U[1:-1] = Un[1:-1] + alpha*(dt/dx**2)*(Un[2:]-2*Un[1:-1]+Un[0:-2])

        #Boundary Conditions

        U[-1]=U[-2]                  #RHS Insulated BC

        t=n*dt                                    #Increasing time = n*timestep(dt)
        U[0]=numpy.exp(beta*t)     #LHS Time dependant heat input in, BC

    return U

#Basic Parameters and Initialization

# Temperature scale: Celsius
# Length scale: meters
# Mass scale: kg
# Time scale: seconds
# Energy scale: Joules
# Power scale: Watts

L = 1                                                         # Length of my ice rod
nt = 40000                                                  # Number of timesteps
nx = 51                                                     # Number of grid space steps
alpha = 1.083e-6                                    # Thermal Diffusivity
dx = L/(nx-1)                                           # grid spacing in "x"
Ui = numpy.ones(nx)*(-10)                    # initialized Temperature array
beta = 0.001                                               #Growth Factor of my Temperature input

sigma = 1/2                                             # Stability*
dt=0.1                                                      # Timestep chosen to be 0.1 seconds*


I am working on the file "Chris Tiu Project/Test2.ipynb".
The first task is:
Results and Discussions
Ok! Now lets see what we get, and more importantly, if it makes any sense:
Can you write Python code for it?

XX = numpy.linspace(0, 1, N)
Max_tempVGM=max(U_VGM)

pyplot.plot(XX,U_VGM, color = '#654322', ls = '--', lw =4)
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(-.5, 2)
pyplot.xlim(0, 1.0)

print('This is our VGM temperature profile after:', nt*dt, 'seconds')
print('The temperature at our LHS boundary is:', Max_tempVGM, 'degrees')
print('Final position of our interface is:', s_VGM[m])
print('The final speed of our interface is:', sdot)
print('Our grid spaceing (dx) after', nt*dt, 'seconds is:', dx_VGM[m] )


I am working on the file "Chris Tiu Project/Test2.ipynb".
The first task is:
The above figure shows us the temperature distribution of the ice rod at a point in time approximately 1 hour into applying our heat source. Now this would be all well and good if ice didn't melt (and therefore change phase) at zero degrees. But it does!!
Let's build a rudimentary function to see what portion of the rod should be water by now, and which part should still be ice:
Can you write Python code for it?

def Phase_graph(U, x, nx):
    phase=numpy.ones(nx)*(-100)
    for n in range(nx):
        if U[n]>0:
            phase[n]=100
    return phase

pyplot.plot(x, U, color = '#003366')
pyplot.ylabel('Temperature', color ='#003366')
Phase1=numpy.ones(nx)
Phase=Phase_graph(U, x, nx)
pyplot.plot(x, Phase, color = '#654321', ls = '-', lw =3)
pyplot.ylim(-12, 10)
pyplot.xlim(0, 0.3)
pyplot.xlabel('Approx Water-Ice Interface', color = '#654321')


I am working on the file "Chris Tiu Project/Test2.ipynb".
The first task is:
Wow! That blew up! Just as we expect it would now that with N=15 our time step is larger than our initial spatial step!
Final Thoughts
And now we know, that the Stefan Problem, a boundary value PDE with  a time-dependant moving boundary, CAN be solved. We have just demonstrated the solution for a 1D, Dimensionless Stefan Problem with specifically chosen (time-dependant) input heat flux in order to give us a simplified exact solution to compare with. You have also seen implementation of the Variable Grid Method, one of many ways inwhich one can numerical simulate the Stefan problem. But we have also seen the limitations, namely in small time-steps ($dt$), smaller numerical domain ($N\cdot dx_{f}$), and large number of iterations ($nt$) for just a small amount of analytical time (t=$nt \cdot dt$ seconds). Perhaps when it comes time for you to model the melting of the Polar Ice caps, you'll choose a more "expedient" method? Just make sure you get the answer BEFORE the caps melt...)
EXERCISE #1: Can you implement the Variable Grid Method discretization governing equations to give us a solution faser? Can it handle millions of time iterations without putting us to sleep?
EXERCISE #2: Can you write a stability condition statement that will maximize our time-step (dt) for a given N and still keep it constant?
References

Kutluay S., The numerical solution of one-phase classical Stefan problem, Journal of Computational and Applied Mathematics 81 (1997) 135-144

Javierre, E., A Comparison of Numeical Models for one-dimensional Stefan problems, Journal of Compuational and Applied Mathematics 192 (2006) 445-459

Vuik, C., "Some historical notes about the Stefan problem". Nieuw Archief voor Wiskunde, 4e serie 11 (2): 157-167 (1993)

Crowley, A. B., Numerical Solution of Stefan Problems, Brunel University, Department of Mathematics, TR/69 December 1976

Caldwell, J., Numerical Solution of Stefan Problem By Variable Space Grid Method and Boundary Immodbilisation Method, Jour. of Mathematical Sciences, Vol. 13 No.1...
Can you write Python code for it?

# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file ='numericalmoocstyle.css'
HTML(open(css_file, "r").read())


I am working on the file "Chris Tiu Project/Test2.ipynb".
The first task is:
and so we see that both the spatial grid size, dx, and the interface location, s(t), increase with time over the course of 1 second. This is exactly what we would expect! Infact, remember back to our anayltical solution, we solved the interface function  to be: $s(t)=t$, which is exactly the graph that our numerical solution has given us!
Well, then! That's a wrap! Looks like everything is all accounted for, right?. . . . . . . . . . . .  . .well, not exactly. There are still quite a few questions left unanswered, and now we must discuss the limitations in the Variable Grid Method of numerical implementation.
Limitations of the Variable Grid Method
A number of questions should be coming to mind, among those are:
1) Why do we only analyze for 1 second of diffusion?
2) Why is your VGM timestep (dt) so small at 2.0e-6?
3) Why is your number of spatial grid points (N) only 10?
These are all VERY good questions, do you already know the answer? Here is a hint: it all has to do with stability!
You see, we have a relatively benign stability statement: $\Delta t \leq (\Delta x^{m})^{2}$, but still, it is a crutial aspect of our numerical analysis. This statement limits the size of our timestep, dt, and is the key reason why we chose our initial, constant, dt to be 2.0e-6 (lets not also forget the fact that this dt and N were also chosen by Caldwell and Savovic [5]) . Remember, because we are using a Central Difference scheme, our stability criteria essentially comes from the CFL condition:
$$\frac{\Delta t}{(\Delta x^{m})^{2}} \leq \sigma = 1$$.
The parameters dt and dx are not just arbitrarilty chosen. They determine the speed of your numerical solution. At all times you need the speed at which your solution progresses to be faster than the speed at which the problem (in this case, thermal diffusion) propagates. Afterall, how can you calculate the solution numerically when the solution is faster than the numerical analysis?!
Here is a pertinent question for you: Which do you determine first: dt, nt, N, or L?...
Can you write Python code for it?

dt=2.0e-6
nt=500000

N = 14

U_VGM14, s_VGM14, dx_VGM14 = VGM(nt, N, U0, dx, s, dt)

XX = numpy.linspace(0, 1, N)
Max_tempVGM14=max(U_VGM14)
pyplot.plot(XX,U_VGM14, color = '#654322', ls = '--', lw =4, label = 'VGM')
pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1, label='Analytical')
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(-.5, 2)
pyplot.xlim(0, 1.0)
pyplot.legend();
print('Time Step dt is:', dt)
print('Initial spatial step, dx[0]^2 is:', dx_VGM14[0]**2 )
print('Max error (@x=0) is:',abs(Max_tempVGM14-Max_tempA)*(100/Max_tempA),'Percent')


I am working on the file "CiC_lecture_01_as_taught.ipynb".
The first task is:
Comma-separated value (CSVs) (files)
LeBron James' first five games of the 2013-2014 NBA season
Can you write Python code for it?

import csv
import urllib

url = "https://gist.githubusercontent.com/aparrish/cb1672e98057ea2ab7a1/raw/13166792e0e8436221ef85d2a655f1965c400f75/lebron_james.csv"
stats = list(csv.reader(urllib.urlopen(url)))
#example courtesy the great Allison Parrish!
#What different things do urllib.urlopen(url) then csv.reader() and then list() do?

stats[0]

len(stats)

stats[74][0]


I am working on the file "CiC_lecture_01_as_taught.ipynb".
The first task is:
In inputing CSV, Pandas parses each column and attempts to discern what sort of data is within. It's good but not infallible.
- Pandas is particularly good with dates: you simply tell it which columns to parse as dates.
Let's refine our reading of the CSV to parse the dates.
Can you write Python code for it?

df=pd.read_csv('HMXPC_13.csv', sep="," , parse_dates=['start_time_DI', 'last_event_DI'])


I am working on the file "CiC_lecture_01_as_taught.ipynb".
The first task is:
Instead of (column, row) we use name_of_dataframe[column name][row #]
Can you write Python code for it?

df[3340:3350] # SLICE a list of ROWS

#This was _not_ in class   PREPARE FOR TERRIBLE ERROR!
#THIS DOESN'T WORK
df[3340]

#That's icky.
#to pick out one row use `.ix`
df.ix[3340]


I am working on the file "CiC_lecture_01_as_taught.ipynb".
The first task is:
Now we can count how many times someone started
Can you write Python code for it?

startdates=df['start_time_DI'].value_counts()
# Exercise to the reader: how might you do this without using the `.value_counts()` method?

startdates

startdates.plot()

startdates.plot(title="I can't it's not butter.")


I am working on the file "CiC_lecture_01_as_taught.ipynb".
The first task is:
Now we read a big csv file using a function from pandas called pd.read_csv()
Can you write Python code for it?

df=pd.read_csv('HMXPC_13.csv', sep=",")

df


I am working on the file "CiC_lecture_01_as_taught.ipynb".
The first task is:
We can pick out columns using their names and with a slice of rows.
Can you write Python code for it?


I am working on the file "CiC_lecture_01_as_taught.ipynb".
The first task is:
note that we pass a list of columns to pick out multiple columns
Can you write Python code for it?


I am working on the file "CiC_lecture_03_text_mining_redux.ipynb".
The first task is:
IMPORTANT: tm is our temporarily helper, not a standard python package!!
download it from my github:
https://github.com/matthewljones/computingincontext
Can you write Python code for it?

#see if package imported correctly
tm.icantbelieve("butter")


I am working on the file "CiC_lecture_03_text_mining_redux.ipynb".
The first task is:
Let's get some text
Let's use the remarkable narratives available from Documenting the American South (http://docsouth.unc.edu/docsouthdata/)
Assuming that you are storing your data in a directory in the same place as your iPython notebook.
Put the slave narratives texts within a data directory in the same place as this notebook
Can you write Python code for it?

title_info=pd.read_csv('data/na-slave-narratives/data/toc.csv')
#this is the "metadata" of these files--we didn't use today
#why does data appear twice?

#Let's use a brittle thing for reading in a directory of pure txt files.
our_texts=tm.readtextfiles('data/na-slave-narratives/data/texts')
#again, this is not a std python package
#returns a simple list of the document as very long strings

#note if you want the following notebook will work on any directory of text files.

len(our_texts)

our_texts[100][:300] # first 300 words of 100th text


I am working on the file "CiC_lecture_03_text_mining_redux.ipynb".
The first task is:
Our first tool: vectorizer from scikit learn
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer=TfidfVectorizer(min_df=0.5, stop_words='english', use_idf=True)

document_term_matrix=vectorizer.fit_transform(our_texts)


I am working on the file "CiC_lecture_03_text_mining_redux.ipynb".
The first task is:
right now stored super efficiently as a sparse matrix
almost all zeros--good for our computers' limited memory
easier for us to see as a dense matrix
Can you write Python code for it?

document_term_matrix_dense=document_term_matrix.toarray()

dtmdf=pd.DataFrame(document_term_matrix_dense, columns=vocab)

dtmdf


I am working on the file "CiC_lecture_04_text_mining_topics.ipynb".
The first task is:
IMPORTANT: tm is our temporarily helper, not a standard python package!!
download it from my github:
https://github.com/matthewljones/computingincontext
Can you write Python code for it?


I am working on the file "CiC_lecture_04_text_mining_topics.ipynb".
The first task is:
Let's keep using the remarkable narratives available from Documenting the American South (http://docsouth.unc.edu/docsouthdata/)
Assuming that you are storing your data in a directory in the same place as your iPython notebook.
Let's look at Victorian novels for a little while
Can you write Python code for it?

our_texts, names=tm.readtextfiles("data/british-fiction-corpus")

names

our_texts=tm.data_cleanse(our_texts)
#more necessary when have messy text
#eliminate escaped characters


I am working on the file "CiC_lecture_04_text_mining_topics.ipynb".
The first task is:
Now we are going to call the topic modeling black box
the key parameter is how many distinct topics we want the computer to find
this will take a while
Can you write Python code for it?

number_topics=40
model = models.LdaModel(corpus, id2word=dictionary, num_topics=number_topics, passes=10) #use gensim multicore LDA

model.show_topics()

topics_indexed=[[b for (a,b) in topics] for topics in model.show_topics(number_topics,10,formatted=False)]
topics_indexed=pd.DataFrame(topics_indexed)

topics_indexed


I am working on the file "CiC_lecture_04_text_mining_topics.ipynb".
The first task is:
Reading at scale:
Martha Ballard's Diary
http://dohistory.org/diary/index.html
http://www.cameronblevins.org/posts/topic-modeling-martha-ballards-diary/
Richmond Dispatch
https://dsl.richmond.edu/dispatch/pages/home
Source: http://dlxs.richmond.edu/d/ddr/index.html
Can you write Python code for it?

from IPython.display import Image
Image("http://journalofdigitalhumanities.org/wp-content/uploads/2013/02/blei_lda_illustration.png")

import textmining_blackboxes as tm
##when I put this on-line you'll have topic modeling helper functions!!


I am working on the file "CiC_lecture_04_text_mining_topics.ipynb".
The first task is:
So which topics most significant for each document?
Pass a bag of words version of each document to the model.
Can you write Python code for it?


I am working on the file "CiC_lecture_04_text_mining_topics.ipynb".
The first task is:
What has this got us?
It suggests that even this crude measure of similarity is able to capture something significant.
Note: the axes don't really mean anything
interesting but what does it mean?
topic modeling
unsupervised algorithm for finding the major topics of texts
unlike hierarchical clustering, assumes texts spring from multiple sets of topics
the big thing in much text modeling, from humanities, to Facebook, to NSA
many variations
fantastic python package gensim
"corpora" = a collection of documents or texts
gensim likes its documents to be a list of lists of words, not a list of strings
Get the stoplist in the data directory in my github.
Can you write Python code for it?

our_texts, names=tm.readtextfiles("Data/PCCIPtext")

our_texts=tm.data_cleanse(our_texts)

#improved stoplist--may be too complete
stop=[]
with open('data/stoplist-multilingual') as f:
    stop=f.readlines()
    stop=[word.strip('\n') for word in stop]

texts = [[word for word in document.lower().split() if word not in stop] for document in our_texts] #gensim requires list of list of words in documents

from gensim import corpora, models, similarities, matutils
"""gensim includes its own vectorizing tools"""
dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]

#doc2bow just means `doc`uments to `b`ag `o`f `w`ords
#ok, this has just vectorized our texts; it's another form


I am working on the file "CiC_lecture_04_text_mining_topics.ipynb".
The first task is:
While this data frame is lovely to look at and useful to think with, it's tough on your computer's memory
Can you write Python code for it?

#easy to program, but let's use a robust version from sklearn!
from sklearn.metrics.pairwise import cosine_similarity

similarity=cosine_similarity(document_term_matrix)

#Note here that the `cosine_similiary` can take
#an entire matrix as its argument

pd.DataFrame(similarity, index=names, columns=names)


I am working on the file "CiC_lecture_04_text_mining_topics.ipynb".
The first task is:
back to vectorizer from scikit learn
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer=TfidfVectorizer(min_df=0.5, stop_words='english', use_idf=True)

document_term_matrix=vectorizer.fit_transform(our_texts)

# now let's get our vocabulary--the names corresponding to the rows
vocab=vectorizer.get_feature_names()

len(vocab)

document_term_matrix.shape

document_term_matrix_dense=document_term_matrix.toarray()

dtmdf=pd.DataFrame(document_term_matrix_dense, columns=vocab)

dtmdf


I am working on the file "CiC_lecture_04_text_mining_topics.ipynb".
The first task is:
can do lots of things with similarity matrix
you've already seen hierarchical clustering
Multidimension scaling

Technique to visualize distances in high dimensional spaces in ways we can cognize.
Keep distances but reduce dimensionality
Can you write Python code for it?

#here's the blackbox
from sklearn.manifold import MDS
mds = MDS(n_components=2, dissimilarity="precomputed", random_state=1)
positions= mds.fit_transform(1-similarity)

positions.shape


I am working on the file "CiC_lecture_04_text_mining_topics.ipynb".
The first task is:
make it pretty with a list comprehension
Can you write Python code for it?

import numpy as np
primarytopics_matrix=pd.DataFrame(np.matrix([matutils.sparse2full(primarytopic, number_topics) for primarytopic in primarytopics]))

primarytopics_matrix.ix[18].plot(kind="bar")

primarytopics_matrix.ix[18].plot(kind="bar", title=names[18])

topics_indexed.ix[[6, 21, 27, 38]]

#FIND which documents focus on a particular topic

primarytopics_matrix[27].plot(kind="bar")

primarytopics_matrix[17].plot(kind="bar", title=str(topics_indexed.ix[17]))

#that's ugly!


I am working on the file "CinC_lecture_02_vectorizing.ipynb".
The first task is:
Here's our help!
Python Libraries
Python has an embarrasment of riches when it comes to working with texts. Some libraries are higher level with simpler, well thought out defaults, namely pattern and TextBlob. Most general, of long development, and foundational is the Natural Language Tool Kit--NLTK. The ideas we'll learn to today are key--they have slightly different instantiations in the different tools. Not everything is yet in Python 3.
For now, we'll play with the cool scientists and use the powerful and fast scikit learn package.
Can you write Python code for it?

vectorizer=TfidfVectorizer(min_df=0.95, stop_words='english')
#.95 is a VERY high threshold--only the most common words--chosen for the form of visualization we're going to do

document_term_matrix=vectorizer.fit_transform(articles)

document_term_matrix.shape

#output is number of documents, then size of remaining vocabulary
rows, terms=document_term_matrix.shape

vocab=vectorizer.get_feature_names()

len(vocab)

dtm=document_term_matrix.toarray()
dtmdf=pd.DataFrame(dtm, columns=vocab)


I am working on the file "CinC_lecture_02_vectorizing.ipynb".
The first task is:
Now we can throw wide variety of mining algorithms at our data!
Similarity and dissimilarity
We reduced our text to a vector of term-weights. What can we do once we've committed this violence on the text?
We can measure distance and similarity
I know. Crazy talk.
Right now our text is just a series of numbers, indexed to words. We can treat it like any other set of words.
And the key way to distinguish two vectors is by measuring their distance or computing their similiarity (1-distance).
You already know how, though you may have buried it along with memories of high school.
Many distance metrics to choose from
key one in textual analysis:
cosine similarity
If $\mathbf{a}$ and $\mathbf{b}$ are vectors, then
$\mathbf{a}\cdot\mathbf{b}=\left\|\mathbf{a}\right\|\left\|\mathbf{b}\right\|\cos\theta$
Or
$\text{similarity} = \cos(\theta) = {A \cdot B \over \|A\| \|B\|} = \frac{ \sum\limits_{i=1}^{n}{A_i \times B_i} }{ \sqrt{\sum\limits_{i=1}^{n}{(A_i)^2}} \times \sqrt{\sum\limits_{i=1}^{n}{(B_i)^2}} }$
Can you write Python code for it?

#easy to program, but let's use a robust version
from sklearn.metrics.pairwise import cosine_similarity

#cosine similarity is vectorized: that means is will operate on an entire matrix, not just its individual elements

similarity=cosine_similarity(dtmdf)

similarity

import matplotlib.pyplot as plt
#we can make a heatmap with no problems within mathplotlib
#pass plt.pcolor our similiarity matrix
plt.pcolor(similarity, norm=None, cmap='Blues')

#we have too many documents for that to be very useful; so
plt.pcolor(similarity[100:110, 100:110], norm=None, cmap='Blues')


I am working on the file "CinC_lecture_02_vectorizing.ipynb".
The first task is:
Once we convert texts into vectors, computer doesn't care or know that we're dealing with something formerly known as texts. It's just another kind of vector . . .
Term frequency [TF]
The "term frequency" measures how often a given term occurs in a given document.
We count up how many times each term appears in a document, then divide it by the number of terms in the document.
For a word $t$ that appears $i_w$ times in a document $D$ with a number of words $n_D$, the term frequency is
$tf(t,D)=\frac{i_w}{n_D}$
Can you think of a problem with this as a measure?
Can you write Python code for it?

def document_vector_freq(wordstring):
    """put yer documentation here friend"""
    wordlist = wordstring.split()
    number_of_words=len(wordlist)
    set_of_words=set(wordlist)
    distinct_words=list(set_of_words)
    wordfreq = [wordlist.count(w) for w in distinct_words]
    wordfreq = [word_freq/number_of_words for word_freq in wordfreq]
    return distinct_words, wordfreq


I am working on the file "CinC_lecture_02_vectorizing.ipynb".
The first task is:
cool beans--we've vectorized our sentence
what has been lost?
Can you write Python code for it?


I am working on the file "CinC_lecture_02_vectorizing.ipynb".
The first task is:
put into pandas just so we could explore it a bit more elegantly
Can you write Python code for it?


I am working on the file "Cincinnati311DataEDA.ipynb".
The first task is:
Download the Cincinnati 311 (Non-Emergency) Service Requests data

Dataset Description
Example of downloading a *.csv file progamatically using urllib2
Can you write Python code for it?

data_dir = "./Data"

csv_file_path = os.path.join(data_dir, "cincinnati311.csv")

if not os.path.exists(csv_file_path):

    if not os.path.exists(data_dir):
        os.mkdir(data_dir)

    url = 'https://data.cincinnati-oh.gov/api/views' +\
          '/4cjh-bm8b/rows.csv?accessType=DOWNLOAD'

    response = urllib2.urlopen(url)

    html = response.read()

    with open(csv_file_path, 'wb') as h_file:
        h_file.write(html)


I am working on the file "Circle.ipynb".
The first task is:
Linear example
The following are four different plots with linear scales to illustrate the problem addressed and the usage of circle:

1.a Using plt.Circle to draw circles on a figure with equal axes.
1.b Using plt.Circle to draw circles on a figure with unequal axes.
1.c Using patches.Ellipse on a figure with unequal axes to draw apparent circles.
1.d Using this custom circle function on a figure with unequal axes to plot circles.
Can you write Python code for it?

# Generate some data to plot
x = np.arange(101)/100*2*np.pi
y = np.sin(x)

# Circle centres
cxy = (np.arange(5)*np.pi/2, np.sin(np.arange(5)*np.pi/2))

## 1.a Using plt.Circle on equal axes

# Create figure
fig1a = plt.figure()

# Set axis to equal
plt.axis('equal')

# Plot data and set limits
plt.plot(x, y, '-', c=clr[6], lw=2)
plt.xlim([min(x), max(x)])

# Draw circles with plt.Circle
# (`clip_on: False` ensures that the circles are not cut-off at fig-border.)
for i in range(5):
    circ = plt.Circle((cxy[0][i], cxy[1][i]), .25, **{'color':clr[i], 'clip_on': False})
    plt.gca().add_artist(circ)

# Set labels
plt.title('1.a plt.Circle with equal axes')
plt.text(1, -1.5, r'$y = \rm{sin}(x)$', fontsize=20)
plt.xlabel('x')
plt.ylabel('y')

# Multiply y-values by 5, to make the effect of unequal axes more apparent
y *= 5
cxy = (np.arange(5)*np.pi/2, 5*np.sin(np.arange(5)*np.pi/2))

## 1.b Using plt.Circle on unequal axes

# Create figure
fig1b = plt.figure()

# Plot data and set limits
plt.plot(x, y, '-', c=clr[6], lw=2)
plt.axis([min(x), max(x), 1.2*min(y), 1.2*max(y)])

# Draw circles with plt.Circle
for i in range(5):
    circ = plt.Circle((cxy[0][i], cxy[1][i]), .25, **{'color':clr[i], 'clip_on': False})
    plt.gca().add_artist(circ)

# Set labels
plt.title('1.b plt.Circle with unequal axes')
plt.text(1, -5, r'$y = 5\times\rm{sin}(x)$', fontsize=20)
plt.xlabel('x')
plt.ylabel('y')

## 1.c Using patches.Ellipse on unequal axes

# Create figure
fig1c = plt.figure()

# Plot data and set limits
plt.plot(x, y, '-', c=clr[6], lw=2)
plt.axis([min(x), max(x), 1.2*min(y), 1.2*max(y)])

# Calculate width and height of Ellipse to create an apparent circle
factor = fig1c.get_figwidth()*(max(1.2*y)-min(1.2*y))/fig1c.get_figheight()/(max(x)-min(x))

# Draw circles with patches.Ellipse
for i in range(5):
    circ = patches.Ellipse((cxy[0][i], cxy[1][i]), .5, .5*factor,
                           **{'color':clr[i], 'clip_on': False})
    plt.gca().add_artist(circ)

# Set labels
plt.title('1.c patches.Ellipse with...


I am working on the file "Circle.ipynb".
The first task is:
Load the circle-function
(You can find it in the notebook adashof.ipynb, in the same repo as this notebook).
Can you write Python code for it?

%load -s circle adashof.py

def circle(xy, radius, kwargs=None):
    """Create circle on figure with axes of different sizes.

    Plots a circle on the current axes using `plt.Circle`, taking into account
    the figure size and the axes units.

    It is done by plotting in the figure coordinate system, taking the aspect
    ratio into account. In this way, the data dimensions do not matter.
    However, if you adjust `xlim` or `ylim` after plotting `circle`, it will
    screw them up; set `plt.axis` before calling `circle`.

    Parameters
    ----------
    xy, radius, kwars :
        As required for `plt.Circle`.

    """

    # Get current figure and axis
    fig = plt.gcf()
    ax = fig.gca()

    # Calculate figure dimension ratio width/height
    pr = fig.get_figwidth()/fig.get_figheight()

    # Get the transScale (important if one of the axis is in log-scale)
    tscale = ax.transScale + (ax.transLimits + ax.transAxes)
    ctscale = tscale.transform_point(xy)
    cfig = fig.transFigure.inverted().transform(ctscale)

    # Create circle
    if kwargs == None:
        circ = patches.Ellipse(cfig, radius, radius*pr,
                transform=fig.transFigure)
    else:
        circ = patches.Ellipse(cfig, radius, radius*pr,
                transform=fig.transFigure, **kwargs)

    # Draw circle
    ax.add_artist(circ)


I am working on the file "Circuitikz-examples.ipynb".
The first task is:
Example: Transmission line
Can you write Python code for it?

%%circuitikz filename=tm dpi=150

\begin{circuitikz}[scale=1.25]

\draw (-1,0) node[anchor=east] {} to [short, *-*] (1,0);
\draw (-1,2) node[anchor=east] {} to [inductor, *-*,  l=$\Delta x L$] (1,2);
\draw (-1,0) to [open, l=$\cdots$] (-1,2);
\draw (3, 0) to (1, 0) to [capacitor, l=$\Delta x C$, *-*] (1, 2) to [inductor, *-*, l=$\Delta x L$] (3, 2);
\draw (5, 0) to (3, 0) to [capacitor, l=$\Delta x C$, *-*] (3, 2) to [inductor, *-*, l=$\Delta x L$] (5, 2);
\draw (7, 0) to (5, 0) to [capacitor, l=$\Delta x C$, *-*] (5, 2) to [inductor, *-*, l=$\Delta x L$] (7, 2);
\draw (9, 0) to (7, 0) to [capacitor, l=$\Delta x C$, *-*] (7, 2) to [inductor, *-*, l=$\Delta x L$] (9, 2);
\draw (9,0) node[anchor=east] {} to [short, *-*] (9,0);
\draw (10,0) to [open, l=$\cdots$] (10,2);

\end{circuitikz}


I am working on the file "Circuitikz-examples.ipynb".
The first task is:
Load the extension
Can you write Python code for it?


I am working on the file "Circuits/Resistance 1.ipynb".
The first task is:
Components
Each component is assumed to be a normal distriubution of exptected value as listed and the 4, 6, 8 Ohm resistors are 1% while the 12 Ohm is a 5% resistor.
Per http://docs.pymc.io/api/bounds.html we will use bounded variables because resistances that are negative are nonsensical.
Can you write Python code for it?

## setup the model

# these are the values and precision of each
Datasheets = {'R1':(6.0, 0.01),
              'R2':(8.0, 0.01),
              'R3':(4.0, 0.01),
              'R4':(12.0, 0.05),
              'V1':(6.0, 0.01),} # 1% on the 12V power supply

with pm.Model() as model:
    BoundedNormal = pm.Bound(pm.Normal, lower=0.0)
    # http://docs.pymc.io/api/distributions/continuous.html#pymc3.distributions.continuous.Normal
    # in Bayes world these are considered prior distributions, they are based on previous information
    #   that is gained in some other manner, from the datasheet in this case.
    R1 = BoundedNormal('R1', mu=Datasheets['R1'][0], sd=Datasheets['R1'][0]*Datasheets['R1'][1])
    R2 = BoundedNormal('R2', mu=Datasheets['R2'][0], sd=Datasheets['R2'][0]*Datasheets['R2'][1])
    R3 = BoundedNormal('R3', mu=Datasheets['R3'][0], sd=Datasheets['R3'][0]*Datasheets['R3'][1])
    R4 = BoundedNormal('R4', mu=Datasheets['R4'][0], sd=Datasheets['R4'][0]*Datasheets['R4'][1])
    # don't bound the voltage as negative is possilbe
    V1 = pm.Normal('V1', mu=Datasheets['V1'][0], sd=Datasheets['V1'][0]*Datasheets['V1'][1])

    # Match should all be done on paper first to get the full answer, but we will do steps here because one can.
    #   all at once would be much faster.

    # just add them, we will not get info on R2_3 at the output unless we wrap them in pm.Deterministic
    R2_3 = R2+R3
    #   R2_3 = pm.Deterministic('R2_3', R2+R3)

    # now get the resistance answer, and we want details
    R_eff = pm.Deterministic('R_eff', 1/(1/R2_3 + 1/R4))

    # total current is then just I=V/R
    I_t = pm.Deterministic('I_t', V1/R_eff)
    # and I_1 and I_2
    I_1 = pm.Deterministic('I_1', I_t*R2_3/R4)
    I_2 = pm.Deterministic('I_2', I_t-I_1)

    # makes it all a bit clearner to start in a good place
    start = pm.find_MAP()
    # run a fair number of samples, I have a 8 core machine so run 6
    trace = pm.sample(5000, start=start, njobs=6)


I am working on the file "Circuits/Resistance 1.ipynb".
The first task is:
Explore and explain the results
In the summary what is shown is the variable name ans summary information about each variable.
We see that:
$R_{eff} = 5.997 \pm 0.3$,  really $R_{eff}$ is between  $[5.705, 6.302]$
or $6\Omega \pm 5\%$
$I_t = 1.001 \pm 0.113$, $[0.892, 1.113]$
or $1.001 \pm 11\%$
So if one is going for a 5% accuracy on the current you cannot say that!
Can you write Python code for it?

I_t_perc = np.percentile(trace['I_t'], (2.5, 97.5))
I_t_mean = trace['I_t'].mean()
print('I_t = {:.4} +/- {:.4}'.format(I_t_mean, I_t_perc[1]-I_t_mean))
print('I_t = {:.4} +/- {:.4}%'.format(I_t_mean, (I_t_perc[1]-I_t_mean)/I_t_mean*100))

pm.summary(trace, varnames=('R_eff', 'I_t', 'I_1', 'I_2'))


I am working on the file "Circuits/Resistance 1.ipynb".
The first task is:
For this simple circuit the questions are
1. what is the end-to-end resistance and its uncertainity?
1. What are the currents I1 and I2 and uncertaianity?
For each we will assume also that the 12V is not perfect.
We will try and answer these questions for the circuit as designed and also for the cuircuit as built with a few measurments of the resistors and see how that impacts the results.
The results are:
$R_{eff} = 6\Omega$
$I_T = 1A, I_1 = I_2 = 0.5A$
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
import pymc3 as pm
import pandas as pd
import seaborn as sns

sns.set()
%matplotlib inline


I am working on the file "Circuits/Resistance 1.ipynb".
The first task is:
Results
So while not a huge difference a single measurment of R1 changed the results:
$R_{eff} = [5.705, 6.302]$ became $R_{eff} = [5.688, 6.282]$
$I_t = [0.892, 1.113]$ became $I_t = [0.891, 1.110]$
A not huge difference but a shift down.
Can you write Python code for it?

I_t_perc = np.percentile(trace['I_t'], (2.5, 97.5))
I_t_mean = trace['I_t'].mean()
print('I_t = {:.4} +/- {:.4}'.format(I_t_mean, I_t_perc[1]-I_t_mean))
print('I_t = {:.4} +/- {:.4}%'.format(I_t_mean, (I_t_perc[1]-I_t_mean)/I_t_mean*100))


I am working on the file "Circuits/Resistance 1.ipynb".
The first task is:
We also good visual diagnostics in terms of traceplots. The right side shows the draws, this should always look like hash, the left side is the Kernel Density Esimator of the output distribution. For this problem it should look pretty darn Normal.
For both summary and traceplot, leavre off varnames to jst get them all...
Can you write Python code for it?

pm.traceplot(trace, combined=True, varnames=('R_eff', 'I_t'));


I am working on the file "Citation_Network_SDP.ipynb".
The first task is:
Finally, we generate a level-3 relaxation, and solve it as a feasibility problem.
Can you write Python code for it?

sdp = nc.SdpRelaxation([alpha0, alphap, alpham, beta0, betap, betam])
sdp.get_relaxation(3, inequalities=inequalities, momentequalities=moments)
sdp.solve(solver="mosek")


I am working on the file "Citation_Network_SDP.ipynb".
The first task is:
If we study the result, for the example set that comes with metaknowledge, the status will be optimal. In this case, we cannot say anything about static latent homophily. Executing the notebook with the actual database used in the paper gives an infeasible status, which allows us to reject the hypothesis.
Can you write Python code for it?


I am working on the file "Citation_Network_SDP.ipynb".
The first task is:
Next we split the corpus in three time periods and record the coauthor relationships in the first time period.
Can you write Python code for it?

years = []
for R in RC:
    if isinstance(R.year, int):
        years.append(R.year)
histogram = np.histogram(np.array(years), bins=3)
coauthors0 = create_directed_coauthor_graph(RC.yearSplit(histogram[1][0],
                                                         histogram[1][1]))

authors_period0 = get_authors_in_period(RC, histogram[1][0], histogram[1][1])
authors_period1 = get_authors_in_period(RC, histogram[1][1], histogram[1][2])
authors_period2 = get_authors_in_period(RC, histogram[1][2], histogram[1][3])
epochs = [authors_period0, authors_period1, authors_period2]


I am working on the file "Citation_Network_SDP.ipynb".
The first task is:
The next couple of functions work on the citation network. The first function attempts to mitigate the arrogance of Thomson-Reuters, who cannot be bothered to normalize author names. The second function picks all authors who made a reference in a time period. The third one defines a directed graph over the co-author network. The last one gets all references
Can you write Python code for it?

def normalize_name(name):
    w = re.sub("([a-z]{2,}) ", "\g<1>X", re.sub("[\.,]", " ", name.lower()))
    return w.replace(" ", "").replace("X", " ")

def get_authors_in_period(RC, year1, year2):
    RC_period = RC.yearSplit(year1, year2)
    authors_period = {}
    for R in RC_period:
        if R.citations is not None:
            reference_IDs = [reference.ID() for reference in R.citations]
            for author in R.authors:
                author = normalize_name(author)
                if author in authors_period:
                    authors_period[author] += reference_IDs
                else:
                    authors_period[author] = reference_IDs
    return authors_period

def create_directed_coauthor_graph(RC):
    coauthors_undirected = RC.coAuthNetwork()
    coauthors = coauthors_undirected.to_directed()
    weights = coauthors_undirected.degree()
    for edge in coauthors_undirected.edges_iter():
        if weights[edge[0]] > weights[edge[1]]:
            coauthors.remove_edge(edge[1], edge[0])
        else:
            coauthors.remove_edge(edge[0], edge[1])
    return coauthors

def get_all_references(author, epochs):
    references = set()
    for epoch in epochs:
        if author in epoch:
            for reference in epoch[author]:
                references.add(reference)
    return references


I am working on the file "Citation_Network_SDP.ipynb".
The first task is:
We also have to make sure that the constraints Eq. (3) are satisfied to ensure that we have a valid probability distribution"
$x\in\mathbb{R}^6: g_i(x)=x_i(1-x_i)\geq 0, i=1,\ldots,6.$
These constraints are not on the moments but on the actual variables, so matching localizing matrices will be generated in the SDP relaxation.
Can you write Python code for it?

inequalities = [alpha0, alphap, alpham, beta0, betap, betam,
                1-alpha0, 1-alphap, 1-alpham, 1-beta0, 1-betap, 1-betam]


I am working on the file "Citation_Network_SDP.ipynb".
The first task is:
We populate the probability class with length-3 chains and add the statistics from the corpus across the three time periods:
Can you write Python code for it?

T = len(epochs)
p = Probability(T)
M = 0
for pair in coauthors0.edges():
    coauthor1 = normalize_name(pair[0])
    coauthor2 = normalize_name(pair[1])
    references = get_all_references(coauthor1, epochs)
    references.union(get_all_references(coauthor1, epochs))
    for reference in references:
        chain1, chain2 = [], []
        for epoch in epochs:
            if coauthor1 in epoch:
                if reference in epoch[coauthor1]:
                    chain1.append(+1)
                else:
                    chain1.append(-1)
            else:
                chain1.append(-1)
            if coauthor2 in epoch:
                if reference in epoch[coauthor2]:
                    chain2.append(+1)
                else:
                    chain2.append(-1)
            else:
                chain2.append(-1)
        p[(chain1, chain2)] += 1
        M += 1
p.normalize(M)


I am working on the file "Citation_Network_SDP.ipynb".
The first task is:
We set up the symbolic variables of the SDP relaxation, and set the moment constraints as outlined in Eq. (2), that is,
$y_j = \sum_{R_A, R_B} P(R_A, R_B|E)f_j(x),$
where
$f_j(x) = \sum_{A,B}P(A_{1:T}|R_A)P(B_{1:T}|R_B)O_j(A,B).$
Can you write Python code for it?

A1 = 1
alpha0 = nc.generate_variables(name='alpha_0')[0]
alphap = nc.generate_variables(name='alpha_p')[0]
alpham = nc.generate_variables(name='alpha_m')[0]
beta0 = nc.generate_variables(name='beta_0')[0]
betap = nc.generate_variables(name='beta_p')[0]
betam = nc.generate_variables(name='beta_m')[0]
chains = generate_chains(T)
moments = []
for chain1 in chains:
    for chain2 in chains:
        if p[(chain1, chain2)] > 0:
            P_A = alphap**Fp(chain1, T) * alpham**Fn(chain1, T) * (1-alpham)**Sn(chain1, T) * (1-alphap)**Sp(chain1, T) * \
                    alpha0**((1+A1)//2) * (1-alpha0)**((1-A1)//2)
            P_B = betap**Fp(chain2, T) * betam**Fn(chain2, T) * (1-betam)**Sn(chain2, T) * (1-betap)**Sp(chain2, T) * \
                    beta0**((1+A1)//2) * (1-beta0)**((1-A1)//2)
            moments.append(P_A*P_B - p[(chain1, chain2)])


I am working on the file "CitiBike_01_Read_Explore.ipynb".
The first task is:
I am going to do a bunch of datetime converting and add columns with more specific and useful info. These will be:
* Start Hour
* Start Minute
* Start Year
* Stop Hour
* Stop Minute
* Weekday
* User Age
Can you write Python code for it?

# Convert to datetime timestamps
cb_trip_df['starttime']=pd.to_datetime(cb_trip_df['starttime'],infer_datetime_format=True)
cb_trip_df['stoptime']=pd.to_datetime(cb_trip_df['stoptime'],infer_datetime_format=True)
#cb_trip_df['birth year']=pd.to_datetime(cb_trip_df['birth year'],infer_datetime_format=True)

cb_trip_df['startyear']=cb_trip_df['starttime'].apply(lambda x: x.year)
cb_trip_df['starthour']=cb_trip_df['starttime'].apply(lambda x: x.hour)
cb_trip_df['startminute']=cb_trip_df['starttime'].apply(lambda x: x.minute)
cb_trip_df['startday']=cb_trip_df['starttime'].apply(lambda x: x.day)
cb_trip_df['stophour']=cb_trip_df['stoptime'].apply(lambda x: x.hour)
cb_trip_df['stopminute']=cb_trip_df['stoptime'].apply(lambda x: x.minute)
cb_trip_df['stopday']=cb_trip_df['stoptime'].apply(lambda x: x.day)
cb_trip_df['weekday']=cb_trip_df['starttime'].apply(lambda x: x.year)
cb_trip_df['userage']=cb_trip_df['startyear']-cb_trip_df['birth year']
cb_trip_df.info()

cb_trip_df.describe()

print "Number of bikes: ",(len(cb_trip_df['bikeid'].unique()))
print "Number of stations: ",(len(cb_trip_df['start station id'].unique()))
print(cb_trip_df['bikeid'].iloc[0])

print(len(cb_trip_df['start station id'].unique()))

cb_trip_df['bikeid'].value_counts()

bid_hist = cb_trip_df[cb_trip_df['bikeid']==21099]
bid_hist.head()


I am working on the file "Clase 01 - Introducción y estructuras selectivas.ipynb".
The first task is:
Además, se pueden armar strings multilínea poniendo tres comillas simples o dobles seguidas:
Can you write Python code for it?

cadena_caracteres = """y si quiero
usar un string
que se escriba en varias
líneas?."""
print cadena_caracteres
print type(cadena_caracteres)


I am working on the file "Clase 01 - Introducción y estructuras selectivas.ipynb".
The first task is:
Ahora, ¿qué pasa cuando ese número entero crece mucho?, por ejemplo, si le asignamos 9223372036854775807
Python
numero_muy_grande = -9223372036854775809
print numero_muy_grande
print type(numero_entero)
print 2**16/2
Can you write Python code for it?


I am working on the file "Clase 01 - Introducción y estructuras selectivas.ipynb".
The first task is:
Cualquier tipo de dato se lo puede evaluar como booleano.
Se toma como falso a:
* None
* False para los bool
* cero para todo tipo de dato numérico: 0, 0L, 0.0, 0j
* vacío para cualquier secuencia o diccionario: '', (), [], {}
Por lo tanto, se puede saber si una lista esta vacía o no con simplemente:
Can you write Python code for it?

if []:
    print 'La lista no esta vacía'

if False or None or [] or () or {} or 0 or '':
    print 'Alguna de las anteriores no era falsa'
else:
    print 'Todos los valores anteriores son consideradas como Falso'

x = 'Este mensaje se va a mostrar porque será evaulado como verdadero'
if x:
    print x
else:
    print 'Esta vacio'