I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/APD-Crime-Data_orig.ipynb".
The first task is:
Fitting the regression line
Suppose there are $n$ data points {{math|{(''xi'', ''yi''), ''i'' {{=}} 1, ..., ''n''}.}} The function that describes x and y is:
$$y_i = \alpha + \beta x_i + \varepsilon_i.$$
The goal is to find the equation of the straight line
$$y = \alpha + \beta x,$$
which would provide a "best" fit for the data points. Here the "best" will be understood as in the [[Ordinary least squares|least-squares]] approach: a line that minimizes the sum of squared residuals of the linear regression model. In other words, {{mvar|α}} (the {{mvar|y}}-intercept) and {{mvar|β}} (the slope) solve the following minimization problem:
$$\text{Find }\min_{\alpha,\,\beta} Q(\alpha, \beta), \qquad \text{for } Q(\alpha, \beta) = \sum_{i=1}^n\varepsilon_i^{\,2} = \sum_{i=1}^n (y_i - \alpha - \beta x_i)^2\ $$
By using either [[calculus]], the geometry of [[inner product space]]s, or simply expanding to get a quadratic expression in {{mvar|α}} and {{mvar|β}}, it can be shown that the values of {{mvar|α}} and {{mvar|β}} that minimize the objective function {{mvar|Q}}<ref>Kenney, J. F. and Keeping, E. S. (1962) "Linear Regression and Correlation." Ch. 15 in ''Mathematics of Statistics'', Pt. 1, 3rd ed. Princeton, NJ: Van Nostrand, pp. 252–285</ref> are
: <math>\begin{align}
  \hat\beta &= \frac{ \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y}) }{ \sum_{i=1}^n (x_i - \bar{x})^2 } \[6pt]
            &= \frac{ \sum_{i=1}^{n} (x_i y_i - x_i \bar{y} - \bar{x} y_i + \bar{x} \bar{y})} { \sum_{i=1}^n (x_i^2 - 2 x_i \bar{x} + \bar{x}^2) } \[6pt]
            &= \frac{ \sum_{i=1}^{n} (x_i y_i) - \bar{y} \sum_{i=1}^{n}  x_i  - \bar{x} \sum_{i=1}^{n} y_i + n  \bar{x} \bar{y}} { \sum_{i=1}^n (x_i^2) - 2 \bar{x}  \sum_{i=1}^n  x_i  + n \bar{x}^2 } \[6pt]
            &= \frac{ \frac{1}{n} \sum_{i=1}^{n} x_i y_i - \bar{x} \bar{y} }{ \frac{1}{n}\sum_{i=1}^n {x_i^2} -   \overline{x}^2 } \[6pt]
            &= \frac{ \overline{xy} - \bar{x}\bar{y} }{ \overline{x^2} - \bar{x}^2 } = \frac{ \operatorname{Cov}[x, y] }{...
Can you write Python code for it?

### in case we want to save a DataFrame
#writer = pd.ExcelWriter('myresults.xlsx')
#df.to_excel(writer,'Results')
#writer.save()

resdf


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/APD-Crime-Data_orig.ipynb".
The first task is:
Let's look at residential burglary.
Can you write Python code for it?

c = 3
crime_types[c]

crime_year_month = dff[dff.occur_year.between(2009, 2015)].groupby(by=['UC2 Literal', 'occur_year', 'occur_month']).offense_id.count()

c = 3 ## 'BURGLARY-RESIDENCE'
resburglaries = crime_year_month.loc[crime_types[c]]
fig = plt.figure(figsize=(20,10))
for y in years:
    plt.plot(resburglaries.loc[y].index, resburglaries.loc[y], label=("%4.0f"%y))
plt.legend()
plt.title("Seasonal Trends - %s"%crime_types[c], fontsize=20)
plt.xticks(range(13), ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.xlim(0,13)
None


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/APD-Crime-Data_orig.ipynb".
The first task is:
Load data (don't change this if you're running the notebook on the cluster)
We have two files
- /home/data/APD/COBRA083016_2015.xlsx for 2015
- /home/data/APD/COBRA083016.xlsx from 2009 to current date
Can you write Python code for it?

%%time
df = pd.read_excel('/home/data/APD/COBRA083016_2015.xlsx', sheetname='Query')

df.shape

for c in df.columns:
    print(c)

df[0:5]

df.describe()

df.offense_id.min(), df.offense_id.max()

df.groupby(['UC2 Literal']).offense_id.count()


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/APD-Crime-Data_orig.ipynb".
The first task is:
Normalized over the annual average
Can you write Python code for it?

c = 3 ## 'BURGLARY-RESIDENCE'
fig = plt.figure(figsize=(20,10))
for y in years:
    avg = resburglaries.loc[y].mean()
    plt.hlines(avg, 1, 13, linestyle='dashed')
    plt.plot(resburglaries.loc[y].index, resburglaries.loc[y], label=("%4.0f"%y))
plt.legend()
plt.title("Seasonal Trends - %s (with annuale averages)"%crime_types[c], fontsize=20)
plt.xticks(list(range(1,13)), ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.xlim(0,13)
None

c = 3 ## 'BURGLARY-RESIDENCE'
fig = plt.figure(figsize=(20,10))
for y in years:
    avg = resburglaries.loc[y].mean()
    std = resburglaries.loc[y].std()
    ##plt.hlines(avg, 1, 13, linestyle='dashed')
    plt.plot(resburglaries.loc[y].index, (resburglaries.loc[y]-avg)/std, label=("%4.0f"%y))
plt.legend()
plt.title("Seasonal Trends - %s (normalized)"%crime_types[c], fontsize=20)
plt.xticks(list(range(1,13)), ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.xlim(0,13)
plt.ylabel("Standard deviations $\sigma_y$")
None

seasonal_adjust = resburglaries.reset_index().groupby(by=['occur_month']).offense_id.agg('mean')


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/APD-Crime-Data_orig.ipynb".
The first task is:
Part 1 - Observations from the data
Part 2 - Seasonal Model
Can you write Python code for it?

## load complete dataset
dff = pd.read_excel('/home/data/APD/COBRA083016.xlsx', sheetname='Query')

dff.shape

for evt in ['occur', 'poss']:
    dff['%s_ts'%evt] = pd.to_datetime(dff['%s_date'%evt]+' '+dff['%s_time'%evt])
dff['rpt_ts'] = pd.to_datetime(dff.rpt_date)

', '.join(dff.columns)

dff['occur_year'] = dff.occur_ts.dt.year
dff['occur_month'] = dff.occur_ts.dt.month
dff['occur_dayweek'] = dff.occur_ts.dt.dayofweek


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/LinearRegression_orig.ipynb".
The first task is:
Evaluation: Root-mean-square Deviation
The root-mean-square deviation (RMSD) or root-mean-square error (RMSE) is a frequently used measure of the differences between values (sample and population values) predicted by a model or an estimator and the values actually observed. The RMSD represents the sample standard deviation of the differences between predicted values and observed values. These individual differences are called residuals when the calculations are performed over the data sample that was used for estimation, and are called prediction errors when computed out-of-sample. The RMSD serves to aggregate the magnitudes of the errors in predictions for various times into a single measure of predictive power. RMSD is a good measure of accuracy, but only to compare forecasting errors of different models for a particular variable and not between variables, as it is scale-dependent.$^1$

$^1$  Hyndman, Rob J. Koehler, Anne B.; Koehler (2006). "Another look at measures of forecast accuracy". International Journal of Forecasting. 22 (4): 679–688. doi:10.1016/j.ijforecast.2006.03.001.
Can you write Python code for it?

import math
RSMD = math.sqrt(np.square(yhat_m-y_m).sum()/n)
print RSMD


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/LinearRegression_orig.ipynb".
The first task is:
In most cases we may rely on packages like scikit-learn to model data https://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-ridge-lasso-regression-python/
Can you write Python code for it?

#Define input array with angles from 60deg to 300deg converted to radians
Xr = np.array([i*np.pi/180 for i in range(60,300,4)])
np.random.seed(10)  #Setting seed for reproducability
y = np.sin(Xr) + np.random.normal(0,0.3,len(Xr))

y_test = np.sin(Xr) + np.random.normal(0,0.3,len(Xr))
plt.plot(Xr, y, 'o', alpha=0.5)
plt.plot(Xr, y_test, 'v', alpha=0.5)

X = np.vstack((np.ones(len(Xr)), Xr)).T

beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)

yhat = X.dot(beta)
yhat.shape

p = X.shape[1]  ## get number of parameters
lam = 10.0

beta2 = np.linalg.inv(X.T.dot(X) + lam*np.eye(p)).dot(X.T).dot(y)
yhat2 = X.dot(beta2)

n = float(X.shape[0])
print "      RMSE = ", math.sqrt(np.square(yhat-y).sum()/n)
print "Ridge RMSE = ", math.sqrt(np.square(yhat2-y).sum()/n)
plt.plot(X[:,1], y, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=1, color="red")
plt.plot(X[:,1], yhat2, "-", alpha=1, color="green")

n = float(X.shape[0])
print "      RMSE = ", math.sqrt(np.square(yhat-y_test).sum()/n)
print "Ridge RMSE = ", math.sqrt(np.square(yhat2-y_test).sum()/n)
plt.plot(X[:,1], y_test, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=1, color="red")
plt.plot(X[:,1], yhat2, "-", alpha=1, color="green")


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/LinearRegression_orig.ipynb".
The first task is:
Let's add the bias, i.e. a column of $1$s to the explanatory variables
Can you write Python code for it?

X = np.vstack((np.ones(n), Xr)).T
print X.shape
X[0:10,:]


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/LinearRegression_orig.ipynb".
The first task is:
Multiple Linear Regression
Can you write Python code for it?

n = 100  # numeber of samples
X1 = np.random.rand(n)*99.0
X2 = np.random.rand(n)*51.0 - 26.8
X3 = np.random.rand(n)*5.0 + 6.1
X4 = np.random.rand(n)*1.0 - 0.5
X5 = np.random.rand(n)*300.0

y_m = -7.3 + 2.5*X1 + -7.9*X2 + 1.5*X3 + 10.0*X4 + 0.13*X5 + np.random.randn(n)*7.0
plt.hist(y_m, bins=20)
;

X_m = np.vstack((np.ones(n), X1, X2, X3, X4, X5)).T
X_m.shape

### β = inv(X'*X)*X'*y
beta_m = np.linalg.inv(X_m.T.dot(X_m)).dot(X_m.T).dot(y_m)
beta_m

yhat_m = X.dot(beta_m)
yhat_m.shape


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/LinearRegression_orig.ipynb".
The first task is:
Regularization, Ridge-Regression
Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, is a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting.
In general, a regularization term $R(f)$ is introduced to a general loss function:

for a loss function  $V$ that describes the cost of predicting $f(x)$ when the label is
$y$, such as the square loss or hinge loss, and for the term
$\lambda$  which controls the importance of the regularization term.
$R(f)$ is typically a penalty on the complexity of
$f$, such as restrictions for smoothness or bounds on the vector space norm.$^1$
A theoretical justification for regularization is that it attempts to impose Occam's razor on the solution, as depicted in the figure. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters.
Regularization can be used to learn simpler models, induce models to be sparse, introduce group structure into the learning problem, and more.
We're going to add the L2 term $\lambda||\beta||_2^2$ to the regression equation, which yields to$^2$
$$ \beta = (X^\prime X + \lambda I)^{-1} X^\prime y $$

$^1$ Bishop, Christopher M. (2007). Pattern recognition and machine learning (Corr. printing. ed.). New York: Springer. ISBN 978-0387310732.
$^2$ http://stats.stackexchange.com/questions/69205/how-to-derive-the-ridge-regression-solution
Can you write Python code for it?

p = X.shape[1]  ## get number of parameters
lam = 10.0
p, lam

### β₂ = inv(X'*X + λ*eye(p))*X'*y
beta2 = np.linalg.inv(X.T.dot(X) + lam*np.eye(p)).dot(X.T).dot(y)

yhat2 = X.dot(beta2)

RSMD2 = math.sqrt(np.square(yhat2-y).sum()/n)
print RSMD2

##n = float(X.shape[0])
print "      RMSE = ", math.sqrt(np.square(yhat-y).sum()/n)
print "Ridge RMSE = ", math.sqrt(np.square(yhat2-y).sum()/n)
plt.plot(X[:,1], y, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=0.7, color="red")
plt.plot(X[:,1], yhat2, "-", alpha=0.7, color="green")

n = 100
Xr_test = np.random.rand(n)*99.0
X_test = np.vstack((np.ones(n), Xr_test)).T
y_test = -7.3 + 2.5*Xr_test + np.random.randn(n)*27.0
yhat_test = X_test.dot(beta)
yhat_test2 = X_test.dot(beta2)

print "Test       RMSE = ", math.sqrt(np.square(yhat_test-y_test).sum()/n)
print "Test Ridge RMSE = ", math.sqrt(np.square(yhat_test2-y_test).sum()/n)
plt.plot(X_test[:,1], y_test, "o", alpha=0.5)
plt.plot(X_test[:,1], yhat_test, "-", alpha=1, color="red")
plt.plot(X_test[:,1], yhat_test2, "-", alpha=1, color="green")


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/LinearRegression_orig.ipynb".
The first task is:
Run regressions with varying λ and σ
Let's compute the (L2 regularized) linear regression for varying levels of noise, and regulization parameters
Can you write Python code for it?

rmse(ŷ, y) = √mean((ŷ-y).^2)

n = 100  # numeber of samples

X₁ = rand(n)*99.0
X₂ = rand(n)*51.0 - 26.8
X₃ = rand(n)*5.0 + 6.1
X₄ = rand(n)*1.0 - 0.5
X₅ = rand(n)*300.0
X̃ = rand(n)*1.0 - 0.5

X = [ones(n) X₁ X₂ X₃ X₄ X₅]

p = size(X,2)

for σ in [50, 100, 200]
    y = -7.3 + 2.5X₁ + -7.9X₂ + 1.5X₃ + 10.0X₄ + 0.13X₅ + 1.0X̃.^3 + rand(Normal(0.0, σ), n)

    for λ in [0.0, 2.0, 10.0, 100.0]
        β = inv(X'*X + λ*eye(p))*X'*y
        ŷ = X*β
        R = rmse(ŷ, y)
        @printf("σ=%8.3lf\tλ=%8.3lf\tRMSD=%8.3lf\n", σ, λ, R)
    end
end

n = 100
σ = 200
λ = 20

X₁ = rand(Normal(0.0, 50), n)  ##linspace(-100, 100.0, n)
X = [ones(n) X₁]

p = size(X,2)

y₁ = -7.3 * 2.5X₁ + rand(Normal(0.0, σ), n)
LL = [layer(x=X₁, y=y₁, Geom.point)]

for λ in [0, 20, 100]

    β = inv(X'*X + λ*eye(p))*X'*y₁
    ŷ₁ = X*β
    R = rmse(ŷ₁, y₁)
    @printf("λ=%8.3lf\tRMSD=%8.3lf\n", λ, R)
    push!(LL, layer(x=X₁, y=ŷ₁, Geom.line))
end

plot(LL...)


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/linear-algebra-python-basics.ipynb".
The first task is:
For loops
Create a $12 \times 2$ matrix and print it out:
Can you write Python code for it?

A = np.arange(24).reshape((12,2))
print A
print A.shape


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/linear-algebra-python-basics.ipynb".
The first task is:
Matrices
Can you write Python code for it?

b = zip(z,c_vector)
print b
print "Note that the length of our zipped list is 2 not (2 by 2):",len(b)

print "But we can convert the list to a matrix like this:"
A = np.array(b)
print A
print type(A)
print "A has shape:",A.shape


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/linear-algebra-python-basics.ipynb".
The first task is:
Multiplying two matricies
Now, consider the $2 \times 1$ vector $C=\bigl( \begin{smallmatrix} c_{11} \
  c_{21}
\end{smallmatrix} \bigr)$
Consider multiplying matrix $A_{2 \times 2}$ and the vector $C_{2 \times 1}$.  Unlike the addition and subtraction case, this product is defined.  Here, conformability depends not on the row and column dimensions, but rather on the column dimensions of the first operand and the row dimensions of the second operand.  We can write this operation as follows
$$
\begin{equation}
    A_{2 \times 2} \times C_{2 \times 1} =
    \begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix}{2 \times 2}
    \times
    \begin{bmatrix}
    c{11} \
    c_{21}
    \end{bmatrix}{2 \times 1}
    =
    \begin{bmatrix}
      a{11}c_{11} + a_{12}c_{21} \
      a_{21}c_{11} + a_{22}c_{21}
    \end{bmatrix}_{2 \times 1}
\end{equation}
$$
Alternatively, consider a matrix C of dimension $2 \times 3$ and a matrix A of dimension $3 \times 2$
$$
\begin{equation}
    A_{3 \times 2}=\begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22} \
      a_{31} & a_{32}
    \end{bmatrix}{3 \times 2}
    ,
    C{2 \times 3} =
    \begin{bmatrix}
          c_{11} & c_{12} & c_{13} \
          c_{21} & c_{22} & c_{23} \
    \end{bmatrix}_{2 \times 3}
    \end{equation}
$$
Here, A $\times$ C is
$$
\begin{align}
    A_{3 \times 2} \times C_{2 \times 3}=&
    \begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22} \
      a_{31} & a_{32}
    \end{bmatrix}{3 \times 2}
    \times
    \begin{bmatrix}
      c{11} & c_{12} & c_{13} \
      c_{21} & c_{22} & c_{23}
    \end{bmatrix}{2 \times 3} \
    =&
    \begin{bmatrix}
      a{11} c_{11}+a_{12} c_{21} & a_{11} c_{12}+a_{12} c_{22} & a_{11} c_{13}+a_{12} c_{23} \
      a_{21} c_{11}+a_{22} c_{21} & a_{21} c_{12}+a_{22} c_{22} & a_{21} c_{13}+a_{22} c_{23} \
      a_{31} c_{11}+a_{32} c_{21} & a_{31} c_{12}+a_{32} c_{22} & a_{31} c_{13}+a_{32} c_{23}
    \end{bmatrix}_{3 \times 3}
\end{align}
$$
So in general, $X_{r_x \times c_x}...
Can you write Python code for it?

# Let's redefine A and C to demonstrate matrix multiplication:
A = np.arange(6).reshape((3,2))
C = np.random.randn(2,2)

print A.shape
print C.shape


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/linear-algebra-python-basics.ipynb".
The first task is:
One important property of transposing a matrix is the transpose of a product of two matrices.  Let matrix A be of dimension $N \times M$ and let B of of dimension $M \times P$.  Then
$$
\begin{equation}
    (AB)^{\prime}=B^{\prime}A^{\prime}
\end{equation}
$$
For more information, see this http://en.wikipedia.org/wiki/Matrix_transposition on matrix transposition.  This is also easy to implement:
Can you write Python code for it?

print B.T.dot(A.T)
print "Is identical to:"
print (A.dot(B)).T


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/linear-algebra-python-basics.ipynb".
The first task is:
The code below loops over the rows (12 of them) of our matrix A.  For each row, it slices A and prints the row values across all columns.  Notice the form of the for loop.  The colon defines the statement we are looping over.  For each iteration of the loop idented lines will be executed:
Can you write Python code for it?

for rows in A:
    print rows

for rows in A:
    print rows

for cols in A.T:
    print cols


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/linear-algebra-python-basics.ipynb".
The first task is:
This code sets up Ipython Notebook environments (lines beginning with %), and loads several libraries and functions.  The core scientific stack in python consists of a number of free libraries.  The ones I have loaded above include:

sympy: provides for symbolic computation (solving algebra problems)
numpy: provides for linear algebra computations
matplotlib.pyplot: provides for the ability to graph functions and draw figures
scipy: scientific python provides a plethora of capabilities
seaborn: makes matplotlib figures even pretties (another library like this is called bokeh).  This is entirely optional and is purely for eye candy.

Creating arrays, scalars, and matrices in Python
Scalars can be created easily like this:
Can you write Python code for it?


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/linear-algebra-python-basics.ipynb".
The first task is:
Transposing a Matrix
At times it is useful to pivot a matrix for conformability- that is in order to matrix divide or multiply, we need to switch the rows and column dimensions of matrices.  Consider the matrix
$$
\begin{equation}
    A_{3 \times 2}=\begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22} \
      a_{31} & a_{32}
    \end{bmatrix}{3 \times 2}
\end{equation}
$$
The transpose of A (denoted as $A^{\prime}$) is
$$
\begin{equation}
   A^{\prime}=\begin{bmatrix}
      a{11} & a_{21} & a_{31} \
      a_{12} & a_{22} & a_{32} \
    \end{bmatrix}_{2 \times 3}
\end{equation}
$$
Can you write Python code for it?

A = np.arange(6).reshape((3,2))
B = np.arange(8).reshape((2,4))
print "A is"
print A

print "The Transpose of A is"
print A.T


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/linear-algebra-python-basics.ipynb".
The first task is:
Vectors and Lists
The numpy library (we will reference it by np) is the workhorse library for linear algebra in python.  To creat a vector simply surround a python list ($[1,2,3]$) with the np.array function:
Can you write Python code for it?

x_vector = np.array([1,2,3])
print x_vector


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/linear-algebra-python-basics.ipynb".
The first task is:
While loops
Again, we have the same basic form for the statement (note the colons and indents).  Here we use the shorthand notation x+=1 for performing the calculation x = x + 1:
Can you write Python code for it?

x=0
while x<10:
    x+=1
    print x<10

print x


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/linear-algebra-python-basics.ipynb".
The first task is:
or, we could grab a particular element (in this case, the second column, last row):
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
1b. Add to the above, the most recent birthdate for all employees
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
1c. Show the above results broken down by gender
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
2a. Show me a list of employees that have a lastname that begins with "R".
Can you write Python code for it?

filter = Employees.LastName.map(lambda x: x[0] == 'R')

Employees[filter]

df = Employees[Employees.LastName.map(lambda x: x[0] == 'R')]
df[['EmployeeID', 'FirstName', 'LastName']]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
2a. What are the average vacation hours for all employees?
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
2b. Add to the above, the minimum vacation hours for all employees
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
2b. Show me a list of employees that have a lastname that ends with "r"
Can you write Python code for it?

Employees.loc[Employees.LastName.map(lambda x: x[-1] == 'r'), ['EmployeeID', 'FirstName', 'LastName']]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
2d. Show the above results broken down by job title, and married/single employees
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
2e. Add to the above, the maximum vacation hours per group
Can you write Python code for it?

Employees.groupby(by=['JobTitle', 'MaritalStatus']).VacationHours.aggregate([np.mean, np.min, np.max])


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
2g. Show the above results but only for American employees
Can you write Python code for it?

Employees[Employees.CountryName=='United States'] \
.groupby(by=['JobTitle', 'MaritalStatus', 'StateProvinceName']).VacationHours.aggregate([np.mean, np.min, np.max])


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
2j. Limit the results above to States where the average vacation hours is greater than 30 and the maximum vacation hours is less than 50
Can you write Python code for it?

Employees[Employees.CountryName=='United States'] \
.groupby(by=['StateProvinceName']).filter(lambda x: (x.VacationHours.mean() >30) and (x.VacationHours.max()<50)) \
.groupby(by=['StateProvinceName']).VacationHours.aggregate([np.mean, np.min, np.max])


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
3a. Provide a list of employees that are on salary and have more than 35 vacation hours left.
Can you write Python code for it?

Employees.loc[(Employees.SalariedFlag==1) & (Employees.VacationHours>35), ['EmployeeID', 'FirstName', 'LastName', 'SalariedFlag', 'VacationHours']]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
3c. Remove any manager ID's that are NULL from the results above
Can you write Python code for it?

Employees.loc[pd.isnull(Employees.EmployeeID)==False] \
.groupby(by=['ManagerID']).EmployeeID.aggregate(lambda x: x.shape[0])


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
3d. Show the same results as above, but only for managers who have at least 5 employees
Can you write Python code for it?

Employees.loc[pd.isnull(Employees.EmployeeID)==False] \
.groupby(by=['ManagerID']).filter(lambda x: x.shape[0]>=5) \
.groupby(by=['ManagerID']).EmployeeID.aggregate(lambda x: x.shape[0])


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
4a. List the married employees with more than 35 vacation hours, only ones living in Washington state.
Can you write Python code for it?

Employees.loc[
    (Employees.SalariedFlag==1) & (Employees.VacationHours>35) & (Employees.StateProvinceName == 'Washington')
    , ['EmployeeID', 'FirstName', 'LastName', 'SalariedFlag', 'VacationHours', 'StateProvinceName']
]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
4b. Change the logic to include anyone who meets any of the 3 conditions (i.e., people who are either married, live in Washington state, or have more than 35 vacation hours left)
Can you write Python code for it?

Employees.loc[
    (Employees.SalariedFlag==1) | (Employees.VacationHours>35) | (Employees.StateProvinceName == 'Washington')
    , ['EmployeeID', 'FirstName', 'LastName', 'SalariedFlag', 'VacationHours', 'StateProvinceName']
]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
4c. Show the same as above, but only for Production Technicians
Can you write Python code for it?

'abc'.startswith('ab')

Employees.loc[ Employees.JobTitle.map(lambda x: x.startswith('Production Technician'))
    & (
        (Employees.SalariedFlag==1) | (Employees.VacationHours>35) | (Employees.StateProvinceName == 'Washington')
    ), ['EmployeeID', 'FirstName', 'LastName', 'SalariedFlag', 'VacationHours', 'StateProvinceName', 'JobTitle']
]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
4d. Add something that shows the number of employees per city
Can you write Python code for it?

aggregations = {
    'VacationHours' : {
        'AverageVacationHours' : 'mean'
    },
    'EmployeeID' : {
        'NumEmployees' : 'count'
    }
}
Employees.groupby(by=['StateProvinceName', 'City'])\
.aggregate(aggregations)


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
4e. Sort the results by the city and state
Can you write Python code for it?

### Since Pandas already sorts the results, we show here how show resutls in DESCENDING order
aggregations = {
    'VacationHours' : {
        'AverageVacationHours' : 'mean'
    },
    'EmployeeID' : {
        'NumEmployees' : 'count'
    }
}
Employees.groupby(by=['StateProvinceName', 'City']).aggregate(aggregations) \
.sort_index(ascending=False, axis=0)

## This adds the row indices back as columns
df = Employees.groupby(by=['StateProvinceName', 'City']).aggregate(aggregations) \
.sort_index(ascending=False, axis=0)

df['SPN'] = df.index.get_level_values(0)
df['C'] = df.index.get_level_values(1)


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
4f. Make city and State a single column in the format of "City, State"
Can you write Python code for it?

# Version 1: first create new column 'CityState' ... rest is easy
aggregations = {
    'VacationHours' : {
        'AverageVacationHours' : 'mean'
    },
    'EmployeeID' : {
        'NumEmployees' : 'count'
    },
}

df = Employees.copy()
df['CityState'] = Employees.City.str.cat(Employees.StateProvinceName, sep=', ')
df.groupby(by = 'CityState').aggregate(aggregations).sort_index(ascending=False, axis=0)

# Version 2: using the multi-indexed result and adding index values as new column 'CityState'
aggregations = {
    'VacationHours' : {
        'AverageVacationHours' : 'mean'
    },
    'EmployeeID' : {
        'NumEmployees' : 'count'
    },
}

df = Employees.groupby(by=['StateProvinceName', 'City']).aggregate(aggregations) \
.sort_index(ascending=False, axis=0)
df['CityState'] = [
    r.City+', '+r.State \
        for i, r in pd.DataFrame({
            'City': df.index.get_level_values(1),
            'State': df.index.get_level_values(0)
            }).iterrows()
    ]
df


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
4g. Add a column that shows the difference between the maximum vacation hours and minimum vacation hours for each city
Can you write Python code for it?

aggregations = {
    'VacationHours' : {
        'AverageVacationHours' : 'mean',
        'MinVacationHours' : min,
        'MaxVacationHours' : max,
        'DeltaVacationHoues' : lambda x: x.max()-x.min()
    },
    'EmployeeID' : {
        'NumEmployees' : 'count'
    },
}

Employees.groupby(by=['StateProvinceName', 'City']).aggregate(aggregations) \
.sort_index(ascending=False, axis=0)


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
4h. Now sort the results by the new column created above
Can you write Python code for it?

aggregations = {
    'VacationHours' : {
        'AverageVacationHours' : 'mean',
        'MinVacationHours' : min,
        'MaxVacationHours' : max,
        'DeltaVacationHours' : lambda x: x.max()-x.min()
    },
    'EmployeeID' : {
        'NumEmployees' : 'count'
    },
}

df = Employees.groupby(by=['StateProvinceName', 'City']).aggregate(aggregations)
df.iloc[np.argsort(df['VacationHours']['DeltaVacationHours'])]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
4i. Limit the results to cities that have more than 1 employee
Can you write Python code for it?

def my_own_delta(x):
    return  x.max()-x.min()

aggregations = {
    'VacationHours' : {
        'AverageVacationHours' : 'mean',
        'MinVacationHours' : np.min,
        'MaxVacationHours' : np.max,
        'DeltaVacationHours' : my_own_delta, ##lambda x: x.max()-x.min(),
        'cnt' : pd.Series.count
    },
    'EmployeeID' : {
        'NumEmployees' : 'count'
    },
}

df = Employees.groupby(by=['StateProvinceName', 'City']).aggregate(aggregations)
df['VacationHours']['DeltaVacationHours']>1
df2 = df[df['VacationHours']['DeltaVacationHours']>1]

df2.columns = df2.columns.droplevel(0)
df2.reset_index()
##df2.iloc[np.argsort(df2['VacationHours']['DeltaVacationHours'])]
##np.argsort(df2['VacationHours']['DeltaVacationHours'])


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
7a. Provide a list of employees who have at least 60 vacation hours left.
Can you write Python code for it?

Employees.loc[Employees.VacationHours >= 60
              , ['EmployeeID', 'FirstName', 'LastName', 'VacationHours', 'SickLeaveHours']]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
Assign intermediate results to new variables, and perform next operation on them
In particular to saticfy filtering ('HAVING') and sorting on aggregated values
Columns are not lists. Methods on columns are defined in Pandas.Series and NumPy
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
Filtering (with)
1. Provide a list of employees that are married.
Can you write Python code for it?

Employees.MaritalStatus == 'M'

Employees.loc[Employees.MaritalStatus == 'M', ['EmployeeID', 'FirstName', 'LastName', 'MaritalStatus']]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
Since expressions return DataFrames we can chain method calls and [...] operators
The order matters!
Can you write Python code for it?

Employees[0:10][Employees.MaritalStatus=='M'][['EmployeeID', 'FirstName', 'MaritalStatus']]

Employees[Employees.MaritalStatus=='M'][['EmployeeID', 'FirstName', 'MaritalStatus']][0:10]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
Summary
In the simplest form one can specify either rows or columns
Can you write Python code for it?

# First ten rows
Employees[0:10]

# Rows satisfying a particular condition
Employees[Employees.MaritalStatus=='M']

# Select sertain columns
Employees[['EmployeeID', 'FirstName']]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/PandasExercise_solution.ipynb".
The first task is:
Use 'aggregate' for more complex groupings and aggregations
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
A where operation for getting.
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Append
Append rows to a dataframe. See the Appending
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(8, 4), columns=['A','B','C','D'])
df

s = df.iloc[3]
df.append(s, ignore_index=True)


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Apply
Applying functions to the data
Can you write Python code for it?

df

df.apply(np.cumsum)

df

df.max() ##df.apply(max)

df.Ann.max()-df.Ann.min()

df.apply(lambda x: x.max() - x.min())


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
By integer slices, acting similar to numpy/python
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Convert the raw grades to a categorical data type.
Can you write Python code for it?

df["grade"] = df["raw_grade"].astype("category")
df["grade"]

Out[124]:
0    a
1    b
2    b
3    a
4    a
5    e
Name: grade, dtype: category
Categories (3, object): [a, b, e]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Converting between period and timestamp enables some convenient arithmetic functions to be used. In the following example, we convert a quarterly frequency with year ending in November to 9am of the end of the month following the quarter end:
Can you write Python code for it?

prng = pd.period_range('1990Q1', '2000Q4', freq='Q-NOV')
ts = pd.Series(np.random.randn(len(prng)), prng)
ts.index = (prng.asfreq('M', 'e') + 1).asfreq('H', 's') + 9
ts.head()


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Converting between time span representations
Can you write Python code for it?

rng = pd.date_range('1/1/2012', periods=5, freq='M')
ts = pd.Series(np.random.randn(len(rng)), index=rng)
ts

ps = ts.to_period()
ps

ps.to_timestamp()


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Creating a DataFrame by passing a numpy array, with a datetime index and labeled columns:
Can you write Python code for it?

dates = pd.date_range('20130101', periods=6)
dates

df = pd.DataFrame(np.random.randn(6,4), index=dates, columns = ['Ann', "Bob", "Charly", "Don"])
                  ## columns=list('ABCD'))
df

df.Charly+df.Don

df2 = pd.DataFrame({ 'A' : 1.,
   ....:                      'B' : pd.Timestamp('20130102'),
   ....:                      'C' : pd.Series(1,index=list(range(4)),dtype='float32'),
   ....:                      'D' : np.array([3] * 4,dtype='int32'),
   ....:                      'E' : pd.Categorical(["test","train","test","train"]),
   ....:                      'F' : 'foo' })
   ....:

df2


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Excel
Reading and writing to MS Excel
Writing to an excel file
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
For getting a value explicitly
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
For getting fast access to a scalar (equiv to the prior method)
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
For slicing columns explicitly
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Getting Data In/Out
CSV
Writing to a csv file
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Grouping
By “group by” we are referring to a process involving one or more of the following steps
- Splitting the data into groups based on some criteria
- Applying a function to each group independently
- Combining the results into a data structure
See the Grouping section
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Grouping by multiple columns forms a hierarchical index, which we then apply the function.
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
HDF5
Reading and writing to HDFStores
Writing to a HDF5 Store
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Histogramming
See more at Histogramming and Discretization
Can you write Python code for it?

s = pd.Series(np.random.randint(0, 7, size=10))
s

s.value_counts()


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Join
SQL style merges. See the Database style joining
Can you write Python code for it?

left = pd.DataFrame({'key': ['foo', 'boo', 'foo'], 'lval': [1, 2, 3]})
#right = pd.DataFrame({'key': ['boo', 'foo', 'foo'], 'rval': [4, 5, 6]})
right = pd.DataFrame({'key': ['foo', 'foo'], 'rval': [5, 6]})

left

right

pd.merge(left, right, on='key', how='left')


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Merge
Concat
pandas provides various facilities for easily combining together Series, DataFrame, and Panel objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join / merge-type operations.
See the Merging section
Concatenating pandas objects together with concat():
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(10, 4))
df

# break it into pieces
pieces = [df[:3], df[3:7], df[7:]]
pd.concat(pieces)


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
On DataFrame, plot() is a convenience to plot all of the columns with labels:
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,
   .....:                   columns=['A', 'B', 'C', 'D'])
   .....:

df = df.cumsum()

%matplotlib inline
plt.figure(); df.plot(); plt.legend(loc='best')


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Operating with objects that have different dimensionality and need alignment. In addition, pandas automatically broadcasts along the specified dimension.
Can you write Python code for it?

s = pd.Series([1,3,5,np.nan,6,8], index=dates).shift(2)
s

df.sub(s, axis='index')


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Operations
See the Basic section on Binary Ops
Stats
Operations in general exclude missing data.
Performing a descriptive statistic
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Pivot Tables
See the section on Pivot Tables.
In [100]:
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Plotting
Plotting docs.
Can you write Python code for it?

ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
ts = ts.cumsum()

%matplotlib inline
ts.plot()


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Reading from a HDF5 Store
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Reading from an excel file
Can you write Python code for it?

pd.read_excel('foo.xlsx', 'Sheet1', index_col=None, na_values=['NA'])


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Reduction in the dimensions of the returned object
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Rename the categories to more meaningful names (assigning to Series.cat.categories is inplace!)
Can you write Python code for it?

df["grade"].cat.categories = ["very good", "good", "very bad"]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Reorder the categories and simultaneously add the missing categories (methods under Series .cat return a new Series per default).
Can you write Python code for it?

df["grade"] = df["grade"].cat.set_categories(["very bad", "bad", "medium", "good", "very good"])
df["grade"]


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Reshaping
See the sections on Hierarchical Indexing and Reshaping.
Stack
Can you write Python code for it?

tuples = list(zip(*[['bar', 'bar', 'baz', 'baz',
   ....:                      'foo', 'foo', 'qux', 'qux'],
   ....:                     ['one', 'two', 'one', 'two',
   ....:                      'one', 'two', 'one', 'two']]))
   ....:

index = pd.MultiIndex.from_tuples(tuples, names=['first', 'second'])

df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=['A', 'B'])
df2 = df[:4]
df2


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Selecting via [], which slices the rows.
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Selection by Label
See more in Selection by Label
For getting a cross section using a label
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Selection by Position
See more in Selection by Position
Select via the position of the passed integers
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Series is equipped with a set of string processing methods in the str attribute that make it easy to operate on each element of the array, as in the code snippet below. Note that pattern-matching in str generally uses regular expressions by default (and in some cases always uses them). See more at Vectorized String Methods.
Can you write Python code for it?

s = pd.Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])
s.str.lower()


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Setting
Setting a new column automatically aligns the data by the indexes
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Setting by assigning with a numpy array
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Setting values by label
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Showing label slicing, both endpoints are included
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Sorting by values
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Sorting is per order in the categories, not lexical order.
Can you write Python code for it?

df.sort_values(by="grade")

Out[128]:
   id raw_grade      grade
5   6         e   very bad
1   2         b       good
2   3         b       good
0   1         a  very good
3   4         a  very good
4   5         a  very good


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
The result of the prior setting operations
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
The stack() method “compresses” a level in the DataFrame’s columns.
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
Time Series
pandas has simple, powerful, and efficient functionality for performing resampling operations during frequency conversion (e.g., converting secondly data into 5-minutely data). This is extremely common in, but not limited to, financial applications.
See the Time Series section
Can you write Python code for it?

rng = pd.date_range('1/1/2012', periods=100, freq='S')
ts = pd.Series(np.random.randint(0, 500, len(rng)), index=rng)
ts.resample('5Min').sum()

Out[105]:
2012-01-01    25083
Freq: 5T, dtype: int64


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
To drop any rows that have missing data.
Can you write Python code for it?


I am working on the file "DataScienceProgramming/04-Pandas-Data-Tables/Ten-Minute-Tutorial.ipynb".
The first task is:
With a “stacked” DataFrame or Series (having a MultiIndex as the index), the inverse operation of stack() is unstack(), which by default unstacks the last level:
Can you write Python code for it?

stacked.unstack()

stacked.unstack(1)

stacked.unstack(0)


I am working on the file "DataScienceProgramming/08-Machine-Learning-I/HW5_orig.ipynb".
The first task is:
Standard Scaler
http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler
Standardize features by removing the mean and scaling to unit variance
Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set. Mean and standard deviation are then stored to be used on later data using the transform method.
Standardization of a dataset is a common requirement for many machine learning estimators: they might behave badly if the individual feature do not more or less look like standard normally distributed data (e.g. Gaussian with 0 mean and unit variance).
For instance many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order. If a feature has a variance that is orders of magnitude larger that others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.
This scaler can also be applied to sparse CSR or CSC matrices by passing with_mean=False to avoid breaking the sparsity structure of the data.
Can you write Python code for it?

datset

from sklearn.metrics import confusion_matrix

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/DecisionTree.ipynb".
The first task is:
Decision Tree
Let's see how well a decision tree can classify the data. Hereby we need to consider
1. the parameters to the classifier, and
2. the features of the data set that will be used.
We may just explore the impact of the maximum depth of the decision tree. Two of the 16 features ('day' and 'month') may not be useful because they reflect a date, and we're not looking for seasonal effects. So, it's fairly safe to take them out.
Once the dataset is loaded we will convert the categorical data into numeric values.
Finding the right parameters and features for the best performing classifier can be a challenge. The number of possible configurations grows quickly, and knowing how they perform requires training and testing with each of them.
We may also run the training and testing on a configuration multiple times with different random splits of the data set. The performance metrics will be avaraged over the iterations.
We use percision, recall, and the F1 score to evaluate each configuration.
Can you write Python code for it?

### Load Packages
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
import sklearn.tree
import pydot_ng as pdot
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from sklearn.metrics import precision_recall_fscore_support
import itertools


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/DecisionTree.ipynb".
The first task is:
Evaluation
Test how Maximum Depth of tree impacts performance
Can you write Python code for it?

for d in [3, 5, 7, 11, 13]:
    clf = DecisionTreeClassifier(max_depth=d)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.4, random_state=42)
    clf.fit(X_train, y_train)
    ŷ = clf.predict(X_test)
    print('Depth %d' % d)
    print(classification_report(y_test, ŷ))


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/DecisionTree.ipynb".
The first task is:
Reading Data
Can you write Python code for it?

### Read data
DATAFILE = '/home/data/archive.ics.uci.edu/BankMarketing/bank.csv'
df = pd.read_csv(DATAFILE, sep=';')

### use sets and '-' difference operation 'A-B'. Also there is a symmetric different '^'
all_features = set(df.columns)-set(['y'])
num_features = set(df.describe().columns)
cat_features = all_features-num_features
print("All features:         ", ", ".join(all_features), "\nNumerical features:   ", ", ".join(num_features), "\nCategorical features: ", ", ".join(cat_features))

### convert to categorical variables to numeric ones
level_substitution = {}

def levels2index(levels):
    dct = {}
    for i in range(len(levels)):
        dct[levels[i]] = i
    return dct

df_num = df.copy()

for c in cat_features:
    level_substitution[c] = levels2index(df[c].unique())
    df_num[c].replace(level_substitution[c], inplace=True)

## same for target
df_num.y.replace({'no':0, 'yes':1}, inplace=True)
df_num

### create feature matrix and target vector
X = df_num[list(all_features-set(['day', 'month']))].as_matrix()
y = df_num.y.as_matrix()
X, y


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/DecisionTree.ipynb".
The first task is:
That took a while (about 2 minutes). Once computations take that long we should look at a different way to implement them ...  outside the notebook .
Let's see what the best performing configuration with respect to the F1-score of 'yes' is:
Can you write Python code for it?

best = performance_df.F1_score.yes.argmax()
print(performance_df.iloc[best])
print("\nFeatures: ", ', '.join([ '"%s"'%f for f in performance_df.iloc[best].Params.Features.split('|') ], ))


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/DecisionTree.ipynb".
The first task is:
Two methods from sklearn.metrics can be helpful:
1. confusion_matrix produces a confusion matrix
2. precision_recall_fscore_support returns a matrix with values for each of them across all target levels.
Can you write Python code for it?

cm = confusion_matrix(y_test, ŷ)
cm

prf1s = precision_recall_fscore_support(y_test, ŷ)
prf1s

perf = None
for i in range(100):
    if type(perf)!=type(None):
        perf = np.vstack((perf, np.array(prf1s).reshape(1,8)))
    else:
        perf = np.array(prf1s).reshape(1,8)
perf_agg = perf.mean(axis=0)
pd.DataFrame(perf_agg.reshape(1,8), columns=[[b for a in ['Precision', 'Recall', 'F1_score', 'Support'] for b in [a, a]], ['no', 'yes']*4])
##pd.DataFrame([5,5, 'a|b|c'] + list(perf.mean(axis=0)), columns=perf_df.columns)

performance_df = pd.DataFrame(columns=[
        ['Params']*3 + [b for a in ['Precision', 'Recall', 'F1_score', 'Support'] for b in [a, a]],
        ['MaxDepth', 'Nfeature', 'Features'] + ['no', 'yes']*4
    ])
tempdf = pd.concat([
        pd.DataFrame({'a': [1], 'b': [2], 'c': ['Hello']}),
        pd.DataFrame(np.zeros((1,8)))
    ], axis=1, ignore_index=True)

tempdf.columns=performance_df.columns
#performance_df
tempdf

pd.DataFrame(np.zeros(8).reshape(1,8))


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/HW6_orig.ipynb".
The first task is:

Please, rename this file to HW6.ipynb and save it in MSA8010F16/HW6

Homework 6: Preprocessing Data
We use a data set from the UCI Machine Learning Repository
https://archive.ics.uci.edu/ml/datasets/Bank+Marketing
to experiment with a Decision Tree classifier http://www.saedsayad.com/decision_tree.htm
Scikit-Learn: http://scikit-learn.org/stable/modules/tree.html#tree
Book slides:
- http://131.96.197.204/~pmolnar/mlbook/BookSlides_4A_Information-based_Learning.pdf
- http://131.96.197.204/~pmolnar/mlbook/BookSlides_4B_Information-based_Learning.pdf
Bank Marketing Data Set
The data is related with direct marketing campaigns (phone calls) of a Portuguese banking institution. The classification goal is to predict if the client will subscribe a term deposit (variable y).
Data Set Information:
The data is related with direct marketing campaigns of a Portuguese banking institution. The marketing campaigns were based on phone calls. Often, more than one contact to the same client was required, in order to access if the product (bank term deposit) would be ('yes') or not ('no') subscribed.
There are four datasets:
1) bank-additional-full.csv with all examples (41188) and 20 inputs, ordered by date (from May 2008 to November 2010), very close to the data analyzed in [Moro et al., 2014]
2) bank-additional.csv with 10% of the examples (4119), randomly selected from 1), and 20 inputs.
3) bank-full.csv with all examples and 17 inputs, ordered by date (older version of this dataset with less inputs).
4) bank.csv with 10% of the examples and 17 inputs, randomly selected from 3 (older version of this dataset with less inputs).
The smallest datasets are provided to test more computationally demanding machine learning algorithms (e.g., SVM).
The classification goal is to predict if the client will subscribe (yes/no) a term deposit (variable y).
Attribute Information:
Input variables:
- bank client data:
        1...
Can you write Python code for it?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

DATAFILE = '/home/data/archive.ics.uci.edu/BankMarketing/bank.csv'
###DATAFILE = 'data/bank.csv'  ### using locally

df = pd.read_csv(DATAFILE, sep=';')
list(df.columns)


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/HW6_orig.ipynb".
The first task is:
Now, let's look at the categorical variables and their distribution...
Can you write Python code for it?

for f in cat_features:
    tab = df[f].value_counts()
    print('%s:\t%s' % (f, ', '.join([ ("%s(%d)" %(tab.index[i], tab.values[i])) for i in range(len(tab))]) ))


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/HW6_orig.ipynb".
The first task is:
Results in a data frame:
Can you write Python code for it?

mat = pd.DataFrame(
    [ df[f].value_counts() for f in list(cat_features) ],
    index=list(cat_features)
    ).stack()

pd.DataFrame(mat.values, index=mat.index)


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/HW6_orig.ipynb".
The first task is:
Step 1: Investigate Data Set

We have a number of categorical data: What's their cardinality? How are the levels distributed?
What's the distribution on numeric values? Do we see any correlations?

Let's first look at columns (i.e. variables) with continuous values. We can get a sense of the distribution from aggregate functions like mean, standard variation, quantiles, as well as, minimum and maximum values.
The Pandas method describe creates a table view of those metrics. (The methods can also be used to identify numeric features in the data frame.
Can you write Python code for it?

### use sets and '-' difference operation 'A-B'. Also there is a symmetric different '^'
all_features = set(df.columns)-set(['y'])
num_features = set(df.describe().columns)
cat_features = all_features-num_features

print("All features:         ", ", ".join(all_features), "\nNumerical features:   ", ", ".join(num_features), "\nCategorical features: ", ", ".join(cat_features))

set(df.columns)-set(df.describe().columns)-set('y')

### Describe Columns
help(pd.DataFrame.describe)

### Let's get the description of the numeric data for each of the target values separately.
### We need to rename the columns before we can properly join the tables. The column names may look strange...
desc_yes = df[df.y=='yes'].describe().rename_axis(lambda c: "%s|A"%c, axis='columns')
desc_no  = df[df.y=='no'].describe().rename_axis(lambda c: "%s|B"%c, axis='columns')

### ...but this way we can get them in the desired order...
desc = desc_yes.join(desc_no).reindex_axis(sorted(desc_yes.columns), axis=1)
### ...because we're changing them anyway:

#desc.set_axis(1, [sorted(list(num_features)*2), ['yes', 'no']*len(num_features)])
#desc


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/HW6_orig.ipynb".
The first task is:
Step 2: Prepare for ML algorithm
The ML algorithms in Scikit-Learn use Matrices (with numeric values). We need to convert our data-frame into a feature matrix X and a target vector y.
Many algorithms also require the features to be in the same range. Decision-trees don't bother because they don't perform any operations across features.
Use the pd.DataFrame.as_matrix method to convert a DataFrame into a matrix.
Can you write Python code for it?

help(pd.DataFrame.as_matrix)

## We copy our original dataframe into a new one, and then perform replacements on categorical levels.
## We may also keep track of our replacement
level_substitution = {}

def levels2index(levels):
    dct = {}
    for i in range(len(levels)):
        dct[levels[i]] = i
    return dct

df_num = df.copy()

for c in cat_features:
    level_substitution[c] = levels2index(df[c].unique())
    df_num[c].replace(level_substitution[c], inplace=True)

## same for target
df_num.y.replace({'no':0, 'yes':1}, inplace=True)

df_num

level_substitution


I am working on the file "DataScienceProgramming/09-Machine-Learning-II/HW6_orig.ipynb".
The first task is:
Step 3: Training
Now that we have our DataFrame prepared, we can create the feature matrix X and target vector y:
1. split data into training and test sets
2. fit the model
Can you write Python code for it?

X = df_num[list(all_features)].as_matrix()
y = df_num.y.as_matrix()
X, y

### Scikit-learn provides us with a nice function to split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.4, random_state=42)

from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(max_depth=5)

clf.fit(X_train, y_train)
score_train = clf.score(X_train, y_train)
score_test = clf.score(X_test, y_test)
print('Ratio of correctly classified samples for:\n\tTraining-set:\t%f\n\tTest-set:\t%f'%(score_train, score_test))


I am working on the file "DataScienceWorkshop.ipynb".
The first task is:
Hyperparameter Tuning
The last step on our journey is to conduct some hyperparameter tuning. We have several parameters that we can set in our Decision Tree, and it is not always obvious what we should set them to. We can use a grid search to find the best model after we change many different parameters.
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV

# initialize our model with the default parameters
tree_model = DecisionTreeClassifier()

# set up a dictionary with lists for the different parameter values
param_grid = {'min_samples_leaf': [1, 2, 10, 20, 100], 'min_impurity_decrease': [0.0, 0.1, 0.2]}
# initialize our grid searcher
grid_searcher = GridSearchCV(tree_model, param_grid)

# can call `.fit` just like a single classifier
grid_searcher.fit(X_train, y_train)

# get our best model
tree_model = grid_searcher.best_estimator_

tree_model.get_params()

y_pred = tree_model.predict_proba(X_test)[:, 1]
make_auc_plot(y_test, y_pred)


I am working on the file "DataScienceWorkshop.ipynb".
The first task is:
I am using Pandas to read an manipulate the data. It makes it easy to work with tabular data and is the default data tool for most data-science work.
The dataformat is a DataFrame.
Can you write Python code for it?

df = pd.read_csv('adult.data', names=column_names)

# quick look at the top few rows
df.head()

# and get a sense of the size of the dataframe
df.shape


I am working on the file "DataScienceWorkshop.ipynb".
The first task is:
Logistic Regression
Let's start with one of the simplest models: logistic regression. By default, this model adds "regularization". In this case an "L2" regularization. In practical terms this makes a model that 1) tries to fit the data as well as possible while 2) keep the coefficients as small as possible. This second part helps to avoid overfitting by not letting the model fit too closely to our training data.
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

# initialize the model by calling the class `LogisticRegression`
model = LogisticRegression()

# now that we have our model we can call the `fit` method
model.fit(X_train, y_train)

# get the predicted classes (0 or 1)
model.predict(X_train)

# get the odds (probability) of each class this is an Nx2 array where the first column contains the probability of the first class
model.predict_proba(X_train)


I am working on the file "DataScienceWorkshop.ipynb".
The first task is:
Sometimes there are lots of categories. For instance the NativeCountry feature has 42 unique values. We want to add this information to our model, but adding 42 features seems like a lot for our simple model. Let's add just an 'indicator' if the person came from the USA.
Can you write Python code for it?

#<TODO: confirm the number of unique features is 42>

# we can add a new column like we would add an entry to a dictionary
X['from_USA'] = (df['NativeCountry'] == ' United-States').astype(int)

#<TODO: split again into training and testing>

model.fit(X_train, y_train)

y_pred = model.predict_proba(X_test)[:, 1]
make_auc_plot(y_test, y_pred)

#<TODO: refit our tree model>

#<TODO: and check our test performance>


I am working on the file "DataScienceWorkshop.ipynb".
The first task is:
Testing our model
We want to see how good our model is at fitting the data. This is a classification model, so one sensible way of seeing how good our model would be would be to look at the "accuracy", that is, how often does something that we say should happen actually happen and how often does something we say shouldn't happen not happen (see: https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers for more).
Accuracy can be a misleading metric based on the base rate of the event. If the event usually happens 99% of the time, a classifier could get 99% accuracy by just guessing that event all the time.
A better metric would balance both the true positive rate (how often the thing we predicted happened) and the false positive rate (how often the thing we predicted didn't happen). A popular metric for that is the "area under the receiving operator curve" or AUROC (sometimes just called AUC).
This metric goes from 0.5 to 1.0, where 0.5 is a random classifier and 1.0 is a perfect classifier.
Can you write Python code for it?

from sklearn.metrics import roc_curve, roc_auc_score

# we are going to get the false positive rates and true positive rates to plot
y_pred = model.predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
# and the total area under the curve
auroc = roc_auc_score(y_test, y_pred)

import matplotlib.pyplot as plt
%matplotlib inline

# plot the ROC
fig, axis = plt.subplots()
axis.plot(fpr, tpr)
axis.set_xlabel('False Positive Rate')
axis.set_ylabel('True Positive Rate')

axis.plot([0, 1], [0, 1], color='black', linestyle='--')
axis.set_xlim([0, 1])
axis.set_ylim([0, 1])
axis.set_title(f'AUROC: {auroc:.2f}')


I am working on the file "DataScienceWorkshop.ipynb".
The first task is:
We can look at the coefficients of our model by examining the coef_ attribute.
Can you write Python code for it?


I am working on the file "DataScienceWorkshop_full.ipynb".
The first task is:
Add Another Feature
Let's add a new feature MaritalStatus to our model. We run into a problem in that Marital Status is a feature of different strings. We need to find a way to 'encode' this value into a numeric value.
The method we will use here is called 'dummy encoding' or 'one-hot' encoding. We will create a new feature for each string in the old feature and then give that row a 1 for the new feature if it takes that value (e.g. we would create a 'Divorced' feature and then place a 1 in each row that had an MaritalStatus value 'Divorced'). Bonus: there are some 'fancier' ways that one could use to encode categorical variables for instance in this blog post.
Can you write Python code for it?

# We could also do this directly in sklearn:
#https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html
marital_status_dummies = pd.get_dummies(df['MaritalStatus'])

marital_status_dummies.head()

use_columns = ['Age', 'HoursPerWeek', 'MaritalStatus']

X = pd.get_dummies(df[use_columns])

X.head()

# now that we have new columns we need to redo our train/test split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=3523523)

model.fit(X_train, y_train)

y_pred = model.predict_proba(X_test)[:, 1]
make_auc_plot(y_test, y_pred)


I am working on the file "DataScienceWorkshop_full.ipynb".
The first task is:
Group time!
Some ideas:

Explore more feature engineering and more features
Explore other model types:
ensemble methods: either gradient boosted classifiers or random forests
GAMs (https://github.com/dswah/pyGAM)
other classifiers (https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html)

Explore the data (or model) using visualizations
Do so clustering analysis of the data
Mix and match from above

A note on making plots
If there is one thing I have learned in Astro grad school it is how to make good plots, things like: labeling axes, having clear style, and choosing the right plot type. These small things will make a big difference when presenting your work to a wide array of audiences.
So you want to make some plots...
What type of plot do you want to make? If you need inspiration or want to see different options you can check out the sites below.
https://www.data-to-viz.com/
https://datavizcatalogue.com/index.html
http://datavizproject.com/
And see some examples of common plots made in different python libraries.
http://pythonplot.com/
Can you write Python code for it?

import xgboost
from xgboost import XGBClassifier

use_cols = [
    'Age', 'WorkClass', 'EducationNum', 'MaritalStatus', 'Occupation', 'Relationship', 'Race', 'Gender',
 'CapitalGain', 'CapitalLoss', 'HoursPerWeek', 'NativeCountry'
]
X = pd.get_dummies(df[use_cols])

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=432345)

model = XGBClassifier(n_estimators=250, max_depth=5)
model.fit(X_train, y_train)

y_pred = model.predict_proba(X_test)[:, 1]

make_auc_plot(y_test, y_pred)


I am working on the file "DataScienceWorkshop_full.ipynb".
The first task is:
I am using Pandas to read an manipulate the data. It makes it easy to work with tabular data and is the default data tool for most data-science work.
The dataformat is a DataFrame.
Can you write Python code for it?


I am working on the file "DataScienceWorkshop_full.ipynb".
The first task is:
Imports
Can you write Python code for it?


I am working on the file "DataScienceWorkshop_full.ipynb".
The first task is:
Logistic Regression
Let's start with one of the simplest models: logistic regression. By default, this model adds "regularization". In this case an "L2" regularization. In practical terms this makes a model that 1) tries to fit the data as well as possible while 2) keep the coefficients as small as possible. This second part helps to avoid overfitting by not letting the model fit too closely to our training data.
Can you write Python code for it?


I am working on the file "DataScienceWorkshop_full.ipynb".
The first task is:
Testing our model
We want to see how good our model is at fitting the data. This is a classification model, so one sensible way of seeing how good our model would be would be to look at the "accuracy", that is, how often does something that we say should happen actually happen and how often does something we say shouldn't happen not happen (see: https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers for more).
Accuracy can be a misleading metric based on the base rate of the event. If the event usually happens 99% of the time, a classifier could get 99% accuracy by just guessing that event all the time.
A better metric would balance both the true positive rate (how often the thing we predicted happened) and the false positive rate (how often the thing we predicted didn't happen). A popular metric for that is the "area under the receiving operator curve" or AUROC (sometimes just called AUC).
This metric goes from 0.5 to 1.0, where 0.5 is a random classifier and 1.0 is a perfect classifier.
Can you write Python code for it?

from sklearn.metrics import roc_curve, roc_auc_score

# we are going to get the false positive rates and true positive rates to plot
fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:, 1])
# and the total area under the curve
auroc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])

import matplotlib.pyplot as plt
%matplotlib inline

# plot the ROC
fig, axis = plt.subplots()
axis.plot(fpr, tpr)
axis.set_xlabel('False Positive Rate')
axis.set_ylabel('True Positive Rate')

axis.plot([0, 1], [0, 1], color='black', linestyle='--')
axis.set_xlim([0, 1])
axis.set_ylim([0, 1])
axis.set_title(f'AUROC: {auroc:.2f}')


I am working on the file "DataScienceWorkshop_full.ipynb".
The first task is:
We are almost ready to fit our first model, but before we do, we need to create training and testing sets. Why is it necessary to split our data? We want to be sure we are not overfitting (wiki).
If we had a model that fit the training perfectly, it may not generalize well to new data.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

# splits both X and y into train and testing data - I am using the `random_state` argument to reproduce the same results
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=432345)


I am working on the file "DataScienceWorkshop_full.ipynb".
The first task is:
We can look at the coefficients of our model by examining the coef_ attribute.
Can you write Python code for it?


I am working on the file "DataScience_Tutorials/AZ/Pandas_DF_Creation.ipynb".
The first task is:
Burada gördüyünüz kimi məlumat DataFrame daxilinə keçsədə, sütunlar istədiyimiz ardıcıllıqda yox, əlifba ardıcıllığı üzrə sıralanmışdır. Bu məqamı aradan qaldırmaq üçün ya yuxarıda DataFrame yaradan zamandaki kimi əvvəlcədən column parametri vasitəsi ilə sütun adların və ardıcıllığın qeyd etməli, və ya sonradan aşaqda qeyd olunmuş əmr ilə sütun yerlərin dəyşməliyik.
Can you write Python code for it?


I am working on the file "DataScience_Tutorials/AZ/Pandas_DF_Creation.ipynb".
The first task is:
CSV
Yuxarıdaki funksiyadaki kimi ilk öncə .csv fayla yol, amma sonra sətr daxilində dəyərləri bir birindən ayıran işarə delimiter parameterinə ötürülməlidir. Ötürülmədikdə standart olaraq vergülü qəbul olunur.
Can you write Python code for it?

df = pd.read_csv('https://raw.githubusercontent.com/limpapud/datasets/master/Tutorial_datasets/csv_to_dataframe.csv',
                 delimiter=',')
df


I am working on the file "DataScience_Tutorials/AZ/Pandas_DF_Creation.ipynb".
The first task is:
Cədvəli yaradaq və sütunların yerlərin dəyişək:
Can you write Python code for it?

df = pd.DataFrame.from_dict(email_list_dct)
df=df[['f_name','l_name','email_adrs','a_status',]]
df


I am working on the file "DataScience_Tutorials/AZ/Pandas_DF_Creation.ipynb".
The first task is:
JSON
Json faylından verilənləri qəbul etmək üçün URL və ya fayl sistemində fayla yol tələb olunur. Json faylı misalı aşağıda qeyd olunub.
Diqqət ilə baxdığınız halda özünüz üçün json faylın Lüğət: yazı əsaslı datafreym yaratma metodunda istifadə etdiyimiz dəyər təyinatından heç fərqi olmadığını görmüş oldunuz.
Can you write Python code for it?

df = pd.read_json('https://raw.githubusercontent.com/limpapud/datasets/master/Tutorial_datasets/json_to_dataframe.json')
df = df[['f_name','l_name','email_adrs','a_status',]]
df


I am working on the file "DataScience_Tutorials/AZ/Pandas_DF_Creation.ipynb".
The first task is:
Lüğət: yazı əsaslı
Növbəti misal mənim ən çox tərcihə etdiyim (əlbəttə ki çox vaxt verilənlər istədiyimiz formada olmur, və bizim nəyə üstünlük verdiyimiz onları heç marağlandırmır) üsula keçirik. Bu üsula üstünlük verməyimin səbəbi çox sadədir: bundan əvvəl və bundan sonra qeyd olunmuş yollardan istifadə edərkən məlumatın əldə edilməsi zamanı və ya təmizləmə zamanı  "NaN" dəyəri almadan bəzi məlumatlar pozula bilər ki bu da sütunun və ya yazının sürüşməsinə gətirə bilər ki o zaman analiz ya qismən çətinləşə, və ya ümumiyyətlə verilənlərin qatışması üzündan mənasın itirə bilər. ( Danışdığım problem ilə tanış olmaq üçün 2017/08 tarixində çalışdığım məlumat əldə edilməsi və analizi işimə baxa bilərsiniz.). Amma bu dəfə hər bir dəyər üzrə hansı sütuna aid olması açıq şəkildə qeyd olunur ki, qeyd olunmadığı halda avtomatik "NaN" olaraq qeyd olunur. Nəticədə əlavə rutin təmizləmə işi aparmağa ehtiyac olmur, olmayan dəyərləri isə araşdırmadan ya yığışdırmaq və ya digər metodlar ilə verilənlər ilə doldurmaq olur.

Sözügedən misala yaxından baxaq:
Can you write Python code for it?

email_list=[{
             'f_name' : 'Omar',
             'l_name': 'Bayramov',
             'email_adrs' : 'omarbayramov@hotmail.com',
             'a_status' : 1
            },
            {'f_name' : 'Ali', 'l_name': 'Aliyev', 'email_adrs':'alialiyev@example.com', 'a_status' : 0},
            {'f_name': 'Dmitry', 'l_name': 'Vladimirov', 'email_adrs':'v.dmitry@koala.kl', 'a_status':1},
            {'f_name': 'Donald', 'l_name': 'Trump', 'email_adrs':'grabthat@pussycatdolls.com', 'a_status':1},
            {'f_name': 'Rashid', 'l_name': 'Maniyev', 'email_adrs':'rashid.maniyev@exponential.az', 'a_status':1},
           ]

df=pd.DataFrame(email_list,)
df


I am working on the file "DataScience_Tutorials/AZ/Pandas_DF_Creation.ipynb".
The first task is:
Məlumat sorğulandıqdan sonra fetchall funksiyası vasitəsi ilə sətrləri "oxuyub" data dəyişkəninə təhkim edək və sonda MB bağlantısın bağlayaq.
Can you write Python code for it?


I am working on the file "DataScience_Tutorials/AZ/Pandas_DF_Creation.ipynb".
The first task is:
Növbəti email_list_lst_cln dəyşəninə isə sütun adlarından ibarət siyahı təhkim edirik.
Can you write Python code for it?


I am working on the file "DataScience_Tutorials/AZ/Pandas_DF_Creation.ipynb".
The first task is:
Python xarici mənbələr:
Standart, Python-un daxili verilən strukturlarından başqa Pandas fayl sistemi, Məlumat bazarı və digər mənbələrdən verilənləri əldə edib cədvəl qurmağa imkan yaradır.
Excel fayl
Cədvəli yaratmaq üçün pandas-ın read_excel funksiyasına Excel faylına işarələyən fayl sistemi yolu, fayl internetdə yerləşən zaman isə URL qeyd etmək bəsdir. Əgər faylda bir neçə səhifə varsa, və ya məhz olaraq müəyyən səhifədə yerləşən məlumatı əldə etmək lazımdırsa o zaman sheet_name parametrinə səhifə adın ötürmək ilə məlumatı cədvələ çevirmək olur.
Can you write Python code for it?

df = pd.read_excel('https://raw.githubusercontent.com/limpapud/datasets/master/Tutorial_datasets/excel_to_dataframe.xlsx',
                   sheet_name='data_for_ttrl')
df


I am working on the file "DataScience_Tutorials/AZ/Pandas_DF_Creation.ipynb".
The first task is:
Sorğulama üçün engine yaradaq və məlumat bazası faylına yolu göstərək.
Can you write Python code for it?


I am working on the file "DataScience_Tutorials/AZ/Pandas_DF_Creation.ipynb".
The first task is:
Əldə olunan məlumatın strukturu tanış qəlir mi? Diqqət ilə baxsanız ilk tanış olduğumuz Siyahı: sətr əsaslı məlumat strukturun tanıyarsınız. Artıq tanış olduğumuz proseduru icra edərək cədvəl qurmaq qaldı:
Can you write Python code for it?


I am working on the file "DataScience_Tutorials/AZ/Pandas_SQL.ipynb".
The first task is:
2-ci indeksdə 'Valentina' adlı istifadəçi sorğulanan parameterlərə uyğun gəldi və "Female"-dən "Male"-yə dəyişdi.
INSERT
SQL-də INSERT əmri məlumatın əlavə olunması üçün istifadə olunur.
Nümaiş üçün staff_df-də yeni işçi əlavə edək:
SQL variantı:
Pandas vasitəsi ilə isə eyni nəticə almaq üçün artıq tanış olan .loc indeks köməkçisindən istifadə edərək sətr əlavə edirik.
Pandas variantı: DatafreymAdı.loc[ indeks ] = ['vergül', 'ilə', 'ayrılmış', 'dəyərlər']
İndeksi əldə etmək üçün datafreymin uzunluğun ölçüb uzunluğunu dəyər kimi ".loc"-a ötürək.
Can you write Python code for it?

len(staff_df)

staff_df.loc[len(staff_df)] = ['6a9472d4-4949-4f98-8543-c6878d558422','Omar','Bayramov','Male','NaN','Limpapud','NaN']

staff_df.tail(3) # son yazıları yoxlayaq


I am working on the file "DataScience_Tutorials/AZ/Pandas_SQL.ipynb".
The first task is:
DELETE
SQL-də DELETE əmri məlumatın silinməsi üçün istifadə olunur.
Hər-hansı bir məlumatı datafreymdən silmək istədikdə SQL-a əks məntiqdən istifadə etməliyik. Nümaiş üçün az əvvəl əlavə etdiyimiz "Ömər" və ayrı "Ömər" işçilərin silək.
SQL variantı:
Yəni əgər SQL sorğu "staff_db-dən 'Ömər' adlı işçiləri poz" əmrin verirsə Pandas sorğusu "staff_db-də "Ömər" adlı işçilərdən başqa hamısın saxla məntiqi istifadə edir.
Can you write Python code for it?

staff_df = staff_df.loc[staff_df['first_name'] != 'Omar']
staff_df.tail(3)


I am working on the file "DataScience_Tutorials/AZ/Pandas_SQL.ipynb".
The first task is:
Gördüyümüz kimi binar_maska 1 ölçülü datafreym yarandı və realda "Male" dəyəri olan sətrdə "True", digər dəyərlər olan sətrlərdə isə "False" dəyərin saxlayır. İndi isə bu maskanı tətbiq edək və cavabları yoxlayaq:
Can you write Python code for it?


I am working on the file "DataScience_Tutorials/AZ/Pandas_SQL.ipynb".
The first task is:
Pandas modulu ilə işin nümaişi 2 hissədən ibarət olacaq:
1. Real SQL əmr sorğusunun yazılması və icrası
2. Yazılmış sorğunun analoqunun datafreymlər üzərində Pandas vasitəsi ilə hazırlanması
SELECT
SQL-də SELECT əmri məlumatın sorğulanması və əldə edilməsi üçün istifadə olunur.
Deyək ki biz staff_db cədvəlindən bütün sütun və sətrləri əldə etmək istəyirik. Bunun üçün yazılmış SQL sorğusu aşağıdaki kimidir:
Pandas vasitəsi ilə bu komandanı sadəcə datafreymin adın yazmaq ilə icra edə bilərsiniz.
Can you write Python code for it?


I am working on the file "DataScience_Tutorials/AZ/Pandas_SQL.ipynb".
The first task is:
Ups! Nədənsə (əslində əlbəttə ki Pandasın standart ayarlarına uyğun, perfomans və praktiklik baxımından) məlumat öyrəşdiyimiz SQL sorğulama zamanı kimi tam olaraq yox, hissə-hissə olaraq çıxdı, yəni yalnız ilk 30 və son 30 sətr əks olundu.
Bu məqamı nəzərə alaraq ayarlarda maksimal sətr və sütun sayın dəyişək və limitsiz edək. (Diqqət! Real həyatda böyük həcmli verilənlər ilə iş zamanı bunu etməyin!!!)
Maksimal sütun və sətr sayın None qeyd edib limitləri aradan qaldırırıq.
Can you write Python code for it?

pd.set_option('display.max_columns',7) # sütün sayı üzərindən limit qaldıran əmr
pd.set_option('display.max_rows',60) # sətr sayı üzərindən limit qaldıran əmr


I am working on the file "DataScience_Tutorials/AZ/Pandas_SQL.ipynb".
The first task is:
Yenidən yoxlayaq ...
Can you write Python code for it?


I am working on the file "DataSetStratification.ipynb".
The first task is:
Print data to file.
Can you write Python code for it?

fname = data['filename'][:-8] + '.csv'
print(fname)
df.to_csv(fname, na_rep='NA', index=False)


I am working on the file "DataSetStratification.ipynb".
The first task is:
Print the labeled to data to a new file.
Can you write Python code for it?

fname = data['filename']
print(fname)
df.to_csv(fname, na_rep='NA', index=False)


I am working on the file "DataSetStratification.ipynb".
The first task is:
This cell can be used for all data sets except colon. colon is special because it has 3 types of events instead of just 2. Just change the first line to run a different data set.
Can you write Python code for it?

#data = ds._pbc
#data = ds._lung
#data = ds._nwtco
data = ds._flchain

df = pd.read_csv(data['filename'][:-4] + "_org.csv",
                 sep=None, engine='python')
k = 4

# flchain has three guys at zero, remove them
if 'flchain' in data['filename']:
    df = df[(df[data['timecol']] > 0)]

# Need shape later
n, d = df.shape

# Random reordering
df = df.reindex(np.random.permutation(df.index))
df.sort(data['eventcol'], inplace=True)

assignments = np.array((n // k + 1) * list(range(0, k)))
assignments = assignments[:n]

print(assignments.shape)
print(df.shape)

# Create a new column that specifies set
df['set'] = 1
# 0 is testing
df.loc[assignments == 0, 'set'] = 'testing'
# rest is training
df.loc[assignments != 0, 'set'] = 'training'

print("Training size:", np.sum(df['set'] == 'training'))
print("Testing size:", np.sum(df['set'] == 'testing'))

df = df.reindex(np.sort(df.index))


I am working on the file "DataStory/OldTaxiData.ipynb".
The first task is:
Loading files
Can you write Python code for it?

path=os.path.expanduser("~/Documents/TaxiTripData/2013taxi_trip_data/")

files=glob.glob(path+'*.csv.zip.gz')

countlist=[]
cunk=100000
for i in files:
    print(i)
    iterate=pd.read_csv(i,usecols=[5],iterator=True,chunksize=cunk,names=['pickup_datetime'],header=0)
    counter=0
    for chunk in iterate:
        a=pd.to_datetime(chunk['pickup_datetime'])
        if a.dtype==np.dtype('<M8[ns]'):
            countlist.append(a.dt.date.value_counts())
        else:
            bads=[]
            for q in range(a.shape[0]):
                b=pd.to_datetime(a.iloc[q])
                if type(b)!=pd.tslib.Timestamp:
                    bads.append(q)
            a.drop(bads,axis=0,inplace=True)
            a=pd.to_datetime(a)
            if a.dtype==np.dtype('<M8[ns]'):
                countlist.append(a.dt.date.value_counts())
            else:
                print('chunk ' + str(counter)+' of file '+i+' encountered a problem and was not processed.')

        counter+=1

cond=pd.concat(countlist)
cond=cond.groupby(cond.index).sum()

cond.index=pd.to_datetime(cond.index)

%pwd

cond.to_pickle('2013_taxi_date_counts.pkl')


I am working on the file "DataStory/When do people take cabs?.ipynb".
The first task is:
Taxi rides, meanwhile, fluctuate surprisingly much throughout the year.
Can you write Python code for it?

from wand.image import Image as WImage
img = WImage(filename='yellowcabyear.pdf')
img


I am working on the file "DataStory/When do people take cabs?.ipynb".
The first task is:
Uber is growing
Unsurprisingly as it has just started up, uber is growing in NYC.  I suspect that doing a regression on this data will show that it isn't quite growing exponentially, at least over the period studied.
Can you write Python code for it?

from wand.image import Image as WImage
img = WImage(filename='ubergrowth.pdf')
img


I am working on the file "DataStructures/array.ipynb".
The first task is:
Alternative Byte Ordering
Can you write Python code for it?

import array
import binascii

def to_hex(a):
    chars_per_item = a.itemsize * 2  # 2 hex digits
    hex_version = binascii.hexlify(a)
    num_chunks = len(hex_version) // chars_per_item
    for i in range(num_chunks):
        start = i * chars_per_item
        end = start + chars_per_item
        yield hex_version[start:end]

start = int('0x12345678', 16)
end = start + 5
a1 = array.array('i', range(start, end))
a2 = array.array('i', range(start, end))
a2.byteswap()

fmt = '{:>12} {:>12} {:>12} {:>12}'
print(fmt.format('A1 hex', 'A1', 'A2 hex', 'A2'))
print(fmt.format('-' * 12, '-' * 12, '-' * 12, '-' * 12))
fmt = '{!r:>12} {:12} {!r:>12} {:12}'
for values in zip(to_hex(a1), a1, to_hex(a2), a2):
    print(fmt.format(*values))


I am working on the file "DataStructures/array.ipynb".
The first task is:
Manuipulating Arrays
Can you write Python code for it?

import array
import pprint
a = array.array('i', range(3))
print('initialize\n', a)

a.extend(range(3))
print('Extend\n',a)

print('Slice \n', a[2:5])

print('Iterartor\n')
print(list(enumerate(a)))


I am working on the file "DataStructures/collection/ChainMap.ipynb".
The first task is:
Updating Values
Can you write Python code for it?

import collections
a = {'a':'A', 'c':'C'}
b = {'b':'B', 'c':'D'}
m = collections.ChainMap(a, b)
print(m.maps)

m['c'] ='E'
print(m.maps)

print(a)

import collections
a = {'a':'A', 'c':'C'}
b = {'b':'B', 'c':'D'}
m1 = collections.ChainMap(a, b)
m2 = m1.new_child()
print('m1 before', m1)
print('m2 before', m2)
m2['c']='E'
print('m1 after', m1)
print('m2 after', m2)


I am working on the file "DataStructures/collection/Counter.ipynb".
The first task is:
Arithmetic
Can you write Python code for it?

import collections
c1 = collections.Counter(['a','b', 'c', 'a', 'b', 'b'])
c2 = collections.Counter('alphabet')
print('C1:', c1)
print('C2:', c2)
print('\n Combined counts')
print(c1+c2)
print('\n subtractions')
print(c1-c2)
print('\n Interaction')
print(c1 & c2)
print('\n union')
print(c1 | c2)


I am working on the file "DataStructures/pprint.ipynb".
The first task is:
Arbitrary Classes
Can you write Python code for it?

from pprint import pprint

class node:

    def __init__(self, name, contents=[]):
        self.name = name
        self.contents = contents[:]

    def __repr__(self):
        return (
            'node(' + repr(self.name) + ', ' +
            repr(self.contents) + ')'
        )

trees = [
    node('node-1'),
    node('node-2', [node('node-2-1')]),
    node('node-3', [node('node-3-1')]),
]
pprint(trees)


I am working on the file "DataStructures/pprint.ipynb".
The first task is:
Controlling output width
Can you write Python code for it?

from pprint import pprint

for width in [80, 5]:
    print('WIDTH =', width)
    pprint(data, width=width)
    print()

from pprint import pprint

print('DEFAULT:')
pprint(data, compact=False)
print('\nCOMPACT:')
pprint(data, compact=True)


I am working on the file "DataStructures/pprint.ipynb".
The first task is:
Formatting
Can you write Python code for it?

import logging
from pprint import pformat

logging.basicConfig(
    level=logging.DEBUG,
    format='%(levelname)-8s %(message)s',
)

logging.debug('Logging pformatted data')
formatted = pformat(data)
for line in formatted.splitlines():
    logging.debug(line.rstrip())


I am working on the file "DataStructures/pprint.ipynb".
The first task is:
Limiting Nested Output
Can you write Python code for it?

from pprint import pprint
pprint(data, depth=1)
pprint(data, depth=2)


I am working on the file "Data_Analysis.ipynb".
The first task is:
If we look at just the first five by five
block of this pivoted data, we can see
what we've done we now have a two
dimensional data frame where each column
is a day in the data set and each row
corresponds to an hour during that day.
Can you write Python code for it?


I am working on the file "Data_Analysis.ipynb".
The first task is:
Now one thing you might want to do is see if
there's any sort of annual trend in the
number of riders, any sort of annual
growth or decline and ridership. We can create a
rolling window. So let's resample and
do a rolling sum of over  days of all
the data and what comes out there is
kind of the annual trend each of these
points is the sum of rides in the
previous days.
Can you write Python code for it?


I am working on the file "Data_Analysis.ipynb".
The first task is:
We can see that somehow that the East
Side and west side of the bridge have
kind of slipped a little bit, i.e., the trends
are reversed so that the the total
counts of biker bikes across the bridge
hover right around one million per year or something like that.
That's been pretty consistent for the
past few years plus or minus a few
percent.
Another thing that we can do is that we can take a look at the
trend within individual days. So we will take a look at "group by" here
and let's group by the time of day
and take the mean and then plot it to
see what that looks like.
Can you write Python code for it?


I am working on the file "Data_Analysis/Data_Analysis.ipynb".
The first task is:
From the above visualizations, only less than half of the 891 passengers survived the sinking of the Titanic.
The age distributions for both groups have means that are very close.
Despite the number of women passengers being much lower than men, in terms of survival, women survivors are more than men.
This shows signs that being a woman increased the chance of survival.

Drilling down into survival by class :
Can you write Python code for it?

g = sns.factorplot(x="Sex", hue = "Survived", col="Pclass",
                   data=titanic_clean, kind="count", size=4, aspect=1, alpha=.65)


I am working on the file "Data_Analysis/Data_Analysis.ipynb".
The first task is:
The table above gives an overview of the dataset. Here are some important points to note.
- There are 891 records of data in total
- The mean age of people onboard was around 30 years
- The average person seems to have traveled second class (mean= 2.3)
- The middle 50% of the passengers were between the age group of 22 to 35 years
Looking at the survival data graphically:
Can you write Python code for it?

f, axs = plt.subplots(figsize=(18, 5), ncols = 3)
sns.set_palette("Set2")

# Fig 1 - Survival Distribution
sns.countplot(x="Survived", data=titanic_clean, alpha=.65,
              ax=axs[0]).set_title("Fig 1: Survival Distribution, (1 = Survived)")

# Fig 2 - Survival by Age
sns.boxplot(x="Survived", y="Age",data=titanic_clean,
            ax=axs[1]).set_title("Fig 2: Survival by Age, (1 = Survived)")

# Fig 3 - Survival by Gender
sns.countplot(y="Survived", hue="Sex", palette={"male":"m","female":"orange"}, data=titanic_clean,
              alpha=.55, ax=axs[2]).set_title("Fig 3: Survival by Gender, 1 = Survived")


I am working on the file "Data_Analysis/Data_Analysis.ipynb".
The first task is:
This shows that those who traveled third class were much more (primarily male passengers) than those traveled by the second and first classes respectively
Among the women who survived, the highest number of them traveled first class. This indicates that the combination of being a woman, and traveling first class may have increased one's chance of survival.
To test the significance between these variables, statistical hypothesis tests have been used.

Statistical Inference and Hypothesis Testing
Chi-Squared Test for Independence: 'Pclass' vs. 'Survived'
$H_0$: Survival is NOT dependent of the travel class of the passenger
$H_a$: Survival IS dependent on the travel class of the passenger
Can you write Python code for it?

# Chi-squared test
def chi_squared_test(col1, col2, isPrint=False):
    contingency_table = pd.crosstab(col1, col2)
    chi, p, dof, expected = stats.chi2_contingency(contingency_table)
    if(isPrint):
        print(contingency_table)
        print "Chi square: ", chi
        print "p-value: ", np.round(p, decimals=4)
        print "Degrees of freedom: ", dof
        print "\nExpected frequency:\n", expected
    return contingency_table, chi, p, dof, expected

#Pclass vs. Survived (Chi-squared test)

contingency_table, chi, p, dof, expected = chi_squared_test(titanic_clean["Pclass"],
                                                            titanic_clean["Survived"], isPrint=True)


I am working on the file "Data_Analysis/Data_Analysis.ipynb".
The first task is:
We reject the null hypothesis since p-value < α level of 0.05
Thus it can be concluded that survival is dependent on the gender of the passenger

Independent two-sample t-test: 'Age' vs. 'Survived'
$H_0 :  \mu_0 = \mu_1 $ There is no difference in mean age between the survivors and the victims
$H_a :   \mu_0 \neq \mu_1 $ The mean age of survivors is signifcantly different than the mean age of victims
Can you write Python code for it?

# Visualizing both the distributions

h = sns.FacetGrid(titanic_clean, col="Survived", hue="Survived").map(sns.distplot, "Age")
h.set_axis_labels("Age", "KDE")

# Mean age of passengers who died vs. survived (survived=1)
print "Variance:"
print titanic_clean.groupby("Survived")["Age"].var()
print ('\n')
print "Number of passengers:"
print titanic_clean.groupby("Survived")["Age"].size()


I am working on the file "Data_Analysis/Data_Analysis.ipynb".
The first task is:
We reject the null hypothesis since p-value < α level of 0.05
Thus it can be concluded that survival is dependent on the travel class of the passenger

Chi-Squared Test for Independence: 'Sex' vs. 'Survived'
$H_0$: Survival is NOT dependent on the passenger's sex
$H_a$: Survival IS dependent on the passenger's sex
Can you write Python code for it?

#Sex vs. Survived (Chi-squared test)

contingency_table, chi, p, dof, expected = chi_squared_test(titanic_clean["Sex"],
                                                           titanic_clean["Survived"],
                                                           isPrint=True)


I am working on the file "Data_Analysis/convert_ascii_to_netcdf.ipynb".
The first task is:
For the sake of simplicity
Can you write Python code for it?


I am working on the file "Data_Analysis/convert_ascii_to_netcdf.ipynb".
The first task is:
Get number of lines and columns
Can you write Python code for it?

nlines = ascii_data.shape[0]
ncols = ascii_data.shape[1]


I am working on the file "Data_Analysis/convert_ascii_to_netcdf.ipynb".
The first task is:
Print some information
Can you write Python code for it?

print('--> data shape        = (%d,%d) ' % ascii_data.shape)
print('--> number of lines   = %d ' % nlines)
print('--> number of columns = %d ' % ncols)


I am working on the file "Data_Analysis/convert_ascii_to_netcdf.ipynb".
The first task is:
Print the information about the transposed data
Can you write Python code for it?

print('\nCorrect shape of data!\n')
print('--> data shape      = (%d,%d) ' % data.shape)
print('--> number of lat   = %d ' % nlat)
print('--> number of lon   = %d ' % nlon)


I am working on the file "Data_Analysis/convert_ascii_to_netcdf.ipynb".
The first task is:
Read Ascii data
Can you write Python code for it?

ascii_data = np.genfromtxt('data/1901_1.csv', dtype=None, delimiter=',')


I am working on the file "Data_Analysis/convert_ascii_to_netcdf.ipynb".
The first task is:
Set NaN to missing value
Can you write Python code for it?


I am working on the file "Data_Analysis/convert_ascii_to_netcdf.ipynb".
The first task is:
Set missing value
Can you write Python code for it?


I am working on the file "Data_Analysis/convert_ascii_to_netcdf.ipynb".
The first task is:
The data is in the wrong shape (columns x lines).
The rows and columns must be swapped (lines x columns).
Can you write Python code for it?

data = ascii_data.T
nlat = data.shape[0]
nlon = data.shape[1]


I am working on the file "Data_Analysis/convert_ascii_to_netcdf.ipynb".
The first task is:
Write data array to file data/var.txt
Can you write Python code for it?

np.savetxt('data/var.txt', data, delimiter=', ', fmt='%1.2e')


I am working on the file "Data_Analysis/convert_csv_to_netcdf.ipynb".
The first task is:
CDO command
Can you write Python code for it?

cdo.settaxis(
        time,
        input='-setname,'+varname+ \
        ' -input,data/gridfile.txt < data/var.txt',
        output='input.nc',
        options = '-f nc')


I am working on the file "Data_Analysis/convert_csv_to_netcdf.ipynb".
The first task is:
Get size of the coordinates
Can you write Python code for it?

nlon = len(lon)
nlat = len(lat)
nlines = nlon