I am working on the file "Data_Analysis/convert_csv_to_netcdf.ipynb".
The first task is:
Get the longitude, latitude and data values of the data columns
Can you write Python code for it?

lon = list(map(float, csv_data['f0']))
lat = list(map(float, csv_data['f1']))
values = list(map(float, csv_data['f2']))


I am working on the file "Data_Analysis/convert_csv_to_netcdf.ipynb".
The first task is:
Print some information
Can you write Python code for it?

print('--> longitudes:   ', lon)
print('--> # longitudes: %d' % nlon)
print('--> latitudes:    ', lat)
print('--> # latitudes:  %d' % nlat)
print('--> values:       ', values)


I am working on the file "Data_Analysis/convert_csv_to_netcdf.ipynb".
The first task is:
Set variable name
Can you write Python code for it?


I am working on the file "Data_Analysis/convert_csv_to_netcdf.ipynb".
The first task is:
Write value array to file data/var.txt
Can you write Python code for it?

fv = open('data/var.txt', 'w')
[fv.write(str(s)+'\n') for s in values]
fv.close()


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
And now we want to mask the land part of the data. The easiest way is to set all zeros to one and all ones to NaN. Matplotlib will automatically use the color grey for the missing values.
Can you write Python code for it?

import numpy as np
import matplotlib as mpl
import copy

lsm = np.where(lsm==0, 1, np.nan)

masked = tsurf * lsm

masked.plot(subplot_kws=dict(projection=ccrs.PlateCarree(), facecolor='gray'), transform=ccrs.PlateCarree(),
           cbar_kwargs={'orientation': 'horizontal'})
p.axes.set_extent([-20.,20.,30.,60.])
p.axes.coastlines()


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Based on CDO version:
Can you write Python code for it?


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Delete temporary files
Can you write Python code for it?


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
In comparison to xarray's dataset information.
Can you write Python code for it?

import xarray as xr

xr.open_dataset(infile)


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Let's see how it looks like.
Can you write Python code for it?

p = masked.plot(subplot_kws=dict(projection=ccrs.PlateCarree(), facecolor='gray'), transform=ccrs.PlateCarree(),)
p.axes.set_extent([-20.,20.,30.,60.])
p.axes.coastlines()


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
List all operators starting with 'sel'
Use list comprehension
Can you write Python code for it?

[key for key, value in cdo.operators.items() if key.startswith('sel')]


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Masking data
Before we can use a mask on a data variable we need to create a mask file using the same grid as the data variable.
Can you write Python code for it?

cdo.setname('lsm', input='-gtc,0.5 -remapbil,'+infile+' -topo', options='-f nc', output='lsm.nc')


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Now, we want to get only tsurf values over land for timestep 1.
Can you write Python code for it?

masked = cdo.setctomiss(0, input='-mul lsm.nc -seltimestep,1 '+infile, returnXArray='tsurf')


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Plot the fieldmean data:
Can you write Python code for it?


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Print the value of the first timestep, first latitude, first longitude:
Can you write Python code for it?


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Remapping
Interpolate the data of input file to a new grid using the bilinear interpolation method.
For the ease of use, we select the variable tsurf and only the first timestep to interpolate the data to a longitude 1 deg x latitude 1 deg (r360x180) grid.
Can you write Python code for it?

cdo.remapbil('r360x180', input='-seltimestep,1 -selvar,tsurf '+infile, output=outfile)


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Select a sub-region
Can you write Python code for it?

cdo.sellonlatbox('20,30,70,80', input='-seltimestep,1 '+infile, output=outfile)
cdo.sinfon(input=outfile)


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Select timesteps
Select timestep 1 and 10:
Can you write Python code for it?

cdo.seltimestep('1,10', input=infile, output=outfile)
cdo.sinfon(input=outfile)


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Select variables
Can you write Python code for it?

cdo.selvar('tsurf', input=infile, output=outfile)
cdo.sinfon(input=outfile)

cdo.selvar('u10,v10', input=infile, output=outfile)
cdo.sinfon(input=outfile)


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Set output file.
Can you write Python code for it?


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Show CDO version
python-cdo version:
Can you write Python code for it?


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Show information about the variables value range, times, levels in more detail
Can you write Python code for it?


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Turn debugging on
Can you write Python code for it?


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Turn off debugging
Can you write Python code for it?

cdo.debug = False

infile = 'data/rectilinear_grid_2D.nc'

cdo.sinfon(input=infile)


I am working on the file "Data_Analysis/data_analysis_with_CDO.ipynb".
The first task is:
Use a variables instead of output files.
Can you write Python code for it?

lsm = cdo.setname('lsm', input='-gtc,0.5 -remapbil,'+infile+' -topo', options='-f nc', returnXArray='lsm')

tsurf = cdo.seltimestep(1, input='-selvar,tsurf '+infile, returnXArray='tsurf')

masked = tsurf * lsm
print(masked)


I am working on the file "Data_Analysis_First_Program.ipynb".
The first task is:
Counts of Employment Rate:
Can you write Python code for it?

employment_bins = pd.cut(data['employrate'], bins=np.linspace(0, 100., num=21))
employment_bins.value_counts(sort=False, dropna=False)


I am working on the file "Data_Analysis_First_Program.ipynb".
The first task is:
Employment rate frequencies
Can you write Python code for it?

display(Markdown("Employment rate (min, max) = ({0:.2f}, {1:.2f})".format(data['employrate'].min(), data['employrate'].max())))


I am working on the file "Data_Analysis_First_Program.ipynb".
The first task is:
Percentages of Employment Rate:
Can you write Python code for it?


I am working on the file "Data_Analysis_First_Program.ipynb".
The first task is:
Suicide rate per 100,000 frequencies
Can you write Python code for it?

display(Markdown("Suicide rate (min, max) = ({0:.2f}, {1:.2f})".format(data['suicideper100th'].min(), data['suicideper100th'].max())))


I am working on the file "Data_Analysis_First_Program.ipynb".
The first task is:
We will now have a look at the frequencies of the variables.
Internet use rate frequencies
First for the Internet usage, the counts are (including data missing):
Can you write Python code for it?


I am working on the file "Data_Analyst_ND_Project0.ipynb".
The first task is:
Let's do a basic statistical calculation on the data using code! Run the block of code below to calculate the average "Food Pinching Efficiency" for all 31 participants and all chopstick lengths.
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
2 二维数组
Can you write Python code for it?

A = np.arange(10,19).reshape((3,3))
print A
print A[1,2]


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
3 索引机制、切片和迭代方法
索引机制
通过方括号[]的索引机制
1 一维数组
Can you write Python code for it?

a = np.arange(10,16)
print a
print a[1]
print a[-1]


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
4 条件和布尔数组
按照一定的条件从数组中筛选出一部分元素
Can you write Python code for it?

A = np.random.random((4,4))
A

A <0.5

# 按照条件抽选出数组
A[A<0.5]


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
5 形状改变
reshape()函数改变数组的形状来形成一个新的数组
Can you write Python code for it?

a = np.random.random(12)
print a
A = a.reshape((3,4))
print A
print a


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
7 数组切分
数组切分是数组连接的逆操作，也分为hsplit()和vsplit()两种
Can you write Python code for it?

A = np.arange(16).reshape((4,4))
print A
# 水平方向切割成两部分
B,C=np.hsplit(A,2)
print B
print C

B,C=np.vsplit(A,2)
print B
print C


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
8 其他概念
副本和视图
numpy为所有的复制运算都不会为数组创建和数组的任何元素创建副本，如果想要创建副本，使用copy()函数
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
9 结构化数组
数组的数据类型不仅可以说会用内置的，也可以自定义一些数据类型，记录一些事数据结构而不是独立的元素，指定dtype对象
bytes ---> b1
int ---> i1,i2,i4,i8
unsigned ints ---> u1,u2,u4,u8
floats ---> f2,f4,f8
complex ---> c8,c16
fix length strings ---> a<n>
Can you write Python code for it?

structured = np.array([(1,'first',0.5,1+2j),(2,'second',1.3,2-2j),(3,'third',0.8,1+3j)],dtype=('i2,a6,f4,c8'))
structured

structured[0]

# 引用同列下
structured['f1']

# 指定每一列的名称
structured = np.array([(1,'first',0.5,1+2j),(2,'second',1.3,2-2j),(3,'third',0.8,1+3j)],dtype=[('id','i2'),('position','a6'),('value','f4'),('complex','c8')])
structured

structured.dtype.names=('id','order','value','complex')
structured['order']


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
split函数更为复杂，可以把数组分割成几个不对称的部分，还指定切分的索引，axis=0,表示为行索引，axis=1表示列索引
Can you write Python code for it?

A1,A2,A3 = np.split(A,[1,3],axis=1)
print A1
print A2
print A3

A1,A2,A3 = np.split(A,[1,3],axis=0)
print A1
print A2
print A3


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
不形成的数组，直接修改数组的shape属性
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
从上面可以看出，对于常见的$+ \space - \space \times $都是每个元素之间的操作
矩阵积
矩阵乘法：$$C=A \times B$$
其中$A_{m \times n},B_{n \times p}$，那么矩阵$C_{m \times p}$,其中$$C_{ij}=\sum_{k=1}^{n}A_{ik}*B_{kj}$$
Can you write Python code for it?

A = np.arange(0,9).reshape(3,3)
B = np.ones((3,3))
print A.dot(B)
print np.dot(A,B)


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
切片操作
抽取数组中一部分形成一个新的数组，在Python列表中进行切片操作是原列表的副本，而在numpy中，则是原数据的一个缓冲。
Can you write Python code for it?

a = range(5)
print a
a_a = a[1:3]
a_a[0]=10
print a_a
print a

b = np.arange(10,16)
b_b1 = b[1:5]
print b_b1
b_b1[0]=20
print b_b1
print b


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
切片的语法：省略第一个元素，表示从0开始；省略第二元素，表示到最大的索引值；省略第三个元素表示间隔为1。
对于二维数组切片，不过要指定切片的维数。
Can you write Python code for it?

A = np.arange(10,19).reshape((3,3))
print A
print A[0,:]
print A[0:2,0:2]


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
创建二维数组
Can you write Python code for it?

b = np.array([[1.3,2.4],[0.3,4.1]])
print b.dtype
print b.ndim
print b.size
print b.shape


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
创建数组既可以通过list也可以通过tuple,也可以list和tuple混合使用
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
广播机制
广播机制对两个及以上的数组进行运算或者是函数处理，满足的条件为：
1. 连个数组的各维度的长度相同
2. 如果不相容，其中那一维是1
Can you write Python code for it?

A = np.arange(16).reshape((4,4))
B = np.arange(4)
print A
print B


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
数组A为$4 \times 4$而数组B为$4 \times 1$，但满足广播的条件
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
数组文件的读写
二进制文件读写
Can you write Python code for it?

data = np.random.random((4,4))
data

np.save('saved_data',data)

loaded_data=np.load('saved_data.npy')
loaded_data


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
数组迭代
在Python中可以通过for迭代的方式进行数据处理，在numpy中同样可以通过该方式
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
方括号内传入多个参数，来获取多个值
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
聚合函数
对数组进行操作，并返回一个数值
Can you write Python code for it?

import numpy as np
a = np.array([3.3,4.5,5.7,0.3])
print a.sum()
print a.max()
print a.min()
print a.std()


I am working on the file "Data_Analytics_in_Action/numpy.ipynb".
The first task is:
自增操作
Can you write Python code for it?

a = np.arange(4)
print a
a +=1
print a
a -=2
print a


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
HDF5数据
HDF文件（hierarchical data from)等级数据格式，用二进制文件存储数据。
Can you write Python code for it?

from pandas.io.pytables import HDFStore
store = HDFStore('mydata.h5')
store['obj1']=frame

store['obj1']


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
HTML读表格
Can you write Python code for it?

web_frames = pd.read_html('myFrame.html')
web_frames[0]

# 以网址作为参数
ranking = pd.read_html('http://www.meccanismocomplesso.org/en/meccanismo-complesso-sito-2/classifica-punteggio/')
ranking[0]


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
JSON数据
Can you write Python code for it?

frame = pd.DataFrame(np.arange(16).reshape((4,4)),
                    index=['white','black','red','blue'],
                    columns=['up','down','right','left'])
frame.to_json('frame.json')

# 读取json
pd.read_json('frame.json')


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
Regexp 解析TXT文件
使用正则表达式指定sep,来达到解析数据文件的目的。
正则元素 | 功能
--- | ---
. | 换行符以外所有元素
\d | 数字
\D | 非数字
\s | 空白字符
\S | 非空白字符
\n | 换行符
\t | 制表符
\uxxxx | 使用十六进制表示ideaUnicode字符
数据文件随机以制表符和空格分隔
white  red  blue  green
1  4 3   2
2 4    6 7
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
pickle数据
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
从TXT文件中读取部分数据
只想读文件的一部分，可明确指定解析的行号，这时候用到nrows和skiprows选项，从指定的行开始和从起始行往后读多少行（norow=i)
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
写入文件

to_csv(filenmae)
to_csv(filename,index=False,header=False)
to_csv(filename,na_rep='NaN')

HTML文件读写
写入HTML文件
Can you write Python code for it?

frame = pd.DataFrame(np.arange(4).reshape((2,2)))
print frame.to_html()


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
创建一个具有等级结构的DataFrame对象，可以添加index_col选项,数据文件格式
colors,status,item1,item2,item3
black,up,3,4,6
black,down,2,6,7
white,up,5,5,5
white,down,3,3,2
red,up,2,2,2
red,down,1,1,4
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
创建复杂的DataFrame
Can you write Python code for it?

frame = pd.DataFrame(np.random.random((4,4)),
                    index=['white','black','red','blue'],
                    columns=['up','down','left','right'])
frame

s = ['']
s.append('<TITLE>MY DATAFRAME</TITLE>')
s.append('')
s.append(frame.to_html())
s.append('')
html=''.join(s)
with open('myFrame.html','w') as html_file:
    html_file.write(html)


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
可以指定header
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
实例 ：
对于一列数据，每隔两行取一个累加起来，最后把和插入到列的Series对象中
Can you write Python code for it?

out = pd.Series()
i=0
pieces = pd.read_csv('myCSV_01.csv',chunksize=3)
for piece in pieces:
    print piece
    out.set_value(i,piece['white'].sum())
    i += 1
out


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
数据库连接
以sqlite3为例介绍
Can you write Python code for it?

frame=pd.DataFrame(np.arange(20).reshape((4,5)),
                  columns=['white','red','blue','black','green'])
frame

from sqlalchemy import create_engine
enegine=create_engine('sqlite:///foo.db')

frame.to_sql('colors',enegine)

pd.read_sql('colors',enegine)


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
读写Excel文件
Can you write Python code for it?

pd.read_excel('data.xlsx')

pd.read_excel('data.xlsx','Sheet2')

frame = pd.DataFrame(np.random.random((4,4)),
                    index=['exp1','exp2','exp3','exp4'],
                    columns=['Jan2015','Feb2015','Mar2015','Apr2015'])
frame

frame.to_excel('data2.xlsx')


I am working on the file "Data_Analytics_in_Action/pandasIO.ipynb".
The first task is:
读取没有head的数据
1,5,2,3,cat
2,7,8,5,dog
3,3,6,7,horse
2,2,8,3,duck
4,4,2,1,mouse
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/scikit_learning.ipynb".
The first task is:
3 绘制散点图
萼片散点图

x轴
萼片的长度
y轴
萼片的宽度
Can you write Python code for it?

import matplotlib.pyplot as plt
import matplotlib.patches as mpathes
from sklearn import datasets
iris = datasets.load_iris()
x = iris.data[:,0]
y = iris.data[:,1]
species = iris.target
x_min, x_max = x.min() - 0.5, x.max() + 0.5
y_min, y_max = y.min() - 0.5, y.max() + 0.5
plt.figure()
plt.title(u'Iris Datase - classification by sepal sizes')
plt.scatter(x,y,c=species)
plt.xlabel('sepal length')
plt.ylabel('sepal width')
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.xticks(())
plt.yticks(())
plt.show()


I am working on the file "Data_Analytics_in_Action/scikit_learning.ipynb".
The first task is:
4 主成分分析
我们用花瓣、萼片的长和宽四个维度表达出测量的数据，但是显示是三维空间，无法再展示四维数据，上面一个步骤中，分别使用萼片和花瓣的测量数据制作的，那么怎样在一幅图中把四个测量数据整合起来。
主成分分析法 (Prinical Component Analysis) 可以降低系统的维数目，但是也能够保留各个数据点的特征信息，其中新生成的各维叫主成分，针对上述数据，我们将四维数据减低到三维后，可以通过3D散点图展示出来。
scikit-learn 库通过fit_transform()函数来降维，再新的数据展示出来。
Can you write Python code for it?

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn import datasets

iris = datasets.load_iris()
x = iris.data[:,1]
y = iris.data[:,2]
species = iris.target
pca = PCA(n_components=3)
x_reduced = pca.fit_transform(iris.data)
print 'Feature Convariance Matrix:'
print pca.get_covariance()
# draw 3d
fig = plt.figure()
ax = Axes3D(fig)
ax.set_title('Iris Dataset by PCA',size=14)
ax.scatter(x_reduced[:,0],x_reduced[:,1],x_reduced[:,2],c=species)
ax.set_xlabel('first eigenvector')
ax.set_ylabel('second eigenvector')
ax.set_zlabel('third eigenvector')
ax.w_xaxis.set_ticklabels(())
ax.w_yaxis.set_ticklabels(())
ax.w_zaxis.set_ticklabels(())
plt.show()


I am working on the file "Data_Analytics_in_Action/scikit_learning.ipynb".
The first task is:
6 支持向量机
SVM（support vector machines) 是一系列机器学习方法， 可分为SVR（support vector regression，支持向量回归）和SVC（support vector classification，支持向量分类）
支持向量机分类
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm
x = np.array([[1,3],[1,2],[1,1.5],[1.5,2],[2,3],[2.5,1.5],
             [2,1],[3,1],[3,2],[3.5,1],[3.5,3]])
y = [0]*6 + [1]*5
plt.scatter(x[:,0],x[:,1],c=y,s=50,alpha=0.9)
plt.show()

svc = svm.SVC(kernel='linear').fit(x,y)
X, Y = np.mgrid[0:4:200j,0:4:200j]
Z = svc.decision_function(np.c_[X.ravel(),Y.ravel()])
Z = Z.reshape(X.shape)
plt.contourf(X,Y,Z>0,alpha=0.4)
plt.contour(X,Y,Z,color=['k'],linestyle=['-'],level=[0])
plt.scatter(x[:,0],x[:,1],c=y,s=50,alpha=0.9)
plt.show()

svc.predict([1.5,2.5])

svc.predict([2.5,1])


I am working on the file "Data_Analytics_in_Action/scikit_learning.ipynb".
The first task is:
Iris数据
SVC 将数据分类成3个数据，使用非线性决策边界。
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets
iris =datasets.load_iris()
x = iris.data[:,:2]
y = iris.target
h = 0.5
svc = svm.SVC(kernel='linear',C=1.0).fit(x,y)
x_min,x_max = x[:,0].min() - .5, x[:,0].max() +.5
y_min,y_max = x[:,1].min() - .5, x[:,1].max() +.5
h = 0.05
X, Y = np.meshgrid(np.arange(x_min,x_max,h), np.arange(y_min,y_max,h))
Z = svc.predict(np.c_[X.ravel(),Y.ravel()])
Z = Z.reshape(X.shape)
plt.contourf(X,Y,Z,alpha=0.4)
plt.contour(X,Y,Z,colors='k')
plt.scatter(x[:,0],x[:,1],c=y)
plt.show()

import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets
iris =datasets.load_iris()
x = iris.data[:,:2]
y = iris.target
h = 0.5
svc = svm.SVC(kernel='poly',C=1.0,degree=3).fit(x,y)
x_min,x_max = x[:,0].min() - .5, x[:,0].max() +.5
y_min,y_max = x[:,1].min() - .5, x[:,1].max() +.5
h = 0.05
X, Y = np.meshgrid(np.arange(x_min,x_max,h), np.arange(y_min,y_max,h))
Z = svc.predict(np.c_[X.ravel(),Y.ravel()])
Z = Z.reshape(X.shape)
plt.contourf(X,Y,Z,alpha=0.4)
plt.contour(X,Y,Z,colors='k')
plt.scatter(x[:,0],x[:,1],c=y)
plt.show()

import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets
iris =datasets.load_iris()
x = iris.data[:,:2]
y = iris.target
h = 0.5
svc = svm.SVC(kernel='rbf',C=1.0,gamma=3).fit(x,y)
x_min,x_max = x[:,0].min() - .5, x[:,0].max() +.5
y_min,y_max = x[:,1].min() - .5, x[:,1].max() +.5
h = 0.05
X, Y = np.meshgrid(np.arange(x_min,x_max,h), np.arange(y_min,y_max,h))
Z = svc.predict(np.c_[X.ravel(),Y.ravel()])
Z = Z.reshape(X.shape)
plt.contourf(X,Y,Z,alpha=0.4)
plt.contour(X,Y,Z,colors='k')
plt.scatter(x[:,0],x[:,1],c=y)
plt.show()


I am working on the file "Data_Analytics_in_Action/scikit_learning.ipynb".
The first task is:
正确率为96%，再绘制出分界面
Can you write Python code for it?

from matplotlib.colors import ListedColormap
x = iris.data[:,:2]
y = iris.target
x_min,x_max = x[:,0].min() -.5, x[:,0].max() + .5
y_min,y_max = x[:,1].min() -.5, x[:,1].max() + .5
cmap_light = ListedColormap(['#AAAAFF','#AAFFAA','#FFAAAA'])
h = .02
xx,yy = np.meshgrid(np.arange(x_min,x_max,h),np.arange(y_min,y_max,h))
knn = KNeighborsClassifier()
knn.fit(x,y)
Z = knn.predict(np.c_[xx.ravel(),yy.ravel()])
Z = Z.reshape(xx.shape)
plt.figure()
plt.pcolormesh(xx,yy,Z,cmap=cmap_light)
plt.scatter(x[:,0],x[:,1],c=y)
plt.xlim(xx.min(),xx.max())
plt.ylim(yy.min(),yy.max())
plt.show()

from matplotlib.colors import ListedColormap
x = iris.data[:,2:4]
y = iris.target
x_min,x_max = x[:,0].min() -.5, x[:,0].max() + .5
y_min,y_max = x[:,1].min() -.5, x[:,1].max() + .5
cmap_light = ListedColormap(['#AAAAFF','#AAFFAA','#FFAAAA'])
h = .02
xx,yy = np.meshgrid(np.arange(x_min,x_max,h),np.arange(y_min,y_max,h))
knn = KNeighborsClassifier()
knn.fit(x,y)
Z = knn.predict(np.c_[xx.ravel(),yy.ravel()])
Z = Z.reshape(xx.shape)
plt.figure()
plt.pcolormesh(xx,yy,Z,cmap=cmap_light)
plt.scatter(x[:,0],x[:,1],c=y)
plt.xlim(xx.min(),xx.max())
plt.ylim(yy.min(),yy.max())
plt.show()


I am working on the file "Data_Analytics_in_Action/scikit_learning.ipynb".
The first task is:
每一个花卉的种类，访问 iris 的 target 属性
Can you write Python code for it?


I am working on the file "Data_Analytics_in_Action/scikit_learning.ipynb".
The first task is:
直方图统计
Can you write Python code for it?

groups = list(frame.groupby('kinds'))
current_group = groups[0]
petal_length = current_group[1]['petal_length']
petal_width = current_group[1]['petal_width']
sepal_length = current_group[1]['sepal_length']
sepal_width = current_group[1]['sepal_width']
plt.figure()
plt.suptitle(current_group[0]);
plt.subplot(221)
plt.hist(petal_length,color='r')
plt.title('petal_length')
plt.subplot(222)
plt.hist(petal_width,color='g')
plt.title('petal_width')
plt.subplot(223)
plt.hist(sepal_length,color='b')
plt.xlabel('sepal_length')
plt.subplot(224)
plt.hist(sepal_width,color='y')
plt.xlabel('sepal_width')
plt.show()

groups = list(frame.groupby('kinds'))
current_group = groups[1]
petal_length = current_group[1]['petal_length']
petal_width = current_group[1]['petal_width']
sepal_length = current_group[1]['sepal_length']
sepal_width = current_group[1]['sepal_width']
plt.figure()
plt.suptitle(current_group[0]);
plt.subplot(221)
plt.hist(petal_length,color='r')
plt.title('petal_length')
plt.subplot(222)
plt.hist(petal_width,color='g')
plt.title('petal_width')
plt.subplot(223)
plt.hist(sepal_length,color='b')
plt.xlabel('sepal_length')
plt.subplot(224)
plt.hist(sepal_width,color='y')
plt.xlabel('sepal_width')
plt.show()

groups = list(frame.groupby('kinds'))
current_group = groups[2]
petal_length = current_group[1]['petal_length']
petal_width = current_group[1]['petal_width']
sepal_length = current_group[1]['sepal_length']
sepal_width = current_group[1]['sepal_width']
plt.figure()
plt.suptitle(current_group[0]);
plt.subplot(221)
plt.hist(petal_length,color='r')
plt.title('petal_length')
plt.subplot(222)
plt.hist(petal_width,color='g')
plt.title('petal_width')
plt.subplot(223)
plt.hist(sepal_length,color='b')
plt.xlabel('sepal_length')
plt.subplot(224)
plt.hist(sepal_width,color='y')
plt.xlabel('sepal_width')
plt.show()


I am working on the file "Data_Inspection.ipynb".
The first task is:
Advanced Grid Inspection
If we want to examine grids only at a given level, we can!  Not only that, but we can load data and take a look at various fields.
This section can be skipped!
Can you write Python code for it?

gs = ds.index.select_grids(ds.index.max_level)

g2 = gs[0]
print (g2)
print (g2.Parent)
print (g2.get_global_startindex())

g2["density"][:,:,0]

print ((g2.Parent.child_mask == 0).sum() * 8)
print (g2.ActiveDimensions.prod())

for f in ds.field_list:
    fv = g[f]
    if fv.size == 0: continue
    print (f, fv.min(), fv.max())


I am working on the file "Data_Inspection.ipynb".
The first task is:
But, you may have to access information about individual grid objects!  Each grid object mediates accessing data from the disk and has a number of attributes that tell you about it.  The index (ds.index here) has an attribute grids which is all of the grid objects.
Can you write Python code for it?

ds.index.grids[1]

g = ds.index.grids[1]
print(g)


I am working on the file "Data_Inspection.ipynb".
The first task is:
Examining Data in Regions
yt provides data object selectors.  In subsequent notebooks we'll examine these in more detail, but we can select a sphere of data and perform a number of operations on it.  yt makes it easy to operate on fluid fields in an object in bulk, but you can also examine individual field values.
This creates a sphere selector positioned at the most dense point in the simulation that has a radius of 10 kpc.
Can you write Python code for it?


I am working on the file "Data_Inspection.ipynb".
The first task is:
Grids have dimensions, extents, level, and even a list of Child grids.
Can you write Python code for it?

g.ActiveDimensions

g.LeftEdge, g.RightEdge

g.Level

g.Children


I am working on the file "Data_Inspection.ipynb".
The first task is:
Let's look at the total mass.  This is how you call a given quantity.  yt calls these "Derived Quantities".  We'll talk about a few in a later notebook.
Can you write Python code for it?


I am working on the file "Data_Inspection.ipynb".
The first task is:
Now that we've loaded yt, we can load up some data.  Let's load the IsolatedGalaxy dataset.
Can you write Python code for it?


I am working on the file "Data_Inspection.ipynb".
The first task is:
We can calculate a bunch of bulk quantities.  Here's that list, but there's a list in the docs, too!
Can you write Python code for it?


I am working on the file "Data_Inspection.ipynb".
The first task is:
yt can also convert this into various units:
Can you write Python code for it?

print (ds.domain_width.in_units("kpc"))
print (ds.domain_width.in_units("au"))
print (ds.domain_width.in_units("mile"))


I am working on the file "Data_Inspection.ipynb".
The first task is:
yt can also tell you the fields it found on disk:
Can you write Python code for it?


I am working on the file "Data_Mining/Local_outlier_factor.ipynb".
The first task is:
局部异常因子计算出每个点的局部密度，通过它与K最近邻的点的距离来评估点的局部密度，并与邻居的密度进行比较，以此找出异常点--异常点比邻居的密度要低得多
为了理解LOF,先了解一些术语的定义
* 对象P的K距离：对象P与它第K个最近邻的距离，K是算法的参数
 P的K距离邻居：到P的距离小于或等于P到第K个最邻近的距离的所有对象的集合Q
* 从P到Q的可达距离：P与它的第K个最近邻的距离和P和Q之间的距离中的最大者。
 P的局部可达密度（local Reachability Density of P）：K距离邻居和K与其邻居的可达距离之和的比值
* P的局部异常因子（Local Outlier Factor of P):P与它的K最近邻的局部可达性的比值的平均值
Can you write Python code for it?

# 获取点两两之间的距离pairwise_distance
distance = 'manhattan'
from sklearn.metrics import pairwise_distances
dist = pairwise_distances(instance,metric=distance)

print dist

# 计算K距离，使用heapq来获得K最近邻
k = 2
# 计算K距离
import heapq
# k_distance的值是tuple
k_distance = defaultdict(tuple)
# 对每个点计算
for i in range(instance.shape[0]):
    # 获取它与所有其点之间的距离
    distances = dist[i].tolist()
    # 获得K最近邻
    ksmallest = heapq.nsmallest(k+1,distances)[1:][k-1]
    # 获取索引号
    ksmallest_idx = distances.index(ksmallest)
    # 记录下每个点到第K个最近邻以及到它的距离
    k_distance[i]=(ksmallest,ksmallest_idx)

# 计算K距离邻居
def all_indices(value,inlist):
    out_indices = []
    idx = -1
    while True:
        try:
            idx = inlist.index(value,idx+1)
            out_indices.append(idx)
        except ValueError:
            break
    return out_indices

# 计算K距离邻居
k_distance_neig = defaultdict(list)
for i in range(instance.shape[0]):
    # 获得它到所有邻居点的距离
    distances = dist[i].tolist()
    print 'k distance neighbourhood',i
    print distances
    # 获得从第1到第k的最近邻
    ksmallest = heapq.nsmallest(k+1,distances)[1:]
    print ksmallest
    ksmallest_set = set(ksmallest)
    print ksmallest_set
    ksmallest_idx = []
    # 获取k里最小的元素的索引号
    for x in ksmallest_set:
        ksmallest_idx.append(all_indices(x,distances))
    # 将列表的列表转换为列表
    ksmallest_idx = [item for sublist in ksmallest_idx for item in sublist]
    # 对每个点保存
    k_distance_neig[i].extend(zip(ksmallest,ksmallest_idx))

print k_distance_neig

# 计算可达距离和LRD
# 局部可达密度
local_reach_density = defaultdict(float)
for i in range(instance.shape[0]):
    # LRD分子，k距离邻居的个数
    no_neighbours = len(k_distance_neig[i])
    denom_sum = 0
    # 可达距离求和
    for neigh in k_distance_neig[i]:
        denom_sum += max(k_distance[neigh[1]][0],neigh[0])
    local_reach_density[i] = no_neighbours/(1.0*denom_sum)

# 计算LOF
lof_list = []
for i in range(instance.shape[0]):
    lrd_sum = 0
    rdist_sum = 0
    for neigh in k_distance_neig[i]:
        lrd_sum +=local_reach_density[neigh[1]]
        rdist_sum +=...


I am working on the file "Data_Ninja_Integration_Hadoop_Spark_Notebook.ipynb".
The first task is:
Accessing the Data Ninja services
Please sign-up at https://market.mashape.com/dataninja/smart-content and obtain your free Data Ninja API key. We will access the Smart Content service to analyze the semantic content of each article obtained in the previous step. The Smart Content service is based on our pre-built knowledge graph database.
Alternatively, you can use the Amazon Web Serivices API Gateway to access our services (using your AWS account): https://auth.dataninja.net/cart
Can you write Python code for it?

import json

with open('mashape_key.txt', 'r') as keyfile:
   mashape_key = keyfile.read().rstrip()

# Please add your own Data Ninja API Mashape key here -->
# mashape_key = <your-mashape-key>

smartcontent_url = 'https://smartcontent.dataninja.net/smartcontent/tag'
headers = {'Content-Type': 'application/json',
           'Accept': 'application/json',
           'X-Mashape-User': 'Newsbot',
           'X-Mashape-Key': mashape_key}

# If you are using AWS API Gateway, please add the X-API-Key: <your-AWS-key>
# in place of 'X-Mashape-Key': mashape_key and use the following link to access
# the service: https://api.dataninja.net/smartcontent/tag

def fetch_smartcontent(link):
    payload = {'url': link, 'max_size': 10}
    response = requests.post(smartcontent_url, headers=headers, data=json.dumps(payload))
    return response.json()


I am working on the file "Data_Ninja_Integration_Hadoop_Spark_Notebook.ipynb".
The first task is:
Apache Spark Integration
We will take advantage of the many data formats Apache Spark natively supports. Recall that we stored the Concepts and Categories returned from Data Ninja in HDFS in the previous step. Here, we will show how to use PySpark to read the data into a Spark Dataframe and perform some simple aggregations to find the top-n Concepts and Categories.
Can you write Python code for it?

from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext

conf = SparkConf().setAppName('dataninja-pyspark')
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)

def dndemo_spark():
    # A JSON dataset in HDFS.
    # The path can be either a single text file or a directory storing text files.
    concepts = sqlContext.read.json(hdfs_location + os.path.sep + 'concepts')
    concepts.printSchema()
    concepts.registerTempTable('concepts')

    categories = sqlContext.read.json(hdfs_location + os.path.sep + 'categories')
    categories.printSchema()
    categories.registerTempTable('categories')

# Run this only once!
# You can only have once SparkContext and SqlContext
dndemo_spark()

count = sqlContext.sql('SELECT count(*) as num_concepts FROM concepts')
print count.show()

count = sqlContext.sql('SELECT count(*) as num_categories FROM categories')
print count.show()

trending_con = sqlContext.sql('SELECT concept_title, sum(score) as total_score ' +
                              'FROM concepts GROUP BY concept_title ORDER BY total_score desc')

trending_cat = sqlContext.sql('SELECT category_title, sum(score) as total_score ' +
                              'FROM categories GROUP BY category_title ORDER BY total_score desc')

# print trending.show()


I am working on the file "Data_Ninja_Integration_Hadoop_Spark_Notebook.ipynb".
The first task is:
Article Text Extraction
Built into our Smart Content Service is the ability to extract the main text from a web page using machine learning techniques. Here is an example:
Can you write Python code for it?

# Dispay the extracted text from Smart Content
print data['text']


I am working on the file "Data_Ninja_Integration_Hadoop_Spark_Notebook.ipynb".
The first task is:
Collect and print the set of links
Can you write Python code for it?

import os

hdfs_location = '/Users/Projects/Current/Notebook'

linkset = set()
links = fetch_links('news.google.com')

# Collect the article links applying the URL filters
for link in links:
    href = link.get('href')
    if str(href).startswith(prefix) and include_url(str(href)):
        linkset.add(link.get('href').strip())
        # print str(href)

# Store links in HDFS
outfile = open(hdfs_location + os.path.sep + 'links' + os.path.sep + 'links.txt', "wb")
outfile.write("\n".join(linkset))

print 'Links harvested: ', str(len(linkset))

# Take 100 links for the demo
links100 = list(linkset)[:100]


I am working on the file "Data_Ninja_Integration_Hadoop_Spark_Notebook.ipynb".
The first task is:
Fetch Smart Content
The Smart Content service analyzes the text to produce concepts, categories, keywords and sentiments in JSON output format. Here is an example:
http://www.macrumors.com/roundup/macbook-pro/
Can you write Python code for it?

data = fetch_smartcontent('http://www.macrumors.com/roundup/macbook-pro/')

# Display the JSON output from Smart Content
print json.dumps(data, indent=4)


I am working on the file "Data_Ninja_Integration_Hadoop_Spark_Notebook.ipynb".
The first task is:
Fetch Smart Content for a set of links
Now we will extract the Smart Content for the list of URLs we obtained from Google News earlier. We will specifically prepare a list of extracted text for topic clustering in the next step.
Can you write Python code for it?

import json

# Call the Smart Content service and collect the article text into a list
documents = []

# Create a MongoDB connection
db = connect_to_db()

con_index = 0
cat_index = 0
for link in linkset:
    data = fetch_smartcontent(link)
    if 'text' in data and len(data['text']) > 100:
        documents.append(data['text'])
        doc_id = write_to_db(json.dumps(data), db)
        if 'concept_list' in data:
            for concept in data['concept_list']:
                write_to_hdfs(json.dumps(concept), hdfs_location + os.path.sep + 'concepts',
                              'concept_' + str(con_index) + '.json')
                con_index += 1
        if 'category_list' in data:
            for category in data['category_list']:
                write_to_hdfs(json.dumps(category), hdfs_location + os.path.sep + 'categories',
                              'category_' + str(cat_index) + '.json')
                cat_index += 1

print 'Documents in collection: ', str(len(documents))


I am working on the file "Data_Ninja_Integration_Hadoop_Spark_Notebook.ipynb".
The first task is:
Store Smart Content output in MongoDB
Using the PyMongo client created earlier, we will show how to save the Data Ninja
output to a MongoDB collection.
Can you write Python code for it?

def write_to_db(data, db):
    return db.insert_one(json_util.loads(data)).inserted_id

def write_to_hdfs(data, location, filename):
    outname = location + os.path.sep + filename
    outfile = open(outname, 'w')
    outfile.write(data)
    outfile.close()
    # return hdfs.write(data, location)


I am working on the file "Data_Ninja_Integration_Hadoop_Spark_Notebook.ipynb".
The first task is:
Visualizing the results
We will convert the final results from Spark Dataframe to a Pandas Dataframe and visualize the trending Concepts and Categories.
Can you write Python code for it?

from IPython.display import display, HTML
from tabulate import tabulate
import pandas as pd

display(trending_con)
display(trending_cat)

df_con = trending_con.toPandas().head(n=40)
df_cat = trending_cat.toPandas().head(n=40)

print 'Top 40 trending concepts:'
print tabulate(df_con)

print 'Top 40 trending categories'
print tabulate(df_cat)

import matplotlib.pyplot as plt
import matplotlib
matplotlib.style.use('ggplot')
%matplotlib inline

df_con.plot(x='concept_title', y='total_score', kind='bar', title='Trending Concepts', color='green', figsize=(20,10))
df_cat.plot(x='category_title', y='total_score', kind='bar', title='Trending Categories', color='orange', figsize=(20,10))


I am working on the file "Data_Preprocessing/Collaboration-appLoan_DataProcessing.ipynb".
The first task is:
1. Data Understanding -- Selected Decriptive Analysis
Can you write Python code for it?


I am working on the file "Data_Preprocessing/Collaboration-appLoan_DataProcessing.ipynb".
The first task is:
2a. Convert data type  on certain columns and create dummies
Nehal
Can you write Python code for it?

# function to create dummies
def create_dummies(column_name,df):
    temp=pd.get_dummies(df[column_name],prefix=column_name)
    df=pd.concat([df,temp],axis=1)
    return df

dummy_list=['emp_length','home_ownership','purpose','sub_grade']
for col in dummy_list:
    df2=create_dummies(col,df2)
for col in dummy_list:
    df2=df2.drop(col,1)

temp=df2['int_rate'].astype(str).str.replace('%', '').replace(' ','').astype(float)
df2=df2.drop('int_rate',1)
df2=pd.concat([df2,temp],axis=1)
df2=df2.drop('loan_status',1)

for col in df2.columns:
    print((df2[col].dtype))


I am working on the file "Data_Preprocessing/Collaboration-appLoan_DataProcessing.ipynb".
The first task is:
3a. Check and remove outliers (methods: MAD)
Can you write Python code for it?

df2.shape

df2['loan_amnt'][sorted(np.random.randint(0, high=10, size=5))]

# Reference:
# http://stackoverflow.com/questions/22354094/pythonic-way-of-detecting-outliers-in-one-dimensional-observation-data

def main(df, col, thres):
    outliers_all = []
    ind = sorted(np.random.randint(0, high=len(df), size=5000)) # randomly pick instances from the dataframe
    #select data from our dataframe
    x = df[col][ind]
    num = len(ind)
    outliers = plot(x, col, num, thres) # append all the outliers in the list
    pl.show()
    return outliers

def mad_based_outlier(points, thresh):
    if len(points.shape) == 1:
        points = points[:,None]
    median = np.median(points, axis=0)
    diff = np.sum((points - median)**2, axis=-1)
    diff = np.sqrt(diff)
    med_abs_deviation = np.median(diff)

    modified_z_score = 0.6745 * diff / med_abs_deviation

    return modified_z_score > thresh

def plot(x, col, num, thres):
    fig, ax = pl.subplots(nrows=1, figsize=(10, 3))
    sns.distplot(x, ax=ax, rug=True, hist=False)
    outliers = np.asarray(x[mad_based_outlier(x, thres)])
    ax.plot(outliers, np.zeros_like(outliers), 'ro', clip_on=False)

    fig.suptitle('MAD-based Outlier Tests with selected {} values'.format(col, num, size=20))
    return outliers

### Find outliers
##
boundries = []
outliers_loan = main(df2, 'loan_amnt', thres=2.2)

boundries.append(outliers_loan.min())

## annual income
outliers_inc = main(df2, 'annual_inc', 8)

boundries.append(outliers_inc.min())

## For total current balance of bank accounts
outliers_bal = main(df2, 'tot_cur_bal', 8)

boundries.append(outliers_bal.min())

columns = ['loan_amnt', 'annual_inc', 'tot_cur_bal']

for col, bound in zip(columns, boundries):
    print ('Lower bound of detected Outliers for {}: {}'.format(col, bound))
    # Use the outlier boundry to "regularize" the dataframe
    df2_r = df2[df2[col] <= bound]


I am working on the file "Data_Preprocessing/Collaboration-appLoan_DataProcessing.ipynb".
The first task is:
4a. Remove or replace missing values of certain columns
Can you write Python code for it?

# df2_r.info()

df2_r.shape

#### Fill NaN with "none"??? ####
#df_filled = df2.fillna(value='none')
#df_filled.head(3)
df2_r = df2_r.dropna()
print (len(df2_r))


I am working on the file "Data_Preprocessing/LendingClub_DataExploratory.ipynb".
The first task is:
Columns Interested
loan_status -- Current status of the loan

loan_amnt -- The listed amount of the loan applied for by the borrower. If at some point in time, the credit        department reduces the loan amount, then it will be reflected in this value.

int_rate -- interest rate of the loan

grade -- LC assigned loan grade

sub_grade -- LC assigned sub loan grade

purpose -- A category provided by the borrower for the loan request.
 -- dummy
annual_inc -- The self-reported annual income provided by the borrower during registration.

emp_length -- Employment length in years. Possible values are between 0 and 10 where 0 means less than one year and 10 means ten or more years.
 -- dummie
fico_range_low
fico_range_high
home_ownership -- The home ownership status provided by the borrower during registration or obtained from the credit report. Our values are: RENT, OWN, MORTGAGE, OTHER

tot_cur_bal -- Total current balance of all accounts
num_actv_bc_tl -- number of active bank accounts

  (avg_cur_bal -- average current balance of all accounts )

mort_acc -- number of mortgage accounts

num_actv_rev_tl -- Number of currently active revolving trades

dti -- A ratio calculated using the borrower’s total monthly debt payments on the total debt obligations, excluding mortgage and the requested LC loan, divided by the borrower’s self-reported monthly income.
pub_rec_bankruptcies - Number of public record bankruptcies

delinq_amnt --

title --
mths_since_last_delinq -- The number of months since the borrower's last delinquency.

mths_since_recent_revol_delinq -- Months since most recent revolving delinquency.

total_cu_tl -- Number of finance trades

last_credit_pull_d -- The most recent month LC pulled credit for this loan

Can you write Python code for it?

## 2015
df_app_2015 = pd.read_csv('data/LoanStats3d_securev1.csv.zip', compression='zip', low_memory=False,\
                         header=1)

df_app_2015.loan_status.unique()

df_app_2015.head(5)

df_app_2015['delinq_amnt'].unique()

df_app_2015.info(max_cols=111)

df_app_2015.groupby('title').loan_amnt.mean()

df_app_2015.groupby('purpose').loan_amnt.mean()

df_app_2015['emp_length'].unique()


I am working on the file "Data_Preprocessing/LendingClub_DataExploratory.ipynb".
The first task is:
Decriptive Analyss

Annual income distribution
Total loan amount groupby interest rate chunks
Average loan amount groupby grade
Average loan amount groupby
Can you write Python code for it?

## selected columns
df = df_app_2015.ix[:, ['loan_status','loan_amnt', 'int_rate', 'grade', 'sub_grade',\
                        'purpose',\
                        'annual_inc', 'emp_length', 'home_ownership',\
                        'fico_range_low','fico_range_high',\
                        'num_actv_bc_tl', 'tot_cur_bal', 'mort_acc','num_actv_rev_tl',\
                        'pub_rec_bankruptcies','dti' ]]

df.head(3)

len(df.dropna())

df.shape

df.loan_status.unique()

len(df[df['loan_status']=='Fully Paid'])

len(df[df['loan_status']=='Default'])

len(df[df['loan_status']=='Charged Off'])

len(df[df['loan_status']=='Late (31-120 days)'])

df.info()

df.loan_status.unique()

## Convert applicable fields to numeric (I only select "Interest Rate" to use for this analysis)
df.ix[:,'int_rate'] = df.ix[:,['int_rate']]\
                        .applymap(lambda e: pd.to_numeric(str(e).rstrip()[:-1], errors='coerce'))
df.info()

df = df.rename(columns={"int_rate": "int_rate(%)"})

df.head(3)

#len(df.dropna(thresh=  , axis=1).columns)

df.describe()

# 1. Loan Amount distribution
# # create plots and histogram to visualize total loan amounts
fig = pl.figure(figsize=(8,10))
ax1 = fig.add_subplot(211)
ax1.plot(range(len(df)), sorted(df.loan_amnt), '.', color='purple')
ax1.set_xlabel('Loan Applicant Count')
ax1.set_ylabel('Loan Amount ($)')
ax1.set_title('Fig 1a - Sorted Issued Loan Amount (2015)', size=15)

# all_ histogram
# pick upper bound 900 to exclude too large numbers
ax2 = fig.add_subplot(212)
ax2.hist(df.loan_amnt, range=(df.loan_amnt.min(), 36000), color='purple')
ax2.set_xlabel('Loan Amount -$', size=12)
ax2.set_ylabel('Counts',size=12)
ax2.set_title('Fig 1b - Sorted Issued Loan Amount (2015)', size=15)


I am working on the file "Data_Preprocessing/LendingClub_DataExploratory.ipynb".
The first task is:
Fig 2a and Fig 2b both show the sorted applicant annual income from low to high. The former indicates extreme values, and the latter plots only those values below the 75% quantile, which looks more sensible.<br/>
Fig 2c is a histogram showing the distribution of the applicants' income (below 75% quantile).
Obeservation
The most frequent annual income amounts of ths applicants are between $40,000 and below $60,000.
Can you write Python code for it?

4.600000e+04

# 3. Loan amount and Applicant Annual Income
# View all
pl.figure(figsize=(6,4))
pl.plot(df.annual_inc, df.loan_amnt, '.')
pl.ylim(0, 40000)
pl.xlim(0, 0.2e7)   # df.annual_inc.max()
pl.title('Fig 3a - Loan Amount VS Applicant Annual Income_all', size=15)
pl.ylabel('Loan Amount ($)', size=15)
pl.xlabel('Applicant Annual Income ($)', size=15)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
And here's an example of how to find the average hwy mpg for each class of vehicle in our dataset.
Can you write Python code for it?

HwyMpgByClass = []

for t in vehicleclass: # iterate over all the vehicle classes
    summpg = 0
    vclasscount = 0
    for d in mpg: # iterate over all dictionaries
        if d['class'] == t: # if the cylinder amount type matches,
            summpg += float(d['hwy']) # add the hwy mpg
            vclasscount += 1 # increment the count
    HwyMpgByClass.append((t, summpg / vclasscount)) # append the tuple ('class', 'avg mpg')

HwyMpgByClass.sort(key=lambda x: x[1])
HwyMpgByClass


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
And this is a slice starting from the 3rd element of the string and going all the way to the end.
Can you write Python code for it?

x[3:]

firstname = 'Christopher'
lastname = 'Brooks'

print(firstname + ' ' + lastname)
print(firstname*3)
print('Chris' in firstname)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Assign function add_numbers to variable a.
Can you write Python code for it?

def add_numbers(x,y):
    return x+y

a = add_numbers
a(1,2)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Copying Data
Be careful with copying and modifying arrays in NumPy!
r2 is a slice of r
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Create an array using repeating list (or see np.tile)
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Creating Arrays
Create a list and convert it to a numpy array
Can you write Python code for it?

mylist = [1, 2, 3]
x = np.array(mylist)
x


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Handy datetime attributes:
Can you write Python code for it?

dtnow.year, dtnow.month, dtnow.day, dtnow.hour, dtnow.minute, dtnow.second # get year, month, day, etc.from a datetime


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Here we are assigning all values in the array that are greater than 30 to the value of 30.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Here we are selecting all the rows up to (and not including) row 2, and all the columns up to (and not including) the last column.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Here's a more complex example where we are grouping the cars by number of cylinder, and finding the average cty mpg for each group.
Can you write Python code for it?

CtyMpgByCyl = []

for c in cylinders: # iterate over all the cylinder levels
    summpg = 0
    cyltypecount = 0
    for d in mpg: # iterate over all dictionaries
        if d['cyl'] == c: # if the cylinder level type matches,
            summpg += float(d['cty']) # add the cty mpg
            cyltypecount += 1 # increment the count
    CtyMpgByCyl.append((c, summpg / cyltypecount)) # append the tuple ('cylinder', 'avg mpg')

CtyMpgByCyl.sort(key=lambda x: x[0])
CtyMpgByCyl


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Here's an example of mapping the min function between two lists.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Indexing / Slicing
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Iterate by index:
Can you write Python code for it?

for i in range(len(test)):
    print(test[i])


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Iterate by row and index:
Can you write Python code for it?

for i, row in enumerate(test):
    print('row', i, 'is', row)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Iterate by row:
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Iterate over all of the items in the list:
Can you write Python code for it?

for name, email in x.items():
    print(name)
    print(email)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Iterate over all of the values:
Can you write Python code for it?

for email in x.values():
    print(email)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Iterating Over Arrays
Let's create a new 4 by 3 array of random numbers 0-9.
Can you write Python code for it?

test = np.random.randint(0, 10, (4,3))
test


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Let's iterate from 0 to 999 and return the even numbers.
Can you write Python code for it?

my_list = []
for number in range(0, 1000):
    if number % 2 == 0:
        my_list.append(number)
my_list


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Let's look at a multidimensional array.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Let's look at transposing arrays. Transposing permutes the dimensions of the array.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Lists are a mutable data structure.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Make sure the number of values you are unpacking matches the number of variables being assigned.
Can you write Python code for it?

x = ('Christopher', 'Brooks', 'brooksch@umich.edu', 'Ann Arbor')
fname, lname, email, location = x


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Make sure you convert objects to strings before concatenating.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Now let's iterate through the map object to see the values.
Can you write Python code for it?

for item in cheapest:
    print (item)

people = ['Dr. Christopher Brooks', 'Dr. Kevyn Collins-Thompson', 'Dr. VG Vinod Vydiswaran', 'Dr. Daniel Romero']

def split_title_and_name(person):
  title = person.split(' ')[0]
  lname = person.split(' ')[-1]
  return title +" "+ lname

list(map(split_title_and_name, people))


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Now the same thing but with list comprehension.
Can you write Python code for it?

my_list = [number for number in range(0,1000) if number % 2 == 0]
my_list

def times_tables():
    lst = []
    for i in range(10):
        for j in range (10):
            lst.append(i*j)
    return lst

times_tables() == [j*i for i in range(10) for j in range(10)]

lowercase = 'abcdefghijklmnopqrstuvwxyz'
digits = '0123456789'

correct_answer = [a+b+c+d for a in lowercase for b in lowercase for c in digits for d in digits]

correct_answer[0:100]


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Now when r_copy is modified, r will not be changed.
Can you write Python code for it?

r_copy[:] = 10
print(r_copy, '\n')
print(r)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Operations
Use +, -, *, / and ** to perform element wise addition, subtraction, multiplication, division and power.
Can you write Python code for it?

print(x + y) # elementwise addition     [1 2 3] + [4 5 6] = [5  7  9]
print(x - y) # elementwise subtraction  [1 2 3] - [4 5 6] = [-3 -3 -3]

print(x * y) # elementwise multiplication  [1 2 3] * [4 5 6] = [4  10  18]
print(x / y) # elementwise divison         [1 2 3] / [4 5 6] = [0.25  0.4  0.5]

print(x**2) # elementwise power  [1 2 3] ^2 =  [1 4 9]


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Or using the indexing operator:
Can you write Python code for it?

i=0
while( i != len(x) ):
    print(x[i])
    i = i + 1


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Python has a built in method for convenient string formatting.
Can you write Python code for it?

sales_record = {
'price': 3.24,
'num_items': 4,
'person': 'Chris'}

sales_statement = '{} bought {} item(s) at a price of {} each for a total of {}'

print(sales_statement.format(sales_record['person'],
                             sales_record['num_items'],
                             sales_record['price'],
                             sales_record['num_items']*sales_record['price']))


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Similarly this is how to find the average hwy fuel economy across all cars.
Can you write Python code for it?

sum(float(d['hwy']) for d in mpg) / len(mpg)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
The Python Programming Language: Dates and Times
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:

The Python Programming Language: Lambda and List Comprehensions

Here's an example of lambda that takes in three parameters and adds the first two.
Can you write Python code for it?

# Single function only
my_function = lambda a, b, c : a + b + c

my_function(1, 2, 3)

people = ['Dr. Christopher Brooks', 'Dr. Kevyn Collins-Thompson', 'Dr. VG Vinod Vydiswaran', 'Dr. Daniel Romero']

def split_title_and_name(person):
    return person.split()[0] + ' ' + person.split()[-1]

#option 1
for person in people:
    print(split_title_and_name(person) == (lambda x: x.split()[0] + ' ' + x.split()[-1])(person))

#option 2
list(map(split_title_and_name, people)) == list(map(lambda person: person.split()[0] + ' ' + person.split()[-1], people))


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
The Python Programming Language: More on Strings
Can you write Python code for it?

print("Chris" + 2)

print('Chris' + str(2))


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
The Python Programming Language: Numerical Python (NumPy)
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:

The Python Programming Language: Objects and map()

An example of a class in python:
Can you write Python code for it?

class Person:
    department = 'School of Information' #a class variable

    def set_name(self, new_name): #a method
        self.name = new_name
    def set_location(self, new_location):
        self.location = new_location

person = Person()
person.set_name('Christopher Brooks')
person.set_location('Ann Arbor, MI, USA')
print('{} live in {} and works in the department {}'.format(person.name, person.location, person.department))


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
The number of rows has swapped with the number of columns.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
The shape of array z is (2,3) before transposing.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
This is a slice from the beginning of the string and stopping before the 3rd element.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
This is how to find the average cty fuel economy across all cars. All values in the dictionaries are strings, so we need to convert to float.
Can you write Python code for it?

sum(float(d['cty']) for d in mpg) / len(mpg)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
This will return the last element of the string.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
This will return the slice starting from the 4th element from the end and stopping before the 2nd element from the end.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Use + to concatenate lists.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Use .T to get the transpose.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Use : to indicate a range. array[start:stop]
Leaving start or stop empty will default to the beginning/end of the array.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Use append to append an object to a list.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Use bracket notation to get the value at a specific index. Remember that indexing starts at 0.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Use hstack to stack arrays in sequence horizontally (column wise).
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Use negatives to count from the back.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Use the in operator to check if something is inside a list.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
Use vstack to stack arrays in sequence vertically (row wise).
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
We can also perform conditional indexing. Here we are selecting values from the array that are greater than 30. (Also see np.where)
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
You can unpack a sequence into different variables:
Can you write Python code for it?

x = ('Christopher', 'Brooks', 'brooksch@umich.edu')
fname, lname, email = x

fname

lname


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
add_numbers is a function that takes two numbers and adds them together.
Can you write Python code for it?

def add_numbers(x, y):
    return x + y

add_numbers(x, y)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
add_numbers updated to take an optional flag parameter.
Can you write Python code for it?

def add_numbers(x, y, z=None, flag=False):
    if (flag):
        print('Flag is true!')
    if (z==None):
        return x + y
    else:
        return x + y + z

print(add_numbers(1, 2, flag=True))


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
arange returns evenly spaced values within a given interval.
Can you write Python code for it?

n = np.arange(0, 30, 2) # start at 0 count up by 2, stop before 30
n


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
csv.Dictreader has read in each row of our csv file as a dictionary. len shows that our list is comprised of 234 dictionaries.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
diag extracts a diagonal or constructs a diagonal array.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
eye returns a 2-D array with ones on the diagonal and zeros elsewhere.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
linspace returns evenly spaced numbers over a specified interval.
Can you write Python code for it?

o = np.linspace(0, 4, 9) # return 9 evenly spaced values from 0 to 4
o


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
ones returns a new array of given shape and type, filled with ones.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
r has also been changed!
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
resize changes the shape and size of array in-place.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
split returns a list of all the words in a string, or a list split on a specific character.
Can you write Python code for it?

firstname = 'Christopher Arthur Hansen Brooks'.split(' ')[0] # [0] selects the first element of the list
lastname = 'Christopher Arthur Hansen Brooks'.split(' ')[-1] # [-1] selects the last element of the list
print(firstname)
print(lastname)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
time returns the current time in seconds since the Epoch. (January 1st, 1970)
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
timedelta is a duration expressing the difference between two dates.
Can you write Python code for it?

delta = dt.timedelta(days = 100) # create a timedelta of 100 days
delta

dt.date.today()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week1/Week+1.ipynb".
The first task is:
<br>
zeros returns a new array of given shape and type, filled with zeros.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week2/Assignment+2 (1).ipynb".
The first task is:
Question 0 (Example)
What is the first country in df?
This function should return a Series.
Can you write Python code for it?

# You should write your whole answer within the function provided. The autograder will call
# this function and compare the return value against the correct solution value
def answer_zero():
    # This function returns the row for Afghanistan, which is a Series object. The assignment
    # question description will tell you the general format the autograder is expecting
    return df.iloc[0]

# You can examine what your function returns by calling it in the cell. If you have questions
# about the assignment formats, check out the discussion forums for any FAQs
answer_zero()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week2/Assignment+2 (1).ipynb".
The first task is:
Question 1
Which country has won the most gold medals in summer games?
This function should return a single string value.
Can you write Python code for it?

def answer_one():
    answer = df.sort(['Gold'], ascending = False)
    return answer.index[0]

answer_one()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week2/Assignment+2 (1).ipynb".
The first task is:
Question 2
Which country had the biggest difference between their summer and winter gold medal counts?
This function should return a single string value.
Can you write Python code for it?

def answer_two():
    df['Gold_difference'] = df['Gold'] - df['Gold.1']
    answer = df.sort(['Gold_difference'], ascending = False)
    return answer.index[0]

answer_two()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week2/Assignment+2 (1).ipynb".
The first task is:
Question 4
Write a function that creates a Series called "Points" which is a weighted value where each gold medal (Gold.2) counts for 3 points, silver medals (Silver.2) for 2 points, and bronze medals (Bronze.2) for 1 point. The function should return only the column (a Series object) which you created.
This function should return a Series named Points of length 146
Can you write Python code for it?

def answer_four():
    df['Points'] = (df['Gold.2'] * 3) + (df['Silver.2'] * 2) + (df['Bronze.2'] * 1)
    return df['Points']

answer_four()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week2/Assignment+2 (1).ipynb".
The first task is:
Question 6
Only looking at the three most populous counties for each state, what are the three most populous states (in order of highest population to lowest population)? Use CENSUS2010POP.
This function should return a list of string values.
Can you write Python code for it?

def answer_six():
    ctydf = census_df[census_df['SUMLEV'] == 50]
    ctydfs = ctydf.sort(['CENSUS2010POP'], ascending = False)
    ctydfst3 = ctydfs.groupby('STNAME').head(3)
    ldf = ctydfst3.groupby(['STNAME']).sum()
    answer = ldf.sort(['CENSUS2010POP'], ascending = False)
    return answer.index[0:3].tolist()

answer_six()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week2/Assignment+2 (1).ipynb".
The first task is:
Question 8
In this datafile, the United States is broken up into four regions using the "REGION" column.
Create a query that finds the counties that belong to regions 1 or 2, whose name starts with 'Washington', and whose POPESTIMATE2015 was greater than their POPESTIMATE 2014.
This function should return a 5x2 DataFrame with the columns = ['STNAME', 'CTYNAME'] and the same index ID as the census_df (sorted ascending by index).
Can you write Python code for it?

def answer_eight():
    ctydf = census_df[(census_df['SUMLEV'] == 50) & (census_df['REGION'] < 3)]
    ctydfwas = ctydf.loc[(ctydf['CTYNAME'] == 'Washington County') & (ctydf['POPESTIMATE2015'] > ctydf['POPESTIMATE2014'])]
    columns_to_keep = ['STNAME',
                   'CTYNAME']
    ansdf = ctydfwas[columns_to_keep]
    return ansdf

answer_eight()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week2/Week+2.ipynb".
The first task is:
Dataframe Indexing and Loading
Can you write Python code for it?

costs = df['Cost']
costs

costs+=2
costs

df

!cat olympics.csv

df = pd.read_csv('olympics.csv')
df.head()

df = pd.read_csv('olympics.csv', index_col = 0, skiprows=1)
df.head()

df.columns

for col in df.columns:
    if col[:2]=='01':
        df.rename(columns={col:'Gold' + col[4:]}, inplace=True)
    if col[:2]=='02':
        df.rename(columns={col:'Silver' + col[4:]}, inplace=True)
    if col[:2]=='03':
        df.rename(columns={col:'Bronze' + col[4:]}, inplace=True)
    if col[:1]=='№':
        df.rename(columns={col:'#' + col[1:]}, inplace=True)

df.head()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week2/Week+2.ipynb".
The first task is:
Missing values
Can you write Python code for it?

df = pd.read_csv('log.csv')
df

df.fillna?

df = df.set_index('time')
df = df.sort_index()
df

df = df.reset_index()
df = df.set_index(['time', 'user'])
df

df = df.fillna(method='ffill')
df.head()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week2/Week+2.ipynb".
The first task is:
Querying a Series
Can you write Python code for it?

sports = {'Archery': 'Bhutan',
          'Golf': 'Scotland',
          'Sumo': 'Japan',
          'Taekwondo': 'South Korea'}
s = pd.Series(sports)
s

s.iloc[3]

s.loc['Golf']

s[3]

s['Golf']

sports = {99: 'Bhutan',
          100: 'Scotland',
          101: 'Japan',
          102: 'South Korea'}
s = pd.Series(sports)

s.iloc[0] #This won't call s.iloc[0] as one might expect, it generates an error instead

s = pd.Series([100.00, 120.00, 101.00, 3.00])
s

total = 0
for item in s:
    total+=item
print(total)

import numpy as np

total = np.sum(s)
print(total)

#this creates a big series of random numbers
s = pd.Series(np.random.randint(0,1000,10000))
s.head()

len(s)

%%timeit -n 100
summary = 0
for item in s:
    summary+=item

%%timeit -n 100
summary = np.sum(s)

s+=2 #adds two to each item in s using broadcasting
s.head()

for label, value in s.iteritems():
    s.set_value(label, value+2)
s.head()

%%timeit -n 10
s = pd.Series(np.random.randint(0,1000,10000))
for label, value in s.iteritems():
    s.loc[label]= value+2

%%timeit -n 10
s = pd.Series(np.random.randint(0,1000,10000))
s+=2

s = pd.Series([1, 2, 3])
s.loc['Animal'] = 'Bears'
s

original_sports = pd.Series({'Archery': 'Bhutan',
                             'Golf': 'Scotland',
                             'Sumo': 'Japan',
                             'Taekwondo': 'South Korea'})
cricket_loving_countries = pd.Series(['Australia',
                                      'Barbados',
                                      'Pakistan',
                                      'England'],
                                   index=['Cricket',
                                          'Cricket',
                                          'Cricket',
                                          'Cricket'])
all_countries = original_sports.append(cricket_loving_countries)

original_sports

cricket_loving_countries

all_countries

all_countries.loc['Cricket']


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week3/Assignment+3.ipynb".
The first task is:
<br>
Answer the following questions in the context of only the top 15 countries by Scimagojr Rank (aka the DataFrame returned by answer_one())
Question 3 (6.6%)
What is the average GDP over the last 10 years for each country? (exclude missing values from this calculation.)
This function should return a Series named avgGDP with 15 countries and their average GDP sorted in descending order.
Can you write Python code for it?


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week3/Assignment+3.ipynb".
The first task is:
Question 10 (6.6%)
Create a new column with a 1 if the country's % Renewable value is at or above the median for all countries in the top 15, and a 0 if the country's % Renewable value is below the median.
This function should return a series named HighRenew whose index is the country name sorted in ascending order of rank.
Can you write Python code for it?

def answer_ten():
    import pandas as pd
    import numpy as np
    Top15 = answer_one()
    Top15['HighRenew'] = np.where(Top15['% Renewable'] >= (Top15['% Renewable'].median()), 1, 0)
    return Top15['HighRenew']

answer_ten()

#df = df.set_index('STNAME').groupby(level=0)['CENSUS2010POP'].agg({'avg': np.average})
#pd.cut(df['avg'],5, labels =['Tiny', 'Small', 'Medium', 'Large', 'Heouge'] )


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week3/Assignment+3.ipynb".
The first task is:
Question 12 (6.6%)
Cut % Renewable into 5 bins. Group Top15 by the Continent, as well as these new % Renewable bins. How many countries are in each of these groups?
This function should return a Series with a MultiIndex of Continent, then the bins for % Renewable. Do not include groups with no countries.
Can you write Python code for it?

def answer_twelve():
    Top15 = answer_one()
    return "ANSWER"


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week3/Assignment+3.ipynb".
The first task is:
Question 13 (6.6%)
Convert the Population Estimate series to a string with thousands separator (using commas). Do not round the results.
e.g. 317615384.61538464 -> 317,615,384.61538464
This function should return a Series PopEst whose index is the country name and whose values are the population estimate string.
Can you write Python code for it?

def answer_thirteen():
    Top15 = answer_one()
    Top15['PopEst'] = Top15['Energy Supply']/Top15['Energy Supply per Capita']
    answer = Top15['PopEst'].sort_values(ascending=False)
    PopEst = answer.map('{:,}'.format)
    return PopEst
#answer.astype('str')

answer_thirteen()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week3/Assignment+3.ipynb".
The first task is:
Question 2 (6.6%)
The previous question joined three datasets then reduced this to just the top 15 entries. When you joined the datasets, but before you reduced this to the top 15 items, how many entries did you lose?
This function should return a single number.
Can you write Python code for it?

%%HTML
<svg width="800" height="300">
  <circle cx="150" cy="180" r="80" fill-opacity="0.2" stroke="black" stroke-width="2" fill="blue" />
  <circle cx="200" cy="100" r="80" fill-opacity="0.2" stroke="black" stroke-width="2" fill="red" />
  <circle cx="100" cy="100" r="80" fill-opacity="0.2" stroke="black" stroke-width="2" fill="green" />
  <line x1="150" y1="125" x2="300" y2="150" stroke="black" stroke-width="2" fill="black" stroke-dasharray="5,3"/>
  <text  x="300" y="165" font-family="Verdana" font-size="35">Everything but this!</text>
</svg>

def answer_two():
    #227
    import pandas as pd
    import numpy as np
    energy = get_energy()
    GDP = get_gdp()
    ScimEn = get_ScimEn()
    base = pd.merge(ScimEn, energy, how='inner')
    answer = pd.merge(base, GDP, how='inner')
    base1 = pd.merge(ScimEn, energy, how='outer')
    answer1 = pd.merge(base, GDP, how='outer')
    return (len(answer1) - len(answer))


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week3/Assignment+3.ipynb".
The first task is:
Question 5 (6.6%)
What is the mean Energy Supply per Capita?
This function should return a single number.
Can you write Python code for it?

def answer_five():
    Top15 = answer_one()
    return Top15['Energy Supply per Capita'].mean()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week3/Assignment+3.ipynb".
The first task is:
Question 6 (6.6%)
What country has the maximum % Renewable and what is the percentage?
This function should return a tuple with the name of the country and the percentage.
Can you write Python code for it?

def answer_six():
    Top15 = answer_one()
    answer = (Top15['% Renewable'].idxmax(), Top15['% Renewable'].max())
    return answer
answer_six()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week3/Assignment+3.ipynb".
The first task is:
Question 7 (6.6%)
Create a new column that is the ratio of Self-Citations to Total Citations.
What is the maximum value for this new column, and what country has the highest ratio?
This function should return a tuple with the name of the country and the ratio.
Can you write Python code for it?

def answer_seven():
    Top15 = answer_one()
    Top15['Cita_ratio'] = Top15['Self-citations']/Top15['Citations']
    answer = (Top15['Cita_ratio'].idxmax(), Top15['Cita_ratio'].max())
    return answer
answer_seven()


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week3/Assignment+3.ipynb".
The first task is:
Question 9 (6.6%)
Create a column that estimates the number of citable documents per person.
What is the correlation between the number of citable documents per capita and the energy supply per capita? Use the .corr() method, (Pearson's correlation).
This function should return a single number.
(Optional: Use the built-in function plot9() to visualize the relationship between Energy Supply per Capita vs. Citable docs per Capita)
Can you write Python code for it?

def answer_nine():
    Top15 = answer_one()
    Top15['Pop_est'] = Top15['Energy Supply']/Top15['Energy Supply per Capita']
    Top15['Est_cite_doc'] = Top15['Citable documents']/Top15['Pop_est']
    answer = Top15.corr().iloc[8,-1]
    return answer
answer_nine()

#

#plot9() # Be sure to comment out plot9() before submitting the assignment!


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/0 Introduction to Data Science in Python/Week4/Week+4.ipynb".
The first task is:
Hypothesis Testing
Can you write Python code for it?

df = pd.read_csv('grades.csv')

df.head()

len(df)

early = df[df['assignment1_submission'] <= '2015-12-31']
late = df[df['assignment1_submission'] > '2015-12-31']

early.mean()

late.mean()

from scipy import stats
stats.ttest_ind?

stats.ttest_ind(early['assignment1_grade'], late['assignment1_grade'])

stats.ttest_ind(early['assignment2_grade'], late['assignment2_grade'])

stats.ttest_ind(early['assignment3_grade'], late['assignment3_grade'])


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/1 Applied Plotting, Charting & Data Representation/Week 2/Week2.ipynb".
The first task is:
Bar Charts
Can you write Python code for it?

plt.figure()
xvals = range(len(linear_data))
plt.bar(xvals, linear_data, width = 0.3)

new_xvals = []

# plot another set of bars, adjusting the new xvals to make up for the first set of bars plotted
for item in xvals:
    new_xvals.append(item+0.3)

plt.bar(new_xvals, exponential_data, width = 0.3 ,color='red')

from random import randint
linear_err = [randint(0,15) for x in range(len(linear_data))]

# This will plot a new set of bars with errorbars using the list of random error values
plt.bar(xvals, linear_data, width = 0.3, yerr=linear_err)

# stacked bar charts are also possible
plt.figure()
xvals = range(len(linear_data))
plt.bar(xvals, linear_data, width = 0.3, color='b')
plt.bar(xvals, exponential_data, width = 0.3, bottom=linear_data, color='r')

# or use barh for horizontal bar charts
plt.figure()
xvals = range(len(linear_data))
plt.barh(xvals, linear_data, height = 0.3, color='b')
plt.barh(xvals, exponential_data, height = 0.3, left=linear_data, color='r')


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/1 Applied Plotting, Charting & Data Representation/Week 2/Week2.ipynb".
The first task is:
Let's see how to make a plot without using the scripting layer.
Can you write Python code for it?

# First let's set the backend without using mpl.use() from the scripting layer
from matplotlib.backends.backend_agg import FigureCanvasAgg
from matplotlib.figure import Figure

# create a new figure
fig = Figure()

# associate fig with the backend
canvas = FigureCanvasAgg(fig)

# add a subplot to the fig
ax = fig.add_subplot(111)

# plot the point (3,2)
ax.plot(3, 2, '.')

# save the figure to test.png
# you can see this figure in your Jupyter workspace afterwards by going to
# https://hub.coursera-notebooks.org/
canvas.print_png('test.png')


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/1 Applied Plotting, Charting & Data Representation/Week 2/Week2.ipynb".
The first task is:
Let's try using pandas
Can you write Python code for it?

import pandas as pd

plt.figure()
observation_dates = np.arange('2017-01-01', '2017-01-09', dtype='datetime64[D]')
observation_dates = map(pd.to_datetime, observation_dates) # trying to plot a map will result in an error
plt.plot(observation_dates, linear_data, '-o',  observation_dates, exponential_data, '-o')

plt.figure()
observation_dates = np.arange('2017-01-01', '2017-01-09', dtype='datetime64[D]')
observation_dates = list(map(pd.to_datetime, observation_dates)) # convert the map to a list to get rid of the error
plt.plot(observation_dates, linear_data, '-o',  observation_dates, exponential_data, '-o')

x = plt.gca().xaxis

# rotate the tick labels for the x axis
for item in x.get_ticklabels():
    item.set_rotation(45)

# adjust the subplot so the text doesn't run off the image
plt.subplots_adjust(bottom=0.25)

ax = plt.gca()
ax.set_xlabel('Date')
ax.set_ylabel('Units')
ax.set_title('Exponential vs. Linear performance')

# you can add mathematical expressions in any text element
ax.set_title("Exponential ($x^2$) vs. Linear ($x$) performance")


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/1 Applied Plotting, Charting & Data Representation/Week 2/Week2.ipynb".
The first task is:
Line Plots
Can you write Python code for it?

import numpy as np

linear_data = np.array([1,2,3,4,5,6,7,8])
exponential_data = linear_data**2

plt.figure()
# plot the linear data and the exponential data
plt.plot(linear_data, '-o', exponential_data, '-o')

# plot another series with a dashed red line
plt.plot([22,44,55], '--r')

plt.xlabel('Some data')
plt.ylabel('Some other data')
plt.title('A title')
# add a legend with legend entries (because we didn't have labels when we plotted the data series)
plt.legend(['Baseline', 'Competition', 'Us'])

# fill the area between the linear data and exponential data
plt.gca().fill_between(range(len(linear_data)),
                       linear_data, exponential_data,
                       facecolor='blue',
                       alpha=0.25)


I am working on the file "Data_Science_Course/Michigan Data Analysis Course/1 Applied Plotting, Charting & Data Representation/Week 2/Week2.ipynb".
The first task is:
We can use html cell magic to display the image.
Can you write Python code for it?

%%html
<img src='test.png' />

# create a new figure
plt.figure()

# plot the point (3,2) using the circle marker
plt.plot(3, 2, 'o')

# get the current axes
ax = plt.gca()

# Set axis properties [xmin, xmax, ymin, ymax]
ax.axis([0,6,0,10])

# create a new figure
plt.figure()

# plot the point (1.5, 1.5) using the circle marker
plt.plot(1.5, 1.5, 'o')
# plot the point (2, 2) using the circle marker
plt.plot(2, 2, '*')
# plot the point (2.5, 2.5) using the circle marker
plt.plot(2.5, 2.5, 'o')

# get current axes
ax = plt.gca()
# get all the child objects the axes contains
ax.get_children()


I am working on the file "Data_Story_1/Adv_vs_Trade.ipynb".
The first task is:
*TV avd has an impact on sell.
Can you write Python code for it?

# Plot the data and fitted line
x_prime = pd.DataFrame({'TV': np.linspace(adv.TV.min(),
                                          adv.TV.max(),
                                          100)})
y_hat = est_m.predict(x_prime)

plt.xlabel("TV")
plt.ylabel("Sales")
plt.title("Example of Heteroskedasticity")
plt.scatter(adv.TV, adv.Sales, alpha=0.3)
plt.plot(x_prime, y_hat, 'r', linewidth=2, alpha=0.9)

# View the residuals
plt.figure()
plt.scatter(est_m.predict(adv), est_m.resid, alpha=0.3)
plt.xlabel("Predicted Sales")
plt.ylabel("Residuals")


I am working on the file "Data_Story_1/Adv_vs_Trade.ipynb".
The first task is:
Determine if TV advertisement has any impact on sell.
Can you write Python code for it?

est_m=smf.ols(formula='Sales ~ TV', data=adv).fit()

est_m.summary()


I am working on the file "Data_Story_1/Adv_vs_Trade.ipynb".
The first task is:
Question
-- While condidering linear regression between Sales and Newspaper Advertisement coefficient is +ve but when it comes with all medium it is negative. What does it mean?
Can you write Python code for it?

est_tv_radio = smf.ols(formula='Sales ~ TV + Radio', data=adv).fit()

est_tv_radio.summary()


I am working on the file "Data_Story_2/linear_regression.ipynb".
The first task is:
Working with Hours.csv

Read in the 'hour.csv' file
Run the regression with: cnt ~ temp + hum + workingday + hour + C(weathersit)
Evaluate the results, how does this compare with the day
Create a binary variable for rush hour defined by 6-9a & 4-6p
Run the regression again. Does this new variable improve the results?
Can you write Python code for it?

# Dataset: How many people rent capitol bikeshare bikes
bike_hour_dat = pd.read_csv("Data/hour.csv")
bike_hour_dat.head(2)

# Multiple regression
est_hour_m = smf.ols(formula='cnt ~ temp + hum + workingday + hr + weathersit', data=bike_hour_dat).fit()
est_hour_m.summary()

est_hour_m.rsquared_adj

est_m.rsquared_adj


I am working on the file "Data_Story_2/logistic_regression.ipynb".
The first task is:
Conclusion

Balace is more closely related to the default rate.

=============================================
            Logistic Regreassion
=============================================
Can you write Python code for it?

# Logistic regression on the balance variable
bal = smf.logit(formula='default ~ balance', data=train).fit()
bal.summary()

# Get confidence interval.
bal.conf_int()

# odds ratio or Beta
# http://www.ats.ucla.edu/stat/mult_pkg/faq/general/odds_ratio.htm
np.exp(bal.params.balance)


I am working on the file "Data_Story_2/logistic_regression.ipynb".
The first task is:
Plot the same graph differently
Can you write Python code for it?

plt.scatter(train_default.balance, train_default.income, alpha=0.5, marker='o', c='g')
plt.scatter(train_no_default.balance, train_no_default.income, marker='+', edgecolors = 'r', facecolors = 'none')
plt.legend( ('Default', 'No Default'), loc='upper right')
plt.ylim([0,80000])
plt.xlim([0,3000])
plt.xlabel('Income')
plt.ylabel('Balance')


I am working on the file "Data_Story_2/logistic_regression.ipynb".
The first task is:
Split data into train and test data
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

train, test = train_test_split(default, test_size=0.33, random_state=1)

# Converting data into the dataframe
train = pd.DataFrame(data=train, columns=default.columns)
test = pd.DataFrame(data=test, columns=default.columns)

train.hist()

plt.scatter(train.income, train.balance)
plt.xlim(0, 75000)
plt.ylim(0, 3000)
plt.xlabel('Income')
plt.ylabel('Balance')
plt.title('Income vs Balance')

train.plot(x='balance', y='income', kind='scatter', alpha=0.3)
plt.ylim([0,80000])
plt.xlim([0,3000])

train_default=train[train.default==1]
train_no_default=train[train.default==0]

plt.figure()

plt.scatter(train_no_default.balance, train_no_default.income, alpha=0.5, marker='+', c='g')
plt.scatter(train_default.balance, train_default.income, marker='o', edgecolors = 'r', facecolors = 'none')
plt.legend( ('No Default', 'Default'), loc='upper right')
plt.ylim([0,80000])
plt.xlim([0,3000])
plt.xlabel('Income')
plt.ylabel('Balance')


I am working on the file "Data_Types_Control_Flow_Operators.ipynb".
The first task is:
Modulo %
Yields the remainder from the division of the first arg by the second.
Basically, for simple calculations - returns the remainder.
It can be used to determine divisibilty. Is 25 divisible by 5?
Can you write Python code for it?

25 % 5

23 % 5

nums = [x for x in range(0,100)]

for entry in nums:
    if not entry % 5 :
        print(entry)


I am working on the file "Data_Types_Control_Flow_Operators.ipynb".
The first task is:
Square **
Unlike other programming languages, Python uses ** and not ^
Can you write Python code for it?

5 ** 2 # or 5 squared

nums = [0,1,2,3,4,5]
nums_squared = [element ** 2 for element in nums] # list comprehension
nums_squared


I am working on the file "Data_Types_Control_Flow_Operators.ipynb".
The first task is:
There can be no elif and the final else is optional.
Note for people with a different language background, Python doesn't have a switch or case statement
For
Python iterates over items of any sequence
Can you write Python code for it?

words = ['cat', 'dog', 'alligator', 'pirate']
for w in words:
    print(w, len(w))

var1 = "this is sstring"
for char in var1:
    print(hex(ord(char)))

help(ord)


I am working on the file "Data_Types_Control_Flow_Operators.ipynb".
The first task is:
True division /
Can you write Python code for it?


I am working on the file "Data_Types_Control_Flow_Operators.ipynb".
The first task is:
While
convention: while true:
Can you write Python code for it?


I am working on the file "Data_Wrangle.ipynb".
The first task is:
Collect and Clean Twitter Data
The twitter data was obtained using the Trump Twitter Archive, the data is from 01/20/2017 - 03/02/2018 2:38 PM MST. I used the Federal Register's website to obtain all of the actions published by the Executive Office for the same time frame.
Can you write Python code for it?

# load json twitter data
twitter_json = r'data/twitter_01_20_17_to_3-2-18.json'
# Convert to pandas dataframe
tweet_data = pd.read_json(twitter_json)


I am working on the file "Data_Wrangle.ipynb".
The first task is:
Create a dictionary using DefaultDict where the date of publication is the key, and the text of the pdf is the value.
Can you write Python code for it?

start_path = r'data/'
end_path = '.pdf'
data_dict = defaultdict(list)
for i in range(0,270):
    file_path = start_path + str(i) + end_path
    txt = convert_pdf_to_txt(file_path)
    date_obj = extract_date(txt)
    data_dict[date_obj].append(txt)


I am working on the file "Data_Wrangle.ipynb".
The first task is:
Create a list of tuples, where the date is the first entry and the text of a pdf is the second entry, skipping over any values of None
Can you write Python code for it?

tuple_lst = []
for k, v in data_dict.items():
    if v != None:
        for text in v:
            tuple_lst.append((k, text))

# create dataframe from list of tuples
fed_reg_dataframe = pd.DataFrame.from_records(tuple_lst, columns=['date','str_text'], index = 'date')

# tokenize all the pdf text
fed_reg_dataframe['token_text'] = fed_reg_dataframe['str_text'].apply(lambda x: word_tokenize(x.lower()))

# final dataframe
fed_reg_dataframe[fed_reg_dataframe.index > '2017-01-20']