I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
按照值排序
Series.sort_values(), DataFrame.sort_values()用于按照值进行排序，参数有by
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
排序
排序方法可以分为两大类： 按照实际的值排序和按照label排序。
按照索引排序 sort_index()
Series.sort_index(), DataFrame.sort_index(), 参数是ascending, axis。
Can you write Python code for it?

unsorted_df = df.reindex(index=['a', 'd', 'c', 'b'],
                         columns=['three', 'two', 'one'])
unsorted_df

unsorted_df.sort_index()

unsorted_df.sort_index(ascending=False)

unsorted_df.sort_index(axis=1)

unsorted_df['three'].sort_index() # Series


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
改变时间的格式也很方便，Series.dt.strftime()
Can you write Python code for it?

s = pd.Series(pd.date_range('20130101', periods=4))

s

s.dt.strftime('%Y/%m/%d')


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
数值数据类型可以在ndarray，Series和DataFrame中传播。
Can you write Python code for it?

df1 = pd.DataFrame(np.random.randn(8, 1), columns=['A'], dtype='float32')

df1

df1.dtypes

df2 = pd.DataFrame(dict( A = pd.Series(np.random.randn(8), dtype='float16'),
                         B = pd.Series(np.random.randn(8)),
                         C = pd.Series(np.array(np.random.randn(8), dtype='uint8')) )) #这里是float16， uint8
df2

df2.dtypes


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
数据类型的默认值
整型的默认值蕾西int64，浮点型的默认类型是float64，和你用的是32位还是64位的系统无关。
Can you write Python code for it?

pd.DataFrame([1, 2], columns=['a']).dtypes

pd.DataFrame({'a': [1, 2]}).dtypes

pd.DataFrame({'a': 1}, index=list(range(2))).dtypes


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
最大/最小值对应的索引值
Series和DataFrame内置的idxmin() idxmax()方法求得 最小值、最大值对应的索引值，看一下例子：
Can you write Python code for it?

s1 = pd.Series(np.random.randn(5))
s1

s1.idxmin(), s1.idxmax() #最小值：-0.296405, 最大值：1.735420

df1 = pd.DataFrame(np.random.randn(5,3), columns=list('ABC'))
df1

df1.idxmin(axis=0)

df1.idxmax(axis=1)


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
最小/最大值
Series有nsmallest() nlargest()方法能够返回最小或最大的n个值。如果Series对象很大，这两种方法会比先排序后使用head()方法快很多。
Can you write Python code for it?

s = pd.Series(np.random.permutation(10))
s

s.sort_values()

s.nsmallest(3)

s.nlargest(3)


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
比较对象是否相等
一个问题通常有多种解法。一个最简单的例子：df+df和df*2。为了检测两个计算结果是否相等，你可能想到：(df+df == df*2).all()，然而，这样计算得到的结果是False：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
注意 f g h三个方法中DataFrame都是作为第一个参数。如果DataFrame作为第二个参数呢？方法是为pipe提供(callable, data_keyword)，pipe会自动调用DataFrame对象。
比如，使用statsmodels处理回归问题，他们的API期望第一个参数是公式，第二个参数是DataFrame对象data。我们使用pipe传递(sm.poisson, 'data'):

pipe灵感来自于Unix中伟大的艺术：管道。pandas中pipe()的实现很简洁，推荐阅读源代码pd.DataFrame.pipe
基于行或者列的函数应用
任意函数都可以直接对DataFrame或Panel某一坐标轴进行直接操纵，只需要使用apply()方法即可，同描述性统计方法一样，apply()方法接收axis参数。
Can you write Python code for it?

df.apply(np.mean)

df.apply(np.mean, axis=1)

df.apply(lambda x: x.max() - x.min())

df.apply(np.cumsum)

df.apply(np.exp)


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
灵活的比较操作
pandas引入了二元比较运算方法：eq, ne, lt, gt, le。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
离散区间也可以用极限定义
Can you write Python code for it?

arr = np.random.randn(20)

factor = pd.cut(arr, [-np.inf, 0, np.inf])

factor


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
移除某些索引值
和reindex()方法很相似的是drop()，用于移除索引的某些取值。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
统计相关 的方法
Series, DataFrame和Panel内置了许多计算统计相关指标的方法。这些方法大致分为两类：
* 返回低维结果，比如sum(),mean(),quantile()
* 返回同原对象同样大小的对象，比如cumsum(), cumprod()
总体来说，这些方法接收一个坐标轴参数:
* Series不需要坐标轴参数
* DataFrame 默认axis=0(index), axis=1(columns)
* Panel 默认axis=1(major), axis=0(items), axis=2(minor)
Can you write Python code for it?

df

df.mean() #axis=0, 计算每一列的平均值

df.mean(1) #计算每一行的平均值


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
虽然前面介绍过mode()方法了，看两个例子吧：
Can you write Python code for it?

s5 = pd.Series([1,1,3,3,3,5,5,7,7,7])

s5.mode()

df5 = pd.DataFrame({"A": np.random.randint(0, 7, size=50),
                    "B": np.random.randint(-10, 15, size=50)})

df5

df5.mode()


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
解释：
对于df1中NaN的元素，用df2中对应位置的元素替换！
DataFrame.combine()
DataFrame.combine()方法接收一个DF对象和一个combiner方法。
Can you write Python code for it?

combiner = lambda x,y: np.where(pd.isnull(x), y,x)

df1.combine(df2, combiner)


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
还好pandas提供了equals()方法解决上面NaN之间不想等的问题。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
迭代操作 Iteration
pandas中迭代操作依赖于具体的对象。迭代Series对象时类似迭代数组,产生的是值，迭代DataFrame或Panel对象时类似迭代字典的键值。
一句话，(for i in object)产生:
* Series: 值
* DataFrame: 列名
* Panel: item名
看一下例子吧：
Can you write Python code for it?

df = pd.DataFrame({'col1' : np.random.randn(3), 'col2' : np.random.randn(3)},
                  index=['a', 'b', 'c'])
df

for col in df: #产生的是列名
    print col


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
重命名索引值
rename() 方法可以对索引值重新命名，命名方式可以是字典或Series，也可以是任意的方法。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
重新索引和改变label
reindex()是pandas中基本的数据对其方法。其他所有依赖label对齐的方法基本都要靠reindex()实现。reindex(重新索引)意味着是沿着某条轴转换数据以匹配新设定的label。具体来说，reindex()做了三件事情：
* 对数据进行排序以匹配新的labels
* 如果新label对应的位置没有数据，插入缺失值NA
* 可以指定调用fill填充数据。
下面是一个简单的例子：
Can you write Python code for it?

s = pd.Series(np.random.randn(5), index=['a','b','c','d','e'])

s

s.reindex(['e', 'b', 'f', 'd'])


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
默认describe()只包含25%, 50%, 75%, 也可以通过percentiles参数进行指定。
Can you write Python code for it?

series.describe(percentiles=[.05, .25, .75, .95])


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/数据结构的内置方法-checkpoint.ipynb".
The first task is:
默认情况下修改的仅仅是副本，如果想对原对象索引值修改，inplace=True.
在0.18.0版本中，rename方法也能修改Series.name
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
==和列表对象一起使用 Special use of the == operator with list objects
可以使用==/!=将列表和列名直接进行比较，等价于使用in/not in.
三种方法功能等价： ==/!= VS in/not in VS isin()/~isin()
Can you write Python code for it?

df.query('b==["a", "b", "c"]')

df[df.b.isin(["a", "b", "c"])] #Python语法

df.query('c == [1, 2]')

df.query('c != [1, 2]')

df.query('[1, 2] in c') #使用in

df.query('[1, 2] not in c')

df[df.c.isin([1, 2])] #Python语法


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
DataFrame同样有isin方法，参数是数组或字典。二者的区别看例子吧：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
DataFrame的示例:
Can you write Python code for it?

df1 = pd.DataFrame(np.random.randn(6,4),index=list(range(0,12,2)), columns=list(range(0,8,2)))
df1

df1.iloc[:3]


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
Index对象 Index objects
pandas中的Index类和它的子类可以被当做一个序列可重复集合(ordered multiset)，允许数据重复。然而，如果你想把一个有重复值Index对象转型为一个集合这是不可以的。创建Index最简单的方法就是通过传递一个列表或者其他序列创建。
Can you write Python code for it?

index = pd.Index(['e', 'd', 'a', 'b'])
index

'd' in index


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
Index对象中也有isin方法.
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
MultiIndex query() 语法
对于DataFrame对象，可以使用MultiIndex，如同操作列名一样。
Can you write Python code for it?

n = 10

colors = np.random.choice(['red', 'green'], size=n)

foods = np.random.choice(['eggs', 'ham'], size=n)

colors

foods

index = pd.MultiIndex.from_arrays([colors, foods], names=['color', 'food'])

df = pd.DataFrame(np.random.randn(n,2), index=index)

df

df.query('color == "red"')


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
Python和Numpy的索引操作符[]和属性操作符‘.’能够快速检索pandas数据。
然而，这两种方式的效率在pandas中可能不是最优的，我们推荐使用专门优化过的pandas数据检索方法。而这些方法则是本节要介绍的。
多种索引方式
pandas支持三种不同的索引方式：
* .loc 是基本的基于label的，当然也可以和一个boolean数组一起使用。
* .iloc 是基本的基于整数位置(从0到axis的length-1)的，当然也可以和一个boolean数组一起使用。当提供检索的index越界时会有IndexError错误，注意切片索引(slice index)允许越界。
* .ix 支持基于label和整数位置混合的数据获取方式。默认是基本label的. .ix是最常用的方式，它支持所有.loc和.iloc的输入。如果提供的是纯label或纯整数索引，我们建议使用.loc或 .iloc。
以 .loc为例看一下使用方式：
对象类型  |  Indexers
Series | s.loc[indexer]
DataFrame | df.loc[row_indexer, column_indexer]
Panel | p.loc[item_indexer, major_indexer, minor_indexer]
最基本的索引和选择
最基本的选择数据方式就是使用[]操作符进行索引，
对象类型 |  Selection | 返回值类型
Series | series[label],这里的label是index名 | 常数
DataFrame| frame[colname],使用列名 | Series对象，相应的colname那一列
Panel | panel[itemname] | DataFrame对象,相应的itemname那一个
下面用示例展示一下
Can you write Python code for it?

dates = pd.date_range('1/1/2000', periods=8)
dates

df  = pd.DataFrame(np.random.randn(8,4), index=dates, columns=list('ABCD'))
df

panel = pd.Panel({'one':df, 'two':df-df.mean()})
panel


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
in 和not in 运算符 The in and not in operators
query()也支持Python中的in和not in运算符，实际上是底层调用isin
Can you write Python code for it?

df = pd.DataFrame({'a': list('aabbccddeeff'), 'b': list('aaaabbbbcccc'),
                  'c': np.random.randint(5, size=12),
                  'd': np.random.randint(9, size=12)})

df

df.query('a in b')

df[df.a.isin(df.b)]

df[~df.a.isin(df.b)]

df.query('a in b and c < d') #更复杂的例子

df[df.b.isin(df.a) & (df.c < df.d)] #Python语法


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
loc同样支持赋值操作
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
lookup()方法 The lookup()方法
输入行label和列label，得到一个numpy数组，这就是lookup方法的功能。
Can you write Python code for it?

dflookup = pd.DataFrame(np.random.randn(20, 4), columns=list('ABCD'))
dflookup

dflookup.lookup(list(range(0,10,2)), ['B','C','A','B','D'])


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
mask
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
query()可以去掉圆括号, 也可以用and 代替&运算符
Can you write Python code for it?

df.query('a < b & b < c')

df.query('a<b and b<c')


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
query()的性能
DataFrame.query()底层使用numexptr，所以速度要比Python快,特别时当DataFrame对象非常大时。

重复数据的确定和删除 Duplicate Data
如果你想确定和去掉DataFrame对象中重复的行，pandas提供了两个方法：duplicated和drop_duplicates. 两个方法的参数都是列名。
* duplicated 返回一个布尔向量，长度等于行数，表示每一行是否重复
* drop_duplicates 则删除重复的行
默认情况下，首次遇到的行被认为是唯一的，以后遇到内容相同的行都被认为是重复的。不过两个方法都有一个keep参数来确定目标行是否被保留。
* keep='first'(默认)：标记/去掉重复行除了第一次出现的那一行
* keep='last': 标记/去掉重复行除了最后一次出现的那一行
* keep=False: 标记/去掉所有重复的行
Can you write Python code for it?

df2 = pd.DataFrame({'a': ['one', 'one', 'two', 'two', 'two', 'three', 'four'],
                        'b': ['x', 'y', 'x', 'y', 'x', 'x', 'x'],
                      'c': np.random.randn(7)})

df2

df2.duplicated('a') #只观察列a的值是否重复

df2.duplicated('a', keep='last')

df2.drop_duplicates('a')

df2.drop_duplicates('a', keep='last')

df2.drop_duplicates('a', keep=False)


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
select()方法 The select() Method
Series, DataFrame和Panel都有select()方法来检索数据，这个方法作为保留手段通常其他方法都不管用的时候才使用。select接受一个函数(在label上进行操作)作为输入返回一个布尔值。
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(10, 3), columns=list('ABC'))

df.select(lambda x: x=='A', axis=1)


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
where方法还有一个可选的other参数，作用是替换返回结果中是False的值，并不会改变原对象。
Can you write Python code for it?

df.where(df<0, 2)

df

df.where(df<0, df) #将df作为other的参数值


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
上面两种方法返回的结果抖一下，那么应该使用哪种方法呢？答案是我们更推荐大家使用方法二。
dfmi['one']选择了第一级列然后返回一个DataFrame对象，然后另一个Python操作dfmi_with_one['second']根据'second'检索出了一个Series。对pandas来说，这两个操作是独立、有序执行的。而.loc方法传入一个元组(slice(None),('one','second')),pandas把这当作一个事件执行，所以执行速度更快。
为什么使用链式索引赋值为报错？
刚才谈到不推荐使用链式索引是出于性能的考虑。接下来从赋值角度谈一下不推荐使用链式索引。首先，思考Python怎么解释执行下面的代码？
Can you write Python code for it?

dfmi.loc[:,('one','second')]=value
#实际是
dfmi.loc.__setitem__((slice(None), ('one', 'second')), value)


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
你可能想基于某种判断条件来赋值。一种直观的方法是：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
使用where方法
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
使用切片检索
Can you write Python code for it?

df1.loc['d':,'A':'C'] #注意是闭集合

df1.loc['a']


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
使用布尔数组检索
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
使用赋值的方式扩充对象 Setting With Enlargement
用.loc/.ix/[]对不存在的键值进行赋值时，将会导致在对象中添加新的元素，它的键即为赋值时不存在的键。
对于Series来说，这是一种有效的添加操作。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
利用列表解析和map方法能够产生更加复杂的选择标准。
Can you write Python code for it?

df2 = pd.DataFrame({'a' : ['one', 'one', 'two', 'three', 'two', 'one', 'six'],
                    'b' : ['x', 'y', 'y', 'x', 'y', 'x', 'x'],
                     'c' : np.random.randn(7)})
df2

criterion = df2['a'].map(lambda x:x.startswith('t'))

df2[criterion]

df2[[x.startswith('t') for x in df2['a']]]

df2[criterion & (df2['b'] == 'x')]


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
如果切片检索时输入的范围越界，没关系，只要pandas版本>=v0.14.0, 就能如同Python/Numpy那样正确处理。
注意：仅限于 切片检索
Can you write Python code for it?

x = list('abcdef')
x

x[4:10] #这里x的长度是6

x[8:10]

s = pd.Series(x)

s

s.iloc[4:10]

s.iloc[8:10]

df1 = pd.DataFrame(np.random.randn(5,2), columns=list('AB'))
df1

df1.iloc[:,2:3]

df1.iloc[:,1:3]

df1.iloc[4:6]


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
如果是对DataFrame对象进行有权重采样，一个简单 的方法是新增一列用于表示每一行的权重
Can you write Python code for it?

df2 = pd.DataFrame({'col1':[9,8,7,6], 'weight_column':[0.5, 0.4, 0.1, 0]})
df2

df2.sample(n=3,weights='weight_column')


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
对于DataFrame对象来说，[]操作符按照行进行切片，非常有用。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
对列进行采样, axis=1
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
对齐
where方法会将输入的布尔条件对齐，因此允许部分检索时的赋值。
Can you write Python code for it?

df2 = df.copy()

df2[df2[1:4] >0]=3

df2

df2 = df.copy()

df2.where(df2>0, df2['A'], axis='index')


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
布尔运算符 Boolean Operators
可以使用not或者~对布尔表达式进行取非。
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(n, 3), columns=list('abc'))
df

df['bools']=np.random.randn(len(df))>0.5

df

df.query('bools')

df.query('not bools')

df.query('not bools') == df[~df.bools]


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
得到DataFrame中的某一个值, 等同于df1.get_value('a','A')
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
我们也可以使用random_state参数 为sample内部的随机数生成器提供种子数。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
我们使用最基本的[]操作符
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
标量值的快速获取和赋值
如果仅仅想获取一个元素，使用[]未免太繁重了。pandas提供了快速获取一个元素的方法：at和iat. 适用于Series、DataFrame和Panel。
如果loc方法，at方法的合法输入是label，iat的合法输入是整型。
Can you write Python code for it?

s.iat[5]

df.at[dates[5],'A']

df.iat[3,0]


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
正确的方式是：老老实实使用.loc
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
比较下面两种访问方式：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
注意下面两个示例，输出是相同的，因为使用了相同的种子数
Can you write Python code for it?

df4.sample(n=2, random_state=2)

df4.sample(n=2,random_state=2)

df4.sample(n=2,random_state=3)


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
注意：由于sample默认进行的是无放回抽样，所以输入必须n<=行数，除非进行可放回抽样。
Can you write Python code for it?

s

s.sample(7) #7不行

s.sample(7,replace=True)


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
看到__getitem__了吗？除了最简单的情况，我们很难预测他到底返回的是视图还是副本(哲依赖于数组的内存布局，这是pandas没有硬性要求的)，因此不推荐使用链式索引赋值！
而dfmi.loc.__setitem__直接对dfmi进行操作。
有时候明明没有使用链式索引，也会引起SettingWithCopy警告，这是Pandas设计的bug~
Can you write Python code for it?

def do_something(df):
   foo = df[['bar', 'baz']]  # Is foo a view? A copy? Nobody knows!
   # ... many lines here ...
   foo['quux'] = value       # We don't know whether this will modify df or not!
   return foo


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
细心地你一定发现了，index='20160104'那一行也被检索出来了，没错，loc检索时范围是闭集合[start,end].
整型可以作为label检索，这是没问题的，不过要记住此时整型表示的是label而不是index中的下标！
.loc操作是检索时的基本操作，以下输入格式都是合法的：
* 一个label，比如：5、'a'. 记住这里的5表示的是index中的一个label而不是index中的一个下标。
* label组成的列表或者数组比如['a','b','c']
* 切片，比如'a':'f'.注意loc中切片范围是闭集合！
* 布尔数组
Can you write Python code for it?

s1 = pd.Series(np.random.randn(6), index=list('abcdef'))
s1

s1.loc['c':]

s1.loc['b']


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
结合isin方法和any() all()可以对DataFrame进行快速查询。比如选择每一列都符合标准的行:
Can you write Python code for it?

values = {'ids': ['a', 'b'], 'ids2': ['a', 'c'], 'vals': [1, 3]}

row_mark = df.isin(values).all(1)

df[row_mark]

row_mark = df.isin(values).any(1)

df[row_mark]


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
结合loc、iloc等方法可以检索多个坐标下的数据.
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
表达式任意复杂都没关系。
Can you write Python code for it?

shorter = df.query('a<b<c and (not bools) or bools>2')
shorter

longer = df[(df.a < df.b) & (df.b < df.c) & (~df.bools) | (df.bools > 2)]
longer

shorter == longer


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
输入string进行检索没问题
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
还可以个Index命名
Can you write Python code for it?

index = pd.Index(['e', 'd', 'a', 'b'], name='something')
index.name

index = pd.Index(list(range(5)), name='rows')

columns = pd.Index(['A', 'B', 'C'], name='cols')

df = pd.DataFrame(np.random.randn(5, 3), index=index, columns=columns)

df

df['A']


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
进行行和列的检索
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
链式索引中顺序也很重要
此外，在链式表达式中，不同的顺序也可能导致不同的结果。这里的顺序指的是检索时行和列的顺序。
Can you write Python code for it?

dfb = pd.DataFrame({'a' : ['one', 'one', 'two',
                               'three', 'two', 'one', 'six'],
                       'c' : np.arange(7)})
dfb

dfb['c'][dfb.a.str.startswith('o')] = 42 #虽然会引起SettingWithCopyWarning 但也能得到正确结果

pd.set_option('mode.chained_assignment','warn')
dfb[dfb.a.str.startswith('o')]['c'] = 42 #这实际上是对副本赋值！


I am working on the file "05其他/pandas文档-zh-master/.ipynb_checkpoints/检索 ,查询数据-checkpoint.ipynb".
The first task is:
默认情况下，where方法并不会修改原始对象，它返回的是一个修改过的原始对象副本，如果你想直接修改原始对象，方法是将inplace参数设置为True
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(6,5), index=list('abcdef'), columns=list('ABCDE'))
df_orig = df.copy()

df_orig.where(df < 0, -df, inplace=True);

df_orig


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
-- Oracle's RANK() analytic function
SELECT * FROM (
SELECT
t.*,

RANK() OVER(PARTITION BY sex ORDER BY tip) AS rnk

FROM tips t
WHERE tip < 2
)
WHERE rnk < 3
ORDER BY sex, rnk;
更新
UPDATE tips
SET tip = tip*2
WHERE tip < 2;
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
-- tips by parties of at least 5 diners OR bill total was more than $45
SELECT *
FROM tips
WHERE size >= 5 OR total_bill > 45;
Can you write Python code for it?

tips[(tips['size'] >= 5) | (tips['total_bill'] > 45)]


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
SELECT *
FROM frame
WHERE col1 IS NOT NULL;
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
SELECT *
FROM frame
WHERE col2 IS NULL;
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
SELECT city, rank
FROM df1
UNION ALL
SELECT city, rank
FROM df2;
/*
     city  rank

  Chicago     1

  Chicago     2

 New York     3

  Chicago     1

   Boston     4

  Angeles     5
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
上述声明是通过給DataFrame传递一系列的真/假对象,返回所有真实的行。
Can you write Python code for it?

is_dinner = tips['time'] == 'Dinner'

is_dinner.value_counts()

tips[is_dinner].head(5)


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
传递一个列表給groupby()方法，完成多个列的分组。
SELECT smoker, day, COUNT(*), AVG(tip)
FROM tips
GROUP BY smoker, day;
/*
smoker day
No     Fri      4  2.812500
   Sat     45  3.102889

   Sun     57  3.167895

   Thur    45  2.673778

Yes    Fri     15  2.714000
   Sat     42  2.875476

   Sun     19  3.516842

   Thur    17  3.030000

*/
Can you write Python code for it?

tips.groupby(['smoker', 'day']).agg({'tip': [np.size, np.mean]})


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
使用notnull()和isnull()方法进行NULL检查。
Can you write Python code for it?

frame = pd.DataFrame({'col1': ['A', 'B', np.NaN, 'C', 'D'],
                      'col2': ['F', np.NaN, 'G', 'H', 'I']})

frame


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
使用rank(method=’first’)函数
Can you write Python code for it?

(tips.assign(rnk=tips.groupby(['day'])['total_bill']
                       .rank(method='first', ascending=False))
       .query('rnk < 3')
     .sort_values(['day','rnk'])
   )


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
假设我们有两个与DataFrames同名同结构的数据库表。现在我们来讨论各种类型的连接
内连接
SELECT *
FROM df1
INNER JOIN df2
ON df1.key = df2.key;
Can you write Python code for it?

# merge performs an INNER JOIN by default
pd.merge(df1, df2, on='key')


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
全连接
pandas还允许全连接,显示两边的数据集,无论是否找到列匹配。全连接不支持所有RDBMS(MySQL)。
-- show all records from both tables
SELECT *
FROM df1
FULL OUTER JOIN df2
ON df1.key = df2.key;
Can you write Python code for it?

pd.merge(df1, df2, on='key', how='outer')


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
分组
在pandas中,执行SQL的GROUP BY操作时使用同样的命名groupby()方法。groupby()通常是指一个过程,我们想把数据集分成组,应用一些函数(通常聚合),然后结合分组在一起。一个常见的SQL操作可以得到数据集中每组记录的数量。例如:
SELECT sex, count(*)
FROM tips
GROUP BY sex;
/*
Female     87
Male      157
*/
同理，The pandas是这样的：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
左外连接
-- show all records from df1
SELECT *
FROM df1
LEFT OUTER JOIN df2
ON df1.key = df2.key;
Can you write Python code for it?

# show all records from df1
pd.merge(df1, df2, on='key', how='left')


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
或者,我们可以用count()方法应用于一个单独的列:
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
每组前n行
-- Oracle's ROW_NUMBER() analytic function
SELECT * FROM (
SELECT
t.*,

ROW_NUMBER() OVER(PARTITION BY day ORDER BY total_bill DESC) AS rn

FROM tips t
)
WHERE rn < 3
ORDER BY day, rn;
Can you write Python code for it?

 (tips.assign(rn=tips.sort_values(['total_bill'], ascending=False)
                      .groupby(['day'])
                    .cumcount() + 1)
     .query('rn < 3')
        .sort_values(['day','rn'])
    )


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
注意,在pandas的代码中，我们使用size()并不是count()。这是因为count()函数适用于每一列,返回每个非空记录的数量。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/与SQL对比-Comparison with SQL.ipynb".
The first task is:
选择
在SQL中， 用逗号将所要选择的列分隔开，来进行选择。（或用*表示选择所有列）：
SELECT total_bill, tip, smoker, time
FROM tips
LIMIT 5;
在pandas中，通过給DataFrame传递列名进行列选择：
Can you write Python code for it?

tips[['total_bill', 'tip', 'smoker', 'time']].head(5)


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:

Can you write Python code for it?

result = pd.merge(left, right, left_index=True, right_index=True, how='inner')


result = df1.append(df4)
#由于df1和df2的索引值交集不为空，导致最后result的索引值有重复！


result = pd.merge(left, right, how='inner', on=['key1', 'key2'])


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
DataFrame.join方法也提供了lsuffix和rsuffix参数来实现同样的功能:
Can you write Python code for it?

left = left.set_index('k')
right = right.set_index('k')

result = left.join(right, lsuffix='_l', rsuffix='_r')


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
append()操作并不是直接对df1对象进行操作，而是在df1副本的基础上进行拼接操作。
拼接操作时忽略索引值
大多数情况下，索引值都是默认生成的一些无意义的id，此时，两个DataFrame对象很可能有重复的索引值，但他们并没有实质物理含义，此时，在进行行拼接操作时我们可以忽略索引，使用ignore_index参数即可。
在进行列拼接时，就不要使用ignore_index参数了，因为99.9999%的数据列名都是有意义的。
Can you write Python code for it?

df1

df4

result = pd.concat([df1, df4],ignore_index=True)
result

df1

df4

result2 = pd.concat([df1, df4],axis=1, ignore_index=True) #99.9999%不推荐大家列拼接时使用ignore_index
result2


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
append()方法可以一次拼接多个对象。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
df1和df2列名完全相同，索引值也相似。
由于df2中不存在索引值=0，所以df1中第一行的值原封不动；
对于df1中不等于NaN的值，也不需要改变；
对于df1中等于NaN的值，用df2中同索引同列名的值替换， 得到结果。
注意combine_first()方法不改变df1的值。
如果想改变df1的值，使用update()方法。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
merge方法中indicator参数说明
0.17.0版本才有的参数
merge方法在0.17.0版本中新增了一个参数indicator。这个参数取值True，False或字符串。如果取值True，会在拼接结果中多出一列'_merge', 显示 每一行的键值来自left还是right。
|    _merge值     | 含义|
| ------------- |:-------------:|
| left_only      | 键值仅来自left|
| right_only      | 键值仅来自right|
| both | 键值来自left和both|
Can you write Python code for it?

df1 = pd.DataFrame({'col1':[0,1], 'col_left':['a','b']})
df2 = pd.DataFrame({'col1':[1,2,2],'col_right':[2,2,2]})

df1

df2

pd.merge(df1, df2, on='col1', how='outer', indicator=True)


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
merge方法中提供了left_index, right_index参数同样实现了基于索引的拼接操作。
Can you write Python code for it?

result = pd.merge(left, right, left_index=True, right_index=True, how='outer')


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
上面提到indicator参数取值也可以是字符串，没啥特别的，就是在拼接结果中用接收的字符串作为'_merge'的列名罢了。
Can you write Python code for it?

pd.merge(df1, df2, on='col1', how='outer', indicator='indicator_column')


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
使用join='inner' 查看结果 ,和outer一样！
Can you write Python code for it?

res = pd.concat(frames, join='inner')
res


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
使用合并方法填充NaN
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
修改df3的index，看看结果
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
修改df5，使得某几行和df2一样，再看看结果
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
先使用combine_first方法：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
再来看看join='inner'的例子：
Can you write Python code for it?

result = pd.concat([df1, df4], axis=1, join='inner')
result


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
合并 有序数据
v0.18.0新增了一个ordered_merge()方法 对时间序列或者其他有序数据进行合并。并且ordered_merge提供了fill_method参数来填充缺失值。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
回顾一下我们的第一个例子:
Can you write Python code for it?

result = pd.concat(frames, keys=['x', 'y', 'z'])


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
如同numpy.concatenate()方法, pandas.concat方法接收一个列表或者字典对象然后进行拼接操作，注意列表或字典中每个对象要是同构的。拼接的本质就是沿着一条轴并且根据对另一条轴上进行的操作 将列表中的对象堆叠。
concat方法的参数:
pd.concat(objs, axis=0, join='outer', join_axes=None, ignore_index=False,
       keys=None, levels=None, names=None, verify_integrity=False)

objs: Series、DataFrame或Panel对象构成的列表或字典。如果传入一个字典对象，它的键值就作为最后结果的键值参数，除非显示说明结果的键值参数。
axis: 轴，取值范围是{0,1,...}，默认值为0。concat操作时沿着的那条轴
join: 取值范围是{'inner', 'outer'},即内连接(交集)和外连接(并集)。 默认是'outer'(外连接)。连接操作决定了如何处理除axis之外的那条轴。
join_axes: 拼接操作得到的结果，它的索引取值。
keys: 序列对象，默认是None。 构建层次化索引时使用传入的keys作为最外层的索引。如果想要传入多级keys，需要以元组形式传入。
levels: 序列构成的列表，默认是None。用于构建MultiIndex。
names：列表，默认是None。层次索引中每个level的名字。
verify_integrity: 布尔变量，默认是False。用于检查新拼接的轴是否包含重复值。这个操作很耗时。
ignore_index: 布尔变量，默认是False。如果值为True，则不使用拼接轴现有的索引值，拼接结果的索引值将会设置为0,1,...,n-1。
copy: 布尔变量，默认是True。 如果取值为False，则不会复制不必要的数据。

如果不结合例子来看，上面介绍的参数真没法解释。
先来看一下keys参数 用于构建层次化索引时起到的作用。
Can you write Python code for it?

#层次化索引， 用处是给列表中每个对象一个map标记，这样在结果中还能方便的调用每个子Series或DataFrame
result = pd.concat(frames, keys=['x','y','z'])

result


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
对Series对象进行拼接时使用keys参数
concat()方法中keys参数除了构建层次化索引外，另一个很常见的使用情景是：对多个Series对象进行拼接操作，我们希望重新制定得到的DataFrame对象中的列名，而不是使用Series中的名字。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
对索引进行拼接
DataFrame.join方法的作用是对两个DataFrame对象进行基于索引值的拼接操作。下面是一个简单的例子:
把索引也看做是一列，再来看下面的操作就就没啥了
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
数据库风格的DataFrame 连接/合并操作
pandas提供了非常高效的DataFrame连接(join)操作，并且使用风格和SQL非常类似。相对于其他的开源实现，pandas无疑是相当高效的, 这要归功于对DataFrame对象的优秀的算法设计和高效的内部数据部署。
如果你之前有过SQL开发经验而首次使用pandas，你可能想先了解二者的对比。
pandas仅提供了一个merge()方法，就能够对DataFrame对象进行类似关系数据库的连接(join)操作，先看merge()参数
merge(left, right, how='inner', on=None, left_on=None, right_on=None,
      left_index=False, right_index=False, sort=True,
      suffixes=('_x', '_y'), copy=True, indicator=False)

left: 参与操作的DataFrame对象
right: 参与操作的另一个DataFrame对象
on: 进行连接操作用到的列名,可以是一列，也可以是多列。这个列名必须在left和right中都存在！ 如果on参数没有赋值，并且left_index和right_index都是False，pandas会将left和right都含有的列作为on的值
left_on: 取left中某一列名作为keys，可以是列名或者长度和DataFrame对象一行长度相等的数组
right_on: 取right中某一列名作为keys, 可以是列名或者长度和DataFrame对象一行长度相等的数组
left_index: 如果取值是True，使用left中的索引作为连接操作的keys
right_index: 意思同上，
how: 取值范围{'left', 'right', 'outer', 'inner'},默认是inner
sort: 是否对结果按照连接keys进行排序，默认是True,如果顺序不重要，可以设置为False，会提高效率
suffixes: 元组。如果left和right中除keys之外，也有列名一样，会用suffixes参数区别, 默认格式('_x', '_y').
copy:是否copy数据，默认为True，这样不会改变left和right
indicator：在pandas 0.17.0中加入的参数，如果设置为True,能够显示结果中的key来自left还是right，

merger方法的返回的对象类型同left。
merge()方法是pandas中的全局方法，你可以通过DataFrame对象调用它。
和merge()方法很相似的是DataFrame.join()方法，它内部使用merge方法来实现 索引与索引、索引与列 的拼接操作，默认是对索引进行拼接，而merge方法默认是对列进行拼接，无疑，对列进行拼接更常见。除非你要对索引进行拼接，通常都是使用merge()方法。
基本的数学知识：关系代数
对数据库操作熟悉的同学肯定对拼接操作不陌生，下面几种情况是必须知道的：

一对一拼接(one-to-one join): 比如对两个DF对象的索引 进行拼接操作，注意此时索引值必须是唯一的
多对一拼接 (many-to-one join): 对一个对象的索引(唯一的)和另一个对象的一列或几列进行拼接操作
多对多拼接 (many-to-many join): 对两个对象的列 进行拼接操作

注意： 显然， 多对多拼接时最常用的，并且在多对多拼接过程中会丢弃索引值。
大家有必要花时间理解多对多拼接操作的情况。在SQL/关系代数中，如果连接时的键值(key)在两个表中值不唯一，拼接结果是两个表的笛卡尔积。
看一个简单的例子来理解笛卡尔积。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
注意到结果中的索引是层次化的。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
注意：concat()方法在拼接使要复制所有的数据，因此对于它的性能你要容忍。为了方便起见，如果对多个数据集拼接，可以使用列表解析式。
Can you write Python code for it?

frames = [process_your_file(f) for f in files]
result = pd.concat(frames)


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
解释:
此时拼接操作的键值是['key1', 'key2'],在left中['key1','key2']的取值是{['K0','K0'], ['K0', 'K1'], ['K1', 'K0'], ['K2', 'K1']},在right中['key1',  'key2']的取值是{['K0', 'K0'], ['K1', 'K0'], ['K2', 'K0']}, 默认进行内连接，所以取二者的交集{['K0', 'K0'], ['K1', 'K0']}, 并且left中['K1', 'K0']出现了两次，所以拼接结果是3行。
merge方法中的参数how决定了哪些键值(keys)会被保留到结果中。注意如果on参数接收的键值在left和right中都没有出现，拼接结果是NA。
下面总结了how参数在SQL中对应的操作。
|    参数how取值     | SQL拼接操作  | 操作描述|
| ------------- |:-------------:| :-------------:|
| left      | LEFT OUTER　JOIN| 只使用来自left中的键值|
| right      | RIGHT OUTER　JOIN      |   只使用来自right中的键值|
| outer | FULL OUTER JOIN      |    使用left、right中键值的并集|
|inner|INNER JOIN|使用left、right中键值的交集|
Can you write Python code for it?

result = pd.merge(left, right, how='left', on=['key1', 'key2'])


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
解释：
axis=1，即拼接轴是横轴，先将df1的列和df4的列名进行堆叠，即每一行有7列。由于join='inner', 故将df1和df4的索引取取并集，得到{2,3},共2行。
还要注意结果的索引值没有排序。
最后，看一下使用 join_axes参数的例子：
Can you write Python code for it?

result = pd.concat([df1, df4], axis=1, join_axes=[df1.index])


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
解释：
axis=1，即拼接轴是横轴，先将df1的列和df4的列名进行堆叠，即每一行有7列。由于join='outer', 故将df1和df4的索引取取并集，得到{0,1,2,3,6,7},共6行。
还要注意结果的索引值是排好序的。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
解释：
on的参数是['key1', 'key2'],所以拼接的键值是['key1', 'key2']，how参数是'left',所以进行的是左拼接，只关注left中键值取值。
过程：left中第一行键值是['K0', 'K1'],发现right有此取值，拼接后得到result中的第一行；
left中第二行键值是['K0', 'K1'],发现right中没有此取值，仍进行拼接得到result中第二行，注意'C' 'D'取值NaN
left中第三行键值是['K1', 'K0'], 发现right中有此取值切出现了两次，分别进行拼接得到result中第三行、第四行
left中第四行取值是['K2', 'K1'], 发现right中没有此取值，仍进行拼接得到result中第五行，'C', 'D'取值NaN
Can you write Python code for it?

result = pd.merge(left, right, how='right', on=['key1', 'key2'])


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
通过keys参数，来设置生成的DataFrame的列名
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
除拼接键值之外，重复的列名用suffixes参数区别
很多情况下，left和right中除了拼接键值之外，还会有其他列名也相同，为了在结果中加以区分，这个时候suffixes参数就派上用场了。好消息是suffixes有默认值，即使不为它赋值在结果中也能够区分。
举例如下：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据合并、连接和拼接-Merge, join, and concat.ipynb".
The first task is:
默认情况下：
Can you write Python code for it?

result = pd.merge(left, right, on='k', suffixes=['_l', '_r']) #为suffixes参数赋值


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
DataFrame.from_items
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
DataFrame与Series进行运算时, 默认行为是DataFrame的column和Series的index对齐，也就是对Series进行行广播。例如:
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
Series
结果是创建一个列数为1的DataFrame，它的index和输入的Series相同，column名是Series的name。
其他的构造函数
DataFrame.from_dict
DataFrame.from_dict接受字典的字典或数组序列的字典，并返回一个DataFrame。
DataFrame.from_records
DataFrame.from_records接受一个元组列表或结构化类型的ndarray。例如：
Can you write Python code for it?

data

pd.DataFrame.from_records(data, index='C')


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
Series vs dict
你可以把Series看做一个长度固定的字典，你可以通过index来获取和设置数值：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
Series vs ndarray
大多数情况下，你都可以把Series看做一个ndarray来处理。甚至，大多数的NumPy函数都支持Series作参数。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
dict字典
如果dict是字典，index的长度可以和data长度不同，也可以不提供index：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
你可以插入ndarray，但其长度必须与DataFrame的index长度相匹配。
缺省情况下，列被插在末端。insert()函数可以允许插入到指定的列位置：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
使用get方法，遇到缺失的label将返回None或指定值：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
元组字典
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
列选择，添加，删除
你可以把DataFrame看做Series构成的字典，使用和字典同样的操作来增加列、删除列、对列重新赋值：
Can you write Python code for it?

df['one']

df['three'] = df['one'] * df['two']

df['flag'] = df['one'] > 2

df


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
向量操作和label对齐
进行数据分析时，你可以把Series看做向量进行整体操作。大多数接受ndarray作参数的NumPy方法也支持传入Series对象。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
在特殊情况下, 处理时间序列数据时, DataFrame的index包含日期，广播会变成列广播:
Can you write Python code for it?

index = pd.date_range('1/1/2000',periods=8)

df = pd.DataFrame(np.random.randn(8,3), index=index, columns=list('ABC'))

df

type(df['A'])

df - df['A']

df.sub(df['A'], axis=0) #

df * 5 + 2

1 / df

df ** 4


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
如果label值不存在，将抛出异常：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
如果两个label不一样的Series参与计算，存储结果的Series的index/label是它们俩的union(并集)。正是自带label对齐，才使得pandas区别于大多数数据分析库，优越性一览无余。
name属性
Series构造函数还有另一个参数：name，
Can you write Python code for it?

s = pd.Series(np.random.randn(5), name='something')

s

s.name


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
字典列表
Can you write Python code for it?

data2 = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]

pd.DataFrame(data2)

pd.DataFrame(data2, index=['first','second'])

pd.DataFrame(data2, columns=['a','b'])


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
布尔运算
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
当插入一个标量值，它自然会被填充到列中
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
当插入不具有相同的index的Series时，将服从该DataFrame的索引：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
注意s和s2是两个不同对象。
DataFrame
DataFrame是一个带label的二维数据结构。你可以把它看做一张SQL表，或是由Series构成的字典。它基本上是最常用的pandas数据结构。
和Series一样，DataFrame构造函数支持许多不同类型的输入：

字典，可以是普通字典，或者一维ndarray字典、列表字典、Series字典
二维numpy.ndarray
结构化或记录化的ndarray
字典列表
一个Series
DataFrame

DataFrame构造函数也支持传入index(行label)和columns(列label)参数。
Series字典或字典
结果的index是各个Series的index的union。如果存在任何嵌套类型的字典，将首先转换成Series。如果不传入columns，则默认把字典的键构成的有序列表看做columns。
Can you write Python code for it?

 d = {'one' : pd.Series([1., 2., 3.], index=['a', 'b', 'c']),
      'two' : pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}

df = pd.DataFrame(d)

df

pd.DataFrame(d, index=['d','b','a'])

pd.DataFrame(d, index=['d', 'b', 'a'], columns=['two', 'three'])


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
注意： NaN(not a number)是pandas中缺失值标记。
标量数值
如果data是一个标量值，必须提供index。为了匹配index的长度，该数值将被重复。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
注意： 从0.8.0版本开始，index的值可以重复。
Can you write Python code for it?

pd.Series(np.random.randn(3), index=['a', 'b', 'a'])


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
用方法链来分配新列
DataFrame有一个assign()方法，利用现有列来创建新列。
iris = pd.read_csv('data/iris.data')
iris.head()
(iris.assign(sepal_ratio = iris['SepalWidth'] / iris['SepalLength'])
         .head())
iris.assign(sepal_ratio = lambda x: (x['SepalWidth'] /
                                        x['SepalLength'])).head()
(iris.query('SepalLength > 5')
         .assign(SepalRatio = lambda x: x.SepalWidth / x.SepalLength,
                 PetalRatio = lambda x: x.PetalWidth / x.PetalLength)
         .plot(kind='scatter', x='SepalRatio', y='PetalRatio'))
检索
检索的基础知识如下：
操作|语法 |结果
-----|------|----
列选择    | df[col]    | Series
通过label来选择行   | df.loc[label]   | Series
通过下标选择行    | df.iloc[loc]    | Series
选择部分行    | df[5:10]    | DataFrame
通过布尔向量选择行   | df[bool_vec]    | DataFrame
行选择,例如,返回一个index为DataFrame columns的Series:
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
类似地,dot方法也能运用在Series上:
Can you write Python code for it?

s1 = pd.Series(np.arange(5,10))

s1.dot(s1)


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
结构化或记录ndarray
Can you write Python code for it?

data = np.zeros((2,), dtype=[('A', 'i4'),('B', 'f4'),('C', 'a10')])

data[:] = [(1,2.,'Hello'), (2,3.,"World")]

pd.DataFrame(data)

pd.DataFrame(data, index=['first','second'])

pd.DataFrame(data, columns=['C','A','B'])


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
行label和列label可分别通过访问index和columns获取：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
转置
转置, T属性(转置函数):
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构.ipynb".
The first task is:
通过pandas.Series.rename()方法，可以给Series重命名。
Can you write Python code for it?

s2 = s.rename("different")

s2.name

s.name


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
## 字符串处理方法
Series带有一系列的字符串处理, 默认不对NaN处理。
Can you write Python code for it?

s = pd.Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])

s

s.str.lower()


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
DataFrame.align()同样能接收Series对象，此时axis指的是DataFrame对象的索引或列。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
Head() Tail()
想要预览Series或DataFrame对象，可以使用head()和tail()方法。默认显示5行数据，你也可以自己设置显示的行数。
Can you write Python code for it?

long_series = pd.Series(np.random.randn(1000))

long_series.head()

long_series.tail(3)


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
Series.map()还有一个功能是模仿merge(), join()
Can you write Python code for it?

s = pd.Series(['six', 'seven', 'six', 'seven', 'six'], index=['a', 'b', 'c', 'd', 'e'])

t = pd.Series({'six':6., 'seven':7.})

s

t

s.map(t)


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
Series同样有dtypes属性
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
align()也含有一个axis参数，指定仅对于某一坐标轴进行对齐。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
astype()方法
使用astype()显示的进行转型。默认会返回原对象的副本，即使数据类型不变。当然可以传递copy=False参数直接对原对象转型。
Can you write Python code for it?

df3

df3.dtypes

df3.astype('float32').dtypes


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
combine_first()
看一下例子：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
descrieb()， 数据摘要
describe()方法非常有用，它计算数据的各种常用统计指标(比如平均值、标准差等)，计算时不包括NA。拿到数据首先要有大概的了解，使用describe()方法就对了。
Can you write Python code for it?

series = pd.Series(np.random.randn(1000))
series[::2]=np.nan

series.describe()

frame = pd.DataFrame(np.random.randn(1000, 5), columns=['a', 'b', 'c', 'd', 'e'])

frame.ix[::2]=np.nan

frame.describe()


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
method参数要求索引必须是有序的：递增或递减。
除了method='nearest',其他method取值也能用fillna()方法实现：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
pandas对象也有类似字典的iteritems()方法来迭代(key, value)。
为了每一行迭代DataFrame对象，有两种方法：
* iterrows(): 按照行来迭代(index, Series)。会把每一行转为Series对象。
* itertuples(): 按照行来迭代namedtuples. 这种方法比iterrows()快，大多数情况下推荐使用此方法。
警告：
迭代pandas对象通常会比较慢。所以尽量避免迭代操作，可以用以下方法替换迭代：
* 数据结构的内置方法，索引或numpy方法等
* apply()
* 使用cython写内循环
警告：
当迭代进行时永远不要有修改操作。
iteritems()
类似字典的接口，iteritems()对键值对进行迭代操作：
* Series: (index, scalar value)
* DataFrame: (column, Series)
* Panel: (item, DataFrame)
看一下例子：
Can you write Python code for it?

for item, frame in df.iteritems():
    print item, frame


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
select_dtypes()有两个参数：include, exclude。含义是要选择的列的dtype和不选择列的dtype。
Can you write Python code for it?

df.select_dtypes(include=[bool])

df.select_dtypes(include=['bool'])

df.dtypes

df.select_dtypes(include=['number', 'bool'], exclude=['unsignedinteger'])


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
upcasting
不同类型结合时会upcast，即得到更通用的类型，看例子吧：
Can you write Python code for it?

df1.dtypes

df2.dtypes

df1.reindex_like(df2).fillna(value=0.0).dtypes

df3 = df1.reindex_like(df2).fillna(value=0.0) + df2

df3.dtypes


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
上面一行代码推荐用下面的等价写法:
Can you write Python code for it?

(df.pipe(h).pipe(g, arg1=1).pipe(f, arg2=2, arg3=3))


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
上面一行代码顺便说明了Series的索引和DataFrame的索引是同一类的实例。
重新索引来和另一个对象对齐 reindex_like()
你可能想传递一个对象，使得原来对象的label和传入的对象一样，使用reindex_like()即可。
Can you write Python code for it?

df2 = pd.DataFrame(np.random.randn(3, 2),index=['a','b','c'],columns=['one', 'two'])

df2

df

df.reindex_like(df2)


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
为什么df + df == df*2 返回的结果含有False？因为NaN和NaN比较厚结果为False！
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
二者的区别是：如果索引不是有序的，reindex()会报错，而fillna()和interpolate()不会检查索引是否有序。
重索引时 有条件地填充NaN
limit和tolerance参数会对填充操作进行条件限制，通常限制填充的次数。
Can you write Python code for it?

ts2.reindex(ts.index, method='ffill', limit=1)

ts2

ts

ts2.reindex(ts.index, method='ffill', tolerance='1 day')


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
从v0.17.0开始，DataFrame也有了以上两个方法。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
但要知道不同shape的numpy数组之间是可以直接比较的！因为广播！即使无法广播，也不会Error而是返回False。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
使用align() 是两个对象相互对齐
align()方法是让两个对象同事对齐的最快方法。它含有join参数，
* join='outer': 取得两个对象的索引并集，这也是join的默认值。
* join='left': 使用调用对象的索引值
* join='right'：使用被调用对象的索引值
* join='inner': 使用两个对象的索引交集
align()方法返回一个元组，元素元素是重新索引的Series对象。
Can you write Python code for it?

s = pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])

s1 = s[:4]
s2 = s[1:]

s1

s2

s1.align(s2)

s1.align(s2, join='inner') #交集是 'b', 'c', 'd'


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
使用empty属性检测一个pandas对象是否为空。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
函数应用
如果你想用自己写的方法或其他库方法操作pandas对象，你应该知道下面的三种方式。 具体选择哪种方式取决于你是想操作整个DataFrame对象还是DataFrame对象的某几行或某几列，或者逐元素操作。

管道 pipe()
基于列或行的函数应用 apply()
对DataFrame对象逐元素计算 applymap()

管道
DataFrame和Series当然能够作为参数传入方法。然而，如果涉及到多个方法的序列调用，推荐使用pipe()。看一下例子：
Can you write Python code for it?

#f, g 和h是三个方法，接收DataFrame对象，返回DataFrame对象
f(g(h(df), arg1=1), arg2=2, arg3=3)


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
区间离散化
cut() qcut()方法可以对连续数据进行离散化
Can you write Python code for it?

arr = np.random.randn(20)

factor = pd.cut(arr, 4)

factor

factor = pd.cut(arr, [-5, -1, 0, 1, 5]) #输入 离散区间

factor


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
另一个有用的特性是对DataFrame对象传递Series方法，然后针对DF对象的每一列或每一行执行 Series内置的方法！
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
填充缺失值
在Series和DataFrame中，算术运算方法(比如add())有一个fill_value参数，含义很明显，计算前用一个值来代替缺失值，然后再参与运算。注意，如果参与运算的两个object同一位置(同行同列)都是NaN，fill_value不起作用，计算结果还是NaN。
看例子：
Can you write Python code for it?

df

df2 = pd.DataFrame({'one' : pd.Series(np.random.randn(3), index=['a', 'b', 'c']),
                       'two' : pd.Series(np.random.randn(4), index=['a', 'b', 'c', 'd']),
                       'three' : pd.Series(np.random.randn(4), index=['a', 'b', 'c', 'd'])})

df2

df + df2

 df.add(df2, fill_value=0) #注意['a', 'three']不是NaN


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
多索引列 排序
如果一列是多索引，你必须指定全部每一级的索引。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
如果DataFrame对象有的列是数值类型，有的列不是数值类型，describe()仅对数值类型的列进行计算。
Can you write Python code for it?

frame = pd.DataFrame({'a':['Yes', 'Yes', 'NO', 'No'], 'b':range(4)})

frame.describe()


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
如果只想改变列或者索引的label，DataFrame也提供了reindex_axis()方法，接收label和axis。
Can you write Python code for it?

rs = s.reindex(df.index)

rs

rs.index is df.index


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
如果多个数值都是最大值或最小值，idxmax() idxmin()返回最大值、最小值第一次出现对应的索引值
Can you write Python code for it?

df3 = pd.DataFrame([2, 1, 1, 3, np.nan], columns=['A'], index=list('edcba'))
df3

df3['A'].idxmin()


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
如果非要知道非数值列的统计指标呢？describe提供了include参数，取值范围{'object', 'number', 'all'}。
看一下例子, 注意'object'和'number'都是在列表中，而'all'不需要放在列表中：
Can you write Python code for it?

frame.describe(include=['object']) #只对非数值列进行统计计算

frame.describe(include=['number'])

frame.describe(include='all')#'all'不是列表


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
实际上，idxmin和idxmax就是NumPy中的argmin和argmax。
value_counts() 数值计数
value_counts()计算一维度数据结构的直方图。
Can you write Python code for it?

data = np.random.randint(0, 7, size=50)
data

s = pd.Series(data)

s.value_counts()

pd.value_counts(data) #也是全局方法


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
对于DataFrame来说，join方法默认会应用到索引和列名。
Can you write Python code for it?

df

df2 = df.iloc[:5,:2]

df2

df.align(df2, join='inner')

df.align(df2)


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
对于DataFrame来说，你可以同时改变列名和索引值。
Can you write Python code for it?

df

df.reindex(index=['c', 'f', 'b'], columns=['three', 'two', 'one'])


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
对单个元素应用Python方法
既然不是所有的方法都能被向量化(接收NumPy数组，返回另一个数组或者值),但是DataFrame内置的applymap()和Series的map()方法能够接收任意的接收一个值且返回一个值的Python方法。
Can you write Python code for it?

df4 = pd.DataFrame(np.random.randn(4, 3),index=['a','b','c','d'],columns=['one', 'two', 'three'])
df4

f = lambda x:len(str(x))

df4['one'].map(f)

df4.applymap(f)


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
属性和 ndarray
pandas对象有很多属性，你可以通过这些属性访问数据。

shape: 显示对象的维度，同ndarray
坐标label

Series: index

DataFrame: index(行)和columns

Panel:  items, major_axis and minor_axis

可以通过属性进行安全赋值。
Can you write Python code for it?

df[:2]

df.columns = [x.lower() for x in df.columns] #将列名重置为小写

df


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
所有的这些方法都有skipna参数，含义是计算过程中是否剔除缺失值，skipna默认值为True。
Can you write Python code for it?

df.sum(0, skipna=False)

df.sum(axis=1, skipna=True)


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
按照值排序
Series.sort_values(), DataFrame.sort_values()用于按照值进行排序，参数有by
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
数据类型的默认值
整型的默认值蕾西int64，浮点型的默认类型是float64，和你用的是32位还是64位的系统无关。
Can you write Python code for it?

pd.DataFrame([1, 2], columns=['a']).dtypes

pd.DataFrame({'a': [1, 2]}).dtypes

pd.DataFrame({'a': 1}, index=list(range(2))).dtypes


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
最大/最小值对应的index
idxmin(), idxmax()返回最小值和最大值对应的index。
Can you write Python code for it?

s1 = pd.Series(np.random.randn(5))
s1

s1.idxmin(), s1.idxmax() #最小值：-0.296405, 最大值：1.735420

df1 = pd.DataFrame(np.random.randn(5,3), columns=list('ABC'))
df1

df1.idxmin(axis=0)

df1.idxmax(axis=1)


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
最小/最大值
Series有nsmallest() nlargest()方法能够返回最小或最大的n个值。如果Series对象很大，这两种方法会比先排序后使用head()方法快很多。
Can you write Python code for it?

s = pd.Series(np.random.permutation(10))
s

s.sort_values()

s.nsmallest(3)

s.nlargest(3)


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
注意 f g h三个方法中DataFrame都是作为第一个参数。如果DataFrame作为第二个参数呢？方法是为pipe提供(callable, data_keyword)，pipe会自动调用DataFrame对象。
比如，使用statsmodels处理回归问题，他们的API期望第一个参数是公式，第二个参数是DataFrame对象data。我们使用pipe传递(sm.poisson, 'data'):

pipe灵感来自于Unix中伟大的艺术：管道。pandas中pipe()的实现很简洁，推荐阅读源代码pd.DataFrame.pipe
基于行或者列的函数应用
任意函数都可以直接对DataFrame或Panel某一坐标轴进行直接操纵，只需要使用apply()方法即可，同描述性统计方法一样，apply()方法接收axis参数。
Can you write Python code for it?

df.apply(np.mean)

df.apply(np.mean, axis=1)

df.apply(lambda x: x.max() - x.min())

df.apply(np.cumsum)

df.apply(np.exp)


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
注意cumsum() cumprod()方法 保留NA值的位置。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
注意：
在使用equals()方法进行比较时，两个对象如果数据不一致必为False。甚至是如果label的顺序不一致也返回False。
Can you write Python code for it?

df1 = pd.DataFrame({'c':['f',0,np.nan]})
df1

df2 = pd.DataFrame({'c':[np.nan, 0, 'f']}, index=[2,1,0])
df2

df1.equals(df2)

df1.equals(df2.sort_index()) #对df2的索引排序，然后再比较


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
离散区间也可以用极限定义
Can you write Python code for it?

arr = np.random.randn(20)

factor = pd.cut(arr, [-np.inf, 0, np.inf])

factor


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
统计相关 的方法
Series, DataFrame和Panel内置了许多计算统计相关指标的方法。这些方法大致分为两类：
* 返回低维结果，比如sum(),mean(),quantile()
* 返回同原对象同样大小的对象，比如cumsum(), cumprod()
总体来说，这些方法接收一个坐标轴参数:
* Series不需要坐标轴参数
* DataFrame 默认axis=0(index), axis=1(columns)
* Panel 默认axis=1(major), axis=0(items), axis=2(minor)
Can you write Python code for it?

df

df.mean() #axis=0, 计算每一列的平均值

df.mean(1) #计算每一行的平均值


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
还好pandas提供了equals()方法解决上面NaN之间不想等的问题。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
迭代操作 Iteration
pandas中迭代操作依赖于具体的对象。迭代Series对象时类似迭代数组,产生的是值，迭代DataFrame或Panel对象时类似迭代字典的键值。
一句话，(for i in object)产生:
* Series: 值
* DataFrame: 列名
* Panel: item名
看一下例子吧：
Can you write Python code for it?

df = pd.DataFrame({'col1' : np.random.randn(3), 'col2' : np.random.randn(3)},
                  index=['a', 'b', 'c'])
df

for col in df: #产生的是列名
    print col


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
通过na_position参数处理NA值。
Can you write Python code for it?

s[2] = np.nan

s.sort_values()

s.sort_values(na_position='first') #将NA放在前面


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
重命名索引值
rename() 方法可以对索引值重新命名，命名方式可以是字典或Series，也可以是任意的方法。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
重索引时顺便填充数值
reindex()方法还有一个method参数，用于填充数值,method取值如下：
* pad/ffill: 使用后面的值填充数值
* bfill/backfill: 使用前面的值填充数值
* nearest: 使用最近的索引值进行填充
以Series为例，看一下：
Can you write Python code for it?

rng = pd.date_range('1/3/2000', periods=8)

ts = pd.Series(np.random.randn(8), index=rng)

ts2 = ts[[0, 3, 6]]

ts

ts2

ts2.reindex(ts.index)

ts2.reindex(ts.index, method='ffill') #索引小那一行的数值填充NaN

ts2.reindex(ts.index, method='bfill') #索引大的非NaN的数值填充NaN

ts2.reindex(ts.index, method='nearest')


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
默认describe()只包含25%, 50%, 75%, 也可以通过percentiles参数进行指定。
Can you write Python code for it?

series.describe(percentiles=[.05, .25, .75, .95])


I am working on the file "05其他/pandas文档-zh-master/数据结构的内置方法.ipynb".
The first task is:
默认情况下修改的仅仅是副本，如果想对原对象索引值修改，inplace=True.
在0.18.0版本中，rename方法也能修改Series.name
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
==和列表对象一起使用 Special use of the == operator with list objects
可以使用==/!=将列表和列名直接进行比较，等价于使用in/not in.
三种方法功能等价： ==/!= VS in/not in VS isin()/~isin()
Can you write Python code for it?

df.query('b==["a", "b", "c"]')

df[df.b.isin(["a", "b", "c"])] #Python语法

df.query('c == [1, 2]')

df.query('c != [1, 2]')

df.query('[1, 2] in c') #使用in

df.query('[1, 2] not in c')

df[df.c.isin([1, 2])] #Python语法


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
DataFrame可以在行或者列上扩充数据
Can you write Python code for it?

dfi = pd.DataFrame(np.arange(6).reshape(3,2),columns=['A','B'])
dfi

dfi.loc[:,'C']=dfi.loc[:,'A'] #对列进行扩充

dfi

dfi.loc[3]=5 #对行进行扩充

dfi


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
DataFrame示例：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
Index对象 Index objects
pandas中的Index类和它的子类可以被当做一个序列可重复集合(ordered multiset)，允许数据重复。然而，如果你想把一个有重复值Index对象转型为一个集合这是不可以的。创建Index最简单的方法就是通过传递一个列表或者其他序列创建。
Can you write Python code for it?

index = pd.Index(['e', 'd', 'a', 'b'])
index

'd' in index


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
MultiIndex query() 语法
对于DataFrame对象，可以使用MultiIndex，如同操作列名一样。
Can you write Python code for it?

n = 10

colors = np.random.choice(['red', 'green'], size=n)

foods = np.random.choice(['eggs', 'ham'], size=n)

colors

foods

index = pd.MultiIndex.from_arrays([colors, foods], names=['color', 'food'])

df = pd.DataFrame(np.random.randn(n,2), index=index)

df

df.query('color == "red"')


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
Python和Numpy的索引操作符[]和属性操作符‘.’能够快速检索pandas数据。
然而，这两种方式的效率在pandas中可能不是最优的，我们推荐使用专门优化过的pandas数据检索方法。而这些方法则是本节要介绍的。
多种索引方式
pandas支持三种不同的索引方式：
* .loc 基于label进行索引，当然也可以和boolean数组一起使用。‘.loc’接受的输入：
*  一个单独的label，比如5、'a'，注意，这里的5是index值，而不是整形下标
*  label列表或label数组，比如['a', 'b', 'c']
* .iloc 是基本的基于整数位置(从0到axis的length-1)的，当然也可以和一个boolean数组一起使用。当提供检索的index越界时会有IndexError错误，注意切片索引(slice index)允许越界。
* .ix 支持基于label和整数位置混合的数据获取方式。默认是基本label的. .ix是最常用的方式，它支持所有.loc和.iloc的输入。如果提供的是纯label或纯整数索引，我们建议使用.loc或 .iloc。
以 .loc为例看一下使用方式：
对象类型  |  Indexers
Series | s.loc[indexer]
DataFrame | df.loc[row_indexer, column_indexer]
Panel | p.loc[item_indexer, major_indexer, minor_indexer]
最基本的索引和选择
最基本的选择数据方式就是使用[]操作符进行索引，
对象类型 |  Selection | 返回值类型
Series | series[label],这里的label是index名 | 常数
DataFrame| frame[colname],使用列名 | Series对象，相应的colname那一列
Panel | panel[itemname] | DataFrame对象,相应的itemname那一个
下面用示例展示一下
Can you write Python code for it?

dates = pd.date_range('1/1/2000', periods=8)
dates

df  = pd.DataFrame(np.random.randn(8,4), index=dates, columns=list('ABCD'))
df

panel = pd.Panel({'one':df, 'two':df-df.mean()})
panel


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
Series使用index索引
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
[]不但可以检索，也可以赋值
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
iloc同样也可以进行赋值
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
in 和not in 运算符 The in and not in operators
query()也支持Python中的in和not in运算符，实际上是底层调用isin
Can you write Python code for it?

df = pd.DataFrame({'a': list('aabbccddeeff'), 'b': list('aaaabbbbcccc'),
                  'c': np.random.randint(5, size=12),
                  'd': np.random.randint(9, size=12)})

df

df.query('a in b')

df[df.a.isin(df.b)]

df[~df.a.isin(df.b)]

df.query('a in b and c < d') #更复杂的例子

df[df.b.isin(df.a) & (df.c < df.d)] #Python语法


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
lookup()方法 The lookup()方法
输入行label和列label，得到一个numpy数组，这就是lookup方法的功能。
Can you write Python code for it?

dflookup = pd.DataFrame(np.random.randn(20, 4), columns=list('ABCD'))
dflookup

dflookup.lookup(list(range(0,10,2)), ['B','C','A','B','D'])


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
query()方法 The query() Method (Experimental)
DataFrame对象拥有query方法，允许使用表达式检索。
比如，检索列'b'的值介于列‘a’和‘c’之间的行。
注意： 需要安装numexptr。
Can you write Python code for it?

n = 10

df = pd.DataFrame(np.random.randn(n, 3), columns=list('abc'))

df

df[(df.a<df.b) & (df.b<df.c)]

df.query('(a < b) & (b < c)') #


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
query()的性能
DataFrame.query()底层使用numexptr，所以速度要比Python快,特别时当DataFrame对象非常大时。

重复数据的确定和删除 Duplicate Data
如果你想确定和去掉DataFrame对象中重复的行，pandas提供了两个方法：duplicated和drop_duplicates. 两个方法的参数都是列名。
* duplicated 返回一个布尔向量，长度等于行数，表示每一行是否重复
* drop_duplicates 则删除重复的行
默认情况下，首次遇到的行被认为是唯一的，以后遇到内容相同的行都被认为是重复的。不过两个方法都有一个keep参数来确定目标行是否被保留。
* keep='first'(默认)：标记/去掉重复行除了第一次出现的那一行
* keep='last': 标记/去掉重复行除了最后一次出现的那一行
* keep=False: 标记/去掉所有重复的行
Can you write Python code for it?

df2 = pd.DataFrame({'a': ['one', 'one', 'two', 'two', 'two', 'three', 'four'],
                        'b': ['x', 'y', 'x', 'y', 'x', 'x', 'x'],
                      'c': np.random.randn(7)})

df2

df2.duplicated('a') #只观察列a的值是否重复

df2.duplicated('a', keep='last')

df2.drop_duplicates('a')

df2.drop_duplicates('a', keep='last')

df2.drop_duplicates('a', keep=False)


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
sample()默认进行的无放回抽样，可以利用replace=True参数进行可放回抽样。
Can you write Python code for it?

s

s.sample(n=6,replace=False)

s.sample(6,replace=True)


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
select()方法 The select() Method
Series, DataFrame和Panel都有select()方法来检索数据，这个方法作为保留手段通常其他方法都不管用的时候才使用。select接受一个函数(在label上进行操作)作为输入返回一个布尔值。
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(10, 3), columns=list('ABC'))

df.select(lambda x: x=='A', axis=1)


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
where()方法 The where() Method and Masking
使用布尔向量对Series对象查询时通常返回的是对象的子集。如果想要返回的shape和原对象相同，可以使用where方法。
使用布尔向量对DataFrame对象查询返回的shape和原对象相同，这是因为底层用的where方法实现。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
也可以给[]传递一个column name组成的的list，形如df[[col1,col2]], 如果给出的某个列名不存在，会报错
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
也可以输入小数，则会随机选择N*frac个样本, 结果进行四舍五入
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
也可以进行赋值操作
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
使用Label进行检索
警告：
.loc要求检索时输入必须严格遵守index的类型，一旦输入类型不对，将会引起TypeError。
Can you write Python code for it?

df1 = pd.DataFrame(np.random.rand(5,4), columns=list('ABCD'), index=pd.date_range('20160101',periods=5))
df1

df1.loc[2:3]


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
使用isin方法检索 Indexing with isin
isin(is in)
对于Series对象来说，使用isin方法时传入一个列表，isin方法会返回一个布尔向量。布尔向量元素为1的前提是列表元素在Series对象中存在。看起来比较拗口，还是看例子吧：
Can you write Python code for it?

s = pd.Series(np.arange(5), index=np.arange(5)[::-1],dtype='int64')

s

s.isin([2,4,6])

s[s.isin([2,4,6])]


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
使用where方法
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
使用布尔数组检索
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
使用赋值的方式扩充对象 Setting With Enlargement
用.loc/.ix/[]对不存在的键值进行赋值时，将会导致在对象中添加新的元素，它的键即为赋值时不存在的键。
对于Series来说，这是一种有效的添加操作。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
再来看看DataFramed的例子
Can you write Python code for it?

df1 = pd.DataFrame(np.random.randn(6,4), index=list('abcdef'),columns=list('ABCD'))
df1

df1.loc[['a','b','c','d'],:]

df1.loc[['a','b','c','d']] #可以省略 ':'


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
利用列表解析和map方法能够产生更加复杂的选择标准。
Can you write Python code for it?

df2 = pd.DataFrame({'a' : ['one', 'one', 'two', 'three', 'two', 'one', 'six'],
                    'b' : ['x', 'y', 'y', 'x', 'y', 'x', 'x'],
                     'c' : np.random.randn(7)})
df2

criterion = df2['a'].map(lambda x:x.startswith('t'))

df2[criterion]

df2[[x.startswith('t') for x in df2['a']]]

df2[criterion & (df2['b'] == 'x')]


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
如果index没有名字，可以给他们命名
Can you write Python code for it?

df.index.names = [None, None]

df

df.query('ilevel_0 == "red"')


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
如果是对DataFrame对象进行有权重采样，一个简单 的方法是新增一列用于表示每一行的权重
Can you write Python code for it?

df2 = pd.DataFrame({'col1':[9,8,7,6], 'weight_column':[0.5, 0.4, 0.1, 0]})
df2

df2.sample(n=3,weights='weight_column')


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
对齐
where方法会将输入的布尔条件对齐，因此允许部分检索时的赋值。
Can you write Python code for it?

df2 = df.copy()

df2[df2[1:4] >0]=3

df2

df2 = df.copy()

df2.where(df2>0, df2['A'], axis='index')


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
布尔运算符 Boolean Operators
可以使用not或者~对布尔表达式进行取非。
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(n, 3), columns=list('abc'))
df

df['bools']=np.random.randn(len(df))>0.5

df

df.query('bools')

df.query('not bools')

df.query('not bools') == df[~df.bools]


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
形似字典的get()方法
Serires, DataFrame和Panel都有一个get方法来得到一个默认值。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
我们也可以使用random_state参数 为sample内部的随机数生成器提供种子数。
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
我们使用最基本的[]操作符
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
标量值的快速获取和赋值
如果仅仅想获取一个元素，使用[]未免太繁重了。pandas提供了快速获取一个元素的方法：at和iat. 适用于Series、DataFrame和Panel。
如果loc方法，at方法的合法输入是label，iat的合法输入是整型。
Can you write Python code for it?

s.iat[5]

df.at[dates[5],'A']

df.iat[3,0]


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
比较下面两种访问方式：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
注意下面两个例子的区别：
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
注意下面两个示例，输出是相同的，因为使用了相同的种子数
Can you write Python code for it?

df4.sample(n=2, random_state=2)

df4.sample(n=2,random_state=2)

df4.sample(n=2,random_state=3)


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
注意：由于sample默认进行的是无放回抽样，所以输入必须n<=行数，除非进行可放回抽样。
Can you write Python code for it?

s

s.sample(7) #7不行

s.sample(7,replace=True)


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
细心地你一定发现了，index='20160104'那一行也被检索出来了，没错，loc检索时范围是闭集合[start,end].
整型可以作为label检索，这是没问题的，不过要记住此时整型表示的是label而不是index中的下标！
.loc操作是检索时的基本操作，以下输入格式都是合法的：
* 一个label，比如：5、'a'. 记住这里的5表示的是index中的一个label而不是index中的一个下标。
* label组成的列表或者数组比如['a','b','c']
* 切片，比如'a':'f'.注意loc中切片范围是闭集合！
* 布尔数组
Can you write Python code for it?

s1 = pd.Series(np.random.randn(6), index=list('abcdef'))
s1

s1.loc['c':]

s1.loc['b']


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
结合isin方法和any() all()可以对DataFrame进行快速查询。比如选择每一列都符合标准的行:
Can you write Python code for it?

values = {'ids': ['a', 'b'], 'ids2': ['a', 'c'], 'vals': [1, 3]}

row_mark = df.isin(values).all(1)

df[row_mark]

row_mark = df.isin(values).any(1)

df[row_mark]


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
结合loc、iloc等方法可以检索多个坐标下的数据.
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
表达式任意复杂都没关系。
Can you write Python code for it?

shorter = df.query('a<b<c and (not bools) or bools>2')
shorter

longer = df[(df.a < df.b) & (df.b < df.c) & (~df.bools) | (df.bools > 2)]
longer

shorter == longer


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
输入string进行检索没问题
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
输入一个字典的情形：
Can you write Python code for it?

values = {'ids': ['a', 'b'], 'vals': [1, 3]}

df.isin(values)


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
输入有切片，有整数，如果越界同样不能处理
Can you write Python code for it?


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
选择随机样本 Selecting Random Samples
使用sample()方法能够从行或者列中进行随机选择，适用对象包括Series、DataFrame和Panel。sample()方法默认对行进行随机选择，输入可以是整数或者小数。
Can you write Python code for it?

s = pd.Series([0,1,2,3,4,5])
s

s.sample()

s.sample(n=6)

s.sample(3) #直接输入整数即可


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
通过属性访问 把column作为DataFrame对象的属性
可以直接把Series的index、DataFrame中的column、Panel中的item作为这些对象的属性使用，然后直接访问相应的index、column、item
Can you write Python code for it?

sa = pd.Series([1,2,3],index=list('abc'))
dfa = df.copy()

sa

sa.b #直接把index作为属性

dfa

dfa.A

panel.one

sa

sa.a = 5
sa

sa

dfa.A=list(range(len(dfa.index))) # ok if A already exists

dfa

dfa['A'] = list(range(len(dfa.index)))  # use this form to create a new column
dfa


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
链式索引中顺序也很重要
此外，在链式表达式中，不同的顺序也可能导致不同的结果。这里的顺序指的是检索时行和列的顺序。
Can you write Python code for it?

dfb = pd.DataFrame({'a' : ['one', 'one', 'two',
                               'three', 'two', 'one', 'six'],
                       'c' : np.arange(7)})
dfb

dfb['c'][dfb.a.str.startswith('o')] = 42 #虽然会引起SettingWithCopyWarning 但也能得到正确结果

pd.set_option('mode.chained_assignment','warn')
dfb[dfb.a.str.startswith('o')]['c'] = 42 #这实际上是对副本赋值！


I am working on the file "05其他/pandas文档-zh-master/检索 ,查询数据.ipynb".
The first task is:
默认情况下，where方法并不会修改原始对象，它返回的是一个修改过的原始对象副本，如果你想直接修改原始对象，方法是将inplace参数设置为True
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(6,5), index=list('abcdef'), columns=list('ABCDE'))
df_orig = df.copy()

df_orig.where(df < 0, -df, inplace=True);

df_orig


I am working on the file "06 - Natural Language Processing.ipynb".
The first task is:
Creating a Corpus
Building models requires gathering multiple documents and performing the processing steps on them that we showed above. We've used a tool called Baleen to ingest data from RSS feeds for the past year. (It currently contains 1,154,100 posts for 373 feeds after 5,566 jobs).
We've provided a small sample of the corpus to start playing with the tool. It has saved documents in the following structure:

Each file stored in the directory of its category
One document per file, stored as a pickle
Document is a list of paragraphs
Paragraph is a list of sentences
Sentence is a list of (token, tag) tuples

We can then create a reader to automatically fetch data from our corpus. This is a bit more complex, but necessary. Also note that we add our normalization process here as well, just so we don't have to repeat steps later on.
Can you write Python code for it?

import string
import pickle

from nltk.corpus.reader.api import CorpusReader
from nltk.corpus.reader.api import CategorizedCorpusReader

CORPUS_PATH = "data/baleen_sample"
PKL_PATTERN = r'(?!\.)[a-z_\s]+/[a-f0-9]+\.pickle'
CAT_PATTERN = r'([a-z_\s]+)/.*'

class PickledCorpus(CategorizedCorpusReader, CorpusReader):

    def __init__(self, root, fileids=PKL_PATTERN, cat_pattern=CAT_PATTERN):
        CategorizedCorpusReader.__init__(self, {"cat_pattern": cat_pattern})
        CorpusReader.__init__(self, root, fileids)

        self.punct = set(string.punctuation) | {'“', '—', '’', '”', '…'}
        self.stopwords = set(nltk.corpus.stopwords.words('english'))
        self.wordnet = nltk.WordNetLemmatizer()

    def _resolve(self, fileids, categories):
        if fileids is not None and categories is not None:
            raise ValueError("Specify fileids or categories, not both")

        if categories is not None:
            return self.fileids(categories=categories)

        if fileids is None:
            return self.fileids()

        return fileids

    def lemmatize(self, token, tag):
        token = token.lower()

        if token not in self.stopwords:
            if not all(c in self.punct for c in token):
                tag =  {
                    'N': wn.NOUN,
                    'V': wn.VERB,
                    'R': wn.ADV,
                    'J': wn.ADJ
                }.get(tag[0], wn.NOUN)
                return self.wordnet.lemmatize(token, tag)

    def tokenize(self, doc):
        # Expects a preprocessed document, removes stopwords and punctuation
        # makes all tokens lowercase and lemmatizes them.
        return list(filter(None, [
            self.lemmatize(token, tag)
            for paragraph in doc
            for sentence in paragraph
            for token, tag in sentence
        ]))

    def docs(self, fileids=None, categories=None):
        # Resolve the fileids and the categories
        fileids =...


I am working on the file "06 - Natural Language Processing.ipynb".
The first task is:
Operationalization
At this point we can save our best performing model to disk and use it to classify new text.
The most important thing to remember is that the input to our model needs to be identical to the input we trained our model upon. Because we preprocessed our text in the experimental phase, we have to preprocess it before we make predictions on it as well.
Can you write Python code for it?

def build_model(path, corpus):
    model = Pipeline([
        ('tfidf', TfidfVectorizer(tokenizer=identity, preprocessor=None, lowercase=False)),
        ('svm', SGDClassifier(loss='log')),
    ])

    # Train model on the entire data set
    X = list(corpus.docs(categories=hobbies))
    y = list(corpus.labels(categories=hobbies))
    model.fit(X, y)

    with open(path, 'wb') as f:
        pickle.dump(model, f)

build_model('data/hobbies.classifier', corpus)

# We can now load our model from disk
with open('data/hobbies.classifier', 'rb') as f:
    model = pickle.load(f)

# Let's create a normalization method for fetching URL content
# that our model expects, based on our methods above.
def fetch(url):
    html = requests.get(url)
    text = extract(html.text)
    tokens = tokenize(text)
    tags = tag(tokens)
    lemmas = lemmatize(tags)
    return list(normalize(lemmas))

def predict(url):
    text = fetch(url)
    probs = zip(model.classes_, model.predict_proba([text])[0])
    label = model.predict([text])[0]

    print("y={}".format(label))
    for cls, prob in sorted(probs, key=lambda x: x[1]):
        print("  {}: {:0.3f}".format(cls, prob))

predict("http://minimalistbaker.com/5-ingredient-white-chocolate-truffles/")


I am working on the file "06 - Natural Language Processing.ipynb".
The first task is:
Step 5: Normalize
Normalization reduces the number of tokens that we pass to our analysis, allowing us to do more effective language inference.
Can you write Python code for it?

from nltk.corpus import wordnet as wn

lemmatizer = nltk.WordNetLemmatizer()

def tagwn(tag):
    return {
        'N': wn.NOUN,
        'V': wn.VERB,
        'R': wn.ADV,
        'J': wn.ADJ
    }.get(tag[0], wn.NOUN)

def lemmatize(tagged_sents):
    for sent in tagged_sents:
        for token, tag in sent:
            yield lemmatizer.lemmatize(token, tagwn(tag))

story = list(lemmatize(story))

print(story)

from string import punctuation
from nltk.corpus import stopwords

punctuation = set(punctuation)
stopwords = set(stopwords.words('english'))

def normalize(tokens):
    for token in tokens:
        token = token.lower()
        if not all(char in punctuation for char in token):
            if token not in stopwords:
                yield token

story = list(normalize(story))

print(story)


I am working on the file "06 - Natural Language Processing.ipynb".
The first task is:
Step Four: Tag Text
Tagging adds information to the data structure we have -- namely the word class for each word (e.g. is it a Noun, Verb, Adjective, etc.). Note that tagging needs a complete sentence to work effectively.
After we have tagged our text, we have completed the non-destructive operations on our text string, it is at this point that the text should be saved as a pickle to disk for use in downstream processing.
Can you write Python code for it?

def tag(sents):
    for sent in sents:
        yield list(nltk.pos_tag(sent))

story = list(tag(story))

for sent in story: print(sent)


I am working on the file "06 - Natural Language Processing.ipynb".
The first task is:
Step Two: Clean Up Data
The HTML that we fetched contains navigation, advertisements, and markup not related to the text. We need to clean it up to extract only the part of the document we're interested in analyzing.
Note that this is also the point that we should consider larger document structures like chapters, sections, or paragraphs. If we want to consider paragraphs, the extract function should return a list of strings that each represent a paragraph.
Can you write Python code for it?

from bs4 import BeautifulSoup
from readability.readability import Document

def extract(html):
    article = Document(html).summary()
    soup = BeautifulSoup(article, 'lxml')

    return soup.get_text()

story = extract(story)

print(story)


I am working on the file "06 - Natural Language Processing.ipynb".
The first task is:
Training a Model
We'll build a model that can classify what hobby a document is about based on our sample corpus.
We'll need to add transformers that vectorize our text, and send them to a classification model.
In this case we will evaluate with 12-part cross validation, using the cross_val_predict function and the classifier_report function.
The function cross_val_predict has a similar interface to cross_val_score, but returns, for each element in the input, the prediction that was obtained for that element when it was in the test set. Only cross-validation strategies that assign all elements to a test set exactly once can be used (otherwise, an exception is raised).
Can you write Python code for it?

hobbies = ['gaming', 'cooking', 'sports', 'cinema', 'books', 'do_it_yourself']

X = list(corpus.docs(categories=hobbies))
y = list(corpus.labels(categories=hobbies))

# Models
from sklearn.linear_model import SGDClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.ensemble import RandomForestClassifier

# Transformers
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline

# Evaluation
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import classification_report

def identity(words):
    return words

# SVM Classifier
svm = Pipeline([
        ('tfidf', TfidfVectorizer(tokenizer=identity, preprocessor=None, lowercase=False)),
        ('svm', SGDClassifier()),
    ])

yhat = cross_val_predict(svm, X, y, cv=12)
print(classification_report(y, yhat))

# Logistic Regression
logit = Pipeline([
        ('tfidf', TfidfVectorizer(tokenizer=identity, preprocessor=None, lowercase=False)),
        ('logit', LogisticRegression()),
    ])

yhat = cross_val_predict(logit, X, y, cv=12)
print(classification_report(y, yhat))

# Naive Bayes
nbayes = Pipeline([
        ('tfidf', TfidfVectorizer(tokenizer=identity, preprocessor=None, lowercase=False)),
        ('nbayes', MultinomialNB()),
    ])

yhat = cross_val_predict(nbayes, X, y, cv=12)
print(classification_report(y, yhat))

# Random Forest
trees = Pipeline([
        ('tfidf', TfidfVectorizer(tokenizer=identity, preprocessor=None, lowercase=False)),
        ('trees', RandomForestClassifier()),
    ])

yhat = cross_val_predict(trees, X, y, cv=12)
print(classification_report(y, yhat))


I am working on the file "06 - Natural Language Processing.ipynb".
The first task is:
Visualizing a Corpus
TSNE - stochastic neighbor embedding, is a useful mechanism for performing high dimensional data visualization on text. We will use our classes to try to visualize groupings of documents on a per-class basis.
Can you write Python code for it?

from sklearn.manifold import TSNE
from sklearn.pipeline import Pipeline
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import CountVectorizer

cluster = Pipeline([
        ('vect', CountVectorizer(tokenizer=lambda x: x, preprocessor=None, lowercase=False)),
        ('svd', TruncatedSVD(n_components=50)),
        ('tsne', TSNE(n_components=2))
    ])

docs = cluster.fit_transform(list(corpus.docs()))

import seaborn as sns
import matplotlib.pyplot as plt

from collections import defaultdict

sns.set_style('whitegrid')
sns.set_context('notebook')

colors = {
    "design": "#e74c3c",
    "tech": "#3498db",
    "business": "#27ae60",
    "gaming": "#f1c40f",
    "politics": "#2c3e50",
    "news": "#bdc3c7",
    "cooking": "#d35400",
    "data_science": "#1abc9c",
    "sports": "#e67e22",
    "cinema": "#8e44ad",
    "books": "#c0392b",
    "do_it_yourself": "#34495e",
}

series = defaultdict(lambda: {'x':[], 'y':[]})
for idx, label in enumerate(corpus.labels()):
    x, y = docs[idx]
    series[label]['x'].append(x)
    series[label]['y'].append(y)

fig = plt.figure(figsize=(12,6))
ax = plt.subplot(111)

for label, points in series.items():
    ax.scatter(points['x'], points['y'], c=colors[label], alpha=0.7, label=label)

# Add a title
plt.title("TSNE Projection of the Baleen Corpus")

# Remove the ticks
plt.yticks([])
plt.xticks([])

# Add the legend
# Shrink current axis by 20%
box = ax.get_position()
ax.set_position([box.x0, box.y0, box.width * 0.8, box.height])

# Put a legend to the right of the current axis
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))


I am working on the file "06 - Reshaping data.ipynb".
The first task is:

REMEMBER:

**stack**: make your data *longer* and *smaller*

**unstack**: make your data *shorter* and *wider*

Mimick pivot table
To better understand and reason about pivot tables, we can express this method as a combination of more basic steps. In short, the pivot is a convenient way of expressing the combination of a groupby and stack/unstack.
Can you write Python code for it?

df = pd.read_csv("data/titanic.csv")

df.head()

df.pivot_table(index='Pclass', columns='Sex',
               values='Survived', aggfunc='mean')


I am working on the file "06 - Reshaping data.ipynb".
The first task is:

REMEMBER:

By default, `pivot_table` takes the **mean** of all values that would end up into one cell. However, you can also specify other aggregation functions using the `aggfunc` keyword.

Can you write Python code for it?

df.pivot_table(index='Sex', columns='Pclass',
               values='Fare', aggfunc='max')

df.pivot_table(index='Sex', columns='Pclass',
               values='Fare', aggfunc='count')


I am working on the file "06 - Reshaping data.ipynb".
The first task is:

EXERCISE:

Define a year as a "Superman year" when films of that year feature more Superman characters than Batman characters. How many years in film history have been Superman years?

Can you write Python code for it?

# %load snippets/06 - Reshaping data51.py

# %load snippets/06 - Reshaping data52.py


I am working on the file "06 - Reshaping data.ipynb".
The first task is:

EXERCISE:

Make a pivot table with the survival rates (= number of persons survived / total number of persons) for Pclass vs Sex.

Plot the result as a bar plot.

Can you write Python code for it?

# %load snippets/06 - Reshaping data20.py

# %load snippets/06 - Reshaping data21.py


I am working on the file "06 - Reshaping data.ipynb".
The first task is:

EXERCISE:

Plot the number of actor roles each year and the number of actress roles each year over the whole period of available movie data.

Can you write Python code for it?

# %load snippets/06 - Reshaping data46.py

# %load snippets/06 - Reshaping data47.py

# %load snippets/06 - Reshaping data48.py


I am working on the file "06 - Reshaping data.ipynb".
The first task is:

    EXERCISE: Group the survey data by sites, date of measurement on each site and the quantity measured. List all readings for `site` DR-1; all readings of radiation using the hierchical index.

stack/unstack
stack — shifts last level of hierarchical rows to columns
unstack — does the opposite, i.e. shifts last level of hierarchical columns to rows
Can you write Python code for it?

result = multi['concentration'].unstack()
result


I am working on the file "06 - Reshaping data.ipynb".
The first task is:

    EXERCISE: Join the `readings` and `visited` tables.

    EXERCISE: Pivot the table such that we have sites in rows and different quantities in columns.

Hierarchical index
Hierarchical index of pandas is a way of introducing another dimension to a (two-dimensional) data frame. This is implemented by having multiple levels of the index. Let's look at an example.
Can you write Python code for it?

multi = df.set_index(['subject', 'treatment'])
multi


I am working on the file "06 - Reshaping data.ipynb".
The first task is:

    EXERCISE: Rearange the data frame from last exercise, such that rows contain sites and dates (hierchical index) and columns different quantities. List all readings of radiation.

Formatting data — Case study
Going further with the time series case study test on the AirBase (The European Air quality dataBase) data.
One of the actual downloaded raw data files of AirBase is included in the repo:
Can you write Python code for it?


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
A better representation might be one where the different subjects are in rows, the applied treatments are in columns and outcomes are in the data frame values.
<img src="img/stack.png" width=70%>
You can achieve this by pivot function:
Can you write Python code for it?

pivoted = df.pivot(index='subject', columns='treatment', values='concentration')
pivoted


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
And try the same pivot (no worries about the try-except, this is here just used to catch a loooong error):
Can you write Python code for it?

try:
    df.pivot(index='Sex', columns='Pclass', values='Fare')
except Exception as e:
    print("Exception!", e)


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Assume we have a DataFrame like the above. The observations (the average Fare people payed) are spread over different columns. In a tidy dataset, each observation is stored in one row. To obtain this, we can use the melt function:
Can you write Python code for it?


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Back to our case study
We can now use stack and some other functions to create a timeseries from the original dataframe:
Can you write Python code for it?

colnames = ['date'] + [item for pair in zip(["{:02d}".format(i) for i in range(24)], ['flag']*24) for item in pair]

data = pd.read_csv("data/BETR8010000800100hour.1-1-1990.31-12-2012",
                   sep='\t', header=None, na_values=[-999, -9999], names=colnames)
data = data.drop('flag', axis=1)

data.head()


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Case study: air quality data of European monitoring stations (AirBase)
Going further with the time series case study test on the AirBase (The European Air quality dataBase) data: the actual data downloaded from the Airbase website did not look like a nice csv file (data/airbase_data.csv).
One of the actual downloaded raw data files of AirBase is included in the repo:
Can you write Python code for it?


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Exercises: use the reshaping methods with the movie data
These exercises are based on the PyCon tutorial of Brandon Rhodes (so credit to him!) and the datasets he prepared for that. You can download these data from here: titles.csv and cast.csv and put them in the /data folder.
Can you write Python code for it?

cast = pd.read_csv('data/cast.csv')
cast.head()

titles = pd.read_csv('data/titles.csv')
titles.head()


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Indexing on the second index only may be slightly involved:
Can you write Python code for it?


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Interested in Grand totals?
Can you write Python code for it?

# sum columns
excelample_pivot.sum(axis=1)

# sum rows
excelample_pivot.sum(axis=0)


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Just reading the tab-delimited data:
Can you write Python code for it?


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Melt
The melt function performs the inverse operation of a pivot. This can be used to make your frame longer, i.e. to make a tidy version of your data.
Can you write Python code for it?

pivoted = df.pivot_table(index='Sex', columns='Pclass', values='Fare').reset_index()
pivoted.columns.name = None

pivoted


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Mimick melt
Like the pivot table above, we can now also obtain the result of melt with stack/unstack.
Let's use the same pivoted frame as above, and look at the final melt result:
Can you write Python code for it?

pivoted = df.pivot_table(index='Sex', columns='Pclass', values='Fare').reset_index()
pivoted.columns.name = None
pivoted

pd.melt(pivoted, id_vars=['Sex'], var_name='Pclass', value_name='Fare')


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Note that it creates a standard data frame with "flat" index.
Can you write Python code for it?


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Pivot is just reordering your data
Small subsample of the titanic dataset:
Can you write Python code for it?

df = pd.DataFrame({'Fare': [7.25, 71.2833, 51.8625, 30.0708, 7.8542, 13.0],
                   'Pclass': [3, 1, 1, 2, 3, 2],
                   'Sex': ['male', 'female', 'male', 'female', 'female', 'male'],
                   'Survived': [0, 1, 0, 1, 0, 1]})

df

df.pivot(index='Pclass', columns='Sex', values='Fare')

df.pivot(index='Pclass', columns='Sex', values='Survived')


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Pivoting data
Cfr. excel
People who know Excel, probably know the Pivot functionality:

The data of the table:
Can you write Python code for it?

excelample = pd.DataFrame({'Month': ["January", "January", "January", "January",
                                  "February", "February", "February", "February",
                                  "March", "March", "March", "March"],
                   'Category': ["Transportation", "Grocery", "Household", "Entertainment",
                                "Transportation", "Grocery", "Household", "Entertainment",
                                "Transportation", "Grocery", "Household", "Entertainment"],
                   'Amount': [74., 235., 175., 100., 115., 240., 225., 125., 90., 260., 200., 120.]})

excelample

excelample_pivot = excelample.pivot(index="Category", columns="Month", values="Amount")
excelample_pivot


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Reshaping data with stack and unstack
Pivoting
Data is often stored in CSV files or databases in so-called “stacked” or “record” format:
Can you write Python code for it?

df = pd.DataFrame({'subject':['A', 'A', 'B', 'B'],
                   'treatment':['CH', 'DT', 'CH', 'DT'],
                   'concentration':range(4)},
                 columns=['subject', 'treatment', 'concentration'])
df


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Reshaping with stack and unstack
The docs say:

Pivot a level of the (possibly hierarchical) column labels, returning a
DataFrame (or Series in the case of an object with a single level of
column labels) having a hierarchical index with a new inner-most level
of row labels.

Indeed...
<img src="img/schema-stack.svg" width=50%>
Before we speak about hierarchical index, first check it in practice on the following dummy example:
Can you write Python code for it?


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
Since pivot is just restructuring data, where would both values of Fare for the same combination of Sex and Pclass need to go?
Well, they need to be combined, according to an aggregation functionality, which is supported by the functionpivot_table

NOTE:

**Pivot** is purely restructuring: a single value for each index/column combination is required.

Pivot tables - aggregating while pivoting
Can you write Python code for it?

df = pd.read_csv("data/titanic.csv")

df.pivot_table(index='Sex', columns='Pclass', values='Fare')


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
So far, so good...
Let's now use the full titanic dataset:
Can you write Python code for it?

df = pd.read_csv("data/titanic.csv")

df.head()


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
This does not work, because we would end up with multiple values for one cell of the resulting frame, as the error says: duplicated values for the columns in the selection. As an example, consider the following rows of our three columns of interest:
Can you write Python code for it?


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
To use stack/unstack, we need the values we want to shift from rows to columns or the other way around as the index:
Can you write Python code for it?

df = df.set_index(['A', 'B'])
df

result = df['C'].unstack()
result

df = result.stack().reset_index(name='C')
df