I am working on the file "Python_Resources/SimulationOptimization.ipynb".
The first task is:
Measure the speed of both fitness function vectorized and not for different population sizes
Can you write Python code for it?

non_vectorized_times = []
vectorized_times = []
p_sizes = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 120, 140, 150, 160, 180, 200, 240, 280, 320, 400, 500, 600]
for p_size in p_sizes:
    population_size = p_size
    variables = InstanceVariables(runs, generations, population_size, mutation_rate,
                                  execution_error, reputation_assignment_error,
                                  private_assessment_error, reputation_update_probability,
                                  socialnorm, cost, benefit)
    non_vector_result = %timeit -o SimulationInstance.fitness_function(0, 1, variables)
    vector_result = %timeit -o SimulationInstanceVectorized.fitness_function(0, 1, variables)
    non_vectorized_times.append(non_vector_result)
    vectorized_times.append(vector_result)

v_times = [time_obj.best for time_obj in vectorized_times]
non_v_times = [time_obj.best for time_obj in non_vectorized_times]
line_vectorized, = plt.plot(p_sizes, v_times, linewidth=1.5, label='Vectorized')
line_non_vectorized, = plt.plot(p_sizes, non_v_times, label='Non vectorized', marker='^')
# plt.ylim((-0.01, 1.05))
# plt.xlim((-0.1, 125))
plt.legend(handles=[line_vectorized, line_non_vectorized], loc='upper left', bbox_to_anchor=(1,1))
plt.grid(b=True, which='both')
plt.rcParams["figure.figsize"][0] = 9
plt.rcParams["figure.figsize"][1] = 5
# plt.xticks([10*i for i in range(13)])
# plt.yticks([0.1*i for i in range(11)])
plt.ylabel('Run time (s)')
plt.xlabel('Population size')
plt.show()


I am working on the file "Python_Resources/SimulationOptimization.ipynb".
The first task is:
Now for the optimized simulation, the time taken is only 1.61 seconds, which is extremely close to the time taken for a population of size 50.
Fitness Function
Let's have a look at the speedup of the fitness function, responsible for determining the average fitness of an agent X against a tournament of other agents.
Can you write Python code for it?

runs = 1
generations = 3*np.power(10, 2)
population_size = 10
socialnorm = [[0, 0], [0, 1]]
mutation_rate = np.power(10.0*population_size, -1.0)

execution_error = 0.08
reputation_assignment_error = 0.01
private_assessment_error = 0.01
reputation_update_probability = 1.0
cost = 1
benefit = 5
variables = InstanceVariables(runs, generations, population_size, mutation_rate,
                              execution_error, reputation_assignment_error,
                              private_assessment_error, reputation_update_probability,
                              socialnorm, cost, benefit)


I am working on the file "Python_Resources/SimulationOptimization.ipynb".
The first task is:
Population optimization
Now let's have a quick look at the speed difference in a population of 50.
Can you write Python code for it?

%%timeit
SSP(50, 3*np.power(10,3), SternJudging, optimized=False)


I am working on the file "Python_Resources/SimulationOptimization.ipynb".
The first task is:
The best case was 2.48 seconds for the optimized simulation. Compared to the 4.73 seconds it took for the unoptimized, we have a almost twice as quick of a simulation. However before we say it is ONLY twice as quick. Let us give the example of a population of size 1000, 20 times larger than what we've tested. Given we are only testing these with 3000 generations compared to the 300 000 generations required for the simulation to be accurate, it is simply to measure the speedup.
Can you write Python code for it?

%%timeit
SSP(1000, 3*np.power(10,3), SternJudging, optimized=False)


I am working on the file "Python_Resources/SimulationOptimization.ipynb".
The first task is:
The best case was 4.73 seconds in the unoptimized simulation, now if we test the optimized simulation.
Can you write Python code for it?

%%timeit
SSP(50, 3*np.power(10,3), SternJudging, optimized=True)


I am working on the file "Python_Resources/SimulationResultVerification.ipynb".
The first task is:
Complete rerun of results with 1...Z loop in each generation
For a simulation with parameters:
R = 8, Z = 20, G = 3x10^3, 48 seconds
R = 8, Z = 20, G = 3x10^4, 432 seconds
R = 8, Z = 40, G = 3x10^3, 92 seconds
Estimated
Z = 20
Estimated R = 8, G = 3x10^5, ~4320 seconds (1.2 hours)
Estimated R = 32 (96//3), G = 3x10^5, ~5 hours
Z = 40
Estimated R = 8, G = 3x10^5, ~9200 seconds (2.5 hours)
Estimated R = 32 (96//3), G = 3x10^5, ~10 hours
Z = 80
Estimated R = 32 (96//3), G = 3x10^5, ~20 hours
Z = 120
Estimated R = 32 (96//3), G = 3x10^5, ~30 hours
Results
Social Norm Simple Standing
Can you write Python code for it?


I am working on the file "Python_Resources/SimulationResultVerification.ipynb".
The first task is:
Stern Judging
Now let us retrieve the results for populations in the ranges 10-150, with simulations at every multiple of 10, and then simulations at multiples of 25 from 150-300.
Let's begin with the social norm of Stern Judging.
Can you write Python code for it?

population_sizes = [5, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120]

stern_judging_results = [0.508516473,0.775682093,0.826424982,0.760320876,0.697826156,0.643724905,0.561129066,0.594354744,0.652819772,0.598121451,0.527160869,0.531084403,0.608492096]

stern_judging_results = np.array(stern_judging_results)
# results_temp_x2
#x = stern_judging_results[:,0]
#y = stern_judging_results[:,1]
plt.plot(population_sizes, stern_judging_results, 'k', population_sizes, stern_judging_results, 'r^', linewidth=1.5)
plt.ylim((-0.01, 1.1))
plt.xlim((-0.01, 125))
plt.xticks([10*i for i in range(13)])
plt.yticks([0.1*i for i in range(11)])
plt.ylabel('Cooperation Index')
plt.xlabel('Population Size')
plt.grid(True)
plt.show()


I am working on the file "Python_Spider_Tutorial_01.ipynb".
The first task is:
3. 用Python简单处理URL
如果要抓取百度上面搜索关键词为Jecvay Notes的网页, 则代码如下
Can you write Python code for it?

import urllib
import urllib.request

data={}
data['word']='Jecvay Notes'

url_values=urllib.parse.urlencode(data)
url="http://www.baidu.com/s?"
full_url=url+url_values

data=urllib.request.urlopen(full_url).read()
data=data.decode('UTF-8')
print(data)


I am working on the file "Python_Spider_Tutorial_01.ipynb".
The first task is:
urllib.request是一个库, 隶属urllib. 点此打开官方相关文档. 官方文档应该怎么使用呢? 首先点刚刚提到的这个链接进去的页面有urllib的几个子库, 我们暂时用到了request, 所以我们先看urllib.request部分. 首先看到的是一句话介绍这个库是干什么用的:

The urllib.request module defines functions and classes which help in opening URLs (mostly HTTP) in a complex world — basic and digest authentication, redirections, cookies and more.

然后把我们代码中用到的urlopen()函数部分阅读完.

urllib.request.urlopen(url, data=None, [timeout, ]*, cafile=None, capath=None, cadefault=False)

重点部分是返回值, 这个函数返回一个 http.client.HTTPResponse 对象, 这个对象又有各种方法, 比如我们用到的read()方法, 这些方法都可以根据官方文档的链接链过去. 根据官方文档所写, 我用控制台运行完毕上面这个程序后, 又继续运行如下代码, 以更熟悉这些乱七八糟的方法是干什么的.
Can you write Python code for it?

a = urllib.request.urlopen(url)
type(a)

a.geturl()

a.info()

a.getcode()


I am working on the file "Python_Spider_Tutorial_02.ipynb".
The first task is:
2. Python的集合
在爬虫程序中, 为了不重复爬那些已经爬过的网站, 我们需要把爬过的页面的url放进集合中, 在每一次要爬某一个url之前, 先看看集合里面是否已经存在. 如果已经存在, 我们就跳过这个url; 如果不存在, 我们先把url放入集合中, 然后再去爬这个页面.
Python提供了set这种数据结构. set是一种无序的, 不包含重复元素的结构. 一般用来测试是否已经包含了某元素, 或者用来对众多元素们去重. 与数学中的集合论同样, 他支持的运算有交, 并, 差, 对称差.
创建一个set可以用 set() 函数或者花括号 {} . 但是创建一个空集是不能使用一个花括号的, 只能用 set() 函数. 因为一个空的花括号创建的是一个字典数据结构. 以下同样是Python官网提供的示例.
Can you write Python code for it?

basket = {'apple', 'orange', 'apple', 'pear', 'orange', 'banana'}

print(basket)                      # 这里演示的是去重功能

print('orange in basket? ', 'orange' in basket)                # 快速判断元素是否在集合内
print('crabgrass in basket? ', 'crabgrass' in basket)

# 下面展示两个集合间的运算.
a = set('abracadabra')
b = set('alacazam')
print(a)
print(b)

print(a & b)  # 交集

print(a | b)  # 并集

print(a - b)  # 差集

print(a ^ b)  # 对称差


I am working on the file "Python_Spider_Tutorial_02.ipynb".
The first task is:
4. Python网络爬虫Ver 1.0 alpha
有了以上铺垫, 终于可以开始写真正的爬虫了. 我选择的入口地址是Fenng叔的Startup News, 我想Fenng叔刚刚拿到7000万美金融资, 不会介意大家的爬虫去光临他家的小站吧. 这个爬虫虽然可以勉强运行起来, 但是由于缺乏异常处理, 只能爬些静态页面, 也不会分辨什么是静态什么是动态, 碰到什么情况应该跳过, 所以工作一会儿就要败下阵来.
Can you write Python code for it?

import re
import urllib.request
import urllib

from collections import deque

queue = deque()
visited = set()

url = 'http://news.dbanotes.net'  # 入口页面, 可以换成别的

queue.append(url)
cnt = 0

while queue:
  url = queue.popleft()  # 队首元素出队
  visited |= {url}  # 标记为已访问

  print('已经抓取: ' + str(cnt) + '   正在抓取 <---  ' + url)
  cnt += 1
  urlop = urllib.request.urlopen(url)
  if 'html' not in urlop.getheader('Content-Type'):
    continue

  # 避免程序异常中止, 用try..catch处理异常
  try:
    data = urlop.read().decode('utf-8')
  except:
    continue

  # 正则表达式提取页面中所有队列, 并判断是否已经访问过, 然后加入待爬队列
  linkre = re.compile('href="(.+?)"')
  for x in linkre.findall(data):
    if 'http' in x and x not in visited:
      queue.append(x)
      print('加入队列 --->  ' + x)


I am working on the file "Python_Spider_Tutorial_02.ipynb".
The first task is:
findall(string[, pos[, endpos]]) | re.findall(pattern, string[, flags])
搜索string，以列表形式返回全部能匹配的子串。
Can you write Python code for it?

p = re.compile(r'\d+')
print(p.findall('one1two2three3four4'))


I am working on the file "Python_Spider_Tutorial_02.ipynb".
The first task is:
re模块方法
match(string[, pos[, endpos]]) | re.match(pattern, string[, flags])
这个方法将从string的pos下标处起尝试匹配pattern；如果pattern结束时仍可匹配，则返回一个Match对象；如果匹配过程中pattern无法匹配，或者匹配未结束就已到达endpos，则返回None。
pos和endpos的默认值分别为0和len(string)；re.match()无法指定这两个参数，参数flags用于编译pattern时指定匹配模式。
注意：这个方法并不是完全匹配。当pattern结束时若string还有剩余字符，仍然视为成功。想要完全匹配，可以在表达式末尾加上边界匹配符'$'。
Can you write Python code for it?

pattern = re.compile(r'hello')
# 使用Pattern匹配文本，获得匹配结果，无法匹配时将返回None
match = pattern.match('hello world!')
if match:
    print(match.group())


I am working on the file "Python_Spider_Tutorial_02.ipynb".
The first task is:
re模块还提供了一个方法escape(string)，用于将string中的正则表达式元字符如*/+/?等之前加上转义符再返回，在需要大量匹配元字符时有那么一点用。
Match
Match对象是一次匹配的结果，包含了很多关于此次匹配的信息，可以使用Match提供的可读属性或方法来获取这些信息。

属性

string: 匹配时使用的文本。
re: 匹配时使用的Pattern对象。
pos: 文本中正则表达式开始搜索的索引。值与Pattern.match()和Pattern.seach()方法的同名参数相同。
endpos: 文本中正则表达式结束搜索的索引。值与Pattern.match()和Pattern.seach()方法的同名参数相同。
lastindex: 最后一个被捕获的分组在文本中的索引。如果没有被捕获的分组，将为None。
lastgroup: 最后一个被捕获的分组的别名。如果这个分组没有别名或者没有被捕获的分组，将为None。

方法

group([group1, …]): 获得一个或多个分组截获的字符串；指定多个参数时将以元组形式返回。group1可以使用编号也可以使用别名；编号0代表整个匹配的子串；不填写参数时，返回group(0)；没有截获字符串的组返回None；截获了多次的组返回最后一次截获的子串。
groups([default]): 以元组形式返回全部分组截获的字符串。相当于调用group(1,2,…last)。default表示没有截获字符串的组以这个值替代，默认为None。
groupdict([default]): 返回以有别名的组的别名为键、以该组截获的子串为值的字典，没有别名的组不包含在内。default含义同上。
start([group]): 返回指定的组截获的子串在string中的起始索引（子串第一个字符的索引）。group默认值为0。
end([group]): 返回指定的组截获的子串在string中的结束索引（子串最后一个字符的索引+1）。group默认值为0。
span([group]): 返回(start(group), end(group))。
expand(template):将匹配到的分组代入template中然后返回。template中可以使用\id或\g<id>、\g<name>引用分组，但不能使用编号0。\id与\g<id>是等价的；但\10将被认为是第10个分组，如果你想表达\1之后是字符'0'，只能使用\g<1>0。
Can you write Python code for it?

m = re.match(r'(\w+) (\w+)(?P<sign>.*)', 'hello world!')

print("m.string:", m.string)
print("m.re:", m.re)
print("m.pos:", m.pos)
print("m.endpos:", m.endpos)
print("m.lastindex:", m.lastindex)
print("m.lastgroup:", m.lastgroup)

print("m.group(1,2):", m.group(1, 2))
print("m.groups():", m.groups())
print("m.groupdict():", m.groupdict())
print("m.start(2):", m.start(2))
print("m.end(2):", m.end(2))
print("m.span(2):", m.span(2))
print(r"m.expand(r'\2 \1\3'):", m.expand(r'\2 \1\3'))


I am working on the file "Python_Spider_Tutorial_02.ipynb".
The first task is:
search(string[, pos[, endpos]]) | re.search(pattern, string[, flags])
这个方法用于查找字符串中可以匹配成功的子串。从string的pos下标处起尝试匹配pattern，如果pattern结束时仍可匹配，则返回一个Match对象；若无法匹配，则将pos加1后重新尝试匹配；直到pos=endpos时仍无法匹配则返回None。
pos和endpos的默认值分别为0和len(string))；re.search()无法指定这两个参数，参数flags用于编译pattern时指定匹配模式。
Can you write Python code for it?

p = re.compile(r'world')
# 使用search()查找匹配的子串，不存在能匹配的子串时将返回None
# 这个例子中使用match()无法成功匹配
match1 = p.search('hello world!')
match2 = p.match('hello world!')
if match1:
    print('pattern.search result: ', match1.group())
if match2:
    print('pattern.match result: ', match2.group())


I am working on the file "Python_Spider_Tutorial_02.ipynb".
The first task is:
split(string[, maxsplit]) | re.split(pattern, string[, maxsplit])
按照能够匹配的子串将string分割后返回列表。maxsplit用于指定最大分割次数，不指定将全部分割。
Can you write Python code for it?

p = re.compile(r'\d+')
print(p.split('one1two2three3four4'))


I am working on the file "Python_Spider_Tutorial_02.ipynb".
The first task is:
sub(repl, string[, count]) | re.sub(pattern, repl, string[, count])
使用repl替换string中每一个匹配的子串后返回替换后的字符串。
当repl是一个字符串时，可以使用\id或\g<id>、\g<name>引用分组，但不能使用编号0。
当repl是一个方法时，这个方法应当只接受一个参数（Match对象），并返回一个字符串用于替换（返回的字符串中不能再引用分组）。
count用于指定最多替换次数，不指定时全部替换。
Can you write Python code for it?

p = re.compile(r'(\w+) (\w+)')
s = 'i say, hello world!'

print(p.sub(r'\2 \1', s))

def func(m):
    return m.group(1).title() + ' ' + m.group(2).title()

print(p.sub(func, s))


I am working on the file "Python_Spider_Tutorial_02.ipynb".
The first task is:
subn(repl, string[, count]) |re.sub(pattern, repl, string[, count])
返回 (sub(repl, string[, count]), 替换次数)。
Can you write Python code for it?

p = re.compile(r'(\w+) (\w+)')
s = 'i say, hello world!'

print(p.subn(r'\2 \1', s))

def func(m):
    return m.group(1).title() + ' ' + m.group(2).title()

print(p.subn(func, s))


I am working on the file "Python_Spider_Tutorial_02.ipynb".
The first task is:
在我们的爬虫中, 只是用到其中的快速判断元素是否在集合内的功能, 以及集合的并运算.
3. Python的正则表达式
在爬虫程序中, 爬回来的数据是一个字符串, 字符串的内容是页面的html代码. 我们要从字符串中, 提取出页面提到过的所有url. 这就要求爬虫程序要有简单的字符串处理能力, 而正则表达式可以很轻松的完成这一任务.
参考资料
1. 正则表达式30分钟入门教程(GitBook)
2. w3cschool 的Python正则表达式部分
3. Python正则表达式指南
虽然正则表达式功能异常强大, 很多实际上用的规则也非常巧妙, 真正熟练正则表达式需要比较长的实践锻炼. 不过我们只需要掌握如何使用正则表达式在一个字符串中, 把所有的url都找出来, 就可以了. 如果实在想要跳过这一部分, 可以在网上找到很多现成的匹配url的表达式, 拿来用即可.
正则表达式简介
Courtesy of AstralWind - cnblogs
正则表达式并不是Python的一部分。正则表达式是用于处理字符串的强大工具，拥有自己独特的语法以及一个独立的处理引擎，效率上可能不如str自带的方法，但功能十分强大。得益于这一点，在提供了正则表达式的语言里，正则表达式的语法都是一样的，区别只在于不同的编程语言实现支持的语法数量不同；但不用担心，不被支持的语法通常是不常用的部分。如果已经在其他语言里使用过正则表达式，只需要简单看一看就可以上手了。
正则表达式的大致匹配过程是：依次拿出表达式和文本中的字符比较，如果每一个字符都能匹配，则匹配成功；一旦有匹配不成功的字符则匹配失败。如果表达式中有量词或边界，这个过程会稍微有一些不同，但也是很好理解的，看下图中的示例以及自己多使用几次就能明白。
正则表达式通常用于在文本中查找匹配的字符串。Python里数量词默认是贪婪的（在少数语言里也可能是默认非贪婪），总是尝试匹配尽可能多的字符；非贪婪的则相反，总是尝试匹配尽可能少的字符。例如：正则表达式"ab*"如果用于查找"abbbc"，将找到"abbb"。而如果使用非贪婪的数量词"ab*?"，将找到"a"。
下图列出了Python支持的正则表达式元字符和语法：

开始使用re
Python通过re模块提供对正则表达式的支持。使用re的一般步骤是先将正则表达式的字符串形式编译为Pattern实例，然后使用Pattern实例处理文本并获得匹配结果（一个Match实例），最后使用Match实例获得信息，进行其他的操作。
Can you write Python code for it?

import re

# 将正则表达式编译成Pattern对象
pattern = re.compile(r'hello')

# 使用Pattern匹配文本，获得匹配结果，无法匹配时将返回None
match = pattern.match('hello world!')

if match:
    # 使用Match获得分组信息
    print(match.group())


I am working on the file "Python_Spider_Tutorial_03.ipynb".
The first task is:
上述代码运行后通过 Fiddler 抓到的 GET 报文如下所示:
GET http://www.baidu.com/ HTTP/1.1
Accept-Encoding: identity
Connection: close
Host: www.baidu.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko
Accept: text/html, application/xhtml+xml, */*
Accept-Language: en-US,en;q=0.8,zh-Hans-CN;q=0.5,zh-Hans;q=0.3
可见我们在代码里写的东西都添加到请求报文里面了.
保存抓回来的报文
顺便说说文件操作. Python 的文件操作还是相当方便的. 我们可以讲抓回来的数据 data 以二进制形式保存, 也可以经过 decode() 处理成为字符串后以文本形式保存. 改动一下打开文件的方式就能保存不同格式的文件了. 下面是参考代码:
Can you write Python code for it?

def saveFile(data):
    save_path = './data/temp.out'
    f_obj = open(save_path, 'wb') # wb 表示打开方式
    f_obj.write(data)
    f_obj.close()


I am working on the file "Python_Spider_Tutorial_03.ipynb".
The first task is:
第二种方法使用了 build_opener 这个方法, 用来自定义 opener, 这种方法的好处是可以方便的拓展功能, 例如下面的代码就拓展了自动处理 Cookies 的功能.
Can you write Python code for it?

import urllib.request
import http.cookiejar

# head: dict of header
def makeMyOpener(head = {
    'Connection': 'Keep-Alive',
    'Accept': 'text/html, application/xhtml+xml, */*',
    'Accept-Language': 'en-US,en;q=0.8,zh-Hans-CN;q=0.5,zh-Hans;q=0.3',
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko'
}):
    cj = http.cookiejar.CookieJar()
    opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
    header = []
    for key, value in head.items():
        elem = (key, value)
        header.append(elem)
    opener.addheaders = header
    return opener

oper = makeMyOpener()
uop = oper.open('http://www.baidu.com/', timeout = 1000)
data = uop.read()
print(data.decode())


I am working on the file "Python_Spider_Tutorial_03.ipynb".
The first task is:
经过上面的改进，我们的爬虫现在可以处理更多类型的网页了，先看一下这一阶段我们的成果吧
Can you write Python code for it?

import re
import urllib.request
import urllib
import http.cookiejar
from collections import deque

# head: dict of header
def makeMyOpener(head = {
    'Connection': 'Keep-Alive',
    'Accept': 'text/html, application/xhtml+xml, */*',
    'Accept-Language': 'en-US,en;q=0.8,zh-Hans-CN;q=0.5,zh-Hans;q=0.3',
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko'
}):
    cj = http.cookiejar.CookieJar()
    opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
    header = []
    for key, value in head.items():
        elem = (key, value)
        header.append(elem)
    opener.addheaders = header
    return opener

def saveFile(data, save_path = './temp.out'):
    f_obj = open(save_path, 'wb') # wb 表示打开方式
    f_obj.write(data)
    f_obj.close()

queue = deque()
visited = set()

url = 'http://news.dbanotes.net'  # 入口页面, 可以换成别的

queue.append(url)
cnt = 0

oper = makeMyOpener()

while queue:
  url = queue.popleft()  # 队首元素出队
  visited |= {url}  # 标记为已访问

  print('已经抓取: ' + str(cnt) + '   正在抓取 <---  ' + url)
  cnt += 1
  urlop = oper.open(url, timeout=5)
  if 'html' not in urlop.getheader('Content-Type'):
    continue

  # 避免程序异常中止, 用try..catch处理异常
  try:
    data = urlop.read().decode('utf-8')
    #saveFile(data, './data/'+url+'.out')
  except:
    print('Error in: ', url)
    continue

  # 正则表达式提取页面中所有队列, 并判断是否已经访问过, 然后加入待爬队列
  linkre = re.compile('href="(.+?)"')
  for x in linkre.findall(data):
    if 'http' in x and x not in visited:
      queue.append(x)
      print('加入队列 --->  ' + x)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
Cookies的类型是RequestsCookieJar
Can you write Python code for it?


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
不过这不是最好的方式，https协议的请求把证书验证都忽略了还有什么意义？
当然你也可以指定一个本地证书用作客户端证书，可以是单个文件（包含密钥和证书）或一个包含两个文件路径的元组。
python
response = requests.get('https://www.12306.cn', cert=('/path/server.crt', '/path/key'))
print(response.status_code)
上面代码是实例，你需要有crt和key文件，指定它们的路径。注意本地私有证书的key必须要是解密状态，加密状态的key是不支持的。
代理设置
对于某些网站，在测试的时候请求几次，能正常获取内容。但是一旦开始大规模爬取，对于大规模且频繁的请求，网站可能会直接登录验证，验证码，甚至直接把IP给封禁掉。为了防止这种情况的发生，我们就需要设置代理来解决这个问题，需要用到proxies这个参数。
```python
import requests
proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
}
requests.get("https://www.taobao.com", proxies=proxies)
```
当然直接运行这个实例可能不行，因为这个代理可能是无效的，请换成自己的有效代理试验一下。
若你的代理需要使用HTTP Basic Auth，可以使用类似 http://user:password@host/ 这样的语法。
```python
import requests
proxies = {
    "http": "http://user:password@10.10.1.10:3128/",
}
requests.get("https://www.taobao.com", proxies=proxies)
```
超时设置
在本机网络状况不好或者服务器网络响应太慢甚至无响应时，我们可能会等待特别久的时间才可能会收到一个响应，甚至到最后收不到响应而报错。为了防止服务器不能及时响应，我们应该设置一个超时时间，即超过了这个时间还没有得到响应，那就报错。
设置超时时间需要用到timeout参数。这个时间的计算是发出request请求到服务器响应response的时间。
下面用一个实例来感受一下：
Can you write Python code for it?

import requests

r = requests.get("https://www.taobao.com", timeout = 1)
print(r.status_code)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
会话维持
在requests中，我们如果直接利用requests.get()或requests.post()等方法的确可以做到模拟网页的请求。但是这实际上是相当于不同的会话，即不同的session，也就是说相当于你用了两个浏览器打开了不同的页面。
设想这样一个场景，你第一个请求利用了requests.post()方法登录了某个网站，第二次想获取成功登录后的自己的个人信息，你又用了一次requests.get()方法。实际上，这相当于打开了两个浏览器，是两个完全不相关的会话，你说你能成功获取个人信息吗？那当然不能。
有小伙伴就说了，我在两次请求的时候都设置好一样的Cookie不就行了？行是行，但是不觉得麻烦吗？每次都要这样。是我我忍不了。其实解决这个问题的主要方法就是维持同一个会话，也就是相当于打开一个新的浏览器选项卡而不是新开一个浏览器。但是我又不想每次设置Cookie，那该咋办？这时候就有了新的利器Session。利用它，我们可以方便地维护一个会话，而且不用担心Cookie的问题，它会帮我们自动处理好。
在下面的实例中我们请求了一个测试网址，http://httpbin.org/cookies/set/number/123456789请求这个网址我们可以设置Cookie，名称叫做number，内容是123456789，后面的网址http://httpbin.org/cookies可以获取当前的Cookie。
你觉得这样能成功获取到设置的Cookie吗？运行结果如下：
Can you write Python code for it?

requests.get('http://httpbin.org/cookies/set/number/123456789')
r = requests.get('http://httpbin.org/cookies')
print(r.text)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
但注意，如果返回结果不是Json格式，便会出现解析错误，抛出 json.decoder.JSONDecodeError 的异常。
抓取网页
如上的请求链接返回的是Json形式的字符串，那么如果我们请求普通的网页，那么肯定就能获得相应的内容了。
下面我们以知乎－发现页面为例来体验一下：
Can you write Python code for it?

import requests
import re

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}
r = requests.get("https://www.zhihu.com/explore", headers=headers)
pattern = re.compile('explore-feed.*?question_link.*?>(.*?)</a>', re.S)
titles = re.findall(pattern, r.text)
print(titles)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
其实这只是 requests 库的冰山一角，更多的还在后面呢。
GET请求
HTTP 中最常见的请求之一就是 GET 请求，我们首先来详细了解下利用 requests 来构建 GET 请求的方法以及相关属性方法操作。
首先让我们来构建一个最简单的GET请求，请求 httpbin.org/get ，它会判断如果你是 GET 请求的话，会返回响应的请求信息。
Can you write Python code for it?

r = requests.get('http://httpbin.org/get')
print(r.text)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
可以发现我们成功发起了get请求，请求的链接和头信息都有相应的返回。
那么 GET 请求，如果要附加额外的信息一般是怎样来添加？没错，那就是直接当做参数添加到url后面。
比如现在我想添加两个参数，名字name是germey，年龄age是22。构造这个请求链接是不是我们要直接写成
r = requests.get("http://httpbin.org/get?name=germey&age=22")
可以是可以，但是不觉得很不人性化吗？一般的这种信息数据我们会用字典来存储，那么怎样来构造这个链接呢？
{"name": "germey", "age": 22}
同样很简单，利用params这个参数就好了。
实例如下：
Can you write Python code for it?

data = {
    'name': 'germey',
    'age': 22
}

r = requests.get("http://httpbin.org/get", params=data)
print(r.text)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
可以得到响应的 HTTP HEADER
Can you write Python code for it?


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
可以看到，首先打印输出了cookie，可以发现它是一个RequestCookieJar类型。然后用items()方法将其转化为元组组成的列表，遍历输出每一个cookie的名和值。
当然，我们也可以直接用Cookie来维持登录状态。~~比如我们以知乎为例，直接利用Cookie来维持登录状态。~~
~~首先登录知乎，将请求头中的Cookie复制下来。~~
（知乎已经更改了登陆方式，现在使用Cookie无法登陆）
Can you write Python code for it?

headers = {
    'Cookie': '_za=dc07d0bb-599c-46e9-8906-f6dd252910b4; d_c0="AIAABz1hvQmPTup3qtT92xkZN2UQoova_cc=|1460107885"; _zap=3c9b8860-2a38-4532-b476-c3617ff3fb0d; _ga=GA1.2.1945944829.1442545470; aliyungf_tc=AQAAANoM4BIxkAMAj67seLZyZFoQgALT; q_c1=a366104786da4623b23af9cd321e4d38|1484577173000|1468254315000; _xsrf=b9564d003dd4bdb7b6f9e6185b8a0b78; l_cap_id="MTZlMjVmODgwYTVlNGEyYzg1ZDBkNzhkMzNjYjMxYmI=|1484577173|1c7374494b0e1a7ef526ea93c7065b2a8b9736eb"; cap_id="MzU3OGUzZjYwYjIwNGNmNWFhMTk2OGU2NjRjOWE3MDk=|1484577173|b5723f8256e35bd66f86d1e070c3a3d4324c655a"; r_cap_id="YmQwYmEwNDIxMDYxNDBkZmI2NzAzNjI2ZjJkMzNmOGQ=|1484577175|01cabbd0c7a481a7e4a907ae60ba3e2c64d07ddc"; login="NWVhMDVlNzZjMzI0NGE3ZGIyMmExODFhNzEzNGVhNmY=|1484577185|45503ef9b8dc4c3468f776c202fbc7fe442f8521"; n_c=1; z_c0=Mi4wQUFDQXhFMGpBQUFBZ0FBSFBXRzlDUmNBQUFCaEFsVk5vV2FrV0FBQkU5RmpvbXljZE15a2FZNU8zQVRjS0g1Qm1n|1484577189|3dc63f1a6bd389c32162f4d901ebf10e6750dffc; nweb_qa=heifetz; __utma=51854390.1945944829.1442545470.1484577176.1484577176.1; __utmb=51854390.0.10.1484577176; __utmc=51854390; __utmz=51854390.1484577176.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=51854390.100-1|2=registration_date=20140101=1^3=entry_date=20140101=1',
    'Host': 'www.zhihu.com',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36',
}
r = requests.get("http://www.zhihu.com", headers=headers)
print(r.text)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
响应体内容
Can you write Python code for it?


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
响应体的类型是字符串str
Can you write Python code for it?


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
在这里分别打印输出了响应状态吗status_code，响应头headers，Cookies，请求连接，请求历史的类型和内容。可以看到，headers还有cookies这两个部分都是特定的数据结构，打开浏览器同样可以发现有同样的响应头信息。
状态码
在这里状态码常用来判断请求是否成功，requests还提供了一个内置的状态码查询对象requests.codes。比如你可以通过if r.status_code == requests.codes.ok来判断请求是否成功。
Can you write Python code for it?

r = requests.get('http://www.jianshu.com')
exit() if not r.status_code == requests.codes.ok else print('Request Successfully')


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
在这里用了open() 函数，第一个参数是文件名称，第二个参数代表以二进制写的形式打开，可以向文件里写入二进制数据，然后保存。
运行结束之后，可以发现在data文件夹中出现了名为favicon.ico的图标。
添加头信息
如urllib.request一样，我们也可以通过headers参数来传递头信息。你可以在headers这个数组中任意添加其他的头信息。
比如上面的知乎的例子，如果不传递头信息，就不能正常请求：
Can you write Python code for it?

r = requests.get("https://www.zhihu.com/explore")
print(r.text)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
如上代码，我们请求了知乎－发现页面 https://www.zhihu.com/explore ，在这里加入了头信息，头信息中包含了 User-Agent 信息，也就是上一节提到的浏览器标识信息。如果不加这个，知乎会禁止抓取。可以看到，我们成功提取出了所有的问题内容。
抓取二进制数据
在上面的例子中，我们抓取的是知乎的一个页面，实际上它返回的是一个HTML文档，那么如果我们想抓去图片、音频、视频等文件的话应该怎么办呢？
我们都知道，图片、音频、视频这些文件都是本质上由二进制码组成的，由于有特定的保存格式和对应的解析方式，我们才可以看到这些形形色色的多媒体。所以想要抓取他们，那就需要拿到他们的二进制码。
下面我们以GitHub的站点图标，也就是在浏览器每一个标签上显示的小图标，为例来感受一下：
Can you write Python code for it?

r = requests.get("https://github.com/favicon.ico")
print(r.text)
print(r.content)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
如果想永久等待，那么你可以直接将timeout设置为None，或者不设置，直接留空，因为默认是None。这样的话，如果服务器还在运行，但是响应特别慢，那就慢慢等吧，它永远不会返回超时错误的。
用法如下：
Can you write Python code for it?

r = requests.get('https://www.taobao.com', timeout=None)
print(r.status_code)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
如果用户名和密码正确的话，认证成功，那么运行结果会返回200，如果认证失败，则会返回401状态码。
当然如果参数都传一个HTTPBasicAuth类，那的确太繁琐了，所以requests提供了一个简单的写法，你可以直接传一个元组，它会默认使用HTTPBasicAuth这个类来认证。
所以上面的代码可以直接简写如下：
Can you write Python code for it?

import requests

r = requests.get('http://120.27.34.24:9001', auth=('user', '123'))
print(r.status_code)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
成功获取！所以，利用Session我们可以做到模拟同一个会话，而且不用担心Cookie的问题，通常用于模拟登录成功之后再进行下一步的操作。
Session在平常用到的非常广泛，可以用于模拟在一个浏览器中打开同一站点的不同页面。
SSL证书验证
reqeusts 提供了证书验证的功能，当一个请求是 https 的时候，它会检查 SSL 证书，可以使用 verify 这个参数。(其实如果不加的话默认是True，会自动验证的)
现在以12306( https://www.12306.cn)为例来感受一下它的用法，我们现在访问它都可以看到一个证书问题的页面，如下：

现在我们用requests来测试一下：
Can you write Python code for it?

import requests

response = requests.get('https://www.12306.cn')
print(response.status_code)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
我们可以发现，使用了 requests.get(url) 方法就成功实现了一个 GET 请求。这倒不算什么，更方便的在于其他的请求类型依然可以用一句话来完成。
下面介绍一个很棒的demo网站，httpbin(1): HTTP Request & Response Service (http://httpbin.org) ，这个网站的介绍大概是这个样子的
```
Testing an HTTP Library can become difficult sometimes. RequestBin is fantastic for testing POST requests, but doesn't let you control the response. This exists to cover all kinds of HTTP scenarios. Additional endpoints are being considered.
All endpoint responses are JSON-encoded.
```
简单的说就是你可以对这个网站发出各类HTTP请求，来测试你的请求报文。
下面用一个实例来感受一下 requests 库的强大威力：
Can you write Python code for it?

r = requests.post('http://httpbin.org/post')
print('----POST----\n', r.text)
r = requests.put('http://httpbin.org/put')
print('----PUT----\n', r.text)
r = requests.delete('http://httpbin.org/delete')
print('----DELETE----\n', r.text)
r = requests.head('http://httpbin.org/get')
print('----HEAD----\n', r.text)
r = requests.options('http://httpbin.org/get')
print('----OPTIONS----\n', r.text)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
或直接不加参数：
Can you write Python code for it?

r = requests.get('https://www.taobao.com')
print(r.status_code)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
提示一个错误，叫做SSLError，证书验证错误。
所以如果我们请求一个https站点，但是证书验证错误的页面时，就会报这样的错误，那么如何避免这个错误呢？很简单，把verify这个参数设置为False即可。
改成如下代码：
Can you write Python code for it?

response = requests.get('https://www.12306.cn', verify=False)
print(response.status_code)


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
状态码是200
Can you write Python code for it?


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
请求响应的类型是 requests.models.Response
Can you write Python code for it?


I am working on the file "Python_Spider_Tutorial_04.ipynb".
The first task is:
这个网站会返回一个响应，里面包含files这个字段，而form是空的，这证明文件上传部分，会单独有一个files来标识。
Cookie处理
在前面我们使用了urllib，让它处理cookie真的是挺麻烦的，而有了requests，获得和提交cookies只需要一步。我们先用一个实例感受一下获取Cookie的过程：
Can you write Python code for it?

import requests

r = requests.get("https://www.baidu.com")
print(r.cookies)
for key, value in r.cookies.items():
    print(key + '=' + value)


I am working on the file "Python_Spider_Tutorial_05.ipynb".
The first task is:
find()和findAll()
文档中的定义是这样的
BeautifulSoup.find(self, name=None, attrs={}, recursive=True, text=None, **kwargs)
BeautifulSoup.findAll(self, name=None, attrs={}, recursive=True, text=None, limit=None, **kwargs)
下面我们通过实例来看一下用法
Can you write Python code for it?

bsObj.find({'span'})

bsObj.findAll({'span'})

bsObj.find({'h1','h2'})

bsObj.findAll({'h1','h2'})


I am working on the file "Python_Spider_Tutorial_05.ipynb".
The first task is:
可以看到，find()只会找出一个的字段，而findAll()则会找出所有符合的字段。
此外，还有一个关键词参数keyword可以选择哪些具有指定属性的标签（就和上面演示的findAll功能一样，这样子设计是为了功能的冗余）
Can you write Python code for it?

allText = bsObj.findAll(id='text')
print(allText[0].get_text())


I am working on the file "Python_Spider_Tutorial_05.ipynb".
The first task is:
可以看到，我们从网页中提取的标签被嵌在BeautifulSoup对象的第二层。但是，当我们从对象里提取h1标签的时候，可以直接这样调用他。
bsObj.h1
事实上，下面的所有函数调用都可以产生同样的结果
bsObj.html.body.h1
bsObj.body.h1
bsObj.html.h1
处理网络连接异常
让我们看看爬虫的第二行代码
python
html = urlopen("http://www.pythonscraping.com/exercises/exercise1.html")
这行代码可能会发生下面两种异常：

网页在服务器上不存在（或者获取页面的时候出现错误）
服务器不存在

当第一种异常发生时，urlopen会抛出HTTPError错误，我们可以用try-catch语句处理这种异常
当第二种异常发生时，urlopen会返回一个None对象，我们用if语句来检查这种错误
Can you write Python code for it?

from urllib.request import urlopen
from urllib.error import HTTPError
from bs4 import BeautifulSoup
import sys

def getTitle(url):
    try:
        html = urlopen(url)
    except HTTPError as e:
        print(e)
        return None
    try:
        bsObj = BeautifulSoup(html.read(), "html.parser")
        title = bsObj.h1
    except AttributeError as e:
        return None
    return title

title = getTitle("http://www.pythonscraping.com/exercises/exercise1.html")
if title == None:
    print("Title could not be found")
else:
    print(title)


I am working on the file "Python_Spider_Tutorial_05.ipynb".
The first task is:
在BeautifulSoup中使用正则表达式
当然bs4是支持正则表达式的，在bs4中使用正则表达式的例子请参考文档，这里不做过多叙述，仅仅给出一个例子：
Can you write Python code for it?

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re

html = urlopen("http://www.pythonscraping.com/pages/page3.html")
bsObj = BeautifulSoup(html, "html.parser")
images = bsObj.findAll("img", {"src":re.compile("\.\.\/img\/gifts/img.*\.jpg")})
for image in images:
    print(image["src"])


I am working on the file "Python_Spider_Tutorial_06.ipynb".
The first task is:
上面的函数还不太能用于实际抓取，我们稍作改进，变成下面这个样子，就可以初步用于抓取页面的所有链接了。因为我们不能无限制地抓取下去，我便设置了10个链接的上限。
Can you write Python code for it?

from urllib.request import urlopen
from bs4 import BeautifulSoup
import datetime
import random
import re

count = 0
random.seed(datetime.datetime.now())
def getLinks(articleUrl):
    html = urlopen("http://en.wikipedia.org"+articleUrl)
    bsObj = BeautifulSoup(html, "html.parser")
    return bsObj.find("div", {"id":"bodyContent"}).findAll("a", href=re.compile("^(/wiki/)((?!:).)*$"))
links = getLinks("/wiki/Python_(programming_language)")
while len(links) > 0 and count < 10:
    newArticle = links[random.randint(0, len(links)-1)].attrs["href"]
    print(newArticle)
    count = count + 1
    links = getLinks(newArticle)


I am working on the file "Python_Spider_Tutorial_06.ipynb".
The first task is:
为了避免一个页面被采集两次，链接去重是非常重要的，下面的代码用Python的set来保存已经采集的链接。下面这段代码将无限制地运行下去，除非set集为空，然而这几乎是不可能的。
Can you write Python code for it?

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re

pages = set()
def getLinks(pageUrl):
    global pages
    html = urlopen("http://en.wikipedia.org"+pageUrl)
    bsObj = BeautifulSoup(html, "html.parser")
    try:
        print(bsObj.h1.get_text())
        print(bsObj.find(id ="mw-content-text").findAll("p")[0])
        print(bsObj.find(id="ca-edit").find("span").find("a").attrs['href'])
    except AttributeError:
        print("This page is missing something! No worries though!")

    for link in bsObj.findAll("a", href=re.compile("^(/wiki/)")):
        if 'href' in link.attrs:
            if link.attrs['href'] not in pages:
                #We have encountered a new page
                newPage = link.attrs['href']
                print("----------------\n"+newPage)
                pages.add(newPage)
                getLinks(newPage)
getLinks("")


I am working on the file "Python_Spider_Tutorial_06.ipynb".
The first task is:
使用Scrapy
写网络爬虫的挑战之一就是你要不断重复一些简单的任务：找出页面上所有的链接，区分外链和内链，跳转到新的页面。掌握这些基本模式非常有用，从零开始也是完全可行的，但是这里有一个工具可以帮助你自动处理这些细节。
https://scrapy.org
关于这个框架的使用，请参考其文档，这里不作更多叙述了。
如果以后有时间再继续补充。
Can you write Python code for it?

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('h2.entry-title'):
            yield {'title': title.css('a ::text').extract_first()}

        next_page = response.css('div.prev-post > a ::attr(href)').extract_first()
        if next_page:
            yield scrapy.Request(response.urljoin(next_page), callback=self.parse)


I am working on the file "Python_Spider_Tutorial_06.ipynb".
The first task is:
可以发现，所有指向Wikipedia词条的链接都是/wiki/开头，所以我们可以用正则表达式来过滤出这些词条，就像这样
Can you write Python code for it?

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re

html = urlopen("https://en.wikipedia.org/wiki/Python_(programming_language)")
bsObj = BeautifulSoup(html.read(), "html.parser")
for link in bsObj.find("div", {"id":"bodyContent"}).findAll("a", href=re.compile("^(/wiki/)((?!:).)*$")):
    if 'href' in link.attrs:
        print(link.attrs['href'])


I am working on the file "Python_Spider_Tutorial_06.ipynb".
The first task is:
我们接下来可以写一个随机找外链的小程序
Can you write Python code for it?

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
import datetime
import random

pages = set()
random.seed(datetime.datetime.now())

#Retrieves a list of all Internal links found on a page
def getInternalLinks(bsObj, includeUrl):
    internalLinks = []
    #Finds all links that begin with a "/"
    for link in bsObj.findAll("a", href=re.compile("^(/|.*"+includeUrl+")")):
        if link.attrs['href'] is not None:
            if link.attrs['href'] not in internalLinks:
                internalLinks.append(link.attrs['href'])
    return internalLinks

#Retrieves a list of all external links found on a page
def getExternalLinks(bsObj, excludeUrl):
    externalLinks = []
    #Finds all links that start with "http" or "www" that do
    #not contain the current URL
    for link in bsObj.findAll("a", href=re.compile("^(http|www)((?!"+excludeUrl+").)*$")):
        if link.attrs['href'] is not None:
            if link.attrs['href'] not in externalLinks:
                externalLinks.append(link.attrs['href'])
    return externalLinks

def splitAddress(address):
    addressParts = address.replace("http://", "").split("/")
    return addressParts

def getRandomExternalLink(startingPage):
    html = urlopen(startingPage)
    bsObj = BeautifulSoup(html, "html.parser")
    externalLinks = getExternalLinks(bsObj, splitAddress(startingPage)[0])
    if len(externalLinks) == 0:
        internalLinks = getInternalLinks(startingPage)
        return getNextExternalLink(internalLinks[random.randint(0,
                                  len(internalLinks)-1)])
    else:
        return externalLinks[random.randint(0, len(externalLinks)-1)]

def followExternalOnly(startingSite):
    externalLink = getRandomExternalLink(startingSite)
    print("Random external link is: "+externalLink)
    #followExternalOnly(externalLink)

followExternalOnly("http://oreilly.com")
followExternalOnly("http://oreilly.com")
followExternalOnly("http://oreilly.com")


I am working on the file "Python_Spider_Tutorial_07.ipynb".
The first task is:
用Python解析JSON
Can you write Python code for it?

import json

jsonString = '{"arrayOfNums":[{"number":0},{"number":1},{"number":2}],"arrayOfFruits":[{"fruit":"apple"},{"fruit":"banana"},{"fruit":"pear"}]}'
jsonObj = json.loads(jsonString)


I am working on the file "Python_Strings.ipynb".
The first task is:
Ascii Strings
default when creating strings
Can you write Python code for it?

# ascii string
a_string = 'this\tis a ascii\nstring'
print(a_string)

# r-string or raw string
r_string = r'this is an r \nstring \t'
r_string

string1 = 'hello'
string2 = 'there'
string3 = 'Mary had a little lamb'


I am working on the file "Python_Strings.ipynb".
The first task is:
Byte String
Working with Hex/Oct/Bin String Data
Common Erros

TypeError: unicode strings are not supported, please encode to bytes

We will work with split, join, rstrip, lstrip, rfind
Slice Operator
Can you write Python code for it?

# to get last element
print(string1)
string1[4]

# to get the THIRD element
print(string2)
string2[2]

# TO get a range of elements
print(string3)
string3[0:4]


I am working on the file "Python_Strings.ipynb".
The first task is:
Formatting Strings
How to build strings properly.
Can you write Python code for it?

'{one}{one}{one}'.format(one=5,three=3, two=4)

stringgggg = '%d%d%d' % (5,4,3)
print(stringgggg)


I am working on the file "Python_Strings.ipynb".
The first task is:
Join
Can you write Python code for it?

help(str.join)

tiene_la_a = 'a'.join(le_falta_la_a)
tiene_la_a

string1 = ['this', 'iss', 'a', 'random', 'string']
'*'.join(string1)


I am working on the file "Python_Strings.ipynb".
The first task is:
Split
Can you write Python code for it?

help(str.split)

print(string3)
string3.split()

le_falta_la_a = string3.split('a')
le_falta_la_a


I am working on the file "Python_Strings.ipynb".
The first task is:
lstrip
Can you write Python code for it?

left_spaces = '             adgfgfsgsfdgsr'
left_spaces

left_spaces.lstrip()


I am working on the file "Python_Strings.ipynb".
The first task is:
rstrip
Can you write Python code for it?

help(str.rstrip)

this_is_a_string = 'adfadfds                                    '

this_is_a_string.rstrip()


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Arithmetic with Strings
Can you write Python code for it?

s = 'spam'
e = "eggs"

s + e

s + " " + e

4 * (s + " ") + e

print(4 * (s + " ") + s + " and\n" + e)     # use \n to get a newline with the print function


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Emoji are unicode characters, so you can use them a well (not all OSs will show all characters!)
Can you write Python code for it?

radio_active = "\U00002622"
wink = "\U0001F609"

print(radio_active + wink)


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Find and Replace
Can you write Python code for it?


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Justification and Cleaning
Can you write Python code for it?

line.center(100)

line.ljust(100)

line.rjust(100, "*")

line2 = "            My hovercraft is full of eels      "

line2.strip()

line3 = "*$*$*$*$*$*$*$*$My hovercraft is full of eels*$*$*$*$"

line3.strip('*$')

line3.lstrip('*$'), line3.rstrip('*$')


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Line Formatting
Can you write Python code for it?

anotherline = "mY hoVErCRaft iS fUlL oF eEELS"

anotherline.upper()

anotherline.lower()

anotherline.title()

anotherline.capitalize()

anotherline.swapcase()


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Nice trick to convert number to a different base
Can you write Python code for it?

"Representation of the number {1:s} - dec: {0:d};  hex: {0:x};  oct: {0:o};  bin: {0:b}".format(A,D)


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
One of the useful tings about regular expressions in Python is using the to search and replace parts of string (re.sub)
Can you write Python code for it?

mynewline = re.sub(regex1, "*TEST*", myline)

mynewline


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Python supports Unicode characters
You can enter unicode characters directly from the keyboard (depends on your operating system), or you can use the ASCII encoding.
A list of ASCII encoding can be found here.
For example the ASCII ecoding for the greek capital omega is U+03A9, so you can create the character with \U000003A9
Can you write Python code for it?

print("This resistor has a value of 100 k\U000003A9")

Ω = 1e3

Ω + np.pi


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Raw strings begin with a special prefix (r) and signal Python not to interpret backslashes and other special metacharacters in the string, allowing you to pass them through directly to the regular expression engine.
Can you write Python code for it?

regex1 = r"test"

match1 = re.search(regex1, myline)

match1

myline[10:14]

match3 = re.findall(regex1, myline)

match3


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Really long strings
Can you write Python code for it?

long_string = (
"""
The planets {0:s} and {1:s} are at a distance
of {2:.1f} AU and {3:.1f} AU from the Sun.
"""
.format(planet_table['Name'][1],planet_table['Name'][4],
        planet_table['a'][1],planet_table['a'][4])
)

print(long_string)


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Regular Expression in Python (re)
Can you write Python code for it?

import re

myline = "This is a test, this in only a test."

print(myline)


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Splitting and Joining
Can you write Python code for it?

line.split()

'_*_'.join(line.split())

' '.join(line.split()[::-1])


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Strings are just arrays of characters
Can you write Python code for it?


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Use explicit formatting to avoid these errors
Python string formatting has the form:
{Variable Index: Format Type}  .format(Variable)
Can you write Python code for it?

A = 42
B = 1.23456
C = 1.23456e10
D = 'Forty Two'

"I like the number {0:d}".format(A)

"I like the number {0:s}".format(D)

"The number {0:f} is fine, but not a cool as {1:d}".format(B,A)

"The number {0:.3f} is fine, but not a cool as {1:d}".format(C,A)       # 3 places after decimal

"The number {0:.3e} is fine, but not a cool as {1:d}".format(C,A)       # sci notation

"{0:g} and {1:g} are the same format but different results".format(B,C)


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Watch out for variable types!
Can you write Python code for it?

n = 4

print("I would like " + n + " orders of spam")

print("I would like " + str(n) + " orders of spam")


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Working with Files and Directories (OS agnostic)

The os package allows you to do operating system stuff without         worrying about what system you are using
Can you write Python code for it?

import os

os.chdir("./MyData")

my_data_dir = os.listdir()

my_data_dir

for file in my_data_dir:

    if file.endswith(".txt"):
        print(file)

for file in my_data_dir:

    if file.endswith(".txt"):
        print(os.path.abspath(file))


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
Working with strings
Can you write Python code for it?


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
You can also find files with glob
Can you write Python code for it?

import glob

my_files = glob.glob('02_*.fits')

my_files

for file in my_files:

    file_size = os.stat(file).st_size
    out_string = "The file {0} as a size of {1} bytes".format(file,file_size)

    print(out_string)


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
You can also use the textwrap module
Can you write Python code for it?

import textwrap

lots_of_spam = (s + " ") * 100

print(lots_of_spam)

textwrap.wrap(lots_of_spam, width=70)


I am working on the file "Python_StringsAndStuff.ipynb".
The first task is:
You can compare strings
Can you write Python code for it?

"spam" == "good"

"spam" != "good"

"spam" == "spam"

"sp" < "spam"

"spam" < "eggs"


I am working on the file "Python_SymPy.ipynb".
The first task is:
$$ \int z^{2}\, dz $$
Astropy can output $\LaTeX$ tables
Can you write Python code for it?

from astropy.io import ascii
from astropy.table import QTable

my_table = QTable.read('./MyData/Zodiac.csv', format='ascii.csv')

my_table[0:3]

ascii.write(my_table, format='latex')


I am working on the file "Python_SymPy.ipynb".
The first task is:
And solve equations - solve
Can you write Python code for it?

yet_another_equation = 2 * x**2 - 5 * x + 30

yet_another_equation

sp.solve(yet_another_equation,x)


I am working on the file "Python_SymPy.ipynb".
The first task is:
Calculus
Can you write Python code for it?

yet_another_equation

sp.diff(yet_another_equation,x)

sp.diff(yet_another_equation,x,2)

sp.integrate(yet_another_equation,x)

sp.integrate(yet_another_equation,(x,0,5))   # limits x = 0 to 5


I am working on the file "Python_SymPy.ipynb".
The first task is:
Limits
$$\lim _{x\to \infty }\left(1+{\frac {1}{x}}\right)^{x}$$
Can you write Python code for it?

limit_equation = (1 + (1 / x)) ** x

limit_equation

sp.limit(limit_equation, x, sp.oo)      # sp.oo = infinity


I am working on the file "Python_SymPy.ipynb".
The first task is:
Solve ODE Equations - dsolve
Can you write Python code for it?

f = sp.Function('f')

equation_ode = sp.Derivative(f(x), x, x) + 9*f(x)

equation_ode

sp.dsolve(equation_ode, f(x))


I am working on the file "Python_SymPy.ipynb".
The first task is:
SymPy has all sorts of ways to manipulates symbolic equations
Can you write Python code for it?

sp.simplify(my_equation / x)

another_equation = (x + 2) * (x - 3)
another_equation

sp.expand(another_equation)

long_equation = 2*y*x**3 + 12*x**2 - x + 3 - 8*x**2 + 4*x + x**3 + 5 + 2*y*x**2 + x*y
long_equation

sp.collect(long_equation,x)

sp.collect(long_equation,y)


I am working on the file "Python_SymPy.ipynb".
The first task is:
Symbolic Mathematics (SymPy)
Can you write Python code for it?

sp.init_printing()     # Turns on pretty printing

np.sqrt(8)

sp.sqrt(8)


I am working on the file "Python_SymPy.ipynb".
The first task is:
System of equations
$$
\begin{array}{c}
x + 3y + 5z = 10 \
2x + 5y + z = 8 \
2x + 3y + 8z = 3 \
\end{array}
\hspace{3cm}
\left[
\begin{array}{ccc}
1 & 3 & 5 \
2 & 5 & 1 \
2 & 3 & 8
\end{array}
\right]
\left[
\begin{array}{c}
x\
y\
z
\end{array}
\right]
=
\left[
\begin{array}{c}
10\
8\
3
\end{array}
\right]
$$
Can you write Python code for it?


I am working on the file "Python_SymPy.ipynb".
The first task is:
Use can use SymPy to make $\LaTeX$ equations for you!
Can you write Python code for it?


I am working on the file "Python_Units.ipynb".
The first task is:
An Example: The velocity of an object in circular orbit around the Sun is
$$ v=\sqrt{GM_{\odot}\over d} $$
Can you write Python code for it?

distance = planet_table['a'][0:3]  # Mercury, Venus, Earth

orbit_v = np.sqrt(const.G * const.M_sun / distance)

orbit_v

orbit_v.decompose()

orbit_v.to(u.km/u.s)

orbit_v.to(ringo/u.ms)


I am working on the file "Python_Units.ipynb".
The first task is:
At example problem: How long would it take to go 100 km at velocities in velocity?
Can you write Python code for it?

distance = 100 * u.km

time_2 = distance / velocity

time_2


I am working on the file "Python_Units.ipynb".
The first task is:
Dimentionless Units
Can you write Python code for it?

dimless_y = (1 * u.m) / (1 * u.km)

dimless_y

dimless_y.unit

dimless_y.decompose()   # returns the scale of the dimentionless quanity


I am working on the file "Python_Units.ipynb".
The first task is:
For any unit you can find all of the built-in units that are equivalent:
Can you write Python code for it?


I am working on the file "Python_Units.ipynb".
The first task is:
Note: because we imported the units package as u, you cannot use u as a variable name.
You can use the units by themselves:
Can you write Python code for it?

u.m    # The unit of meters

u.s    # The unit of seconds

u.m / u.s    # combine them into a composite unit


I am working on the file "Python_Units.ipynb".
The first task is:
Notice that the units are a bit strange. We can simplify this using .decompose()
Can you write Python code for it?


I am working on the file "Python_Units.ipynb".
The first task is:
Or they expect the correct type of unit!
Can you write Python code for it?


I am working on the file "Python_Units.ipynb".
The first task is:
QTables vs. DataFrames

Astropy tables are constructed to use units.
The units of the column will be saved with the table

Part II - Advantage Astropy
Can you write Python code for it?

planet_table = QTable.read('Planets.csv', format='ascii.csv')

planet_table[0:3]

planet_table['a'].unit = u.AU
planet_table[0:3]


I am working on the file "Python_Units.ipynb".
The first task is:
Units make math with Time easy!
Can you write Python code for it?

u.s.find_equivalent_units()

time_3 = 29 * u.day + 7 * u.h + 56 * u.min + 12 * u.s
time_4 = 2 * u.fortnight + 1.33 * u.day

time_3 - time_4

(time_3 - time_4).to(u.min)


I am working on the file "Python_Units.ipynb".
The first task is:
You can access the number and unit part of the Quantity separately:
Can you write Python code for it?


I am working on the file "Python_Units.ipynb".
The first task is:
You can define your own units
Can you write Python code for it?

ringo = u.def_unit('Ringos', 3.712 * imperial.yd)

position.to(ringo)

velocity.to(ringo / u.s)


I am working on the file "Python_and_physics/Lorena_codigo.ipynb".
The first task is:
Creamos un diccionario vacío que se llama p

Le agregamos los diferentes parámetros que necesitamos usando la siguiente sintaxis:

p ['Llave'] = valor

El nombre del diccionario, entre corchetes el nombre de la llave (en mi caso el nombre del parámetro) signo de igual y el valor (el valor podría ser una str también, sólo es necesario ponerlo entre comillas).

Can you write Python code for it?


I am working on the file "Python_and_physics/Lorena_codigo.ipynb".
The first task is:
Escribimos la función solve, esta función integra las dos ecuaciones diferenciales que describimos arriba ('rhs'), utilizando como condiciones iniciales los valores del diccionario w0 y V0 y el tiempo descrito en el diccionario como SampTimes
Can you write Python code for it?

def solve (p):
    orbit =  sc.integrate.odeint(p['rhs'],p['z0'], p['sampTimes'],args=(p,)).transpose()
    vorbit = orbit[0]
    worbit = orbit[1]
    xx = {'v':vorbit, 'w':worbit, 'sampTimes':p['sampTimes']}
    return xx


I am working on the file "Python_and_physics/Lorena_codigo.ipynb".
The first task is:
Hasta ahora lo que hicimos fue definir las funciones que queremos resolver con valores de parámetros únicos (los dados en el diccionario) y resolver las ecuaciones utilizando una herramienta de scipy para integrar.

Pero lo que yo quería hacer era resolver la ecuación para distintos valores de un mismo parámetros y luego graficar el comportamiento del sistema para poder visualizar las diferencias.

Para esto creo una función que varíe los parámetros (paramVar)
Can you write Python code for it?

def paramVars(pa, key = 'V12w', outString = r'$V12w$'):
    vals = sc.arange(-50,-20,3)  # es un arreglo (como una lista) con los valores que voy a utlizar para el parámetro que escogí.
    simulations = list()  # es una lista vacía en la que se van a ir agregando las soluciones para cada valor del parámetro.
    nsims = len(vals)    # nsims nos da cuantos valores vamos a usar, cuantas veces vamos a resolver la ecuación.
    for n in sc.arange(nsims): # Creamos un for loop para cada uno de los valores en el arreglo que construimos arriba.
        p = pa.copy()   #Hacemos una copia del diccionario p sobre el cual vamos a variar el valor del parámetro.
        p[key] = vals[n]   # Cambiamos el valor del parámetro por el valor n de nuestro arreglo.
        #print('Performing simulations with %s=%g'%(key,p[key]))
        xx = solve(p)  # resolvemos las ecuaciones usando la función solve para el valor del parámetro n.
        simulations.append(xx)  # Agregamos la solución a la lista (antes vacía)
        #Ahora tenemos una lista llamada 'simulations' en la que se encuentran las soluciones para cada uno de los valores n
        # que tomó el parámetro que escogimos.
        #print simulations

    if 1:
        cols = 3; rows = sc.ceil(nsims/sc.float32(cols))
        fig = py.figure(figsize = (14,8)) #creamos una figura en la que van a estar las gráficas y le damos un tamaño.
        py.ioff()
        ax = []  #Creamos una lista vacía
        for n in sc.arange(nsims): #Creamos un for loop, que va a hacer una gráfica por cada valor n del parámetro.
            ax.append(fig.add_subplot(rows,cols,n+1))  #Agregamos a la lista un subplot una gráfica dentro de la figura.
            ax[n].plot(simulations[n]['sampTimes'], simulations[n]['v']) #Aquí le decimos que graficar en la figura que acaba de crear
            str1 = r'%s=%g'%(outString, vals[n])
            ax[n].set_ylim(-90,70)  # Le damos los límites del eje y
            ax[n].set_xlabel('tiempo (ms)') # El nombre del eje x
           ...


I am working on the file "Python_jupyter_utilities/Magic_notebook.ipynb".
The first task is:
Manage packages in anaconda
Can you write Python code for it?

# check all packages in conda
conda_packages = !conda list
conda_packages

# check packages in pip
pip_packages = !pip freeze
pip_packages

## check which module are in python
help('modules')


I am working on the file "Python_jupyter_utilities/Magic_notebook.ipynb".
The first task is:
Other display utilities
Can you write Python code for it?

from IPython.display import YouTubeVideo, display
YouTubeVideo("yH-7XWLm_Po")
# vid = YouTubeVideo("yH-7XWLm_Po")
# display(vid)

# diplay in interactive way embedded in jupyer notebook
%matplotlib notebook

import pandas as pd
df = pd.DataFrame({
        'Letter': ['a', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c'],
        'X': [4, 3, 5, 2, 1, 7, 7, 5, 9],
        'Y': [0, 4, 3, 6, 7, 10, 11, 9, 13],
        'Z': [1, 2, 3, 1, 2, 3, 1, 2, 3]
    })

df.plot()


I am working on the file "Python_jupyter_utilities/Magic_notebook.ipynb".
The first task is:
Other usefuls:

Nice shortcuts for jupyter notebook: https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/
Can you write Python code for it?

## allow to reload the module when the module is modified (useful to debug your own imported scripts)
%load_ext autoreload
%autoreload 2

# magic and bash call return python variables
names = !ls
names

# print all statements as if they would have been the last one
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

from pydataset import data
quakes = data('quakes')
quakes.head()
quakes.tail()

# By adding a semicolon at the end, the output is suppressed.
%matplotlib inline
from matplotlib import pyplot as plt
import numpy
x = numpy.linspace(0, 1, 1000)**1.5
plt.hist(x)

plt.hist(x); # <<<<< semicolumn here


I am working on the file "Python_jupyter_utilities/myFunctions.ipynb".
The first task is:
Check separations by histogram and scatter plot
Can you write Python code for it?

def plot_classcolor(df, x='y', y='x', hue=None):
    sns.lmplot(x, y, data=df, hue=hue, fit_reg=False)
    sns.plt.title("({} vs {})".format(y, x))
    plt.show()

plot_classcolor(df, 'sepal_length', 'sepal_width', hue='species')

def plot_histo_per_class(df, var, target):
    t_list = df[target].unique()
    for t in t_list:
        sns.distplot(
            df[df[target] == t][var], kde=False, norm_hist=True, label=str(t))
    sns.plt.legend()
    sns.plt.show()

plot_histo_per_class(df, 'sepal_length', "species_id")


I am working on the file "Python_jupyter_utilities/myFunctions.ipynb".
The first task is:
Control plots: ROC, Precision-Recall, ConfusionMatrix, top_k, classification report,
Can you write Python code for it?

### VERSIONE CORRETTA A LAVORO

def plot_ROC_PrecisionRecall(y_test, y_pred):
    """
    Plot ROC curve and Precision-Recall plot
    numpy arrays are required.
    """
    fpr_clf, tpr_clf, _ = roc_curve(y_test, y_pred)
    precision, recall, thresholds = precision_recall_curve(y_test, y_pred)
    f1 = np.array([2 * p * r / (p + r) for p, r in zip(precision, recall)])
    f1[np.isnan(f1)] = 0
    t_best_f1 = thresholds[np.argmax(f1)]
    roc_auc = auc(fpr_clf, tpr_clf)
    plt.figure(figsize=(25, 25))
    # plot_ROC
    plt.subplot(221)
    plt.plot(
        fpr_clf,
        tpr_clf,
        color='r',
        lw=2,
        label='ROC curve (area = %0.2f)' % roc_auc)
    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='-')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver operating characteristic')
    plt.legend(loc="lower right")
    # plot_PrecisionRecall
    plt.subplot(222)
    plt.plot(
        recall, precision, color='r', lw=2, label='Precision-Recall curve')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.title('Precison-Recall curve')
    plt.legend(loc="lower right")

    plt.show()
    return {"roc_auc": roc_auc, "t_best_f1": t_best_f1}

def plot_ROC_PR_test_train(y_train, y_test, y_test_pred, y_train_pred):
    """Plot ROC and Precision-Recall curve for test and train.
    Return the auc for test and train"""
    roc_auc_test = plot_ROC_PrecisionRecall(y_test, y_test_pred)
    roc_auc_train = plot_ROC_PrecisionRecall(y_train, y_train_pred)
    return roc_auc_test, roc_auc_train


I am working on the file "Python_jupyter_utilities/myFunctions.ipynb".
The first task is:
FastFourier Transformation
Can you write Python code for it?

from scipy.fftpack import fft, rfft, irfft, fftfreq
def rfourier_transformation(df, var, pass_high=-1, pass_low=-1, verbose=True, plot=True):
    """Return the signal after low and high filter applied.
    Use verbose and plot to see stats and plot the signal before and after the filter.
    """
    low = pass_high
    high = pass_low

    if (high < low) and (high>0):
        print("Cannot be pass_low < pass_high!!")
        return -1

    time = pd.Series(df.index.values[1:10] -
                     df.index.values[:10 - 1])  # using the first 10 data
    dt = time.describe()['50%']
    if (verbose):
        print("""
          sampling time: {0} s
          sampling frequency: {1} hz
          max freq in rfft: {2} hz
          """.format(dt, 1 / dt, 1 / (dt * 2), 1 / (dt)))

    signal = df[var]
    freq = fftfreq(signal.size, d=dt)
    f_signal = rfft(signal)

    m = {}
    if (low > 0):
        f_signal_lowcut = f_signal.copy()
        f_signal_lowcut[(freq < low)] = 0
        cutted_signal_low = irfft(f_signal_lowcut)
        m['low'] = 1

    if (high > 0):
        f_signal_highcut = f_signal.copy()
        f_signal_highcut[(freq > high)] = 0
        cutted_signal_high = irfft(f_signal_highcut)
        m['high'] = 1

    if (high > 0) & (low > 0):
        f_signal_bwcut = f_signal.copy()
        f_signal_bwcut[(freq < low) | (freq > high)] = 0
        cutted_signal_bw = irfft(f_signal_bwcut)
        m['bw'] = 1
        m['low'] = 2
        m['high'] = 3

    n = len(freq)

    if (plot):
        f, axarr = plt.subplots(len(m) + 1, 1, sharex=True, figsize=(18,15))
        f.canvas.set_window_title(var)

        # time plot
        axarr[0].plot(signal)
        axarr[0].set_title('Signal')
        if 'bw' in m:
            axarr[m['bw']].plot(df.index, cutted_signal_bw)
            axarr[m['bw']].set_title('Signal after low-high cut')
        if 'low' in m:
            axarr[m['low']].plot(df.index, cutted_signal_low)
            axarr[m['low']].set_title('Signal after high filter (low...


I am working on the file "Python_jupyter_utilities/myFunctions.ipynb".
The first task is:
GBM skitlearn
Can you write Python code for it?

def plot_var_imp_skitlearn(features, clf_fit):
    """Plot var_imp for a skitlearn fitted model"""
    my_ff = np.array(features)
    importances = clf_fit.feature_importances_
    indices = np.argsort(importances)
    pos = np.arange(len(my_ff[indices])) + .5
    plt.figure(figsize=(20, 0.75 * len(my_ff[indices])))
    plt.barh(pos, importances[indices], align='center')
    plt.yticks(pos, my_ff[indices], size=25)
    plt.xlabel('rank')
    plt.title('Feature importances', size=25)
    plt.grid(True)
    plt.show()
    importance_dict = dict(zip(my_ff[indices], importances[indices]))
    return importance_dict


I am working on the file "Python_jupyter_utilities/myFunctions.ipynb".
The first task is:
LightGBM
New way to install the package:
- pip install lightgbm
Old way to install the package
- build the package from https://github.com/Microsoft/LightGBM/wiki/Installation-Guide
- add the library to python using: python setup.py install inside the python package of the github clone
Problem: when you compile a package with the linux compiler and then you use it with the anaconda compiler you need to have the same compiler on both.
This cannot be the default and to fix this do:
- cd ~/anaconda3/lib
- mv -vf libstdc++.so.6 libstdc++.so.6.old
- ln -s /usr/lib/x86_64-linux-gnu/libstdc++.so.6 ./libstdc++.so.6
So now the shared library seen from anaconda is the same that have been usage when the package was compiled.
These steps were necessary to install lightgbm
Can you write Python code for it?


I am working on the file "Python_jupyter_utilities/myFunctions.ipynb".
The first task is:
PCA
Can you write Python code for it?

from sklearn import decomposition

def pca_decomposition(df):
    """Perform sklearn PCA. The returned components are already ordered by the explained variance"""
    pca = decomposition.PCA()
    pca.fit(df)
    return pca

def pca_stats(pca):
    print("variance explained:\n", pca.explained_variance_ratio_)
    print("pca components:\n", pca.components_)

def plot_classcolor(df, x='y', y='x', hue=None):
    sns.lmplot(x, y, data=df, hue=hue, fit_reg=False)
    sns.plt.title("({} vs {})".format(y, x))
    plt.show()

def add_pca_to_df(df, allvars, pca):
    df[["pca_" + str(i) for i, j in enumerate(pca.components_)
        ]] = pd.DataFrame(pca.fit_transform(df[allvars]))

pca = pca_decomposition( df[['sepal_length', 'sepal_width', 'petal_length', 'petal_width']] )

pca_stats(pca)

add_pca_to_df(df, ['sepal_length', 'sepal_width', 'petal_length', 'petal_width'], pca)

plot_classcolor(df, 'pca_0', 'pca_1', 'species_id')


I am working on the file "Python_jupyter_utilities/myFunctions.ipynb".
The first task is:
Plot Cumulative Lift
Can you write Python code for it?

def plotLift(df, features, target, ascending=False, multiclass_level=None):
    """
    Plot the Lift function for all the features.
    Ascending can be a list of the same feature length or a single boolean value.
    For the multiclass case you can give the value of a class and the lift is calculated
    considering the select class vs all the other
    """
    if multiclass_level != None:
        df = df[features+[target]].copy()
        if multiclass_level != 0:
            df.loc[df[target] != multiclass_level, target] = 0
            df.loc[df[target] == multiclass_level, target] = 1
        else :
            df.loc[df[target] == multiclass_level, target] = 1
            df.loc[df[target] != multiclass_level, target] = 0

    npoints = 100
    n = len(df)
    st = n / npoints
    df_shuffled = df.sample(frac=1)

    flat = np.array([[(i * st) / n, df_shuffled[0:int(i * st)][target].sum()]
                     for i in range(1, npoints + 1)])
    flat = flat.transpose()
    to_leg = []

    if not isinstance(features, list):
        features = [features]
    if not isinstance(ascending, list):
        ascending = [ascending for i in features]

    for f, asc in zip(features, ascending):
        a = df[[f, target]].sort_values(f, ascending=asc)
        b = np.array([[(i * st) / n, a[0:int(i * st)][target].sum()]
                      for i in range(1, npoints + 1)])
        b = b.transpose()
        to_leg.append(plt.plot(b[0], b[1], label=f)[0])
    to_leg.append(plt.plot(flat[0], flat[1], label="no_gain")[0])
    plt.legend(handles=to_leg, loc=4)
    plt.xlabel('faction of data', fontsize=18)
    plt.ylabel(target+' (cumulative sum)', fontsize=16)

    plt.show()

# Lift for regression
titanic = sns.load_dataset("titanic")
plotLift(titanic, ['sibsp', 'survived', 'class'], 'fare', ascending=[False,False, True])

# Lift plot example for multiclass
plotLift(
    df, ['sepal_length', 'sepal_width', 'petal_length'],
    'species_id',
    ascending=[False, True, False],
    multiclass_level=3)


I am working on the file "Python_jupyter_utilities/myFunctions.ipynb".
The first task is:
Rotations
Can you write Python code for it?

def rotMat3D(a,r):
    """Return the matrix that rotate the a vector into the r vector. numpy array are required"""

    a = a/np.linalg.norm(a)
    r = r/np.linalg.norm(r)

    I = np.eye(3)
    v = np.cross(a,r)
    c = np.inner(a,r)
    v_x = np.array([[0,-v[2],v[1]],[v[2],0,-v[0]],[-v[1],v[0],0]])

    return I + v_x + np.matmul(v_x,v_x)/(1+c)

# example usage
z_old = np.array([0, 0, 1])
z = np.array([1, 1, 1])
R = rotMat3D(z, z_old)

print(z, R.dot(z))
print(z_old, R.dot(z_old))
print(np.linalg.norm(z), np.linalg.norm(R.dot(z)))

def createR2D(vector):
    """rotate the vector to [0,1], require numpy array"""
    m = np.linalg.norm(vector)
    c, s = vector[1]/m , vector[0]/m
    R2 = np.array([c, -s, s, c]).reshape(2,2)
    return R2

# example usage
y_old = np.array([3,4])
R2 = createR2D(y_old)

print(y_old, R2.dot(y_old))


I am working on the file "Python_jupyter_utilities/myFunctions.ipynb".
The first task is:
Xgboost
To instal xgboost: https://github.com/dmlc/xgboost/tree/master/python-package
- conda install py-xgboost (pip install xgboost)
Can you write Python code for it?

import xgboost

#### VERSIONE GIUSTA A LAVORO

def plot_var_imp_xgboost(model, mode='gain', ntop=-1):
    """Plot the vars imp for xgboost model, where mode = ['weight','gain','cover']
    'weight' - the number of times a feature is used to split the data across all trees.
    'gain' - the average gain of the feature when it is used in trees
    'cover' - the average coverage of the feature when it is used in trees
    """
    importance = model.get_score(importance_type=mode)
    importance = sorted(
        importance.items(), key=operator.itemgetter(1), reverse=True)
    if ntop == -1: ntop = len(importance)
    importance = importance[0:ntop]
    my_ff = np.array([i[0] for i in importance])
    imp = np.array([i[1] for i in importance])
    indices = np.argsort(imp)
    pos = np.arange(len(my_ff[indices])) + .5
    plt.figure(figsize=(20, 0.75 * len(my_ff[indices])))
    plt.barh(pos, imp[indices], align='center')
    plt.yticks(pos, my_ff[indices], size=25)
    plt.xlabel('rank')
    plt.title('Feature importances (' + mode + ')', size=25)
    plt.grid(True)
    plt.show()
    return


I am working on the file "Python_jupyter_utilities/pyHDF5.ipynb".
The first task is:
From pandas dataframe to hdf5 and viceversa
Can you write Python code for it?

!rm iris.hdf5

import pandas as pd
df = pd.read_csv('iris.csv')
df.to_hdf(path_or_buf='iris.hdf5', key='iris_', mode='w', format='fixed')
# fixed format: Fast writing/reading. Not-appendable, nor searchable
# table forma: Write as a PyTables Table structure which may perform worse
#              but allow more flexible operations like searching selecting subsets of the data

# reopen the file and check the structure and how it has stored the columns etc..
newf = h5py.File('iris.hdf5')

newf.visit(printname)

for i in newf.keys():
    for j in newf[i].keys():
        print(newf[i + '/' + j])

for i in newf.keys():
    for j in newf[i].keys():
        print(newf[i + '/' + j].name)
        print(newf[i + '/' + j].value, end='\n\n')

newf.close()

# read an hdf5 file to pandas dataframe
df_hdf5 = pd.read_hdf('iris.hdf5')
df_hdf5.head()

## append data to the previous data on the hdf5
df.to_hdf('iris.hdf5', 'data', append=True, format='table')
df.to_hdf('iris.hdf5', 'data', append=True, format='table') # since the format is table you can append stak data

# to retrive the data you have to use the key (here data or iris_)
df_hdf5 = pd.read_hdf('iris.hdf5',key='data')
print(len(df_hdf5))
df_hdf5


I am working on the file "Python_jupyter_utilities/pyHDF5.ipynb".
The first task is:
dataset
Can you write Python code for it?

# you can create group or dataset into the file
# Using existing data:
data = np.random.logistic(size=100)
dsetdata = f.create_dataset("dsetdata", data=data)
print(dsetdata)

# Creating the dataset without data
dsetname = f.create_dataset(
    "dsetname", shape=(100, ), dtype='i')  # i, f, etc..
print(dsetname)

dsetdata.dtype, dsetdata.shape, dsetdata.size

# check data in filename
for i in f.items():
    print(i)

# support array-style slicing, example of read/write in this cell
dsetname[0] = 5
dsetname[5:15] = np.random.uniform(size=10,low=0, high=5)
dsetname[10:20:2]

dsetname.value, dsetname.name, dsetname.fillvalue, dsetname.shuffle


I am working on the file "Python_scientific/.ipynb_checkpoints/numpy_matplot_ejemplos-checkpoint.ipynb".
The first task is:

#Matplotlib

## Es el modulo para realizar y visualizar graficos

[Pagina oficial](http://matplotlib.org/)
Can you write Python code for it?

#para que los graficos queden empotrados
%pylab inline

X = np.linspace(-np.pi, np.pi, 256, endpoint=True)
C, S = np.cos(X), np.sin(X)

plot(X, C)
plot(X, S)

t = 2 * np.pi / 3

plot(X, C, color="blue", linewidth=2.5, linestyle="-", label="cosine")
plot(X, S, color="red",  linewidth=2.5, linestyle="-", label="sine")
plot([t, t], [0, np.cos(t)], color='blue', linewidth=2.5, linestyle="--")
scatter([t, ], [np.cos(t), ], 50, color='blue')

annotate(r'$sin(\frac{2\pi}{3})=\frac{\sqrt{3}}{2}$',
            xy=(t, np.sin(t)), xycoords='data',
            xytext=(+10, +30), textcoords='offset points', fontsize=16,
            arrowprops=dict(arrowstyle="->", connectionstyle="arc3,rad=.2"))

plot([t, t],[0, np.sin(t)], color='red', linewidth=2.5, linestyle="--")
scatter([t, ],[np.sin(t), ], 50, color='red')

annotate(r'$cos(\frac{2\pi}{3})=-\frac{1}{2}$',
            xy=(t, np.cos(t)), xycoords='data',
            xytext=(-90, -50), textcoords='offset points', fontsize=16,
            arrowprops=dict(arrowstyle="->", connectionstyle="arc3,rad=.2"))


I am working on the file "Python_scientific/.ipynb_checkpoints/numpy_matplot_ejemplos-checkpoint.ipynb".
The first task is:
Ahora los ponemos a prueba
Can you write Python code for it?

a = np.random.rand(10000)

%%timeit
py_dcumsum(a)

%%timeit
dcumsum.dcumsum(a)

%%timeit
a.cumsum()


I am working on the file "Python_scientific/.ipynb_checkpoints/numpy_matplot_ejemplos-checkpoint.ipynb".
The first task is:
Caracteristicas y utilidades principales:
Can you write Python code for it?

#Creando arrays

a = np.array([1,1,1]) # 1D
b = np.array([[1,1,1],[1,1,1]]) #2d (Matrix)
c = np.array([[[1,1,1],[1,1,1],[1,1,1]]]) #3D (Tensor...)

print a.shape
print b.shape
print c.shape

#Podemos crear arrays predeterminados con funciones muy utiles

a = np.arange(10) # un array de enteros 0 a 10(no lo incluye)
a

# Array de 1 a 9 con paso 2
b = np.arange(1, 9, 2)
b

#Como en Matlab
a = np.ones((3,3))
a

b = np.zeros((5,5))
b

c = np.eye(10)
c

d = np.diag(np.array([1, 2, 3, 4]))
d

#Complex numbers
e = np.array([1+2j, 3+4j, 5+6*1j])
e

#boolean
e = np.array([True, False, False, True])
e

#String
f = np.array(['Bonjour', 'Hello', 'Hallo',])
f


I am working on the file "Python_scientific/.ipynb_checkpoints/numpy_matplot_ejemplos-checkpoint.ipynb".
The first task is:
Compilamos directamente a un modulo de python
Can you write Python code for it?

!f2py -c dcumsum.f -m dcumsum

#importamos el modulo recien creado

import dcumsum

a = np.array([1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0])

py_dcumsum(a)

dcumsum.dcumsum(a)


I am working on the file "Python_scientific/.ipynb_checkpoints/numpy_matplot_ejemplos-checkpoint.ipynb".
The first task is:
Cubo magico hecho solo con Matplotlib
Can you write Python code for it?


I am working on the file "Python_scientific/.ipynb_checkpoints/numpy_matplot_ejemplos-checkpoint.ipynb".
The first task is:
Ejemplo 2: suma acumulativa, entrada vector, salida vector
Primero hacemos la implementacion en Python puro, este tipo de suma es particularmente costoso ya que se necesitan loops for
Can you write Python code for it?

# Esta no es la mejor implementacion
#Porque el loop esta implementado en Python
def py_dcumsum(a):
    b = np.empty_like(a)
    b[0] = a[0]
    for n in range(1,len(a)):
        b[n] = b[n-1]+a[n]
    return b


I am working on the file "Python_scientific/.ipynb_checkpoints/numpy_matplot_ejemplos-checkpoint.ipynb".
The first task is:
Elementwise operations
Operar con funciones en cada elemento de un array(NO USE FOR)
Can you write Python code for it?

a = np.array([1, 2, 3, 4])
a

#todas las operaciones aritmeticas funcionan
a+1

j = np.arange(5)
2**(j + 1) - j

#Multiplicacion ES ELEMENTO A ELEMENTO
a * a

#Para hacer multiplicaciones matriciales usamos dot

b = np.random.rand(3,3)
c = np.random.rand(3,3)

np.dot(b,c)

#cado objeto ndarray tiene muuchos metodos eje
c.sum()


I am working on the file "Python_scientific/.ipynb_checkpoints/numpy_matplot_ejemplos-checkpoint.ipynb".
The first task is:
Guauuuu¡¡¡¡
Ejemplo de vectorizacion
Can you write Python code for it?

%run srinivasan_pruebas.py

%run srinivasan_pruebas_vec.py


I am working on the file "Python_scientific/.ipynb_checkpoints/numpy_matplot_ejemplos-checkpoint.ipynb".
The first task is:
Python posee por defecto un tipo de datos que se asemeja(listas), pero es numéricamente ineficiente
Can you write Python code for it?

#Ejemplo creo una lista de Python de 0 a 1000 y calculo el cuadrado de cada elemento
L = range(1000)

%%timeit
[i**2 for i in L]

#Ahora hago lo mismo con Numpy
a = np.arange(1000)

%%timeit
a**2

print 'Numpy es '
print  111/5.4
print 'veces mas rapido'


I am working on the file "Python_scientific/numpy_matplot_ejemplos.ipynb".
The first task is:

#Matplotlib

## Es el modulo para realizar y visualizar graficos

[Pagina oficial](http://matplotlib.org/)
Can you write Python code for it?


I am working on the file "Python_scientific/numpy_matplot_ejemplos.ipynb".
The first task is:
Ahora los ponemos a prueba
Can you write Python code for it?


I am working on the file "Python_scientific/numpy_matplot_ejemplos.ipynb".
The first task is:
Compilamos directamente a un modulo de python
Can you write Python code for it?


I am working on the file "Python_scientific/numpy_matplot_ejemplos.ipynb".
The first task is:
Ejemplos
Ejemplo 1 Ecuaciones de Navier-Stokes
con convección no-lineal en 2D
Now we solve 2D Convection, represented by the pair of coupled partial differential equations below:
$$\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} + v \frac{\partial u}{\partial y} = 0$$
$$\frac{\partial v}{\partial t} + u \frac{\partial v}{\partial x} + v \frac{\partial v}{\partial y} = 0$$
Discretizing these equations using the methods we've applied previously yields:
$$\frac{u_{i,j}^{n+1}-u_{i,j}^n}{\Delta t} + u_{i,j}^n \frac{u_{i,j}^n-u_{i-1,j}^n}{\Delta x} + v_{i,j}^n \frac{u_{i,j}^n-u_{i,j-1}^n}{\Delta y} = 0$$
$$\frac{v_{i,j}^{n+1}-v_{i,j}^n}{\Delta t} + u_{i,j}^n \frac{v_{i,j}^n-v_{i-1,j}^n}{\Delta x} + v_{i,j}^n \frac{v_{i,j}^n-v_{i,j-1}^n}{\Delta y} = 0$$
Initial Conditions
The initial conditions are the same that we used for 1D convection, applied in both the x and y directions.
$$u,\ v\ = \begin{cases}\begin{matrix}
2 & \text{for } x,y \in (0.5, 1)\times(0.5,1) \cr
1 & \text{everywhere else}
\end{matrix}\end{cases}$$
Boundary Conditions
The boundary conditions hold u and v equal to 1 along the boundaries of the grid
.
$$u = 1,\ v = 1 \text{ for } \begin{cases} \begin{matrix}x=0,2\cr y=0,2 \end{matrix}\end{cases}$$
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np

###variable declarations
nx = 101
ny = 101
nt = 80
c = 1
dx = 2.0/(nx-1)
dy = 2.0/(ny-1)
sigma = .2
dt = sigma*dx

x = np.linspace(0,2,nx)
y = np.linspace(0,2,ny)

u = np.ones((ny,nx)) ##create a 1xn vector of 1's
v = np.ones((ny,nx))
un = np.ones((ny,nx))
vn = np.ones((ny,nx))

###Assign initial conditions

u[.5/dy:1/dy+1,.5/dx:1/dx+1]=2 ##set hat function I.C. : u(.5<=x<=1 && .5<=y<=1 ) is 2
v[.5/dy:1/dy+1,.5/dx:1/dx+1]=2 ##set hat function I.C. : u(.5<=x<=1 && .5<=y<=1 ) is 2

for n in range(nt+1): ##loop across number of time steps
    un[:] = u[:]
    vn[:] = v[:]

    u[1:,1:]=un[1:,1:]-(un[1:,1:]*dt/dx*(un[1:,1:]-un[0:-1,1:]))-vn[1:,1:]*dt/dy*(un[1:,1:]-un[1:,0:-1])
    v[1:,1:]=vn[1:,1:]-(un[1:,1:]*dt/dx*(vn[1:,1:]-vn[0:-1,1:]))-vn[1:,1:]*dt/dy*(vn[1:,1:]-vn[1:,0:-1])

    u[0,:] = 1
    u[-1,:] = 1
    u[:,0] = 1
    u[:,-1] = 1

    v[0,:] = 1
    v[-1,:] = 1
    v[:,0] = 1
    v[:,-1] = 1

from matplotlib import cm ##cm = "colormap" for changing the 3d plot color palette
fig = plt.figure(figsize=(11,7), dpi=100)
ax = fig.gca(projection='3d')
X,Y = np.meshgrid(x,y)

ax.plot_surface(X,Y,u, cmap=cm.coolwarm)


I am working on the file "Python_scientific/numpy_matplot_ejemplos.ipynb".
The first task is:
Generamos un modulo de Python con f2py
Can you write Python code for it?


I am working on the file "Python_scientific/numpy_matplot_ejemplos.ipynb".
The first task is:
Guauuuu¡¡¡¡
Ejemplo de vectorizacion
Can you write Python code for it?


I am working on the file "Python_scientific/numpy_matplot_ejemplos.ipynb".
The first task is:
Importamos el modulo que generamos y lo utilizamos
Can you write Python code for it?

%%file hello.py
import hellofortran

hellofortran.hellofortran(5)

# corremos el script
!python hello.py


I am working on the file "Python_scientific/numpy_matplot_ejemplos.ipynb".
The first task is:
Indexing and slicing
Los items de un array pueden accederse de una manera natural como en Matlab(hay que tener en cuenta que los indices comienzan en 0)
Can you write Python code for it?

a = np.arange(10)
a

#creo una tupla
a[0],a[2],a[-1]

# Slicing [comienzo:final:paso]
# Los tres no son necesarios explicitamente ya que por default comienzo=0, final=[-1] y el paso=1

a[2:8:2]

a[::4] # solo cambiamos el paso


I am working on the file "Python_scientific/numpy_matplot_ejemplos.ipynb".
The first task is:
Optimizaciones con Fortran y C

Can you write Python code for it?

%%file hellofortran.f
C File  hellofortran.f
        subroutine hellofortran (n)
        integer n

        do 100 i=0, n
            print *, "Hola Soy Fortran tengo muuchos años"
100     continue
        end


I am working on the file "Python_scientific/numpy_matplot_ejemplos.ipynb".
The first task is:
Python posee por defecto un tipo de datos que se asemeja(listas), pero es numéricamente ineficiente
Can you write Python code for it?


I am working on the file "Python_tutorial.ipynb".
The first task is:
Au and Ag variables reassigned to floating point values
Floats use a decimal point or exponential notation. Note that values such as 20.0, 1.0, 0.0 etc. are floats, not integers. Also, floating point values may not be a true representation of the number since computer use a binary (base-2) number system. This leads to floating point errors when working with floating point values.
Can you write Python code for it?

Ag = 106.9

Au = 197.0

type(Ag)

type(Au)


I am working on the file "Python_tutorial.ipynb".
The first task is:
Boolean operators
Boolean operators asses the truth or falseness of a statement.
Can you write Python code for it?

Ag > Au

Ag < Au

Ag == 106.9

Au >= 100

Ag <= Au and Ag > 200

Ag <= Au or Ag > 200


I am working on the file "Python_tutorial.ipynb".
The first task is:
Collection data types
Lists
Lists are mutable (i.e., modifiable), ordered collections of items. Lists are created by enclosing a collection of items with square brackets. An empty list may also be created simply by assigning [] to a variable, i.e., empty_list = [].
Can you write Python code for it?


I am working on the file "Python_tutorial.ipynb".
The first task is:
Conditional statements
Code is only executed if the conditional statement is evaluated as True. In the following example, Ag has a value of greater than 100 and therefore only the "Ag is greater than 100 Da" string is printed. A colon follows the conditional statement and the following code block is indented by 4 spaces (always use 4 spaces rather than tabs - errors will resulting when mixing tabs with spaces!). Note, the elif and else statements are optional.
Can you write Python code for it?

if Ag < 100:
    print("Ag is less than 100 Da")
elif Ag > 100:
    print("Ag is greater than 100 Da.")
else:
    print("Ag is equal to 100 Da.")


I am working on the file "Python_tutorial.ipynb".
The first task is:
Dictionaries
Dictionaries are mutable, unordered collections of key: value pairs. Dictionaries are created created by enclosing key: value pairs with curly brackets. Importantly, keys must be hashable. This means, for example, that lists can't be used as keys since the items inside a list may be modified.
Can you write Python code for it?


I am working on the file "Python_tutorial.ipynb".
The first task is:
Dictionary methods
Can you write Python code for it?

carbon_isotopes.keys()

carbon_isotopes.values()

carbon_isotopes.items()


I am working on the file "Python_tutorial.ipynb".
The first task is:
Exercise: Calculate peptide masses
In the following example, we will calculate the mass of a peptide from a string containing one letter amino acid residue codes. For example, peptide = "GASPV". To do this, we will first need a dictionary containing the one letter codes as keys and the masses of the amino acid residues as values. We will then need to create a variable to store the mass of the peptide and use a for loop to iterate over each amino acid residue in the peptide.
Can you write Python code for it?


I am working on the file "Python_tutorial.ipynb".
The first task is:
Fetching the value for a certain key
Can you write Python code for it?


I am working on the file "Python_tutorial.ipynb".
The first task is:
Functions
Functions perform a specified task when called during the execution of a program. Functions reduce the amount of code that needs to be written and greatly improves code readability. (Note: readability matters!) The for loop created above is better placed in a function so that the for loop doesn't need to be re-written everytime we wish to calculate the mass of a peptide. Pay careful attention to the syntax below.
Can you write Python code for it?

def peptide_mass(peptide):
    mass = 18.010565
    for amino_acid in peptide:
        mass += amino_dict[amino_acid]
    return mass

peptide_mass(peptide_name)


I am working on the file "Python_tutorial.ipynb".
The first task is:
Indexing
Can you write Python code for it?


I am working on the file "Python_tutorial.ipynb".
The first task is:
Math operations
In Python 3, math operations behave as you would expect. In Python 2, division is the same as floor division when working with integers!
Can you write Python code for it?

Au + Ag # addition (Note: You can add comments to your code by using the # symbol)

Au - Ag # subtraction

Au * 5 # multiplication

Ag ** 2 # exponential - mass of silver squared in this case

Au / Ag # division

Au // Ag # floor division


I am working on the file "Python_tutorial.ipynb".
The first task is:
Sets
Sets are another data type which are like an unordered list with no dublicates. They are especially useful for finding all the unique items from a list as shown below.
Can you write Python code for it?

phospholipids = ["PA(16:0/18:1)", "PA(16:0/18:2)", "PC(14:0/16:0)", "PC(16:0/16:1)", "PC(16:1/16:2)"]
# Lets assume we apply a function that finds the type of phospholipid name to
phospholipid_fatty_acids = ["16:0", "18:1", "16:0", "18:2", "14:0", "16:0", "16:0", "16:1", "16:1", "16:2"]

unique_fatty_acids = set(phospholipid_fatty_acids)

unique_fatty_acids

num_unique_fa = len(unique_fatty_acids)

num_unique_fa


I am working on the file "Python_tutorial.ipynb".
The first task is:
Several list 'methods' exist for manipulating lists
Can you write Python code for it?

MS_files.remove("MS2_405")

MS_files

MS_files.append("MS3_225")

MS_files


I am working on the file "Python_tutorial.ipynb".
The first task is:
Strings and indexing
In the following example, we have a file name represented as a string. From this string, we can use to indexing to select certain characters or substrings from the file name string.
Strings
String are simply strings of characters (letters, numbers, symbols etc.). Strings are indicated using either single quotes ('This is a string') or double quotes ("This is another string"). Multiline strings can be made using triple quotes (i.e. """Multiline string""").
Can you write Python code for it?

MS2_spectrum = "Liver_MS2_406.raw"

MS2_spectrum


I am working on the file "Python_tutorial.ipynb".
The first task is:
Type conversion
We can convert a float to an integer or an integer to a float. Conversions of string representations of numbers to actual numbers (integers and floats) is also common in Python programming.
Can you write Python code for it?

integer_Ag = int(Ag)

integer_Ag

type(integer_Ag)


I am working on the file "Python_tutorial.ipynb".
The first task is:
Type conversion - string to float
Can you write Python code for it?


I am working on the file "Python_tutorial.ipynb".
The first task is:
User input
A simple means to gather user inputted data is to use input. This will prompt the user to enter data which may be used within the program. In the example below, we prompt the user to enter a peptide name. The peptide name is then used for the function call to calculate the peptide's mass.
Can you write Python code for it?

user_peptide = input("Enter peptide name: ")

peptide_mass(user_peptide)


I am working on the file "Python_tutorial.ipynb".
The first task is:
We could also add a predicate to a list comprehension. Here, we calculate the mass of lipids less than 700 Da.
Can you write Python code for it?

adducts_comp = [mass + Na for mass in lipid_masses if mass < 700]

adducts_comp


I am working on the file "Python_tutorial.ipynb".
The first task is:
While and for loops with conditional statements
Both while and for loops can be combined with conditional statements for greater control of flow within a program.
Can you write Python code for it?

mass_spectrometers = 0
while mass_spectrometers < 5:
    mass_spectrometers += 1
    print("Number of mass spectrometers equals", mass_spectrometers)
    if mass_spectrometers == 1:
        print("Woohoo, the first of many!")
    elif mass_spectrometers == 5:
        print("That'll do for now.")
    else:
        print("More!!")

for MS_file in MS_files:
    if "spectrum" in MS_file:
        print("MS file:", MS_file)
    elif "MS2" in MS_file:
        print("MS2 file:", MS_file)
    else:
        print("MS3 file:", MS_file)


I am working on the file "Python教程.ipynb".
The first task is:
1.2 注释：python 中可以进行单行注释和多行注释
Can you write Python code for it?

# 这是一个单行注释

'''
这是一个多行注释
'''

print('Hello world')


I am working on the file "Python教程.ipynb".
The first task is:
1.3 缩进：python中强制采用四个空格作为缩进，作用等同于{}
Can you write Python code for it?

for i in range(3):
    print(i)

for i in range(3):
print(i)


I am working on the file "Python教程.ipynb".
The first task is:
1.4 多行语句：多行语句使用反斜杠(\)实现
Can you write Python code for it?

item_one = 1
item_two = 2
item_three = 3

total = item_one + \
        item_two + \
        item_three
total


I am working on the file "Python教程.ipynb".
The first task is:
1.5 格式化输出
Can you write Python code for it?

a = 1
b = 'apple'
print('a = %d and b = %s' % (a, b))

print('a = {} and b = {}'.format(a, b))

print('a = {1} and b = {0}'.format(a, b))


I am working on the file "Python教程.ipynb".
The first task is:
2. 数据类型
python中有六个标准的数据类型：

Number(数字)
String(字符串)
List(列表)
Tuple(元组)
Sets(集合)
Dict(字典)

所有变量不需要提前定义，直接赋值即可
2.1 数字&字符串
Can you write Python code for it?

a = 1
b = 0.2
c = 'Hello'

print('type(a) = {}, a = {}'.format(type(a), a))
print('type(b) = {}, b = {}'.format(type(b), b))
print('type(c) = {}, c = {}'.format(type(c), c))


I am working on the file "Python教程.ipynb".
The first task is:
2.2 元组：元组用来存储一族数据，定义以后不可修改
Can you write Python code for it?

tupleA = (1, 2, 3, 4)
tupleA = (1, 2, 3, 'apple')
print(tupleA[0])
print(tupleA[0:2])

tupleA[0] = 1


I am working on the file "Python教程.ipynb".
The first task is:
2.4 集合：用来存储一族互不相同的数据，可以做集合运算

|: 并集
&: 交集
-: 差集
Can you write Python code for it?

a = {1, 2}
b = {2, 3}

print('a和b的并集为：{}'.format(a | b))
print('a和b的交集为：{}'.format(a & b))
print('a和b的差集为：{}'.format(a - b))


I am working on the file "Python教程.ipynb".
The first task is:
2.5 字典：用于保存键值对，实现"switch"的功能
Can you write Python code for it?

data = {'name':'马旭辉', 'gender':'男'}
print(data['name'])


I am working on the file "Python教程.ipynb".
The first task is:
3. 扩展包
3.1 安装扩展包
pip install numpy
pip3 install numpy
conda install numpy
3.2 环境管理
由于python中各种扩展包之间存在依赖关系强烈建议使用conda进行环境管理，开发环境可以安装anaconda会自带这些工具

新建一个环境
conda create -n myenv_name python = 3.5
启动一个环境
activate myenv_name(windows);source activate myenv_name(linux)

3.3 导入扩展包
Can you write Python code for it?

import time
import numpy as np

print(time.asctime())
print(np.array([1, 2, 3, 4]))


I am working on the file "Python教程.ipynb".
The first task is:
4. 条件控制
if condition_1:
    statement_block_1
elif condition_2:
    statement_block_2
else:
    statement_block_3
Can you write Python code for it?

a = 5

if a>3:
    print('a > 3')
elif a==3:
    print('a == 3')
else:
    print('a < 3')

a = 100
if a:
    print('a != 0')
else:
    print('a == 0')


I am working on the file "Python教程.ipynb".
The first task is:
5. 循环语句

for
while

5.1 while
while condition:
    do ...
Can you write Python code for it?

count = 0

while count<10:
    count += 1
    print(count)


I am working on the file "Python教程.ipynb".
The first task is:
6. 函数
def 函数名(参数列表):
    函数体
Can you write Python code for it?

def myadd(a, b):
    result = a + b
    return result

print(myadd(1, 2))


I am working on the file "Python数据科学101.ipynb".
The first task is:
10 PySpark简介
扩展我们的算法：有时我们需要处理大量数据，并且采样已经无效，这个时候可以通过把数据分到多个机器来处理。
Spark是一个用来并行进行大数据处理的API。它将数据切割到集群来处理。在开发阶段，我们可以只在本地运行。
我们使用PySpark Shell来连接到集群。
运行下面路径的pyspark，会启动PySpark Shell
~/spark/bin/pyspark (Max/Linux)
C:\spark\bin\pyspark (Windows)
此时，可以在Shell中运行文件加载：
lines = sc.textFile("README.md")
lines.first() # 加载第一行
可以在http://localhost:4040查看PySpark运行的Job
大多数情况下，我们希望能够在Jupyter Notebook中运行PySpark，为此，我们需要设置环境变量：
PYSPARK_PYTHON=python3
PYSPARK_DRIVER_PYTHON="jupyter"
PYSPARK_DRIVER_PYTHON_OPTS="notebook"
然后运行~/spark/bin/pyspark，最后一个命令会启动一个jupyter server,样子跟我们用的一样。
Can you write Python code for it?

lines = sc.text('README.md')
lines.take(5)


I am working on the file "Python数据科学101.ipynb".
The first task is:
4. Pandas简介
最重要的是DataFrame和Series
Can you write Python code for it?


I am working on the file "Python数据科学101.ipynb".
The first task is:
4.1 Series
创建一个series，包含空值NaN
Can you write Python code for it?


I am working on the file "Python数据科学101.ipynb".
The first task is:
4.4 给行起名(Naming Rows)
Can you write Python code for it?

rain = rain.set_index(rain['City'] + rain['Month'])


I am working on the file "Python数据科学101.ipynb".
The first task is:
6.2 稀疏矩阵(Sparse Matrices)
Can you write Python code for it?

import numpy, scipy.sparse
n = 100000
x = (numpy.random.rand(n) * 2).astype(int).astype(float) #50%稀疏矩阵
x_csr = scipy.sparse.csr_matrix(x)
x_dok = scipy.sparse.dok_matrix(x.reshape(x_csr.shape))
x_dok


I am working on the file "Python数据科学101.ipynb".
The first task is:
7.1 特征向量
Can you write Python code for it?

pred = 'dep_delay'
features = ['month', 'day', 'dep_time', 'arr_time', 'carrier', 'dest', 'air_time',
           'distance', 'lat', 'lon', 'alt', 'dewp', 'humid', 'wind_speed', 'wind_gust',
           'precip', 'pressure', 'visib']
features_v = df[features]
pred_v = df[pred]

pd.options.mode.chained_assignment = None #default='warn'

# 因为航空公司不是一个数字，我们把它转化为数字哑变量
features_v['carrier'] = pd.factorize(features_v['carrier'])[0]

# dest也不是一个数字，我们也把它转为数字
features_v['dest'] = pd.factorize(features_v['dest'])[0]

features_v


I am working on the file "Python数据科学101.ipynb".
The first task is:
7.3 特征降维(Reducing Dimensions)
我们使用PCA(Principle Component Analysis主成分析)把特征降维为2个
Can you write Python code for it?

from sklearn.decomposition import PCA

pca = PCA(n_components=2)
X_r = pca.fit(scaled_features).transform(scaled_features)

X_r


I am working on the file "Python数据科学101.ipynb".
The first task is:
7.4 画图(Plotting)
Can you write Python code for it?

import matplotlib.pyplot as plt

print('explained variance ratio (first two components): %s'
      % str(pca.explained_variance_ratio_))

plt.figure()
lw = 2

plt.scatter(X_r[:,0], X_r[:,1], alpha=.8, lw=lw)
plt.title('PCA of flights dataset')


I am working on the file "Python数据科学101.ipynb".
The first task is:
8 构建分类器(Build a classifier)
我们来预测一个航班是否会晚点
Can you write Python code for it?

%matplotlib inline

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import sklearn
from sklearn import linear_model, cross_validation, metrics, svm, ensemble
from sklearn.metrics import classification_report, confusion_matrix, precision_recall_fscore_support, accuracy_score
from sklearn.cross_validation import train_test_split, cross_val_score, ShuffleSplit
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler, OneHotEncoder

flights = pd.read_csv('data/nycflights13/flights.csv.gz')
weather = pd.read_csv('data/nycflights13/weather.csv.gz')
airports = pd.read_csv('data/nycflights13/airports.csv.gz')

df_withweather = pd.merge(flights, weather, how='left', on=['year', 'month', 'day', 'hour'])
df = pd.merge(df_withweather, airports, how='left', left_on='dest', right_on='faa')

df = df.dropna()

df

pred = 'dep_delay'
features = ['month', 'day', 'dep_time', 'arr_time', 'carrier', 'dest', 'air_time',
           'distance', 'lat', 'lon', 'alt', 'dewp', 'humid', 'wind_speed', 'wind_gust',
           'precip', 'pressure', 'visib']
features_v = df[features]
pred_v = df[pred]

how_late_is_late = 15.0

pd.options.mode.chained_assignment = None #default='warn'

# 因为航空公司不是一个数字，我们把它转化为数字哑变量
features_v['carrier'] = pd.factorize(features_v['carrier'])[0]

# dest也不是一个数字，我们也把它转为数字
features_v['dest'] = pd.factorize(features_v['dest'])[0]

scaler = StandardScaler()
scaled_features_v = scaler.fit_transform(features_v)

features_train, features_test, pred_train, pred_test = train_test_split(
    scaled_features_v, pred_v, test_size=0.30, random_state=0)

# 使用logistic回归来执行分类

clf_lr = sklearn.linear_model.LogisticRegression(penalty='l2',
                                                class_weight='balanced')
logistic_fit = clf_lr.fit(features_train, np.where(pred_train >= how_late_is_late, 1, 0))

predictions = clf_lr.predict(features_test)

# summary Report

# Confusion Matrix
cm_lr = confusion_matrix(np.where(pred_test >= how_late_is_late, 1, 0),
            ...


I am working on the file "Python数据科学101.ipynb".
The first task is:
从上面的图表中我们可以直观的看到一些特征。现在我们再来看看每个航空公司晚点的情况，并进行一些可视化。
Can you write Python code for it?

# 看看是否不同的航空公司对晚点会有不同的影响
df[['carrier', 'arr_delay']].groupby('carrier').mean().plot(kind='bar', figsize=(12, 8))
plt.xticks(rotation=0)
plt.xlabel('Carrier')
plt.ylabel('Average Delay in Min')
plt.title('Average Arrival Delay by Carrier in 2008, All airports')

df[['carrier', 'dep_delay']].groupby('carrier').mean().plot(kind='bar', figsize=(12, 8))
plt.xticks(rotation=0)
plt.xlabel('Carrier')
plt.ylabel('Average Delay in Min')
plt.title('Average Departure Delay by Carrier in 2008, All airports')


I am working on the file "Python数据科学101.ipynb".
The first task is:
我们也可以使用shape方法从一维数组创建多维数组
Can you write Python code for it?


I am working on the file "Python数据科学101.ipynb".
The first task is:
我们看看http://localhost:4040 可以查看运行的Job
Can you write Python code for it?

linesWithSpark = lines.filter(lambda line: 'spark' in line)
linesWithSpark.count()


I am working on the file "Python数据科学101.ipynb".
The first task is:
更大的数据应当从文件里获取
Can you write Python code for it?

rain = pd.read_csv('data/rainfall/rainfall.csv')
rain

# 加载一列
rain['City']

# 加载一行(第二行)
rain.loc[[1]]

# 第一行和第二行
rain.loc[0:1]


I am working on the file "Python数据科学101.ipynb".
The first task is:
查找4月份的降雨
Can you write Python code for it?


I am working on the file "Python数据科学101.ipynb".
The first task is:
注意这里我们传的是列表，而不是np.array(1, 2, 3)。
现在我们创建一个arange
Can you write Python code for it?

np.arange(10)

# 给序列乘以一个系数
np.arange(10) * np.pi


I am working on the file "Python数据科学101.ipynb".
The first task is:
注意，这里9、10月均值会有负值。
Can you write Python code for it?

mean_delay_by_month_ord = df[(df.dest == 'ORD')].groupby(['month'])['arr_delay'].mean()
print("Flights to Chicago (ORD)")
print(mean_delay_by_month_ord)

mean_month_plt_ord = mean_delay_by_month_ord.plot(kind='bar', title="Mean Delay By Month (Chicago)")
mean_month_plt_ord

# 再看看Los Angeles进行比较一下
mean_delay_by_month_lax = df[(df.dest == 'LAX')].groupby(['month'])['arr_delay'].mean()
print("Flights to Chicago (LAX)")
print(mean_delay_by_month_lax)

mean_month_plt_lax = mean_delay_by_month_lax.plot(kind='bar', title="Mean Delay By Month (Los Angeles)")
mean_month_plt_lax


I am working on the file "Pytorch Gradient reversal.ipynb".
The first task is:
Gradient reversal
Here the modifying function $g(x)$ is simply the $-2*f(x)$, this will make the gradients $-f'(x)$.
Can you write Python code for it?

gradient_reversal_op = (tf.stop_gradient(2*f(tf_X)) - f(tf_X))
gradient_reversal_grad_op = tf.gradients(gradient_reversal_op, tf_X)

sess.run(gradient_reversal_op)

sess.run(gradient_reversal_grad_op)

sess.run((gradient_op[0] + gradient_reversal_grad_op[0])) # This should be zero. Signifying grad is reversed.


I am working on the file "Pytorch Gradient reversal.ipynb".
The first task is:
Modify gradients
Can you write Python code for it?

modified_gradients_forward = lambda x: f(x) + g(x) - g(x).detach()

zero_grad(torch_X)
modified_grad = modified_gradients_forward(torch_X)
modified_grad.backward(torch.ones(modified_grad.size()))
torch_X.grad.data.numpy()

2*X + 3*(X*X) # It should be same as above