I am working on the file "Pysegreg/Pysegreg_notebook_time.ipynb".
The first task is:
Input file
Attention to the new data structure for input !!!
Change your input file with path/name in the cell below to be processed.
Data Format
ID | X | Y | group 1 | group 2 | group n
Can you write Python code for it?


I am working on the file "Pysegreg/Pysegreg_notebook_time.ipynb".
The first task is:
Results
Prepare data for saving on a local file
Can you write Python code for it?

# Concatenate local values from measures
if len(cc.locality) == 0:
    results = np.concatenate((expo_local, diss_local, entro_local, idxh_local), axis=1)
else:
    results = np.concatenate((cc.locality, expo_local, diss_local, entro_local, idxh_local), axis=1)

# Concatenate the results with original data
output = np.concatenate((cc.tract_id, cc.attributeMatrix, results),axis = 1)

names = ['id','x','y']

for i in range(cc.n_group):
    names.append('group_'+str(i))

if len(cc.locality) == 0:
    for i in range(cc.n_group):
        for j in range(cc.n_group):
            if i == j:
                names.append('iso_' + str(i) + str(j))
            else:
                names.append('exp_' + str(i) + str(j))

    names.append('dissimil')
    names.append('entropy')
    names.append('indexh')

else:
    for i in range(cc.n_group):
        names.append('intens_'+str(i))

    for i in range(cc.n_group):
        for j in range(cc.n_group):
            if i == j:
                names.append('iso_' + str(i) + str(j))
            else:
                names.append('exp_' + str(i) + str(j))

    names.append('dissimil')
    names.append('entropy')
    names.append('indexh')


I am working on the file "Pysegreg/Pysegreg_notebook_time.ipynb".
The first task is:
Save Local and global results to a file
The paramenter fname corresponds to the folder/filename, change it as you want.
To save on a diferent folder, use the "/" to pass the directory.
The local results will be saved using the name defined and adding the "_local" postfix to file's name.
The global results are automatically saved using the same name with the addiction of the postfix "_globals".
It's recommended to save on a different folder from the code, e.g.: a folder named result.
The fname value should be changed for any new executions or the local file will be overwrited!
Can you write Python code for it?

fname = "/Users/sandrofsousa/Downloads/valid/result"

output = pd.DataFrame(output, columns=names)
output.to_csv("%s_local.csv" % fname, sep=",", index=False)
with open("%s_global.txt" % fname, "w") as f:
    f.write('Global dissimilarity: ' + str(diss_global))
    f.write('\nGlobal entropy: ' + str(entro_global))
    f.write('\nGlobal Index H: ' + str(idxh_global))
    f.write('\nGlobal isolation/exposure: \n')
    f.write(str(expo_global))


I am working on the file "Pyspark_SQL_Magic_Jupyter/IPython_Pyspark_SQL_Magic.ipynb".
The first task is:
Define test tables
Can you write Python code for it?

# Define test data and register it as tables
# This is a classic example of employee and department relational tables
# Test data will be used in the examples later in this notebook

from pyspark.sql import Row

Employee = Row("id", "name", "email", "manager_id", "dep_id")
df_emp = sqlContext.createDataFrame([
        Employee(1234, 'John', 'john@mail.com', 1236, 10),
        Employee(1235, 'Mike', 'mike@mail.com', 1237, 10),
        Employee(1236, 'Pat', 'pat@mail.com', 1237, 20),
        Employee(1237, 'Claire', 'claire@mail.com', None, 20),
        Employee(1238, 'Jim', 'jim@mail.com', 1236, 30)
        ])

df_emp.registerTempTable("employee")

Department = Row("dep_id", "dep_name")
df_dep = sqlContext.createDataFrame([
        Department(10, 'Engineering'),
        Department(20, 'Head Quarter'),
        Department(30, 'Human resources')
        ])

df_dep.registerTempTable("department")


I am working on the file "Pyspark_SQL_Magic_Jupyter/IPython_Pyspark_SQL_Magic.ipynb".
The first task is:
Example of cell magic to run SQL spanning multiple lines
Can you write Python code for it?

%%sql_show
select emp.id, emp.name, emp.email, emp.manager_id, dep.dep_name
from employee emp, department dep
where emp.dep_id=dep.dep_id


I am working on the file "Pyspark_SQL_Magic_Jupyter/IPython_Pyspark_SQL_Magic.ipynb".
The first task is:
Examples of how to use %SQL magic functions with Spark
Use %sql to run SQL and return a DataFrame, lazy evaluation
Can you write Python code for it?

# Example of line magic, a shortcut to run SQL in pyspark
# Pyspark has lazy evaluation, so the query is not executed in this exmaple

df = %sql select * from employee
df


I am working on the file "Pyspark_SQL_Magic_Jupyter/IPython_Pyspark_SQL_Magic.ipynb".
The first task is:
Use %sql_display to run SQL and display the results as a HTML table
Example of cell magic that runs SQL and then transforms it to Pandas. This will display the output as a HTML table in Jupyter notebooks
Can you write Python code for it?

%%sql_display
select emp.id, emp.name, emp.email, emp2.name as manager_name, dep.dep_name
from employee emp
     left outer join employee emp2 on emp2.id=emp.manager_id
     join department dep on emp.dep_id=dep.dep_id


I am working on the file "Pyspark_SQL_Magic_Jupyter/IPython_Pyspark_SQL_Magic.ipynb".
The first task is:
Use %sql_explain to display the execution plan
Can you write Python code for it?

%%sql_explain
select emp.id, emp.name, emp.email, emp2.name as manager_name, dep.dep_name
from employee emp
     left outer join employee emp2 on emp2.id=emp.manager_id
     join department dep on emp.dep_id=dep.dep_id


I am working on the file "Pyspark_SQL_Magic_Jupyter/IPython_Pyspark_SQL_Magic.ipynb".
The first task is:
Use %sql_show to run SQL and show the top lines of the result set
Can you write Python code for it?

# Example of line magic, the SQL is executed and the result is displayed
# the maximum number of displayed lines is configurable (max_show_lines)

%sql_show select * from employee


I am working on the file "Python Challenge.ipynb".
The first task is:
1. What about making trans?
URL: http://www.pythonchallenge.com/pc/def/map.html
Flag: http://www.pythonchallenge.com/pc/def/ocr.html
Python docs
* str.maketrans()
* str.translate()
* printf-style string formating
* Literal string interpolation
* Lists slices (and other operations)
Other resources
* Caesar cipher
Can you write Python code for it?

from string import ascii_lowercase as al

# Build Caesar table using a shift of 2
table = str.maketrans(al, al[2:] + al[:2])

hint = "g fmnc wms bgblr rpylqjyrc gr zw fylb. rfyrq ufyr amknsrcpq ypc dmp. bmgle gr gl zw fylb gq glcddgagclr ylb rfyr'q ufw rfgq rcvr gq qm jmle. sqgle qrpgle.kyicrpylq() gq pcamkkclbcb. lmu ynnjw ml rfc spj."
print("Hint: \"%s\"\n" % hint.translate(table))

cypher_flag = 'map'
flag = cypher_flag.translate(table)
print(f'Flag: \"{flag}\"')


I am working on the file "Python Challenge.ipynb".
The first task is:
3. re
URL: http://www.pythonchallenge.com/pc/def/equality.html
Flag: http://www.pythonchallenge.com/pc/def/linkedlist.html
Can you write Python code for it?

import requests
import re

# Load website and comment
webpage = requests.get('http://www.pythonchallenge.com/pc/def/equality.html').text
cypher_flag = re.findall(r'<!--([^>]*)-->', webpage)[0]

# One letter surrounded by EXACTLY three big bodyguards on each of its side = 3 upper, 1 lower, 3 upper (EXACTLY)
matches = re.findall(r'[a-z][A-Z]{3}[a-z][A-Z]{3}[a-z]', cypher_flag)

''.join([x[4] for x in matches])


I am working on the file "Python Course - 002a - And so we begin.ipynb".
The first task is:
You want to add in .weekday()
Lets you output LaTeX-style (formatted) maths
Example calculating the output of $ \int x^3 dx $:
Can you write Python code for it?

from sympy import *
init_printing()
x = Symbol("x")
integrate(x ** 3, x)


I am working on the file "Python Dictionaries.ipynb".
The first task is:
Dictionary Comprehension
Can you write Python code for it?

{x: x**2 for x in range(10) if x%2 == 0}


I am working on the file "Python Hacks/PythonHacks.ipynb".
The first task is:
Misc
Getting list of files and subdirectories Reference link
Can you write Python code for it?

"""Solution top 1"""
import os
path = "/Users/jesusenriquealdanasigona/Documents/Documents - Jesus’s Mac mini/Github/nanoHUB_roles/nanohub/Literature support/1 Escience"
arr_txt = [filename for filename in os.listdir(path) if (".pdf") in filename]

arr_txt[:5]


I am working on the file "Python Pandas US Census names/Vinit_Nalawade_Project_Pandas.ipynb".
The first task is:
Calculating the last letter frequency for female names.
Can you write Python code for it?

female_last_letter_freq = Counter()
for name in female_names['name']:
    female_last_letter_freq[name[-1]] += 1
female_last_letter_freq


I am working on the file "Python Pandas US Census names/Vinit_Nalawade_Project_Pandas.ipynb".
The first task is:
Calculating the letter frequency for female names.
Can you write Python code for it?

female_letter_freq = Counter()
#converting every letter to lowercase
for name in map(lambda x:x.lower(),female_names['name']):
    for i in name:
        female_letter_freq[i] += 1
female_letter_freq


I am working on the file "Python Pandas US Census names/Vinit_Nalawade_Project_Pandas.ipynb".
The first task is:
Calculating the letter frequency for male names.
Can you write Python code for it?

male_letter_freq = Counter()
#converting every letter to lowercase
for name in map(lambda x:x.lower(),male_names['name']):
    for i in name:
        male_letter_freq[i] += 1
male_letter_freq


I am working on the file "Python Pandas US Census names/Vinit_Nalawade_Project_Pandas.ipynb".
The first task is:
Evaluate how stable this statistic is. Speculate on why it is is stable, if it is, or on what demographic facts might explain any changes, if there are any.
We can normalize the table by total births in each particular decades to compute a new table containing proportion
of total births for each decade ending in each letter.
Can you write Python code for it?

decades_table.sum()

#plot the decades as bars and the female line for all years as a line
c = ['m','g','c']
decades_table_prop = decades_table/decades_table.sum().astype(float)
decades_table_prop['frequency'].plot(kind = 'bar', rot = 0,color = c, title = 'Normalized Frequency of Last letter of Female names by Female Births')
#the female line for all years taken from part 2
#plt.plot(range(len(female_last_letter_freq_asc)), female_last_letter_freq_asc.values(), c = 'r', label = 'All Female births')
plt.xlabel('Letters')
plt.ylabel('Normalized Frequency')
plt.legend(loc = 'best')
#double the size of plot for visibility
size = 2
params = plt.gcf()
plSize = params.get_size_inches()
params.set_size_inches((plSize[0]*size, plSize[1]*size))
plt.show()


I am working on the file "Python Pandas US Census names/Vinit_Nalawade_Project_Pandas.ipynb".
The first task is:
Part One: Go to the Social Security Administration US births website and select the births table there and copy it to your clipboard. Use the pandas read_clipboard function to read the table into Python, and use matplotlib to plot male and female births for the years covered in the data.
Can you write Python code for it?

#informing python that ',' indicates thousands
df = pd.read_clipboard(thousands = ',')

df

#plot male and female births for the years covered in the data
plt.plot(df['Year of birth'], df['Male'], c = 'b', label = 'Male')
plt.plot(df['Year of birth'], df['Female'],c = 'r', label = 'Female')
plt.legend(loc = 'upper left')
#plt.axis([1880, 2015, 0, 2500000])
plt.xlabel('Year of birth')
plt.ylabel('No. of births')
plt.title('Total births by Sex and Year')
#double the size of plot for visibility
size = 2
params = plt.gcf()
plSize = params.get_size_inches()
params.set_size_inches((plSize[0]*size, plSize[1]*size))
plt.show()


I am working on the file "Python Pandas US Census names/Vinit_Nalawade_Project_Pandas.ipynb".
The first task is:
Plot for each letter showing the frequency of that letter as the last letter for both for male and female names.
<br>I use the OrderedDict function from collections here to arrange the letters present in counter in acsending order for plotting.
Can you write Python code for it?

#for ordering items of counter in ascending order
from collections import OrderedDict
#plot of last letter frequency of male names in ascending order of letters
male_last_letter_freq_asc = OrderedDict(sorted(male_last_letter_freq.items()))
plt.bar(range(len(male_last_letter_freq_asc)), male_last_letter_freq_asc.values(), align='center')
plt.xticks(range(len(male_last_letter_freq_asc)), male_last_letter_freq_asc.keys())
plt.xlabel('Letters')
plt.ylabel('Frequency')
plt.title('Frequency of last letter for Male names')
plt.show()

#plot of last letter frequency of female names in ascending order of letters
female_last_letter_freq_asc = OrderedDict(sorted(female_last_letter_freq.items()))
plt.bar(range(len(female_last_letter_freq_asc)), female_last_letter_freq_asc.values(), align='center')
plt.xticks(range(len(female_last_letter_freq_asc)), female_last_letter_freq_asc.keys())
plt.xlabel('Letters')
plt.ylabel('Frequency')
plt.title('Frequency of last letter for Female names')
plt.show()

female_last_letter_freq_asc = OrderedDict(sorted(female_last_letter_freq.items()))
plt.plot(range(len(female_last_letter_freq_asc)), female_last_letter_freq_asc.values(), c = 'r', label = 'Female')
plt.plot(range(len(male_last_letter_freq_asc)), male_last_letter_freq_asc.values(), c = 'b', label = 'Male')
plt.xticks(range(len(male_last_letter_freq_asc)), male_last_letter_freq_asc.keys())
plt.xlabel('Letters')
plt.ylabel('Frequency')
plt.legend(loc = 'upper right')
plt.title('Frequency of last letter in names by Sex')
#double the size of plot for visibility
size = 2
params = plt.gcf()
plSize = params.get_size_inches()
params.set_size_inches((plSize[0]*size, plSize[1]*size))
plt.show()


I am working on the file "Python Pandas US Census names/Vinit_Nalawade_Project_Pandas.ipynb".
The first task is:
Plot of last letter of females for 1880s , 1940s, 1990s, and for all years (from part 2).
Can you write Python code for it?

#plot the decades as bars and the female line for all years as a line
c = ['m','g','c']
decades_table['frequency'].plot(kind = 'bar', rot = 0,color = c, title = 'Frequency of Last letter of Female names by Female Births')
#the female line for all years taken from part 2
plt.plot(range(len(female_last_letter_freq_asc)), female_last_letter_freq_asc.values(), c = 'r', label = 'All Female births')
plt.xlabel('Letters')
plt.ylabel('Frequency')
plt.legend(loc = 'best')
#double the size of plot for visibility
size = 2
params = plt.gcf()
plSize = params.get_size_inches()
params.set_size_inches((plSize[0]*size, plSize[1]*size))
plt.show()


I am working on the file "Python Pandas US Census names/Vinit_Nalawade_Project_Pandas.ipynb".
The first task is:
Preparing data for the 1940s.
<br>A counter for last letter frequencies.
Can you write Python code for it?

female_1940_freq = Counter()
for name in female_1940['name']:
    female_1940_freq[name[-1]] += 1
female_1940_freq


I am working on the file "Python Pandas US Census names/Vinit_Nalawade_Project_Pandas.ipynb".
The first task is:
Preparing data for the 1990s.
<br>A counter for last letter frequencies.
Can you write Python code for it?

female_1990_freq = Counter()
for name in female_1990['name']:
    female_1990_freq[name[-1]] += 1
female_1990_freq


I am working on the file "Python Pandas US Census names/Vinit_Nalawade_Project_Pandas.ipynb".
The first task is:
plot xkcd style :)
with plt.xkcd():
#plt.plot(df['Year of birth'], df['Male'], c = 'b', label = 'Male')
#plt.plot(df['Year of birth'], df['Female'],c = 'r', label = 'Female')
#plt.legend(loc = 'upper left')
#plt.xlim(xmax = 2015)
#plt.xlabel('Year of birth')
#plt.ylabel('No. of births')
#plt.title('Male and Female births from 1880 to 2015')
#plt.show()

In the same notebook, use Python to get a list of male and female names from these files. This data is broken down by year of birth.

The files contain names data of the years from 1881 to 2010.

Aggregating this data in "names" dataframe below.
Can you write Python code for it?

years = range(1881,2011)
pieces = []
columns = ['name','sex','births']

for year in years:
    path = 'names/yob{0:d}.txt'.format(year)
    frame = pd.read_csv(path,names=columns)
    frame['year'] = year
    pieces.append(frame)

names = pd.concat(pieces, ignore_index=True)

names.head()

names.tail()


I am working on the file "Python Programming hands on-1.ipynb".
The first task is:
Prime number (2)
write a function, which takes 1 parameter (n), and returns n-th prime number.
example: n=1 output=2, n=2 output=3, n=3 output=5, n=5 output=11
nth_primt(10)
Can you write Python code for it?

#!/usr/bin/python3
# -*- coding: utf-8 -*-

'''
find N-th prime number
'''

prime = [2, 3]

def is_prime(candidate):
    sqrt_c = int(candidate ** 0.5)

    for p in prime:
        if p > sqrt_c:
            return True
        if candidate % p == 0:
            return False    # Not Prime number
    return True

def nth_prime(n):
    if not isinstance(n, int):
        raise TypeError
    if n < 1:
        raise ValueError
    prime_list_len = len(prime)
    if n <= prime_list_len:
        print('Prime List cache hit', prime_list_len)
        return prime[n-1]

    candidate = prime[-1]
    while prime_list_len < n:
        candidate += 2
        if is_prime(candidate):
             candidate += 2 prime.append(candidate)
            prime_list_len += 1

    return prime[-1]

print(nth_prime(5))
print(nth_prime(100))
print(nth_prime(25))


I am working on the file "Python Programming hands on-1.ipynb".
The first task is:
Programming hands on
習うより慣れろ！(narau yori narero)  more practice, less learning;  practice makes perfect.
a + b
input: each line contains 2 integer
output: print sum of the 2 integer
input: input_ab.txt, output: standard out
```
sample input:
1 1
100 150
123 321
11112222 22223333
sample output:
2
250
444
33335555
```
Can you write Python code for it?


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
2. Simple Data Types
Determine type of an object
Can you write Python code for it?

print(type(2)) #int

type(2.0) #float

type("two") #string

type(True) # bool

type(None) #NoneType


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
4. Comparisons and Boolean Operations
Can you write Python code for it?


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
5. Conditional Statements
Can you write Python code for it?

if x > 0:
    print('positive')
elif x == 0:
    print('zero')
else:
    print('negative')

#single-line if
if x > 0: print('positive')

#single line if/else statement (ternary operator)
'positive' if x > 0 else 'zero or negative'


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
8. With Statement
Can you write Python code for it?

# the With statement (like using in C#) is used to issue cleanup code when a variable comes out of scope
class cleanup_thing:
        def __enter__(self):
            self.open=True
            return self
        def __exit__(self, type, value, traceback):
            self.open=False
        def isOpen(self):
            return self.open

with cleanup_thing() as thing:
    print(thing.isOpen())
print(thing.isOpen())


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
9. DateTime
Can you write Python code for it?

from datetime import datetime
from datetime import timedelta

#get now
datetime.today()

a = datetime(2012, 9, 23)
a

b = timedelta(days=2, hours=6)
a+b

a-b

d=b+timedelta(hours=4.5)
d.days

d.seconds

# total timedelta in seconds
d.total_seconds()


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
Check if an object is of a given type
Can you write Python code for it?

isinstance(2.0, float)

isinstance("two", (float, str))


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
Convert an object to a given type
Can you write Python code for it?


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
Converting to / from strings
Can you write Python code for it?


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
Non-Empty containers and non-zeros are converted to True
Can you write Python code for it?

bool(2)

bool('two')

 bool([2])

bool({'key':'val'})


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
Time Zones
Can you write Python code for it?

from pytz import timezone

d = datetime(2012, 12, 21, 9, 30, 0)

# localize the date for Chicago
central = timezone('US/Central')
loc_d = central.localize(d)
print(loc_d)

# Convert to Bangalore time
bang_d = loc_d.astimezone(timezone('Asia/Kolkata'))
print(bang_d)


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
for loops:
Can you write Python code for it?

# not the recommended style
fruits = ['apple', 'banana', 'cherry']
for i in range(len(fruits)):
    print(fruits[i].upper())

# recommended style
for fruit in fruits:
    print(fruit.upper())

# iterate through two things at once (using tuple unpacking)
family = {'dad':'homer', 'mom':'marge', 'size':6}
for key, value in family.items():
    print( key,value)

# use enumerate if you need to access the index value within the loop
for index,fruit in enumerate(fruits):
    print(index,fruit)


I am working on the file "Python Quick Reference/Base Language.ipynb".
The first task is:
for/else loop:
Can you write Python code for it?

for fruit in fruits:
    if fruit == 'banana':
        print('Found the banana!')
        break    # exit the loop and skip the 'else' block
else:
    # this block executes ONLY if the for loop completes without hitting 'break'
    print("Can't find the banana")


I am working on the file "Python Quick Reference/Classes.ipynb".
The first task is:
2. Member Variables
Can you write Python code for it?

# declare variable

class Car:
    make = 'unknown car type'
    def drive(self):
        print('driving my %s vroom vroom...' % self.make)

miata = Car()
miata.make = 'miata'
miata.drive()


I am working on the file "Python Quick Reference/Collections.ipynb".
The first task is:
2. Heapq
heapq provides O(1) access to the smallest item in the heap.
Can you write Python code for it?

import heapq
nums = [1, 8, 2, 23, 7, -4, 18, 23, 42, 37, 2]

#heapq is created from a list

heap = list(nums)
heapq.heapify(heap)

#now the 1st element is guarenteed to be the smallest
heap

heapq.heappop(heap)

heap

heapq.heappush(heap, -10)

heap


I am working on the file "Python Quick Reference/Collections.ipynb".
The first task is:
3. Counter
Can you write Python code for it?

words = [
'look', 'into', 'my', 'eyes', 'look', 'into', 'my', 'eyes',
'the', 'eyes', 'the', 'eyes', 'the', 'eyes', 'not', 'around', 'the',
'eyes', "don't", 'look', 'around', 'the', 'eyes', 'look', 'into',
'my', 'eyes', "you're", 'under'
]

from collections import Counter
word_counts = Counter(words)  #Works with any hashable items, not just strings!
word_counts.most_common(3)

morewords = ['why','are','you','not','looking','in','my','eyes']
for word in morewords:
    word_counts[word] += 1
word_counts.most_common(3)

evenmorewords = ['seriously','look','into','them','while','i','look','at', 'you']
word_counts.update(evenmorewords)
word_counts.most_common(3)

a = Counter(words)
b = Counter(morewords)
c = Counter(evenmorewords)

# combine counters
d = b + c
d

# subtract counts
e = a-d
e


I am working on the file "Python Quick Reference/Collections.ipynb".
The first task is:
Using maxlen to limit the num of items in a deque
Can you write Python code for it?

dq = deque(maxlen = 3)
for n in range(10):
    dq.append(n)
dq


I am working on the file "Python Quick Reference/Collections.ipynb".
The first task is:
nlargest / nsmallest wraps creation of a heap for one-time access
Can you write Python code for it?

# nlargest and nsmallest wrap a heapq to provide its results
print(heapq.nlargest(3, nums)) # Prints [42, 37, 23]
print(heapq.nsmallest(3, nums)) # Prints [-4, 1, 2]

# providing an alternate sort key to nlargest/nsmallest
portfolio = [
{'name': 'IBM', 'shares': 100, 'price': 91.1},
{'name': 'AAPL', 'shares': 50, 'price': 543.22},
{'name': 'FB', 'shares': 200, 'price': 21.09},
{'name': 'HPQ', 'shares': 35, 'price': 31.75},
{'name': 'YHOO', 'shares': 45, 'price': 16.35},
{'name': 'ACME', 'shares': 75, 'price': 115.65}
]

heapq.nsmallest(3, portfolio, key=lambda s: s['price'])


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
12. reversed
Can you write Python code for it?

#iterates in reverse
a = [1, 2, 3, 4]
for x in reversed(a):
    print(x)

#you can customize the behavior of reversed for your class by implementing __reversed()__ method
class Counter:
    def __init__(self, start):
        self.start = start
    # Forward iterator
    def __iter__(self):
        n = 1
        while n <= self.start:
            yield n
            n += 1
    # Reverse iterator
    def __reversed__(self):
        n = self.start
        while n > 0:
            yield n
            n -= 1

foo = Counter(5)
for x in reversed(foo):
    print(x)


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
13. Generators with State
Can you write Python code for it?

# To expose state available at each step of iteration, use a classs that implements __iter__()
class countingiterator:
    def __init__(self, items):
        self.items=items
    def __iter__(self):
        self.clear_count()
        for item in self.items:
            self.count+=1
            yield item
    def clear_count(self):
        self.count=0

foo = countingiterator(["aaa","bbb","ccc"])

for i in foo:
    print("{}:{}".format(foo.count, i))


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
16. Iterating with Indexes
Can you write Python code for it?

# enumerate returns the iterated item and an index
my_list = ['a', 'b', 'c']
for idx, val in enumerate(my_list):
    print(idx, val)

# pass a starting index to enumerate
for idx, val in enumerate(my_list, 7):
    print(idx, val)


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
17. chain
Can you write Python code for it?

# chain iterates over several sequences, one after the other
# making them look like one long sequence

from itertools import chain
a = [1, 2]
b = ['x', 'y', 'z']
for x in chain(a, b):
    print(x)


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
18. Flatten a Nested Sequence
Can you write Python code for it?

# you want to traverse a sequence with nested sub sequences as one big sequence
from collections import Iterable

def flatten(items, ignore_types=(str, bytes)):
    for x in items:
        if isinstance(x, Iterable) and not isinstance(x, ignore_types): # ignore types treats iterable string/bytes as simple values
            yield from flatten(x)
        else:
            yield x

items = [1, 2, [3, 4, [5, 6], 7], 8]

for x in flatten(items):
    print(x)


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
19. Merging Presorted Iterables
Can you write Python code for it?

import heapq
a = [1, 4, 7]
b = [2, 5, 6]
for c in heapq.merge(a, b):
    print(c)


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
2. Delegating Iterator
Can you write Python code for it?

# if you write a container class, and want to expose an iterator over an internal collection use the __iter()__ method
class Node:
    def __init__(self):
        self._children = [1,2,3]
    def __iter__(self):
        return iter(self._children)

root = Node()
for x in root:
    print(x)


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
4. Filter
filter returns an iterator containing the elements from a sequence for which a condition is True:
Can you write Python code for it?

nums = range(5)
filter(lambda x: x % 2 == 0, nums) # returns [0, 2, 4]

# equivalent list comprehension
[num for num in nums if num % 2 == 0]


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
5. Named Slices
Can you write Python code for it?


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
7. itemgetter
Can you write Python code for it?

from operator import itemgetter

rows = [
{'fname': 'Brian', 'lname': 'Jones', 'uid': 1003},
{'fname': 'David', 'lname': 'Beazley', 'uid': 1002},
{'fname': 'John', 'lname': 'Cleese', 'uid': 1001},
{'fname': 'Big', 'lname': 'Jones', 'uid': 1004}
]

rows_by_fname = sorted(rows, key=itemgetter('fname'))
rows_by_fname

rows_by_uid = sorted(rows, key=itemgetter('uid'))
rows_by_uid

# itemgetter() function can also accept multiple keys
rows_by_lfname = sorted(rows, key=itemgetter('lname','fname'))
rows_by_lfname


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
9. groupby
The groupby() function works by scanning a sequence and finding sequential “runs”
of identical values (or values returned by the given key function). On each iteration, it
returns the value along with an iterator that produces all of the items in a group with
the same value.
Can you write Python code for it?

from operator import itemgetter
from itertools import groupby

rows = [
{'address': '5412 N CLARK', 'date': '07/01/2012'},
{'address': '5148 N CLARK', 'date': '07/04/2012'},
{'address': '5800 E 58TH', 'date': '07/02/2012'},
{'address': '2122 N CLARK', 'date': '07/03/2012'},
{'address': '5645 N RAVENSWOOD', 'date': '07/02/2012'},
{'address': '1060 W ADDISON', 'date': '07/02/2012'},
{'address': '4801 N BROADWAY', 'date': '07/01/2012'},
{'address': '1039 W GRANVILLE', 'date': '07/04/2012'},
]

# important!  must sort data on key field first!
rows.sort(key=itemgetter('date'))

#iterate in groups
for date, items in groupby(rows, key=itemgetter('date')):
    print(date)
    for i in items:
        print('   %s' % i)


I am working on the file "Python Quick Reference/Data Algorithms.ipynb".
The first task is:
zip can only be iterated over once!
Can you write Python code for it?

prices_and_names = zip(prices.values(), prices.keys())
print(min(prices_and_names))

# running the following code would fail
#print(min(prices_and_names))

# zip usually stops when any individual iterator ends (it iterates only until the end of the shortest sequence)

a = [1, 2, 3]
b = ['w', 'x', 'y', 'z']
for i in zip(a,b):
    print(i)

# use zip_longest to keep iterating through longer sequences
from itertools import zip_longest
for i in zip_longest(a,b):
    print(i)

# zip can run over more then 2 sequences

c = ['aaa', 'bbb', 'ccc']
for i in zip(a,b,c):
    print(i)


I am working on the file "Python Quick Reference/Dictionaries.ipynb".
The first task is:
2. Getting Data
Can you write Python code for it?

# pass a key ti return its value
family['dad']

# return number of key-value pairs
len(family)

# check if key exists in dictionary
'mom' in family

# only keys, not values, are checked
'marge' in family

# returns an iterable view of keys
family.keys()

# returns an iterable view of values
family.values()

# returns an iterable view of key-value pairs
family.items()


I am working on the file "Python Quick Reference/Dictionaries.ipynb".
The first task is:
3. Safer Access With Get
Can you write Python code for it?

# equivalent to a dictionary lookup
family.get('mom')

# this would throw an error since the key does not exist
# family['grandma']

# return None if not found
family.get('grandma')

# provide a default return value if not found
family.get('grandma', 'not found')

# accesss an element within a list inside a dictionary
family['kids'][0]


I am working on the file "Python Quick Reference/Dictionaries.ipynb".
The first task is:
4. Adding and Changing Data
Can you write Python code for it?

# edit an existing entry
family['cat'] = 'snowball'
family

# edit an existing entry
family['cat'] = 'snowball ii'
family

# delete an entry
del family['cat']
family

# dictionary value can be a list
family['kids']=['bart','lisa']
family

# remove an entry and return the value
family.pop('dad')

# add multiple-entries
family.update({'baby':'maggie', 'mom':'Marge', 'grandpa':'abe'})
family

family['kids'].remove('lisa')
family


I am working on the file "Python Quick Reference/Dictionaries.ipynb".
The first task is:
5. String Substitution With a  Dictionary
Can you write Python code for it?

# replaces %(baby)s with the result of looking up 'baby' in family
'youngest child is %(baby)s' % family


I am working on the file "Python Quick Reference/Dictionaries.ipynb".
The first task is:
6. Default Elements
Ways to get a default element add to the dictionary on first access
Can you write Python code for it?

# using defaultdict
from collections import defaultdict

d=defaultdict(list)
d['a'].append(1)
d['a'].append(2)
d['b'].append(3)
d

# using setdefault
d={}
d.setdefault('a',[]).append(1)
d.setdefault('a',[]).append(2)
d.setdefault('b',[]).append(3)
d


I am working on the file "Python Quick Reference/Dictionaries.ipynb".
The first task is:
7. Set Operations On Keys and Items
Can you write Python code for it?

a = {
'x' : 1,
'y' : 2,
'z' : 3
}

b = {
'w' : 10,
'x' : 11,
'y' : 2
}

# Find keys in common
a.keys() & b.keys() # { 'x', 'y' }

# Find keys in a that are not in b
a.keys() - b.keys() # { 'z' }

# Find (key,value) pairs in common
a.items() & b.items() # { ('y', 2) }


I am working on the file "Python Quick Reference/Dictionaries.ipynb".
The first task is:
8. Dictionary Comprehensions
Can you write Python code for it?

prices = {
'ACME': 45.23,
'AAPL': 612.78,
'IBM': 205.55,
'HPQ': 37.20,
'FB': 10.75
}

# Make a dictionary of all prices over 200
{ key:value for key, value in prices.items() if value > 200 }

# Make a dictionary of tech stocks
tech_names = { 'AAPL', 'IBM', 'HPQ', 'MSFT' }
{ key:value for key,value in prices.items() if key in tech_names }


I am working on the file "Python Quick Reference/Dictionaries.ipynb".
The first task is:
These kinds of operations can also be used to alter or filter dictionary contents. For
example, suppose you want to make a new dictionary with selected keys removed. Here
is some sample code using a dictionary comprehension:
Can you write Python code for it?

{key:a[key] for key in a.keys() - {'z', 'w'}}


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
10. Nonlocal and global
 nonlocal allows you to modify a variable outside your scope (but not global scope)
Can you write Python code for it?

def outside():
    msg = "Outside!"
    def inside():
        msg = "Inside!"
        print(msg)
    inside()
    print(msg) # this prints 'Outside!' even though Inside() mosifies a variable called msg (its a local copy)

outside()

# to have a variable refer to something outside local scope use nonlocal
def outside():
    msg = "Outside!"
    def inside():
        nonlocal msg
        msg = "Inside!"
        print(msg)
    inside()
    print(msg)

outside()

# the global keyword makes a variable reference a global variable rather then a copy
msg = 'Global!!'

def outside():
    msg = "Outside!"
    def inside():
        global msg
        msg = "Inside!"
        print(msg)
    inside()
    print(msg) # this prints 'Outside!' because the copy in Inside() references the global variable

outside()
msg


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
2. Return Values
Can you write Python code for it?

def say_hello():
    return 'hello'

say_hello()


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
3. Parameters
Can you write Python code for it?

def print_this(x):
    print (x)

print_this(3)


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
4. DocStrings
Can you write Python code for it?

def calc(a, b, op='add'):
    """
    calculates the result of a simple math operation.

    :param a: the first parameter in the math operation
    :param b: the first parameter in the math operation
    :param op: which type of math operation (valid values are 'add', 'sub')
    :returns: this result of applying the math argument to the two parameters
    :raises keyError: raises an exception
    """
    if op == 'add':
        return a+b
    elif op == 'sub':
        return a-b
    else:
        print('valid operations are add and sub')

help(calc)


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
5. Unpacking Parameters
Unpacking iterables into positional function arguments (star operator)
Can you write Python code for it?

# range takes start and stop parameters
list(range(3, 6))            # normal call with separate arguments
[3, 4, 5]

# can also pass start and stop arguments using argument unpacking
args = [3,6]  #can be used on tuple too
list(range(*args))


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
6. Generator Functions
Generator functions with yield
Can you write Python code for it?

def myrange(n):
    for i in range(n):
        yield i

max(myrange(5))


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
7. Lambas Anonymous Functions
Can you write Python code for it?

squared = lambda x: x**2

squared(3)

simpsons = ['bart', 'maggie', 'homer', 'lisa', 'marge']
sorted(simpsons, key = lambda word: word[-1])

# no parameter lambda
say_hello = lambda : 'hello'

say_hello()


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
8. Partial
 Partial allows you convert an n-parameter function into a function with less arguments
Can you write Python code for it?


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
9. Closures Nested Functions
Can you write Python code for it?

# this inner closure is used to carry state around
def name_func_from_family(last_name):
    def print_name(first_name):
        print('{} {}'.format(first_name, last_name))
    return print_name #the key here is that the outer function RETURNS the inner function / closure

print_saltwell = name_func_from_family('saltwell')
print_saltwell('erik')
print_saltwell('kasia')
print_saltwell('jacob')


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
Default Parameters in Lambdas
Can you write Python code for it?

talkback = lambda message='hello' : message

talkback()

talkback('hello world')


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
Default values should always be const values, or you can get in trouble
Can you write Python code for it?

def spam(a, b=[]): # b escapes the function as a return variable, which can be altered!
    return b

x = spam(1)
x

x = spam(1)
x.append(99)
x.append('Yow!')

spam(1) # Modified list gets returned!


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
Define a function that only accepts keyword arguments
Can you write Python code for it?

def recv(maxsize, *, block):
    'Receives a message'
    pass

recv(1024, block=True) # Ok

# the following will fail if uncommented
#recv(1024, True) # TypeError


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
Define a function with a default value:
Can you write Python code for it?

def calc(a, b, op='add'):
    if op == 'add':
        return a+b
    elif op == 'sub':
        return a-b
    else:
        print('valid operations are add and sub')

calc(10, 4)

calc(10,4, op='add')

# unnamed arguments are inferred by position
calc(10, 4, 'add')

x = 42
def spam(a, b=x):
    print(a, b)

spam(1)

x = 23 # Has no effect
spam(1)


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
Generator function that uses an internal iterator
Can you write Python code for it?

#using yield from

def myrange(n):
    yield from range(n)

print(max(myrange(5)))


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
Return two values from a single function:
Can you write Python code for it?

def min_max(nums):
    return min(nums), max(nums)

# return values can be assigned into multiple variables using tuple unpacking
nums = [3, 6, 5, 8, 2, 19, 7]

min_num, max_num = min_max(nums)
print(min_num)
print(max_num)


I am working on the file "Python Quick Reference/Functions.ipynb".
The first task is:
Unpacking dictionaries into named arguments (double-star operator)
Can you write Python code for it?

# a dictionary can be unpacked into names arguments with the ** / double star operator

from collections import namedtuple

Stock = namedtuple('Stock', ['name', 'shares', 'price', 'date', 'time'])

# Create a prototype instance
stock_prototype = Stock('', 0, 0.0, None, None)

# Function to convert a dictionary to a Stock
def dict_to_stock(s):
    return stock_prototype._replace(**s)

a = {'name': 'ACME', 'shares': 100, 'price': 123.45}
dict_to_stock(a)


I am working on the file "Python Quick Reference/Lists.ipynb".
The first task is:
2. Accessing Data
Can you write Python code for it?

simpsons[0]

len(simpsons)

# counts the number of instances
simpsons.count('bart')

# returns index of the first reference
simpsons.index('marge')


I am working on the file "Python Quick Reference/Lists.ipynb".
The first task is:
3. Modifying
Can you write Python code for it?

#append to end
simpsons.append('lisa')
simpsons

#append multiple elments to end
simpsons.extend(['itchy','scratchy'])
simpsons

#insert at index (shifts all elements to the right)
simpsons.insert(0, 'maggie')
simpsons

#remove element 0 and return it
simpsons.pop(0)

#remove element at 0 (does not return it)
del simpsons[0]
simpsons

#search for first instance and remove it
simpsons.remove('bart')
simpsons

#replace elmeent
simpsons[0]='krusty'
simpsons

#conctonate lists (slower then 'extend' method)
neighbors = simpsons + ['ned','rod','todd']
neighbors


I am working on the file "Python Quick Reference/Lists.ipynb".
The first task is:
5. Deep Copy
Can you write Python code for it?

new_num = num[:]
new_num=list(num)
new_num


I am working on the file "Python Quick Reference/Lists.ipynb".
The first task is:
6. Equality and Identity
Can you write Python code for it?

# Identity compare: are they referencing the same object?
num is new_num

# Equality compare: do they have the same data?
num == new_num


I am working on the file "Python Quick Reference/Lists.ipynb".
The first task is:
Insert into an already sorted list, and keep it sorted:
Can you write Python code for it?

num = [10, 20, 40, 50]
from bisect import insort
insort(num, 30)
num


I am working on the file "Python Quick Reference/Lists.ipynb".
The first task is:
Return a sorted list (does not modify the original list):
Can you write Python code for it?

sorted(simpsons)

sorted(simpsons, reverse=True, key=len)


I am working on the file "Python Quick Reference/Lists.ipynb".
The first task is:
Setwise list manipulations:
Can you write Python code for it?


I am working on the file "Python Quick Reference/Metaprogramming.ipynb".
The first task is:
2. Advanced Decorators
Can you write Python code for it?

# Decorator that takes arguments
def logged(prefix):
    prefix = prefix if prefix else ''
    logprefix = prefix + ':' if len(prefix) > 0 else prefix
    def decorate(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            print('{}Enter:{}'.format(logprefix, func.__name__))
            result = func(*args, **kwargs)
            print('{}Exit:{}'.format(logprefix, func.__name__))
            return result
        return wrapper
    return decorate

@logged('logger')
def hello_world():
    print('hello world')

hello_world()

# always declare classmethod and staticmethod decorators first (so they are applied last)

class Spam:
    @logthis
    def instance_method(self, n):
        print(self, n)
        while n > 0:
            n -= 1

    @classmethod
    @logthis
    def class_method(cls, n):
        print(cls, n)
        while n > 0:
            n -= 1

    @staticmethod
    @logthis
    def static_method(n):
        print(n)
        while n > 0:
            n-=1

s = Spam()
s.instance_method(4)

Spam.class_method(10)


I am working on the file "Python Quick Reference/Numbers.ipynb".
The first task is:
2. Decimal
Use decimal for floating points that handle high precision base-10 numbers better
Can you write Python code for it?

# example of the problem
a = 4.2
b = 2.1
a + b

(a + b) == 6.3

# with Decimal
from decimal import Decimal

# Decimal is constructed with strings!
a = Decimal('4.2')
b = Decimal('2.1')
a + b

(a + b) == Decimal('6.3')

# to change precision use a scoped local context
from decimal import localcontext

a = Decimal('1.3')
b = Decimal('1.7')
print(a / b) #0.7647058823529411764705882353

with localcontext() as ctx:
    ctx.prec = 3
    print(a / b)


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
2. Get Parts Of A String
Can you write Python code for it?


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
3. isdigit
Can you write Python code for it?

# checks if every character in the string is a digit
s.isdigit()


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
4. Common Modifications
 These functions return a new string.  The original is unmodified.
Can you write Python code for it?

s.lower() # also upper()

# replaces all instances of 'like' with 'love'
s.replace('like', 'love')

# remove leading and trailing whitespace
s5 = '    ham and cheese '
s5.strip()

s5.lstrip()

s5.rstrip()


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
5. Splitting and Joining
Split a string:
Can you write Python code for it?

# split a string into a list of substrings seperated by a delimiter
s.split(' ')

# default delimiter is a space
s.split()


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
7. Raw Strings
Can you write Python code for it?

# normal strings allow for escaped characters
print('first line\nsecond line')

# raw strings treat backslashes as literal characters
print(r'first line\nsecond line')


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
8. Regular Expressions
Can you write Python code for it?

import re

text1 = '11/27/2012'
text2 = 'Nov 27, 2012'

# use a raw string to define reg exps
m = re.match(r'\d+/\d+/\d+', text1)
if m:
    print(m.group())  # returns the entire matched string
else:
    print('no match')


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
9. Handling XML Entities
Converting to entities
Can you write Python code for it?

s = 'Elements are written as "<tag>text</tag>".'

import html
html.escape(s)

# disable escaping of quotes

html.escape(s, quote=False)


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
Capture groups
Can you write Python code for it?

datepat = re.compile(r'(\d+)/(\d+)/(\d+)') # parantheses define capture groups.
m = datepat.match(text1)
m.group()

m.group(0)

m.group(1)

m.group(2)

m.group(3)

m.groups()  # returns a tuple


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
Formatting parameters (more examples):
Can you write Python code for it?

# use 2 decimal places
'pi is {:.2f}'.format(3.14159)

#adding thousands seperator to format
'average hieght is {:,.2f} mm'.format(45678789.7653)


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
If you are going to reuse the expression a bunch, precompile it
Can you write Python code for it?

datepat = re.compile(r'\d+/\d+/\d+')
if(datepat.match(text1)):
    print('yes')
else:
    print('no')


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
Using regular expressions to split with multiple delimiters and whitespace handling.
See Regular Expressions for more on regular expressions.
Can you write Python code for it?

# splitting with a regex allows using multiple delimiters and whitespace handling
import re
line = 'asdf fjdk; afed, fjek,asdf, foo'
re.split(r'[;,\s]\s*', line)

# don't use capturing groups (parenthesis) or the delimiters are captured as fields in the split string
re.split(r'(;|,|\s)\s*', line)

# instead use non-capturing groups / parenthesis
re.split(r'(?:,|;|\s)\s*', line)  # note the '(?:' starting the group, which denotes a non-capturing group


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
fmatch mathes strings using ok-like wildcards
Can you write Python code for it?

# use this when you just want to add a simple wildcard

from fnmatch import fnmatch, fnmatchcase
#fnmatch uses the case sensitivity of the OS, fnmatchcase lets you specify case sensitivity

names = ['Dat1.csv', 'Dat2.csv', 'config.ini', 'foo.py']
[name for name in names if fnmatch(name, 'Dat*.csv')]


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
text justification
Can you write Python code for it?

# useful when outputing fixed-width text records
text = 'Hello World'
text.ljust(20)

text.rjust(20)

text.center(20)

# fill with a specific char
text.center(20,'*')


I am working on the file "Python Quick Reference/Strings.ipynb".
The first task is:
using format for text justification
Can you write Python code for it?

format(text, '>20')

format(text, '<20')

format(text, '^20')

# fill with a specific character
format(text, '*^20s')

#format works with non strings
x = 1.2345
format(x, '*^10')

# format is useful when formatting multiple values or non-strings
'{:>10s}{:>10s}{:*>10f}'.format('Hello', 'World', 1.23)


I am working on the file "Python Quick Reference/Tuples.ipynb".
The first task is:
2. Basic Operations
Can you write Python code for it?

# elements of a tuple cannot be modified (this would throw an error)
# digits[2] = 2

#concatenate tuples
digits = digits + (3,4)
digits

# Create a single tuple with elements repeated (also workss with lists)
(3, 4) * 2

# sort a list of tuples
tens = [(20,60), (10,40), (20,30)]
sorted(tens) #sorts by first element in tuple, then second element


I am working on the file "Python Quick Reference/Tuples.ipynb".
The first task is:
3. Unpacking
Can you write Python code for it?

bart = ('male', 10, 'simpson') # create a tuple
(sex, age, surname) = bart
print(sex)
print(age)
print(surname)

#use the star expression to load multiple values into a list
record = ('Dave', 'dave@example.com', '773-555-1212', '847-555-1212')
name, email, *phone_numbers = record
phone_numbers

# star expressions can be used in the middle of the iterable too!
def drop_first_last(grades):
    first, *middle, last = grades
    return float(sum(middle)) / len(middle)

drop_first_last([3, 8, 8, 8, 8, 8,8, 100])


I am working on the file "Python Quick Reference/Tuples.ipynb".
The first task is:
4. Comparing
Can you write Python code for it?

# Tuples are compared by comparing their members in order
(3,9) < (5,3)

(3,'banana') < (3,'apple')

(3,'banana') == (3,'ban')

# Tuple members are only compared if needed.
# Here is an uncomparable type
class mytype:
    def go():
        pass

a = mytype()
b = mytype()

(3, a) < (5,b)

# but the folllowing code would generate an error because a and b would need to be evaluated

#(3,a) < (3,b)


I am working on the file "Python Quick Reference/Tuples.ipynb".
The first task is:
5. Named Tuples
collections.namedtuple() is a factory method that returns a subclass of the standard Python tuple type. You feed it a type name, and the fields it should have, and it returns a class that you can instantiate, passing in values for the fields you’ve defined, and so on.
Can you write Python code for it?

from collections import namedtuple
Subscriber = namedtuple('Subscriber', ['addr','joined']) # namedtuple(<tuple-name>, [field1, field2, field3...])
jonesy = Subscriber('jonesy@example.com','2012-10-19')
jonesy

len(jonesy)

jonesy.addr

# because a namedtuple is a tuple it is immutable.
# the following code would fail if uncommented

#jonesy.addr='another@example.com'

#instead, use replace()
jonesy = jonesy._replace(addr='another@example.com')
jonesy


I am working on the file "Python Quick Reference/Tuples.ipynb".
The first task is:
A subtle use of the _replace() method is that it can be a convenient way to populate named tuples that have optional or missing fields. To do this, you make a prototype tuple containing the default values and then use _replace() to create new instances with values replaced.
Can you write Python code for it?

Stock = namedtuple('Stock', ['name', 'shares', 'price', 'date', 'time'])

# Create a prototype instance
stock_prototype = Stock('', 0, 0.0, None, None)

# Function to convert a dictionary to a Stock
def dict_to_stock(s):
    return stock_prototype._replace(**s)

a = {'name': 'ACME', 'shares': 100, 'price': 123.45}
dict_to_stock(a)

b = {'name': 'ACME', 'shares': 100, 'price': 123.45, 'date': '12/17/2012'}
dict_to_stock(b)


I am working on the file "Python Workshop/Containers.ipynb".
The first task is:
As we can see, the key returns us the value. This can be useful if you have a bunch of items that need to be paired together.
Accessing Just the Keys or Values
Want to get a list of the keys in a dictionary? How about the values? Fret not, there is a way!
Can you write Python code for it?


I am working on the file "Python Workshop/Containers.ipynb".
The first task is:
Converting Between Containers
But what if I want my set to be a list or my tuple to be a set? To convert between types of containers, you can use any of the following functions:
list() : Converts any container type into a list, for dictionaries it will be the list of keys.
tuple() : Converts any container type into a tuple, for dictionaries it will be the tuple of keys.
set() : Converts any container type into a set, for dictionaries it will be the set of keys. Note that as above, this will remove all duplicates.
Can you write Python code for it?

a = [1,2,3]
b = (1,2,3)
c = {1,2,3}
d = {1:2,3:4}

print(list(b))
print(list(c))
print(list(d))

print(tuple(a))
print(tuple(c))
print(tuple(d))

print(set(a))
print(set(b))
print(set(d))


I am working on the file "Python Workshop/Containers.ipynb".
The first task is:
Dictionaries
Dictionaries are quite different from any container we have seen so far. A dictionary is a bunch of unordered key/value pairs. That is, each element of a dictionary has a key and a value and the elements are in no particular order. It is good to keep this unorderedness in mind later on, for now, let's look at some examples. To create a dictionary we use the following syntax, { key:value}.
Can you write Python code for it?

#Let's say we have some students take a test and we want to store their scores
scores = {'Sally':89, 'Lucy':75, 'Jeff':45, 'Jose':96}
print(scores)
#We can, however, not combine two different dictionaries
scores2 = {'Devin':64, 'John':23, 'Jake':75}
print(scores2)
print(scores+scores2)


I am working on the file "Python Workshop/Containers.ipynb".
The first task is:
Exercises

Build a dictionary of some constants in physics and math. Print out at least two of these values.
Give an example of something that would be best represented by a dictionary, think pairs.

In and Not In
No, this section isn't about fashion. It is about the in and not in operators. They return a boolean value based on whether or not a value is in or not in a container. Note that for dictionaries, this refers to the keys, no the values.
Can you write Python code for it?

print('Devin' in scores2)
print(2 in a)
print('Hello World' not in scores)


I am working on the file "Python Workshop/Containers.ipynb".
The first task is:
Slicing
At one point, you will want to access certain elements of a list, this is done by slicing. There are a couple of ways to do this.
1. la[0] will give the first element of the list la. Note that lists are indexed starting at zero, not one.
2. la[n] will give the (n+1)th element of the list la.
3. la[-1] will give the last element of the list la.
4. la[-n] will give the nth to last element of the list la.
5. la[p:q:k] will give you every kth element starting at p and ending at q of the list la. Ex/la[1:7:2] will give you every second element starting at 1 and ending at 7.
5b. If p is omitted, it is assumed to be 0. If k is omitted, it is assumed to be 1, if q is omitted, it is assumed to be the last index or -1, which is really the same thing as we see above.
Can you write Python code for it?


I am working on the file "Python Workshop/Functions.ipynb".
The first task is:
So clearly we can do some powerful things. Now let's see why these functions have significant power over loops.
Can you write Python code for it?

def fib(n):
 a,b = 1,1
 for i in range(n-1):
  a,b = b,a+b
 return a

def printfib(n):
    for i in range(0,n):
        print(fib(i))

printfib(15)


I am working on the file "Python Workshop/IPython Widgets.ipynb".
The first task is:
Okay! So now you are ready to analyze the world! Just kidding. Let's make a simpler example. Consider the best fitting straight line through a set of points. When a curve fitter fits a straight line, it tries to minimize the sum of the "errors" from all the data points and the fit line. Mathematically this is represented as
$$\sum_{i=0}^{n}(f(x_i)-y_i)^2$$
Now, $f(x_i)=mx_i+b$. Your task is to write a function that plots a line and prints out the error, make an interact that allows you to vary the m and b parameters, then vary those parameters until you find the smallest error.
Can you write Python code for it?

#Make a function that takes two parameters m and b and prints the total error and plots the the line and the data.
#Use this x and y into your function to use as the data
x=np.linspace(0,1,10)
y=(np.random.rand(10)+4)*x+5

#Make an interact as above that allows you to vary m and b.


I am working on the file "Python Workshop/Logic.ipynb".
The first task is:
Again, a neat little example. The true power of for loops comes when we have lists that are not numerical. Let's make every string in a list uppercase.
Can you write Python code for it?

words = ['i', 'am', 'sorry', 'dave', 'i', 'can\'t', 'do', 'that']
upperwords = []
for word in words: #remember that word will take on the value of every element of words
    print(word)
    upperwords.append(word.upper()) # to make a string uppercase, you can use the .upper() function.
print(upperwords)


I am working on the file "Python Workshop/Logic.ipynb".
The first task is:
Elif
Let's say you want to check a different condition before just saying, "The first condition was false, let's do the else statement." We could just use a second if statement, but instead we have the else-if statement, elif. It allows us to check a second condition after the first one fails. Let us concrete this idea with an example.
Can you write Python code for it?

#I love food, let's take a look in my fridge
fridge = ['bananas', 'apples', 'water', 'tortillas', 'cheese']
#I want some pizza, but if I don't have any I will settle for a quesadilla which requires tortillas and cheese
if('pizza' in fridge):
    print('Patrick ate pizza and was happy')
elif('tortillas' in fridge and 'cheese' in fridge):
    print('Patrick didn\'t get his pizza, but he did get a quesadilla and is still happy!')
else:
    print('Patrick is still hungry')


I am working on the file "Python Workshop/Logic.ipynb".
The first task is:
Exercises

Write some "dummy" if, if else, and if elif else statements that will print out exactly what you expect until you feel comfortable with them.

What will be the output of the following code sample:
if(2<4):
    if(len([1,2,3])<=len(set([1,1,1,2,2,3,3,3]))):
        print("This will certainly print")
    elif(2>1):
        print("Or will this print?")
    else:
        print("It's gotta be this one...")
else:
    print("This won't print...or will it.")

Loops
"I feel like I'm doing this over and over again" -Your computer on loops. Wanna do something 10, 100, n times? Loops are your best friend! Want to loop through a list containing all of your data? Loops are your bestest friend! We will look at two different types of loops, while loops and for loops.
While Loops
While loops will continue to loop until some condition is false. While loops follow the format:
while (some condition):
    #some code here

While loops can go on forever if the condition is never false. This is not the end of the world and you won't crash your computer. To stop a cell that is running, you can click on the stop button in the Jupyter toolbar. Let's see what we can do with this.
Can you write Python code for it?

t=15
while(t>0):
    print("t-minus " + str(t))
    t-=1


I am working on the file "Python Workshop/Logic.ipynb".
The first task is:
My god, it worked! Think of the possibilities! With these new tools we can do 90% of all programming we will ever do. Pretty neat huh. I would like to show you one more example of list comprehensions.
Can you write Python code for it?

# I only want words with length less than 3
newwords = [word for word in words if len(word)<3]
print(newwords)


I am working on the file "Python Workshop/Logic.ipynb".
The first task is:
Notice that, although I had the fixings for a quesadilla in my fridge, I had pizza so I never needed to check for a tortilla and cheese. This illustrates the fact that elif wont run unless the if statements before it fails. Further, you can stack elif statements forever. Let's see that.
Can you write Python code for it?

#I love food, let's take a look in my fridge
fridge = ['bananas', 'apples', 'water', 'tortillas', 'beer']
#I want some pizza, but if I don't have any I will settle for a quesadilla which requires tortillas and cheese
if('pizza' in fridge):
    print('Patrick ate pizza and was happy')
elif('tortillas' in fridge and 'cheese' in fridge):
    print('Patrick didn\'t get his pizza, but he did get a quesadilla and is still happy!')
elif('beer' in fridge):
    print('Patrick is still hungry, but he has beer so he is happy')
else:
    print('Patrick is still hungry')


I am working on the file "Python Workshop/Logic.ipynb".
The first task is:
We have one more special type of loop to cover. List comprehensions; a quick way to make a list in one line.
List Comprehensions
A list comprehension is essentially a for loop sandwiched into a list. The syntax for a list comprehension is as follows:
X = [(expression involving special variable) for (special variable) in (some list)]

For example, we want a list containing x^2 for x in [0,1,2,3,4,5,6,7,8,9,10], we can create this list by using:
Y = [x**2 for x in range(0,11)]

Does this actually work?
Can you write Python code for it?

y = [x**2 for x in range(0,11)]
print(y)


I am working on the file "Python Workshop/Logic.ipynb".
The first task is:
What about something wierder?
Can you write Python code for it?

print(words)
wordslength = [len(word) for word in words]
print(wordslength)


I am working on the file "Python Workshop/Logic.ipynb".
The first task is:
While loops are really good if you want to do something over and over again. Let's generate some fake data with this. I introduce here the range() function. This generates a list of numbers. Let's see briefly how it works.
Can you write Python code for it?

# Let's make a list of numbers starting at zero and going to 99. range() by default uses a step size of 1
#so this will yield integers from 0 to 99
x = range(0,100)
print(x)
# Unfortunately range does some strange things and doesn't return a list, if you want a list, you already know how to convert it.
print(list(x))

y = []
x = 1
while(x<100):
    y.append(x**5-27*x**2-2300*x**-1+x%(x+1))
    x+=1
print(y)


I am working on the file "Python Workshop/MatPlotLib.ipynb".
The first task is:
Aha! According to the plot above, my Grand Unified Theory is complete. When your advisor looks at the graph, they might pass out due to its poor quality. This graph needs x labels, y labels, a title and maybe even a legend! Doing those things is astoundingly easy! Let's check it out.
Can you write Python code for it?

plt.plot(x,y, label="Some Physics or Something")
plt.xlabel("I dun measured this!")
plt.ylabel("I dun calculated this!")
plt.title("I dun made this plot!")
plt.legend()


I am working on the file "Python Workshop/NumPy.ipynb".
The first task is:
A list of all the universal functions is included at the end of this notebook.
Exercises

Create a couple of arrays of various type and size and play with them until you feel comfortable moving on.

You know that a certain quantity can be calculated using the following formula:
f(x)=x^e^sin(x^2)-sin(x*ln(x))
Given that you measured x in the cell below, calculate f(x)

Using the same x as above, write a function to transform on x. Then create a mask that will keep any value of the reuslting array whose value is greater than 2*$\pi^2$
Can you write Python code for it?


I am working on the file "Python Workshop/NumPy.ipynb".
The first task is:
Let's say that we have measured some quantity with a computer and generated a really long numpy array, like, really long. It just so happens that we are interested in how many of these numbers are greater than zero. We could try to make a mask with the methods used above, but the people who made masks gave us a tool to do it without ever making a mask.(That's what it does behind the scenes, but we'll just ignore that) I have the data in the next cell, let's check it out.
Can you write Python code for it?

data = np.random.normal(0,3,10000) #Wow, I made 10,000 measurements, wouldn't mastoridis be proud.

data[data>0].size #This returns only the elements of data that are greater than 0


I am working on the file "Python Workshop/NumPy.ipynb".
The first task is:
Some Other Interesting NumPy Functions

np.dot(array1, array2) #Computes the dot product of array1 and array2.
np.cross(array1, array2) #Computes the cross product of array1 and array2.
np.eye(size) #Creates a size by size identity matrix/array

NumPy Array Slicing
NumPy slicing is only slightly different than list slicing. With NumPy arrays, to access a single element you perform the typical
array = np.array([1,2,3,4])

array[index]
array[0] #Returns 1

When you want to take a range of elements, you use the following syntax
array[start:stop+1]
array[1:3] #Returns [2,3]

Or if you would like to move in steps.
array[start:stop+1:step]
array[0:4:2] #Returns [1,3]
Can you write Python code for it?

array = np.array([1,2,3,4])
print(array[0])
print(array[1:4])
print(array[0:4:2])


I am working on the file "Python Workshop/Variables and Operators.ipynb".
The first task is:
Note that our a and b didn't go away. Variables will stick around until you change them.
Can you write Python code for it?


I am working on the file "Python Workshop/Variables and Operators.ipynb".
The first task is:
We have the usual arithmetic operators, $+, -, /, $. On top of that we also have the power operator, $*$, the modulo operator $\%$ which returns the remainder from a quotient, the additive equality operator $+=$, and the equality operator, $=$. We also have the comparison operators; $>,<,==, !=, and, or$. Which mean, greater than, less than, is equal to, is not equal to, and, or.
Can you write Python code for it?

#addition
print(2+2)
#division or quotient
print(42/42)
#multiplication
print(2*3)
#subtraction
print(2-8)
#exponent or power
print(2**3)
#modulo or remainder
print(10%7)
#additive equality
a=56
a+=10
print(a)
#greater than
print(10>5)
#less than
print(10<5)
#is equal to
print(11==10)
#is not equal to
print(10!=10)
#and
print(11==11 and 12>10)
#or
print(10==11 or 5<7)


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
A few quick examples of easy logic statements
Can you write Python code for it?

print L

2 in L

4 in L

(2 in L) and (4 not in L)


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
Addition concatenates lists.
Lists can be a combination of different object types, including other lists
Can you write Python code for it?


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
Better way to generate a list
Don't do this for very complex calculations - gets hard to read
Can you write Python code for it?

%%timeit
[value**2 for value in range(500)]


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
I defined a function above. Functions start with def, end the first line with a colon, and use return if you are returning values
Dictionaries
Dictionaries are flexible mappings of keys to values. They are created via a comma-separated list of key:value pairs within curly braces.
Can you write Python code for it?

numbers = {'one':1, 'two':2, 'three':3}
numbers

numbers['two'] # Use a key to return the matching value

numbers.keys() # Keys don't stay ordered


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
I said list-like objects, which points to an important feature of Python: types don't matter so long as they behave correctly (duck typing)
Can you write Python code for it?

l_array = np.array(l) # make a numpy array from the list
l_array
# l_array.reshape((2,3))

set(l_array)


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
Iterators
Iterators are objects with multiple values that can be looped through. Sometimes you can also do indirect iteration, which will provide a single value at a time.
It's helpful to know about the enumerate and zip functions when using iterators.
Enumerate
Return both the index and value for each item in the iterator
Can you write Python code for it?

for idx, value in enumerate(['dog', 'cat', 'pig', 'sheep']):
    print idx, value


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
List comprehensions
Lists are great basic objects, and can be extremely fast if used correctly
Slower way to add values to a list
Can you write Python code for it?

# range generates a list from 0 to n-1
# Behavior is different in Python 3
range(5)

%%timeit
l = [] # Iniitalize an empty list
for value in range(500):
    l.append(value**2)


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
List indexing and slicing
Python provides access to elements through indexing (single elements), and slicing (multiple elements).
Both are indicated by a square-bracket syntax.
Can you write Python code for it?

L = [2, 3, 5, 7, 11]

L[0] # Python is 0 indexed

L[:-2]

L[0:3] # element 3 is not included [)


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
Off-topic note
Python 3 does away with int/float division issues. You can add this behavior to Python 2.
Can you write Python code for it?

# 3/2
3/2.

from __future__ import division
3/2


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
Python variables are pointers
Fortunately, simple objects are immutable - you can't change the value of "10" in memory, but you can point a variable to a different value
Can you write Python code for it?

x = 10
y = x
# add 5 to x's value, and assign it to x
x += 5
print "x =", x
print "y =", y


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
Tuples
Tuples are in many ways similar to lists, but they are defined with parentheses rather than square brackets.
Or they can be defined without any brackets.
Can you write Python code for it?


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
Tuples are often used to pass groups of values
Pass a group of objects/values into or out of a function
Can you write Python code for it?

def f(x):
    a = x**2
    b = x**3
    return a, b

f(2)

a, b = f(2)
print 'a =', a
print 'b =', b


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
Types of objects (data structures)
Common basic Python objects include:
- Simple objects like ints, floats, strings, etc
- Lists
- Sets
- Dictionaries
- Tuples
| Type      | Example                 |Description                            |
|-----------|-------------------------|---------------------------------------|
|   list    | [1, 2, 3]           | Ordered collection                    |
| tuple | (1, 2, 3)           | Immutable ordered collection          |
| dict  | {'a':1, 'b':2, 'c':3} | Unordered (key,value) mapping         |
| set   | {1, 2, 3}           | Unordered collection of unique values |
Lists
Some basic list operations
Can you write Python code for it?

L = [2, 3, 5, 7] # Define a list
print L
print len(L)


I am working on the file "Python and Jupyter basics/Python basics - RISE presentation.ipynb".
The first task is:
Zip
Combine two or more sets of iterators into a list of tuples
Can you write Python code for it?

L = [2, 4, 6, 8, 10]
R = [3, 6, 9, 12, 15]
for lval, rval in zip(L, R):
    print (lval, rval)
# zip(L, R)[0]
# Doing the same thing with indexing
# for i in range(len(L)):
#     print (L[i], R[i])


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
%prun takes the same “command line options” as cProfile but will profile an arbitrary Python statement instead of a while .py file
Profiling a Function Line-by-Line
%lprun that computes a line-by-line-profiling of one or more functions.
IPython HTML Notebook
It has a JSON-based .ipynb document format that enables easy sharing of code, output, and figures. The notebook application runs as a lightweight server process on the command line.
Tips for Productive Code Development Using IPython

Reloading Module Dependencies: Python’s “load-once” module system makes it difficult to reflect changes in imported libraries. Use reload(some_lib) and dreload(some_lib) after import statement.
Keep relevant objects and data alive
Flat is better than nested
Overcome a fear of longer files

Advanced IPython Features

Making Your Own Classes IPython-friendly: IPython takes the string returned by the _repr_ magic method (by doing output = repr(obj)) and prints that to the console. Thus, we can add a simple _repr_ method to the above class to get a more helpful output

Profiles and Configuration
All of these configuration options are specified in a special ipython_config.py file which will be found in the ~/.config/ipython/ directory on UNIX-like systems and %HOME %/.ipython/ directory on Windows.
NumPy Basics: Arrays and Vectorized Computation
NumPy, short for Numerical Python, is the fundamental package required for high performance scientific computing and data analysis.
Features:

ndarray, a fast and space-efficient multidimensional array providing vectorized arithmetic operations and sophisticated broadcasting capabilities
Standard mathematical functions for fast operations on entire arrays of data without having to write loops
Tools for reading/writing array data to disk and working with memory-mapped files
Linear algebra, random number generation, and Fourier transform capabilities
Tools for integrating code written in C, C++, and Fortran

The NumPy ndarray: A Multidimensional Array Object
One of the key features of NumPy...
Can you write Python code for it?

data1 = [6, 7.5, 8, 0, 1]
arr1 = np.array(data1)

arr1

arr1.shape

arr1.dtype

np.arange(15)


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Adding legends
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)

ax.plot(np.random.randn(1000).cumsum(), 'k', label='one')
ax.plot(np.random.randn(1000).cumsum(), 'k--', label='two')
ax.plot(np.random.randn(1000).cumsum(), 'k.', label='three')

ax.legend(loc='best')


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Annotations and Drawing on a Subplot
Annotations and text can be added using the text , arrow , and annotate functions. text draws text at given coordinates (x, y) on the plot with optional custom styling.
Drawing shapes requires some more care. matplotlib has objects that represent many common shapes, referred to as patches. Some of these, like Rectangle and Circle are found in matplotlib.pyplot , but the full set is located in matplotlib.patches.
Saving Plots to File
The active figure can be saved to file using plt.savefig . This method is equivalent to the figure object’s savefig instance method.
matplotlib Configuration
matplotlib comes configured with color schemes and defaults that are geared primarily toward preparing figures for publication. Fortunately, nearly all of the default behavior can be customized via an extensive set of global parameters governing figure size, subplot spacing, colors, font sizes, grid styles, and so on. There are two main ways to interact with the matplotlib configuration system. The first is programmatically from Python using the rc method.
For more extensive customization and to see a list of all the options, matplotlib comes with a configuration file matplotlibrc in the matplotlib/mpl-data directory. If you customize this file and place it in your home directory titled .matplotlibrc , it will be loaded each time you use matplotlib.
Plotting Functions in pandas
matplotlib is actually a fairly low-level tool. Therefore, pandas has an increasing number of high-level plotting methods for creating standard visualizations that take advantage of how data is organized in DataFrame objects.
Line Plots
Series and DataFrame each have a plot method for making many different plot types. By default, they make line plots.
Can you write Python code for it?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline

s = pd.Series(np.random.randn(10).cumsum(), index=np.arange(0, 100, 10))

s.plot()

df = pd.DataFrame(np.random.randn(10, 4).cumsum(0),
               columns=['A', 'B', 'C', 'D'],
               index=np.arange(0, 100, 10))

df

df.plot()


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Arithmetic and data alignment
One of the most important pandas features is the behavior of arithmetic between objects with different indexes. When adding together objects, if any index pairs are not the same, the respective index in the result will be the union of the index pairs.
Can you write Python code for it?


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Bar Plots
Making bar plots instead of line plots is a simple as passing kind='bar' (for vertical bars) or kind='barh' (for horizontal bars). In this case, the Series or DataFrame index will be used as the X ( bar ) or Y ( barh ) ticks.
Can you write Python code for it?

data = pd.Series(np.random.rand(16), index=list('abcdefghijklmnop')).plot(kind="bar")

data = pd.Series(np.random.rand(16), index=list('abcdefghijklmnop')).plot(kind="barh")


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Boolean indexing
Can you write Python code for it?

names = np.array(['Bob', 'Joe', 'Will', 'Bob', 'Will', 'Joe', 'Joe'])
data = np.random.randn(7, 4)

data[names == "Bob", :]


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
By default merge does an 'inner' join; the keys in the result are the intersection. Other possible options are 'left' , 'right' , and 'outer'.
Merging on Index
In some cases, the merge key or keys in a DataFrame will be found in its index. In this case, you can pass left_index=True or right_index=True (or both) to indicate that the index should be used as the merge key.
Concatenating Along an Axis
Another kind of data combination operation is alternatively referred to as concatenation, binding, or stacking. NumPy has a concatenate function for doing this with raw NumPy arrays.
Can you write Python code for it?

arr = np.arange(12).reshape((3, 4))

np.concatenate([arr, arr], axis=1)


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Combining Data with Overlap
Another data combination situation can’t be expressed as either a merge or concatenation operation. You may have two datasets whose indexes overlap in full or part.
Can you write Python code for it?

a = Series([np.nan, 2.5, np.nan, 3.5, 4.5, np.nan],
           index=['f', 'e', 'd', 'c', 'b', 'a'])

b = Series(np.arange(len(a), dtype=np.float64),
           index=['f', 'e', 'd', 'c', 'b', 'a'])

b[-1] = np.nan

np.where(pd.isnull(a), b, a)

b[:-2].combine_first(a[2:])


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Correlation and Covariance
Can you write Python code for it?

df = DataFrame(np.random.randn(4, 3), index=list("0123"), columns=list("abc"))

df

df.corr()

df.cov()


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Counting Time Zones in pandas
Can you write Python code for it?

df = pd.DataFrame(records)

df.head()

df.tz.value_counts().head()

clean_tz = df.tz.fillna('Missing')
clean_tz[clean_tz == ''] = 'Unknown'

tz_counts = clean_tz.value_counts()
tz_counts.head()

tz_counts.head(10).plot(kind='barh')

df.shape

clean_df = df[df.a.notnull()]
clean_df.shape

operating_system = np.where(clean_df.a.str.contains('Windows'), 'Windows', 'Not Windows')
operating_system

operating_system.shape


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Counting Time Zones in pure Python
Can you write Python code for it?

type(records[0])

records[0].keys()

time_zones = [record['tz'] for record in records if 'tz' in record]

len(time_zones)

time_zones[:10]

from collections import Counter
time_zone_counter = Counter(time_zones)
time_zone_counter

sorted_time_zone_counter = time_zone_counter.most_common()

sorted_time_zone_counter


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Data Types for ndarrays
Can you write Python code for it?

numeric_strings = np.array(['1.25', '-9.6', '42'], dtype=np.string_)
numeric_strings.astype(float)


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Dataframe
A DataFrame represents a tabular, spreadsheet-like data structure containing an ordered collection of columns, each of which can be a different value type (numeric, string, boolean, etc.). The DataFrame has both a row and column index; it can be thought of as a dict of Series (one for all sharing the same index)
Can you write Python code for it?


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Directory Bookmark System
IPython has a simple directory bookmarking system to enable you to save aliases for common directories so that you can jump around very easily.
Can you write Python code for it?

%bookmark notebooks /usr/local/src/notebooks

cd notebooks


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Dropping entries from an axis
Dropping one or more entries from an axis is easy if you have an index array or list without those entries.
Can you write Python code for it?

obj = Series(np.arange(5.), index=['a', 'b', 'c', 'd', 'e'])
obj

obj.drop('c', axis=0)


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Essential Functionality
Reindexing
reindex is to create a new object with the data conformed to a new index.
Can you write Python code for it?


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Figures and Subplots
Plots in matplotlib reside within a Figure object. You can create a new figure with plt.figure
Can you write Python code for it?

fig = plt.figure()
ax1 = fig.add_subplot(2, 2, 1)
ax2 = fig.add_subplot(2, 2, 2)
ax3 = fig.add_subplot(2, 2, 3)

plt.plot(np.random.randn(50).cumsum(), 'k--')
_ = ax1.hist(np.random.randn(100), bins=20, color='k', alpha=0.3)
ax2.scatter(np.arange(30), np.arange(30) + 3 * np.random.randn(30))


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Function application and mapping
Can you write Python code for it?

import numpy as np

frame = DataFrame(np.random.randn(4, 3), columns=list("bde"), index=["Utah", "Ohio", "Texas", "Oregon"])

np.abs(frame)

f = lambda x: x.max() - x.min()

frame.apply(f)

frame.apply(f, axis=1)


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Grouping by Index Levels
Can you write Python code for it?

columns = pd.MultiIndex.from_arrays([['US', 'US', 'US', 'JP', 'JP'],
                                     [1, 3, 5, 1, 3]], names=['cty', 'tenor'])

hier_df = pd.DataFrame(np.random.randn(4, 5), columns=columns)

hier_df

hier_df.groupby(level='cty', axis=1).count()


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Grouping with Dicts and Series
Can you write Python code for it?

people = pd.DataFrame(np.random.randn(5, 5),
                   columns=['a', 'b', 'c', 'd', 'e'],
                   index=['Joe', 'Steve', 'Wes', 'Jim', 'Travis'])

mapping = {'a': 'red', 'b': 'red', 'c': 'blue',
           'd': 'blue', 'e': 'red', 'f' : 'orange'}

people.groupby(mapping, axis=1).mean()

map_series = pd.Series(mapping)

people.groupby(map_series, axis=1).count()


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Grouping with Functions
Can you write Python code for it?


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Hierarchical Indexing
Hierarchical indexing is an important feature of pandas enabling you to have multiple (two or more) index levels on an axis. Somewhat abstractly, it provides a way for you to work with higher dimensional data in a lower dimensional form.
Can you write Python code for it?


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Histograms and Density Plots
A histogram, with which you may be well-acquainted, is a kind of bar plot that gives a discretized display of value frequency. The data points are split into discrete, evenly spaced bins, and the number of data points in each bin is plotted.
A related plot type is a density plot, which is formed by computing an estimate of a continuous probability distribution that might have generated the observed data. A usual procedure is to approximate this distribution as a mixture of kernels , that is, simpler distributions like the normal (Gaussian) distribution.
Scatter Plots
Scatter plots are a useful way of examining the relationship between two one-dimensional data series. matplotlib has a scatter plotting method that is the workhorse of making these kinds of plots.
Plotting Maps: Visualizing Haiti Earthquake Crisis Data
Ushahidi is a non-profit software company that enables crowdsourcing of information related to natural disasters and geopolitical events via text message.
Python Visualization Tool Ecosystem

Chaco: Chaco developed by Enthought, is a plotting toolkit suitable both for static plotting and interactive visualizations. It is especially wellsuited for expressing complex visualizations with data interrelationships.
mayavi is a 3D graphics toolkit built on the open source C++ graphics library VTK. The plots can be panned, rotated, and zoomed using the mouse and keyboard.

Data Aggregation and Group Operations
GroupBy Mechanics
GroupBy uses split-apply-combine:
In the first stage of the process, data contained in a pandas object, whether a Series, DataFrame, or otherwise, is split into groups based on one or more keys that you provide. The splitting is performed on a particular axis of an object. Once this is done, a function is applied to each group, producing a new value. Finally, the results of all those function applications are combined into a result object.
Can you write Python code for it?

df = pd.DataFrame({'key1' : ['a', 'a', 'b', 'b', 'a'],
                'key2' : ['one', 'two', 'one', 'two', 'one'],
                'data1' : np.random.randn(5),
                'data2' : np.random.randn(5)})

df

df['data1'].groupby(df['key1']).mean()

df['data1'].groupby([df['key1'], df['key2']]).mean()


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
IPython: An Interactive Computing and Development Environment
Key features

Tab Completion
Introspection (?)
The %run command
Executing code from clipboard (%paste and %cpaste)
Magic Commands (%timeit)

Qt-based Rich GUI Console
$ ipython qtconsole --pylab=inline
Matplotlib Integration and Pylab Mode
$ ipython --pylab
Input and Output variables
The previous two outputs are stored in the _ (one underscore) and __ (two underscores) variables, respectively.
Input variables are stored in variables named like _iX, where X is the input line number
Logging the input and output
IPython is capable of logging the entire console session including input and output. Logging is turned on by typing %logstart.
Companion functions %logoff, %logon, %logstate, and %logstop.
Interacting with the Operating System
One can perform most standard command line actions as you would in the Windows or UNIX (Linux, OS X) shell without having to exit IPython. This includes executing shell commands, changing directories, and storing the results of a command in a Python object (list or string).
Shell command and aliases
Starting a line in IPython with an exclamation point !, or bang, tells IPython to execute everything after the bang in the system shell.
Can you write Python code for it?


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Index objects
pandas’s Index objects are responsible for holding the axis labels and other metadata (like the axis name or names). Any array or other sequence of labels used when constructing a Series or DataFrame is internally converted to an Index. Index objects are immutable and thus can’t be modified by the user
Can you write Python code for it?

obj = Series(range(3), index=['a', 'b', 'c'])
index = obj.index
index


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Introduction to pandas Data Structures
Series
A Series is a one-dimensional array-like object containing an array of data (of any NumPy data type) and an associated array of data labels, called its index.
Can you write Python code for it?


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Iterating Over Groups
Can you write Python code for it?

for name, group in df.groupby('key1'):
    print name, group

dict(list(df.groupby('key1')))


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
JSON is very nearly valid Python code with the exception of its null value null and some other nuances (such as disallowing trailing commas at the end of lists). The basic types are objects (dicts), arrays (lists), strings, numbers, booleans, and nulls. All of the keys in an object must be strings. There are several Python libraries for reading and writing JSON data. I’ll use json here as it is built into the Python standard library. To convert a JSON string to Python form, use json.loads.
json.dumps on the other hand converts a Python object back to JSON
XML and HTML: Web Scraping
Python has many libraries for reading and writing data in the ubiquitous HTML and XML formats. lxml (http://lxml.de) is one that has consistently strong performance in parsing very large files. lxml has multiple programmer interfaces; first I’ll show using lxml.html for HTML, then parse some XML using lxml.objectify.
Binary Data Formats
One of the easiest ways to store data efficiently in binary format is using Python’s built-in pickle serialization. Conveniently, pandas objects all have a save method which writes the data to disk as a pickle.
You read the data back into Python with pandas.load , another pickle convenience function.
Using HDF5 Format
There are a number of tools that facilitate efficiently reading and writing large amounts of scientific data in binary format on disk. A popular industry-grade library for this is HDF5, which is a C library with interfaces in many other languages like Java, Python, and MATLAB. The “HDF” in HDF5 stands for hierarchical data format. Each HDF5 file contains an internal file system-like node structure enabling you to store multiple datasets and supporting metadata. Compared with simpler formats, HDF5 supports on-the-fly compression with a variety of compressors, enabling data with repeated patterns to be stored more efficiently. For very large datasets that don’t fit into memory, HDF5 is a good choice as you can efficiently read and write small sections of much larger arrays.
There are not...
Can you write Python code for it?

df1 = DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'a', 'b'],
                 'data1': range(7)})

df2 = DataFrame({'key': ['a', 'b', 'd'],
                 'data2': range(3)})

df1.merge(df2)

pd.merge(df1, df2, on="key")


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
MovieLens 1M Data Set
Can you write Python code for it?

unames = ['user_id', 'gender', 'age', 'occupation', 'zip']
users = pd.read_table('data/ml-1m/users.dat', delimiter='::', header=None, names=unames, engine='python')

rnames = ['user_id', 'movie_id', 'rating', 'timestamp']
ratings = pd.read_table('data/ml-1m/ratings.dat', delimiter='::', header=None, names=rnames, engine='python')

mnames = ['movie_id', 'title', 'genres']
movies = pd.read_table('data/ml-1m/movies.dat', delimiter='::', header=None, names=mnames, engine='python')

users.head()

ratings.head()

movies.head()

data = pd.merge(pd.merge(ratings, users), movies)
data.head()

data.dtypes

data.loc[0, :]

data.columns

data.loc[0, 'user_id']

mean_ratings = data.pivot_table('rating', index='title', columns='gender')
mean_ratings.head()

ratings_by_title = data.groupby('title').size().sort_values(ascending=False)
ratings_by_title.head()

active_titles = ratings_by_title.index[ratings_by_title >= 250]
active_titles.size

mean_ratings = mean_ratings.loc[active_titles, :]
mean_ratings.head()

top_female_ratings = mean_ratings.sort_values(by='F', ascending=False)
top_female_ratings.head()

mean_ratings['diff'] = mean_ratings['M'] - mean_ratings['F']
mean_ratings.head()

mean_ratings.sort_values(by='diff', ascending=False).head()

rating_std_by_title = data.groupby('title').rating.std()
rating_std_by_title.head()

rating_std_by_title.loc[active_titles].sort_values(ascending=False).head()


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Pivoting “long” to “wide” Format
Note that pivot is just a shortcut for creating a hierarchical index using set_index and reshaping with unstack.
Data Transformation
Removing Duplicates
Can you write Python code for it?

data = DataFrame({'k1': ['one'] * 3 + ['two'] * 4,
                  'k2': [1, 1, 2, 3, 3, 4, 4]})

data

data.duplicated()

data.drop_duplicates()


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Plotting and Visualization
It may be a part of the exploratory process; for example, helping identify outliers, needed data transformations, or coming up with ideas for models. matplotlib is a (primarily 2D) desktop plotting package designed for creating publication-quality plots. matplotlib has a number of add-on toolkits, such as mplot3d for 3D plots and basemap for mapping and projections.
Can you write Python code for it?

import matplotlib.pyplot as plt

%matplotlib inline

plt.plot(np.arange(10))


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Regular expressions
Can you write Python code for it?

import re

text = "foo      bar\t baz \tqux"

re.split('\s+', text)

regex = re.compile('\s+')

regex.split(text)

regex.findall(text)


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Renaming Axis Indexes
Can you write Python code for it?


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Reordering and Sorting Levels
Can you write Python code for it?

frame = DataFrame(np.arange(12).reshape((4, 3)),
                  index=[['a', 'a', 'b', 'b'], [1, 2, 1, 2]],
                  columns=[['Ohio', 'Ohio', 'Colorado'],
                           ['Green', 'Red', 'Green']])

frame.index.names = ['key1', 'key2']
frame.columns.names = ['state', 'color']

frame

frame.swaplevel("key1", "key2")

frame.swaplevel(0, 1).sortlevel(0)


I am working on the file "Python for Data Analysis.ipynb".
The first task is:
Software Development Tools
IPython has closely integrated and enhanced the built-in Python pdb debugger. IPython has easy-to-use code timing and profiling tools.
Interactive Debugger
The %debug command, when entered immediately after an exception, invokes the “post-mortem” debugger and drops you into the stack frame where the exception was raised.
Executing the %pdb command makes it so that IPython automatically invokes the debugger after any exception, a mode that many users will find especially useful.
Timing Code: %time and %timeit
%time runs a statement once, reporting the total execution time.
Can you write Python code for it?

strings = ['foo', 'foobar', 'baz', 'qux', 'python', 'Guido Van Rossum'] * 100000

%time method1 = [x for x in strings if x.startswith('foo')]

%time method2 = [x for x in strings if x[:3] == 'foo']