I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Opening files. For example, print the headers in the first markdown file in the current directory.
Note that list(Path(cwd).glob('./*.md'))[0] comprises the following: glob returns a generator to paths, which must be made to generate a list, of which the 0 element is used.
When opening the file, the name is already present in p.
Can you write Python code for it?

p = list(Path(cwd).rglob('./*.md'))[0]
print(type(p))
with p.open(mode='r') as fid:
    headers = [line.strip() for line in fid if line.startswith('#')]
print('\n'.join(headers))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Operations on Numpy arrays
NumPy operations are usually done on pairs of arrays on an element-by-element basis. In the simplest case, the two arrays must have exactly the same shape, as in the following example.  Normally, broadcasting is transparent and operates as one would intuitively think.
Broadcasting is where numpy can figure out how to handle arrays of different shape, it the operation is well defined, e.g., adding a constant to an array - the constant is added to all elements. Broadcasting is very important and a rudimentary understanding is necessary.
See http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html
Note that in this case the transpose had no effect.
Can you write Python code for it?

e = (d + 2).T
print(e.shape)
print(e)

a = np.array([1.0, 2.0, 3.0])
b = np.array([2.0, 2.0, 2.0])
print('a * b  ={}'.format(a * b))
print('a - b  ={}'.format(a - b))
a = np.array([1.0, 2.0])
b = np.array([2.0, 2.0, 2.0])
try:
    print('a * b  ={}'.format(a * b))
except:
    print('{} * {} is not allowed, broadcasting not possible'.format(a,b))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Overloaded multiplication for non-numbers
Can you write Python code for it?


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Python 2 has limited list unpacking capabilities.  The following examples unpacks lists using the itertools module.
Can you write Python code for it?

import itertools

list2d = [[1,2,3],[4,5,6], [7], [8,9]]
merged = list(itertools.chain.from_iterable(list2d))
print(merged)

list2d = [['foo']*3,['bar']*3,['ton']*3]
merged = list(itertools.chain.from_iterable([['foo']*3,['bar']*3,['ton']*3]))
print(merged)

# python 3
#a,*b,c = [0, 1, 2, 3, 4, 5]


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Python Classes
Not presently covered here. Please google.
Function Objects and Operator Overloading
From wikipedia: In Python, functions are first-class objects, just like strings, numbers, lists etc. This feature eliminates the need to write a function object in many cases. Any object with a call() method can be called using function-call syntax. An example is this accumulator class (based on Paul Graham's study on programming language syntax and clarity):
Can you write Python code for it?

class Accumulator(object):
    def __init__(self, n):
        self.n = n

    def __call__(self, x):
        self.n += x
        return self.n

a = Accumulator(4)
print(a(3))
print(a(2))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Python Standard Library Logger
The logging module in the standard library provides a reasonably powerful logger capability.  This examples shows how to log to two files at the same time.  The case in point is for a session-level logfile, plus any number of lower level detailed log files.
The logging module provides a global-scope object, so there is no need to pass the object along between functions or objects.
The global logging object provides the functionality to have any number of individual loggers, which can be accessed by using a  key (the filename) to indicate which logger you want to use.  The file that is physically created on disk is named using the key, appended with '.log'.
The logger shown here opens handlers to cout and a disk file, each providing different levels of logging.  The logger is created by the function setUpLogFile and is subsequently activated by code of the form logging.getLogger(filename), where filename is the logfile filename (which also serves as the key to the logger).
The example code below provides the functionality where the user can provide a path to where the logfile must be created on disk (relative to where the current working directory).
The Python logging module provides the following threshold levels:  CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET in order of severity.  When logging, all messages marked with a level equal to, or higher than the threshold level, will be printed.  At the highest level only critical messages will be printed, whereas at the debug level all higher level prints are also performed.
The class SpecialFormatter defines how the different levels of output are formatted.
This code is based on an example by Lynette van der Westhuizen.
Can you write Python code for it?

import logging
import datetime as dt
import sys
import os.path

#===============================================================================
def setUpLogFile(logfilename, saveDir, logLevelCmnd=logging.WARNING, logLevelFile=logging.DEBUG):
    """Set up the log file for IDA.

    Args:
        | logfilename (string) : Name of the log file - this is also the key to access the logger.
        | saveDir (string) : Directory to save the log file, relative to cwd.
        | logLevel (string): log level (logging.ERROR, logging.WARNING, logging.INFO, logging.DEBUG)

    Returns:
        | newLogger (logging file) : Retruns a log file with the name [logfilename]Log.
                             A log file [logfilename]Log.log is created in saveDir.
    Raises:
        | None
    """

    #--- Get the log file and to log all logLevel
    newLogger = logging.getLogger(logfilename)
    newLogger.setLevel(logLevelCmnd)
    newLogger.handlers=[] #Ensure no handlers so that duplicates aren't added in the event the code crashed and logger didn't close

    #--- Set a file handler log from level :
    filepath = os.path.join(saveDir, logfilename.lower())
    fileHdlr = logging.FileHandler("{}.log".format(filepath),'w')
    fileHdlr.setLevel(logLevelFile)
    fileHdlr.setFormatter(SpecialFormatter())
    newLogger.addHandler(fileHdlr)

    #--- Set a command window handler log from level WARNING
    cmdHdlr = logging.StreamHandler()
    cmdHdlr.setLevel(logLevelCmnd)
    cmdHdlr.setFormatter(SpecialFormatter())
    newLogger.addHandler(cmdHdlr)

    #--- Print the log file creation time
    printCreationTime = '\n==========================================================\n'+\
                        'Date of execution: {} {}\n'.format(dt.datetime.now(), logfilename) +\
                        '==========================================================\n\n'.format(logfilename)
    newLogger.info(printCreationTime)

    return newLogger

#===============================================================================
class...


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Python's lists, tuples and dictionaries are very powerful structures.
Python uses zero-based indexing, meaning that the first element is indicated as element 0.
Lists lst=[1,2] are mutable, which means that you can change them in place, by appending and inserting elements.
Can you write Python code for it?

#append to an existing list
lstNumbers = [4, 1, 2,  3, 4, ]
lstNumbers.append(5)
lstNumbers.append([2, 6, 7, 8])
print(lstNumbers)
#insert into specific locations into list
lstDiff = [1, 'a', 2]
lstDiff.insert(0,'zero')
lstDiff.insert(2,'wow')
print(lstDiff)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Rather create multiple instances add them one at a time
Can you write Python code for it?

dic = []
lst = []
for i in [0,1,2]:
    dic.append({})
    lst.append([])

lst[0].append('mystring')
dic[0]['test'] = 'mystring'
print(lst)
print(dic)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Reading and writing raw binary files
Binary files are handled the same as text files, but with file open modes rb or wb:
# Read the entire file as a single byte string
with open('binaryfile.bin', 'rb') as f:
    data = f.read()

# Write binary data to a file
with open('binaryfile.bin', 'wb') as f:
    f.write(b'Hello World')

When reading binary all data returned will be in the form of byte strings, not text strings. Similarly, when writing, you must supply data in the form of objects that expose data as bytes.
Using Python arrays, values can be written without conversion to a bytes object. The data type must be declared in the first parameter of the array.array function.  These data types include signed and unsigned shorts, ints, longs, doubles, and unicode.
import array
nums = array.array('i', [1, 2, 3, 4])
with open('data.bin','wb') as f:
    f.write(nums)

https://docs.python.org/2/library/array.html
http://chimera.labs.oreilly.com/books/1230000000393/ch05.html#_discussion_75
The following code writes two unsigned integers, two doubles and a numpy array to raw binary format:
Can you write Python code for it?

import numpy as np
import array
img = np.ones((3,3))
with open('binfile.bin','wb') as fo:
    header = array.array('I',[512, 1024])
    fo.write(header)
    header = array.array('d',[700, 1250])
    fo.write(header)
    img.tofile(fo)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Reading user keyboard input
Reading user input - this even works in the IPython notebook! Strings must be in quotes. You can even enter lists and dictionaries!
Can you write Python code for it?

# foo=input('Please enter a value:')
# print(foo)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Regular Expressions
https://www3.ntu.edu.sg/home/ehchua/programming/howto/Regexe.html
Optimizing and performance tricks
The Top Mistakes Developers Make When Using Python for Big Data Analytics
References and garbage collection
Python has built-in garbage collection, which for most tasks appear to work reasonably well.  In some cases however the standard garbage collection methodology may not suffice and user intervention may be required.
Python's garbage collection and its intricacies requires a complete, separate notebook, but here is a summary.  What follows is my present understanding, which is incomplete and may even be wrong in places.
Python's memory model uses a reference-to-a-data-item with associated reference counting approach.  Some data item is allocated memory and a reference to that memory is given when you 'assign' the data item to a name. So,
a = ['gg','hh']

creates the data item (the list) and allocates a reference to this data item to the variable a. Note that the data item and the variable are two different things, a merely refers to the data item, it isn't the data item.
An object is automatically marked to be collected when its reference count drops to zero.
gc – Python's Garbage Collector
Memory Management and Limits
http://stackoverflow.com/questions/9617001/python-garbage-collection-fails
http://python.6.x6.nabble.com/question-about-garbage-collection-td1661865.html
http://stackoverflow.com/questions/16261240/releasing-memory-of-huge-numpy-array-in-ipython
The sys.getrefcount() function shows the number of references attached/assigned to the data item. The count is actually one higher than expected because there is a temporary reference to the object held by getrefcount() itself.
References are removed by assigning the value None to the variable.
Can you write Python code for it?

import sys
a = None
print('before assigning ref  a  {}'.format(sys.getrefcount(a)))
a = ['gg','hh']
print('after  assigning ref  a  {}'.format(sys.getrefcount(a)))
b = a
print('after  assigning ref  b  {}'.format(sys.getrefcount(a)))
b = None
print('after  removing  ref  b  {}'.format(sys.getrefcount(a)))
a = None
print('after  removing  ref  a  {}'.format(sys.getrefcount(a)))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Rotation around the $x$ axis:
Can you write Python code for it?


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
SciPy has a number of interpolation options in scipy.interpolate.  The one-dimensional interpolation scheme enables the definition of the lookup table as a function by using scipy.interpolate.interp1d.  The user can define the interpolation scheme to be used - the default is kind='linear', to do cubic interpolation use kind='cubic'.  In the example below obsserve the difference between the two schemes. This is an ill-posed problem: normally one would have a much denser input data set.  The purpose with using this poor data set it to illustrate the dangers of interpolation.
Can you write Python code for it?

from scipy import interpolate
import pyradi.ryplot as ryplot
import numpy as np
%matplotlib inline

# create input data in wavelength and wavenumber domain
wl = np.asarray([1, 2, 5, 10])
fn = np.asarray([2,3,1, 2])

# create the plotting samples - much tighter than input data
wli = np.linspace(1,10,100)

#create the lookup functions
fwll = interpolate.interp1d(wl,fn)
fwlc = interpolate.interp1d(wl,fn, kind='cubic')

p = ryplot.Plotter(1,1,1,figsize=(8,4))
p.plot(1,wli,fwll(wli),label=['linear'])
p.plot(1,wli,fwlc(wli),label=['cubic'])


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
See also here:
https://www.ics.uci.edu/~pattis/ICS-33/lectures/operatoroverloading1.txt
https://docs.python.org/2/c-api/function.html
http://python-3-patterns-idioms-test.readthedocs.io/en/latest/FunctionObjects.html
https://newcircle.com/bookshelf/python_fundamentals_tutorial/functional_programming
Building function parameters as a **kwargs dictionary
Requirement:

send  function fnpassed as parameter to function fncall
send the parameters to the passed to fnpassed also as parameters to fncall
make it possible to add new fnpassed-parameters inside fncall before fnpassed is called.

We want this to have a calling function fncall that can call some other function fnpassed with a user-supplied list of parameters, but also adding some more parameters in fncall.  In other words, the execution of fnpassed must be with parameters passed to fncall from outside, plus new parameters added inside fncall.
In the code below the function testfn must be executed with four parameters, three of which (1,2,3) are passed as tryy parameters , with the fourth (10) added within fncall, because the fourth is only available from within fncall (not beforehand).
The parameters passed to a function can be built up a dictionary and .   is not an operator, so it doesn't really have a name, but it is defined as a "syntactic rule". So it should be called: "the keyword argument unpacking syntax". It unpacks a dictionary into a function parameter list.
So we can build up the dictionary in different places with new information and when complete, present this to the function using the **** unpacking.
https://pythontips.com/2013/08/04/args-and-kwargs-in-python-explained/
http://softwareengineering.stackexchange.com/questions/131403/what-is-the-name-of-in-python/131415
http://stackoverflow.com/questions/36901/what-does-double-star-and-star-do-for-parameters
http://stackoverflow.com/questions/1769403/understanding-kwargs-in-python
Can you write Python code for it?

import numpy as np

def fnpassed(a,b,c,d):
    # also return the input parameters as a tuple
    return (a * b + c) / d, (a,b,c,d)

def fncall(fn,kwargs):
    # add the fourth parameter
    kwargs['d'] = 10.
    print('Building up kwargs: {}\n'.format(kwargs))
    aa = fn(**kwargs)
    return aa

rtnval,inpval = fncall(fnpassed,kwargs={'a':1,'b':2,'c':3})
print('Return value = {}, using input values {}'.format(rtnval,inpval))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Selecting elements with array indices
Numpy provides the facility to extract only certain array indices, based on conditional values.
Some experiment requires a gaussian-distributed random number sequence, but with a twist: only values in a specific amplitude range is required.  The following script creates a large number of gaussian random variables, then filters them based on amplitude by selecting a sub set of all values by the logical test rndAll > 0.4. This test creates a logical array which selects elements from rndAll. The histograms of the two distributions are then calculated and displayed.
The np.histogram function can return either the counts in the bins (density=False), or the estimated probability function (density=True).
Can you write Python code for it?

import numpy as np
import pyradi.ryplot as ryplot
%matplotlib inline

rndAll = np.random.randn(10000)
rndSel = rndAll[rndAll > 0.4]
print('Number of random numbers in input set is {}'.format(rndAll.shape[0]))
print('Number of random numbers in filtered set is {}'.format(rndSel.shape[0]))

hAll, bAll = np.histogram(rndAll,bins=50)
hSel, bSel = np.histogram(rndSel,bins=bAll) # use the bins

p = ryplot.Plotter(1,1,2,figsize=(12,4))
p.plot(1,bAll[1:],hAll,'', 'Amplitude', 'Count', label=['Full dataset'])
p.plot(1,bSel[1:],hSel,'', 'Amplitude', 'Count', label=['Filtered dataset'])

hAll, bAll = np.histogram(rndAll,bins=50, density=True)
hSel, bSel = np.histogram(rndSel,bins=bAll, density=True) # use the bins
p.plot(2,bAll[1:],hAll,'', 'Amplitude', 'Probability', label=['Full dataset'])
p.plot(2,bSel[1:],hSel,'', 'Amplitude', 'Probability', label=['Filtered dataset'])

print('Integrated PDF for all data = {}'.format(np.trapz(hAll,bAll[1:])))
print('Integrated PDF for selected data = {}'.format(np.trapz(hSel,bSel[1:])))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Selections into array
Numpy has a rich set of functions to support conditional testing on element level. This section only touches on the surface.
In the example below a signal is created that only contains the positive values of a sine wave, being zero elsewhere.
Can you write Python code for it?

phase = np.linspace(0, 4 * np.pi, 101)
signal = np.sin(phase)
signalpos = np.where(signal>0, signal, 0)

p = ryplot.Plotter(1,figsize=(9,4))
p.plot(1,phase,signal,'', 'Phase rad', 'Signal', label=['Full sine wave'])
p.plot(1,phase,signalpos,'', 'Phase rad', 'Signal', label=['Positive sine wave'])


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Selective operations on elements in an array: Poisson distribution for large $\lambda$
The Poisson probability distribution describe the arrival/generation of discrete events, e.g., generation or arrival of photons in optical flux.  The noise variance $\sigma^2$ in such a photon stream is equal to the mean value $\lambda$.  The Poisson distribution is a discrete distribution and is defined only for integers.  A practical problem arises when the photon count is large: the numerical limits of integer arithmetic is exceeded in the Numpy np.random.poisson function.
For mean values exceeding 1000 the Normal distribution with mean and variance equal to $\lambda$ is a good approximation to the Poisson distribution.  The Normal distribution is a real valued distribution and is not restricted by integer computation limits.  So we can use the Normal distribution for large $\lambda$. However, for small $\lambda$ it is a poor approximation, so we must use both, each in the appropriate case.  This can be done with array code of the form shown here, where in is the value of $\lambda$:
out =  (inp<=1000) * np.random.poisson(inp * (inp<=1000) )\
      + (inp>1000) * np.random.normal(loc=inp, scale=np.sqrt(inp))

where (inp<=1000) and (inp>1000) are boolean arrays of the same size as the input, but with values corresponding with the element-wise conditional truth. The result therefore comprises a combination of Poisson and Normal elements, element-wise conditional on the value of $\lambda$.
To prevent the integer overflow for large $\lambda$ we also have to scale the value of $\lambda$ with the conditional in the code in * (inp<=1000), reducing the value to zero where it exceeds the threshold of 1000.
The above code works well, provided that $\lambda$ (inp) exceeds zero, because the Normal distribution is not kind to zero variance.  Conceptually it should just return the mean value, but computationally the implementation seems to want to divide by zero.  So we must prevent passing a zero standard...
Can you write Python code for it?

import numpy as np
tpoint = 1000
sdelta = 1e-10
asize = 100000 # you need values of more than 10000000 to get really good stats
for lam in [0, 10, tpoint-5, tpoint-1, tpoint, tpoint+1, tpoint+5, 20000]:
    inp = lam * np.ones((asize,1))
    out =  (inp<=tpoint) * np.random.poisson(inp * (inp<=tpoint) )\
          + ((inp>tpoint) & (inp!=0)) * np.random.normal(loc=inp, scale=np.sqrt(inp+sdelta))

    print('lam={} mean={} var={} err-mean={} err-var={}'.format(lam,
       np.mean(out),np.var(out), (lam-np.mean(out))/lam, (lam-np.var(out))/lam))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Sequence ascending or descending?
Can you write Python code for it?


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Slicing and Stacking
Numpy arrays can be sliced to extract portions of the array. Start by creating a rank-two array with 15 elements, then reshape to a 3 row/5 column array. Then slice different parts from the array. A slice is specified in the form: start:end:stride, where the start value is included but the end value is excluded and the stride definest stride from te starting position (inclusive).
http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html
Observe that the slice [:,1] returns a (N,) rank one vector whereas the slice [:,:1] returns a (N,1) rank two vector
Can you write Python code for it?

a = np.arange(0,15).reshape(3,5)
print(a)
print('Original array: {} has shape {}'.format(a, a.shape))
print('All the rows of second column: {} has shape {}'.format(a[:,1],a[:,1].shape))
print('All the colums of second row: {} has shape {}'.format(a[1,:],a[1,:].shape))
print('Portion of the array: {} has shape {}'.format(a[0:2,1:3],a[0:2,1:3].shape))
print('Strided portion of the array: {} has shape {}'.format(a[0:2,0:5:2],a[0:2,0:5:2].shape))
print('---------------')
print('Sliced by {} all the rows of second column: {} has shape {}'.format('[:,1]',a[:,1],a[:,1].shape))
print('Sliced by {} all the rows of second column: {} has shape {}'.format('[:,:1]',a[:,1],a[:,:1].shape))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Subclassing is not useful for adding new, stored fields. Instead, simply create a new named tuple type from the _fields attribute:
Point3D = namedtuple('Point3D', Point._fields + ('z',))

Default values can be implemented by using _replace() to customize a prototype instance:
Can you write Python code for it?

Account = namedtuple('Account', 'owner balance transaction_count')
default_account = Account('<owner name>', 0.0, 0)
johns_account = default_account._replace(owner='John', balance=50)
print(johns_account)

# def preparefiles(filename):
#     if os.path.isfile(filename):
#         os.remove(filename)
#     conn = sqlite3.connect(filename)
#     conn.execute('''CREATE TABLE COMPANY
#            (ID INT PRIMARY KEY     NOT NULL,
#            NAME           TEXT    NOT NULL,
#            AGE            INT     NOT NULL,
#            ADDRESS        CHAR(50),
#            SALARY         REAL);''')
#     conn.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) \
#           VALUES (1, 'Paul', 32, 'California', 20000.00 )");
#     conn.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) \
#           VALUES (2, 'Allen', 25, 'Texas', 15000.00 )");
#     conn.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) \
#           VALUES (3, 'Teddy', 23, 'Norway', 20000.00 )");
#     conn.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) \
#           VALUES (4, 'Mark', 25, 'Rich-Mond ', 65000.00 )");
#     conn.commit()
#     conn.close()


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
The Fick rotation sequence rotates first around the $z$ axis, then the $y$ axis and finally the $x$ axis.
Can you write Python code for it?


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
The Zen of Python
Can you write Python code for it?


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
The above is however not always true. For example, it does not apply to simple integers.  Assigning to b now created a new storage location with a new ID.
Can you write Python code for it?

a = 5
b = a
print(a, type(a), id(a), a is b, a==b)
print(b, type(b), id(b), a is b, a==b)
b = 6
print(a, type(a), id(a), a is b, a==b)
print(b, type(b), id(b), a is b, a==b)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
The array can also be output as LaTeX for inclusion in a report:
Can you write Python code for it?


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
The concept of a single spectral variable, such as the filter above, can be extended to many spectral variables, all sharing the same wavelength values. One such example is the Planck law radiation at different temperatures. In the next example, the radiation is calculated at a range of temperatures and plotted on the graph.  Note in this case that the value returned from the ryplanck.planck() function is a rank-two array.  Along the first dimension is the spectral axis and along the second dimension is the radiance for the different temperatures.
The plotting functions can plot more than one line if a rank-two array is passed. However many columns there are in the array, so many lines will be drawn.  The only requirement is that the size of the radiance (y) array along the zero axis much match the size of the wavelength (x) array along its zero axis.
Observe how the line labels are created using list comprehension, properly formatted with units.
Can you write Python code for it?

import pyradi.ryplanck as ryplanck

temperatures = [5000, 6000, 8000]
radiance = ryplanck.planck(wl,temperatures, 'el') / np.pi

print('wl.shape = {}'.format(wl.shape))
print('len(temperatures) = {}'.format(len(temperatures)))
print('radiance.shape = {}'.format(radiance.shape))

labels = ['{} K'.format(tmp) for tmp in temperatures]
print('labels = {}'.format(labels))

p = ryplot.Plotter(1,figsize=(9,4))
p.semilogY(1,wl,radiance * sfilter.reshape(-1,1),'Planck Radiance', 'Wavelength $\mu$m', 'Radiance W/(m$^2$.sr.$\mu$m)',label=labels)
p.saveFig('filtered-planck.png')

for i,temperature in enumerate(temperatures):
    integral = np.trapz((radiance[:,i]).reshape(-1,1) * sfilter.reshape(-1,1), wl, axis=0)
    print('Temperature = {} K,  Integrated radiance = {:.3e} W/(m2.sr)'.format(temperature, integral[0]))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
The following code collects all filenames in a dir and sub dirs and write the details to a data frame
Can you write Python code for it?

import time
from pathlib import PureWindowsPath, PurePosixPath
p = Path(cwd)
pstr = './data/**/*'
all_files = []
for i in p.rglob(pstr):
    all_files.append((i.name, i.parent, time.ctime(i.stat().st_ctime)))

# # Windows -> Posix
# win = r'foo\bar\file.txt'
# posix = str(PurePosixPath(PureWindowsPath(win)))
# print(posix)  # foo/bar/file.txt

columns = ["File_Name", "Parent", "Created"]
df = pd.DataFrame.from_records(all_files, columns=columns)
# convert to posix and get rid of the drive
df['Parent'] = df.apply(lambda x:str(PurePosixPath(PureWindowsPath(x['Parent'])))[3:] , axis = 1)

display(HTML(df.head().to_html()))
display(HTML(df.tail().to_html()))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
The following code multiplies two rank-2 arrays, note the different results from different multiplications:
Can you write Python code for it?

import numpy as np
a = np.asarray([0,1,2,3]).reshape(-1,1)
b = a.reshape(1,-1)
print('a={} a.shape={}\n'.format(a, a.shape))
print('b={} b.shape={}\n'.format(b, b.shape))

print('(N,1) x (N,1)  - element by element product')
aa = a * a
print('a*a={} aa.shape={}\n'.format(aa, aa.shape))
print('(1,N) x (1,N)  - element by element product')
bb = b * b
print('b*b={} bb.shape={}\n'.format(bb, bb.shape))

print('(N,1) x (1,N) - this is an outer product')
ab = a * b
print('a * b={} ab.shape={}\n'.format(ab, ab.shape))

print('(1,N) x (N,1) - this is an outer product')
ba = b * a
print('b * a={} ba.shape={}\n'.format(ba, ba.shape))

print('(N,1) . (N,1)  - dot product')
adb = np.dot(a,b)
print('a.b={} (a.b).shape={}\n'.format(adb, adb.shape))

print('(1,N) . (N,1)  - dot product')
bda = np.dot(b,a)
print('b.a={} (b.a).shape={}\n'.format(bda, bda.shape))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
The numpy.meshgrid function was conceived to create a two-dimensional grid, e.g., for a cartesian grid. Because the two arrays returned contains variations along rows and along columns only, these arrays can be used in any function that requires to operate on two parameters.
In the following example a square cartesian grid on [-2,2] is formed.  numpy.meshgrid is used to create two arrays for varying x and varying y.  The two arrays are flattened (made into (N,)) and then a relatively complex function is applied to the data. After the function call the data is reshaped back to the original shape.  Note that the calculation over two dimensions is done without any for loops.
Can you write Python code for it?

import numpy as np
import pyradi.ryplot as ryplot
from matplotlib import cm

def myFunc(x,y):
  scale = np.sqrt(np.exp(-(x**2 +y**2)))
  return np.sin(2 * x) * np.cos(4 * y) * scale

x = np.linspace(-2, 2, 101)
y = np.linspace(-2, 2, 101)
varx, vary = np.meshgrid(x, y)
zdata = myFunc(varx.flatten(), vary.flatten()).reshape(varx.shape)

p = ryplot.Plotter(1,1,1,figsize=(10,6))
p.mesh3D(1, varx, vary, zdata, ptitle='Title', xlabel='x', ylabel='y', zlabel='z',
  rstride=3, cstride=3, linewidth= 0.3, maxNX=5, maxNY=5, maxNZ=0,
  drawGrid=True, cbarshow=True, alpha=0.5)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
The same applies to float variables
Can you write Python code for it?

a = 5.
b = a
print(a, type(a), id(a), a is b, a==b)
print(b, type(b), id(b), a is b, a==b)
b = 6.
print(a, type(a), id(a), a is b, a==b)
print(b, type(b), id(b), a is b, a==b)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
The switch/case Statement in Python
All of this comes from http://www.pydanny.com/why-doesnt-python-have-switch-case.html
Python does not have a switch or case statement. To get around this fact, we use dictionary mapping:
Can you write Python code for it?

def numbers_to_strings(argument):
    switcher = {
        0: "zero",
        1: "one",
        2: "two",
    }
    return switcher.get(argument, "nothing")

print(numbers_to_strings(1))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
There are a few different ways to list many files. The simplest is the .iterdir() method, which iterates over all files in the given directory. The following example combines .iterdir() with the collections.Counter class to count how many files there are of each filetype in the current directory.  This code does not recurse into lower level folders, nor is is very accurate with files or folders without extensions:  .git and .gitignore seem to be counted as  file/folder with no extension/suffix.
Can you write Python code for it?

import collections
collections.Counter(p.suffix for p in Path.cwd().iterdir())

collections.Counter(p.suffix for p in Path.cwd().iterdir() if 'git' in p.name)

collections.Counter(p.suffix for p in Path.cwd().iterdir() if 'ipynb_checkpoints' in p.name)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
This example shows how to construct a unique numbered file name based on a template. First, specify a pattern for the file name, with room for a counter. Then, check the existence of the file path created by joining a directory and the file name (with a value for the counter). If it already exists, increase the counter and try again
Can you write Python code for it?

def unique_path(directory, name_pattern):
    counter = 0
    while True:
        counter += 1
        path = directory / name_pattern.format(counter)
        if not path.exists():
            return path

path = unique_path(Path.cwd(), 'test{:03d}.txt')
print(path)
with path.open('w') as fout:
    fout.write('This is a test file\n')
    fout.write('Line 2\n')


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
To find the file in a directory that was last modified, you can use the .stat() method to get information about the underlying files. For instance, .stat().st_mtime gives the time of last modification of a file:
Can you write Python code for it?

from datetime import datetime
directory = Path.cwd()
print(directory)
times, file_path = max((f.stat().st_mtime, f) for f in directory.iterdir())
print(datetime.fromtimestamp(times), file_path)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Upright micro symbol
In Python 3 all strings are unicode. The upright mu symbol (as required by the SI system to denote 0.000001) is '\u00B5'
Can you write Python code for it?

##
if sys.version_info[0] > 2:
    print('\u00B5m')


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Variable id(), is and ==
http://blog.lerner.co.il/why-you-should-almost-never-use-is-in-python/
Every object (variable) has an ID, which can be determined using id(variable). This ID is a handle to the variable, not the value of the variable.
Assigning one variable to another does not create a new variable, the same ID is used, just with a second variable name.
The is keyword tests if the same ID is shared, whereas equality == tests the value of the variable.
In the example below a and b share the same ID handle and are hence the same variable storage location. c is a different storage location but with the same contents.
Always use == in logical tests to test for equality of value in the variable; if you use is you are testing the ID not the value.
You should only use is to test for ID or handle equality such as in is None or is not None.
Can you write Python code for it?

a = [5, 9]
b = a
c = [5, 9]
print(a, type(a), id(a), a is b, a==b)
print(b, type(b), id(b), a is b, a==b)
print(c, type(c), id(c), a is c, a==c)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Walking directories is not quite possible in pathlib, but glob may help a little.
If you want to recursively walk through all directories, use the **/ glob syntax. rglob returns a list of all the filenames as Path objects.  The [!*] syntax can be used to exclude portions of a file.
The  p.rglob('./data/**/*.[!png;!txt!ltn]*')  call returns Path objects in all sub directories below data, but excludes the files with the extensions shown.
glob and rglob are limited in the sense that these functions cannot support multiple file types.
Note that rglob and glob returns a generator that can be made into a list (if necessary) by
list(p.glob('./*.md'))
Can you write Python code for it?

p.rglob('./data/**/*.[!png;!txt!ltn]*')
print(list)

p = Path(cwd)
print(type(p.glob('**/cmd-here/*.*')))

for i in p.glob('**/cmd-here/*.*'):
    print(i.name)
print(' ')
print(list(p.rglob('./cmd-here/*.*')),end='\n\n')
print(list(p.rglob('./data/**/*.[!png;!txt!ltn]*')),end='\n\n')


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Watch out when using the n * construct with lists or dicts:
Can you write Python code for it?

print(5 *[None])

# the same single dict object repeated with 3 x references in the list
dic = 3 * [{}]
print(dic)
# assigning to the single dict once,
# but all entries in the list reference the same dict
dic[0]['test'] = 'mystring'
print(dic)

lst = 3 * [[]]
print(lst)
lst[0].append('mystring')
print(lst)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
You can build lists of list or tuples to any depth and access them by indexing the elements.  But be careful not to try and construct multi-dimensional arrays this way - numpy provides much better functionality.
Can you write Python code for it?


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
You can enumerate over the zipped list to extract the paired tuples
Can you write Python code for it?

print('Enumerate over all zipped tuples')
for i,(ai,bi) in enumerate(zip(a,b)):
    print(i, ai, bi)

print('Alternative method, not quite as clear and instructive')
for i,aa in enumerate(a):
    print( 1, a[i], b[i])

print('C-Style programming to achieve the same effect')
if len(a)==len(b):
    for i in range(len(b)):
        print(i, a[i],b[i])


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
Zipping pairs two lists element by element into a new list of matched tuples.
You can even enumerate over zipped lists:
Can you write Python code for it?


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
defaultdict allows you to assign to a dictionary[key] element even if the key'ed entry does not exist.
Can you write Python code for it?

# from collections import defaultdict
# dictionary = defaultdict(list) # defaults to list
# for k, v in ls:
#     dictionary[k].append(v)

a = {'x': 1, 'y':2, 'z':3}
b = {'y': 5, 's': 10, 'x': 3, 'z': 6}
c = a
c.update(b)
print(c)

aa = {k: sum(range(k)) for k in range(10)}
print(aa)
# get the max value and the key where this occurs
print(max(zip(aa.values(), aa.keys())))
# traverse dict from max value to min value, with key
print(sorted(zip(aa.values(), aa.keys()), reverse=True))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
http://stackoverflow.com/questions/2970608/what-are-named-tuples-in-python
namedtuples can be used similarly to struct or other common record types, except that they are immutable.
You should use named tuples instead of tuples anywhere you think object notation will make your code more pythonic and more easily readable. For example to represent very simple value types, particularly when passing them as parameters to functions. It makes the functions more readable, without seeing the context of the tuple packing.
Furthermore, you can also replace ordinary immutable classes that have no functions, only fields with them. You can even use your named tuple types as base classes:
class Point(namedtuple('Point', 'x y')):
    [...]
Can you write Python code for it?

from collections import namedtuple
Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)

from math import sqrt
line_length = sqrt((pt1.x-pt2.x)**2 + (pt1.y-pt2.y)**2)


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
https://docs.python.org/2/library/collections.html#collections.namedtuple
Named tuples are especially useful for assigning field names to result tuples returned by the csv or sqlite3 modules. Note that the table names and the name tuple member names are not the same:
Can you write Python code for it?

from collections import namedtuple

EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, address, paygrade')

import csv
for emp in map(EmployeeRecord._make, csv.reader(open('data/test.csv', "rt"))):
    print('from csv: {} {} '.format(emp.name, emp.paygrade))

import sqlite3
conn = sqlite3.connect('data/test.db3')
cursor = conn.cursor()
cursor.execute('SELECT NAME,AGE,ADDRESS,SALARY FROM COMPANY')
for emp in map(EmployeeRecord._make, cursor.fetchall()):
    print('from sqlite3: {} {} {} {}'.format(emp.name, emp.age, emp.address, emp.paygrade))
conn.close ;


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
https://stackoverflow.com/questions/19777612/python-range-and-zip-object-type
https://www.journaldev.com/15891/python-zip-function
In Python 2, the objects zip and range behave  as returning lists.
In Python 3 they were changed to generator-like objects which produce the elements on demand rather than expand an entire list into memory. One advantage was greater efficiency in their typical use-cases (e.g. iterating over them). To retrieve all the output at once into a familiar list object, you may simply call list() to iterate and consume the contents.
list()    Convert an iterable (tuple, string, set, dictionary) to a list. It seems that the change is done in-place.
We can also extract data from the Python zip function. To extract zip, we have to use the same zip() function. But we have add an asterisk(*) in front of the name of the zip object.
Can you write Python code for it?

data = [(1, 2, 3), ('a', 'b', 'c')]
print(data)
print('---------')
zipped = zip(*data)
print(zipped)
print(type(zipped))
print(list(zipped)) # ??? list(zipped) changes zipped to a list, no longer zipped object
print(type(list(zipped)))
print(type(zipped))
print('---------')
# zipped = zip(*data)
unzipped = zip(*zipped)
print(unzipped)
print(type(unzipped))
print(list(unzipped))


I am working on the file "02-PythonWhirlwindCheatSheet.ipynb".
The first task is:
import matplotlib as mpl

mpl.rcParams['text.latex.preamble'] = [
       r'\usepackage{siunitx}',   # i need upright \micro symbols, but you need...
       r'\sisetup{detect-all}',   # ...this to force siunitx to actually use your fonts
       r'\usepackage{helvet}',    # set the normal font here
       r'\usepackage{sansmath}',  # load up the sansmath so that math -> helvet
       r'\sansmath'               # <- tricky! -- gotta actually tell tex to use!
]
mpl.rcParams['text.usetex'] = True

p = ryplot.Plotter(1,1,2,figsize=(12,6))
p.plot(1,spec[:,0],spec[:,1],'Normalised responsivity','Wavelength \si{\micro\metre}','Responsivity')

Esoteric Python tidbits
http://nbviewer.ipython.org/github/rasbt/python_reference/blob/master/not_so_obvious_python_stuff.ipynb?create=1
http://sahandsaba.com/thirty-python-language-features-and-tricks-you-may-not-know.html
Python variables can have data types, see the link for more details.
Can you write Python code for it?

# a = int
# print(type(a))
# print a('3') + 4

# b = str
# print b(12) * 5

# class Foo:
#      def bar(self):
#          print "in Foo.bar()"

# f = Foo
# f().bar()

# Collect input and output arguments into one bunch
class Bunch(object):
    def __init__(self, **kwds): self.__dict__.update(kwds)

arg = Bunch(recurse=1, pattern_list='*.xml;*.py', return_folders=0, results=[])

print(arg.__dict__)


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
1.3 Find Elements
BeautifulSoup has a number of functions to find things on a page. Like other scraping tools, BeautifulSoup lets you find elements by their:

HTML tags
HTML Attributes
CSS Selectors

Let's search first for HTML tags.
The function find_all searches the soup tree to find all the elements with a particular HTML tag, and returns all of those elements.
What does the example below do?
Can you write Python code for it?


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
2. Collecting information

Believe it or not, that's all you need to scrape a website. Let's apply these skills to scrape the 98th general assembly.
Our goal is to scrape information on each senator, including their:
* name
* district
* party
2.1 First, make the GET request and soup it
Can you write Python code for it?

# make a GET request
response = requests.get('http://www.ilga.gov/senate/default.asp?GA=98')

# read the content of the server’s response
page_source = response.text

# soup it
soup = BeautifulSoup(page_source, "html5lib")


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
2.2 Find the right elements and text
Now let's try to get a list of rows in that table. Remember that rows are identified by the tr tag.
Can you write Python code for it?

# get all tr elements
rows = soup.find_all("tr")
print(len(rows))


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
2.3 Loop it all together
Challenge 4
Let's use a for loop to get 'em all! We'll start at the beginning with the request:
Can you write Python code for it?

# make a GET request
response = requests.get('http://www.ilga.gov/senate/default.asp?GA=98')

# read the content of the server’s response
page_source = response.text

# soup it
soup = BeautifulSoup(page_source, "html5lib")

# create empty list to store our data
members = []

# returns every ‘tr tr tr’ css selector in the page
rows = soup.select('tr tr tr')

# loop through all rows
for row in rows:

    # select only those 'td' tags with class 'detail'

    # get rid of junk rows

    # keep only the text in each of those cells

    # collect information

    # store in a tuple

    # append to list

print(len(members))
print()
print(members)


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
4. Export to CSV
We can write this to a CSV too:
Can you write Python code for it?

# manually decide on header names
header = ['Senator', 'District', 'Party', 'Bills Link', 'Description', 'Chamber', 'Last Action', 'Last Action Date']

with open('all-bills.csv', 'w') as output_file:
    csv_writer = csv.writer(output_file)
    csv_writer.writerow(header)
    csv_writer.writerows(bills_info)

pandas.read_csv('all-bills.csv')


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
It's a tag! Which means it has a text member:
Can you write Python code for it?


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
Most of the time, we're interested in the actual text of a website, not its tags. Remember, to get the text of an HTML element, use the text member.
Can you write Python code for it?

# Keep only the text in each of those cells
row_data = [cell.text for cell in detail_cells]
print(row_data)


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
NB: Because find_all() is the most popular method in the BeautifulSoup search library, you can use a shortcut for it. If you treat the BeautifulSoup object as though it were a function, then it’s the same as calling find_all() on that object.
Can you write Python code for it?


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
Nice, but that doesn't look like a full URL! Don't worry, we'll get to this soon.
Challenge 3
Find all the href attributes (url) from the mainmenu by writing a list comprehension and assign to it rel_paths.
Can you write Python code for it?


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
Oftentimes a more efficient way to search and find things on a website is by CSS selector. For this we have to use a different method, select(). Just pass a string into the .select() to get all elements with that string as a valid CSS selector.
In the example above, we can use "a.sidemenu" as a CSS selector, which returns all a tags with class sidemenu.
Can you write Python code for it?

# get elements with "a.sidemenu" CSS Selector.
soup.select("a.sidemenu")


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
That's a lot! Many elements on a page will have the same HTML tag. For instance, if you search for everything with the a tag, you're likely to get a lot of stuff, much of which you don't want. What if we wanted to search for HTML tags ONLY with certain attributes, like particular CSS classes?
We can do this by adding an additional argument to the find_all. In the example below, we are finding all the a tags, and then filtering those with class = "sidemenu".
Can you write Python code for it?

# get only the 'a' tags in 'sidemenu' class
soup("a", class_="sidemenu")


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
Using CSS is one way to organize how we stylize a website. They allow us to categorize and label certain HTML elements, and use these categories and labels to apply specfic styling. CSS selectors are what we use to identify these elements, and then decide what style to apply. We won't have time today to go into detail about HTML and CSS, but it's worth talking about the three most important CSS selectors:

element selector: simply including the element type, such as a above, will select all elements on the page of that element type. Try using your development tools (Chrome, Firefox, or Safari) to change all elements of the type a to a background color of red.

class selector: if you put a period (.) before the name of a class, all elements belonging to that class will be selected. Try using your development tools to change all elements of the class detail to a background color of red.

ID selector: if you put a hashtag (#) before the name of an id, all elements with that id will be selected. Try using the development tools to change all elements with the id Senate to a background color of red.

The above three examples will take all elements with the given property, but oftentimes you only want certain elements within the hierarchy. We can do that by simply placing elements side-by-side separated by a space.
Challenge 1
Using your developer tools, change the background-color of all a elements in only the "Current Senate Members" table.
Challenge 2
Find all the <a> elements in class mainmenu
Can you write Python code for it?


I am working on the file "02-Scraping/01-BS_workbook.ipynb".
The first task is:
You'll see there is some extra spacing here, we can use the strip method to remove that:
Can you write Python code for it?


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
4. Exporting data to CSV
We can then loop through all the combinations of the dropdown menu we want, collect the information from the generated table, and append it to the data list. Once we're done, we can write it to a CSV.
Can you write Python code for it?

header = data[0].keys()

with open('WBS-table.csv', 'w') as output_file:
    dict_writer = csv.DictWriter(output_file, header)
    dict_writer.writeheader()
    dict_writer.writerows(data)

pandas.read_csv('WBS-table.csv')


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
But the first row is the header so we don't want that.
Can you write Python code for it?


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
Each cell in the row corresponds to the data we want.
Can you write Python code for it?


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
First we'll get all the rows of the table using the tr selector.
Can you write Python code for it?

# get list of rows
rows = [row for row in table.select("tr")]


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
Now it's just a matter of looping through the rows and getting the information we want from each one.
Can you write Python code for it?

data = []
for row in rows:
    d = {}
    seat_names = row.select('td')[0].find_all("span")
    d['seat'] = ' '.join([x.text for x in seat_names])
    d['electors'] = row.select('td')[1].text.strip()
    d['polled'] = row.select('td')[2].text.strip()
    d['rejected'] = row.select('td')[3].text.strip()
    d['osn'] = row.select('td')[4].text.strip()
    d['candidate'] = row.select('td')[5].text.strip()
    d['party'] = row.select('td')[6].text.strip()
    d['secured'] = row.select('td')[7].text.strip()
    data.append(d)

print(data[1])


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
Now we can initialize the Selenium web driver, giving it the path to the Firefox binary code and the driver:
Can you write Python code for it?

# setup driver
driver = webdriver.Firefox(firefox_binary='./firefox/firefox', executable_path="./geckodriver")


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
Now we'll make a dictionary associating each name with its value.
Can you write Python code for it?

d_options = {option.text.strip(): option.get_attribute("value") for option in district_options if option.get_attribute("value").isdigit()}
print(d_options)


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
Once we selected the last dropdown menu parameter, the website automatically generate a table below. This table could not have been called up by a URL, as you can see that the URL in the browser did not change. This is why Selenium is so helpful.
3. Collecting generated data
Now that the table has been rendered, it exists as HTML in our page source. If we wanted to, we could send this to BeautifulSoup using the driver.page_source method to get the text. But we can also use Selenium's parsing methods.
First we'll identify it by its CSS selector, and then use the get_attribute method.
Can you write Python code for it?

soup = BeautifulSoup(driver.page_source, 'html5lib')

# get the html for the table
table = soup.select('#DataGrid1')[0]


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
We can then select a district by using its name and our dictionary. First we'll make our own function using Selenium's Select, and then we'll call it on "Bankura".
Can you write Python code for it?

district_select = Select(district)
district_select.select_by_value(d_options["Bankura"])


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
You can navigate Selenium to a URL by using the get method, exactly the same way we used the requests.get before:
Can you write Python code for it?


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
You should have seen the dropdown menu select 'Bankura' by running the previous cell.
Challenge
Using the code above as a guide, choose dropdown options for Panchayat Samity and Gram Panchayat:
Panchayat Samity
Can you write Python code for it?

# your code here

print(Select(driver.find_element_by_name("ddlblock")).first_selected_option.text)


I am working on the file "02-Scraping/02-Selenium_workbook.ipynb".
The first task is:
You'll notice that some of the information, such as total electors, is not supplied for each canddiate. This code will add that information for the candidates who don't have it.
Can you write Python code for it?

i = 0
while i < len(data):
    if data[i]['seat']:
        seat = data[i]['seat']
        electors = data[i]['electors']
        polled = data[i]['polled']
        rejected = data[i]['rejected']
        i = i+1
    else:
        data[i]['seat'] = seat
        data[i]['electors'] = electors
        data[i]['polled'] = polled
        data[i]['rejected'] = rejected
        i = i+1

data


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
1. Using BeautifulSoup

1.1 Make a GET request and parse the HTML response
We use the requests library just as we did with APIs, but this time we won't get JSON or XML back, but we'll get an HTML response.
Can you write Python code for it?

# make a GET request
response = requests.get('http://www.ilga.gov/senate/default.asp')

# read the content of the server’s response as a string
page_source = response.text
print(page_source[:1000])


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
1.3 Find Elements
BeautifulSoup has a number of functions to find things on a page. Like other scraping tools, BeautifulSoup lets you find elements by their:

HTML tags
HTML Attributes
CSS Selectors

Let's search first for HTML tags.
The function find_all searches the soup tree to find all the elements with a particular HTML tag, and returns all of those elements.
What does the example below do?
Can you write Python code for it?


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
1.4 Get Attributes and Text of Elements
Once we identify elements, we want to access information in that element. Oftentimes this means two things:

Text
Attributes

Getting the text inside an element is easy. All we have to do is use the text member of a tag object:
Can you write Python code for it?

# this is a list
soup.select("a.sidemenu")

# we first want to get an individual tag object
first_link = soup.select("a.sidemenu")[0]

# check out its class
print(type(first_link))


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
2. Collecting information

Believe it or not, that's all you need to scrape a website. Let's apply these skills to scrape the 98th general assembly.
Our goal is to scrape information on each senator, including their:
* name
* district
* party
2.1 First, make the GET request and soup it
Can you write Python code for it?

# make a GET request
response = requests.get('http://www.ilga.gov/senate/default.asp?GA=98')

# read the content of the server’s response
page_source = response.text

# soup it
soup = BeautifulSoup(page_source, "html5lib")


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
2.2 Find the right elements and text
Now let's try to get a list of rows in that table. Remember that rows are identified by the tr tag.
Can you write Python code for it?

# get all tr elements
rows = soup.find_all("tr")
print(len(rows))


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
4. Export to CSV
We can write this to a CSV too:
Can you write Python code for it?

# manually decide on header names
header = ['Senator', 'District', 'Party', 'Bills Link', 'Description', 'Chamber', 'Last Action', 'Last Action Date']

with open('all-bills.csv', 'w') as output_file:
    csv_writer = csv.writer(output_file)
    csv_writer.writerow(header)
    csv_writer.writerows(bills_info)

pandas.read_csv('all-bills.csv')


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
Cool! Now you can probably guess how to loop it all together by iterating through the links we just extracted.
3. Following links to scrape bills

3.1 Writing a scraper function
Now we want to scrape the webpages corresponding to bills sponsored by each senator.
Challenge 6
Write a function called get_bills(url) to parse a given bill's URL. This will involve:

requesting the URL using the requests library
using the features of the BeautifulSoup library to find all of the <td> elements with the class billlist
return a list of tuples, each with:
description (2nd column)
chamber (S or H) (3rd column)
the last action (4th column)
the last action date (5th column)

I've started the function for you. Fill in the rest.
Can you write Python code for it?

# your code here

def get_bills(url):

    # make the GET request
    response = requests.get(url)
    page_source = response.text
    soup = BeautifulSoup(page_source, "html5lib")

    # get the table rows
    rows = soup.select('tr tr tr')

    # make empty list to collect the info
    bills = []
    for row in rows:

        # get columns
        detail_cells = row.select('td.billlist')
        if len(detail_cells) is not 5:
            continue

        # get text in each column
        row_data = [cell.text for cell in row]

        # append data in columns 2-5
        bills.append(tuple(row_data[2:6]))

    return(bills)

# uncomment to test your code:
test_url = members[0][3]
print(test_url)
get_bills(test_url)[0:5]


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
Most of the time, we're interested in the actual text of a website, not its tags. Remember, to get the text of an HTML element, use the text member.
Can you write Python code for it?

# Keep only the text in each of those cells
row_data = [cell.text for cell in detail_cells]
print(row_data)


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
Now we can combine the BeautifulSoup tools with our basic python skills to scrape an entire web page.
Can you write Python code for it?

# check it out
print(row_data[0]) # name
print(row_data[3]) # district
print(row_data[4]) # party


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
Sometimes we want the value of certain attributes. This is particularly relevant for a tags, or links, where the href attribute tells us where the link goes.
You can access a tag’s attributes by treating the tag like a dictionary:
Can you write Python code for it?


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
That's a lot! Many elements on a page will have the same HTML tag. For instance, if you search for everything with the a tag, you're likely to get a lot of stuff, much of which you don't want. What if we wanted to search for HTML tags ONLY with certain attributes, like particular CSS classes?
We can do this by adding an additional argument to the find_all. In the example below, we are finding all the a tags, and then filtering those with class = "sidemenu".
Can you write Python code for it?

# get only the 'a' tags in 'sidemenu' class
soup("a", class_="sidemenu")


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
Using CSS is one way to organize how we stylize a website. They allow us to categorize and label certain HTML elements, and use these categories and labels to apply specfic styling. CSS selectors are what we use to identify these elements, and then decide what style to apply. We won't have time today to go into detail about HTML and CSS, but it's worth talking about the three most important CSS selectors:

element selector: simply including the element type, such as a above, will select all elements on the page of that element type. Try using your development tools (Chrome, Firefox, or Safari) to change all elements of the type a to a background color of red.

a {
    background-color: red
}

class selector: if you put a period (.) before the name of a class, all elements belonging to that class will be selected. Try using your development tools to change all elements of the class detail to a background color of red.

.detail {
    background-color: red
}

ID selector: if you put a hashtag (#) before the name of an id, all elements with that id will be selected. Try using the development tools to change all elements with the id Senate to a background color of red.

```
Senate {
background-color: red

}
```
The above three examples will take all elements with the given property, but oftentimes you only want certain elements within the hierarchy. We can do that by simply placing elements side-by-side separated by a space.
Challenge 1
Using your developer tools, change the background-color of all a elements in only the "Current Senate Members" table.
tr tr tr a {
    background-color: red
}
Challenge 2
Find all the <a> elements in class mainmenu
Can you write Python code for it?

# your code here
soup.select("a.mainmenu")


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
We can use the select method on anything. Let's say we want to find everything with the CSS selector td.detail in an item of the list we created above.
Can you write Python code for it?

# select only those 'td' tags with class 'detail'
row = rows[2]
detail_cells = row.select('td.detail')
detail_cells


I am working on the file "02-Scraping/solutions/01-BS_solutions.ipynb".
The first task is:
You'll see there is some extra spacing here, we can use the strip method to remove that:
Can you write Python code for it?


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
1. Launching the webdriver
Since we are in different environment and we can't use our regular graphical desktop, we need to tell Python to start a virutal display, onto which Selenium can project the Firefox web browser (though we won't actually see it).
Can you write Python code for it?

display = Display(visible=0, size=(1024, 768))
display.start()


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
4. Exporting data to CSV
We can then loop through all the combinations of the dropdown menu we want, collect the information from the generated table, and append it to the data list. Once we're done, we can write it to a CSV.
Can you write Python code for it?

header = data[0].keys()

with open('WBS-table.csv', 'w') as output_file:
    dict_writer = csv.DictWriter(output_file, header)
    dict_writer.writeheader()
    dict_writer.writerows(data)

pandas.read_csv('WBS-table.csv')


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
But the first row is the header so we don't want that.
Can you write Python code for it?


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
Each cell in the row corresponds to the data we want.
Can you write Python code for it?


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
First we'll get all the rows of the table using the tr selector.
Can you write Python code for it?

# get list of rows
rows = [row for row in table.select("tr")]


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
Great! One dropdown menu to go.
Gram Panchayat
Can you write Python code for it?

# get options
gp = driver.find_element_by_name("ddlgp")
gp_options = gp.find_elements_by_tag_name("option")

print(gp_options[1].get_attribute("value"))
print(gp_options[1].text)

gp_options = {option.text.strip(): option.get_attribute("value") for option in gp_options if option.get_attribute("value").isdigit()}
print(gp_options)

gram_select = Select(gp)
gram_select.select_by_value(gp_options["ANCHURI"])


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
Now if we want to get the different options in this drop down, we can do the same. You'll notice that each name is associated with a unique value. Since we're getting multiple elements here, we'll use find_elements_by_tag_name
Can you write Python code for it?

# find options in "disrict" drop down
district_options = district.find_elements_by_tag_name("option")

print(district_options[1].get_attribute("value"))
print(district_options[1].text)


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
Now it's just a matter of looping through the rows and getting the information we want from each one.
Can you write Python code for it?

data = []
for row in rows:
    d = {}
    seat_names = row.select('td')[0].find_all("span")
    d['seat'] = ' '.join([x.text for x in seat_names])
    d['electors'] = row.select('td')[1].text.strip()
    d['polled'] = row.select('td')[2].text.strip()
    d['rejected'] = row.select('td')[3].text.strip()
    d['osn'] = row.select('td')[4].text.strip()
    d['candidate'] = row.select('td')[5].text.strip()
    d['party'] = row.select('td')[6].text.strip()
    d['secured'] = row.select('td')[7].text.strip()
    data.append(d)

print(data[1])


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
Now we can initialize the Selenium web driver, giving it the path to the Firefox binary code and the driver:
Can you write Python code for it?

# setup driver
driver = webdriver.Firefox(firefox_binary='./firefox/firefox', executable_path="./geckodriver")


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
Selenium actually uses our web browser, and since the JupyterHub doesn't come with Firefox, we'll download the binaries:
Can you write Python code for it?

# download firefox binaries
!wget http://ftp.mozilla.org/pub/firefox/releases/54.0/linux-x86_64/en-US/firefox-54.0.tar.bz2

# untar binaries
!tar xvjf firefox-54.0.tar.bz2


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
We can then select a district by using its name and our dictionary. First we'll make our own function using Selenium's Select, and then we'll call it on "Bankura".
Can you write Python code for it?

district_select = Select(district)
district_select.select_by_value(d_options["Bankura"])


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
You'll notice that some of the information, such as total electors, is not supplied for each canddiate. This code will add that information for the candidates who don't have it.
Can you write Python code for it?

i = 0
while i < len(data):
    if data[i]['seat']:
        seat = data[i]['seat']
        electors = data[i]['electors']
        polled = data[i]['polled']
        rejected = data[i]['rejected']
        i = i+1
    else:
        data[i]['seat'] = seat
        data[i]['electors'] = electors
        data[i]['polled'] = polled
        data[i]['rejected'] = rejected
        i = i+1

data


I am working on the file "02-Scraping/solutions/02-Selenium_solutions.ipynb".
The first task is:
Zilla Parishad
Similar to BeautifulSoup, Selenium has methods to find elements on a webpage. We can use the method find_element_by_name to find an element on the page by its name.
Can you write Python code for it?

# find "district" drop down menu
district = driver.find_element_by_name("ddldistrict")

district


I am working on the file "02-ShapeFunctions.ipynb".
The first task is:
For a $10$th order polynomial:
Can you write Python code for it?


I am working on the file "02-ShapeFunctions.ipynb".
The first task is:
For a $5$th order polynomial:
Can you write Python code for it?


I am working on the file "02-ShapeFunctions.ipynb".
The first task is:
For the $10$th order polynomial, note the oscillations near the ends of the element. This is known as Runge's phenomena when interpolating points with a polynomial. This element, with equally spaces nodes, wouldn't be recommended in a simulation.
Hermitian shape functions
The below function plots Hermitian shape functions of arbitrary degree.
Can you write Python code for it?

def row(x, n):
    "For a point x, compute the 'N' and 'M' rows in the interpolation matrix for degree n"
    a = np.zeros((2, n + 1))
    for i in range(n + 1):
        a[0, i] = np.power(x, i)
    for i in range(1, n + 1):
        a[1, i] = i*np.power(x, i - 1)
    return a

# Element nodes
num_nodes = 3
x_n = np.linspace(-1.0, 1.0, num_nodes)
plt.plot(x_n, np.zeros(len(x_n)), '-o', color='k');

# Polynomial degree
degree = 2*num_nodes -1

# Build Vandermonde matrix
V = np.zeros((0, 2*len(x_n)))
#V = None
for i in range(len(x_n)):
    V = np.vstack((V, row(x_n[i], degree)))

# Points for plotting
x = np.linspace(-1.0, 1.0, 200)

# Solve Vandermonde matrix while cycling through RHS, and plot
for i in range(2*len(x_n)):
    b = np.zeros(2*len(x_n))
    b[i] = 1.0

    # Solve for polynomial coefficients
    c = np.linalg.solve(V, b)

    # Reverse order of coefficients since NumPy polyn1d functions wants coefficients
    # in order [x^4, x^3, . . . 1], etc.
    c = c[::-1]

    # Create polynimial for plotting
    p = np.poly1d(c)
    N = p(x)
    plt.plot(x, N);


I am working on the file "02-ShapeFunctions.ipynb".
The first task is:
Lagrange polynomials in 1D
We start with the exmaple of a cubic finite element basis, and then develop a function for plotting shape functions of any order. In all cases we consider the interval $(-1, 1)$.
Cubic basis
Cubic shape functions will have the form:
$$
N_{i}(x) = c_{0} + c_{1} x + c_{2} x^{2} + c_{2}x^{3}
$$
Recall the shape function $N_{i}$ should be equal to one at its own node ($N_{i}(x_{i}) = 1$) and zero at all other nodes ($N_{i}(x_{i}) = 0$ when  $i \ne j$).
The cubic function has four coefficients, so we need four nodes. First step is to create the nodes on the interval $(-1, 1)$. We will consider equally spaced nodes, in which case we can use the linspace function:
Can you write Python code for it?

x_n = np.linspace(-1.0, 1.0, 4)
print(x_n)


I am working on the file "02-ShapeFunctions.ipynb".
The first task is:
We can now compute the four shape functions by solving $\boldsymbol{A} \boldsymbol{c}{i} = \boldsymbol{f}{i}$ to get the polynomial coefficients $\boldsymbol{c}{i}$ for the shape function $N{i}$. For node $i$, $f_{j=1} = 1$ if $i=j$ and $f_{j} = 0$ if $i \ne j$. We use a loop to compute the four shape functions at once:
Can you write Python code for it?

shape_functions = []
for i in range(4):
    f = np.zeros(4)
    f[i] = 1.0
    c = np.linalg.solve(A, f)

    shape_functions.append(np.poly1d(c))
    print("-Shape function for node {}: \n{}".format(i, shape_functions[-1]))


I am working on the file "02-approximate-retrieval.ipynb".
The first task is:
Scale remains problematic; we can't just do a double loop over all $N$ elements...
Hashing works well for exact duplicates, can it work with near duplicates?
Yes, we have locality sensitive hashing (LSH)

Min-hashing
Reorder shingle matrix rows with random permutation $\pi$
$\operatorname{hash}(C) =$ minimum row number in which permuted column contains a one (C represents a column, i.e. a document in shingle form)
$\operatorname{hash}(C) = h_\pi(C) = \min_{i:C(i)=1}\pi(i)$ (hence the min in min-hashing)
Turns out that the probability of two documents sharing a hash is equal to their Jaccard similarity: $P[h(C_1) = h(C_2)] = Sim(C_1, C_2)$ (trivial but interesting proof; see slides).
This means we can use many hash functions, see how often they clash for a pair of documents, and we have a decent estimate for the documents' Jaccard similarity.
An alternative to min-hashing is sim-hashing (see Information Retrieval)
Can you write Python code for it?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

shingle_matrix = np.array([
    [1, 0, 1, 0],
    [1, 0, 0, 1],
    [0, 1, 0, 1],
    [0, 1, 0, 1],
    [0, 1, 0, 1],
    [1, 0, 1, 0],
    [1, 0, 1, 0]
])


I am working on the file "02-approximate-retrieval.ipynb".
The first task is:
Scaling up min-hash
While the above method works well, it's quite problematic to actually permute a huge dataset
We therefore want to implement the permutations as linear hash funtions (see your local linear algebra course for more info about why this works!).
\begin{equation}
    \pi(r) = a \cdot r + b \operatorname{mod} n =: h_{a,b}(r), > a, b > \text{random}
\end{equation}
This is huge! It allows us to iteratively process any shingle matrix (row-wise), without having to actually move (huge) data around (necessary for performing actual permutations).
Moreover, $h_{a,b}$ is efficient to store.
Note: on slide 17, try to really figure out the prof's corrections, if time allows it.
Increasing the accuracy

More hash functions, any collision $\implies$ similar leads to fewer false negatives, but more false positives
Can you write Python code for it?

index = np.arange(0, 1, 0.01)
curves = {'k = %d' % k: 1 - (1 - index) ** k for k in range(1, 20, 2)}
fr = pd.DataFrame(data=curves, index=pd.Index(index))

ax = fr.plot()
plt.xlabel("Similarity")
plt.ylabel("P(hit)")


I am working on the file "02-basics/04-structuring_code.ipynb".
The first task is:
Absolute and relative imports
A more valid solution to avoid the module/package collisions is to import the modules using absolute (or relative, although absolute are prefeered) paths.
This is an "absolute" import:
Can you write Python code for it?


I am working on the file "02-basics/04-structuring_code.ipynb".
The first task is:
For this example, we can see that sys usually is implemented (in C) as a built-in module.
Next, the interpreter checks sys.path:
Can you write Python code for it?


I am working on the file "02-basics/04-structuring_code.ipynb".
The first task is:
Module/package structures are accesible from the invoker code using the notation: <module/package_name>.<structure>. Examples:
Can you write Python code for it?


I am working on the file "02-basics/04-structuring_code.ipynb".
The first task is:
Notice also that the hash bang line is considered as a comment by the Python interpreter.
More about importing
Python's import search for the packages through (in this order):
1. The built-in Python modules of the Python’s Standard Library (PSL), which are compiled directly into the Python interpreter.
2. The list of paths returned by sys.path.
3. The rest of modules of the PSL.
The built-in modules can be found by:
Can you write Python code for it?


I am working on the file "02-basics/04-structuring_code.ipynb".
The first task is:
Notice that in this case the module does not know its name.
Can you write Python code for it?

!python my_package # Running a Python package as an script


I am working on the file "02-basics/04-structuring_code.ipynb".
The first task is:
Notice that the file __main__py has ben run and again, the name of the module is the relative dotted-path (the modules has been invoked by __main__.py).
Using the hash bang available from the Shell
In Bash, the first line of a shell script can be used to specify the script interpreter that should be used to process the rest of the script file. Therefore, if we place in the first line of the script:
```
!/usr/bin/env python
```
the rest of the script will be run by the Python interpreter. Example (notice that the permissions of the file allows to run it directly):
Can you write Python code for it?


I am working on the file "02-basics/04-structuring_code.ipynb".
The first task is:
Only the local modules:
Can you write Python code for it?

import pkgutil
search_path = ['.']
[x[1] for x in pkgutil.iter_modules(path=search_path)]


I am working on the file "02-basics/04-structuring_code.ipynb".
The first task is:
Sumarizing: import loads and runs the code of a module, and loads and runs the code of the __init__.py file of a package.
Using the Python interpreter
Can you write Python code for it?

!python -m my_module # Running a Python module as an script


I am working on the file "02-basics/04-structuring_code.ipynb".
The first task is:
This is an "relative" import:
Can you write Python code for it?

from my_package import a_module
a_module.a


I am working on the file "02-basics/04-structuring_code.ipynb".
The first task is:
Which is populated with (and in this order):
1. The directory containing the input script (or the current directory when no file is specified).
2. The environment variable PYTHONPATH (as a list of directory names, with the same syntax as the shell variable PATH).
3. The installation-dependent defaults.
Therefore, if we create a module in our current directory with the same name that a module of the PSL, the interpreter will use our module and it will not be way of using the module of the PSL. On the contrary, if we create a module with the same name that a built-in module, the interpreter will ignore our module. To do the things more complicated, sys.path can be modified in running time, making more difficult to figure out which module will be imported when there is an overlaping between module names.
A way of overcoming this is to name our modules in such a way that they cannot overwrite a PSL module. To do this, we need a way of cheking the current list of importable modules. Notice however, that this is not an definitive solution (in the future, the PSL could implement a module with the name we have selected).
Getting the list of importable modules
All available modules:
Can you write Python code for it?

import pkgutil
search_path = ['.'] # set to None to see all modules importable from sys.path
search_path = None  # Set to
[x[1] for x in pkgutil.iter_modules(path=search_path)]


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
3 [0] Using parameters
Can you write Python code for it?

def max(a, b):
    if a > b:
        return a
    else:
        return b

max(1,2)


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
3.4 [1] Working with an undefined number of unnamed parameters
Can you write Python code for it?

def variable_args(*vargs):
    print(vargs)
    print(type(vargs))
    for i in vargs:
        print(i)
    print('first argument =', vargs[0])

variable_args("hola", "caracola", ("hi", "folks"))


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
3.6 [1] Functions can be arguments to functions
Can you write Python code for it?

def add(x, y):
    return x + y

def compute(function, x, y):
    return function(x, y)

compute(add, 1, 2)


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
3.7 [1] Of course, functions are objects!
Functions are objects which can, for example, be "copied" to other objects:
Can you write Python code for it?

print(type(say_hello))
a = say_hello
a()

id(say_hello)

id(a)


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
5 [1] Nesting functions
Can you write Python code for it?

def outter():
    def inner():
        print('Hello world')
    inner()

outter()


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
6 [1] Decorating functions
Extending the behavior of functions that we don't want (or cannot) to modify:
Can you write Python code for it?

def divide(numerator, denominator):
    return numerator/denominator

def safe_division(function):
    def wrapper(numerator, denominator):
        if denominator != 0:
            return function(numerator, denominator)
    return wrapper

# Function "decoration".
divide = safe_division(divide)

print(divide(1,2))
print(divide(1,0))

None == 0


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
7 [1] Lambda functions
$\lambda$-funcions are "anonymous" functions:
Can you write Python code for it?

# Standard function:
def power(x,y):
    return x**y

power(2,3)

# Lambda function:
power = lambda x,y: x**y

power(2,3)


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
8 [0] Namespaces and scopes
8.1 [0] Names
Python objects are referenced by names:
Can you write Python code for it?

id(1) # The `id()` funtion returns the address of a object

a = 1
id(a) # "a" is the same object than "1"


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
8.4 [0] When there is a coincidence, local scopes create their own names
Can you write Python code for it?

a = 1
def func():
    a = 2
    print(a) # This is the internal 'a'
func()
print(a)


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
8.5 [1] To avoid creating a new name when there is a coincidence, use global
However, this is DISCOURAGED. It is confusing and isn’t thread-safe (among other things).
Can you write Python code for it?

a = 1
def func():
    global a
    a = 2 # This is the external 'a'
func()
print(a)


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
A namespace is a collection of names. All packages, modules, classes and functions define their own namespace (basically, the variables -- or names -- locally defined). The scope of a name is the region of code where that name can be referenced without using any prefix.
8.2 [0] Functions create their own namespace
(Please, restart this kernel) Do you think that the second print(a) of in the following code should work?
Can you write Python code for it?

def func():
    a = 1
    print(a)
func()
print(a)


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
Different objects have different id's:
Can you write Python code for it?


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
In other words ..., filter() creates a sequences of elements for which a function returns true.
Can you write Python code for it?

for i in filter(lambda x: x < 0, range(-3,3)):
    print (i)

tuple(filter(lambda x: x%2, range(5*2)))


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
In other words ..., map() applies a $\lambda$-function to all the items in an input sequence.
Can you write Python code for it?


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
In other words ..., reduce() process iteratively a sequence of items. One of them is item of the sequence and the other, the previus output of the reduce().
Can you write Python code for it?

n=10; list(range(1, n+1))

def factorial(n):
    return reduce(lambda x,y: x*y, list(range(1, n+1)))
#                 ---------------  ------------------
#                    function           sequence

print(factorial(3))


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
It's better to do:
Can you write Python code for it?

a = 1
def func(a):
    a = 2
    return a
a = func(a)
print(a)


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
Lambda functions are useful because they can be defined inline and we don't need give a name to use them:
Can you write Python code for it?


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
Now using $\lambda$-functions:
Can you write Python code for it?

# A different implementation of the Sieve of Eratosthenes
# (http://stackoverflow.com/questions/27990094/finding-primes-with-modulo-in-python)
primes = list(range(2, N))
print(primes)
for i in range(2, int(math.sqrt(N))):
    primes = list(filter(lambda x: x == i or x % i, primes))
    print(primes, i)

# This code produces a list


I am working on the file "02-basics/07-functions.ipynb".
The first task is:
Only to see the performance:
Can you write Python code for it?

import math
print(math.factorial(3))

now = time.time()
factorial(100000)
print ("Time =", time.time() - now)

now = time.time()
math.factorial(100000)
print ("Time =", time.time() - now)


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
2. Defining (instantiating) an object
Can you write Python code for it?

x = A()
print(isinstance(x,A))

s = "hola"
isinstance(s, str)


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
3. Class check
To know the class of an object:
Can you write Python code for it?


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
4. Defining class attributes
Class attributes (variables and methods) are accesible even if the class is not instantiated.
4.1 Class variables
Because class variables do no depends on any instance, they can be used, for example, to share data between instances:
Can you write Python code for it?

class A:
    class_variable = 1

A.class_variable

A.class_variable = 2
A.class_variable

a = A()
a.class_variable

b = A()
b.class_variable


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
4.2 Class methods
By default, (instance) methods are created when the class is instantiated. On the other hand, class methods are created when the class is defined. In order to distinguish them, class methods must be decorated with @classmethod or @staticmethod. The only difference between both types of methods is that in a @classmethod, the name of the class is passed automatically by the interpreter as the first argument.
Can you write Python code for it?

class A:
    def instance_method(self, x):
        print('executing instance method with args (%s,%s)' % (self, x))

    @classmethod
    def class_method(cls, x):
        print('executing class method class_method with args (%s,%s)' % (cls, x))

    @staticmethod
    def static_method(x):
        print('executing static (class) method static_method with arg (%s)' % x)

A.instance_method(1)

A().instance_method(1)

A.class_method(1)

A.static_method(1)


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
4.2 Defining instance attributes
Instance attributes (methods and variables) are only accesible when the class has been instantiated. Of the instance methods, one of the most important is the constructor of the class. All instance methods must include a parameter which holds the data instantiated. By convention, this parameter is called self.
Can you write Python code for it?

class A:
    def __init__(self):
        print('Constructor called')

    def setter(self, x=1):
        self.x = x

    def getter(self):
        return self.x
a = A()

a.setter(2)
print(a.getter())

hasattr(A, 'setter')


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
5. Inheritance
Extends functionality of a (previously defined) class.
5.1 Simple inheritance
Can you write Python code for it?

class Class_A:
    '''Base class A'''

    def method_1(self):
        self.x = 1
        print('Class_A.method_1 called')

class Class_B(Class_A): # Notice that Class_B depends on (inherits from) Class_A
    '''Derived class B'''

    def method_1(self):
        print('Class_B.method_1 called')

    # This method extends "Class_A"
    def method_2(self):
        Class_A.x = 2 # "Class_A" is the prefix of "x" in "Class_B" for "Class_A.x"
        print('Class_B.method_2 called')
        Class_A.method_1(self)
        super().method_1() # This is identical to the previous
        self.method_1()

Class_A().method_1()

Class_B().method_2()

Class_B().method_1()

print(Class_A.x)


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
5.2 Multiple inheritance
Can you write Python code for it?

class Class_A:
    '''Base class A'''
    def method_1(self):
        print('Class_A.method_1 called')

class Class_B:
    '''Base class B'''
    def method_1(self):
        print('Class_B.method_1 called')

class Class_C(Class_A, Class_B): # Class_C inherits from Class_A and Class_B
    '''Derived class C'''
    def method_1(self):
        Class_A.method_1(self)
        Class_B.method_1(self)

Class_C().method_1()


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
6. Operator overloading
Can you write Python code for it?

class Point():
    def __init__(self, x, y):
        self.x = x
        self.y = y
    def __add__(self, point):
        '''Overloads "+"'''
        return Point(self.x + point.x, self.y + point.y)
    def __str__(self):
        '''Overloads print()'''
        return '(' + str(self.x) + ',' + str(self.y) + ')'

a = Point(1,2)
b = Point(3,4)
print(a+b)


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
7. "Abstract" methods
Python does not define a special syntax for declaring methods that should be implemented in a child class. However, an "abstract" method can be implemented in the "abstract" class as:
Can you write Python code for it?

class A():
    def do(self):
        raise NotImplementedError

x = A()
x.do()

class B(A):
    def do(self):
        print('Do something useful')

x = B()
x.do()


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
A curiosity:
Objects store the instance variables in a dictionary:
Can you write Python code for it?


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
Example
Can you write Python code for it?

class Stack:
     def __init__(self):
         self.items = []

     def put(self, item):
         self.items.append(item)

     def get(self):
         return self.items.pop()

     def size(self):
         return len(self.items)

class Queue:
    def __init__(self):
        self.items = []

    def put(self, item):
        self.items.insert(0,item)

    def get(self):
        return self.items.pop()

    def size(self):
        return len(self.items)


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
Remember! In Python everything is an object:
Can you write Python code for it?


I am working on the file "02-basics/09-OOP.ipynb".
The first task is:
We can create a new entry in the (class) dictionary (a new instance variable) with:
Can you write Python code for it?


I am working on the file "02-basics/10-exceptions.ipynb".
The first task is:
1.2 Defining alternatives with else
The else statement solves the previous problem (run unwanted code).
Can you write Python code for it?

try:
    text = input('Please, enter something: ')
except:
    print('Sorry, something wrong happened :-(')
else:
    # Now this statement is executed only if you provided an input
    print('You entered "{}".'.format(text))


I am working on the file "02-basics/10-exceptions.ipynb".
The first task is:
1.3 Finally, the finally code is executed yes or yes
No matter what happend in the rest of sections of the try statement (even if an exception is thrown in these sections).
Can you write Python code for it?

try:
    text = input('Please, enter something: ')
except:
    print('Sorry, something wrong happened :-(')
else:
    print('You entered "{}".'.format(text))
finally:
    # This will always executed, with exception or not.
    print('Thanks for your interaction!')


I am working on the file "02-basics/10-exceptions.ipynb".
The first task is:
1.4 Discriminating exceptions
The except statement accepts the type of exception as an argument. This is suitable to refine the exception handling.
Can you write Python code for it?

try:
    text = input('Please, enter something: ')
except EOFError: # Exception specific for input()
    print('Sorry, you didn\'t enter anything (<ctrl>+d) :-(')
except KeyboardInterrupt: # Exception raised when a program is interrupted
    print('Sorry, you cancelled the input (<ctrl>+c) :-(')
else:
    print('You entered "{}".'.format(text))
finally:
    print('Thanks for your interaction!')


I am working on the file "02-basics/10-exceptions.ipynb".
The first task is:
1.5 Handling several exceptions in the same way
Can you write Python code for it?

try:
    x = 1/0
except (ArithmeticError, ZeroDivisionError):
    print('Some arithmetic issue has been arised :-/')


I am working on the file "02-basics/10-exceptions.ipynb".
The first task is:
3. Creating (new type of) exceptions
Exceptions can be created, for example, to increase the functionality of an existing one. All exceptions must be derived (directly or indirectly) from the Exception class.
Can you write Python code for it?

class SmallStack_full(Exception):
    pass

class SmallStack_empty(Exception):
    pass

class SmallStack():
    '''A stack structure with 10 slots.'''

    def __init__(self):
        '''Create the stack.'''
        self.stack = [None]*10
        self.counter = 0

    def push(self, x)->None:
        '''Put "x" on the stack.

        Raises SmallStackFull upon fullness.
        '''
        if self.counter < 10:
            self.stack[self.counter] = x
            self.counter += 1
        else:
            raise SmallStack_full

    def pop(self)->object:
        '''Remove the last element inserter in the stack.

        Raises SmallStackEmpty upon emptyness.
        '''
        if self.counter > 0:
            self.counter -= 1
            return self.stack[self.counter]
        else:
            raise SmallStack_empty

s = SmallStack()

try:
    for i in range(100):
        s.push(i)
        print(i)
except SmallStack_full:
    print('The stack is full. i={}'.format(i))

try:
    for i in range(100):
        print(i, s.pop())
except SmallStack_empty:
    print('The stack is empty. i={}'.format(i))


I am working on the file "02-basics/11-iterators.ipynb".
The first task is:
Be aware!
Iterators spend memory.
Can you write Python code for it?

list_of_squares = [x*x for x in range(3)]

list_of_squares


I am working on the file "02-basics/11-iterators.ipynb".
The first task is:
Iterators are commonly used in for structures:
Can you write Python code for it?

for i in yrange(10):
    print(i, end=' ')


I am working on the file "02-basics/11-iterators.ipynb".
The first task is:
Strings also support the iterator protocol:
Can you write Python code for it?


I am working on the file "02-basics/12-generators.ipynb".
The first task is:
6.  Generators and Coroutines
Coroutines can be classified as generators that consume data (and, as expected, generate some data). Equivalently, generators (and generator expression) can be considered as coroutines that does not consume data.
Can you write Python code for it?

def minimize():
    current = yield
    while True:
        value = yield current  # Receives "value" and returns "current"
        current = min(value, current)

it = minimize()
next(it)  # "Prime" the coroutine (neccesary to reach the second yield).
print(it.send(10))
print(it.send(4))
print(it.send(22))
print(it.send(-1))


I am working on the file "02-basics/12-generators.ipynb".
The first task is:
Default behaviour
When we iterate over a (function) generator or a generator expression, the next() function is automatically invoked.
Can you write Python code for it?

powers_of_myself_generator_expression = (x**x for x in range(10))

for i in powers_of_myself_generator_expression:
    print(i)

def generate_powers_of_myself(exp):
    for x in exp:
        yield x**x
for i in generate_powers_of_myself(iter(range(10))):
    print(i)


I am working on the file "02-basics/12-generators.ipynb".
The first task is:
Generator expressions
Generator expressions can be considered as memory efficient generalization of list comprehensions, and also as compact representations of simple generators.
Can you write Python code for it?

list_comprehension = [x*x for x in range(100)]
print(list_comprehension)

import sys
sys.getsizeof(list_comprehension)

generator_expression = (x*x for x in range(100))
generator_expression

sys.getsizeof(generator_expression)


I am working on the file "02-basics/12-generators.ipynb".
The first task is:
Generator expressions can be used for defining loops
Can you write Python code for it?

for x in (i*2 for i in range(10)):
    print(x, end=' ')


I am working on the file "02-basics/12-generators.ipynb".
The first task is:
Nesting generator expressions
Can you write Python code for it?

import time
c = 0
now = time.time()
# Notice that this is a memoryless process whilst list compressions produce lists.
for i in (x for x in range(2, 2000) if all(x % y != 0 for y in range(2, int(x ** 0.5) + 1))):
    c += 1
    print(i, end=' ')
print('\n{} primes found in {} seconds'.format(c,time.time() - now))


I am working on the file "02-basics/14-commenting_code.ipynb".
The first task is:
2. Using help()
Getting help for a whole class.
Can you write Python code for it?


I am working on the file "02-basics/14-commenting_code.ipynb".
The first task is:
A different way of getting help (in Ipython):
Can you write Python code for it?


I am working on the file "02-basics/14-commenting_code.ipynb".
The first task is:
Getting help for a single member function:
Can you write Python code for it?


I am working on the file "02-basics/14-commenting_code.ipynb".
The first task is:
The same queries can be carried out with an instance.
Can you write Python code for it?

a = ClassExample()

help(a)

help(a.set)

print(a.__doc__)

print(a.get.__doc__)