I am working on the file "ListaEX_05.ipynb".
The first task is:
Exercício 06: Mostre o elemento com a menor distância de X e qual é essa distância.
Can you write Python code for it?

# encontre o ponto mais próximo do exercicio anterior


I am working on the file "ListaEX_05.ipynb".
The first task is:
Exercício 07:  Esse é o Mario, personagem famoso da Nintendo:
Can you write Python code for it?

# antes de tentar esse exercício, digite no prompt de comando: conda install scikit-image e reinicie o notebook
%matplotlib inline
import matplotlib.pyplot as plt
from skimage import data, io, filters

mario = io.imread('mario.gif')
plt.figure(figsize=(4, 4))
plt.imshow(mario)
plt.axis('on')
plt.grid(True)
plt.show()


I am working on the file "ListaExercicios.ipynb".
The first task is:
Exercício 01: Calcule a distância média, o diâmetro e o coeficiente de agrupamento das redes abaixo.
Can you write Python code for it?

G1 = nx.erdos_renyi_graph(10,0.4)
nx.draw_shell(G1)

G2 = nx.barabasi_albert_graph(10,3)
nx.draw_shell(G2)

G3 = nx.barabasi_albert_graph(10,4)
nx.draw_shell(G3)


I am working on the file "ListaExercicios.ipynb".
The first task is:
Exercício 02: Calcule a centralidade de grau, betweenness e pagerank dos nós das redes abaixo:
Can you write Python code for it?

G4 = nx.barabasi_albert_graph(10,3)
plt.pyplot.figure(figsize=(10,10))
pos = nx.shell_layout(G4)
nx.draw_networkx_nodes(G4,pos);
nx.draw_networkx_edges(G4,pos);
nx.draw_networkx_labels(G4,pos);
plt.pyplot.axis('off')


I am working on the file "Lists.ipynb".
The first task is:
Basic List Methods
If you are familiar with another programming language, you might start to draw parallels between arrays in another language and lists in Python. Lists in Python however, tend to be more flexible than arrays in other languages for a two good reasons: they have no fixed size (meaning we don't have to specify how big a list will be), and they have no fixed type constraint (like we've seen above).
Let's go ahead and explore some more special methods for lists:
Can you write Python code for it?


I am working on the file "Lists.ipynb".
The first task is:
Example 2
Lists can also hold multiple object types
Can you write Python code for it?

list1 = [1,'a',"This is a list",5.25]
print(list1)


I am working on the file "Lists.ipynb".
The first task is:
Example 4 - Slicing & Indexing
Can you write Python code for it?

# Get the element using the index
print(list1[0])
print(list1[2])

# Grab index 1 and everything after it
print(list1[1:])

# Grab the element from index position 1 to 3 (1 less than given)
print(list1[1:3])

# Grab elements upto 3rd item
print(list1[:3])

# Grab the last item in the list
print(list1[-1])

print(list1[-1:])

# Third parameter is the jump parameter
list2 = ['a','b','c','d','e','f','g']
list2[1:4:2]

# Concatenate Elements
list2 + ["added"]

# unless reassigned the added item is not permanently added
list2

list2 = list2 + ["added permanently"]
list2

list3 = list2 * 2
list3


I am working on the file "Lists.ipynb".
The first task is:
Indexing and Slicing
Indexing and slicing works just like in strings. Let's make a new list to remind ourselves of how this works:
Can you write Python code for it?

my_list = ['one','two','three',4,5]

# Grab element at index 0
my_list[0]

# Grab index 1 and everything past it
my_list[1:]

# Grab everything UP TO index 3
my_list[:3]


I am working on the file "Lists.ipynb".
The first task is:
It should also be noted that lists indexing will return an error if there is no element at that index. For example:
Can you write Python code for it?


I am working on the file "Lists.ipynb".
The first task is:
Just like strings, the len() function will tell you how many items are in the sequence of the list.
Can you write Python code for it?


I am working on the file "Lists.ipynb".
The first task is:
List Comprehensions
Python has an advanced feature called list comprehensions. They allow for quick construction of lists. To fully understand list comprehensions we need to understand for loops. So don't worry if you don't completely understand this section, and feel free to just skip it since we will return to this topic later.
But in case you want to know now, here are a few examples!
Can you write Python code for it?

# Build a list comprehension by deconstructing a for loop within a []
first_col = [row[0] for row in matrix]

first_col


I am working on the file "Lists.ipynb".
The first task is:
Now we can again use indexing to grab elements, but now there are two levels for the index. The items in the matrix object, and then the items inside that list!
Can you write Python code for it?

# Grab first item in matrix object
matrix[0]

# Grab first item of the first item in the matrix object
matrix[0][0]


I am working on the file "Lists.ipynb".
The first task is:
Use pop to "pop off" an item from the list. By default pop takes off the last index, but you can also specify which index to pop off. Let's see an example:
Can you write Python code for it?

# Pop off the 0 indexed item
l.pop(0)

# Show
l

# Assign the popped element, remember default popped index is -1
popped_item = l.pop()

popped_item

# Show remaining list
l


I am working on the file "Lists.ipynb".
The first task is:
We can also use + to concatenate lists, just like we did for strings.
Can you write Python code for it?


I am working on the file "Lists.ipynb".
The first task is:
We can also use the * for a duplication method similar to strings:
Can you write Python code for it?

# Make the list double
my_list * 2

# Again doubling not permanent
my_list


I am working on the file "Lists.ipynb".
The first task is:
We can use the sort method and the reverse methods to also effect your lists:
Can you write Python code for it?

new_list = ['a','e','x','b','c']

#Show
new_list

# Use reverse to reverse order (this is permanent!)
new_list.reverse()

new_list

# Use sort to sort the list (in this case alphabetical order, but for numbers it will go ascending)
new_list.sort()

new_list


I am working on the file "LiterateProgramming.ipynb".
The first task is:
File magic
Instead of writing SConstruct as one file, we are going to break this file into different parts and include them in the IPython notebook using %%file magic. Each part will be a file with .scons suffix.
In the first part, we are downloading input data (a short gather) and converting it to RSF format.
Can you write Python code for it?

%%file data.scons

# Download data
#Fetch('wz.25.H','wz')

# Convert and window
Flow('data','/Users/sergey/geo/fomels/cise/tpow/wz.25.H',
     '''
     dd form=native | window min2=-2 max2=2 |
     put label1=Time label2=Offset unit1=s unit2=km
     ''')


I am working on the file "LiterateProgramming.ipynb".
The first task is:
How does it work?
The function view() is specified in m8r.py as follows:
python
def view(name):
    try:
        from IPython.display import Image
        png = name+'.png'
        makefile = os.path.join(rsf.prog.RSFROOT,'include','Makefile')
        os.system('make -f %s %s' % (makefile,png))
        return Image(filename=png)
    except:
        print 'No IPython Image support'
        return None
It runs the make command to generate the image and imports it into IPython. The corresponding Makefile is
```
SConstruct: .scons
    echo "from rsf.proj import " > $@
    cat $^ >> $@
    echo "\nEnd()" >> $@
%.png: SConstruct
    scons Fig/$.vpl
    vpconvert pen=gd fat=3 serifs=n bgcolor=w Fig/$.vpl $@
```
It collects all *.scons files into one SConstruct file, uses scons to create the result figure, and converts the figure to the PNG format using vpconvert.
Cleaning up
Can you write Python code for it?


I am working on the file "LiterateProgramming.ipynb".
The first task is:
The shot gather comes from the collection of shot gathers by Yilmaz and Cumro.
The pipeline in the second Flow converts the data to the native format, windows the offsets to the range from -2 to +2 km, and adds apppropriate labels and units. Next, let us display the data with and without the time-power correction.
Can you write Python code for it?

%%file display.scons

# two plots displayed side by side
Plot('data','grey title="(a) Original Data"')
Plot('tpow2','data',
     'pow pow1=3 | grey title="(b) Time Power Correction" ')

Result('tpow','data tpow2','SideBySideAniso')


I am working on the file "Liver Interactive.ipynb".
The first task is:
Add in new seeds using IndexMouseCapture above, and then create a new input level set image:
Can you write Python code for it?

# get array from previous geodesic active contour
X_gac = sitk.GetArrayFromImage(binaryImg)

# get array from user-input seed clicks
X_click = sitk.GetArrayFromImage(input_level_set_click(imgSigmoid, coords))
X_click[np.where(X_click == -0.5)] = 1.0
X_click[np.where(X_click == 0.5)] = 0.0

# combine into a single array
X_input = X_gac.astype(bool) + X_click.astype(bool)

# write array into new input level set
initImg = sitk.Cast(sitk.GetImageFromArray(X_input.astype(int)), imgSigmoid.GetPixelIDValue()) * -1 + 0.5
initImg.SetSpacing(imgSigmoid.GetSpacing())
initImg.SetOrigin(imgSigmoid.GetOrigin())
initImg.SetDirection(imgSigmoid.GetDirection())


I am working on the file "Liver Interactive.ipynb".
The first task is:
Create initial level set image for input into segmentation:
Can you write Python code for it?

initImg = sitk.Cast(sitk.GetImageFromArray(X_man), imgSigmoid.GetPixelIDValue()) * -1 + 0.5
initImg.CopyInformation(imgSigmoid)


I am working on the file "Liver Interactive.ipynb".
The first task is:
Display the initial input level set:
Can you write Python code for it?


I am working on the file "Liver Interactive.ipynb".
The first task is:
For subsequent rounds, create a new level set from segmentation of downsampled image. Start by converting the segmentation result into a workable format:
Can you write Python code for it?

binaryThresh = sitk.BinaryThresholdImageFilter()
binaryThresh.SetLowerThreshold(-3.0)
binaryThresh.SetUpperThreshold(2.0)
binaryThresh.SetInsideValue(1)
binaryThresh.SetOutsideValue(0)
binaryImg = binaryThresh.Execute(imgGac)

show_img(binaryImg)


I am working on the file "Liver Interactive.ipynb".
The first task is:
I. Read in DICOM Images
The case being analyzed here is a resectable hepatocellular carcinoma from The Cancer Imaging Archive. The patient ID is TCGA-BC-4073. Ultimately, the liver along with the tumor and blood vessels are to be reconstructed. Here, the focus is only on the liver.
Read in the DICOM series:
Can you write Python code for it?

dicomPath = join(expanduser('~'), 'Documents', 'SlicerDICOMDatabase', 'TCIALocal', '0', 'images', '')
reader = sitk.ImageSeriesReader()
seriesIDread = reader.GetGDCMSeriesIDs(dicomPath)[1]
dicomFilenames = reader.GetGDCMSeriesFileNames(dicomPath, seriesIDread)
reader.SetFileNames(dicomFilenames)
imgSeries = reader.Execute()


I am working on the file "Liver Interactive.ipynb".
The first task is:
II. Filtering
First, the original image is smoothed with an edge-preserving filter. Two options for filtering are used here: either curvature anisotropic diffusion or mean filtering.
Curvature anisotropic diffusion
Can you write Python code for it?

timeStep_, conduct, numIter = (0.04, 9.0, 5)
imgRecast = sitk.Cast(imgSlice, sitk.sitkFloat32)
curvDiff = sitk.CurvatureAnisotropicDiffusionImageFilter()
curvDiff.SetTimeStep(timeStep_)
curvDiff.SetConductanceParameter(conduct)
curvDiff.SetNumberOfIterations(numIter)
imgFilter = curvDiff.Execute(imgRecast)

show_img(imgFilter)


I am working on the file "Liver Interactive.ipynb".
The first task is:
III. Edge Potential
Gradient magnitude recursive Gaussian
Can you write Python code for it?

sigma_ = 1.0
imgGauss = sitk.GradientMagnitudeRecursiveGaussian(image1=imgFilter, sigma=sigma_)

show_img(imgGauss)


I am working on the file "Liver Interactive.ipynb".
The first task is:
IV. Feature Image
Sigmoid mapping
Following Section 4.3.1 "Fast Marching Segmentation" on pages 373-374 from The ITK Software Guide Book 2: Design and Functionality, 4th ed. for the setup of the sigmoid filter. The output will be passed along to a segmentation algorithm below.
Note that the plan is to conduct multiple rounds of segmentation, to "start with a downsampled volume and work back to the full resolution using the results at each intermediate scale as the initialization for the next scale." (pg. 370) Therefore, K1 and K2 below for the sigmoid mapping are first set loosely set and will become more strict in later segmentations.
Can you write Python code for it?

K1, K2 = 10.0, 4.0

alpha_ = (K2 - K1)/6
beta_ = (K1 + K2)/2

sigFilt = sitk.SigmoidImageFilter()
sigFilt.SetAlpha(alpha_)
sigFilt.SetBeta(beta_)
sigFilt.SetOutputMaximum(1.0)
sigFilt.SetOutputMinimum(0.0)
imgSigmoid = sigFilt.Execute(imgGauss)

show_img(imgSigmoid)


I am working on the file "Liver Interactive.ipynb".
The first task is:
If needed, combine manual label map with segmentation from above:
Can you write Python code for it?

X_gac = sitk.GetArrayFromImage(imgGac)

X_bool = X_gac < np.amax(X_gac)
X_final = X_man.astype(bool) + X_bool
X_final = X_final.astype(int)

show_img(imgSlice, sitk.GetImageFromArray(X_final))


I am working on the file "Liver Interactive.ipynb".
The first task is:
Pick a slice to work with. Only the 2D case is considered here.
Can you write Python code for it?

sliceNum = 42

imgSlice = imgSeries[:,:,sliceNum]


I am working on the file "Liver Interactive.ipynb".
The first task is:
The 2nd tuple element corresponds to series 9.
Display the slice:
Can you write Python code for it?


I am working on the file "Liver Interactive.ipynb".
The first task is:
V. Input Level Set
Using ideas from the SimpleITK geodesic active contour example to create an initial input level set. Instead of computing a signed Maurer distance map and then applying a binary threshold, the approach here simply draws a circle of a given radius around each user-chosen seed coordinate. Following the SimpleITK Notebook on Levelset Segmentation, a binary dilation with a kernel size of 3 is performed. Finally, as was done in the example (line 60), all image values are multiplied by -1 and added to 0.5. The results is the input level set.
Use the class IndexMouseCapture (defined above) to capture coordinates from mouse clicks for seeds. The radii are all assumed to be of the same size at the moment.
Can you write Python code for it?

X = sitk.GetArrayFromImage(imgSigmoid)

fig = plt.figure()
ax = fig.add_subplot(111)
capture = IndexMouseCapture(ax, X)
fig.canvas.mpl_connect('button_press_event', capture.onclick)

coords = pickle.load(open(os.path.join('Liver Segmentation Data', 'TCGA-BC-4073', 'slice42_2nd_round_seeds.p'), 'rb'))

initImg = input_level_set_click(imgSigmoid, coords)

pickleDir = os.path.join('Liver Segmentation Data', 'TCGA-BC-4073', '')

pickle.dump(capture.coords, open(pickleDir + 'slice51_1st_round_seeds.p', 'wb'))


I am working on the file "Liver Interactive.ipynb".
The first task is:
VI. Segmentation
Geodesic active contour
Can you write Python code for it?

gac = sitk.GeodesicActiveContourLevelSetImageFilter()
gac.SetPropagationScaling(1.0)
gac.SetCurvatureScaling(0.2)
gac.SetAdvectionScaling(4.0)
gac.SetMaximumRMSError(0.01)
gac.SetNumberOfIterations(200)

imgGac = gac.Execute(initImg, imgSigmoid)

show_img(imgSlice, imgGac)


I am working on the file "Liver Interactive.ipynb".
The first task is:
VII. Write out to file
If needed, binarize segmentation image:
Can you write Python code for it?

X_gac = sitk.GetArrayFromImage(imgGac)

X_bool = X_gac < np.amax(X_gac)
X_final = X_bool.astype(int)

show_img(imgSlice, sitk.GetImageFromArray(X_final))


I am working on the file "Liver Segmentation.ipynb".
The first task is:
Add in a 3rd set of seeds:
Can you write Python code for it?

coords3 = [(225, 177), (246, 114), (83, 229), (78, 208), (82, 183), (238, 126)]
radii3 = [5, 10, 5, 5, 5, 15]

seed2radius3 = {tuple(reversed(p[0])): p[1] for p in zip(coords3, radii3)}

X_1 = sitk.GetArrayFromImage(binaryImg)

# create a 3rd seed matrix from the 3rd set of coordinates
setupImg = sitk.Image(imgSigmoid.GetSize()[0], imgSigmoid.GetSize()[1], sitk.sitkUInt8)
X_2 = sitk.GetArrayFromImage(setupImg)
for i in range(X_2.shape[0]):
    for j in range(X_2.shape[1]):
        for s in seed2radius3.keys():
            if euclidean((i,j), s) <= seed2radius3[s]:
                X_2[i,j] = 1

X = X_1.astype(bool) + X_2.astype(bool)
initImg3 = sitk.Cast(sitk.GetImageFromArray(X.astype(int)), imgSigmoid.GetPixelIDValue()) * -1 + 0.5
initImg3.SetSpacing(imgSigmoid.GetSpacing())
initImg3.SetOrigin(imgSigmoid.GetOrigin())
initImg3.SetDirection(imgSigmoid.GetDirection())

liversegmentation.sitk_show(initImg3)


I am working on the file "Liver Segmentation.ipynb".
The first task is:
Add in new seeds:
Can you write Python code for it?


I am working on the file "Liver Segmentation.ipynb".
The first task is:
By comparing images between OsiriX and plots of the SimpleITK images, the 2nd tuple element corresponds to series 9.
Can you write Python code for it?


I am working on the file "Liver Segmentation.ipynb".
The first task is:
Display overlay of segmentation over original slice:
Can you write Python code for it?

labelLowThresh = -2.3438
labelUpThresh = 0.0

binarySegImg3 = liversegmentation.binary_threshold(imgGac3, labelLowThresh, labelUpThresh)

liversegmentation.sitk_show(sitk.LabelOverlay(imgSliceUInt8, binarySegImg3, backgroundValue=255))


I am working on the file "Liver Segmentation.ipynb".
The first task is:
Edge potential
Gradient magnitude recursive Gaussian
Can you write Python code for it?

#sigma = 3.0
sigma = 1.0
imgGauss = liversegmentation.gradient_magnitude(imgFilter, sigma)

liversegmentation.sitk_show(imgGauss)


I am working on the file "Liver Segmentation.ipynb".
The first task is:
Median filter
Can you write Python code for it?

med = sitk.MedianImageFilter()
med.SetRadius(3)
imgFilter = med.Execute(imgSlice)

liversegmentation.sitk_show(imgFilter)


I am working on the file "Liver Segmentation.ipynb".
The first task is:
Note that the TCGA-BC-4073 patient has 2 series of images (series 9 & 10). The series IDs are:
Can you write Python code for it?


I am working on the file "Liver Segmentation.ipynb".
The first task is:
Now create new level set image:
Can you write Python code for it?

X_1 = sitk.GetArrayFromImage(binaryImg)

# create a 2nd seed matrix from the 2nd set of coordinates
setupImg = sitk.Image(imgSigmoid.GetSize()[0], imgSigmoid.GetSize()[1], sitk.sitkUInt8)
X_2 = sitk.GetArrayFromImage(setupImg)
for i in range(X_2.shape[0]):
    for j in range(X_2.shape[1]):
        for s in seed2radius2.keys():
            if euclidean((i,j), s) <= seed2radius2[s]:
                X_2[i,j] = 1

X = X_1.astype(bool) + X_2.astype(bool)
initImg2 = sitk.Cast(sitk.GetImageFromArray(X.astype(int)), imgSigmoid.GetPixelIDValue()) * -1 + 0.5
initImg2.SetSpacing(imgSigmoid.GetSpacing())
initImg2.SetOrigin(imgSigmoid.GetOrigin())
initImg2.SetDirection(imgSigmoid.GetDirection())

liversegmentation.sitk_show(initImg2)


I am working on the file "Liver Segmentation.ipynb".
The first task is:
Read in DICOM images
Can you write Python code for it?

sliceNum = 42

dicomPath = join(expanduser('~'), 'Documents', 'SlicerDICOMDatabase', 'TCIALocal', '0', 'images', '')
reader = sitk.ImageSeriesReader()
seriesIDread = reader.GetGDCMSeriesIDs(dicomPath)[1]
dicomFilenames = reader.GetGDCMSeriesFileNames(dicomPath, seriesIDread)
reader.SetFileNames(dicomFilenames)
imgSeries = reader.Execute()
imgSlice = imgSeries[:,:,sliceNum]


I am working on the file "Liver Segmentation.ipynb".
The first task is:
Segmentation
Geodesic Active Contour
Can you write Python code for it?

#gacParams = (1.0, 0.2, 4.5, 0.01, 250)
#gacParams = (1.0, 0.2, 4.5, 0.01, 200)
gacParams = (1.0, 0.2, 5.0, 0.01, 350)

imgGac3 = liversegmentation.geodesic_active_contour(initImg3, imgSigmoid, *gacParams)

liversegmentation.sitk_show(imgGac)


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
Adding a white noise for numbers
This is very simple.
But I want to be safe, so it will only works if the frame indicate that the number does not come from a file, as previously.
Can you write Python code for it?

def add_white_noise_to_numbers(frame, obj):
    if isinstance(obj, Number):
        info = getframeinfo(frame)
        if '<stdin>' in info.filename or '<ipython-' in info.filename:
            return obj + np.random.normal()
    return obj


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
And it works as expected:
Can you write Python code for it?

np.random.seed(120193)
print(120193, 120193j)
with WhiteNoiseComplex():
    print(120193, 120193j)  # Huhoo, noisy!
print(120193, 120193j)

print(0*1j)
with WhiteNoiseComplex():
    print(0*1j)  # Huhoo, noisy!
print(0*1j)


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
Anyhow, if rwatch is installed, we can import it, and it enables two new functions in the sys module:
Can you write Python code for it?

import rwatch
from sys import setrwatch, getrwatch

setrwatch({})  # clean any previously installed rwatch
getrwatch()


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
But obviously, having this for all objects is incredibly verbose!
Can you write Python code for it?

def debug_view_for_any_object(frame, obj):
    info = getframeinfo(frame)
    if '<stdin>' in info.filename or '<ipython-' in info.filename:
        msg = '- Access to {!r} (@{}) at {}:{}:{}'
        print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
    return obj


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
Defining a generic noisy context manager
This will be a very simple change from the previous one, by letting the Noisy class accept any noisy function, which takes obj and return a noisy version of obj, only for complex-valued objects.
Can you write Python code for it?

class Noisy(object):
    def __init__(self, noise):
        def add_white_noise_to_complex(frame, obj):
            if isinstance(obj, complex):
                info = getframeinfo(frame)
                if '<stdin>' in info.filename or '<ipython-' in info.filename:
                    return noise(obj)
            return obj

        self.rwatch = add_white_noise_to_complex

    def __enter__(self):
        setrwatch(defaultdict(lambda: self.rwatch))

    def __exit__(self, exc_type, exc_val, exc_tb):
        setrwatch({})

print(1j)
with Noisy(lambda obj: obj + np.random.normal()):
    print(1j)
print(1j)

print(1j)
with Noisy(lambda obj: obj * np.random.normal()):
    print(1j)
print(1j)

print(1j)
with Noisy(lambda obj: obj + np.random.normal(10, 0.1) + np.random.normal(10, 0.1) * 1j):
    print(1j)
print(1j)


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
Finally, we need the collections module and its defaultdict magical datastructure.
Can you write Python code for it?


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
It seems to work!
Let's do it for any number then...
... Sadly, it's actually breaking the interpreter, which obviously has to have access to non-noisy constants and numbers to work !
We can lower the risk by only adding noise to complex numbers.
I guess the interpreter doesn't need complex numbers, write?
Can you write Python code for it?

def add_white_noise_to_complex(frame, obj):
    if isinstance(obj, complex):
        info = getframeinfo(frame)
        if '<stdin>' in info.filename or '<ipython-' in info.filename:
            return obj + np.random.normal() + np.random.normal() * 1j
    return obj

np.random.seed(1234)
setrwatch({})
y = 1234j
print(y)
setrwatch(defaultdict(lambda: add_white_noise_to_complex))
print(y)  # huhoww, that's noisy!
setrwatch({})
print(y)


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
It will probably break everything in the notebook, but works in a basic Python interpreter.
Can you write Python code for it?


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
Let check that one, on a very simple example (which runs in less than 20 micro seconds):
Can you write Python code for it?

print(x)
%time 123 + 134

setrwatch({})
setrwatch(defaultdict(lambda: debug_view_for_any_object))
print(x)
%time 123 + 134
setrwatch({})


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
More useful debuggin information
What is this frame thing?
It is described in the documentation of the inspect module.
We can actually use it to display some useful information about the object and where was it called etc.
Can you write Python code for it?

from inspect import getframeinfo

def debug_view(frame, obj):
    info = getframeinfo(frame)
    msg = '- Access to {!r} (@{}) at {}:{}:{}'
    print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
    return obj

setrwatch({})
setrwatch({
    id(x): debug_view
})
getrwatch()

print(x)


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
The first debug information shows line 5, which is the line where print(z) is.
A second context manager to debug any object
Easy:
Can you write Python code for it?

class InspectAllObjects(object):
    def __init__(self):
        pass

    def __enter__(self):
        setrwatch(defaultdict(lambda: debug_view_for_any_object))

    def __exit__(self, exc_type, exc_val, exc_tb):
        setrwatch({})


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
Then, the core part will be to install and import James Powell (dutc) rwatch module.
If you don't have it installed :

Be sure to have CPython 3.5. rwatch patches the CPython eval loop, so it's fixed to specific versions of Python & the author lazy about keeping it updated.
Then pip install dutc-rwatch. It should work, but it fails for me.
(alternative) You can just cd /tmp/ && git clone https://github.com/dutc/rwatch && cd rwatch/src/ && make and copy the rwatch.so dynamic library wherever you need...
Can you write Python code for it?

%%bash
tmpdir=$(mktemp -d)
cd $tmpdir
git clone https://github.com/dutc/rwatch
cd rwatch/src/
make
ls -larth ./rwatch.so
file ./rwatch.so
# cp ./rwatch.so /where/ver/you/need/  # ~/publis/notebook/ for me


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
We also see here the None and {} objects being given to the context manager (see the __enter__ method at first, and __exit__ at the end).

Defining a context manager to add white noise
Basically, we will do as above, but instead of debug information, a white noise sampled from a Normal distribution (i.e., $\sim \mathcal{N}(0, 1)$) will be added to any number.
Capturing any numerical value
To capture both integers and float numbers, the numbers.Number abstract class is useful.
Can you write Python code for it?


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
We can also delete rwatches that are not defined, without a failure:
Can you write Python code for it?

y = "I am Zorro !"
print(y)
delrwatch(y)  # No issue!
print(y)


I am working on the file "Living_in_a_noisy_world_with_James_Powell_rwatch_module.ipynb".
The first task is:
We can check it:
Can you write Python code for it?

z = "I am Batman!"
print(z)

with InspectThisObject(z):
    print(z)

print(z)


I am working on the file "Load and Manipulate Sheet by Adding Logic.ipynb".
The first task is:
Combine DecisionF1F2 and Decision2
Can you write Python code for it?

# use a different way to access the columns (using .colname - this only works for
# easy-to-read ASCII names, it won't work with funny characters)
# and use the logical and (&) to do pairwise logic, assigning the result to our new column
df['Decision3'] = df.DecisionF1F2 & df.Decision2
df.head(10)


I am working on the file "Load and Manipulate Sheet by Adding Logic.ipynb".
The first task is:
Let's be clever - we'll do some text processing
We'll try the external fuzzywuzzy library to do some very simple text processing.
If needed install using conda install -c wpb fuzzywuzzy.
Can you write Python code for it?

from fuzzywuzzy import fuzz
# http://pbpython.com/excel-pandas-comp.html side reading on fuzzywuzzy
# http://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/ other fuzzywuzzy metrics for
# other ways of measuring similarity

# quick demo:
item1 = "hollyoaks"
target = "hollyoaks"
print(target, fuzz.ratio(item1, target))
target = 'holly-oaks'
print(target, fuzz.ratio(item1, target))
target = 'holly oak'
print(target, fuzz.ratio(item1, target))
target = "tv's best moments"
print(target, fuzz.ratio(item1, target))

def decision_tvshow_is_hollyoaks_with_smarts(cell):
    cleaned_cell = cell
    cleaned_cell = cleaned_cell.lower() # lowercase
    cleaned_cell = cleaned_cell.strip() # remove superflous whitespace
    is_hollyoaks =  fuzz.ratio("hollyoaks", cleaned_cell) > 85
    return is_hollyoaks

# we can assign the result back to the DataFrame as a new column
df['TVShow'].apply(decision_tvshow_is_hollyoaks_with_smarts)

df.head(10) # show that we *haven't* yet manipulated the DataFrame

df['Decision2'] = df['TVShow'].apply(decision_tvshow_is_hollyoaks_with_smarts)
df.head(10)
# note that we *haven't re-run Decision3*!

df['Decision3'] = df.DecisionF1F2 & df.Decision2
df.head(10)

# We could copy the Excel code down here and write out a new spreadsheet to disk...


I am working on the file "LoadingData.ipynb".
The first task is:
.csv — Comma-Separated Values
Data type with the least overhead (aka file size) and very common
Std Python
Conceptually, the basics of a CSV are just like splitting a string by commas:
Can you write Python code for it?

print("Split on comma as strings")
csv_row = '1,2.0,Three point five,True'
print(csv_row.split(','))

print("\nSplit on comma and converted to ints")
csv_row = '1,2,3,4,5,6,7,8,9'
print([int(c) for c in csv_row.split(',')])


I am working on the file "LoadingData.ipynb".
The first task is:
.json — JavaScript Object Notation
JSON is a common web and API communication format and easy to read. It also happens to be copy/paste compatible as native Python code (with some restrictions the other way around)
Std Python
Can you write Python code for it?

import json

color_data = [
    {"name": "Michael", "fav_color": "purple"},
    {"name": "Casey", "fav_color": "turquoise"}
]

#load, loads, dump, and dumps are common library APIs
#load/dump tkae file objects while loads/dumps take strings
print(json.dumps(color_data))

json_data = json.load(open('data/volcanoes.json'))
print('Rows:', len(json_data))


I am working on the file "LoadingData.ipynb".
The first task is:
.xls .xlsx — Excel Spreadsheet
Excel spreadsheets are more complicated than CSV (ex they can contain multiple datasets), but they still follow the 2D array format of a CSV and DataFrames
Std Python
Can you write Python code for it?

import openpyxl
# You can also use xlrd, xlsxwriter, and a host of others.
# See http://www.python-excel.org/ for all your options.
wb = openpyxl.Workbook()
ws = wb.create_sheet("NewSheet")

ws['A1'] = "Name"
ws['A2'] = "Michael"
ws['B1'] = "fav_color"
ws['B2'] = "Purple"

wb.save("data/ExcelData.xlsx")

excel_row = 'data/ExcelData.xlsx'
data = openpyxl.load_workbook(excel_row)
for sheet in data:
    for row in sheet:
        print(row)
        print(*[cell.value for cell in row])


I am working on the file "LoadingData.ipynb".
The first task is:
.xml — eXtensible Markup Language
XML looks and acts just like HTML. It is a popular web format but is being superseded by JSON
Std Python
Can you write Python code for it?

import xml.etree.ElementTree as ET

tree = ET.parse('data/volcano.xml')
#ET.fromstring(some_xml_string)
root = tree.getroot()

for item in root:
    print(item.tag, ' -- ', item.text)


I am working on the file "LoadingData.ipynb".
The first task is:
Bonus
Here are some scripts we used to create the different datasets
Can you write Python code for it?

#Create full JSON from CSV
with open('data/volcanoes.csv') as fin:
    data = [dict(row) for row in csv.DictReader(fin)]
    json.dump(data, open('data/volcanoes.json', 'w'))

#Create single element XML from the JSON data
#This library is not included
from dicttoxml import dicttoxml
print(dicttoxml(json_data[0]), file=open('data/volcano.xml'))

#Create full YAML from the JSON data
yaml.dump(json_data, open('data/volcanoes.yml', 'w'))


I am working on the file "LoadingData.ipynb".
The first task is:
Pandas
Because CSV matches the structure of pandas.DataFrame, it's really easy to import the data.
Can you write Python code for it?

csv_df = pd.read_csv(open('data/volcanoes.csv'))
csv_df.head()


I am working on the file "LoadingData.ipynb".
The first task is:
Pandas
Can you write Python code for it?

color_df = pd.DataFrame(pd.read_pickle('data/colors.pkl'))
color_df

color_df.to_pickle('data/color_df.pkl')


json_df = pd.read_json('data/volcanoes.json')
json_df.head()


I am working on the file "LoadingData.ipynb".
The first task is:
Pandas
There's not a native way to read in YAML data to Pandas, but you could use it's JSON lib to do so
Can you write Python code for it?

yaml_df = pd.io.json.json_normalize(yaml.load(open('data/volcanoes.yml')))
yaml_df.head()


I am working on the file "LoanDefaultPrediction.ipynb".
The first task is:
Now we can look for data to do some exploratory stuff.
Can you write Python code for it?


I am working on the file "LoanDefaultPrediction.ipynb".
The first task is:
Remove the rows with NA for loan status
Can you write Python code for it?

df3a.dropna(subset=['loan_status'],how='any',inplace=True)

df3a.loan_status.value_counts().plot(kind='bar')

default_status = {'Fully Paid':1,'Charged Off':0,'Current':1,'In Grace Period':0,'Late (31-120 days)':0,'Default':0,'Does not meet the credit policy. Status:Fully Paid':1,'Does not meet the credit policy. Status:Charged Off':0,'Late (16-30 days)':0}

df3a['defaultYN']=df3a.loan_status.map(default_status)

df3a.defaultYN.value_counts()

#sns.countplot(x="defaultYN",data=df3a)
df3a.grade.value_counts()

sns.distplot(df3a.loan_amnt)

sns.distplot(df3a.annual_inc)

sns.countplot(df3a.defaultYN)


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Calculate the average balance on the account.
Can you write Python code for it?

avg_balance = pd.DataFrame(trans_scope.groupby('account_id').agg('mean')['balance'])
df = pd.merge(left = df, right = avg_balance, left_on = 'account_id', right_index = True)


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Card info

We will only look at the type of card
We will try to make two groups. We have no information on what 'junior', 'classic' or gold is. Thus we will make a plot to see whether they have any influence on the loan status.
We only have 107 credit cards for all customers having a loan. Only 5 of those have no debt.
Thus we deicd to drop card info for now.
Can you write Python code for it?

card_scope = client_info.loc[:,['status','type_card']].dropna()
card_scope['status'].value_counts()


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Client charasteristics
We look at the following charasteristics:
- Number of clients per account.
- Average balance on the account.
- Birthyear of the owner
- Sex of the owner
- Startyear of the account
Note that we will need to cut the data into 75% running time for finished clients when we accumulate different data.
Scope on only the clients with a loan:
Can you write Python code for it?

client_scope = client_info[client_info['account_id'].isin(scope)]
client_scope.describe()


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Get the average balance on the account
For finished clients, cut all data after 75% of the running time.
NB: This actually comes from the transaction_scope data.
Can you write Python code for it?

# Get the scope
trans_scope = transaction_info[transaction_info['account_id'].isin(scope)]
trans_scope = pd.merge(left = trans_scope, right = df.loc[:,['account_id','start_date','duration','status']])
trans_scope['start_date'] = pd.to_datetime(trans_scope['start_date'], format='%Y-%m-%d')
trans_scope['date'] = pd.to_datetime(trans_scope['date'], format='%Y-%m-%d')
# Split the data in running and finished
trans_scope_running = trans_scope[trans_scope['status'].isin(['run_no_problem','run_but_debt'])].copy()
trans_scope_finished = trans_scope[trans_scope['status'].isin(['fin_no_problem','fin_unpaid'])].copy()
# Remove the transactions after the third quarter
trans_scope_finished['cutoff_duration'] = trans_scope_finished['duration']*3/4
trans_scope_finished['cutoff_date'] = trans_scope_finished['start_date'] + \
    trans_scope_finished['cutoff_duration'].values.astype("timedelta64[M]")
trans_scope_finished = trans_scope_finished[trans_scope_finished['cutoff_date'] > trans_scope_finished['date']]
# Remove the helpcolumns
del trans_scope_finished['cutoff_duration']
del trans_scope_finished['cutoff_date']

trans_scope = pd.concat([trans_scope_finished, trans_scope_running])


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Get the average birthyear of the owner.
Can you write Python code for it?

owner_scope = client_scope[client_scope['type_client']=="OWNER"].copy()
owner_scope['birth_number'] = pd.to_datetime(owner_scope['birth_number'],
                                                format='%Y-%m-%d')
owner_scope['birthyear_owner'] = owner_scope['birth_number'].dt.year
df = pd.merge(left = df,
              right = owner_scope.loc[:,['account_id','birthyear_owner']],
              left_on = 'account_id', right_on = 'account_id')


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Load data:
Can you write Python code for it?

client_info = pd.read_csv('data/client_info.csv')
demographic_info = pd.read_csv('data/demographic_data.csv')
transaction_info = pd.read_csv('data/transction_info.csv')
order_info = pd.read_csv('data/order_info.csv')
loan_info = pd.read_csv('data/loan_info.csv')


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Loan data
Look at the loan data and see which features can be made.
Can you write Python code for it?


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Max number of consecutively missed payments
Can you write Python code for it?

consecutive = monthly_missed.copy()
consecutive[consecutive > -2] =0
consecutive[consecutive < -1] = -1

# From: https://stackoverflow.com/questions/36441521
import itertools
tmplist = []
for acct in consecutive.columns:
    a = consecutive[acct].tolist()
    z = [(x[0], len(list(x[1]))) for x in itertools.groupby(a)]
    z = [x for x in z if x[0] !=0]
    if len(z)==0:
        tmp = 0
    else:
        tmp = max(z, key=lambda x:x[1])[1]
    tmplist.append(tmp)
max_missed = pd.DataFrame()
max_missed['account_id'] = consecutive.columns
max_missed['max_missed'] = tmplist

df = pd.merge(left = df, right = max_missed, left_on = 'account_id', right_on = 'account_id')


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Merge the total misses
Can you write Python code for it?

tmp = pd.DataFrame(monthly_missed.apply(lambda column: (column < -1).sum()))
tmp.columns = ['total_missed']
df = pd.merge(left = df, right = tmp, left_on = 'account_id', right_index = True)


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Missing payments into loan.
Now that we know this, we can do our final transformation on the missed payments. Even though we have information on the unemployment rates of '95 and '96, we act as if all our periods are equal. Hence we can now start making features.
We will use the features:
- Loan amount
- Loan duration
- Number of total missed payments
- Number of missed payments in first, second, third and last quarter of the loan
- Max number of consecutively missed payments
As we will provide the split into the 80/20 estimate, we will first split and then generate the features on the first 80%. The last 20% will only be provided a "missed payment" column which is either 0 or 1.
Let's first remake the above picture for 'months into loan'.
This means moving all starts to 0. My implementation seems suboptimal, but it will do the job. I could have also moved all transactions and redo the pivoting.
Can you write Python code for it?

max_duration = loan_info['duration'].max()
monthly_missed = pd.DataFrame(index=range(0,60), columns=list(payment_diff))

for acct in scope:
    if acct in running:
        loan_start = loan_info[loan_info['account_id']==acct]['date'].unique()[0]
        tmp = payment_diff.loc[payment_diff.index > loan_start, acct].tolist()
        tmp = [round(i,0) for i in tmp]
        tmp = tmp + [0]*(60 - len(tmp))
        tmp = [int(i) for i in tmp]
        monthly_missed[acct] = tmp
    else:
        loan_start = loan_info[loan_info['account_id']==acct]['date'].unique()[0]
        loan_date = datetime.strptime(loan_start, '%Y-%m-%d')
        loan_date_end = add_months(loan_date,loan_info[loan_info['account_id']==acct]['duration'].unique()[0])
        loan_end = loan_date_end.strftime('%Y-%m-%d')
        tmp = payment_diff.loc[(payment_diff.index > loan_start) & (payment_diff.index < loan_end), acct].tolist()
        tmp = [round(i,0) for i in tmp]
        tmp = tmp + [0]*(60 - len(tmp))
        tmp = [int(i) for i in tmp]
        monthly_missed[acct] = tmp

plt.figure()
ax = monthly_missed.plot(legend = False)
ax.set_ylabel("amount")
ax.set_xlabel("month")


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
On first inspection, it seems that we missed some initial payments. This is not a big deal as we are only intersted in missed payments and not so much in early payments (which may or may not have been refunded).
Note also that in this graph, spikes are one-time missers, where plateaus are multiple consecutive missed payments.
Now, the main bulk of missed payments is at the end of the graph, this is because we look only at running loans, which are generally started later.
Therefore we choose to revert this to once again looking at all loans and later look at only the running ones. This essentially means running the same code again, with the adjustment that for ended loans, we need to not assume that a zero payment after the startdate is a missed one.
Can you write Python code for it?

scope = loan_info['account_id'].unique()
running = loan_info[loan_info['status']=='run_but_debt']['account_id'].unique()

transaction_scope = transaction_info[transaction_info['account_id'].isin(scope)]
transaction_scope = transaction_scope[transaction_scope['k_symbol']=='loan']

transaction_scope = transaction_scope.loc[:,['account_id','date','amount']]
transaction_scope = transaction_scope.pivot(index = 'date',
                                            columns = 'account_id',
                                            values = 'amount')
transaction_scope = transaction_scope.fillna(0)


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Plot all payments.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib
matplotlib.style.use('ggplot')

trans_plot = transaction_scope.cumsum()
plt.figure()
ax = trans_plot.plot(legend = False)
ax.set_ylabel("amount")


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Save the data frame:
Can you write Python code for it?

df.to_csv('data/loan_prep.csv', index = False)


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Scope only on loans and pivot.
Can you write Python code for it?

transaction_scope = transaction_scope[transaction_scope['k_symbol']=='loan']
transaction_scope = transaction_scope.loc[:,['account_id','date','amount']]
transaction_scope = transaction_scope.pivot(index = 'date',
                                            columns = 'account_id',
                                            values = 'amount')
transaction_scope = transaction_scope.fillna(0)


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Split the dataset on loan duration
Can you write Python code for it?

durations = df['duration'].value_counts().to_dict()
quarter_missed = {}
misses = pd.DataFrame(columns = ['first_quarter','second_quarter','third_quarter','last_quarter'])
for key in durations.keys():
    tmp =  df[df['duration']==int(key)]['account_id'].tolist()
    quarter_missed[key] = monthly_missed.loc[:,tmp]
    tmp_df = pd.DataFrame(quarter_missed[key].loc[range(0,key/4),].apply(lambda column: (column < -1).sum()))
    tmp_df['second_quarter'] = quarter_missed[key].loc[range(key/4, 2*key/4),].apply(lambda column: (column < -1).sum())
    tmp_df['third_quarter'] = quarter_missed[key].loc[range(2*key/4, 3*key/4),].apply(lambda column: (column < -1).sum())
    tmp_df['last_quarter'] = quarter_missed[key].loc[range(3*key/4, 4*key/4),].apply(lambda column: (column < -1).sum())
    tmp_df.columns = ['first_quarter','second_quarter','third_quarter','last_quarter']
    misses = pd.concat([misses,tmp_df])
for col in misses.columns:
    misses[col] = misses[col].astype(int)
df = pd.merge(left = df, right = misses, left_on = 'account_id', right_index = True)

quarter_missed.keys()

for col in misses.columns:
    misses[col] = misses[col].astype(int)


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
This confirms the fact that payment differences which are positive occur in the first month. Hence, this is the prepayment mismatch we have (but at least we don't set ones to missed which aren't).
Let's make the loan table.
Can you write Python code for it?

df = loan_info.copy()
df.columns = ['loan_id','account_id','start_date','amount','duration','payments','status']


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
Transactional behavior
Let's look at all the transactions
- The total number transactions
- The average volume per transaction
- The total number of cash transactions
- The aveage volume per cash transaction
- The total number of cc withdrawals
- The aveage volume per cc withdrawal
NB: We already filtered the finsihed contracts on the 75/25% split above.
Can you write Python code for it?

trans_scope['operation'].value_counts()

average_volume = pd.DataFrame(trans_scope.groupby('account_id').agg('mean')['amount'])
average_volume.columns = ['average_volume']
total_number = pd.DataFrame(trans_scope.groupby('account_id').agg(lambda x: len(x.unique()))['trans_id'])
total_number.columns = ['total_number_trans']

total_number_cash = pd.DataFrame(trans_scope[trans_scope['operation'].isin(['cash_credit','cash_withdrawl'])].groupby('account_id').agg(lambda x: len(x.unique()))['trans_id'])
total_number_cash.columns = ['total_number_cash']
average_cash_volume = pd.DataFrame(trans_scope[trans_scope['operation'].isin(['cash_credit','cash_withdrawl'])].groupby('account_id').agg('mean')['amount'])
average_cash_volume.columns = ['average_volume_cash']

total_number_cc = pd.DataFrame(trans_scope[trans_scope['operation']=="cc_withdrawal"].groupby('account_id').agg(lambda x: len(x.unique()))['trans_id'])
total_number_cc.columns = ['total_number_cc']
average_cc_volume = pd.DataFrame(trans_scope[trans_scope['operation']=="cc_withdrawal"].groupby('account_id').agg('mean')['amount'])
average_cc_volume.columns = ['average_volume_cc']

df = pd.merge(left = df, right = average_volume,
              left_on = 'account_id', right_index = True,
              how ='left')
df = pd.merge(left = df, right = total_number,
              left_on = 'account_id', right_index = True,
              how ='left')
df = pd.merge(left = df, right = total_number_cash,
              left_on = 'account_id', right_index = True,
              how ='left')
df = pd.merge(left = df, right = average_cash_volume,
              left_on = 'account_id', right_index = True,
              how ='left')
df = pd.merge(left = df, right = total_number_cc,
              left_on = 'account_id', right_index = True,
              how ='left')
df = pd.merge(left = df, right = average_cc_volume,
              left_on = 'account_id', right_index = True,
              how ='left')

df = df.fillna(0)


I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
We start by looking at running loans with payment problems to see how we can define a model their behavior.
Scope on run_but_dept clients (for plots)
Can you write Python code for it?

scope = loan_info[loan_info['status']=='run_but_debt']['account_id'].unique()
transaction_scope = transaction_info[transaction_info['account_id'].isin(scope)]


I am working on the file "Lobatto_quadrature.ipynb".
The first task is:
Comparison of Lobatto and Chebyshev nodes
Can you write Python code for it?

norder = 10

min_cheb = []
max_cheb = []
min_lob = []
max_lob = []
plt.figure(figsize=(10, 4))
plt.subplot(1, 2, 1)
x_cheb = []
x_lob = []
y = []
for n in range(1, norder + 1):
    x_ch = [np.cos((2*i - 1)/(2*n)*np.pi) for i in range(n, 0, -1)]
    x_ch = [-1] + x_ch + [1]
    x_cheb += x_ch
    x_lo, _ = gauss_lobatto(n + 2, 16)
    x_lob += x_lo
    dx_cheb = [x_ch[k + 1] - x_ch[k] for k in range(0, n + 1)]
    dx_lob = [x_lo[k + 1] - x_lo[k] for k in range(0, n + 1)]
    min_cheb.append(min(dx_cheb))
    max_cheb.append(max(dx_cheb))
    min_lob.append(min(dx_lob))
    max_lob.append(max(dx_lob))
    y += [n for i in range(n + 2)]

plt.plot(x_cheb, y, marker='s', lw=0)
plt.plot(x_lob, y, marker='v', lw=0)

#%%
plt.xlim(-1.2, 1.2)
plt.ylim(0, norder)
plt.subplot(1, 2, 2)
plt.plot(range(2, norder + 2), min_cheb, label="Min distance Chebyshev",
         marker="s", lw=0)
plt.plot(range(2, norder + 2), max_cheb, label="Max distance Chebyshev",
         marker="o", lw=0)
plt.plot(range(2, norder + 2), min_lob, label="Min distance Lobatto",
         marker="v", lw=0)
plt.plot(range(2, norder + 2), max_lob, label="Max distance Lobatto",
         marker="^", lw=0)
plt.legend(loc="best")
plt.show()

from IPython.core.display import HTML
def css_styling():
    styles = open('./styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()


I am working on the file "Lobatto_quadrature.ipynb".
The first task is:
Gauss-Lobatto quadrature can integrate exactly polynomials up to $2n-3$, where $n$ is the number of points used
Fourth order polynomial
Can you write Python code for it?

poly = 4*y**4 + 3*y**3 + y - 1
sym.plot(poly, (y, -1, 1));

print(sym.integrate(poly,(y,-1,1)))
print(sym.integrate(poly,(y,-1,1)).n(8))

for k in range(2,10):
    xi, w = gauss_lobatto(k, 8)
    poly_int = sum(w[j]*poly.subs(y, xi[j]) for j in range(k))
    print(k, poly_int)


I am working on the file "Lobatto_quadrature.ipynb".
The first task is:
Trascendent function
Can you write Python code for it?

fun = y*sym.sin(5*y) + y**2
sym.plot(fun, (y, -1, 1));

fun_int = sym.integrate(fun,(y,-1,1))
print(fun_int)
print(fun_int.n(8))

print("{}\t{}\t{}".format('n', 'Integral', 'Relative error'))
print("===========================================")
for k in range(2,20):
    xi, w = gauss_lobatto(k, 8)
    poly_int = sum(w[j]*fun.subs(y, xi[j]) for j in range(k))
    print("{}\t{}\t{}".format(k, float(poly_int), float((fun_int - poly_int)/poly_int)))


I am working on the file "LocalCoordAperturePhotometry.ipynb".
The first task is:
Annotate plot axis with coordinate positions and designations
Invoked by show_image()
Can you write Python code for it?

def annotate_image(axis, designation, xy, aperture_size):
    axis.plot(xy[0], xy[1], 'o', markersize=aperture_size,
              markeredgecolor='r', markerfacecolor='none',
              markeredgewidth=2)

    offsetbox = TextArea(designation, minimumdescent=False)

    ab = AnnotationBbox(offsetbox, xy,
                        xybox=(-20, 40+random()*10-10),
                        xycoords='data',
                        boxcoords="offset points",
                        arrowprops=dict(arrowstyle="->"))

    axis.add_artist(ab)


I am working on the file "LocalCoordAperturePhotometry.ipynb".
The first task is:
Aperture photometry
Can you write Python code for it?

# Create empty table with time and object headers
pd.options.display.float_format = '{:,.6f}'.format
instr_mag_df = pd.DataFrame()
names = [name for name in position_map]
instr_mag_df['name'] = [time_name] + names
instr_mag_df.set_index('name', inplace=True)

# Carry out photometry on B, G, R FITS files, yielding instrumental magnitudes
positions = position_map.values()

multi_file_photometry(fits_root, fits_files, data_index, positions, instr_mag_df,
                      measurement_aperture, inner_sky_annulus, outer_sky_annulus, gain=gain)

# Save photometry table as CSV
instr_mag_csv_path = os.path.join(instr_mag_file_root, instr_mag_csv_file)
instr_mag_df.T.to_csv(instr_mag_csv_path)

# Display photometry table
instr_mag_df.T

# TODO: to do this properly, create separate IM DFs above and summarise each here
# The multiple DFs above can be merged for display and written as a single file for
# input into next stage of analysis (spreadsheet)

# cols = [col for col in instr_mag_df.columns]# if col.startswith(fits_prefixes[0])]
# mean_df = pd.DataFrame(columns=cols)
# stdev_df = pd.DataFrame(columns=cols)

# for fits_prefix in fits_prefixes:
#     row_names = instr_mag_df.T.columns
#     row = [fits_prefix]
#     for row_index, row_name in enumerate(row_names):
#         row.append(instr_mag_df[cols].loc[row_name])
#     #mean_df.loc[row_index] = [row_name, np.mean(row)]
#     #stdev_df.loc[row_index] = [row_name, np.mean(row)]

# mean_df
# stdev_df


I am working on the file "LocalCoordAperturePhotometry.ipynb".
The first task is:
Aperture radii and gain
Can you write Python code for it?

# Aperture radii
measurement_aperture = 12
inner_sky_annulus = 15
outer_sky_annulus = 20

# ph/ADU
# Note: PythonPhot's aperture photometry function takes a phadu parameter.
# Assumption: this is photons/ADU or e-/ADU, i.e. gain.
gain=1.67


I am working on the file "LocalCoordAperturePhotometry.ipynb".
The first task is:
Display an image with target and reference stars annotated, to sanity check local coordinates
Can you write Python code for it?

def show_image(image_data, coord_map, aperture_size, annotate=True, vmin=10, vmax=200, figx=20, figy=10):
    fig = plt.figure(figsize=(figx, figy))
    plt.imshow(image_data, cmap='gray', vmin=vmin, vmax=vmax)
    plt.gca().invert_yaxis()
    plt.colorbar()

    if annotate:
        for designation in coord_map:
            xy = coord_map[designation]
            annotate_image(fig.axes[0], designation, xy, aperture_size)

    plt.show()


I am working on the file "LocalCoordAperturePhotometry.ipynb".
The first task is:
Functions
Photometry of a list of FITS files, creating a table of times and instrumental magnitudes
Can you write Python code for it?

def multi_file_photometry(fits_root, fits_files, data_index, coords, dataframe,
                          aperture_radius, inner_sky_radius, outer_sky_radius,
                          gain=1, zeropoint=0, suffix='.fit'):

    for fits_file in fits_files:
        fits_file_path = os.path.join(fits_root, fits_file)
        hdus = fits.open(fits_file_path)

        instr_mags = []
        for x, y in coords:
            time, mag = aperture_photometry(hdus[data_index], x, y,
                                            aperture_radius, inner_sky_radius, outer_sky_radius,
                                            gain=gain, zeropoint=zeropoint)
            instr_mags.append(mag)

        dataframe[fits_file[0:fits_file.rindex(suffix)]] = [time] + instr_mags


I am working on the file "LocalCoordAperturePhotometry.ipynb".
The first task is:
Inputs
Change these to suit your environment
File settings
Can you write Python code for it?

# Instrumental magnitude output file path
instr_mag_file_root = "/Users/david/aavso/dslr-photometry/working"
instr_mag_csv_file = "instr_mags.csv"

# FITS file directory
fits_root = "/Users/david/aavso/dslr-photometry/working"

# B, G, and R FITS file prefixes to identify files,
# e.g. stk-median-g matches stk-median-g1.fit, stk-median-g2.fit, ...
fits_prefixes = ["stk-median-b", "stk-median-g", "stk-median-r"]

# FITS file data HDU index
data_index = 0

# Time column name
time_name = "JD"


I am working on the file "LocalCoordAperturePhotometry.ipynb".
The first task is:
Map of object designations to local coordinates
Can you write Python code for it?


I am working on the file "LocalCoordAperturePhotometry.ipynb".
The first task is:
Outputs
Find B, G, R files in the FITS file directory
Can you write Python code for it?

files = os.listdir(fits_root)

fits_files = []
for fits_prefix in fits_prefixes:
    fits_files += sorted([file for file in files if fits_prefix in file])


I am working on the file "LocalCoordAperturePhotometry.ipynb".
The first task is:
Single image+coordinate photometry, returning a time and instrumental magnitude
Invoked by multi_file_photometry()
Can you write Python code for it?

def aperture_photometry(hdu, x, y,
                        aperture_radius, inner_sky_radius, outer_sky_radius,
                        gain=1, zeropoint=0):

    image_data = hdu.data
    time = hdu.header[time_name]

    mag, magerr, flux, fluxerr, sky, skyerr, badflag, outstr = \
                aper.aper(image_data, x, y, phpadu=gain,
                          apr=aperture_radius, zeropoint=zeropoint,
                          skyrad=[inner_sky_radius, outer_sky_radius],
                          exact=True)

    return time, mag[0]


I am working on the file "Log analysis presentation.ipynb".
The first task is:
Converting the data to a pandas DataFrame only requires passing the list to pd.DataFrame
Can you write Python code for it?

log = pd.DataFrame(log_list)

print(log.head(1))


I am working on the file "Log analysis presentation.ipynb".
The first task is:
Currently, all columns are object type. The data will be easier to work with if the types are converted where appropriate

time_local -> datetime
upstream_response_time -> float
request_time -> float
body_bytes_sent -> integer

status is a numerical code so could be converted to an integer but as this is a categorical variable we probably don't need to convert it
Can you write Python code for it?

log['time_local'] = pd.to_datetime(log['time_local'].apply(lambda x : x.split(' ')[0]), format='%d/%b/%Y:%H:%M:%S')

log.loc[log['upstream_response_time'] == '-', 'upstream_response_time'] = np.nan
log[['upstream_response_time', 'request_time']] = log[['upstream_response_time', 'request_time']].astype(np.float64)
log['body_bytes_sent'] = log['body_bytes_sent'].astype(np.int64)

log.info()


I am working on the file "Log analysis presentation.ipynb".
The first task is:
Data import - here be regex
The log format I use for nginx:
'$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time $pipe';

The following regex will match
Can you write Python code for it?

pattern = re.compile(r"""(?P<remote_addr>.+?) \s-\s
            (?P<remote_user>.+?) \s
            \[(?P<time_local>.+?)\] \s
            "(?P<method>.+?) \s (?P<uri>.+?) \s (?P<protocol>.+?)" \s
            (?P<status>.+?) \s
            (?P<body_bytes_sent>.+?) \s
            "(?P<http_referer>.+?)"? \s # Optional final " as I initially missed this when setting the format
            "(?P<http_user_agent>.*?)" \s
            (?P<request_time>.+?) \s
            (?P<upstream_response_time>.+?) \s
            (?P<pipe>.+?)""", re.VERBOSE)

test = """195.211.155.181 - - [18/Jan/2015:22:43:05 +0000] "GET /blog/email-templates-using-zend-framework/ HTTP/1.1" """ + \
        """200 7408 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; """ + \
        """Windows NT 5.1; SV1) )" 0.039 0.039 ."""

m = re.match(pattern, test)
m.groupdict()


I am working on the file "Log analysis presentation.ipynb".
The first task is:
Inspecting the dataframe
Can you write Python code for it?


I am working on the file "Log analysis presentation.ipynb".
The first task is:
Some of the requests take a long time. Let's take a closer look
Response time analysis
Can you write Python code for it?

mask = log['upstream_response_time'] > 1.0
log[mask].shape

selected_columns = ['request_time', 'status', 'upstream_response_time', 'uri']
log.loc[mask, selected_columns].sort('upstream_response_time', ascending=False)


I am working on the file "Log analysis presentation.ipynb".
The first task is:
The slowest requests are on the back end but a couple of blog posts and the atom feed also appear to be slow. Are they always slow or were these requests anomalous?
Can you write Python code for it?

mask = log['uri'] == '/blog/atom/'
log[mask].describe()

np.percentile(log.loc[log['uri'] == '/blog/atom/', 'upstream_response_time'], 95)


I am working on the file "Log analysis presentation.ipynb".
The first task is:
Traffic analysis
Daily requests
Can you write Python code for it?

log_freq = log['body_bytes_sent'].resample('D', how='count')

log_freq.plot()


I am working on the file "Log analysis presentation.ipynb".
The first task is:
With our working regex pattern we can parse every line in the log file and store the data in a list of dicts
Can you write Python code for it?

log_file = '/home/jonathan/dev/access.log'

with open(log_file) as f:
    log_list = []
    for i in f.readlines():
        m = re.match(pattern, i)
        log_list.append(m.groupdict())


I am working on the file "LogGabor_fit_example.ipynb".
The first task is:
Back to top
performing a fit
Can you write Python code for it?

from LogGabor import LogGaborFit
lg = LogGaborFit(parameterfile)
lg.set_size((32, 32))

x_pos, y_pos, theta, sf_0  = 14.6, 8.5, 12 * np.pi / 180., .1
data = lg.invert(lg.loggabor(x_pos, y_pos, sf_0=np.absolute(sf_0), B_sf=lg.pe.B_sf, theta=theta, B_theta=lg.pe.B_theta))
data /= np.abs(data).max()
data_noisy = data + .25*np.random.normal(size=data.shape)

data_fitted, params = lg.LogGaborFit(data_noisy.reshape(lg.pe.N_X, lg.pe.N_Y))

data_fitted.shape

params.pretty_print()

extent = (0, lg.pe.N_X, 0, lg.pe.N_Y)
print ('params :', params, ', true : ', x_pos, y_pos, theta, sf_0)
fig, axs = plt.subplots(1, 3, figsize=(15, 5))
_ = axs[0].contourf(data.reshape(lg.pe.N_X, lg.pe.N_Y), 8, extent=extent, cmap=plt.cm.viridis, origin='upper')
_ = axs[1].imshow(data_noisy.reshape(lg.pe.N_X, lg.pe.N_Y), cmap=plt.cm.viridis, extent=extent)
_ = axs[2].contourf(data_fitted.reshape(lg.pe.N_X, lg.pe.N_Y), 8, extent=extent, cmap=plt.cm.viridis, origin='upper')
for ax in axs: ax.axis('equal')


I am working on the file "LogGabor_fit_example.ipynb".
The first task is:
Perspectives: Better fits of the filters
Basically, it is possible to infer the best possible log-Gabor function, even if it's parameters do not fall on the grid
Defining a reference log-gabor  (look in the corners!)
Can you write Python code for it?

def twoD_Gaussian(xy, x_pos, y_pos, theta, sf_0):
    FT_lg = lg.loggabor(x_pos, y_pos, sf_0=np.absolute(sf_0), B_sf=lg.pe.B_sf, theta=theta, B_theta=lg.pe.B_theta)
    return lg.invert(FT_lg).ravel()

# Create x and y indices
x = np.arange(lg.pe.N_X)
y = np.arange(lg.pe.N_Y)
x, y = xy =  np.meshgrid(x, y)

#create data
x_pos, y_pos, theta, sf_0  = 14.6, 8.5, 12 * np.pi / 180., .1
data = twoD_Gaussian(xy, x_pos, y_pos, theta=theta, sf_0=sf_0)

# plot twoD_Gaussian data generated above
#plt.figure()
#plt.imshow(data.reshape(lg.pe.N_X, lg.pe.N_Y))
#plt.colorbar()

# add some noise to the data and try to fit the data generated beforehand
data /= np.abs(data).max()
data_noisy = data + .25*np.random.normal(size=data.shape)
# getting best match
C = lg.linear_pyramid(data_noisy.reshape(lg.pe.N_X, lg.pe.N_Y))
idx = lg.argmax(C)
initial_guess = [idx[0], idx[1], lg.theta[idx[2]], lg.sf_0[idx[3]]]
print ('initial_guess :', initial_guess, ', idx :', idx)

import scipy.optimize as opt

popt, pcov = opt.curve_fit(twoD_Gaussian, xy, data_noisy, p0=initial_guess)

data_fitted = twoD_Gaussian(xy, *popt)

extent = (0, lg.pe.N_X, 0, lg.pe.N_Y)
print ('popt :', popt, ', true : ', x_pos, y_pos, theta, sf_0)
fig, axs = plt.subplots(1, 3, figsize=(15, 5))
_ = axs[0].contourf(data.reshape(lg.pe.N_X, lg.pe.N_Y), 8, extent=extent, cmap=plt.cm.viridis, origin='upper')
_ = axs[1].imshow(data_noisy.reshape(lg.pe.N_X, lg.pe.N_Y), cmap=plt.cm.viridis, extent=extent)
_ = axs[2].contourf(data_fitted.reshape(lg.pe.N_X, lg.pe.N_Y), 8, extent=extent, cmap=plt.cm.viridis, origin='upper')
for ax in axs: ax.axis('equal')


I am working on the file "LogGabor_fit_example.ipynb".
The first task is:
To install the dependencies related to running this notebook, see Installing notebook dependencies.
Back to top
Can you write Python code for it?

import os
import numpy as np
np.set_printoptions(formatter={'float': '{: 0.3f}'.format})
%matplotlib inline
import matplotlib.pyplot as plt
fig_width = 12
figsize=(fig_width, .618*fig_width)


I am working on the file "LogReg-sklearn.ipynb".
The first task is:
As one can see, the "dataset cannot be separated into positive and negative examples by a straight-line through the plot." cf. ex2.pdf
We're going to need polynomial terms to map onto.
Use this code: cf. Underfitting vs. Overfitting¶
Can you write Python code for it?

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures

polynomial_features = PolynomialFeatures(degree=6,include_bias=False)

pipeline = Pipeline([("polynomial_features", polynomial_features),("logistic_regression",logreg2)])

pipeline.fit(X_ex2data2,Y_ex2data2)

Z_ex2data2 = pipeline.predict(np.c_[xx_ex2data2.ravel(),yy_ex2data2.ravel()])

Z_ex2data2 = Z_ex2data2.reshape(xx_ex2data2.shape)
plt.figure(3)
plt.pcolormesh(xx_ex2data2,yy_ex2data2,Z_ex2data2)
plt.scatter(X_ex2data2[:, 0], X_ex2data2[:, 1], c=Y_ex2data2, edgecolors='k')
plt.show()


I am working on the file "LogReg-sklearn.ipynb".
The first task is:
Get the probability estimates; say a student has an Exam 1 score of 45 and an Exam 2 score of 85.
Can you write Python code for it?

logreg.predict_proba(np.array([[45,85]])).flatten()
print "The student has a probability of no admission of %s and probability of admission of %s" % tuple( logreg.predict_proba(np.array([[45,85]])).flatten()  )


I am working on the file "LogReg-sklearn.ipynb".
The first task is:
Let's load the data for Exercise 2 of Machine Learning, taught by Andrew Ng, of Coursera.
Can you write Python code for it?

ex2data1 = np.loadtxt("./Data/ex2data1.txt",delimiter=',')  # you, the user, may have to change this, if the directory that you're running this from is somewhere else
ex2data2 = np.loadtxt("./Data/ex2data2.txt",delimiter=',')

X_ex2data1 = ex2data1[:,0:2]
Y_ex2data1 = ex2data1[:,2]
X_ex2data2 = ex2data2[:,:2]
Y_ex2data2 = ex2data2[:,2]

logreg.fit(X_ex2data1,Y_ex2data1)

def trainingdat2mesh(X,marginsize=.5, h=0.2):
    rows, features = X.shape
    ranges = []
    for feature in range(features):
        minrange = X[:,feature].min()-marginsize
        maxrange = X[:,feature].max()+marginsize
        ranges.append((minrange,maxrange))
    if len(ranges) == 2:
        xx, yy = np.meshgrid(np.arange(ranges[0][0], ranges[0][1], h), np.arange(ranges[1][0], ranges[1][1], h))
        return xx, yy
    else:
        return ranges

xx_ex2data1, yy_ex2data1 = trainingdat2mesh(X_ex2data1,h=0.2)

Z_ex2data1 = logreg.predict(np.c_[xx_ex2data1.ravel(),yy_ex2data1.ravel()])

Z_ex2data1 = Z_ex2data1.reshape(xx_ex2data1.shape)
plt.figure(2)
plt.pcolormesh(xx_ex2data1,yy_ex2data1,Z_ex2data1)
plt.scatter(X_ex2data1[:, 0], X_ex2data1[:, 1], c=Y_ex2data1, edgecolors='k')
plt.show()


I am working on the file "Logic and Control Flow.ipynb".
The first task is:
Note that when the loop is exited via a break clause the else clause will not be executed.
Can you write Python code for it?

a, b = 0, 5
while a < b:
    print(a)
    a += 1
    if a > 3:
        break
else:
    print('loop over')


I am working on the file "Logic and Control Flow.ipynb".
The first task is:
Unlike most other programming languages however Python adds an else clause to the while loop. This takes the form:
python
a, b = 0, 5
while a < b:
    print(a)
    a += 1
else:
    print('loop over')
Can you write Python code for it?


I am working on the file "Logic and Control Flow.ipynb".
The first task is:
We can also unpack using wild cards:
python
tup = 1, 2, 3, 4
a, b, * = tup
This can also be done in the middle:
python
tup = 1, 2, 3, 4
a, *, b = tup
By convention if we want to disregard we use the underscore:
python
tup = 1, 2, 3, 4
a, _, _, b = tup
It is also possible to use a slice:
python
tup = 3, 7, 4, 1, 2
print(max(*tup[2:4]))
Can you write Python code for it?


I am working on the file "Logic and Control Flow.ipynb".
The first task is:
You can also provide an else clause that will be executed if the statement is not true:
python
x = int(input("Please enter an integer: "))
if x == 42:
    print('That is the answer!')
else:
    print('Wrong')
You can also chain together several conditions using another condition chained on to the else. Python supports this directly with the elif clause:
python
x = int(input("Is this a 5 or a 30 minute argument?"))
if x == 5:
    print('That will be five pounds please')
elif x == 30:
    print('That will be ten pounds please')
else:
    print('Sorry I can only offer 5 or 30 minutes')
Can you write Python code for it?


I am working on the file "Logic and Control Flow.ipynb".
The first task is:
range does not return the sequence directly, instead it returns an iterable that can be repeatedly called to get the next item in the sequence. This lazy evaluation allows you to use a very big sequence without the cost of creating all the items at once.
Python Tuple Unpacking
Tuples can be upacked:
python
tup = 1, 2
a, b = tup
Tuple unpacking can also be used for function calls:
python
tup = 3, 7
print(max(*tup))
Can you write Python code for it?


I am working on the file "Logic.ipynb".
The first task is:
All, Not All, Exist, Not Exist
When thinkging about the relationship between 2 statements $A$ and $B$, using Benn Diagram, and logic fomula makes it clear.

A is B, If A then B:  $$\forall _x(A(_x) \Rightarrow B(_x)) \ A \Rightarrow B$$
A is B and B is A: $$\forall _x(A(_x) \Leftrightarrow B(_x))$$
Some A is B: $$\exists _x(A(_x) \Rightarrow B(_x)) \ \exists _x(A(_x) \land B(_x))$$
A is not B: $$\lnot \exists _x(A(_x) \Rightarrow B(_x)) \ \lnot \exists _x(A(_x) \land B(_x))$$
Not all A is B:  $$\lnot \forall _x(A(_x) \Rightarrow B(_x)) \ \exists _x(A(_x) \Rightarrow \lnot B(_x)) \ \exists _x(A(_x) \land \lnot B(_x))$$
Can you write Python code for it?

# A is B

from matplotlib import pyplot as plt
from matplotlib_venn import venn2

s = (
    1,    # Ab
    0,    # aB
    0.3   # AB
)

v = venn2(subsets=s, set_labels=('(All)A is B; if A then B', '', ''))

# Subset labels
v.get_label_by_id('10').set_text('B')
v.get_label_by_id('01').set_text('')
v.get_label_by_id('11').set_text('A')
# v.get_label_by_id('111').set_text('')

plt.show()

# A is equivalent to B

from matplotlib import pyplot as plt
from matplotlib_venn import venn2

s = (
    0,    # Ab
    0,    # aB
    1   # AB
)

v = venn2(subsets=s, set_labels=('', '', 'A is equivalent to B'))

# Subset labels
v.get_label_by_id('10').set_text('')
v.get_label_by_id('01').set_text('')
v.get_label_by_id('11').set_text('A and B')

plt.show()

# Some A is B

from matplotlib import pyplot as plt
from matplotlib_venn import venn2

s = (
    1,    # Ab
    1,    # aB
    0.3   # AB
)

v = venn2(subsets=s, set_labels=('Some A is B', '', ''))

# Subset labels
v.get_label_by_id('10').set_text('A but Not B')
v.get_label_by_id('01').set_text('B')
v.get_label_by_id('11').set_text('A and B')

plt.show()

# No A is B

from matplotlib import pyplot as plt
from matplotlib_venn import venn2

s = (
    1,    # Ab
    1,    # aB
    0   # AB
)

v = venn2(subsets=s, set_labels=('No A is B', '', ''))

# Subset labels
v.get_label_by_id('10').set_text('A')
v.get_label_by_id('01').set_text('B')
#v.get_label_by_id('11').set_text('Some A')

plt.show()


I am working on the file "Logic.ipynb".
The first task is:
Please install matplotlib-venn package in Anaconda before executing Python scripts in this notebook.
$ conda install -c conda-forge matplotlib-venn
Logic
Terminology

Statement: (Proposition). a sentence that is  either True or False. (e.g. "A is B")
Premise: Precondition (Assumption). Statement that provide reason or support for conclution.
Conclution: a statement that the argument is trying to prove.
Argument: a group of statements that include one or more premises and 1 conclusion.

Example:
All humans are mortal ($P_1$). Socrates is human ($P_2$). Therefore: Socrates is mortal ($C$).

Argument: $P_1 \land P_2 \Rightarrow C$
Premise: $P_1$, $P_2$
Conclusion $C$

This is a typical Syllogism.
Can you write Python code for it?

# Draw benn diaglam of Socrates

from matplotlib import pyplot as plt
from matplotlib_venn import venn3, venn3_circles

s = (
    0,    # Abc
    0,    # aBc
    0,    # ABc
    4,    # abC
    0,    # AbC
    2,    # aBC
    0.2,    # ABC
)

v = venn3(subsets=s, set_labels=('Socrates', '', ''))

# Subset labels
# v.get_label_by_id('100').set_text('')
# v.get_label_by_id('010').set_text('aBc')
#v.get_label_by_id('110').set_text('Human')
v.get_label_by_id('001').set_text('Mortal')
# v.get_label_by_id('101').set_text('aBc')
v.get_label_by_id('011').set_text('Human')
v.get_label_by_id('111').set_text('')

# Subset colors
# v.get_patch_by_id('100').set_color('c')
# v.get_patch_by_id('010').set_color('#993333')
# v.get_patch_by_id('110').set_color('blue')

# Subset alphas
# v.get_patch_by_id('101').set_alpha(0.4)
# v.get_patch_by_id('011').set_alpha(1.0)
# v.get_patch_by_id('111').set_alpha(0.7)

# Border styles
#c = venn3_circles(subsets=s, linestyle='solid')
#c[0].set_ls('dotted')  # Line style
#c[1].set_ls('dashed')
#c[2].set_lw(1.0)       # Line width

plt.show()


I am working on the file "Logic.ipynb".
The first task is:
Problem 2

$P_1$: If you overslept, you'll be late
$P_2$: You aren't late

Therefore:

You did oversleep
You didn't oversleep
You're late
None of these validly follows
Can you write Python code for it?

from matplotlib import pyplot as plt
from matplotlib_venn import venn3, venn3_circles

s = (
    1,    # Abc
    0,    # aBc
    0,    # ABc
    2,    # abC
    0.5,    # AbC
    1,    # aBC
    0.5,    # ABC
)

v = venn3(subsets=s, set_labels=('You', '', ''))

# Subset labels
v.get_label_by_id('100').set_text('P2')
# v.get_label_by_id('010').set_text('aBc')
# v.get_label_by_id('110').set_text('')
v.get_label_by_id('001').set_text('Late')
v.get_label_by_id('101').set_text('')
v.get_label_by_id('011').set_text('Sleep, P1')
v.get_label_by_id('111').set_text('P1')

# Subset colors
v.get_patch_by_id('100').set_color('c')
# v.get_patch_by_id('010').set_color('#993333')
v.get_patch_by_id('101').set_color('white')

# Subset alphas
# v.get_patch_by_id('101').set_alpha(0.4)
# v.get_patch_by_id('011').set_alpha(1.0)
# v.get_patch_by_id('111').set_alpha(0.7)

plt.show()


I am working on the file "Logic.ipynb".
The first task is:
Problem 3

$P_1$: No one held for murder is given bail
$P_2$: Smith isn't held for murder

Therefore:

Smith is given bail
Smith isn't given bail
Smith is innocent
None of these validly follows

M: (Held for murder), B: (is given bail) $$(M \cap B = \emptyset) \equiv (M \rightarrow \lnot B)$$
Can you write Python code for it?

from matplotlib import pyplot as plt
from matplotlib_venn import venn2

s = (
    1,    # Ab
    1,    # aB
    0.00001     # AB
)

v = venn2(subsets=s, set_labels=('', '', ''))

# Subset labels
v.get_label_by_id('10').set_text('Murder')
v.get_label_by_id('01').set_text('Given Bail')
v.get_label_by_id('11').set_text('')
# v.get_label_by_id('111').set_text('')

plt.show()

from matplotlib import pyplot as plt
from matplotlib_venn import venn3, venn3_circles

s = (
    1,    # Abc
    0,    # aBc
    0,    # ABc
    2,    # abC
    0.5,    # AbC
    1,    # aBC
    0,    # ABC
)

v = venn3(subsets=s, set_labels=('', 'No Bail', ''))

# Subset labels
v.get_label_by_id('100').set_text('Smith, P2')
# v.get_label_by_id('010').set_text('aBc')
# v.get_label_by_id('110').set_text('')
v.get_label_by_id('001').set_text('')
v.get_label_by_id('101').set_text('')
v.get_label_by_id('011').set_text('Murder, P1')
# v.get_label_by_id('111').set_text('')

# Subset colors
v.get_patch_by_id('100').set_color('c')
# v.get_patch_by_id('010').set_color('#993333')
v.get_patch_by_id('101').set_color('white')

# Subset alphas
# v.get_patch_by_id('101').set_alpha(0.4)
# v.get_patch_by_id('011').set_alpha(1.0)
# v.get_patch_by_id('111').set_alpha(0.7)

plt.show()


I am working on the file "Logic.ipynb".
The first task is:
Problem 4

$P_1$: No court that suppresses evidence is impartial
$P_2$: Some court subject to politicalpressure suppress evidence

Therefore:

Some courts subject to political pressure aren't impartial
No courts subject to political pressure are impartial
Some courts subject to political pressure are impartial
None of these validly follows
Can you write Python code for it?

from matplotlib import pyplot as plt
from matplotlib_venn import venn3, venn3_circles

s = (
    1,    # Abc
    0,    # aBc
    0,    # ABc
    2,    # abC
    0.5,    # AbC
    1,    # aBC
    0.5,    # ABC
)

v = venn3(subsets=s, set_labels=('', '', ''))

# Subset labels
v.get_label_by_id('100').set_text('some court')
# v.get_label_by_id('010').set_text('aBc')
# v.get_label_by_id('110').set_text('')
v.get_label_by_id('001').set_text('Not impartial')
v.get_label_by_id('101').set_text('')
v.get_label_by_id('011').set_text('evidence, P1')
v.get_label_by_id('111').set_text('P1, P2')

# Subset colors
v.get_patch_by_id('100').set_color('c')
# v.get_patch_by_id('010').set_color('#993333')
v.get_patch_by_id('101').set_color('white')

# Subset alphas
# v.get_patch_by_id('101').set_alpha(0.4)
# v.get_patch_by_id('011').set_alpha(1.0)
v.get_patch_by_id('111').set_alpha(0.7)

plt.show()


I am working on the file "Logic.ipynb".
The first task is:
Problem 5
If you overslept, you'll be late $P_1$. You didn't oversleep $P_2$. Therefore:

You're late
You aren't late
You did oversleep
None of these validly follows

(see the diagram of Problem 2)
Problem 6

$P_1$: Every revolution is a trade disruption
$P_2$: Some trade disruptions cause financial anxiety

Therefore:

Some revolutions cause financial anxiety
Some revolutions don't cause financial anxiety
All trade disruptions cause financial anxiety
Non of above validly follows

As for relation between Revolution and Financial Anxiety, there are 2 possible combinations. See below.
Can you write Python code for it?

from matplotlib import pyplot as plt
from matplotlib_venn import venn3, venn3_circles

s = (
    1,    # Abc
    0,    # aBc
    0,    # ABc
    2,    # abC
    0.5,    # AbC
    1,    # aBC
    0.5,    # ABC
)

v = venn3(subsets=s, set_labels=('', '', ''))

# Subset labels
v.get_label_by_id('100').set_text('Financial Anxiety')
# v.get_label_by_id('010').set_text('aBc')
# v.get_label_by_id('110').set_text('')
v.get_label_by_id('001').set_text('Trade disruption')
v.get_label_by_id('101').set_text('P2')
v.get_label_by_id('011').set_text('Revolution P1')
v.get_label_by_id('111').set_text('P1, P2')

# Subset colors
v.get_patch_by_id('100').set_color('c')
# v.get_patch_by_id('010').set_color('#993333')
v.get_patch_by_id('111').set_color('white')

# Subset alphas
# v.get_patch_by_id('101').set_alpha(0.4)
# v.get_patch_by_id('011').set_alpha(1.0)
v.get_patch_by_id('111').set_alpha(0.7)

plt.show()

from matplotlib import pyplot as plt
from matplotlib_venn import venn3, venn3_circles

s = (
    1,    # Abc
    0,    # aBc
    0,    # ABc
    2,    # abC
    0.5,    # AbC
    1,    # aBC
    0,    # ABC
)

v = venn3(subsets=s, set_labels=('', '', 'Trade disruption'))

# Subset labels
v.get_label_by_id('100').set_text('Financial Anxiety')
# v.get_label_by_id('010').set_text('aBc')
# v.get_label_by_id('110').set_text('')
v.get_label_by_id('001').set_text('')
v.get_label_by_id('101').set_text('P2')
v.get_label_by_id('011').set_text('Revolution P1')
# v.get_label_by_id('111').set_text('')

# Subset colors
v.get_patch_by_id('100').set_color('c')
# v.get_patch_by_id('010').set_color('#993333')
# v.get_patch_by_id('110').set_color('blue')

# Subset alphas
# v.get_patch_by_id('101').set_alpha(0.4)
# v.get_patch_by_id('011').set_alpha(1.0)
# v.get_patch_by_id('111').set_alpha(0.7)

plt.show()


I am working on the file "Logic.ipynb".
The first task is:
Problem 8

$P_1$: If there's knowledge, then either some things are known without proof or we can prove every premise by previous arguments infinitely
$P_2$: We can't prove every premise by previous arguments infinitely
$P_3$: There's knowledge

Therefore:

Everything that's known is provable
There's no knowledge
Some things are known without proof
None of these validly follows

From Premise $P_1$ and $P_3$, we can make a new premise, and discard $P_1$ and $P_3$.

$P_4$: Either some things are known without proof or we can prove every premise by previous arguments infinitely
Can you write Python code for it?

from matplotlib import pyplot as plt
from matplotlib_venn import venn2

s = (
    1,    # Ab
    0,    # aB
    0.3   # AB
)

v = venn2(subsets=s, set_labels=('', '', ''))

# Subset labels
v.get_label_by_id('10').set_text('with/wo Proof')
v.get_label_by_id('01').set_text('')
v.get_label_by_id('11').set_text('Knowledge, P1')
# v.get_label_by_id('111').set_text('')

plt.show()

from matplotlib import pyplot as plt
from matplotlib_venn import venn3, venn3_circles

s = (
    1,    # Abc
    1,    # aBc
    0.5,    # ABc
    1,    # abC
    0.5,    # AbC
    0,    # aBC
    0.0001,    # ABC
)

v = venn3(subsets=s, set_labels=('', '', ''))

# Subset labels
v.get_label_by_id('100').set_text('')
v.get_label_by_id('010').set_text('Some: No proof')
v.get_label_by_id('110').set_text('P1,P3')
v.get_label_by_id('001').set_text('Every: with proof, P2')
v.get_label_by_id('101').set_text('P1,P2,P3')
#v.get_label_by_id('011').set_text('')
v.get_label_by_id('111').set_text('')

# Subset colors
v.get_patch_by_id('100').set_color('white')
# v.get_patch_by_id('010').set_color('#993333')
v.get_patch_by_id('001').set_color('red')
#v.get_patch_by_id('011').set_color('red')
#v.get_patch_by_id('111').set_color('red')
v.get_patch_by_id('101').set_color('red')

# Subset alphas
# v.get_patch_by_id('101').set_alpha(0.4)
# v.get_patch_by_id('011').set_alpha(1.0)
# v.get_patch_by_id('111').set_alpha(0.7)

plt.show()


I am working on the file "Logic.ipynb".
The first task is:
Problem 9

$P_1$: No person desiring to help others is reluctant to make scrifices
$P_2$: Some masochists aren't reluctant to make sacrifices

Therefore:

Some masochists desire to help others
Some masochists don't desire to help others
All masochists desire to help others
None of these validly follows
Can you write Python code for it?

from matplotlib import pyplot as plt
from matplotlib_venn import venn3, venn3_circles

s = (
    2,    # Abc
    2,    # aBc
    0,    # ABc
    1,    # abC
    0.5,    # AbC
    0.5,    # aBC
    0,    # ABC
)

v = venn3(subsets=s, set_labels=('', '', ''))

# Subset labels
v.get_label_by_id('100').set_text('des-help, P1')
v.get_label_by_id('010').set_text('r-sacri, P1')
# v.get_label_by_id('110').set_text('')
v.get_label_by_id('001').set_text('Maso,P2')
v.get_label_by_id('101').set_text('')
v.get_label_by_id('011').set_text('')
#v.get_label_by_id('111').set_text('P1, P2')

# Subset colors
v.get_patch_by_id('100').set_color('c')
# v.get_patch_by_id('010').set_color('#993333')
v.get_patch_by_id('101').set_color('white')

# Subset alphas
# v.get_patch_by_id('101').set_alpha(0.4)
# v.get_patch_by_id('011').set_alpha(1.0)
# v.get_patch_by_id('111').set_alpha(0.7)

plt.show()


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Algoritmo de Davis-Putnam
Can you write Python code for it?

def inconsistente(cs):
    """Detecta si cs es satisfacible usando Davis-Putnam"""
    css = quita_tautologias(quita_redundantes(cs))
    unitpuro=True
    while unitpuro:
        cssn = clausula_unit(css)
        if cssn== css:
            cssn = literal_puro(css)
            if cssn==css:
                unitpuro=False
        if len(cssn)==0:
            print("Hemos llegado al conjunto vacío")
            return False
        if False in cssn:
            print("Hemos encontrado la cláusula vacía")
            return True
        css=cssn

    d = divide(css)
    return inconsistente(d[0]) and inconsistente(d[1])

inconsistente([~a,a])

clausula_unit([a,b])

clausula_unit([b])

inconsistente([a,b])

inconsistente([~a|b,a,~b])


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Davis-Putnam
Forma clausular y operaciones con literales
Vamos a definir la forma clausular de un conjunto de proposiciones
Can you write Python code for it?

G= {c>>(a|b), b>>(c>>a), d&~(c>>a)}

def forma_clausular(G):
    fc = set([])
    for g in G:
        q = to_cnf(g)
        if isinstance(q,And):
            fc.update(q.args)
        else:
            fc.add(q)
    return fc

forma_clausular(G)

def es_literal(p):
    if isinstance(p,Symbol):
        return True
    if isinstance(p,Not):
        ar = p.args
        if len(ar)==1 and isinstance(ar[0],Symbol):
            return True
    return False

es_literal(a)

es_literal(~a)

es_literal(a&b)

v=Or(*[])

type(v)

v==False

def es_clausula(p):
    if p==False:
        return True
    if es_literal(p):
        return True
    if isinstance(p,Or):
        ar = p.args
        return all(es_literal(a) for a in ar)
    return False

es_clausula(a&b)

es_clausula(a|~b)

def complemento(p):
    if isinstance(p,Symbol):
        return ~p
    if isinstance(p,Not):
        return p.args[0]
    return None

complemento(a)

complemento(~a)

l=[1,2,3]
l.remove(3)
2 in l

def quita(c,l):
    if c==False:
        return False
    if es_literal(c):
        if c==l:
            return False
        return c
    ar = list(c.args)
    if l in ar:
        ar.remove(l)
        return Or(*ar)
    return c

quita(a|b,b)

quita(a|b,~b)

def primero(xs,cond):
    for x in xs:
        if cond(x):
            return x
    return None

primero([1,2,3],lambda x:(x%2)==0)

def clausula_a_lista(c):
    if c==False:
        return []
    if es_literal(c):
        return [c]
    return list(c.args)

clausula_a_lista(a|b|~c)


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Detectando una clausula unit
Can you write Python code for it?

def clausula_unit(cs):
    l = primero(cs,es_literal)
    if l!=None:
        print("Hemos encontrado una clausula unit ", l)
        csp = [c for c in cs if not(l in clausula_a_lista(c))]
        cspq = [quita(c,complemento(l)) for c in csp]
        pprint(cspq)
        return cspq
    return cs

clausula_unit([a,a|b, ~a|c, c|b])

Or(a,~a,b)

simplify(_)


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Dividiendo cuando no hay literales puros ni cláuslas unit
Can you write Python code for it?

def divide(cs):
    literales = set([])
    for c in cs:
        literales.update(clausula_a_lista(c))
    l = primero(literales, lambda x:(complemento(x) in literales))
    cl = complemento(l)
    csl = set([quita(c,l) for c in cs if l in clausula_a_lista(c)])
    csnl= set([quita(c,cl) for c in cs if cl in clausula_a_lista(c)])
    csc = set([c for c in cs if not(l in clausula_a_lista(c)) and not(cl in clausula_a_lista(c))])
    parte1=csl.union(csc)
    parte2=csnl.union(csc)
    print("Dividimos usando ",l)
    pprint(parte1)
    pprint(parte2)
    return [csl.union(csc),csnl.union(csc)]

divide([a|b,~a|b,a|c, b|c])


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Ejemplo Veamos que $${(a\to \neg b\vee d)\wedge(b\wedge \neg d\to a\vee c), (d\to (a \leftrightarrow \neg b)\vee (b\wedge \neg c)}\models (\neg b\to (d\wedge (c\vee \neg d)))\to c\wedge d$$
Can you write Python code for it?

Gamma={(a>>(~b|d))&((b&~d)>>(a|c)), (d>>(Equivalent(a,~b))|(b|~c)),
       (~b>>(d&(c|~d))),~(c&d)}

forma_clausular(Gamma)

inconsistente(forma_clausular(Gamma))


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Este último ejemplo es precisamente el modus ponens
Can you write Python code for it?


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Implicación semántica
Podemos detectar si $\Gamma\models a$ con la siguiente función
Can you write Python code for it?

def implica(gamma,a):
    """ Determina si Gamma implica semanticamente a"""
    p = True
    for x in gamma:
        p = p & x
    p=p&(~a)
    return not(satisfiable(p))

implica([a,a>>b],b)

implica([a>>b,b>>c],a>>c)


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
La siguiente función determina si todos los literales de una cláusula están contenidos en otra cláusula
Can you write Python code for it?

def contenido(c1,c2):
    return all((l in clausula_a_lista(c2)) for l in clausula_a_lista(c1))

contenido(a|b,a|b|c)

contenido(a|b,a|c)

def quita_redundantes(cs):
    return [c for c in cs if not(any(contenido(d,c) and d!=c for d in cs))]

quita_redundantes([a|b,a|b|c])


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Otro ejemplo
Can you write Python code for it?


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Quitando tautologías y cláusulas redundantes
Can you write Python code for it?

def quita_tautologias(cs):
    return [c for c in cs if simplify(c)!=True]

quita_tautologias([a|b|~a,a|c])

l=[1,2]


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Resolventes
Can you write Python code for it?

Or(Or(a,b),Or(a,c))

def resolventes(a,b):
    literales=[l for l in clausula_a_lista(a) if complemento(l) in clausula_a_lista(b)]
    return [Or(quita(a,l),quita(b,complemento(l))) for l in literales]

resolventes(~a|b,a|~b)


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Satisfacibilidad
Ejemplo Si es cierto $a\to b$, ¿podemos inferir algo sobre $a\vee c\to b\vee c$? ¿Y sobre $a\wedge c\to b\wedge c$?
Can you write Python code for it?


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
También podíamos ver si $p\to q$ es una tautología por el teorema de la deducción, o lo que es lo mismo, mostrar que siempre es verdadera
Can you write Python code for it?


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Tautologías
Una proposición $a$ es tautología si $\models a$
Can you write Python code for it?

implica([], a>>(b>>a))

implica([],~a>>(a>>b))

def tautologia(a):
    return not(satisfiable(~a))

tautologia(~a>>(a>>b))

tautologia(a>>b)

(a>>b).args

to_cnf(a>>(b&c))

type(_)

to_cnf(a>>b)

type(_)


I am working on the file "Logica Proposicional/Logica proposicional.ipynb".
The first task is:
Volvamos al primer ejemplo
Can you write Python code for it?


I am working on the file "Logistic Matrix Factorization.ipynb".
The first task is:
Binary relationships
Can you write Python code for it?

%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import scipy as sc
import pandas as pd

M = 8
L = 2

A = np.zeros((M,M,L))
Mask = np.ones((M,M))
for i in range(M):
    for j in range(M):
        k = ((i+1)*j)%L | int(j>i+2)
        #k = 1 if i==j else 0
        A[i,j,k] = 1

Y = A[:,:,1]
plt.imshow(Y, cmap='gray_r')
plt.show()

W, H = LogisticMF(Y, K=4, Mask=Mask, MAX_ITER = 20000)


I am working on the file "Logistic Matrix Factorization.ipynb".
The first task is:
Now, we will pretend as if we don't know the underlying relation and, moreover that only a subset of the entries of the matrix are unknown to us. We will denote this by a mask matrix defined as
$$
M_{ij} = \left{\begin{array}{cc} 1 & \text{if}\;X_{ij}\; \text{is observed} \ 0 & \text{if not observed} \end{array} \right.
$$
We will assume every element is observed with a probability $\pi$; this will mimic a real situation where some
items are unknown.
Can you write Python code for it?

Mask = np.random.rand(M, N) < 0.4
Y = Y_full.copy()
Y[Mask==False] = np.nan
plt.imshow(Y, cmap='bwr')
plt.show()

print('Missing(%) : ', np.sum(1-Mask)/(N*M))


I am working on the file "Logistic Matrix Factorization.ipynb".
The first task is:
The following function implements an alternating gradient ascent algorithm for Logistic Matrix Factorization
Can you write Python code for it?

def LogisticMF(Y, K, Mask, eta=0.005, nu=0.1, MAX_ITER = 5000, PRINT_PERIOD=500):
    M = Y.shape[0]
    N = Y.shape[1]

    W = np.random.randn(M,K)
    H = np.random.randn(K,N)

    YM = Y.copy()
    YM[Mask==False] = 0

    for epoch in range(MAX_ITER):
        dLh = np.dot(W.T, YM-Mask*sigmoid(np.dot(W,H))) - nu*H
        H = H + eta*dLh
        dLw = np.dot(YM-Mask*sigmoid(np.dot(W,H)),H.T ) - nu*W
        W = W + eta*dLw

        if epoch % PRINT_PERIOD == 0:
            LL = np.sum( (YM*np.log(sigmoid(np.dot(W,H))) +  (Mask-YM)*np.log(1 - sigmoid(np.dot(W,H)))) ) - nu*np.sum(H**2)/2. - nu*np.sum(W**2)/2.
            print(epoch, LL)

    return W,H

W, H = LogisticMF(Y, K=2, Mask=Mask, MAX_ITER = 50000, PRINT_PERIOD=10000)

thr = 0.5

fig=plt.figure(figsize=(12, 6))
plt.subplot(1,5,1)
plt.imshow(Y_full, cmap='bwr', vmin=0, vmax=1)
plt.subplot(1,5,2)
plt.imshow(Y, cmap='bwr', vmin=0, vmax=1)
plt.subplot(1,5,3)
Y_pred = sigmoid(W.dot(H))
plt.imshow(Y_pred, cmap='bwr', vmin=0, vmax=1)
ax = plt.subplot(1,5,4)
Y_rec = Y_pred > thr
plt.imshow(Y_rec, cmap='bwr', vmin=0, vmax=1)
ax2 = plt.subplot(1,5,5)
plt.imshow(Y_full - Y_rec, cmap='PiYG', vmin=-1, vmax=1)
#plt.show()
plt.close(fig)

def change_thr(thr):
    ax.cla()
    Y_rec = Y_pred > thr
    ax.imshow(Y_rec, cmap='bwr', vmin=0, vmax=1)
    ax2.imshow(Y_full - Y_rec, cmap='PiYG', vmin=-1, vmax=1)
    #plt.show()
    display(fig)

interact(change_thr, thr=(0.0, 1.0,0.01))

#change_thr(0.01)

def evaluate_results(target=target, probabilities=Y_pred, thr=0.5):
    estimate = probabilities > thr

    conf_matrix = {'TP': 0, 'TN': 0, 'FP':0, 'FN': 0}

    for i in range(len(target)):
        if target[i]==True and estimate[i]==True:
            conf_matrix['TP'] += 1
        elif target[i]==True and estimate[i]==False:
            conf_matrix['FN'] += 1
        elif target[i]==False and estimate[i]==False:
            conf_matrix['TN'] += 1
        elif target[i]==False and estimate[i]==True:
            conf_matrix['FP'] += 1

   ...