I am working on the file "11-notebook-solution.ipynb".
The first task is:
Aufgabe 4     Mehr Semantik für IE
Zusätzlich zu den in Aufgabe 2 behandelten Konstruktionen sollen jetzt auch negierte und komplexe Sätze mit Konjunktionen sinnvoll verarbeitet werden.
Eingabe:
I see an elephant.
You didn't see the elephant.
Peter saw the elephant and drank wine.

Gewünschte Ausgabe:
see(I, elephant)
not_see(You, elephant)
saw(Peter, elephant)
drank(Peter, wine)

Kopieren Sie am besten Ihren aktuellen Stand von oben herunter und fügen Sie Ihre Erweiterungen dann hier ein.
Can you write Python code for it?

def generate_predicates_for_sentence(sentence):
    # verbs contains everything that should be treated as verb (i.e. become a predicate)
    verbs = set()
    # sbj (obj) maps from verbs/predicates to its first (second) argument
    sbj = {}
    obj = {}
    # sbj_candidates maps from a verb to potential subjects
    # it is used when the verb-noun relation is not clear (e.g. nmod)
    sbj_candidates = defaultdict(list)
    # in case, we store information about the kind of PP some noun is found in
    case = {}
    # the negated-set should contain everything we find under negation
    negated = set()
    # the conj-dict should connect verbs that are coordinated
    # the mapping goes from later verbs to former ones
    conj = {}
    for result in dep_parser.raw_parse(sentence):
        relcl_trips = []
        for triple in result.triples():

            if triple[1] == "nsubj":
                verbs.add(triple[0][0])
                sbj[triple[0][0]] = triple[2]
            if triple[1] == "dobj":
                verbs.add(triple[0][0])
                obj[triple[0][0]] = triple[2]
            if triple[1] == "nsubjpass":
                verbs.add(triple[0][0])
                obj[triple[0][0]] = triple[2]
            if triple[0][1].startswith("V"):
                verbs.add(triple[0][0])
                if triple[1] == "nmod":
                    sbj_candidates[triple[0][0]].append(triple[2])
            if triple[1] == "acl:relcl":
                verbs.add(triple[2][0])
                relcl_trips.append(triple)
            if triple[1] == "case":
                case[triple[0]] = triple[2]
            if triple[1] == "neg":
                negated.add(triple[0][0])
            if triple[1] == "conj":
                verbs.add(triple[0][0])
                verbs.add(triple[2][0])
                conj[triple[2][0]] = triple[0][0]

        for triple in relcl_trips:
            if triple[2][0] in sbj:
                if sbj[triple[2][0]][1] in ["WP", "WDT"]:
                    sbj[triple[2][0]] =...


I am working on the file "11-notebook-solution.ipynb".
The first task is:
Hausaufgaben
Aufgabe 3     Parent Annotation
Parent Annotation kann die Performanz einer CFG wesentlich verbessern. Schreiben Sie eine Funktion, die einen gegebenen Syntaxbaum dieser Optimierung unterzieht. Auf diese Art und Weise transformierte Bäume können dann wiederum zur Grammatikinduktion verwendet werden.
parentHistory soll dabei die Anzahl der Vorgänger sein, die zusätzlich zum direkten Elternknoten berücksichtigt werden. (Kann bei der Lösung der Aufgabe auch ignoriert werden.)
parentChar soll ein Trennzeichen sein, das bei den neuen Knotenlabels zwischen dem ursprünglichen Knotenlabel und der Liste von Vorgängern eingefügt wird.
Can you write Python code for it?

def parent_annotation(tree, parentHistory=0, parentChar="^"):
    def pa_rec(node, parents):
        originalNode = node.label()

        parentString = parentChar + '<' + "-".join(parents) + '>'
        node.set_label(node.label() + parentString)

        for child in node:
            pa_rec(child, [originalNode] + parents[:parentHistory])

        return node

    return pa_rec(tree, [])

test_tree = nltk.Tree(
    "S",
    [
        nltk.Tree("NP", [
            nltk.Tree("DET", []),
            nltk.Tree("N", [])
        ]),
        nltk.Tree("VP", [
            nltk.Tree("V", []),
            nltk.Tree("NP", [
                nltk.Tree("DET", []),
                nltk.Tree("N", [])
            ])
        ])
    ]
)

parent_annotation(
   test_tree
)


I am working on the file "11-notebook.ipynb".
The first task is:
Aufgabe 4     Mehr Semantik für IE
Zusätzlich zu den in Aufgabe 2 behandelten Konstruktionen sollen jetzt auch negierte und komplexe Sätze mit Konjunktionen sinnvoll verarbeitet werden.
Eingabe:
I see an elephant.
You didn't see the elephant.
Peter saw the elephant and drank wine.

Gewünschte Ausgabe:
see(I, elephant)
not_see(You, elephant)
saw(Peter, elephant)
drank(Peter, wine)

Kopieren Sie am besten Ihren aktuellen Stand von oben herunter und fügen Sie Ihre Erweiterungen dann hier ein.
Can you write Python code for it?

def generate_predicates_for_sentence(sentence):
    pass
def generate_predicates_for_text(text):
    pass

text = """
I see an elephant.
You didn't see the elephant.
Peter saw the elephant and drank wine.
"""


I am working on the file "11-notebook.ipynb".
The first task is:
Hausaufgaben
Aufgabe 3     Parent Annotation
Parent Annotation kann die Performanz einer CFG wesentlich verbessern. Schreiben Sie eine Funktion, die einen gegebenen Syntaxbaum dieser Optimierung unterzieht. Auf diese Art und Weise transformierte Bäume können dann wiederum zur Grammatikinduktion verwendet werden.
parentHistory soll dabei die Anzahl der Vorgänger sein, die zusätzlich zum direkten Elternknoten berücksichtigt werden. (Kann bei der Lösung der Aufgabe auch ignoriert werden.)
parentChar soll ein Trennzeichen sein, das bei den neuen Knotenlabels zwischen dem ursprünglichen Knotenlabel und der Liste von Vorgängern eingefügt wird.
Can you write Python code for it?

def parent_annotation(tree, parentHistory=0, parentChar="^"):
    pass

test_tree = nltk.Tree(
    "S",
    [
        nltk.Tree("NP", [
            nltk.Tree("DET", []),
            nltk.Tree("N", [])
        ]),
        nltk.Tree("VP", [
            nltk.Tree("V", []),
            nltk.Tree("NP", [
                nltk.Tree("DET", []),
                nltk.Tree("N", [])
            ])
        ])
    ]
)

parent_annotation(
   test_tree
)


I am working on the file "11. Basic Statistics.ipynb".
The first task is:
Including DIAG9 is (for some reason) especially beneficial
What if claims were linked?
This section tries to predict how many triplets could be extracted from the data, if we had a fixed number of linked claims for each patient.
The PUF from DSHS does not contain linkable claims. This is just an estimate of what linkable claims would be able to give us.
Can you write Python code for it?

pdf2 = np.convolve(pdf, pdf)

plt.bar(range(0, len(pdf2)), pdf2)

pdf4 = np.convolve(pdf2, pdf2)

plt.bar(range(0, len(pdf4)), pdf4)

pdf8 = np.convolve(pdf4, pdf4)

pdf16 = np.convolve(pdf8, pdf8)

plt.bar(range(0, len(pdf16)), pdf16)

def calcNumberOfTrigramsPMF(nvars, pmf, claimsperpatient, nclaims=nclaims):
    pmfcut = pmf.copy()
    pmfcut[nvars+1:]=0
    pmfcut[nvars] = 1 - sum(pmfcut[:nvars])
    pconv = pmfcut.copy()
    for k in range(1,claimsperpatient):
        pconv = np.convolve(pconv.copy(),pmfcut)
    trigrams=[(nclaims/float(claimsperpatient))*pconv[i]*i*(i-1)*(i-2)/6 for i in range(3, len(pconv))]
    return sum(trigrams)

ntri = calcNumberOfTrigramsPMF(5, pdf, 1)
ntri

nvars = range(3,15)


I am working on the file "11. Basic Statistics.ipynb".
The first task is:
It consists of ~740K claims.
Can you write Python code for it?

nclaims = data.Discharge.count()

nclaims


I am working on the file "11. Basic Statistics.ipynb".
The first task is:
Using 2007 Q1 for now
Can you write Python code for it?


I am working on the file "11. Non supervised learning I.ipynb".
The first task is:
Example: Image denoising

(Source: https://blog.keras.io/building-autoencoders-in-keras.html)

Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np
from keras.datasets import mnist
from keras.layers import Dense
from keras.models import Sequential

(x_train, _), (x_test, _) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = np.reshape(x_train, (len(x_train), 784))
x_test = np.reshape(x_test, (len(x_test), 784))

noise_factor = 0.5
x_train_noisy = x_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train.shape)
x_test_noisy = x_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test.shape)
x_train_noisy = np.clip(x_train_noisy, 0., 1.)
x_test_noisy = np.clip(x_test_noisy, 0., 1.)

n = 10
plt.figure(figsize=(20, 2))
for i in range(n):
    ax = plt.subplot(1, n, i+1)
    plt.imshow(x_test_noisy[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()

model = Sequential()
model.add(Dense(128, activation='relu', input_dim=784))
model.add(Dense(64, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(784, activation='sigmoid'))
model.compile(optimizer='adam', loss='binary_crossentropy')

model.fit(x_train_noisy, x_train,
          nb_epoch=100,
          batch_size=256,
          shuffle=True,
          validation_data=(x_test_noisy, x_test))

decoded_imgs = model.predict(x_test)
n = 10
plt.figure(figsize=(20, 6))
for i in range(1, n):
    # display original
    ax = plt.subplot(3, n, i)
    plt.imshow(x_test[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # display noisy
    ax = plt.subplot(3, n, i + n)
    plt.imshow(x_test_noisy[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # display reconstruction
    ax = plt.subplot(3, n, i + 2*n)
    plt.imshow(decoded_imgs[i].reshape(28, 28))
    plt.gray()
   ...


I am working on the file "11. Non supervised learning I.ipynb".
The first task is:
Let's also create a separate encoder model and a separate decoder model:
Can you write Python code for it?

# this model maps an input to its encoded representation
encoding_model = Model(input_img, encoded)

# create a placeholder for an encoded input
# and create the decoder model
encoded_input = Input(shape=(encoding_dim,))
decoding_model = Model(encoded_input, decoding_layer(encoded_input))

autoencoder.compile(optimizer='adam', loss='mse')


I am working on the file "11. Non supervised learning I.ipynb".
The first task is:
Now we can create the decoder net:
Can you write Python code for it?

decoder_hidden = Dense(512, activation='relu')
h_p = decoder_hidden(z)

decoder_out = Dense(784, activation='sigmoid')
outputs = decoder_out(h_p)


I am working on the file "11. Non supervised learning I.ipynb".
The first task is:
Variational Autoencoders
A variational autoencoder is an autoencoder that adds probabilistic constraints on the representations being learned.

When using probabilistic models, compressed representation is called latent variable model.

So, instead of learning a function this model is learning a probabilistic distribution function that models your data.

Why? Standard autoencoders are not suited to work as a generative model. If you pick a random value for your decoder you won't get necessarily a good reconstruction: the value can far away from any previous value the network has seen before! That's why attaching a probabilistic model to the compressed representation is a good idea!

For sake of simplicity, let's use a standard normal distribution to define the distribution of inputs ($\mathbf V$) the decoder will receive.
The architecture of a variational autoencoder (VAE) is thus:

(Source: http://ijdykeman.github.io/ml/2016/12/21/cvae.html)

We want the decoder to take any point taken from a standard normal distribution to return a reasonable element of our dataset:

(Source: http://ijdykeman.github.io/ml/2016/12/21/cvae.html)

Let's consider the encoder role in this architecture.
In a traditional autoencoder, the encoder model takes a sample from data and returns a single point in the latent space, which is then passed to the decoder.

What information is encoded in the latent space?

In VAE the encoder instead produces (the parameters of) a probability distribution in the latent space:

(Source: http://ijdykeman.github.io/ml/2016/12/21/cvae.html)

These distributions are (non standard) Gaussians of the same dimensionality as the latent space.
First, let’s implement the encoder net, which takes input $X$ and outputs two things: $\mu(X)$ and $\Sigma(X)$, the parameters of the...
Can you write Python code for it?

# vae architecture

from tensorflow.examples.tutorials.mnist import input_data
from keras.layers import Input, Dense, Lambda
from keras.models import Model
from keras.objectives import binary_crossentropy
from keras.callbacks import LearningRateScheduler

import numpy as np
import matplotlib.pyplot as plt
import keras.backend as K
import tensorflow as tf

m = 50
n_z = 2
n_epoch = 100

# encoder
inputs = Input(shape=(784,))
h_q = Dense(512, activation='relu')(inputs)
mu = Dense(n_z, activation='linear')(h_q)
log_sigma = Dense(n_z, activation='linear')(h_q)


I am working on the file "11. 기초 확률론4 - 상관 관계/01. 결합, 주변, 조건부 확률 밀도 함수.ipynb".
The first task is:
동일한 결합 확률 밀도 함수를 3차원으로 그리면 아래와 같다.
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = Axes3D(fig)
ax.contour(XX, YY, ZZ)
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.set_title("Joint Probability Density")
ax.view_init(50, -80)
plt.show()


I am working on the file "11. 기초 확률론4 - 상관 관계/01. 결합, 주변, 조건부 확률 밀도 함수.ipynb".
The first task is:
위에서 예로 든 연속 확률 변수의 경우에 주변 확률 밀도 함수를 계산하면 다음과 같다.
Can you write Python code for it?

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.contour(XX, YY, ZZ, levels=np.linspace(0, 0.04, 30), alpha=0.3)
ax.plot(yy, ZZ.mean(axis=1), zdir='x', lw=3)
ax.plot(xx, ZZ.mean(axis=0), zdir='y', lw=3)
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.set_title("Marginal Probability Density")
ax.view_init(50, -80)
plt.show()


I am working on the file "11. 기초 확률론4 - 상관 관계/01. 결합, 주변, 조건부 확률 밀도 함수.ipynb".
The first task is:
이산 확률 변수의 결합 확률 질량 함수
다변수 이산 확률 변수에 대해서는 결합 확률 질량 함수를 바로 구할 수 있다. 결합 확률 질량 함수는 모든 확률 변수의 값이 특정 숫자가 될 확률을 듯한다.
$$ f_{XY}(x,y) = P(X=x,Y=y) $$
결합 확률 질량 함수는 댜음과 같이 2차원 표의 형태가 된다.
Can you write Python code for it?

pmf = np.array([[0, 0, 0, 0, 1, 1],
                [0, 0, 1, 2, 1, 0],
                [0, 1, 3, 3, 1, 0],
                [0, 1, 2, 1, 0, 0],
                [1, 1, 0, 0, 0, 0]])
pmf = pmf/pmf.sum()
pmf

sns.heatmap(pmf)
plt.xlabel("x")
plt.ylabel("y")
plt.title("Joint Probability Mass Function")
plt.show()


I am working on the file "11. 기초 확률론4 - 상관 관계/01. 결합, 주변, 조건부 확률 밀도 함수.ipynb".
The first task is:
조건부 확률 밀도 함수
조건부 확률 밀도 함수(conditional probability density function)는 다변수 확률 변수 중 하나의 값이 특정 값으로 고정되어 상수가 되어 버린 경우, 나머지 변수에 대한 확률 밀도 함수를 말한다.
$$ f_{X \mid Y}(x \mid y_0) = \dfrac{f_{XY}(x, y=y_0)}{f_{Y}(y_0)} $$
$$ f_{Y \mid X}(y \mid x_0) = \dfrac{f_{XY}(x, y=y_0)}{f_{X}(x_0)} $$
주변 확률 밀도 함수와 마찬가지로 차원이 감소하지만 평균 연산(적분)으로 변수가 없어진 주변 확률 밀도 함수와는 값이 달라진다.
이산 확률 변수의 경우에는 조건부 확률 질량 함수를 사용하여 정의한다.
위에서 예로 든 이산 확률 변수의 경우에 조건부 확률 질량 함수를 계산하면 다음과 같다.
Can you write Python code for it?

pmf

cond_y0 = pmf[0, :]/pmf_marginal_y[0]
cond_y0

cond_y1 = pmf[1, :]/pmf_marginal_y[1]
cond_y1


I am working on the file "11. 기초 확률론4 - 상관 관계/02. 확률 밀도 함수의 독립.ipynb".
The first task is:
여러 가지 Y값을 바꾸어도 조건부 확률은 변하지 않는 것을 확인할 수 있다.
Can you write Python code for it?

cond_x_y0 = pmf1[0, :]/pmf1_marginal_y[0]
cond_x_y0

cond_x_y1 = pmf1[1, :]/pmf1_marginal_y[1]
cond_x_y1

cond_x_y2 = pmf1[2, :]/pmf1_marginal_y[2]
cond_x_y2


I am working on the file "11. 기초 확률론4 - 상관 관계/02. 확률 밀도 함수의 독립.ipynb".
The first task is:
이 확률 분포는 다음 식에서 보다시피 주변 확률 분포의 곱으로 표현된다.
Can you write Python code for it?

pmf1_marginal_x = pmf1.sum(axis=0)
pmf1_marginal_y = pmf1.sum(axis=1)
pmf = pmf1_marginal_x * pmf1_marginal_y[:, np.newaxis]
pmf/pmf.sum()


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
A final example now, with a (5,1) and (3) array. Read the rule once again - and it will be clear that the new array will be a 5X3 array.
Can you write Python code for it?

arr4 = np.arange(1,6)
arr4

arr4 = arr4.reshape(5,1).astype('float')
arr4.shape

arr2

arr4 * arr2


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
A quick digression, in case you are wondering, the .astype('float') was just a quick operation to convert integers to floats as you are already familiar with. If you want to find out what the data type of an element in a numpy array is, simply use the suffix .dtype
Can you write Python code for it?

arr1.dtype

arr_example = np.array([1,2,3,4])
print(arr_example)
print('arr_example is an',arr_example.dtype)
arr_example = arr_example.astype('float')
print('arr_example is now a',arr_example.dtype)


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Array of Ones
Can you write Python code for it?


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Broadcasting
Broadcasting is a way for Numpy to work with arrays of different shapes.
The easiest example to explain broadcasting would be to use a scalar value. What's a scalar? A quantity having only magnitutde, but not direction. Speed is a scalar, velocity is a vector. For our practical Numpy purposes, scalars are real numbers - 1,2,3,4.....
Broadcasting is fast and efficient because all the underlying looping occurs in C, and happens on the fly without making copies of the data.
Can you write Python code for it?

arr_1 = np.arange(1,11)
print(arr_1)
print(arr_1 * 10)


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Broadcasting Rule
What about arrays of different dimensions and/or sizes? Well, for that, we have the broadcasting rule.

In order to broadcast, the size of the trailing axes for both arrays in an operation must either be the same size or one of them must be one.

Umm....
<img src="images/english.jpg">
Ok, let's not offend our friend Samuel Jackson here, so here's what it means in plain English.
Can you write Python code for it?

arr1 = np.arange(1,13)
arr1

arr1.shape

arr1 = arr1.reshape(4,3).astype('float')

arr1


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Do you see what happened here? Our row with 3 elements, was sequentially added to each 3-element row in arr1.
The 1d array is represented as (3,), but think of it as simple a (3). The trailing axes have to match. So (4,3) and (3) match. What happens with it's (4,3) and (4)? It won't work! Let's prove it here.
Can you write Python code for it?

arr3 = np.arange(0,4)
arr3 = arr3.astype('float')
print(arr3)
print(arr3.shape)

# Let's generate our error
arr1 + arr3


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Exercise
Create atleast 4 challenges for yourself, so you can practice indexing.
Can you write Python code for it?


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Exercise
Select 26 to 30
Can you write Python code for it?

# Step 1 - Narrow down the row
arr1[2] # 3rd row

# 26 is at index 5, we need all the numbers from thr 6th column onwards
arr1[2,5:]


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Exercise
Select:

[ 44,  45]

[ 54,  55]
Can you write Python code for it?


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Exercise
Select:

[71,  72,  73]

[81,  82,  83]

[91,  92,  93]
Can you write Python code for it?

# Step 1: Identify the Row
arr1[7:]

# Now we need the first three columns
arr1[7:,:3]


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Fancy Indexing
Method 1: Boolean Masks
While there are many ways to index, one of the more common methods that Data Scientists use will is Boolean Indexing. You can read more about indexing methods here.
Can you write Python code for it?

import numpy as np
np.random.seed(42)

arr1 = np.random.randint(0,1000,100)
arr1

# We check what values are greater than 150
arr1>150

# Assign this operation to a variable x
mask = arr1>150
# Create a new array which subsets arr1 based on a boolean operation
arr2 = arr1[mask]
arr2

# Check the shape
arr2.shape


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Identity Matrix
An identity matrix is a square matrix, with all the values on the diagonal equal to 1, and the remaining values equal to 0.
Can you write Python code for it?


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Method 2: Indexing with Array of Integers
Don't get intimidated by the big words - it just means indexing by using a Python list.
Can you write Python code for it?

list1 = [1,3,5,7]
list2 = [2,4,6,8]

arr1 = np.arange(1,101)
arr1

arr_even = arr1[list1]
arr_even

# Alternatively
arr_even = arr1[[1,3,5,7]]
arr_even

arr_odd = arr1[list2]
arr_odd


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Quick Operations on Numpy Arrays
Can you write Python code for it?

import numpy as np
np.random.seed(42)

arr1 = np.random.randint(1,1000,100)
arr1 = arr1.reshape(10,10)

arr1.shape

arr1


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Remember, rows before columns. Always!

How do we get entire rows, or snippets of values from rows?

Exactly the same as before. Nothing to worry about here!
Can you write Python code for it?

import numpy as np
np.random.seed(42)

arr1 = np.arange(1,101)
arr1

arr1 = arr1.reshape(10,10)
arr1


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Remember, the first number always represents the number of rows.
Random Array with a Uniform Distribution
From the official documentation:

Create an array of the given shape and populate it with random samples from a uniform distribution over [0, 1).

A uniform distribution, is a distribution that has constant probability.
Can you write Python code for it?

np.random.seed(42)
np.random.rand(5)

np.random.seed(42)
np.random.rand(3,2)


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Selecting Values
Secret! You already know how to select values from a numpy array.
Can you write Python code for it?

import numpy as np
np.random.seed(42)

arr1 = np.arange(1,6)
arr1

arr1[0]

arr1[0:3]

arr1[-1]


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Square Roots
Can you write Python code for it?


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Take
This is similar to Fancy Indexing, but is arguably easier to use, atleast for me. I am sure you might develop a preference for this technique too. Additionally, Wes McKinney - the creator of Pandas, reports that "take" is faster than "fancy indexing".
Can you write Python code for it?


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Translating from Python to English, "call the randint module from the random module of numpy, then select 20 numbers between 0 and 999 at random, and assign that to an array named rand_arr i.e. 0 is included, 1000 is excluded.

Can you write Python code for it?

rand_arr.reshape(5,4)

rand_arr.reshape(4,5)

rand_arr.reshape(2,10)


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
We can even reshape these arrays into our desired shape. But remember, when we say desired shape, we are not speaking of circles or pentagons. Think square, reactangles, cubes and the like.
Can you write Python code for it?

arr1.reshape(4,4)

arr1.reshape(2,8)

arr1.reshape(8,2)

arr1.reshape(16,1)


I am working on the file "11.Introduction_to_Numpy.ipynb".
The first task is:
Why did that happen?! We never touched a1, and even went on to create a whole new array!
This is because Numpy is very efficient in the way it uses memory. If you want a copy, be explicit, else Numpy will make changes to the original array too. Here's how you make a copy.
Can you write Python code for it?

a1_copy = a1.copy()
a1_copy

a1_copy = np.arange(1,21)
a1_copy = a1_copy.reshape(4,5)
a1_copy

a1


I am working on the file "11/.ipynb_checkpoints/Homework_11_Emelike-checkpoint.ipynb".
The first task is:
1. I want to make sure my Plate ID is a string. Can't lose the leading zeroes!
Can you write Python code for it?

# checks data type of each value in series Plate ID by printing if type does not equal string
# all values are strings
for x in df['Plate ID']:
    if type(x) != str:
        print(type(x))


I am working on the file "11/.ipynb_checkpoints/Homework_11_Emelike-checkpoint.ipynb".
The first task is:
2. I don't think anyone's car was built in 0AD. Discard the '0's as NaN.
Can you write Python code for it?

df['Vehicle Year'] = df['Vehicle Year'].replace(0, np.nan)
df['Vehicle Year'] = pd.to_datetime(df['Vehicle Year'], format='%Y', errors = 'coerce')


I am working on the file "11/.ipynb_checkpoints/Homework_11_Emelike-checkpoint.ipynb".
The first task is:
3. I want the dates to be dates! Read the read_csv documentation to find out how to make pandas automatically parse dates.
Can you write Python code for it?

# see where I read in csv for inclusion of parse_date arg


I am working on the file "11/.ipynb_checkpoints/Homework_11_Emelike-checkpoint.ipynb".
The first task is:
4. "Date first observed" is a pretty weird column, but it seems like it has a date hiding inside. Using a function with .apply, transform the string (e.g. "20140324") into a Python date. Make the 0's show up as NaN.
Can you write Python code for it?

df['Date First Observed'] = df['Date First Observed'].apply(lambda x: pd.to_datetime(str(x), format='%Y%m%d', errors='coerce'))


I am working on the file "11/.ipynb_checkpoints/Homework_11_Emelike-checkpoint.ipynb".
The first task is:
5. "Violation time" is... not a time. Make it a time.
Can you write Python code for it?

def remove_char(x):
    x = x[:-1]
    return x
def insert_char(x):
    x = x[:2] + ':' + x[2:]
    return x

# removes 'A' and 'P' from original values in Violation Time
df['Violation Time2'] = df['Violation Time'].apply(lambda x: remove_char(str(x)))
# inserts colon in to string
df['Violation Time3'] = df['Violation Time2'].apply(lambda x: insert_char(str(x)))

#makes values datetime format, allowing .to_datetime to infer format from string
df['Violation Time4'] = df['Violation Time3'].apply(lambda x: pd.to_datetime(str(x), infer_datetime_format=True, errors='coerce'))
#selects only time from timestamp
df['Violation Time'] = df['Violation Time4'].dt.time

# deletes temporary variables used to make violation time a time
df=df.drop(df.columns[[43, 44, 45]], axis=1)
df.columns


I am working on the file "11/.ipynb_checkpoints/Homework_11_Emelike-checkpoint.ipynb".
The first task is:
7. Join the data with the Parking Violations Code dataset from the NYC Open Data site.
Can you write Python code for it?

df2 = pd.read_csv('DOF_Parking_Violation_Codes.csv')

df2

# change format to string to match violation codes data set format
df['Violation Code'] = df['Violation Code'].apply(lambda x: str(x).strip())

# check formatting and string length
for x in df['Violation Code']:
    if type(x) != str:
        print(type(x))
    if len(x) > 2:
        print(x)

df3 = pd.merge(df, df2, left_on='Violation Code', right_on='CODE', how='left', indicator=True)

# unsuccessful merges bc Violation Code 41 does not exist in Violation codes data.
# and bc 38 in documented at 37-38 in Violation codes data
df3[df3['_merge']=='left_only']


I am working on the file "11/.ipynb_checkpoints/Homework_11_Emelike-checkpoint.ipynb".
The first task is:
8. How much money did NYC make off of parking violations?
Can you write Python code for it?

# removes unsuccessful merges
df4 = df3[df3['_merge'] !='left_only']

# Create boolean variable to indicate 0/1: Parking violation
df4['Parking Violation'] = df4['DEFINITION'].str.contains('[Pp]arking') | df4['DEFINITION'].str.contains('[Pp]arked')
df4['Parking Violation'].value_counts()
df4[df4['Parking Violation'] == False]

# df4.groupby('Parking Violation')[''].sum()[True]
df4['Street Name'].value_counts()
above_96_lst=[]
for x in df4['Street Name']:
    if re.search(r'\S \d \S\w ', str(x)):
        above_96 =  False
        above_96_lst.append(above_96)
    elif re.search(r'\d[[Aa]-[Zz]][[Aa]-[Zz]] ', str(x)):
        above_96 =  False
        above_96_lst.append(above_96)
#         print(x, above_96)
    elif re.search(r'\w\s\d\d\s', str(x)):
        above_96 =  False
        above_96_lst.append(above_96)
#         print(x, above_96)
    elif re.search(r'[A-Z]\s\d\d[A-Z][A-Z] [A-Z][A-Z]', str(x)):
        above_96 =  False
        above_96_lst.append(above_96)
#         print(x, above_96)
    elif re.search(r'[0-2][A-Z][A-Z] AVE', str(x)):
        above_96 =  False
        above_96_lst.append(above_96)
#         print(x, above_96)
    else:
        above_96 =  True
        above_96_lst.append(above_96)
#         print(x, above_96)

# above_96_lst.count(False)
df4['Above_96'] = above_96_lst

df4 .h


I am working on the file "11/.ipynb_checkpoints/Parking Data-checkpoint.ipynb".
The first task is:
"Date first observed" is a pretty weird column, but it seems like it has a date hiding inside. Using a function with .apply, transform the string (e.g. "20140324") into a Python date. Make the 0's show up as NaN.
Can you write Python code for it?

# print(violations_df['Date First Observed'].head())

import datetime
violations_df.head()['Issue Date'].astype(datetime.datetime)

import datetime
violations_df.head()['Issue Date'].astype(datetime.datetime)

#violations_df['Issue Date'] = violations_df['Issue Date'].astype('datetime64[ns]')

# violations_df.dtypes


I am working on the file "11/.ipynb_checkpoints/Parking Data-checkpoint.ipynb".
The first task is:
"Violation time" is... not a time. Make it a time.
Can you write Python code for it?

import re
# violations_df['Violation Time'].head()

import numpy as np
# Build a function using that method
def time_to_datetime(str_time):
    try:
    #str_time = re.sub('^0', '', str_time)
        if isinstance(str_time, str):
            str_time = str_time + "M"
            #print("Trying to convert", str_time, "into a time")
            return datetime.datetime.strptime(str_time.strip(), "%I%M%p").time()
    #try:
     #   if str_time == -999:
      #      print("It's -999")
       #     return np.nan

    except:
        return np.nan

# Apply that method to the 'Time' column of the dataframe
violations_df['Violation Time'] = violations_df['Violation Time'].apply(time_to_datetime)

print(violations_df['Violation Time'].head())

# def remove_1900(year):
#     year = str(year)
#     year = re.sub('^1900-01-01', '', year)
#     return year

# print(violations_df['Violation Time'].apply(remove_1900))


I am working on the file "11/.ipynb_checkpoints/Parking Data-checkpoint.ipynb".
The first task is:
Join the data with the Parking Violations Code dataset from the NYC Open Data site.
Can you write Python code for it?

violations_df = violations_df.merge(dof_violations_df, left_on='Violation Code', right_on='CODE')


print(violations_df.columns)
violations_df['Violation Code'].head()

violations_df

print(dof_violations_df.columns)
dof_violations_df['CODE'].head()

# violations_df.merge(dof_violations_df, left_on='Violation Code', right_on='CODE')

#print(dof_violations_df['CODE'][0])
#test = dof_violations_df['CODE'][0]

def to_int(test):
    try:
        # test = re.sub('$', '', test)
        test = test.strip("$")
        #print(test)
        test = int(test)
        #print(test, " is now an int")
        return test
    except:
        print(test, " coult not be converted to an int")
        return np.nan

dof_violations_df['CODE'] = dof_violations_df['CODE'].apply(to_int)


I am working on the file "11/.ipynb_checkpoints/Parking Data-checkpoint.ipynb".
The first task is:
New Jersey has bad drivers, but does it have bad parkers, too?
Can you write Python code for it?

nj_violations_df = violations_df[violations_df['Registration State'] != 'NJ']

print("There were", len(nj_violations_df['All Other Areas']), "parking violations in NYC by drivers registered in New Jersey")
print("These violations totaled", nj_violations_df['All Other Areas'].sum(), "of revenue")


I am working on the file "11/.ipynb_checkpoints/Parking Data-checkpoint.ipynb".
The first task is:
The most frequent parking violation is:
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
freqNYviolations = violations_df.groupby('CODE')['All Other Areas'].count().sort_values(ascending=False).head().plot(kind="bar", color = ['#624ea7', '#599ad3', '#f9a65a', '#9e66ab', 'purple'])
freqNYviolations.set_title('Most Frequent NYC Parking Violations by Code')
freqNYviolations.set_xlabel('Parking Violation Code')
freqNYviolations.set_ylabel('Frequency of Citations')
plt.savefig('freqParkingViolations.png')

violations_df.groupby('CODE')['All Other Areas'].count().sort_values(ascending=False).head(1)

# violations_df.sort_values('All Other Areas', ascending=False)


I am working on the file "11/.ipynb_checkpoints/Parking Data-checkpoint.ipynb".
The first task is:
There sure are a lot of colors of cars, too bad so many of them are the same. Make "BLK" and "BLACK", "WT" and "WHITE", and any other combinations that you notice.
Can you write Python code for it?

import re

def vehicle_colors(vehicle):
    if isinstance(vehicle, str):
        #print(vehicle)
        vehicle = re.sub('^GY$', 'GREY', vehicle)
        #vehicle = vehicle.replace("GY", "GREY")
        vehicle = re.sub('^WH$', 'WHITE', vehicle)
        vehicle = re.sub('^BR$', 'BROWN', vehicle)
        vehicle = re.sub('^RD$', 'RED', vehicle)
        vehicle = re.sub('^B[LK]$', 'BLACK', vehicle)
        vehicle = re.sub('^TN$', 'TAN', vehicle)
        vehicle = re.sub('^YW$', 'YELLOW', vehicle)
        vehicle = re.sub('^SIL$', 'SILVER', vehicle)
        vehicle = re.sub('^GR$', 'GREEN', vehicle)
        vehicle = re.sub('^SILVE$', 'SILVER', vehicle)

#         vehicle = vehicle.replace("WH","WHITE")
#         vehicle = vehicle.replace("BR","BROWN")
#         vehicle = vehicle.replace("RD","RED")
#         vehicle = vehicle.replace("BL","BLACK")
#         vehicle = vehicle.replace("BK","BLACK")
#         vehicle = vehicle.replace("TN","TAN")
#         vehicle = vehicle.replace("YW","YELLOW")
#         vehicle = re.sub('SIL$', 'SILVER', vehicle)
#         vehicle = vehicle.replace("SILVR","SILVER")
#         vehicle = vehicle.replace("SIL","SILVER")
    return vehicle

violations_df['Vehicle Color'] = violations_df['Vehicle Color'].apply(vehicle_colors)

violations_df['Vehicle Color'].head()


I am working on the file "11/.ipynb_checkpoints/Parking Data-checkpoint.ipynb".
The first task is:
What time of day do people usually get their tickets? You can break the day up into several blocks - for example 12am-6am, 6am-12pm, 12pm-6pm, 6pm-12am.

What's the average ticket cost in NYC?
Can you write Python code for it?

print("The average ticket cost in NYC is $", violations_df['All Other Areas'].mean())


I am working on the file "11/.ipynb_checkpoints/Parking Data-checkpoint.ipynb".
The first task is:
What's the most lucrative kind of parking violation? Below lists all the ones that chaarge $115, which is the highest amount
Can you write Python code for it?


I am working on the file "11/Homework11-Devulapalli-Graded.ipynb".
The first task is:
11. Make a chart of the top few.
Can you write Python code for it?

import matplotlib.pyplot as plt

%matplotlib inline

plt.style.use('ggplot')
toptenviolations=nonnyviolations['Violation Code'].value_counts()
toptenviolations.head(10).plot(kind='bar')
#I USE THE VIOLATION CODE INSTEAD OF ACTUAL VIOLATION NAME BECAUSE THE NAMES OF SOME OF THEM ARE TOO LONG.


I am working on the file "11/Homework11-Devulapalli-Graded.ipynb".
The first task is:
4. "Date first observed" is a pretty weird column, but it seems like it has a date hiding inside. Using a function with .apply, transform the string (e.g. "20140324") into a Python date. Make the 0's show up as NaN.
Can you write Python code for it?

## WRITING A FUNCTION TO CONVERT STRING INTO A DATE

def string_to_date(string):
    from dateutil import parser
    if pd.isnull(string):
        return None
    else:
        dt = parser.parse(string)
        return dt.date()

string_to_date('20160808') #Testing it out.

df['Date First Observed']= df['Date First Observed'].apply(string_to_date) #Applying it to the dataframe column


I am working on the file "11/Homework11-Devulapalli-Graded.ipynb".
The first task is:
6, There sure are a lot of colors of cars, too bad so many of them are the same. Make "BLK" and "BLACK", "WT" and "WHITE", and any other combinations that you notice.
Can you write Python code for it?

df['Vehicle Color'].value_counts() #LOOKING AT COLORS IN THE ORIGINAL DATASET

df['Vehicle Color'] = df['Vehicle Color'].replace(['WHT', 'WH','WT','WHI'], 'WHITE')

df['Vehicle Color'] = df['Vehicle Color'].replace(['GRAY', 'GY','GRY'], 'GREY')

df['Vehicle Color'] = df['Vehicle Color'].replace(['BLK', 'BK'], 'BLACK')

df['Vehicle Color'] = df['Vehicle Color'].replace('BL', 'BLUE')

df['Vehicle Color'] = df['Vehicle Color'].replace(['BR', 'BN','BRWN'], 'BROWN')

df['Vehicle Color'] = df['Vehicle Color'].replace('RD', 'RED')

df['Vehicle Color'] = df['Vehicle Color'].replace(['GR', 'GN','GRN'], 'GREEN')

df['Vehicle Color'] = df['Vehicle Color'].replace('TN', 'TAN')

df['Vehicle Color'] = df['Vehicle Color'].replace('GL', 'GOLD')

df['Vehicle Color'] = df['Vehicle Color'].replace('BRN', 'BROWN')

df['Vehicle Color'] = df['Vehicle Color'].replace(['YW', 'YELLO','YELL'], 'YELLOW')

df['Vehicle Color'] = df['Vehicle Color'].replace(['BL', 'BLU'], 'BLUE')

df['Vehicle Color'] = df['Vehicle Color'].replace('MR', 'MAROON')

df['Vehicle Color'] = df['Vehicle Color'].replace(['SIL', 'SILVR','SILVE','SILV'], 'SILVER')

df['Vehicle Color'] = df['Vehicle Color'].replace(['OR', 'ORANG'], 'ORANGE')

df['Vehicle Color'].value_counts() #LOOKING AT THE COLORS AFTER CLEANING UP, MUCH BETTER!


I am working on the file "11/Homework11-Devulapalli-Graded.ipynb".
The first task is:
7. Join the data with the Parking Violations Code dataset from the NYC Open Data site.
Can you write Python code for it?

#READING PARKING VIOLATION CODES FROM NYC DATASET
codesdf=pd.read_csv("DOF_Parking_Violation_Codes.csv",dtype={'Manhattan\xa0 96th St. & below':int, 'All Other Areas': int})

codesdf.columns

codesdf['Manhattan\xa0 96th St. & below'].value_counts() #CHECKING IF THE IMPORT WORKED WELL

newdf=df.join(codesdf, on='Violation Code', how='left') #MERGING THE TWO DATAFRAMES INTO A NEW DATAFRAME Unimaginatively CALLED NEWDF

newdf.head() #DID IT WORK? SEEMS TO HAVE, THE NANS IN THE END OF THIS FRAME ARE THERE IN THE ORIGINAL ONE TOO.

newdf.columns #THE NEW COLUMNS SEEM TO HAVE BEEN ADDED


I am working on the file "11/Homework11-Devulapalli-Graded.ipynb".
The first task is:
9. What's the most lucrative kind of parking violation? The most frequent?
Can you write Python code for it?

newdf['Manhattan\xa0 96th St. & below'].value_counts()
#IT IS EVIDENT THAT THE 265 DOLLARS IS THE MOST LUCRATIVE FINE in terms of monetary amount, but 115 DOlLARS IS THE MOST COMMON

lucrative=newdf[newdf['Manhattan\xa0 96th St. & below']==265]

lucrative['DEFINITION'].value_counts()
#TRACTOR TRAILER PARKING SEEMS TO MAKE MONIES

frequent=newdf['DEFINITION'].value_counts()
#IDENTIFYING THE MOST FREQUENT SETS OF VIOLATIONS

frequent.head() #THE TOP FIVE SETS OF VIOLATIONS


I am working on the file "11/Homework11-Devulapalli-Graded.ipynb".
The first task is:
From the graph above, it is very evident that most people get tickets in the 6 AM to 12 PM ie the morning slot. More than any other segments of the day.
13. What's the average ticket cost in NYC?
Can you write Python code for it?

newdf['Manhattan\xa0 96th St. & below'].describe()
#AVERAGE COST OF TICKETS IN MANHATTAN is 95 DOLALRS

newdf['All Other Areas'].describe()
#AVERAGE COST OF TICKET ELSEWHERE IS 85 DOLLARS

import numpy
a=[84.716996,94.962425]
numpy.mean(a)
#SO THE AVERAGE TICKET COST IN NEW YORK CITY IS ABOUT 90 DOLLARS


I am working on the file "11/Homework11-Devulapalli-Graded.ipynb".
The first task is:
Now, we know the locations of the violation and from the merged dataset we know that there the fine rates vary upon the location of the violation, with certain rates for everything under 96th street in Manhattan and other rates for elsewhere. So using the Violation County numbers, and the assumption that NY county is everything under 96th - we calculate.
Can you write Python code for it?

manhattanviolations=newdf[newdf['Violation County']=='NY']

#FINES COLLECTED IN MANHATTAN
manhattanviolations['Manhattan\xa0 96th St. & below'].sum()

allotherviolations=newdf[newdf['Violation County']!= 'NY']

#FINES NOT COLLECTED IN MANHATTAN

allotherviolations['All Other Areas'].sum()

#FINES COLLECTED IN ALL OF NYC
manhattanviolations['Manhattan\xa0 96th St. & below'].sum()+allotherviolations['All Other Areas'].sum()


I am working on the file "11/Homework11-Devulapalli-Graded.ipynb".
The first task is:
We can then inspect what 21,38,14 violation codes are.
12. What time of day do people usually get their tickets? You can break the day up into several blocks - for example 12am-6am, 6am-12pm, 12pm-6pm, 6pm-12am.
Can you write Python code for it?


I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
11. Make a chart of the top few.
Can you write Python code for it?


I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
17. What's the parking-ticket in dollars-per-licensed-driver in each borough of NYC? Do this with pandas and the dataframe you just made, not with your head!
Can you write Python code for it?

#MAKING DATAFRAMES FOR EACH OF THESE BOROUGHS/COUNTIES
bronxviolations=newdf[newdf['Violation County']=='BRONX']
brooklynviolations=newdf[newdf['Violation County']=='KINGS']
manhattanviolations=newdf[newdf['Violation County']=='NY']
queensviolations=newdf[newdf['Violation County']=='Q']
statenislandviolations=newdf[newdf['Violation County']=='RICHMOND']

#GETTING THE TOTAL NUMBER OF DRIVERS FROM THE NEW DATA FRAME
numberofdriversinbronx=driversdf.iloc[0]['Total']
numberofdriversinbrooklyn=driversdf.iloc[1]['Total']
numberofdriversinmanhattan=driversdf.iloc[2]['Total']
numberofdriversinqueens=driversdf.iloc[3]['Total']
numberofdriversinstatenisland=driversdf.iloc[4]['Total']

bronxviolations['All Other Areas'].sum()/numberofdriversinbronx #AVERAGE PARKING TICKET PER DRIVER IN BRONX

brooklynviolations['All Other Areas'].sum()/numberofdriversinbrooklyn #AVERAGE PARKING TICKET PER DRIVER IN BROOKLYN

manhattanviolations['Manhattan\xa0 96th St. & below'].sum()/numberofdriversinmanhattan #AVERAGE PARKING TICKET PER DRIVER IN MANHATTAN

queensviolations['All Other Areas'].sum()/numberofdriversinqueens #AVERAGE PARKING TICKET PER DRIVER IN QUEENS

statenislandviolations['All Other Areas'].sum()/numberofdriversinstatenisland #AVERAGE PARKING TICKET PER DRIVER IN STATEN ISLAND


I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
4. "Date first observed" is a pretty weird column, but it seems like it has a date hiding inside. Using a function with .apply, transform the string (e.g. "20140324") into a Python date. Make the 0's show up as NaN.
Can you write Python code for it?


I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
5. "Violation time" is... not a time. Make it a time.
Can you write Python code for it?

def string_to_time(string): #CONVERTING THE VIOLATION TIME STRING INTO TIME
    from dateutil import parser
    import re
    if pd.isnull(string):
        return None
    if string =='Nan' or string =='nan' or string =='0':
        return None
    if string[0:4].isnumeric()==True:
        if int(string[0:2]) <=12 and int(string[2:4])<=59:
            regex=re.search(r"\d\d\d\d[AaPp]",string)
            if regex:
                time=string[0:2]+":"+string[2:4]+" "+string[4]+"M"
                return parser.parse(time).time()
            else:
                return None

def float_to_int(float): #THE VIOLATION CODE COLUMN IS ACTUALLY A FLOAT, SO WE'RE CONVERTING INTO AN INTEGER FIRST.
    if pd.isnull(float):
        return None
    else:
         return int(float)

df['Violation Code']=df['Violation Code'].apply(float_to_int)

df['Violation Time']=df['Violation Time'].apply(string_to_time)

df['Violation Time'].head() #TESTING IF IT WORKED


I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
6, There sure are a lot of colors of cars, too bad so many of them are the same. Make "BLK" and "BLACK", "WT" and "WHITE", and any other combinations that you notice.
Can you write Python code for it?


I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
8. How much money did NYC make off of parking violations?
Can you write Python code for it?

newdf['Violation County'].value_counts() #LET US SEE THE VIOLATION COUNTY DATA, THIS NEEDS TO BE CLEANED UP A BIT

newdf['Violation County'] = newdf['Violation County'].replace('BX', 'BRONX')

newdf['Violation County'] = newdf['Violation County'].replace(['R', 'RICH'], 'RICHMOND')

newdf['Violation County'] = newdf['Violation County'].replace('K', 'KINGS')

newdf['Violation County'].value_counts() #ALL CLEANED


I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
From the graph above, it is very evident that most people get tickets in the 6 AM to 12 PM ie the morning slot. More than any other segments of the day.
13. What's the average ticket cost in NYC?
Can you write Python code for it?


I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
Looks like, the month of August is when the Police seem to have gone on a fining spree.
15. Make a graph of the amount of revenue collected per day.
<< Not too sure on how to get this done>>
16. Manually construct a dataframe out of https://dmv.ny.gov/statistic/2015licinforce-web.pdf (only NYC boroughts - bronx, queens, manhattan, staten island, brooklyn), having columns for borough name, abbreviation, and number of licensed drivers.
Can you write Python code for it?


I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
Now, we know the locations of the violation and from the merged dataset we know that there the fine rates vary upon the location of the violation, with certain rates for everything under 96th street in Manhattan and other rates for elsewhere. So using the Violation County numbers, and the assumption that NY county is everything under 96th - we calculate.
Can you write Python code for it?


I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
We can then inspect what 21,38,14 violation codes are.
12. What time of day do people usually get their tickets? You can break the day up into several blocks - for example 12am-6am, 6am-12pm, 12pm-6pm, 6pm-12am.
Can you write Python code for it?


I am working on the file "11/Homework_11_Emelike.ipynb".
The first task is:
1. I want to make sure my Plate ID is a string. Can't lose the leading zeroes!
Can you write Python code for it?

# checks data type of each value in series Plate ID by printing if type does not equal string
# all values are strings
for x in df['Plate ID']:
    if type(x) != str:
        print(type(x))


I am working on the file "11/Homework_11_Emelike.ipynb".
The first task is:
2. I don't think anyone's car was built in 0AD. Discard the '0's as NaN.
Can you write Python code for it?

df['Vehicle Year'] = df['Vehicle Year'].replace(0, np.nan)
df['Vehicle Year'] = pd.to_datetime(df['Vehicle Year'], format='%Y', errors = 'coerce')


I am working on the file "11/Homework_11_Emelike.ipynb".
The first task is:
3. I want the dates to be dates! Read the read_csv documentation to find out how to make pandas automatically parse dates.
Can you write Python code for it?

# see where I read in csv for inclusion of parse_date arg


I am working on the file "11/Homework_11_Emelike.ipynb".
The first task is:
4. "Date first observed" is a pretty weird column, but it seems like it has a date hiding inside. Using a function with .apply, transform the string (e.g. "20140324") into a Python date. Make the 0's show up as NaN.
Can you write Python code for it?

df['Date First Observed'] = df['Date First Observed'].apply(lambda x: pd.to_datetime(str(x), format='%Y%m%d', errors='coerce'))


I am working on the file "11/Homework_11_Emelike.ipynb".
The first task is:
5. "Violation time" is... not a time. Make it a time.
Can you write Python code for it?

def remove_char(x):
    x = x[:-1]
    return x
def insert_char(x):
    x = x[:2] + ':' + x[2:]
    return x

# removes 'A' and 'P' from original values in Violation Time
df['Violation Time2'] = df['Violation Time'].apply(lambda x: remove_char(str(x)))
# inserts colon in to string
df['Violation Time3'] = df['Violation Time2'].apply(lambda x: insert_char(str(x)))

#makes values datetime format, allowing .to_datetime to infer format from string
df['Violation Time4'] = df['Violation Time3'].apply(lambda x: pd.to_datetime(str(x), infer_datetime_format=True, errors='coerce'))
#selects only time from timestamp
df['Violation Time'] = df['Violation Time4'].dt.time

# deletes temporary variables used to make violation time a time
df=df.drop(df.columns[[43, 44, 45]], axis=1)
df.columns


I am working on the file "11/Homework_11_Emelike.ipynb".
The first task is:
6.  There sure are a lot of colors of cars, too bad so many of them are the same. Make "BLK" and "BLACK", "WT" and "WHITE", and any other combinations that you notice.
Can you write Python code for it?

def clean_color(x):
    if x == 'BLK':
        print(x)
        x = x.replace('BLK','BLACK')
        print('Now', x)
    elif x == 'BK':
        print(x)
        x = x.replace('BK','BLACK')
        print('Now', x)
    elif x == 'W':
        print(x)
        x = x.replace('W','WHITE')
        print('Now', x)
    elif x == 'WH':
        print(x)
        x = x.replace('WH','WHITE')
        print('Now', x)
    elif x == 'WHT':
        print(x)
        x = x.replace('WHT','WHITE')
        print('Now', x)
    elif x == 'WT':
        print(x)
        x = x.replace('WT','WHITE')
        print('Now', x)
    elif x == 'GY':
        print(x)
        x = x.replace('GY','GREY')
        print('Now', x)
    elif x == 'GRAY':
        print(x)
        x = x.replace('GRAY','GREY')
        print('Now', x)
    elif x == 'GRY':
        print(x)
        x = x.replace('GRY','GREY')
        print('Now', x)
    elif x == 'G/Y':
        print(x)
        x = x.replace('G/Y','GREY')
        print('Now', x)
    elif x == 'BR':
        print(x)
        x = x.replace('BR','BROWN')
        print('Now', x)
    elif x == 'BRW':
        print(x)
        x = x.replace('BRW','BROWN')
        print('Now', x)
    elif x == 'TN':
        print(x)
        x = x.replace('GY','TAN')
        print('Now', x)
    elif x == 'RD':
        print(x)
        x = x.replace('RD','RED')
        print('Now', x)
    elif x == 'BL':
        print(x)
        x = x.replace('BL','BLUE')
        print('Now', x)
    elif x == 'BLU':
        print(x)
        x = x.replace('BLU','BLUE')
        print('Now', x)
    elif x == 'SIL':
        print(x)
        x = x.replace('SIL','SILVER')
        print('Now', x)
    elif x == 'SILVE':
        print(x)
        x = x.replace('SILVE','SILVER')
        print('Now', x)
    elif x == 'BURGA':
        print(x)
        x = x.replace('BURGA','BURGANDY')
        print('Now', x)
    elif x == 'PURPL':
        print(x)
        x = x.replace('PURPL','PURPLE')
        print('Now', x)
    elif x == 'PUR':
       ...


I am working on the file "11/Homework_11_Emelike.ipynb".
The first task is:
8. How much money did NYC make off of parking violations?
Can you write Python code for it?

# removes unsuccessful merges
df4 = df3[df3['_merge'] !='left_only']

# Create boolean variable to indicate 0/1: Parking violation
df4['Parking Violation'] = df4['DEFINITION'].str.contains('[Pp]arking') | df4['DEFINITION'].str.contains('[Pp]arked')
df4['Parking Violation'].value_counts()
df4[df4['Parking Violation'] == False]

df4['Street Name'].value_counts()
#create empty list
above_96_lst=[]
#create bool object
for x in df4['Street Name']:
    if re.search(r'\S \d \S\w ', str(x)):
        above_96 =  False
        above_96_lst.append(above_96)
    elif re.search(r'\d[[Aa]-[Zz]][[Aa]-[Zz]] ', str(x)):
        above_96 =  False
        above_96_lst.append(above_96)
#         print(x, above_96)
    elif re.search(r'\w\s\d\d\s', str(x)):
        above_96 =  False
        above_96_lst.append(above_96)
#         print(x, above_96)
    elif re.search(r'[A-Z]\s\d\d[A-Z][A-Z] [A-Z][A-Z]', str(x)):
        above_96 =  False
        above_96_lst.append(above_96)
#         print(x, above_96)
    elif re.search(r'[0-2][A-Z][A-Z] AVE', str(x)):
        above_96 =  False
        above_96_lst.append(above_96)
#         print(x, above_96)
    else:
        above_96 =  True
        above_96_lst.append(above_96)
#         print(x, above_96)

# above_96_lst.count(False)
df4['Above_96'] = above_96_lst

# rename column so easier to reference
df4.rename(columns={"Manhattan\xa0 96th St. & below": "Manhattan_Below_96"}, inplace=True)

df4[['Violation Code', 'Street Name', 'Above_96', 'Manhattan_Below_96', 'All Other Areas', 'Parking Violation']].head(20)

# remove dollar signs
df4['Manhattan_Below_96'] = df4['Manhattan_Below_96'].str.replace('$', '')

# replace long descriptions of cost with only the amount
# df4['Manhattan_Below_96'] = df4['Manhattan_Below_96'].str.replace('100\n(Regular Tow, plus violation fine)', '100')
df4.ix[df4['Manhattan_Below_96'] == '100\n(Regular Tow, plus violation fine)', 'Manhattan_Below_96'] = '100'

# replace 'vary' with zero
df4['Manhattan_Below_96'] =...


I am working on the file "11/Homework_11_Paolo_Rivas.ipynb".
The first task is:
10. New Jersey has bad drivers, but does it have bad parkers, too? How much money does NYC make off of all non-New York vehicles?
Can you write Python code for it?

new_df.groupby('Registration State')['All Other Areas 2'].sum().sort_values(ascending=False).head(10)

print('The city has made $274810 of the non newyorkers')
NY_fines= new_df.groupby('Registration State')['All Other Areas 2'].sum().sort_values(ascending=False).head(1)
outcome1 - NY_fines


I am working on the file "11/Homework_11_Paolo_Rivas.ipynb".
The first task is:
12. What time of day do people usually get their tickets? You can break the day up into several blocks - for example 12am-6am, 6am-12pm, 12pm-6pm,6pm-12am.
Can you write Python code for it?

new_df['New Violation Time'].head(10)

def hour_transformer(x):
    try:
        time = int(x[:2])
        if time  <= 6:
            return '12am-6am'
        elif time <= 12:
            return '6am-12pm'
        elif time <= 18:
            return '12pm-6pm'
        elif time <= 24:
            return '6pm-12am'
        else:
            pass
    except:
        pass

day_time = new_df['New Violation Time'].apply(hour_transformer)

day_time.value_counts().plot.pie(title='Ticket time!')


I am working on the file "11/Homework_11_Paolo_Rivas.ipynb".
The first task is:
14. Make a graph of the number of tickets per day.
Can you write Python code for it?

new_df['Issue Date'].describe()

new_df.groupby('Issue Date')['Issue Date'].value_counts(sort=False).plot.bar(figsize=(15, 6))
plt.ylabel('Number of tickets')
plt.xlabel('Days')
#it seems like all the data is concentrated
#only in a few years

new_df.groupby('Issue Date')['All Other Areas 2'].sum().plot(kind="bar", figsize=(15, 6))
plt.ylabel('Amount in $')
plt.xlabel('Days')


I am working on the file "11/Homework_11_Paolo_Rivas.ipynb".
The first task is:
16. Manually construct a dataframe out of https://dmv.ny.gov/statistic/2015licinforce-web.pdf (only NYC boroughts - bronx, queens, manhattan, staten island, brooklyn), having columns for borough name, abbreviation, and number of licensed drivers.
Can you write Python code for it?


I am working on the file "11/Homework_11_Paolo_Rivas.ipynb".
The first task is:
4. "Date first observed" is a pretty weird column, but it seems like it has a date hiding inside. Using a function with .apply, transform the string (e.g. "20140324") into a Python date. Make the 0's show up as NaN.
Can you write Python code for it?

df['Date First Observed'].value_counts()

import dateutil.parser

def first_observed_function(x):
    try:
        x= str(x)
        print("NaN")
        if x == '0':
            return np.nan
        else:
            print("transforming...")
            date_clean = dateutil.parser.parse(x)
            return date_clean.strftime("%Y-%d-%m")
    except:
        return None

first_observed_function('20130731')

df['Clean Date First Observed']= df['Date First Observed'].apply(first_observed_function)

df['Clean Date First Observed'].value_counts()


I am working on the file "11/Homework_11_Paolo_Rivas.ipynb".
The first task is:
6. There sure are a lot of colors of cars, too bad so many of them are the same. Make "BLK" and "BLACK", "WT" and "WHITE", and any other combinations that you notice.
Can you write Python code for it?

df['Vehicle Color'].value_counts()

def color(color):
    if (color == "BK") or (color == "BL"):
            return 'BLACK'
    if (color == "WHT") or (color == "WT") or (color == 'WH'):
            return 'WHITE'
    else:
            return color

#example
color('BK'), color('WHT'), color('BL'), color('WT'), color('WH')

df['B&W Clean Vehicle Color'] = df['Vehicle Color'].apply(color)

df['B&W Clean Vehicle Color'].value_counts()


I am working on the file "11/Homework_11_Paolo_Rivas.ipynb".
The first task is:
7. Join the data with the Parking Violations Code dataset from the NYC Open Data site
Can you write Python code for it?

!head -n 10000 DOF_Parking_Violation_Codes.csv > small_DOF_Parking_Violation_Codes.csv

violations_data = pd.read_csv("small_DOF_Parking_Violation_Codes.csv")

violations_data.head(2)

type(violations_data['CODE'])

violations_data['CODE'].value_counts()

def transform_code(x):
    try:
        new_code = x[0:2]
        return new_code
    except:
        return None

single_code = violations_data['CODE'].apply(transform_code)

violations_data['int CODE'] = single_code.astype(int)

violations_data['int CODE'].dtype #now is an integer

violations_data.head(129)

#I need to do this same process to the df['Violation Code'] because to transform it to a INT
old_df = df["Violation Code"].apply(transform_code)

df.head(10)

df['Violation Code 2'] = old_df.astype(int)

#Merging the two data sets
new_df= df.merge(violations_data, left_on="Violation Code 2", right_on="int CODE")

new_df.head(40)


I am working on the file "11/Homework_11_Paolo_Rivas.ipynb".
The first task is:
8. How much money did NYC make off of parking violations?
Can you write Python code for it?

new_df['All Other Areas'].value_counts()

new_df[ 'Manhattan\xa0 96th St. & below'].value_counts()

#First, I will transfrom all values into integers
def money_transformer(money_string):
    if money_string == '200 (Heavy Tow plus violation fine)':
        string_only = money_string[:3]
        return int(string_only)
    if money_string == '100\n(Regular Tow, plus violation fine)':
        string_only = money_string[:3]
        return int(string_only)
    try:
        return int(money_string.replace("$","").replace(",",""))
    except:
        return None

new_df['All Other Areas 2'] = new_df['All Other Areas'].apply(money_transformer)

new_df['Manhattan\xa0 96th St. & below 2'] = new_df['Manhattan\xa0 96th St. & below'].apply(money_transformer)

outcome1 = new_df['All Other Areas 2'].sum()
outcome2 = new_df['Manhattan\xa0 96th St. & below 2'].sum()

print("NYC makes between","$", outcome1, "US dollars and","$", outcome2, "US dollars of parking violations")
#PS. Data set has been cut to 10000 rows for memory saving reasons. Output would be considerably higher with the complete DF.


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
1. I want to make sure my Plate ID is a string. Can't lose the leading zeroes!
Can you write Python code for it?

#As the datatypes in all of the columns vary, I decided to to make all the values, except for the ones I specify
#into str. This also takes care of questions 1. -> dtype=str


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
10. New Jersey has bad drivers, but does it have bad parkers, too? How much money does NYC make off of all non-New York vehicles?
Can you write Python code for it?

df_merged['Registration State'].value_counts().head(10)

Fines_for_New_York_Cars = df_merged.groupby('Registration State')['All Other Areas'].sum().sort_values(ascending=False).head(1)

Fines_for_New_York_Cars

Mininum_Amount #Total fines

print("New York City makes ${:,.2f} off non-New-York cars.".format(Mininum_Amount - 221829545.0))

#Double Check:
df_merged[df_merged['Registration State'] != 'NY'].sum()


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
11. Make a chart of the top few.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use("ggplot")

df_merged['Registration State'].value_counts().sort_values(ascending=True).tail(10).plot(kind='barh')


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
12. What time of day do people usually get their tickets? You can break the day up into several blocks - for example 12am-6am, 6am-12pm, 12pm-6pm,6pm-12am.
Can you write Python code for it?

import dateutil

def Time_of_day(x):
    x = str(x)
    #x = (':'.join(a+b for a,b in zip(x[::2], x[1::2])))
    if x == 'NaN':
        pass
    try:
        x = str(x[:2] + ':' + x[2:])
        date = dateutil.parser.parse(x)
        time = int(str(date.strftime('%H')))
        if time <= 6:
            return '12am - 6am'
        elif time <= 12:
            return '6am - 12pm'
        elif time <= 18:
            return '12pm - 6pm'
        elif time <= 24:
            return '6pm - 12am'
    except:
        return 'NaN'

df_merged['Time of Day'] = df['Violation Time'].apply(Time_of_day)

df_merged['Time of Day'].value_counts()


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
15. Make a graph of the amount of revenue collected per day.
Can you write Python code for it?

df_merged['Issue Date'].head()

plt.plot(range(5))
plt.gca().xaxis.set_major_locator(plt.NullLocator())
df_merged_only_2013_dates.groupby('Issue Date')['All Other Areas'].sum().plot(kind="bar", figsize=(15, 6), legend=None, title='Total daily fine amount in 2013')
plt.show()


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
16. Manually construct a dataframe out of https://dmv.ny.gov/statistic/2015licinforce-web.pdf (only NYC boroughts - bronx, queens, manhattan, staten island, brooklyn), having columns for borough name, abbreviation, and number of licensed drivers.
Can you write Python code for it?

df_boroughs = pd.read_csv("NYC_Boroughs3.csv")

df_boroughs.head()


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
17. What's the parking-ticket-$-per-licensed-driver in each borough of NYC? Do this with pandas and the dataframe you just made, not with your head!
Can you write Python code for it?

def county_harm(county):
    county = str(county)
    if county == 'NaN':
        return
    else:
        return county.replace("BRONX","BX").replace("KINGS", "K").replace("NYC", "NY").replace("QUEEN", "Q").replace("RC", "R").replace("RICH", "R")

df_merged['Violation County harm'] = df_merged['Violation County'].apply(county_harm)

summed_fine_amounts_by_county = df_merged.groupby('Violation County harm')['All Other Areas'].sum()

#creating dataframe
fine_amounts = pd.DataFrame(summed_fine_amounts_by_county)

#changing the index into column
fine_amounts.reset_index(inplace=True)

fine_amounts

#Joining the two new datasets
df_ticket_dollars_per_licence = fine_amounts.merge(df_boroughs, left_on='Violation County harm', right_on='abbreviation')

df_ticket_dollars_per_licence

df_ticket_dollars_per_licence['Ticket Dollars per Licence'] = df_ticket_dollars_per_licence['All Other Areas'] / df_ticket_dollars_per_licence['Total']

#Dividing one dataset by the other
df_ticket_dollars_per_licence[['borough name', 'abbreviation', 'Ticket Dollars per Licence']]


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
2. I don't think anyone's car was built in 0AD. Discard the '0's as NaN.
Can you write Python code for it?

# The syntax for this is really very nice and clear, an example na_values= {'Vehicle Year' : ['0']}
# A lot more here: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
3. I want the dates to be dates! Read the read_csv documentation to find out how to make pandas automatically parse dates.
Can you write Python code for it?

#parse_dates=True,keep_date_col = True

import datetime
import datetime as dt

dt.datetime.strptime('08/04/2013', '%m/%d/%Y')
datetime.datetime(2013, 8, 4, 0, 0)

parser = lambda date: pd.datetime.strptime(date, '%m/%d/%Y')

#taken from here: http://stackoverflow.com/questions/23797491/parse-dates-in-pandas
#adding this, column counts starts at 0, without index, parse_dates=[4], date_parser=parser

#nrows=100000,

df = pd.read_csv("violations.csv", na_values= {'Vehicle Year' : ['0']}, parse_dates=[4], date_parser=parser, dtype=str)

df.head()

#df.info()


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
5. "Violation time" is... not a time. Make it a time.
Can you write Python code for it?

#adding n_values = {'Violation Time' : ['0']}
#df = pd.read_csv("violations.csv", na_values= {'Vehicle Year' : ['0'], 'Violation Time' : ['0']}, parse_dates=[4], date_parser=parser, dtype=str)

df['Violation Time'].head(5)

#df['Violation Time'].value_counts()

import dateutil

def pmam(x):
    x = str(x)
    #x = (':'.join(a+b for a,b in zip(x[::2], x[1::2])))
    if x == 'NaN':
        pass
    try:
        x = str(x[:2] + ':' + x[2:])
        date = dateutil.parser.parse(x)
        return str(date.strftime('%H:%M %p'))
    except:
        return 'NaN'

print(pmam('NaN'))
print(pmam("0752A"))
print(pmam("1240P"))
print(pmam("0232P"))

df['Violation Time pmam'] = df['Violation Time'].apply(pmam)


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
6. There sure are a lot of colors of cars, too bad so many of them are the same. Make "BLK" and "BLACK", "WT" and "WHITE", and any other combinations that you notice.
Can you write Python code for it?

df['Vehicle Color'].head(10)

def black_and_white(car_color):
    car_color = str(car_color)
    if car_color == 'NaN':
        return 'NaN'
    elif car_color == 'WHITE':
        return car_color
    elif car_color == 'BLACK':
        return car_color
    else:
        return car_color.replace("BK","BLACK").replace("WH", "WHITE")

def rainbow(car_color):
    car_color = str(car_color)
    if car_color == 'NaN':
        return car_color
    elif car_color == 'BROWN':
        return car_color
    elif car_color == 'BLUE':
        return car_color
    elif car_color == 'RED':
        return car_color
    elif car_color == 'GREY':
        return car_color
    elif car_color == 'GREEN':
        return car_color
    else:
        return car_color.replace("BR","BROWN").replace("RD", "RED").replace("GY", "GRAY").replace("TN", "TAN").replace("BLU", "BLUE").replace("GN", "GREEN")

df['Vehicle Color'].apply(black_and_white).apply(rainbow).head(5)


I am working on the file "11/Homework_11_working_with_parking_data-Master.ipynb".
The first task is:
7.Join the data with the Parking Violations Code dataset from the NYC Open Data site.
Can you write Python code for it?

DOF_Parking_Violation_Codes_df = pd.read_csv("Parking_Fine_Codes_Man_Quota.csv")
DOF_Parking_Violation_Codes_df.head()

def make_code_int(code):
    return int(code)

df['Violation Code int'] = df['Violation Code'].apply(make_code_int)

df_merged = df.merge(DOF_Parking_Violation_Codes_df, left_on='Violation Code int', right_on='CODE')

df_merged.head(1)


I am working on the file "11/Homework_11_working_with_parking_data.ipynb".
The first task is:
10. New Jersey has bad drivers, but does it have bad parkers, too? How much money does NYC make off of all non-New York vehicles?
Can you write Python code for it?

df_merged['Registration State'].value_counts().head(10)

Fines_for_New_York_Cars = df_merged.groupby('Registration State')['All Other Areas int'].sum().sort_values(ascending=False).head(1)

Fines_for_New_York_Cars

Mininum_Amount #Total fines

print("New York City makes ${:,.2f} off non-New-York cars.".format(Mininum_Amount - 220943620.0))

#Double Check:
df_merged[df_merged['Registration State'] != 'NY'].sum()


I am working on the file "11/Homework_11_working_with_parking_data.ipynb".
The first task is:
11. Make a chart of the top few.
Can you write Python code for it?


I am working on the file "11/Homework_11_working_with_parking_data.ipynb".
The first task is:
14. Make a graph of the number of tickets per day.
Can you write Python code for it?

df_merged['Issue Date'].head()

#Filtering out all the dates that were in the future and before 2013, obviously wrong dates.
df_merged_only_2013_dates = df_merged[(df_merged['Issue Date'] > '2013-01-01') & (df_merged['Issue Date'] < '2013-12-31')]

#More on histograms
#http://stackoverflow.com/questions/27365467/python-pandas-plot-histogram-of-dates
#http://stackoverflow.com/questions/29672375/histogram-in-matplotlib-time-on-x-axis
df_merged_only_2013_dates.groupby('Issue Date')['Issue Date'].count().plot(kind="bar", figsize=(15, 6))


I am working on the file "11/Homework_11_working_with_parking_data.ipynb".
The first task is:
15. Make a graph of the amount of revenue collected per day.
Can you write Python code for it?

df_merged['Issue Date'].head()

df_merged_only_2013_dates.groupby('Issue Date')['All Other Areas int'].sum().plot(kind="bar", figsize=(15, 6))


I am working on the file "11/Homework_11_working_with_parking_data.ipynb".
The first task is:
17. What's the parking-ticket-$-per-licensed-driver in each borough of NYC? Do this with pandas and the dataframe you just made, not with your head!
Can you write Python code for it?

df_merged['Violation County'].value_counts()

df_merged.groupby('Violation County')['All Other Areas int'].sum()

def boroughs_trans(borough):
    borough = str(borough)
    if borough == 'NaN':
        pass
    elif borough == 'RICH':
        return 'R'
    elif borough == 'BRONX':
        return 'BX'
    else:
        return None


I am working on the file "11/Homework_11_working_with_parking_data.ipynb".
The first task is:
3. I want the dates to be dates! Read the read_csv documentation to find out how to make pandas automatically parse dates.
Can you write Python code for it?


I am working on the file "11/Homework_11_working_with_parking_data.ipynb".
The first task is:
4. "Date first observed" is a pretty weird column, but it seems like it has a date hiding inside. Using a function with .apply, transform the string (e.g. "20140324") into a Python date. Make the 0's show up as NaN.
Can you write Python code for it?

#df['Date First Observed'].value_counts()

import dateutil.parser

def date_transfer(x):
    x = str(x)
    if x == '0':
        return 'NaN'
    try:
        date = dateutil.parser.parse(x)
        return str(date.strftime('%d/%m/%Y'))
    except:
        return 'NaN'
    #    dt.datetime.strptime('20130719', '%Y%m%d')
    #    datetime.datetime(2013, 7, 19, 0, 0)
    #    year_month_tag = dateutil.parser.parse(x)
    return x
print(date_transfer(20151198))

df['Date First Observed'].apply(date_transfer).head()


I am working on the file "11/Homework_11_working_with_parking_data.ipynb".
The first task is:
6. There sure are a lot of colors of cars, too bad so many of them are the same. Make "BLK" and "BLACK", "WT" and "WHITE", and any other combinations that you notice.
Can you write Python code for it?


I am working on the file "11/Homework_11_working_with_parking_data.ipynb".
The first task is:
7.Join the data with the Parking Violations Code dataset from the NYC Open Data site.
Can you write Python code for it?

DOF_Parking_Violation_Codes_df = pd.read_csv("DOF_Parking_Violation_Codes.csv")
DOF_Parking_Violation_Codes_df.head()

df_merged = df.merge(DOF_Parking_Violation_Codes_df, left_on='Violation Code', right_on='CODE')


I am working on the file "11/Homework_11_working_with_parking_data.ipynb".
The first task is:
9. What's the most lucrative kind of parking violation? The most frequent?
Can you write Python code for it?

def def_strip(definition):
    return definition.strip()

df_merged['DEFINITION_stripped'] = df_merged['DEFINITION'].apply(def_strip)

#Most frequent kind of parking violation
df_merged['DEFINITION'].value_counts().head(1)

count_by_violation_code = df_merged['CODE'].value_counts()

count_by_violation_code.head()

df_merged.groupby('CODE')['All Other Areas int'].sum().sort_values(ascending=False).head()

Most_lucrative_amount = df_merged.groupby('CODE')['All Other Areas int'].sum().sort_values(ascending=False)[1]

Most_lucrative_fine_in_words = df_merged[df_merged['CODE'] == '14']

Most_lucrative_fine_in_words = Most_lucrative_fine_in_words['DEFINITION'].head(1)

print("The most lucrative fine regarded ${:,.2f} on parking fines.".format(Most_lucrative_amount))
print("It regarded " + Most_lucrative_fine_in_words)


I am working on the file "11/Parking Data.ipynb".
The first task is:
"Date first observed" is a pretty weird column, but it seems like it has a date hiding inside. Using a function with .apply, transform the string (e.g. "20140324") into a Python date. Make the 0's show up as NaN.
Can you write Python code for it?

# print(violations_df['Date First Observed'].head())

import datetime
violations_df.head()['Issue Date'].astype(datetime.datetime)

import datetime
violations_df.head()['Issue Date'].astype(datetime.datetime)

#violations_df['Issue Date'] = violations_df['Issue Date'].astype('datetime64[ns]')

# violations_df.dtypes


I am working on the file "11/Parking Data.ipynb".
The first task is:
"Violation time" is... not a time. Make it a time.
Can you write Python code for it?

import re
# violations_df['Violation Time'].head()

import numpy as np
# Build a function using that method
def time_to_datetime(str_time):
    try:
    #str_time = re.sub('^0', '', str_time)
        if isinstance(str_time, str):
            str_time = str_time + "M"
            #print("Trying to convert", str_time, "into a time")
            return datetime.datetime.strptime(str_time.strip(), "%I%M%p").time()
    #try:
     #   if str_time == -999:
      #      print("It's -999")
       #     return np.nan

    except:
        return np.nan

# Apply that method to the 'Time' column of the dataframe
violations_df['Violation Time'] = violations_df['Violation Time'].apply(time_to_datetime)

print(violations_df['Violation Time'].head())

# def remove_1900(year):
#     year = str(year)
#     year = re.sub('^1900-01-01', '', year)
#     return year

# print(violations_df['Violation Time'].apply(remove_1900))


I am working on the file "11/Parking Data.ipynb".
The first task is:
How much money does NYC make off of all non-New York vehicles?
11. Make a chart of the top few.
Can you write Python code for it?

violations_df.groupby('Registration State')['All Other Areas'].sum().sort_values(ascending=False).head(8)

import matplotlib.pyplot as plt
%matplotlib inline

nonny_violations_df = violations_df[violations_df['Registration State'] != 'NY']
nonny_violations_df = nonny_violations_df[violations_df['Registration State'] != '99']
nonNYviolations = nonny_violations_df.groupby('Registration State')['All Other Areas'].sum().sort_values(ascending=False).head(20).plot(kind="bar", color = ['#624ea7', '#599ad3', '#f9a65a', '#9e66ab', 'purple'])
nonNYviolations.set_title('Money Earned from NYC Parking Violations by State')
nonNYviolations.set_xlabel('Vehicle Registration State (NON-NYS)')
nonNYviolations.set_ylabel('Amount Earned')
plt.savefig('nonNYParkingViolations.png')

import matplotlib.pyplot as plt
%matplotlib inline

nonny_violations_df = violations_df[violations_df['Registration State'] != 'NY']
nonny_violations_df = nonny_violations_df[violations_df['Registration State'] != '99']
freqnonNYviolations = nonny_violations_df.groupby('Registration State')['All Other Areas'].count().sort_values(ascending=False).head(20).plot(kind="bar", color = ['#624ea7', '#599ad3', '#f9a65a', '#9e66ab', 'purple'])
freqnonNYviolations.set_title('Frequency of NYC Parking Violations by State')
freqnonNYviolations.set_xlabel('Vehicle Registration State (NON-NYS)')
freqnonNYviolations.set_ylabel('Number of Citations')
plt.savefig('freqnonNYParkingViolations.png')


I am working on the file "11/Parking Data.ipynb".
The first task is:
I don't think anyone's car was built in 0AD. Discard the '0's as NaN.
Can you write Python code for it?

violations_df = violations_df[violations_df['Vehicle Year'] != 0]
# print(violations_df['Vehicle Year'].head())


I am working on the file "11/Parking Data.ipynb".
The first task is:
Join the data with the Parking Violations Code dataset from the NYC Open Data site.
Can you write Python code for it?

violations_df = violations_df.merge(dof_violations_df, left_on='Violation Code', right_on='CODE')


I am working on the file "11/Parking Data.ipynb".
The first task is:
Make a graph of the amount of revenue collected per day.
Can you write Python code for it?

freqnonNYviolations = violations_df.groupby('Issue Date')['All Other Areas'].sum().plot(kind="bar", color = ['#624ea7', '#599ad3', '#f9a65a', '#9e66ab', 'purple'])
freqnonNYviolations.set_title('Revenue of NYC Parking Violations by Date')
freqnonNYviolations.set_xlabel('Date')
freqnonNYviolations.set_ylabel('Revenue Generated from Citations')
plt.savefig('datedrevenueonNYParkingViolations.png')


I am working on the file "11/Parking Data.ipynb".
The first task is:
New Jersey has bad drivers, but does it have bad parkers, too?
Can you write Python code for it?

nj_violations_df = violations_df[violations_df['Registration State'] != 'NJ']

print("There were", len(nj_violations_df['All Other Areas']), "parking violations in NYC by drivers registered in New Jersey")
print("These violations totaled", nj_violations_df['All Other Areas'].sum(), "of revenue")


I am working on the file "11/Parking Data.ipynb".
The first task is:
The most frequent parking violation is:
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
freqNYviolations = violations_df.groupby('CODE')['All Other Areas'].count().sort_values(ascending=False).head().plot(kind="bar", color = ['#624ea7', '#599ad3', '#f9a65a', '#9e66ab', 'purple'])
freqNYviolations.set_title('Most Frequent NYC Parking Violations by Code')
freqNYviolations.set_xlabel('Parking Violation Code')
freqNYviolations.set_ylabel('Frequency of Citations')
plt.savefig('freqParkingViolations.png')

violations_df.groupby('CODE')['All Other Areas'].count().sort_values(ascending=False).head(1)

# violations_df.sort_values('All Other Areas', ascending=False)


I am working on the file "11/Parking Data.ipynb".
The first task is:
There sure are a lot of colors of cars, too bad so many of them are the same. Make "BLK" and "BLACK", "WT" and "WHITE", and any other combinations that you notice.
Can you write Python code for it?

import re

def vehicle_colors(vehicle):
    if isinstance(vehicle, str):
        #print(vehicle)
        vehicle = re.sub('^GY$', 'GREY', vehicle)
        #vehicle = vehicle.replace("GY", "GREY")
        vehicle = re.sub('^WH$', 'WHITE', vehicle)
        vehicle = re.sub('^BR$', 'BROWN', vehicle)
        vehicle = re.sub('^RD$', 'RED', vehicle)
        vehicle = re.sub('^B[LK]$', 'BLACK', vehicle)
        vehicle = re.sub('^TN$', 'TAN', vehicle)
        vehicle = re.sub('^YW$', 'YELLOW', vehicle)
        vehicle = re.sub('^SIL$', 'SILVER', vehicle)
        vehicle = re.sub('^GR$', 'GREEN', vehicle)
        vehicle = re.sub('^SILVE$', 'SILVER', vehicle)

#         vehicle = vehicle.replace("WH","WHITE")
#         vehicle = vehicle.replace("BR","BROWN")
#         vehicle = vehicle.replace("RD","RED")
#         vehicle = vehicle.replace("BL","BLACK")
#         vehicle = vehicle.replace("BK","BLACK")
#         vehicle = vehicle.replace("TN","TAN")
#         vehicle = vehicle.replace("YW","YELLOW")
#         vehicle = re.sub('SIL$', 'SILVER', vehicle)
#         vehicle = vehicle.replace("SILVR","SILVER")
#         vehicle = vehicle.replace("SIL","SILVER")
    return vehicle

violations_df['Vehicle Color'] = violations_df['Vehicle Color'].apply(vehicle_colors)

violations_df['Vehicle Color'].head()


I am working on the file "11/Parking Data.ipynb".
The first task is:
What time of day do people usually get their tickets? You can break the day up into several blocks - for example 12am-6am, 6am-12pm, 12pm-6pm, 6pm-12am.

What's the average ticket cost in NYC?
Can you write Python code for it?

print("The average ticket cost in NYC is $", violations_df['All Other Areas'].mean())


I am working on the file "11/Parking Data.ipynb".
The first task is:
What's the most lucrative kind of parking violation? Below lists all the ones that chaarge $115, which is the highest amount
Can you write Python code for it?


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
'All Other Areas' and 'Manhattan 96th st and below' were not integers:
Can you write Python code for it?

def int_dollar(str_dollar):
    try:
        if pd.isnull(str_dollar):
            return None
        else:
            return int(str_dollar)
    except:
        return None

df_merged['All Other Areas Int'] = df_merged['All Other Areas'].apply(int_dollar)
df_merged['Manhattan\xa096th st and below Int'] = df_merged['Manhattan\xa096th st and below'].apply(int_dollar)

df_merged.columns


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
1. I want to make sure my Plate ID is a string. Can't lose the leading zeroes!
Can you write Python code for it?

# test to make sure we haven't lost leading zeroes
# type(df['Plate ID'][0])

# test to make sure we haven't lost leading zeroes
# df['Plate ID'][0][0]

# test to make sure we haven't lost leading zeroes
# number_of_plates = range(len(df['Plate ID']))
# for number in number_of_plates:
#     if str(df['Plate ID'][number])[0] == '0': #used str as a workaround for the nan value at 2743
#         print(df['Plate ID'][number])

# # troubleshooting bc 2743 is na
# type(df['Plate ID'][2743])
# # if df['Plate ID'][2743]:
# #     print(df['Plate ID'][2743])


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
10. New Jersey has bad drivers, but does it have bad parkers, too? How much money does NYC make off of all non-New York vehicles?
Can you write Python code for it?

df_non_NY = df_merged[df_merged['Registration State'] != 'NY']

df_non_NY['Fee for This Precinct'].sum()


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
11. Make a chart of the top few.
Make a list of all of the non-NY registration states and grab the top 10.
<p> for each of the top 10, find the frequency
    `[
    {'Registration State': '99', 'total fees': '999' },
    {'Registration State': 'AB', 'total fees': '999' } ...
    ]`
<p> then put them in a dataframe and use matplotlib
Can you write Python code for it?

df_non_NY['Registration State'].value_counts()

top_out_of_state = ['NJ', 'PA', 'CT', 'FL', 'MA', 'VA', 'MD', '99', 'NC', 'IL']

out_of_state_list = []
out_of_state_dict = {}

for state in top_out_of_state:
    df_this_state = df_merged[df_merged['Registration State'] == state]
    fees = df_this_state['Fee for This Precinct'].sum()
    out_of_state_dict['Registration State'] = state
    out_of_state_dict['total fees'] = fees
    out_of_state_list.append(out_of_state_dict)
    out_of_state_dict = {}

df_out_of_state = pd.DataFrame(out_of_state_list)

%matplotlib inline

ax = df_out_of_state.plot(kind = 'bar', x = 'Registration State', y = 'total fees', legend = None)
ax.set_title('Violation Fees Collected by NY from Non-NY Cars from 1973 to 2060')


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
13. What's the average ticket cost in NYC?
Can you write Python code for it?


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
15. Make a graph of the amount of revenue collected per day.
Let's try this:

We already have the issue dates into a list

We can also group the information by date and find the sum of that revenue. Let's do that and see where we can take it from there.
Can you write Python code for it?

df_revenue_per_day = df_merged[['Issue Date', 'Fee for This Precinct']].groupby('Issue Date').sum()

df_revenue_per_day

date_and_revenue_dict = df_revenue_per_day.to_dict()['Fee for This Precinct']

rev_today = []
rev_dict = {}
for date in sorted_issue_dates_list:
    rev_dict['Issue Date'] = date
    if date_and_revenue_dict[date]:
        rev_dict['Revenue Today'] = date_and_revenue_dict[date]
    else:
        rev_dict['Revenue Today'] = 0
    rev_today.append(rev_dict)
    rev_dict = {}

print(rev_today)


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
16. Manually construct a dataframe out of https://dmv.ny.gov/statistic/2015licinforce-web.pdf (only NYC boroughts - bronx, queens, manhattan, staten island, brooklyn), having columns for borough name, abbreviation, and number of licensed drivers.
Can you write Python code for it?

df_licenses = pd.read_csv('driver_licenses.csv')
df_licenses


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
17. What's the parking-ticket-$-per-licensed-driver in each borough of NYC? Do this with pandas and the dataframe you just made, not with your head!
Can you write Python code for it?

df_fees_per_county = df_merged[['Violation County', 'Fee for This Precinct']].groupby('Violation County').sum()

df_fees_per_county['Violation County'] = df_fees_per_county.index

df_fees_per_county

df_last_question = df_licenses.merge(df_fees_per_county, right_on = 'Violation County', left_on = 'abbreviation')

df_last_question

df_last_question['Fee for This Precinct'] / df_last_question['number of licensed drivers']


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
3. I want the dates to be dates! Read the read_csv documentation to find out how to make pandas automatically parse dates.
Can you write Python code for it?

# seeing if parse_dates = True worked:
# df['Issue Date'].head()


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
4. "Date first observed" is a pretty weird column, but it seems like it has a date hiding inside. Using a function with .apply, transform the string (e.g. "20140324") into a Python date. Make the 0's show up as NaN.
Can you write Python code for it?

df['Date First Observed'][30: 35]

# 33 is a date
# Did it for the whole dataframe instead of for each row
# def year(row):
#     return df['Date First Observed'][row][:4]
# def month(row):
#     return df['Date First Observed'][row][4:6]
# def day(row):
#     return df['Date First Observed'][row][-2:]
# def date(row):
#     return {
#         'year': year(row),
#         'month': month(row),
#         'day': day(row)
#         }

def year(row):
    return row[:4]
def month(row):
    return row[4:6]
def day(row):
    return row[-2:]
def date(row):
    if len(row) == 8:
        dictionary = {
            'year': year(row),
            'month': month(row),
            'day': day(row)
            }
        return '{year}-{month}-{day}'.format(**dictionary)
    else:
        return None

df['Date First Observed Formatted'] = df['Date First Observed'].apply(date)

df['Date First Observed Formatted'][30: 35]


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
6.  There sure are a lot of colors of cars, too bad so many of them are the same. Make "BLK" and "BLACK", "WT" and "WHITE", and any other combinations that you notice.
Can you write Python code for it?

df['Vehicle Color'].value_counts()

def color(row):
    # white
    if row == 'WH' or row == 'WHT' or row == 'WT' or row == 'WHI' or row == 'W':
        return 'WHITE'
    # gray
    elif row == 'GY' or row == 'GREY' or row == 'GRY' or row == 'GREW':
        return 'GRAY'
    # black
    elif row == 'BK' or row == 'BLK' or row == 'BCK':
        return 'BLACK'
    # blue. I'm guessing BL is BLUE since there are two-letter codes and BK is BLACK
    elif row == 'NAVYB' or row == 'BL' or row == 'B LU' or row == 'BLU':
        return 'BLUE'
     # silver
    elif row == 'SILVE' or row == 'SLVIR' or row == 'SL' or row == 'SIL' or row == 'SILVR' or row == 'SILV' or row == 'SILIV':
        return 'SILVER'
    # red
    elif row == 'RD' or row == 'REDD':
        return 'RED'
    # green
    elif row == 'GR' or row == 'GRN' or row == 'GN':
        return 'GREEN'
    # tan
    elif row == 'TN' or row == 'BEIGE':
        return 'TAN'
    # brown
    elif row == 'BR' or row == 'BRN' or row == 'BN' or row == 'BRWN' or row == 'BRW' or row == 'BRO':
        return 'BROWN'
    # yellow
    elif row =='YW' or row == 'YELLO' or row == 'YE' or row == 'YELL' or row == 'YEL':
        return 'YELLOW'
    # gold
    elif row == 'GL' or row == 'GD' or row == 'GLD':
        return 'GOLD'
    # orange
    elif row == 'ORANG' or row == 'ORNAG' or row == 'OR':
        return 'ORANGE'
    # purple
    elif row == 'PURPL' or row == 'LAVEN':
        return 'PURPLE'
    # turquoise
    elif row == 'TQ' or row == 'TURQ':
        return 'TURQUOISE'
    # maroon
    elif row == 'MAROO' or row == 'BRGND' or row == 'BURG':
        return 'MAROON'
    else:
        return row

df['Vehicle Color Formatted'] = df['Vehicle Color'].apply(color)
df['Vehicle Color Formatted'].value_counts()


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
8. How much money did NYC make off of parking violations?
Aside: Let's Fix the Counties and Integers
'Violation County' included 'RICH', 'BRONX', and 'KINGS':
Can you write Python code for it?

def fix_counties(county_name):
    if county_name == 'RICH':
        return 'R'
    elif county_name == 'BRONX':
        return 'BX'
    elif county_name == 'KINGS':
        return 'K'
    else:
        return county_name

df_merged['Violation County'] = df_merged['Violation County'].apply(fix_counties)


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
How I Tried to Do It Before I Made 'Fee for This Precinct'
I want:
    [
    {'DEFINITION': 'Street Cleaning...', 'freq': 140329, 'All Other Areas Int': 45},
    {'DEFINITION': 'Failing to show a receipt', 'freq': 125716, 'All Other Areas Int': 35},
    ...
    ]
Then I'll turn that to a DataFrame.
<p> _Then I'll multiply 'freq' and 'All Other Areas Int' to make a 'How Lucrative' column_
Can you write Python code for it?

# def_list = df_merged['DEFINITION'].unique()

# def first_fee_result(partial_df):
#     number = 0
#     for row in partial_df:
#         number += 1
#         if number == 1:
#             return row

# lucrative_list = []
# lucrative_dict = {}
# for definition in def_list:
#     all_rows_in_df_with_def = df_merged[df_merged['DEFINITION'] == definition]
#     how_many_rows = all_rows_in_df_with_def['DEFINITION'].count()
#     fee = first_fee_result(all_rows_in_df_with_def['All Other Areas Int'])
#     if how_many_rows > 0:
#         lucrative_dict['DEFINITION'] = definition
#         lucrative_dict['freq'] = how_many_rows
#         lucrative_dict['All Other Areas Int'] = fee
#         lucrative_dict['How Lucrative'] = how_many_rows * fee
#         lucrative_list.append(lucrative_dict)
#         lucrative_dict = {}

# df_lucrative = pd.DataFrame(lucrative_list)

# df_lucrative.sort_values('How Lucrative', ascending=False).head()


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
Let's try this:
<p> We can turn the issue dates into a list
<p> We can also turn the value_counts into a list
<p> See if they are both the same length, and then match them up into a dictionary:
<p>
    `tix_per_diem = [
    { 'Issue Date': 07/30/2013 , 'tickets this day': 36500},
    { 'Issue Date': 08/06/2013 , 'tickets this day': 36080}, ...
    ]`
Can you write Python code for it?

issue_dates_list = [unique_date for unique_date in set(df_merged['Issue Date'])]
len(issue_dates_list)

import datetime

def to_datetime_version(str_date):
    datetime_version = datetime.datetime.strptime(str_date, "%m/%d/%Y")
    return datetime_version

def to_string_version(datetime_date):
    string_version = datetime.datetime.strftime(datetime_date, "%m/%d/%Y")
    return string_version

def list_to_sorted_datetime(str_list):
    datetime_version = [to_datetime_version(item) for item in str_list]
    sorted_datetime_version = sorted(datetime_version)
    sorted_string_version = [to_string_version(item) for item in sorted_datetime_version]
    return sorted_string_version

sorted_issue_dates_list = list_to_sorted_datetime(issue_dates_list)

value_counts_list = [value_count for value_count in df_merged['Issue Date'].value_counts()]
len(value_counts_list)

tix_per_diem = []
tix_dict = {}

for unique_date in sorted_issue_dates_list:
    tix_dict['Issue Date'] = unique_date
    tix_dict['tickets this day'] = len(df_merged[df_merged['Issue Date'] == unique_date])
    tix_per_diem.append(tix_dict)
    tix_dict = {}

df_tix_per_diem = pd.DataFrame(tix_per_diem)

df_tix_per_diem.plot(kind = 'line', x = 'Issue Date', y = 'tickets this day', legend = None)


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
Manhattan in Precincts Entirely Below 96th St Money
Can you write Python code for it?

# total_fees = 0
# for fee in df_merged[df_merged['Below 96th St?'] == True]['Fee for This Precinct']:
#     try:
#         total_fees += int(fee)
#     except:
#         pass
# total_fees

# Midtown North is 18
# Midtwon South is 22
# part of 24 is also below 96, but I'm not counting it
# precincts_below_96 = ["22", "20", "19", "18", "17", "13", "10", "9", "7", "6", "5", "1"]

# # go through rows in precincts. If row is in precinct list, save row because we want to get the sum of what was charged for those rows.
# row_numbers = range(len(df_merged['Violation Precinct']))
# row_numbers_for_violations_occuring_below_96 = []

# for row_number in row_numbers:
#     precinct = df_merged['Violation Precinct'][row_number]
#     if precinct in precincts_below_96:
#         row_numbers_for_violations_occuring_below_96.append(row_number)

# # fines_occuring_below_96 = [df_merged['Manhattan\xa096th st and below Int'][row_number] for row_number in row_numbers_for_rows_occuring_below_96]

# def fines_occuring_below_96(row_number):
#     try:
#         return int(df_merged['Manhattan\xa096th st and below Int'][row_number])
#     except:
#         return None

# below_96_fee_sum = 0
# for row_number in row_numbers_for_violations_occuring_below_96:
#     fine = fines_occuring_below_96(row_number)
#     if pd.isnull(fine):
#         pass
#     else:
#         below_96_fee_sum += fine
# below_96_fee_sum


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
Most Lucrative
Can you write Python code for it?

df_merged[['DEFINITION', 'Fee for This Precinct']].groupby('DEFINITION').sum().sort_values('Fee for This Precinct', ascending = False)


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
Okay, Back to the Question
But since there's a different fee for violations in Manhattan below 96th St, first we need to know:
Is It Below 96th St?
Can you write Python code for it?

precincts = df_merged['Violation Precinct'].unique()


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
One precinct straddles 96th St, but for simplicity, I'm only going to county precincts entirely below 96th.
Can you write Python code for it?

precincts_below_96 = ["22", "20", "19", "18", "17", "13", "10", "9", "7", "6", "5", "1"]
# row_numbers = range(len(df_merged['Violation Precinct']))

# Make below_96_list = [
#       {'Violation Precinct': '22', 'Below 96th St?': True},
#       {'Violation Precinct': '20', 'Below 96th St?': True},
#       ...
#      ]

below_96_list = []
below_96_dict = {}

for precinct in precincts:
    below_96_dict['Violation Precinct'] = precinct
    if precinct in precincts_below_96:
        below_96_dict['Below 96th St?'] = True
    else:
        below_96_dict['Below 96th St?'] = False
    below_96_list.append(below_96_dict)
    below_96_dict = {}

# Make below_96_list a DataFrame and then merge it with df_merged

df_below_96 = pd.DataFrame(below_96_list)

df_merged = df_below_96.merge(df_merged, how = 'left', on = 'Violation Precinct')


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
So How Much Is Each Violation Fee?
Can you write Python code for it?

# make it so
# if Below 96th St? == True, Fee for This Precinct = Manhattan 96th st and below Int
# else, Fee for This Precinct = 'All Other Areas Int'

row_numbers = range(len(df_merged['Below 96th St?']))
fees_list = []
fees_dict = {}

for row in row_numbers:
    fees_dict['Summons Number'] = df_merged['Summons Number'][row]
    if df_merged['Below 96th St?'][row] == True:
        fees_dict['Fee for This Precinct'] = df_merged['Manhattan\xa096th st and below Int'][row]
    else:
        fees_dict['Fee for This Precinct'] = df_merged['All Other Areas Int'][row]
    fees_list.append(fees_dict)
    fees_dict = {}

df_fees = pd.DataFrame(fees_list)


I am working on the file "11/homework-11-schuetz.ipynb".
The first task is:
That's because some fees in the columns 'Manhattan 96th st and below' and 'All Other Areas' are nan
Can you write Python code for it?

df_merged['Manhattan\xa096th st and below'].unique()