I am working on the file "06 - Reshaping data.ipynb".
The first task is:
pivot is similar to unstack, but let you specify column names:
Can you write Python code for it?


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
pivot_table is similar as pivot, but can work with duplicate indices and let you specify an aggregation function:
Can you write Python code for it?

df = pd.DataFrame({'A':['one', 'one', 'two', 'two', 'one', 'two'], 'B':['a', 'b', 'a', 'b', 'a', 'b'], 'C':range(6)})
df

df.pivot_table(index='A', columns='B', values='C', aggfunc='count') #'mean'


I am working on the file "06 - Reshaping data.ipynb".
The first task is:
unstack reverses the operation:
Can you write Python code for it?


I am working on the file "06 - Stable Roommates, Marriages, and Gender.ipynb".
The first task is:
Despite looking quite long and convoluted function, this thing is wicked fast. It's basically just a couple of ifs stretched out into a very tall function
Can you write Python code for it?


I am working on the file "06 - Stable Roommates, Marriages, and Gender.ipynb".
The first task is:
However, we need to include all nodes and ignore colour, so we'll use the following code
Can you write Python code for it?

G = nx.from_pandas_dataframe(book,
                             source='source',
                             target='target')

genders = {character: random.choice(['M', 'F']) for character in G.nodes()}


I am working on the file "06 - Stable Roommates, Marriages, and Gender.ipynb".
The first task is:
Stable Marriages
There's a lovely piece of classic game theory known as the the 'Stable Marriage Problem' which is famously solved by the Gale-Shapley Algorithm.
The story given to provide intuition often changes from telling to telling, but here I'll use a story of a town's villagers. For the sake of mathematical simplicity we're going to have to make some reductive assumptions about the townspeople's genders and sexulaities - there are two genders in this town, the population is evenly split between the two genders, and everyone is heterosexual. The townspeople need to be married off to one another for the sake of good and wholesome society, and each person has an ordered list of spousal preferences. Men propose to women, and women can accept or reject any of those proposals at any time until overall stability is achieved. For example:
| Proposor |   |   |   |   |
|----------|---|---|---|---|
| Man 1    | 2 | 1 | 3 | 4 |
| Man 2    | 4 | 1 | 2 | 3 |
| Man 3    | 1 | 3 | 2 | 4 |
| Man 4    | 2 | 3 | 1 | 4 |
| Acceptor |   |   |   |   |
|----------|---|---|---|---|
| Woman 1  | 1 | 3 | 2 | 4 |
| Woman 2  | 3 | 4 | 1 | 2 |
| Woman 3  | 4 | 2 | 3 | 1 |
| Woman 4  | 3 | 2 | 1 | 4 |
Man 1 would ideally like to marry Woman 2, followed by Woman 1, Woman 3 and finally Woman 4, etc.
The problem, now, is to find a set of pairings between townspeople which leaves the population maximally satisfied. Note that we're maximising satisfaction for the whole population here, not individuals - Some people will inevitably miss out on a marriage with their ideal partner because they've already chosen to marry somebody else, or because their preferences are misaligned. The solution we want to approach here is one of equilibrium or stability - ie no person will deviate from their marriage becuase, due to the stability of the other couples around them, their interests are maximised by remaining in the marriage. There are no two people of opposite gender who would both rather have each other than their current partners.
The...
Can you write Python code for it?

def gale_shapley(proposors, acceptors):
    '''
    Returns stable marriage pairings for a given set of input
    candidates and their lists of spousal preferences

    Parameters
    ----------
    proposors : dict (required)
        the set of men and their list of spousal preferences
    acceptors : dict (required)
        the set of women and their list of spousal preferences

    Returns
    -------
    marriages : dict
        the list of stable marriage pairings
    '''
    is_man_available = {man: True for man in proposors}
    is_woman_available = {woman: True for woman in acceptors}

    proposals_tried = {man: [] for man in proposors}
    engagements = {person: None for person in list(proposors) + list(acceptors)}

    while any(is_man_available[man] is True for man in proposors):
        available_men = [man for man in list(proposors) if is_man_available[man]]

        for man in available_men:
            untried_proposals = [w for w in proposors[man] if w not in proposals_tried[man]]

            preferred_woman = untried_proposals[0]

            if is_woman_available[preferred_woman]:
                engagements[man] = preferred_woman

                is_man_available[man] = False
                is_woman_available[preferred_woman] = False

                engagements[man] = preferred_woman
                engagements[preferred_woman] = man

            elif engagements[preferred_woman] is not None:
                current_fiance = engagements[preferred_woman]

                current_fiance_index = acceptors[preferred_woman].index(current_fiance)
                new_proposor_index = acceptors[preferred_woman].index(man)

                if new_proposor_index < current_fiance_index:
                    engagements[current_fiance] = None
                    is_man_available[current_fiance] = True

                    engagements[preferred_woman] = man

                   ...


I am working on the file "06 - Stable Roommates, Marriages, and Gender.ipynb".
The first task is:
The Stable Marriages problem requires a bipartite graph, which is something we don't have naturally here - our network is genderless at the beginning and it has to be awkwardly imposed.
We can adapt our problem to ignore gender. The genderless version of the problem is the lesser known Stable Roommates problem, solved by Irving in 1985.
Stable Roommates
Can you write Python code for it?

def irving(population):
    roommates = {p: None for p in population}
    return roommates


I am working on the file "06 - Stable Roommates, Marriages, and Gender.ipynb".
The first task is:
We get an error, as there aren't enough acceptors to propose to and the algorithm relies on all the proposors being matched at the end of the process. It doesn't really matter if an acceptor is left behind, but the proposors must all be married.
Stable Marriages in Novels
I thought it might be fun to invoke Gale-Shapley on our bookworm networks. We have a list of characters with some kind of preference metric enumerated already - the only thing we're missing is the gender of each character. A problem in modern English literature is that character networks are often vastly asymmetric regarding gender.
We can ascribe gender randomly and use the algorithm outlined above:
visualised, the random splitting looks something like this
Can you write Python code for it?

book = bookworm('data/raw/hp_philosophers_stone.txt', threshold=-1)

G = nx.from_pandas_dataframe(book[book['value'] > 2],
                             source='source',
                             target='target')

genders = {character: random.choice(['#69B4FF', '#FF69B4'])
           for character in G.nodes()}

nx.draw(G,
        cmap=plt.get_cmap("RdYlBu"),
        node_color=list(genders.values()),
        with_labels=True)


I am working on the file "06 - Stable Roommates, Marriages, and Gender.ipynb".
The first task is:
and switch the genders of the proposors / acceptors
Can you write Python code for it?


I am working on the file "06 - Stable Roommates, Marriages, and Gender.ipynb".
The first task is:
we can add a new woman to the mix and see what happens
Can you write Python code for it?


I am working on the file "06 - Stable Roommates, Marriages, and Gender.ipynb".
The first task is:
we have more men than women this time around, so we'll have to have the women proposing to men
Can you write Python code for it?

marriages = gale_shapley(female_preferences, male_preferences)


I am working on the file "06 - Working With Text Data.ipynb".
The first task is:
Look at SpaCy and NLTK
Exercises

Create a pipeine using the count vectorizer and SVM (see 07). Train and score using the pipeline.
vizualize the coefficients of the svm inside the pipeline using pipeline.named_steps['linearsvc'] and pipeline.named_steps['countvectorizer']

Grid search the C in the LinearSVC using the pipeline.

Vary the n_gram_range in the count vectorizer, visualize the changed coefficients.

Grid search the C in the LinearSVC together with the ngram_range (try (1,1), (1, 2), (2, 2))
Visualize the parameter again for the grid-searched model
Can you write Python code for it?


I am working on the file "06 - Working With Text Data.ipynb".
The first task is:
Subsample for interactivity
Can you write Python code for it?

text_train, y_train = text_train[::10], y_train[::10]
text_test, y_test = text_test[::10], y_test[::10]

from IPython.display import HTML

print(text_train[0])

HTML(text_train[0].decode("utf-8"))

print(y_train[0])

from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer()
cv.fit(text_train)

len(cv.vocabulary_)

print(cv.get_feature_names()[:50])
print(cv.get_feature_names()[5000:5020])

X_train = cv.transform(text_train)
X_train

print(text_train[739])

X_train[739].nonzero()[1]

X_test = cv.transform(text_test)

from sklearn.svm import LinearSVC

svm = LinearSVC()
svm.fit(X_train, y_train)

svm.score(X_train, y_train)

svm.score(X_test, y_test)

def visualize_coefficients(classifier, feature_names, n_top_features=25):
    # get coefficients with large absolute values
    coef = classifier.coef_.ravel()
    positive_coefficients = np.argsort(coef)[-n_top_features:]
    negative_coefficients = np.argsort(coef)[:n_top_features]
    interesting_coefficients = np.hstack([negative_coefficients, positive_coefficients])
    # plot them
    plt.figure(figsize=(15, 5))
    colors = ["red" if c < 0 else "blue" for c in coef[interesting_coefficients]]
    plt.bar(np.arange(2 * n_top_features), coef[interesting_coefficients], color=colors)
    feature_names = np.array(feature_names)
    plt.subplots_adjust(bottom=0.3)
    plt.xticks(np.arange(1, 1 + 2 * n_top_features), feature_names[interesting_coefficients], rotation=60, ha="right");

visualize_coefficients(svm, cv.get_feature_names())

cv = CountVectorizer(ngram_range=(1, 2))
cv.fit(text_train)
len(cv.vocabulary_)

X_test = cv.transform(text_test)
X_train = cv.transform(text_train)

svm = LinearSVC()
svm.fit(X_train, y_train)

visualize_coefficients(svm, cv.get_feature_names())


I am working on the file "06 - Working With Text Data.ipynb".
The first task is:
Text Classification of Movie Reviews
Unpack data - this only works on linux and (maybe?) OS X. Unpack using 7zip on Windows.
Can you write Python code for it?

#! tar -xf data/aclImdb.tar.bz2 --directory data

from sklearn.datasets import load_files

reviews_train = load_files("data/aclImdb/train/")
text_train, y_train = reviews_train.data, reviews_train.target

print("Number of documents in training data: %d" % len(text_train))
print(np.bincount(y_train))

reviews_test = load_files("data/aclImdb/test/")
text_test, y_test = reviews_test.data, reviews_test.target
print("Number of documents in test data: %d" % len(text_test))
print(np.bincount(y_test))

from IPython.display import HTML

print(text_train[1])

HTML(text_train[1].decode("utf-8"))

print(y_train[1])

from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer()
cv.fit(text_train)

len(cv.vocabulary_)

print(cv.get_feature_names()[:50])
print(cv.get_feature_names()[50000:50050])

X_train = cv.transform(text_train)
X_train

print(text_train[19726])

X_train[19726].nonzero()[1]

X_test = cv.transform(text_test)

from sklearn.svm import LinearSVC

svm = LinearSVC()
svm.fit(X_train, y_train)

svm.score(X_train, y_train)

svm.score(X_test, y_test)

def visualize_coefficients(classifier, feature_names, n_top_features=25):
    # get coefficients with large absolute values
    coef = classifier.coef_.ravel()
    positive_coefficients = np.argsort(coef)[-n_top_features:]
    negative_coefficients = np.argsort(coef)[:n_top_features]
    interesting_coefficients = np.hstack([negative_coefficients, positive_coefficients])
    # plot them
    plt.figure(figsize=(15, 5))
    colors = ["red" if c < 0 else "blue" for c in coef[interesting_coefficients]]
    plt.bar(np.arange(2 * n_top_features), coef[interesting_coefficients], color=colors)
    feature_names = np.array(feature_names)
    plt.subplots_adjust(bottom=0.3)
    plt.xticks(np.arange(1, 1 + 2 * n_top_features), feature_names[interesting_coefficients], rotation=60, ha="right");

visualize_coefficients(svm, cv.get_feature_names())

svm = LinearSVC(C=0.001)
svm.fit(X_train, y_train)

visualize_coefficients(svm, cv.get_feature_names())

from...


#! tar -xf data/aclImdb.tar.bz2 --directory data

from sklearn.datasets import load_files

reviews_train = load_files("data/aclImdb/train/")
text_train, y_train = reviews_train.data, reviews_train.target

print("Number of documents in training data: %d" % len(text_train))
print(np.bincount(y_train))

reviews_test = load_files("data/aclImdb/test/")
text_test, y_test = reviews_test.data, reviews_test.target
print("Number of documents in test data: %d" % len(text_test))
print(np.bincount(y_test))


#! tar -xf data/aclImdb.tar.bz2 --directory data

from sklearn.datasets import load_files

reviews_train = load_files("data/aclImdb/train/")
text_train, y_train = reviews_train.data, reviews_train.target

print("Number of documents in training data: %d" % len(text_train))
print(np.bincount(y_train))

reviews_test = load_files("data/aclImdb/test/")
text_test, y_test = reviews_test.data, reviews_test.target
print("Number of documents in test data: %d" % len(text_test))
print(np.bincount(y_test))

print(text_train[1])

print(y_train[1])

from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer()
cv.fit(text_train)

len(cv.vocabulary_)

print(cv.get_feature_names()[:50])
print(cv.get_feature_names()[50000:50050])

X_train = cv.transform(text_train)
X_train

print(text_train[19726])

X_train[19726].nonzero()[1]

X_test = cv.transform(text_test)

from sklearn.svm import LinearSVC

svm = LinearSVC()
svm.fit(X_train, y_train)

svm.score(X_train, y_train)

svm.score(X_test, y_test)

def visualize_coefficients(classifier, feature_names, n_top_features=25):
    # get coefficients with large absolute values
    coef = classifier.coef_.ravel()
    positive_coefficients = np.argsort(coef)[-n_top_features:]
    negative_coefficients = np.argsort(coef)[:n_top_features]
    interesting_coefficients = np.hstack([negative_coefficients, positive_coefficients])
    # plot them
    plt.figure(figsize=(15, 5))
    colors = ["red" if c < 0 else "blue" for c in coef[interesting_coefficients]]
    plt.bar(np.arange(2 * n_top_features), coef[interesting_coefficients], color=colors)
    feature_names = np.array(feature_names)
    plt.subplots_adjust(bottom=0.3)
    plt.xticks(np.arange(1, 1 + 2 * n_top_features), feature_names[interesting_coefficients], rotation=60, ha="right");

visualize_coefficients(svm, cv.get_feature_names())

svm = LinearSVC(C=0.001)
svm.fit(X_train, y_train)
svm.score(X_test, y_test)

visualize_coefficients(svm, cv.get_feature_names())

from sklearn.pipeline import...


I am working on the file "06 Deep Residual Network.ipynb".
The first task is:
By increasing the depth of the network and the number of epochs, we can improve the performance to match state of the art.
This was quite a lot of code! Generally, to set up a new model from scratch it is best to follow one of the examples from the neon/examples directory. It's easy to mix and match parts!
Inference
Now we want to grab a few new images from the internet and classify them through our network.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import urllib
from PIL import Image
import numpy as np

# download images from the web
imgs = {
       'frog': "https://upload.wikimedia.org/wikipedia/commons/thumb/5/55/Atelopus_zeteki1.jpg/440px-Atelopus_zeteki1.jpg",
       'airplane': "https://img0.etsystatic.com/016/0/5185796/il_570xN.433414910_p5n3.jpg",
       'cat': "https://s-media-cache-ak0.pinimg.com/236x/8e/d7/41/8ed7410285f101ba5892ff723c91fa75.jpg",
       'car': "http://static01.nyt.com/images/2012/09/09/automobiles/09REFI2/09REFI2-articleLarge.jpg",
       }

# empty buffer to use for inference dataset
# dims [minibatch, imgsize]
x_new = np.zeros((128, 32*32*3), dtype=np.float32)

# crop/resize images and assign them to slots in x_new
# also display with true labels
plt.figure(1)
for i, name in enumerate(imgs):
    imgs[name] = urllib.urlretrieve(imgs[name], filename="data/{}.jpg".format(name))
    plt.subplot(100 + (10 * len(imgs)) + 1 + i)

    img = Image.open("data/{}.jpg".format(name))
    crop = img.crop((0,0,min(img.size),min(img.size)))
    crop.thumbnail((32, 32))

    plt.imshow(crop, interpolation="nearest")
    plt.title(name)
    plt.axis('off')

    x_new[i,:] = np.asarray(crop, dtype=np.float32)[:,:,(2,0,1)].transpose(2,0,1).reshape(1,3072) -127


I am working on the file "06 Deep Residual Network.ipynb".
The first task is:
Callbacks
To provide feedback while the model is training, neon lets the user specify a set of callbacks that get evaluated at the end of every iteration (minibatch) or pass through the dataset (epoch). Callbacks include evaluating the model on a validation set or computing missclassification percentage. There are also callbacks for saving to disk and for generating visualizations. Here we will set up a progress bar to monitor training.
Can you write Python code for it?

# set up callbacks. By default sets up a progress bar
from neon.transforms import Misclassification
from neon.callbacks.callbacks import Callbacks

valmetric = Misclassification()
callbacks = Callbacks(model, eval_set=test_set, metric=valmetric)


I am working on the file "06 Deep Residual Network.ipynb".
The first task is:
Congrats! If you made it this far you have trained a convolutional network in neon.
Evaluating the model
We can now compute the misclassification on the test set to see how well we did.
Can you write Python code for it?

# Check the performance on the supplied test set
from neon.transforms import Misclassification

error_pct = 100 * model.eval(test_set, metric=Misclassification())
print 'Misclassification error = %.1f%%' % error_pct


I am working on the file "06 Deep Residual Network.ipynb".
The first task is:
Cost function
The cost function compares network outputs with ground truth labels, and produces and error that we can backpropagate through the layers of the network.
For our binary classification task, we use a cross entropy cost function.
Can you write Python code for it?

from neon.transforms import CrossEntropyMulti
from neon.layers import GeneralizedCost

cost = GeneralizedCost(costfunc=CrossEntropyMulti())


I am working on the file "06 Deep Residual Network.ipynb".
The first task is:
Create a dataset with this image for inference
Can you write Python code for it?

from neon.data import ArrayIterator

# create a minibatch with the new image
inference_set = ArrayIterator(x_new, None, nclass=10,
                             lshape=(3, 32, 32))
# inference_set = ArrayIterator(x_train, None, nclass=10,
#                              lshape=(3, 32, 32))
classes =["airplane", "auto", "bird", "cat", "deer",
          "dog", "frog", "horse", "ship", "truck"]
out = model.get_outputs(inference_set)


I am working on the file "06 Deep Residual Network.ipynb".
The first task is:
Generating layers
The core of the model is the layers. This can be as simple as a list, but merging and branching makes it easy to specify complex topologies.
Can you write Python code for it?

from neon.initializers import Uniform
from neon.transforms import Rectlin, Softmax
from neon.layers import Activation, Conv, Pooling, Affine, MergeSum

# This is a simple convnet with a one conv layer,
# max-pooling, and a fully connected layer.
#
# input - Conv - ReLu - Pooling - Affine - ReLu - Affine - Softmax
#
layers = [Conv((5, 5, 16), init=Uniform(-0.1, 0.1), activation=Rectlin()),
          Pooling((2, 2)),
          Affine(nout=500, init=Uniform(-0.1, 0.1), activation=Rectlin()),
          Affine(nout=10, init=Uniform(-0.1, 0.1), activation=Softmax())]

# We can use a MergeSum layer to combine differnt layers in parallel
#
#             - Conv3 - ReLu -
#           /                  \
# input   -                     Sum - ReLu - ...
#           \                  /
#             - Conv5 - ReLu -
#
conv3 = Conv((3, 3, 16), init=Uniform(-0.1, 0.1), activation=Rectlin())
conv5 = Conv((5, 5, 16), padding=1, init=Uniform(-0.1, 0.1), activation=Rectlin())

layers = [MergeSum([conv3, conv5]), Activation(Rectlin()),
          Pooling((2, 2)),
          Affine(nout=500, init=Uniform(-0.1, 0.1), activation=Rectlin()),
          Affine(nout=10, init=Uniform(-0.1, 0.1), activation=Softmax())]


I am working on the file "06 Deep Residual Network.ipynb".
The first task is:
Get model outputs on the inference data
Can you write Python code for it?

plt.figure(2)
for i, name in enumerate(imgs):
    plt.subplot(100 + (10 * len(imgs)) + 1 + i)

    img = Image.open("data/{}.jpg".format(name))
    crop = img.crop((0,0,min(img.size),min(img.size)))
    crop.thumbnail((32, 32))

    title = "{} ({:.2})".format(classes[out[i].argmax()], out[i].max())

    plt.imshow(crop, interpolation="nearest")
    plt.title(title)
    plt.axis('off')


I am working on the file "06 Deep Residual Network.ipynb".
The first task is:
The Deep Residual Network
A resnet module is a MergeSum layer containing a main path with conv layers, and a side path with a SkipNode() configured as the identity function. This allows earlier layer activations to bypass a series of layers.
We use some helper functions to succinclty define the deep network:
Can you write Python code for it?

from neon.initializers import Kaiming, IdentityInit
from neon.layers import SkipNode
from neon.models import Model

# helper functions simplify init params for conv and identity layers
def conv_params(fsize, nfm, stride=1, relu=True, batch_norm=True):
    return dict(fshape=(fsize, fsize, nfm),
                strides=stride,
                padding=(1 if fsize > 1 else 0),
                activation=(Rectlin() if relu else None),
                init=Kaiming(local=True),
                batch_norm=batch_norm)

def id_params(nfm):
    return dict(fshape=(1, 1, nfm),
                strides=2,
                padding=0,
                activation=None,
                init=IdentityInit())

# A resnet module
#
#             - Conv - Conv -
#           /                \
# input   -                   Sum - Relu - output
#           \               /
#            -  Identity -
#
def module_factory(nfm, stride=1):
    mainpath = [Conv(**conv_params(3, nfm, stride=stride)),
                Conv(**conv_params(3, nfm, relu=False))]
    sidepath = [SkipNode() if stride == 1 else Conv(**id_params(nfm))]

    module = [MergeSum([mainpath, sidepath]),
              Activation(Rectlin())]
    return module


I am working on the file "06 Deep Residual Network.ipynb".
The first task is:
Training the model
Now all the pieces are in place to run the network. We use the fit function and pass it a dataset, cost, optmizer, and the callbacks we set up.
Can you write Python code for it?

# And  run the model
epochs = 10
model.fit(train_set, optimizer=opt, num_epochs=epochs,
          cost=cost, callbacks=callbacks)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
Exercises
Pet Names

Create a dictionary to hold information about pets. Each key is an animal's name, and each value is the kind of animal.
For example, 'ziggy': 'canary'

Put at least 3 key-value pairs in your dictionary.
Use a for loop to print out a series of statements such as "Willie is a dog."

Polling Friends

Think of a question you could ask your friends. Create a dictionary where each key is a person's name, and each value is that person's response to your question.
Store at least three responses in your dictionary.
Use a for loop to print out a series of statements listing each person's name, and their response.
Can you write Python code for it?

# Ex 7.1 : Pet Names

# put your code here

# Ex 7.2 : Polling Friends

# put your code here


I am working on the file "06 Dictionaries.ipynb".
The first task is:
As long as we have a nice clean function to work with, let's clean up our output a little:
Can you write Python code for it?

def show_words_meanings(python_words):
    # This function takes in a dictionary of python words and meanings,
    #  and prints out each word with its meaning.
    print("\n\nThese are the Python words I know:")
    for word, meaning in python_words.items():
        print("\n%s: %s" % (word, meaning))

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

show_words_meanings(python_words)

# Remove the word 'list' and its meaning.
del python_words['list']

show_words_meanings(python_words)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
If you were going to work with this code, you would certainly want to put the code for displaying the dictionary into a function. Let's see what this looks like:
Can you write Python code for it?

def show_words_meanings(python_words):
    # This function takes in a dictionary of python words and meanings,
    #  and prints out each word with its meaning.
    print("\n\nThese are the Python words I know:")
    for word, meaning in python_words.items():
        print("\nWord: %s" % word)
        print("Meaning: %s" % meaning)

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

show_words_meanings(python_words)

# Remove the word 'list' and its meaning.
del python_words['list']

show_words_meanings(python_words)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
In this example, the keys have been put into alphabetical order in the for loop only; Python has not changed the way the dictionary is stored at all. So the next time the dictionary is accessed, the keys could be returned in any order. There is no way to permanently specify an order for the items in an ordinary dictionary, but if you want to do this you can use the OrderedDict structure.
top
Exercises
Mountain Heights

Wikipedia has a list of the tallest mountains in the world, with each mountain's elevation. Pick five mountains from this list.
Create a dictionary with the mountain names as keys, and the elevations as values.
Print out just the mountains' names, by looping through the keys of your dictionary.
Print out just the mountains' elevations, by looping through the values of your dictionary.
Print out a series of statements telling how tall each mountain is: "Everest is 8848 meters tall."

Revise your output, if necessary.
Make sure there is an introductory sentence describing the output for each loop you write.
Make sure there is a blank line between each group of statements.

Mountain Heights 2

Revise your final output from Mountain Heights, so that the information is listed in alphabetical order by each mountain's name.
That is, print out a series of statements telling how tall each mountain is: "Everest is 8848 meters tall."
Make sure your output is in alphabetical order.
Can you write Python code for it?

# Ex 7.5 : Mountain Heights

# put your code here

# Ex 7.6 : Mountain Heights 2

# put your code here


I am working on the file "06 Dictionaries.ipynb".
The first task is:
Let's show how we might use this in our Python words program. This kind of loop provides a straightforward way to show only the words in the dictionary:
Can you write Python code for it?

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

# Show the words that are currently in the dictionary.
print("The following Python words have been defined:")
for word in python_words:
    print("- %s" % word)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
Since the keys and values in dictionaries can be long, we often write just one key-value pair on a line. You might see dictionaries that look more like this:
Can you write Python code for it?

dictionary_name = {key_1: value_1,
                   key_2: value_2,
                   key_3: value_3,
                   }


I am working on the file "06 Dictionaries.ipynb".
The first task is:
The output is identical, but we did it in 3 lines instead of 6. If we had 100 terms in our dictionary, we would still be able to print them out with just 3 lines.
The only tricky part about using for loops with dictionaries is figuring out what to call those first two variables. The general syntax for this for loop is:
Can you write Python code for it?

for key_name, value_name in dictionary_name.items():
    print(key_name) # The key is stored in whatever you called the first variable.
    print(value_name) # The value associated with that key is stored in your second variable.


I am working on the file "06 Dictionaries.ipynb".
The first task is:
The resulting list is not in order. The list of keys can be put in order by passing the list into the sorted() function, in the line that initiates the for loop:
Can you write Python code for it?

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

for word in sorted(python_words.keys()):
    print(word)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
This allows the user to ask for as many meanings as they want, but it takes the word "quit" as a requested word. Let's add an elif clause to clean up this behavior:
Can you write Python code for it?

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

# Show the words that are currently in the dictionary.
print("The following Python words have been defined:")
for word in python_words:
    print("- %s" % word)

requested_word = ''
while requested_word != 'quit':
    # Allow the user to choose a word, and then display the meaning for that word.
    requested_word = raw_input("\nWhat word would you like to learn about? (or 'quit') ")
    if requested_word in python_words.keys():
        # This is a word we know, so show the meaning.
        print("\n  %s: %s" % (requested_word, python_words[requested_word]))
    elif requested_word != 'quit':
        # This is not in python_words, and it's not 'quit'.
        print("\n  Sorry, I don't know that word.")
    else:
        # The word is quit.
        print "\n  Bye!"


I am working on the file "06 Dictionaries.ipynb".
The first task is:
This allows the user to select one word that has been defined. If we enclose the input part of the program in a while loop, the user can see as many definitions as they'd like:
Can you write Python code for it?

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

# Show the words that are currently in the dictionary.
print("The following Python words have been defined:")
for word in python_words:
    print("- %s" % word)

requested_word = ''
while requested_word != 'quit':
    # Allow the user to choose a word, and then display the meaning for that word.
    requested_word = raw_input("\nWhat word would you like to learn about? (or 'quit') ")
    if requested_word in python_words.keys():
        print("\n  %s: %s" % (requested_word, python_words[requested_word]))
    else:
        # Handle misspellings, and words not yet stored.
        print("\n  Sorry, I don't know that word.")


I am working on the file "06 Dictionaries.ipynb".
The first task is:
This code is a lot longer, and now we have nested if statements as well as nested for loops. But keep in mind, this structure would work if there were 1000 pets in our dictionary, and it would work if we were storing 1000 pieces of information about each pet. One level of nesting lets us model an incredible array of information.
An important note about nesting
While one level of nesting is really useful, nesting much deeper than that gets really complicated, really quickly. There are other structures such as classes which can be even more useful for modeling information. In addition to this, we can use Python to store information in a database, which is the proper tool for storing deeply nested information.
Often times when you are storing information in a database you will pull a small set of that information out and put it into a dictionary, or a slightly nested structure, and then work with it. But you will rarely, if ever, work with Python data structures nested more than one level deep.
Exercises
Mountain Heights 3

This is an extension of Mountain Heights. Make sure you save this program under a different filename, such as mountain_heights_3.py, so that you can go back to your original program if you need to.
The list of tallest mountains in the world provided all elevations in meters. Convert each of these elevations to feet, given that a meter is approximately 3.28 feet. You can do these calculations by hand at this point.
Create a new dictionary, where the keys of the dictionary are still the mountains' names. This time however, the values of the dictionary should be a list of each mountain's elevation in meters, and then in feet: {'everest': [8848, 29029]}
Print out just the mountains' names, by looping through the keys of your dictionary.
Print out just the mountains' elevations in meters, by looping through the values of your dictionary and pulling out the first number from each list.
Print out just the mountains' elevations in feet, by...
Can you write Python code for it?

# Ex 7.7 : Mountain Heights 3

# put your code here

# Ex 7.8 : Mountain Heights 4

# put your code here


I am working on the file "06 Dictionaries.ipynb".
The first task is:
This code looks pretty repetitive, and it is. Dictionaries have their own for-loop syntax, but since there are two kinds of information in dictionaries, the structure is a bit more complicated than it is for lists. Here is how to use a for loop with a dictionary:
Can you write Python code for it?

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

# Print out the items in the dictionary.
for word, meaning in python_words.items():
    print("\nWord: %s" % word)
    print("Meaning: %s" % meaning)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
This is a bit easier to read, especially if the values are long.
Example
A simple example involves modeling an actual dictionary.
Can you write Python code for it?

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }


I am working on the file "06 Dictionaries.ipynb".
The first task is:
This is actually the default behavior of looping through the dictionary itself. So you can leave out the .keys() part, and get the exact same behavior:
Can you write Python code for it?

my_dict = {'key_1': 'value_1',
    'key_2': 'value_2',
    'key_3': 'value_3',
    }

for key in my_dict:
    print('Key: %s' % key)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
This is much more realistic code.
Modifying keys in a dictionary
Modifying a value in a dictionary was straightforward, because nothing else depends on the value. Modifying a key is a little harder, because each key is used to unlock a value. We can change a key in two steps:

Make a new key, and copy the value to the new key.
Delete the old key, which also deletes the old value.

Here's what this looks like. We will use a dictionary with just one key-value pair, to keep things simple.
Can you write Python code for it?

# We have a spelling mistake!
python_words = {'list': 'A collection of values that are not connected, but have an order.'}

# Create a new, correct key, and connect it to the old value.
#  Then delete the old key.
python_words['list'] = python_words['lisst']
del python_words['lisst']

# Print the dictionary, to show that the key has changed.
print(python_words)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
This is starting to work, but we can see from the output that the user does not get the chance to take a second guess if they guess wrong for any meaning. We can use a while loop around the guessing code, to let the user guess until they get it right:
Can you write Python code for it?

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

# Print each meaning, one at a time, and ask the user
#  what word they think it is.
for meaning in python_words.values():
    print("\nMeaning: %s" % meaning)

    # Assume the guess is not correct; keep guessing until correct.
    correct = False
    while not correct:
        guessed_word = input("\nWhat word do you think this is? ")

        # The guess is correct if the guessed word's meaning matches the current meaning.
        if python_words[guessed_word] == meaning:
            print("You got it!")
            correct = True
        else:
            print("Sorry, that's just not the right word.")


I am working on the file "06 Dictionaries.ipynb".
The first task is:
This structure is fairly complex, so don't worry if it takes a while for things to sink in. The dictionary itself probably makes sense; each person is connected to a list of their favorite numbers.
This works, but we'd rather not print raw Python in our output. Let's use a for loop to print the favorite numbers individually, rather than in a Python list.
Can you write Python code for it?

# This program stores people's favorite numbers, and displays them.
favorite_numbers = {'eric': [3, 11, 19, 23, 42],
                    'ever': [2, 4, 5],
                    'willie': [5, 35, 120],
                    }

# Display each person's favorite numbers.
for name in favorite_numbers:
    print("\n%s's favorite numbers are:" % name.title())
    # Each value is itself a list, so we need another for loop
    #  to work with the list.
    for favorite_number in favorite_numbers[name]:
        print(favorite_number)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
This works because the method .items() pulls all key-value pairs from a dictionary into a list of tuples:
Can you write Python code for it?

my_dict = {'key_1': 'value_1',
    'key_2': 'value_2',
    'key_3': 'value_3',
    }

print(my_dict.items())


I am working on the file "06 Dictionaries.ipynb".
The first task is:
We are really just working our way through each key in the dictionary, so let's use a for loop to go through the keys in the dictionary:
Can you write Python code for it?

# Example for a Sparse Matrix Implementation

sparse_matrix = {}
sparse_matrix[0] = {1: 12.3, 23: 25.5}
sparse_matrix[1] = {3: 12.0, 15: 25.5}

sparse_matrix[1][15]

full_matrix = [[1, 3, 4],
               [2, 5, 3]]

full_matrix[1][2]

# This program stores people's favorite numbers, and displays them.
favorite_numbers = {'eric': [3, 11, 19, 23, 42],
                    'ever': [2, 4, 5],
                    'willie': [5, 35, 120],
                    }

# Display each person's favorite numbers.
for name in favorite_numbers:
    print("\n%s's favorite numbers are:" % name.title())
    print(favorite_numbers[name])


I am working on the file "06 Dictionaries.ipynb".
The first task is:
top
Looping through a dictionary in order
Dictionaries are quite useful because they allow bits of information to be connected. One of the problems with dictionaries, however, is that they are not stored in any particular order. When you retrieve all of the keys or values in your dictionary, you can't be sure what order you will get them back. There is a quick and easy way to do this, however, when you want them in a particular order.
Let's take a look at the order that results from a simple call to dictionary.keys():
Can you write Python code for it?

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

for word in python_words.keys():
    print(word)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
top
Common operations with dictionaries
There are a few common things you will want to do with dictionaries. These include adding new key-value pairs, modifying information in the dictionary, and removing items from dictionaries.
Adding new key-value pairs
To add a new key-value pair, you give the dictionary name followed by the new key in square brackets, and set that equal to the new value. We will show this by starting with an empty dictionary, and re-creating the dictionary from the example above.
Can you write Python code for it?

# Create an empty dictionary.
python_words = {}

# Fill the dictionary, pair by pair.
python_words['list'] ='A collection of values that are not connected, but have an order.'
python_words['dictionary'] = 'A collection of key-value pairs.'
python_words['function'] = 'A named set of instructions that defines a set of actions in Python.'

# Print out the items in the dictionary.
for word, meaning in python_words.items():
    print("\nWord: %s" % word)
    print("Meaning: %s" % meaning)

# Dealing with Hashing Functions
hashings = {}
hashings['function'] = hash('function')
hashings['list'] = hash('list')
hashings['dictionary'] = hash('dictionary')

for k, v in hashings.items():
    print(k,': ',v)

from collections import OrderedDict
hashings = OrderedDict()
hashings['function'] = hash('function')
hashings['list'] = hash('list')
hashings['dictionary'] = hash('dictionary')

for k, v in hashings.items():
    print(k,': ',v)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
top
Looping through all keys in a dictionary
Python provides a clear syntax for looping through just the keys in a dictionary:
Can you write Python code for it?

my_dict = {'key_1': 'value_1',
    'key_2': 'value_2',
    'key_3': 'value_3',
    }

for key in my_dict.keys():
    print('Key: %s' % key)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
top
Looping through all values in a dictionary
Python provides a straightforward syntax for looping through all the values in a dictionary, as well:
Can you write Python code for it?

my_dict = {'key_1': 'value_1',
    'key_2': 'value_2',
    'key_3': 'value_3',
    }

for value in my_dict.values():
    print('Value: %s' % value)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
top
Looping through a dictionary
Since dictionaries are really about connecting bits of information, you will often use them in the ways described above, where you add key-value pairs whenever you receive some new information, and then you retrieve the key-value pairs that you care about. Sometimes, however, you will want to loop through the entire dictionary. There are several ways to do this:

You can loop through all key-value pairs;
You can loop through the keys, and pull out the values for any keys that you care about;
You can loop through the values.

Looping through all key-value pairs
This is the kind of loop that was shown in the first example. Here's what this loop looks like, in a general format:
Can you write Python code for it?

my_dict = {'key_1': 'value_1',
    'key_2': 'value_2',
    'key_3': 'value_3',
    }

for key, value in my_dict.items():
    print('\nKey: %s' % key)
    print('Value: %s' % value)


I am working on the file "06 Dictionaries.ipynb".
The first task is:
top
Modifying values in a dictionary
At some point you may want to modify one of the values in your dictionary. Modifying a value in a dictionary is pretty similar to modifying an element in a list. You give the name of the dictionary and then the key in square brackets, and set that equal to the new value.
Can you write Python code for it?

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

print('dictionary: ' + python_words['dictionary'])

# Clarify one of the meanings.
python_words['dictionary'] = 'A collection of key-value pairs. \
                              Each key can be used to access its corresponding value.'

print('\ndictionary: ' + python_words['dictionary'])


I am working on the file "06 Dictionaries.ipynb".
The first task is:
top
Nesting
Nesting is one of the most powerful concepts we have come to so far. Nesting involves putting a list or dictionary inside another list or dictionary. We will look at two examples here, lists inside of a dictionary and dictionaries inside of a dictionary. With nesting, the kind of information we can model in our programs is expanded greatly.
Lists in a dictionary
A dictionary connects two pieces of information. Those two pieces of information can be any kind of data structure in Python. Let's keep using strings for our keys, but let's try giving a list as a value.
The first example will involve storing a number of people's favorite numbers. The keys consist of people's names, and the values are lists of each person's favorite numbers. In this first example, we will access each person's list one at a time.
Can you write Python code for it?

# This program stores people's favorite numbers, and displays them.
favorite_numbers = {'eric': [3, 11, 19, 23, 42],
                    'ever': [2, 4, 5],
                    'willie': [5, 35, 120],
                    }

# Display each person's favorite numbers.
print("Eric's favorite numbers are:")
print(favorite_numbers['eric'])

print("\nEver's favorite numbers are:")
print(favorite_numbers['ever'])

print("\nWillie's favorite numbers are:")
print(favorite_numbers['willie'])


I am working on the file "06 Dictionaries.ipynb".
The first task is:
top
Overall Challenges
Word Wall

A word wall is a place on your wall where you keep track of the new words and meanings you are learning. Write a terminal app that lets you enter new words, and a meaning for each word.
Your app should have a title bar that says the name of your program.
Your program should give users the option to see all words and meanings that have been entered so far.
Your program should give users the option to enter a new word and meaning.
Your program must not allow duplicate entries.

Your program should store existing words and meanings, even after the program closes.
Your program should give users the option to modify an existing meaning.

Bonus Features
Allow users to modify the spelling of words.
Allow users to categorize words.
Turn the program into a game that quizzes users on words and meanings.
(later on) Turn your program into a website that only you can use.
(later on) Turn your program into a website that anyone can register for, and use.
Add a visualization feature that reports on some statistics about the words and meanings that have been entered.

Periodic Table App

The Periodic Table of the Elements was developed to organize information about the elements that make up the Universe. Write a terminal app that lets you enter information about each element in the periodic table.
Make sure you include the following information:
symbol, name, atomic number, row, and column

Choose at least one other piece of information to include in your app.
Provide a menu of options for users to:
See all the information that is stored about any element, by entering that element's symbol.
Choose a property, and see that property for each element in the table.

Bonus Features
Provide an option to view the symbols arranged like the periodic table. (hint)
Can you write Python code for it?

# Challenge: Word Wall

# put your code here

# Challenge: Periodic Table App

# put your code here


I am working on the file "06 Dictionaries.ipynb".
The first task is:
top
Removing key-value pairs
You may want to remove some key-value pairs from one of your dictionaries at some point. You can do this using the same del command you learned to use with lists. To remove a key-value pair, you give the del command, followed by the name of the dictionary, with the key that you want to delete. This removes the key and the value as a pair.
Can you write Python code for it?

python_words = {'list': 'A collection of values that are not connected, but have an order.',
                'dictionary': 'A collection of key-value pairs.',
                'function': 'A named set of instructions that defines a set of actions in Python.',
                }

# Remove the word 'list' and its meaning.
_ = python_words.pop('list')

# Show the current set of words and meanings.
print("\n\nThese are the Python words I know:")
for word, meaning in python_words.items():
    print("\nWord: %s" % word)
    print("Meaning: %s" % meaning)


I am working on the file "06 Rückblick, Scraping Beispiele/01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
10.Schreibe nun ein Programm, das alle Ziffern 4 mit dem Buchstaben A ersetzte, alle Ziffern 5 mit dem Buchtaben B.
Can you write Python code for it?

newnewlist = []
for elem in new_lst:
    if '4' in elem:
        elem = elem.replace('4', '')
    if '5' in elem:
        elem = elem.replace('5', '')
    newnewlist.append(elem)

newnewlist


I am working on the file "06 Rückblick, Scraping Beispiele/01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
2.Drucke alle die Zahlen von 0 bis 4 aus:
Can you write Python code for it?


I am working on the file "06 Rückblick, Scraping Beispiele/01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
4.Baue einen For-Loop, indem Du alle geraden Zahlen ausdruckst, die tiefer sind als 237.
Can you write Python code for it?


I am working on the file "06 Rückblick, Scraping Beispiele/01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
5.Addiere alle Zahlen in der Liste
Can you write Python code for it?

sum(numbers)

count = 0
for x in numbers:
    count += x
print(count)

#Lösung:


I am working on the file "06 Rückblick, Scraping Beispiele/01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
8.Entwickle ein Programm, das alle Nummern zwischen 2000 und 3200 findet, die durch 7, aber nicht durch 5 teilbar sind. Das Ergebnis sollte auf einer Zeile ausgedruckt werden. Tipp: Schaue Dir hier die Vergleichsoperanden von Python an.
Can you write Python code for it?


I am working on the file "06 Rückblick, Scraping Beispiele/01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
9.Schreibe einen For Loop, der die Nummern in der folgenden Liste von int in str verwandelt.
Can you write Python code for it?

lst = range(45,99)

new_lst = []

for x in lst:
    x = str(x)
    new_lst.append(x)

print(new_lst)


I am working on the file "06 Rückblick, Scraping Beispiele/03 Python Functions, 10 Übungen.ipynb".
The first task is:
1.Schreibe eine Funktion, die aus einer Liste, die grösste Zahl herauszieht. Es ist verboten mit "max" zu arbeiten. :-)
Can you write Python code for it?

lst = [3,5,6,2,6,89,90]

max(lst)

def maxi(element):
    element.sort()
    element.reverse()
    return element[0]

maxi(lst)

count = 0
def maxi(element):
    if element > count:

    return element[-1]

maxi(lst)


I am working on the file "06 Rückblick, Scraping Beispiele/03 Python Functions, 10 Übungen.ipynb".
The first task is:
3.Schreibe eine Funktion, die alle Elemente einer Liste multipliziert.
Can you write Python code for it?

lst = [23,213, 234,234, 3323]

def numbers(lst):
    added = 1
    for number in lst:
        added *= number
    return added

numbers(lst)


I am working on the file "06 Rückblick, Scraping Beispiele/03 Python Functions, 10 Übungen.ipynb".
The first task is:
4.Schreibe eine Funktion, die einen String nimmt, und spiegelt. Also "hallo" zu "ollah".
Can you write Python code for it?

hallo = 'hallo'

def mirrorstring(string):
    mirrored = []
    new_string = ''
    for char in string:
        mirrored.insert(0,char)
    for char in mirrored:
        new_string = new_string + char
    return new_string

mirrorstring(hallo)

def mirror(mylist):
        return mylist[::-1]

list(hallo).reverse()

[2,4,5,5,6][::-1]

mirror(hallo)

def umkehr(wort):
    lm = input('bitte geben Sie ein Wort ein.')
    wort = list(lm)
    wort.reverse()
    print(wort)

umkehr('')


I am working on the file "06 Rückblick, Scraping Beispiele/03 Python Functions, 10 Übungen.ipynb".
The first task is:
6.Lösche die mehrfach genannten Elemente aus der folgenden Liste.
Can you write Python code for it?


I am working on the file "06 Rückblick, Scraping Beispiele/03 Python Functions, 10 Übungen.ipynb".
The first task is:
7.Drucke die geraden Zahlen aus der folgenden Liste aus:
Can you write Python code for it?


I am working on the file "06 Rückblick, Scraping Beispiele/03 Python Functions, 10 Übungen.ipynb".
The first task is:
8.Prüfe mit einer Funktionen, wieviele Grossbuchstaben in folgendem Satz zu finden sind.
Can you write Python code for it?

satz = "In Oesterreich zeichnet sich ein Rechtsrutsch ab. OeVP und FPOe haben stark zugelegt. Gemaess der neusten Hochrechnung ist die Partei von Sebastian Kurz mit 31,6 Prozent der Stimmen Wahlsiegerin, auf Platz zwei folgt die SPÖ (26,9 Prozent) vor der FPOe (26,0 Prozent)."

def counting_caps(XXXXX):
    XXXXXX = 0
    for XXXXXX in elem:
        if XXXXXX.isupper():
            XXXXXXX += 1
    return  XXXXXXXX

 counting_caps(satz)


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
A very similar slicing syntax is also used in many data science-oriented packages, including NumPy and Pandas (mentioned in the introduction).
Now that we have seen Python lists and how to access elements in ordered compound types, let's take a look at the other three standard compound data types mentioned earlier.
Tuples
Tuples are in many ways similar to lists, but they are defined with parentheses rather than square brackets:
Can you write Python code for it?


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
Finally, it is possible to specify a third integer that represents the step size; for example, to select every second element of the list, we can write:
Can you write Python code for it?


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
If you're familiar with the mathematics of sets, you'll be familiar with operations like the union, intersection, difference, symmetric difference, and others.
Python's sets have all of these operations built-in, via methods or operators.
For each, we'll show the two equivalent methods:
Can you write Python code for it?

# union: items appearing in either
primes | odds      # with an operator
primes.union(odds) # equivalently with a method

# intersection: items appearing in both
primes & odds             # with an operator
primes.intersection(odds) # equivalently with a method

# difference: items in primes but not in odds
primes - odds           # with an operator
primes.difference(odds) # equivalently with a method

# symmetric difference: items appearing in only one set
primes ^ odds                     # with an operator
primes.symmetric_difference(odds) # equivalently with a method


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
Items are accessed and set via the indexing syntax used for lists and tuples, except here the index is not a zero-based order but valid key in the dictionary:
Can you write Python code for it?

# Access a value via the key
numbers['two']


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
Keep in mind that dictionaries do not maintain any sense of order for the input parameters; this is by design.
This lack of ordering allows dictionaries to be implemented very efficiently, so that random element access is very fast, regardless of the size of the dictionary (if you're curious how this works, read about the concept of a hash table).
The python documentation has a complete list of the methods available for dictionaries.
Sets
The fourth basic collection is the set, which contains unordered collections of unique items.
They are defined much like lists and tuples, except they use the curly brackets of dictionaries:
Can you write Python code for it?


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
Like the lists discussed before, tuples have a length, and individual elements can be extracted using square-bracket indexing:
Can you write Python code for it?


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
Similarly, if we leave out the last index, it defaults to the length of the list.
Thus, the last three elements can be accessed as follows:
Can you write Python code for it?


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
The main distinguishing feature of tuples is that they are immutable: this means that once they are created, their size and contents cannot be changed:
Can you write Python code for it?


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
These multiple return values can be individually assigned as follows:
Can you write Python code for it?

numerator, denominator = x.as_integer_ratio()
print(numerator / denominator)


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
They can also be defined without any brackets at all:
Can you write Python code for it?


I am working on the file "06-Built-in-Data-Structures.ipynb".
The first task is:
Tuples are often used in a Python program; a particularly common case is in functions that have multiple return values.
For example, the as_integer_ratio() method of floating-point objects returns a numerator and a denominator; this dual return value comes in the form of a tuple:
Can you write Python code for it?


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
Convert Columns
Can you write Python code for it?

for col in ['rpt', 'occur', 'poss']:
    datser = df.apply(fixdatetime(col), axis=1)
    df['%s_dt'%col] = pd.to_datetime(datser, format="%m/%d/%Y %H:%M:%S", errors='coerce')

df.head()


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
Convert Time Columns
Please refer to the following resources for working with time series data in pandas:
- https://pandas.pydata.org/pandas-docs/stable/timeseries.html
- https://pandas.pydata.org/pandas-docs/stable/api.html#id10
Can you write Python code for it?

for col in ['rpt', 'occur', 'poss']:
    datser = df.apply(fixdatetime(col), axis=1)
    df['%s_dt'%col] = pd.to_datetime(datser, format="%m/%d/%Y %H:%M:%S", errors='coerce')

df.head()

def fun1(x):
    def _inner1(y):
        print x, " + ", y, " = ", x+y
    return _inner1

showname = fun1('name:')

type(showname)

showname('Bob')

addthree = fun1(3)

addthree(5)


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
Crimes by Zone
In the beat ID, the hundreds place indicates the zone number...
Let's create a separate column for zone.
Can you write Python code for it?

df['Zone'] = df['beat']//100

pd.DataFrame(df.groupby(['Zone','UC2 Literal']).offense_id.count())


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
Date and Time

Working with dates can be tricky. Often dates and times are coded as strings and need to be converted to a date and time data format.
Python provides a module datetime to deal with converting parsing and formatting dates and times. See https://docs.python.org/2/library/datetime.html
The pandas package provides functionality to convert text fields into date/time fields...given the values adhere to a given format. See http://pandas.pydata.org/pandas-docs/version/0.20/generated/pandas.to_datetime.html

Create a proper text field
In order to use the text to date/time converter our text columns need to have the appropriate format.
Can you write Python code for it?

# function currying
def fixdatetime(fld):
    def _fix(s):
        date_col = '%s_date' % fld
        time_col = '%s_time' % fld
        if time_col in s.index:
            return str(s[date_col])+' '+str(s[time_col])
        else:
            return str(s[date_col])+' 00:00:00'
    return _fix

df.apply(fixdatetime('rpt'), axis=1)[:10]


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
Descriptive Statistics
https://pandas.pydata.org/pandas-docs/stable/basics.html#descriptive-statistics
Beats and Zones
The City of Atlanta is divided into 6 zones, each with 12 to 14 beats.
Can you write Python code for it?

# let's create a colormap for the different colors...
crime_col = dict(zip(sorted(df['UC2 Literal'].unique()),
                     ['red','olive','blue','black','black','royalblue','slategrey','purple','purple','darkblue','coral']))

crime_col

violent = df[df['UC2 Literal'].apply(lambda x: x in set(['AGG ASSAULT','HOMICIDE','RAPE']))]
plt.scatter(violent.x, violent.y, s=7, c=violent['UC2 Literal'].apply(lambda x: crime_col[x]), edgecolors='none')

df[df['UC2 Literal']=='ROBBERY-RESIDENCE']

violent = df[df['UC2 Literal'].apply(lambda x: x in ['ROBBERY-COMMERCIAL', 'ROBBERY-PEDESTRIAN', 'ROBBERY-RESIDENCE'])]
plt.scatter(violent.x, violent.y, s=10, c=violent['UC2 Literal'].apply(lambda x: crime_col[x]), edgecolors='none', marker='^')


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
This is an odd assortment of years... let's look at number of crimes reported each year
Can you write Python code for it?

# resample is like "groupby" for time
df.resample('A-DEC', closed='right', on='occur_dt').offense_id.count()
# df['Year'] = df.occur_dt.map(lambda d: d.year)
# df2 = df[(df.Year>=2010) & (df.Year<=2017)]
# df2.shape, df.shape


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
This shows that "shift" is a unicode string...
Date and Time

Working with dates can be tricky. Often dates and times are coded as strings and need to be converted to a date and time data format.
Python provides a module datetime to deal with converting parsing and formatting dates and times. See https://docs.python.org/2/library/datetime.html
The pandas package provides functionality to convert text fields into date/time fields...given the values adhere to a given format. See http://pandas.pydata.org/pandas-docs/version/0.20/generated/pandas.to_datetime.html

Create a proper text field
In order to use the text to date/time converter our text columns need to have the appropriate format.
Can you write Python code for it?

# function currying
def fixdatetime(fld):
    def _fix(s):
        date_col = '%s_date' % fld # "rpt_date"
        time_col = '%s_time' % fld # "rpt_time"
        if time_col in s.index:
            return str(s[date_col])+' '+str(s[time_col])
        else:
            return str(s[date_col])+' 00:00:00'
    return _fix

df.apply(fixdatetime('rpt'), axis=1)[:10]
# df.apply(_fix, axis=1)


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
Time Series and Plotting
The Pandas package provides a number of plotting features. Let's try them out.
- https://pandas.pydata.org/pandas-docs/stable/api.html#api-dataframe-plotting
- https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.boxplot.html
What time of day do different types of crimes most often occur at?
Types of crimes in the database:
Can you write Python code for it?

print "\n".join(sorted(df['UC2 Literal'].unique()))

df['oct'] = pd.to_timedelta(df.occur_time) # estimated time of ocurence in timedelta format

# Let's take a closer look at a few selected examples of crime...
vehicle_larceny = df[df['UC2 Literal']=='LARCENY-FROM VEHICLE'].resample('1H', closed='right', on='oct').offense_id.count()
pedestrian_robbery = df[df['UC2 Literal']=='ROBBERY-PEDESTRIAN'].resample('1H', closed='right', on='oct').offense_id.count()
residence_burglary = df[df['UC2 Literal']=='BURGLARY-RESIDENCE'].resample('1H', closed='right', on='oct').offense_id.count()
assault = df[df['UC2 Literal']=='AGG ASSAULT'].resample('1H', closed='right', on='oct').offense_id.count()

fig = plt.figure(figsize=(8,4));
plt.plot_date(vehicle_larceny.index, vehicle_larceny.values);
plt.title('Vehilce Larceny by Time of Day');
plt.xlabel('Time (24h format)');
plt.ylabel('Number of incidents');

fig = plt.figure(figsize=(8,4));
plt.plot_date(pedestrian_robbery.index, pedestrian_robbery.values);
plt.title('Pedestrian Robbery by Time of Day');
plt.xlabel('Time (24h format)');
plt.ylabel('Number of incidents');

fig = plt.figure(figsize=(8,4));
plt.plot_date(residence_burglary.index, residence_burglary.values);
plt.title('Residence Burglary by Time of Day');
plt.xlabel('Time (24h format)');
plt.ylabel('Number of incidents');

fig = plt.figure(figsize=(8,4));
plt.plot_date(assault.index, assault.values);
plt.title('Aggrivated Assault by Time of Day');
plt.xlabel('Time (24h format)');
plt.ylabel('Number of incidents');


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
We need to enter the descriptions for each entry in our dictionary manually. However, why not just create a the Python code automatically...
Run the code below only if you haven't edited the datadict.py file in a different way, since it will overwrite what you have so far. (That's why the code is commented-out.)
Can you write Python code for it?

# with open("datadict.py", "w") as io:
#     for i in dataDict.index:
#         io.write("dataDict.loc['%s'].Description = '' # type: %s\n" % (i, str(dataDict.loc[i].DataType)))

# %load datadict.py
dataDict.loc['MI_PRINX'].Description = '' # type: int64
dataDict.loc['offense_id'].Description = 'Unique ID in the format YYDDDNNNN with the year YY, the day of the year DDD and a counter NNNN' # type: int64
dataDict.loc['rpt_date'].Description = 'Date the crime was reported' # type: object
dataDict.loc['occur_date'].Description = 'Estimated date when the crime occured' # type: object
dataDict.loc['occur_time'].Description = 'Estimated time when the crime occured' # type: object
dataDict.loc['poss_date'].Description = '' # type: object
dataDict.loc['poss_time'].Description = '' # type: object
dataDict.loc['beat'].Description = '' # type: int64
dataDict.loc['apt_office_prefix'].Description = '' # type: object
dataDict.loc['apt_office_num'].Description = '' # type: object
dataDict.loc['location'].Description = '' # type: object
dataDict.loc['MinOfucr'].Description = '' # type: int64
dataDict.loc['MinOfibr_code'].Description = '' # type: object
dataDict.loc['dispo_code'].Description = '' # type: object
dataDict.loc['MaxOfnum_victims'].Description = '' # type: float64
dataDict.loc['Shift'].Description = 'Zones have 8 or 10 hour shifts' # type: object
dataDict.loc['Avg Day'].Description = '' # type: object
dataDict.loc['loc_type'].Description = '' # type: float64
dataDict.loc['UC2 Literal'].Description = '' # type: object
dataDict.loc['neighborhood'].Description = '' # type: object
dataDict.loc['npu'].Description = '' # type: object
dataDict.loc['x'].Description = '' # type: float64
dataDict.loc['y'].Description = '' # type: float64
dataDict.to_csv("COBRA_Data_Dictionary.csv")


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
We need to enter the descriptions for each entry in our dictionary manually...
Can you write Python code for it?

dataDict.loc['MI_PRINX'].Description = '' # type: int64
dataDict.loc['offense_id'].Description = 'Unique ID in the format YYDDDNNNN with the year YY, the day of the year DDD and a counter NNNN' # type: int64
dataDict.loc['rpt_date'].Description = 'Date the crime was reported' # type: object
dataDict.loc['occur_date'].Description = 'Estimated date when the crime occured' # type: object
dataDict.loc['occur_time'].Description = 'Estimated time when the crime occured' # type: object
dataDict.loc['poss_date'].Description = '' # type: object
dataDict.loc['poss_time'].Description = '' # type: object
dataDict.loc['beat'].Description = '' # type: int64
dataDict.loc['apt_office_prefix'].Description = '' # type: object
dataDict.loc['apt_office_num'].Description = '' # type: object
dataDict.loc['location'].Description = '' # type: object
dataDict.loc['MinOfucr'].Description = '' # type: int64
dataDict.loc['MinOfibr_code'].Description = '' # type: object
dataDict.loc['dispo_code'].Description = '' # type: object
dataDict.loc['MaxOfnum_victims'].Description = '' # type: float64
dataDict.loc['Shift'].Description = 'Zones have 8 or 10 hour shifts' # type: object
dataDict.loc['Avg Day'].Description = '' # type: object
dataDict.loc['loc_type'].Description = '' # type: float64
dataDict.loc['UC2 Literal'].Description = '' # type: object
dataDict.loc['neighborhood'].Description = '' # type: object
dataDict.loc['npu'].Description = '' # type: object
dataDict.loc['x'].Description = '' # type: float64
dataDict.loc['y'].Description = '' # type: float64
dataDict.to_csv("COBRA_Data_Dictionary.csv")

dataDict


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
What is "object?"
Can you write Python code for it?


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
What is object
Let's take a look at what we've got...
Can you write Python code for it?


I am working on the file "06-Descriptive-Statistics/CrimeData_orig.ipynb".
The first task is:
What's the date range of the data?
Can you write Python code for it?

print df.occur_dt.min(), '---', df.occur_dt.max()


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
Absolute humidity
The ryutils.abshumidity function  calculates atmopsheric absolute humidity [g/m3] for temperature in [K] between 248 K and 342 K. This function provides two similar equations, but with different constants.
Atmospheric humidity is commonly expressed in relative humidity (RH). Relative humidity is the quantity of water vapor in the atmosphere, expressed as a percentage of the maximum absolute humidity.  The maximum absolute humidity is determined by the water-vapor partial pressure at the atmospheric temperature and is given by this equation (Equation 1 in the function):
$$
q=\frac{1325.252}{T} 10^{7.5892(T - 273.15)/(T -32.44)},
$$
in units of [g/m$^3$] and temperature in [K]. This equation has an error of less than 1% over the range $-$20 $^\circ$C to 0 $^\circ$C, and less than 0.1% for the temperature range 0 $^\circ$C to 50 $^\circ$C.
The alternative equation  (Equation 1 in the function) is:
$$
q=\frac{1324.37872}{T} \exp[
17.67*(T - 273.16)/(T - 29.66))
$$
The function signature is:
    abshumidity(T, equationSelect = 1)

temperature (np.array[N,] or [N,1])  in  [K].
equationSelect (int) select the equation to be used.

In this example the values from the equations are compared to data from:
Can you write Python code for it?

import pyradi.ryplot as ryplot
%matplotlib inline

data=np.asarray([
[   50  ,   82.78  ]   ,
[   45  ,   65.25    ]   ,
[   40  ,   50.98    ]   ,
[   35  ,   39.47    ]   ,
[   30  ,   30.26    ]   ,
[   25  ,   22.97  ]   ,
[   20  ,   17.24    ]   ,
[   15  ,   12.8    ]   ,
[   10  ,   9.38 ]   ,
[   5   ,   6.79 ]   ,
[   0   ,   4.85 ]   ,
[   -5  ,   3.41 ]   ,
[   -10 ,   2.36 ]   ,
[   -15 ,   1.61 ]   ,
[   -20 ,   1.08 ]   ,
[   -25 ,   0.71 ] ])
temperature = data[:,0]+273.15
absh = ryutils.abshumidity(temperature).reshape(-1,1)
data = np.hstack((data,absh))
# absh = ryutils.abshumidity(temperature, 2).reshape(-1,1)
# data = np.hstack((data,absh))
data = np.hstack((data, 100 * np.reshape((data[:,1]-data[:,2])/data[:,2],(-1,1))))
print('        deg C          Testvalue           Fn value       % Error')
print(data)

p=ryplot.Plotter(1, 2, 1, figsize=(12,6))
p.plot(1,data[:,0].T, data[:,2].T,'Absolute humidity vs temperature','Temperature [$^\circ$C]','Absolute humidity g/m$^3$]')
p.plot(2,data[:,0].T, data[:,3].T,'\% humidity error vs temperature','Temperature [$^\circ$C]','\% error')
# p.saveFig('abshum.eps')

#highest ever recorded absolute humidity was at dew point of 34C
print('Highest recorded absolute humidity was {0}, dew point {1} deg C'.\
    format(ryutils.abshumidity(np.asarray([34 + 273.15]))[0],34))


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
Calculating coordinate index arrays for a Numpy array
The
ryutils.index_coords function calculates zero-based (x,y) coordinate arrrays for a numpy array indices, relative to some origin.
This function calculates two meshgrid arrays containing the coordinates of the
input array.  The origin of the new coordinate system  defaults to the
center of the image, unless the user supplies a new origin.
The data format can be data.shape = (rows, cols, frames) or
data.shape = (frames, rows, cols), the format of which is indicated by the
framesFirst parameter.
The function signature is:
`index_coords(data, origin=None, framesFirst=True)`

data (np.array) array for which coordinates must be calculated.
origin ( (x-orig, y-orig) ) data-coordinates of where origin should be
framesFirst (bool) True if data.shape is (frames, rows, cols), False if data.shape is (rows, cols, frames)

returns

x (float np.array) x coordinates in array format.
y (float np.array) y coordinates in array format.

This transformation code was originally written by Joe Kington.
The following example demonstrates the variations obtained for frames first or frames last, and the location of the origin.
Can you write Python code for it?

import pyradi.ryutils as ryutils
import numpy as np

data = np.ones([2, 3, 4])

x, y = ryutils.index_coords(data, origin=None, framesFirst=True)
print('framesFirst=True, origin=None')
print('x={}\ny={}'.format(x,y))

x, y = ryutils.index_coords(data, origin=None, framesFirst=False)
print('\nframesFirst=False, origin=None')
print('x={}\ny={}'.format(x,y))

x, y = ryutils.index_coords(data, origin=(0,0), framesFirst=True)
print('\nframesFirst=True, origin=(0,0)')
print('x={}\ny={}'.format(x,y))

x, y = ryutils.index_coords(data, origin=(3,2), framesFirst=True)
print('\nframesFirst=True, origin=(3,2)')
print('x={}\ny={}'.format(x,y))


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
Create clean filenames
The ryfiles.cleanFilename
function cleans a string by removing selected characters.  It creates a legal and 'clean' source string from a string by removing some
clutter and  characters not allowed in filenames.     A default set is given but the user can override the default string.
The function signature is:
    cleanFilename(sourcestring,  removestring =" %:/,.\\[]")

sourcestring (string) the string to be cleaned.
removestring (string) remove all these characters from the string (optional).

It is quite useful to use a figure caption as filename, first cleaning it up.
Can you write Python code for it?

print ('\nTest CleanFilename function:')
inString="aa bb%cc:dd/ee,ff.gg\\hh[ii]jj"
print('{0}\n{1}\n'.format(inString, ryfiles.cleanFilename(inString) ))
inString="aa bb%cc:dd/ee,ff.gg\\hh[ii]jj"
print('{0}\n{1}'.format(inString, ryfiles.cleanFilename(inString, "abcd") ))


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
Downloading compressed and uncompressed files from the internet
The ryfiles.downloadFileUrl function downloads a file from a URL.  The URL is used to download a file, to the saveFilename specified. If no saveFilename is given, the basename of the URL is used.  The name of the downloaded file is returned, or None if the download file.
The function signature is:
`downloadFileUrl(url,  saveFilename=None)`

url (string) the url to be accessed.
saveFilename (string) path to where the file must be saved (optional).

The ryfiles.downloadUntar downloads and untars a compressed tar archive, and saves all files to the specified directory.    The tarfilename is used to open the tar file, extracting to the destinationDir specified. If no destinationDir is given, the local directory '.' is used.  Returns a list of the intarred filenames, or None of not succesful. Before downloading, a check is done to determine if the file was already downloaded  and exists in the local file system.
The file signature is:
`downloadUntar(tgzFilename, url, destinationDir=None,  tarFilename=None)`

tgzFilename (string) the name of the tar archive file.
url (string) url where to look for the file (not including the filename).
destinationDir (string) to where the files must be extracted  (optional).
tarFilename (string) downloaded tar filename (optional).

The ryfiles.untarTarfile untars a tar archive, and save all files to the specified directory. The tarfilename is used to open a file, extraxting to the saveDirname specified. If no saveDirname is given, the local directory '.' is used. Returns a list of filenames saved, or None if failed.
The file signature is:
`untarTarfile(tarfilename, saveDirname=None)`

tarfilename (string) the name of the tar archive.
saveDirname (string) to where the files must be extracted

The ryfiles.unzipGZipfile
unzips a file that was compressed using the gzip format.  The zipfilename is used to open a file, to the saveFilename specified. If no saveFilename is given, the basename of the zipfilename is...
Can you write Python code for it?

tgzFilename = 'arrayplotdemo.tgz'
destinationDir = '.'
tarFilename = 'arrayplotdemo.tar'
url = 'https://raw.githubusercontent.com/NelisW/pyradi/master/pyradi/data/'
dlNames = ryfiles.downloadUntar(tgzFilename, url, destinationDir, tarFilename)

if dlNames:
    print('filesAvailable are {}'.format(dlNames))


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
File and directory services
List files in a directory
The ryfiles.listFiles function
returns a list of file paths to files in a file system, searching a directory structure along the specified path, looking for files that matches the glob pattern. If specified, the search will continue into sub-directories.  A list of matching names is returned.
The function supports a local or network reachable filesystem, but not URLs.
The function signature is:
listFiles(root, patterns='*', recurse=1, return_folders=0, useRegex=False)

root (string) directory root from where the search must take place.
patterns (string) glob/regex pattern for filename matching.
recurse (unt)  flag to indicate if subdirectories must also be searched (optional).
return_folders (int)  flag to indicate if folder names must also be returned (optional).
useRegex (bool)  flag to indicate if patterns areregular expression strings (optional).
Can you write Python code for it?

import pyradi.ryfiles as ryfiles
print(ryfiles.listFiles('.', '05*.ipynb'))

import pyradi.ryfiles as ryfiles
print(ryfiles.listFiles('.', '05*', return_folders=1))

print(ryfiles.listFiles('.', '05*.ipynb', useRegex=1))

print(ryfiles.listFiles('.', '05[ab].*', useRegex=1))

print(ryfiles.listFiles('.', '^05[ab].*\.[^h].*', useRegex=1))


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
Probability Density by Kernel Density Estimate
Jake vdPlas makes a good point why and how histograms are not really good probability density estimators here.  Depending on how the data points fall into the bins strange effects may arise.  He writes 'The problem with our two binnings stems from the fact that the height of the block stack often reflects not on the actual density of points nearby, but on coincidences of how the bins align with the data points. This mis-alignment between points and their blocks is a potential cause of the poor histogram results.
The histogram approach is to have (normally same-size) fixed bin edges, in to which the data points are falling and being counted. The Kernel Density method works the other way around make fixed-width block around each data point and add up the blocks' heights.  This way the weights add up around the data points' locations.  This is somewhat making a bin centred around each data point and then adding up the bin counts at a fine resolution.
Working with rectangular blocks the new density function has rough edges at the blocks' corners. The next logical step is to use smooth functions (called kernels) instead of rectangular blocks. This yields smooth functions with more appropriate density distributions than does the traditional histogram.  In this approach you must define/select the kernel shape and width. If the kernel is too wide a kernel's influence can be too wide, masking the data points. If the kernel is too narrow it will leave 'holes' between the data points. Just image a dirac delta kernel function which simply repeats the data set.
scikit-learn provides an efficient library for kernel density estimation. The library provides six different kernel shapes and a method whereby an appropriate  kernel width can be found. The pyradi.ryprob.KDEbounded function packages the scikit learn functionality in an easily usable single function.
The gaussian kernels typically used in KDE reaches to infinity and the density function domain therefore spans infinity...
Can you write Python code for it?

import numpy as np
import pyradi.ryprob as ryprob
import pyradi.ryplot as ryplot

def make_data(N, f=0.3, rseed=1):
    rand = np.random.RandomState(rseed)
    x = rand.randn(N) #mean-0 var=1
    # create bimodal data by adding const to  entries with index > f*N
    # this means that some of the date keep old values and some shift
    x[int(f * N):] += 5
    return x

# helper function to set up experiments
def showexperiment(bounds=(np.nan,np.nan),bandwidth=np.nan,scale=1,offset=0,kernel='gaussian'):

    x = scale * make_data(20) + offset

    if not np.isnan(bounds[0]) and not np.isnan(bounds[1]):
        lowbnd = bounds[0]
        uppbnd = bounds[1]
        x_d = np.linspace(lowbnd,uppbnd, 1000)
    else:
        lowbnd = np.nan
        uppbnd = np.nan
        tenp = (np.max(x) - np.min(x)) / 10
        x_d = np.linspace(np.min(x)-tenp,np.max(x)+tenp, 1000)

    x_d,kde,bandwidth,kernel = ryprob.KDEbounded(x_d,x,bandwidth=bandwidth,lowbnd=lowbnd,uppbnd=uppbnd, kernel=kernel)

    print(f'x.min={np.min(x):.4e}  x.max={np.max(x):.4e} x.shape={x.shape}')
    print(f'x.min={np.min(x_d):.4e}  x.max={np.max(x_d):.4e} x.shape={x_d.shape}')
    print(f'Bandwidth={bandwidth}')
    print(f'kernel={kernel}')
    print(f'Integral of PDF={np.trapz(kde,x_d)}')

    p = ryplot.Plotter(1,1,1)
    p.plot(1,x,np.full_like(x, 0.011*np.max(kde)),linestyle='',markers='^')
    p.plot(1,x_d,kde,'Kernel Density Estimate','$x_d$','$p$')

showexperiment(bounds=[-3.301,7.74481], bandwidth=np.nan, scale=1,offset=0)


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
Pulse detection: probability of detection and false alarm rate
The ryutils.detectSignalToNoise
function solves for the  signal to noise ratio, given the threshold to noise ratio and probability of detection.  Using the theory of matched filter design, calculate the  signal to noise ratio, to achieve a required probability of detection. The function returns the  signal to noise ratio required to achieve the probability of detection.
Reference: "Electro-optics handbook," Tech. Rep. EOH-11, RCA, 1974. RCA Technical Series Publication.
When there is a signal present, the probability of detection (signal plus noise exceeds the threshold) is given by
$$
P_d =
\frac{1}{2}\left[
1+{\rm erf}\left(
\frac{i_s-i_t}{\sqrt{2}i_n}
\right)
\right]
$$
where ${\rm erf}$ is the error function:
$$
{\rm erf}(z)=\frac{2}{\sqrt{\pi}}\int_0^z e^{-t^2}dt.
$$
The function signature is:
`detectSignalToNoise(ThresholdToNoise, pD)`

ThresholdToNoise (float) the threshold to noise ratio (unitless).
pD (float) the probability of detection (unitless).

The ryutils.detectThresholdToNoise
function solve for threshold to noise ratio, given pulse width and FAR, for matched filter.  Using the theory of matched filter design, calculate the     threshold to noise ratio, to achieve a required false alarm rate. The function returns the threshold to noise ratio.  Reference   R. D. Hippenstiel, Detection Theory: Applications and Digital Signal Pro-cessing, CRC Press, 2002.
The average false alarm rate is given by
$$
{FAR}=
\frac{1}{2 t_p \sqrt{3}}\exp^{-i_t^2/(2i_n^2)},
$$
where
$t_p$ is the pulse width,
$i_t$ is the threshold value, and
$i_n$ is the rms noise value at the input to the threshold detector.
The function signature is:
    detectThresholdToNoise(pulseWidth, FAR)

pulseWidth (float) the signal pulse width in [s].
FAR (float) the false alarm rate in [alarms/s].
Can you write Python code for it?

import pyradi.ryutils as ryutils

pulsewidth = 100e-9
FAR = 15
probDetection = 0.999
ThresholdToNoise = ryutils.detectThresholdToNoiseTpFAR(pulsewidth,FAR)
SignalToNoise = ryutils.detectSignalToNoiseThresholdToNoisePd(ThresholdToNoise, probDetection)
print('For a laser pulse with width={0}, a FAR={1} and Pd={2},'.format(pulsewidth,FAR,probDetection))
print('the Threshold to Noise ratio must be {0}'.format(ThresholdToNoise))
print('and the Signal to Noise ratio must be {0}'.format(SignalToNoise))
print(' ')


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
Reading two-dimensional lookup tables
The ryfiles.read2DLookupTable
Reads a 2D lookup table and extract the data. The table has the following format:
line 1: xlabel ylabel title
line 2: 0 (vector of y (col) abscissa)
lines 3 and following: (element of x (row) abscissa), followed
by table data.

From line/row 3 onwards the first element is the x abscissa value
followed by the row of data, one point for each y abscissa value.
The file format can depicted as follows:
x-name y-name ordinates-name
0 y1 y2 y3 y4
x1 v11 v12 v13 v14
x2 v21 v22 v23 v24
x3 v31 v32 v33 v34
x4 v41 v42 v43 v44
x5 v51 v52 v53 v54
x6 v61 v62 v63 v64

The function has the signature:
`read2DLookupTable(filename)`

fname (string) input path and filename

The function returns the following:

xVec ((np.array[N])) x abscissae
yVec ((np.array[M])) y abscissae
data ((np.array[N,M])) data corresponding the x,y
xlabel (string) x abscissa label
ylabel (string) y abscissa label
title (string) dataset title
Can you write Python code for it?

filecont = ['x-name y-name ordinates-name\n',
'0 1   2   3  4\n',
'10 11 12 13 14\n',
'20 21 22 23 24\n',
'30 31 32 33 34\n',
'40 41 42 43 44\n',
'50 51 52 53 54\n',
'60 61 62 63 64\n']

with open('2dfile.dat', 'wt') as f:
    f.writelines(filecont)

import os.path
if os.path.sep == '/':
    !cat 2dfile.dat
else:
    !type 2dfile.dat

xVec,yVec,data,xlabel, ylabel, title = ryfiles.read2DLookupTable('2dfile.dat')
print('\n')
print('xVec={}'.format(xVec))
print('yVec={}'.format(yVec))
print('data={}'.format(data))
print('xlabel={}'.format(xlabel))
print('ylabel={}'.format(ylabel))
print('title={}'.format(title))


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
Solving the range equation
It is frequently necessary to determine the operational detection distance of a source and sensor combination. The problem is usually stated as follows: "What operating detection range can be achieved with a given source intensity, atmospheric attenuation, and sensor sensitivity?"
The inband irradiance is given by (Sec. 7.5)
$$
E_{\cal S}=\frac{v_{\cal S} }{k \,\widehat{{\cal R}}\,Z_t\,A_{1}}
=
\int_{A_0} \left(
\frac{1}{R_{01}^2}
\int_{0}^{\infty}
 \epsilon_{0\lambda} L_{0\lambda}
\,\tau_{a\lambda}(R) {\cal S}\lambda\,
d\lambda\right) dA{0}\,\cos\theta_0
$$
The objective is to solve for $R$ where the irradiance $E_{{\cal S}}=E_{{\cal S}\theta}$ is at the threshold at which the range must be determined.  Normally, $E_{{\cal S}\theta}=SNR\times NEE$ where the SNR is selected to yield a given probability of detection.
Assuming a target smaller than the sensor field of view, and using the effective transmittance the above equation can be simplified to the form
$$
E_{{\cal S}\theta}=\frac{I_{{\cal S}} \tau_{\rm eff}(R)}{R^2}
$$
where $I_{{\cal S}}$ is the source intensity.
This equation can be solved is $\tau_{\rm eff}(R)$ is available in lookup table form.
The ryutils.rangeEquation function solves the range equation for arbitrary transmittance vs range, with the equation given by
$$
E=\frac{I_{{\cal S}} \tau_{\rm eff}(R)}{R^n}
$$
where $E$ is the threshold irradiance in [W/m2], and $I$ is the intensity in [W/sr]. This range equation holds for
the case where the target is smaller than the field of view.
The range $R$ must be in [m], and $\tau_a(R)$
is calculated from a lookup table of atmospheric transmittance vs. range.
The transmittance lookup table  can be calculated from the simple Bouguer law,
or it can have any abritrary shape, provided it decreases with increasing range.
The user supplies the lookup table in the form of an array of range values and
an associated array of transmittance values.  The range values need not be on
constant linear range increment.
The parameter...
Can you write Python code for it?

import  numpy as np
from scipy.interpolate import interp1d

rangeTab = np.linspace(0, 10000, 1000)
tauTab = np.exp(- 0.00015 * rangeTab)
Intensity=200
Irradiancetab=[10e-100, 10e-6, 10e-1]
for Irradiance in Irradiancetab:
    r = ryutils.rangeEquation(Intensity = Intensity, Irradiance = Irradiance, rangeTab = rangeTab,
          tauTab = tauTab, rangeGuess = 1, n = 2)

    #test the solution by calculating the irradiance at this range.
    tauTable = interp1d(rangeTab, tauTab, kind = 'linear')

    if np.abs(r[0]) < rangeTab[2]:
        rr = rangeTab[2]
        strError = "Check range resolution in lookup table"
    elif np.abs(r[0]) > rangeTab[-1]:
        rr = rangeTab[-1]
        strError = "Check maximum range in lookup table"
    else:
        rr = r[0]
        strError = ""

    irrad = Intensity * tauTable(rr) / rr ** 2

    print('Range equation solver with irradiance threshold of {} W/m2:'.format(Irradiance))
    print('   intensity of {4} W/sr at range {0} m the irradiance is {1} W/m2, error is {2:.2e}. \n{3}\n'.format(
        r[0],irrad, (irrad - Irradiance) / Irradiance, strError, Intensity))


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
To join a folder and a filename use  os.path.join, but there is a catch here;  os.path.join does not expect the tuple, but simply a number of arguments.  Study the documentation carefully, because os.path.join may discard some arguments.
Alternatively, use os.sep or os.path.sep to get the pathname separator ('/' for POSIX and '\' for Windows) and then use this character to join the list.
Can you write Python code for it?


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
Upright micro symbol in LaTeX
The ryutils.upMu
function returns an upright $\mu$ symbol, as required in the SI system for the $10^{-6}$ prefix.
The upright symbol requires that the siunitx LaTeX package be installed on the computer running the code.  This function also changes the Matplotlib rcParams file.  This symbol can be used in graphs or text.
The function signature is:
upMu(uprightMu=True, textcomp=False)

uprightMu (bool) signals upright (True, default) or regular (False) symbol (optional).
textcomp (bool) if True use the textcomp package, else use siunitx package (optional).

The ryutils.upMu function returns a string with LaTeX code for the micro symbol.
Can you write Python code for it?

import pyradi.ryutils as ryutils
print('Using siunitx package:')
print('{} renders in LaTeX as an upright symbol'.format(ryutils.upMu()))
print('{} renders in LaTeX as an upright symbol'.format(ryutils.upMu(True)))
print('{} renders in LaTeX as an italic/slanted symbol'.format(ryutils.upMu(False)))
print('\nUsing textcomp package:')
print('{} renders in LaTeX as an upright symbol'.format(ryutils.upMu(textcomp=True)))
print('{} renders in LaTeX as an upright symbol'.format(ryutils.upMu(True,textcomp=True)))
print('{} renders in LaTeX as an italic/slanted symbol'.format(ryutils.upMu(False,textcomp=True)))


I am working on the file "06-Diverse-pyradi-utilities.ipynb".
The first task is:
Working with filenames and directories
The information in this section does not describe any pyradi functionality, but may be useful in this context.
To split a qualified path into the folder and filename use os.path.split, yielding a tuple with the path as the first element and the filename as the second element.
Can you write Python code for it?


I am working on the file "06-Literary Distinction (Probably)/Literary Patterns (Probably).ipynb".
The first task is:
1. Review
Can you write Python code for it?

# Read Moby Dick
moby_string = open('Melville - Moby Dick.txt').read()

# Inspect the text
moby_string

# Make the text lower case
moby_lower = moby_string.lower()

# Tokenize Moby Dick

moby_tokens = moby_lower.split()

# Check out the tokens
moby_tokens

# Just how long is Moby Dick anyway?
len(moby_tokens)

# Create a list comprehension, including an 'if' statement
just_whales = [token for token in moby_tokens if token=='whale']

# Hast seen the White Whale?
just_whales

# Make a new list
maritime = ['ship','harpoon','sail']

# Multiply it
maritime * 2

# Another list
whaling = ['whiteness','whale','ambergris']

# Concatenate
maritime + whaling


I am working on the file "06-Literary Distinction (Probably)/Literary Patterns (Probably).ipynb".
The first task is:
2. Pre-Process
In their paper, Long and So describe their pre-processing as consisting of three major steps: stop word removal, lemmatization of nouns, and feature selection (based on document frequency). In this workshop, we will focus on the first and third steps, since they can be integrated seamlessly with our workflow and Underwood and Sellers use them as well.
Lemmatization -- the transformation of words into their dictionary forms; e.g. plural nouns become singular -- is particularly useful to Long and So, since they partly aim to study imagery. That is, they find it congenial to collapse the words mountains and mountain into the same token, since they express a similar image. For an introduction to Lemmatization (and a related technique, Stemming), see NLTK: http://www.nltk.org/book/ch03.html#sec-normalizing-text
Import Corpus
Note that due to issues of copyright, volumes' word order has not been retained, although their total word counts have been. Fortunately, our methods do not require word-order information.
Underwood and Sellers's literary corpus has been divided into three folders: "reviewed", "random", "canonic". (The last of these are canonic poets but who did not have the opportunity to be reviewed, such as Emily Dickinson.)
Can you write Python code for it?

import os

# Assign file paths to each set of poems

review_path = 'poems/reviewed/'
random_path = 'poems/random/'

# Get lists of text files in each directory

review_files = os.listdir(review_path)
random_files = os.listdir(random_path)

# Inspect

review_files

# Read-in texts as strings from each location

review_texts = [open(review_path+file_name).read() for file_name in review_files]
random_texts = [open(random_path+file_name).read() for file_name in random_files]

# Inspect

review_texts[0]

# Collect all texts in single list

all_texts = review_texts + random_texts

# Get all file names together

all_file_names = review_files + random_files

# Keep track of classes with labels

all_labels = ['reviewed'] * len(review_texts) + ['random'] * len(random_texts)

## EX. How many file names are listed in the directory for reviewed texts?

## EX. How many texts got read into 'review_texts'? Does it match the number of files in the directory?