I am working on the file "04--comps-gens-sparse.ipynb".
The first task is:
Example: Generating email objects
To see how generators can be useful, let's return to the problem of mining an email archive.
(Review) Parsing an email. First, recall that Python has an email module, which you can use to parse plaintext emails.

As a minor technical aside, these emails should be formatted according to the RFC 822 standard.

Given a string formatted in such a way, you can create a structured email object. You can then query the object to extract fields of the email, like the sender, the recipient, the date, the subject, or the body, among others.

Here is some sample code to look for email addresses in certain fields of the email; run it to see that it produces the expected result.
Can you write Python code for it?

email_msg_text = """Message-ID: aslkj42t90wdk23o5uxc@jinx
Date: Tue Aug 25 23:44:06 EDT 2015
From: Richard (Rich) Vuduc <richie@cc.gatech.edu>
To: Jebediah Springfield <jebby@mindsprang.com>
Reply-To: junk@vuduc.org
Subject: Spam -- delete me

Please read the subject, or reply me at prez@whitehouse.gov or junk@vuduc.org

Thanks,

-- Rich"""

import re
import email.parser
import re

# This line was missing in class
EMAIL_PATTERN = re.compile (r'[\w+.]+@[\w.]+')

# Parses email text into an object
email_msg_obj = email.parser.Parser ().parsestr (email_msg_text)

# Poke around for email addresses in the From, To, Reply-To, and body:
addrs = set (EMAIL_PATTERN.findall (email_msg_obj['From']))
addrs.update (EMAIL_PATTERN.findall (email_msg_obj['Reply-To']))
addrs.update (EMAIL_PATTERN.findall (email_msg_obj['To']))
addrs.update (EMAIL_PATTERN.findall (email_msg_obj.get_payload ()))

print ("Email addresses found: %s" % str (addrs))


I am working on the file "04--comps-gens-sparse.ipynb".
The first task is:
Generators
You've undoubtedly noticed the common use of for ... in ...: constructs in Python programs. The in part is quite flexible, allowing you to iterate in a compact and readable way over many kinds of collections.
```python
Characters in a string
text = 'The quick brown fox jumps over the lazy dog'
for letter in text:
    ...
Lists
x = ['a', 'b', 'c']
for y in x:
    ...
Dictionaries
x = {'a': 1, 'b': 2, 'c': 3}
for key, value in x.items (): # also possible: x.keys(), x.values()
    ...
Range objects
for i in range (0, 10):
    ...
Files (line-by-line)
for line in open ('filename.txt', 'r'):
    ...
```
The last two examples---range objects and files---are especially interesting. They are in fact examples of a special kind of custom iteration that you can design. Here, you'll see an example of one technique called a generator.
Generators. A generator is a special kind of "interruptible" function that yields zero or more values or objects. During its execution, a generator may produce an object or value, temporarily transfer control back to the caller with that object or value, and then resume control from the same place when called again. These control transfer points are marked by yield statements.
For example, suppose you are given a dictionary, and you wish to find all keys whose values match or exceed some threshold. Here is how you might use a generator to do the job.
Can you write Python code for it?

def keys_geq_threshold (Dict, threshold):
    """
    (Generator) Given a dictionary, yields the keys whose values
    are at or above (greater than or equal to) a given threshold.
    """
    for key, value in Dict.items ():
        if value >= threshold:
            yield key


I am working on the file "04--comps-gens-sparse.ipynb".
The first task is:
There is an analogous concept for constructing a set, which is called a set comprehension. A set comprehension constructs a set from an input list or set.
Can you write Python code for it?

list1 = """
how much wood could a woodchuck chuck
if a woodchuck could chuck wood
""".split ()
set_from_list = {x for x in list1}
list_from_set = [x for x in set_from_list]

print (list1)
print (set_from_list)
print (list_from_set)


I am working on the file "04--comps-gens-sparse.ipynb".
The first task is:
This code is a loop over key-value pairs. However, when a matching key is detected, the function yields control back to the caller. If the caller calls this function again, it will resume after the yield statement.
You can call such a generator function as part of the in statement of a for loop to get a sequence of keys. The range() and open() functions are themselves generators!

Take a look at the code and run it to verify that it produces the expected result.
Can you write Python code for it?

inventory = {'apples': 6, 'bananas': 3, 'milk': 5, 'peanuts': 10}

# Apply the generator:
overstock = [key for key in keys_geq_threshold (inventory, 5)]

print ("=== Overstock, via list comprehensions ===")
print (overstock)

# Reminder: Here is a different way to implement the above
overstock2 = []
for key in keys_geq_threshold (inventory, 5):
    overstock2.append (key)

print ("\n=== Overstock, via explicit loops ===")
print (overstock2)


I am working on the file "04-Coroutines.ipynb".
The first task is:
Can a generator both produce and receive?
Yes, but it can make the behaviour of the code difficult to understand.
The example below is a natural use of taking in and returning values in one function. track_largest is a function that yields the largest value that it's seen so far. The semantics of how this works is somewhat confusing, and it would not be appropriate to use this in a for loop.
The execution pauses after a value has been produced, but before a value has been recieved. This partly explains why you must first initialize a coroutine with a call to next.
Can you write Python code for it?

def track_largest(start=0):
    largest = start
    while True:
        next_val = yield largest
        largest = next_val if next_val > largest else largest

tracker = track_largest(0)
print(next(tracker))
print(tracker.send(3))
print(tracker.send(2))
print(tracker.send(100))
print(tracker.send(9000))


I am working on the file "04-Coroutines.ipynb".
The first task is:
Sending values
When using a yield expression, the coroutine receives a value through a call to its send method. Like next, sending a value progresses to the next yield and suspends execution. In fact, given an instance of a generator, say my_gen, calling next(my_gen) is the same as my_gen.send(None).
Also note that before a value can be sent to a coroutine, it must be initialized with a call to next. This can be automated with a decorator.
Can you write Python code for it?

logs = [
    "[WARN]: A new user subscribed to cat facts!",
    "[INFO]: Todays fact: Adult cats only meow to communicate with humans.",
    "[ERR]: The user has unsubscribed from cat facts!",
    "[ERR]: 0 users are subscribed",
    "[WARN]: shutting down."
]

err_printer = print_if_error("[ERR]")
next(err_printer)

for log in logs:
    err_printer.send(log)


I am working on the file "04-Dictionaries/00-DictionaryMethod.ipynb".
The first task is:

2. Creating Dictionary Counts
I created two text files, one is a list of positive words from the MPQA dictionary, the other is a list of negative words. One word per line. Our goal here is to count the number of positive and negative words in each row of our dataframe, and add two columns to our dataset with the count of positive and negative words.
First, read in the positive and negative words and create list variables for each.
Can you write Python code for it?

pos_sent = open("../Data/positive_words.txt", encoding='utf-8').read()
neg_sent = open("../Data/negative_words.txt", encoding='utf-8').read()

#view part of the pos_sent variable, to see how it's formatted.
print(pos_sent[:101])

#remember the split function? We'll split on the newline character (\n) to create a list
positive_words=pos_sent.split('\n')
negative_words=neg_sent.split('\n')

#view the first elements in the lists
print(positive_words[:10])
print(negative_words[:10])
positive_words

#count number of words in each list
print(len(positive_words))
print(len(negative_words))


I am working on the file "04-Dictionaries/00-DictionaryMethod.ipynb".
The first task is:
Pre-processing is done. What other pre-processing steps might we use?
One more step before getting to the dictionary method. We want a total token count for each row, so we can normalize the dictionary counts. To do this we simply create a new column that contains the length of the token list in each row.
Can you write Python code for it?

df['token_count'] = df['body_tokens'].apply(lambda x: len(x))

print(df[['body_tokens','token_count']])


I am working on the file "04-Dictionaries/00-DictionaryMethod.ipynb".
The first task is:
That's the dictionary method! You can do this with any dictionary you want, standard or you can create your own.

3. Dictionary Method using Scikit-learn
We can also do this using the document term matrix. We'll again do this in pandas, to make it conceptually clear. As you get more comfortable with programming you may want to eventually shift over to working with sparse matrix format.
Can you write Python code for it?

#import the function CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer

countvec = CountVectorizer()

#create our document term matrix as a pandas dataframe
dtm_df = pandas.DataFrame(countvec.fit_transform(df.body).toarray(), columns=countvec.get_feature_names(), index = df.index)
dtm_df


I am working on the file "04-Dictionaries/00-DictionaryMethod_ExerciseSolutions.ipynb".
The first task is:
3. Dictionary Method using Scikit-learn
We can also do this using the document term matrix. We'll again do this in pandas, to make it conceptually clear. As you get more comfortable with programming you may want to eventually shift over to working with sparse matrix format.
Can you write Python code for it?

#import the function CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer

countvec = CountVectorizer()

#create our document term matrix as a pandas dataframe
dtm_df = pandas.DataFrame(countvec.fit_transform(df.body).toarray(), columns=countvec.get_feature_names(), index = df.index)


I am working on the file "04-Dictionaries/00-DictionaryMethod_ExerciseSolutions.ipynb".
The first task is:
Great! You know what to do now.
Exercise:
1. Create a column with the number of positive words, and another with the proportion of positive words
2. Create a column with the number of negative words, and another with the proportion of negative words
3. Print the average proportion of negative and positive words by genre
4. Compare this to the average score by genre
Can you write Python code for it?

#exercise code here
#1. Create a column with the number of positive words and another with the proportion of positive words
df['pos_num'] = df['body_tokens'].apply(lambda x: len([word for word in x if word in positive_words]))
df['pos_prop'] = df['pos_num']/df['token_count']

#2. Create a column with the number of negative words, and another with the proportion of negative words
df['neg_num'] = df['body_tokens'].apply(lambda x: len([word for word in x if word in negative_words]))
df['neg_prop'] = df['neg_num']/df['token_count']

df

#3. Print the average proportion of negative and positive words by genre
grouped = df.groupby('genre')
print("Averge proportion of positive words by genre")
print(grouped['pos_prop'].mean().sort_values(ascending=False))
print()
print("Averge proportion of negative words by genre")
grouped['neg_prop'].mean().sort_values(ascending=False)

# 4. Compare this to the average score by genre
print("Averge score by genre")
grouped['score'].mean().sort_values(ascending=False)


I am working on the file "04-Dictionaries/00-DictionaryMethod_ExerciseSolutions.ipynb".
The first task is:
Now we can keep only those columns that occur in our positive words list. To do this, we'll first save a list of the columns names as a variable, and then only keep the elements of the list that occur in our positive words list. We'll then create a new dataframe keeping only those select columns.
Can you write Python code for it?

#create a columns variable that is a list of all column names
columns = list(dtm_df)
pos_columns = [word for word in columns if word in positive_words]

#create a dtm from our dtm_df that keeps only positive sentiment columns
dtm_pos = dtm_df[pos_columns]

#count the number of positive words for each document
dtm_pos['pos_count'] = dtm_pos.sum(axis=1)
#dtm_pos.drop('pos_count',axis=1, inplace=True)
dtm_pos['pos_count']


I am working on the file "04-Dictionaries/00.1-DictionaryMethod_AdditionalExercises.ipynb".
The first task is:
Question 2
[your hypothesis here]
Question 3
Can you write Python code for it?


I am working on the file "04-Dictionaries/00.1-DictionaryMethod_AdditionalExercises_Solutions.ipynb".
The first task is:
Question 2
Since these literatures are written for children, the overall sentiment rating is probably positive.
Question 3
Can you write Python code for it?

# Your code here
df['text_lc'] = df['text'].str.lower()
df['text_split'] = df['text_lc'].apply(nltk.word_tokenize)
df['text_split_clean'] = df['text_split'].apply(lambda x : [word for word in x if word not in string.punctuation])
df

df['text_length'] = df['text_split_clean'].apply(len)
df

# Your code here
pos_sent = open("../Data/positive_words.txt", encoding='utf-8').read()
neg_sent = open("../Data/negative_words.txt", encoding='utf-8').read()
positive_words = pos_sent.split('\n')
negative_words = neg_sent.split('\n')

df['num_pos_words'] = df['text_split_clean'].apply(lambda x: len([word for word in x if word in positive_words]))
df['num_neg_words'] = df['text_split_clean'].apply(lambda x: len([word for word in x if word in negative_words]))
df

df['prop_pos_words'] = df['num_pos_words']/df['text_length']
df['prop_neg_words'] = df['num_neg_words']/df['text_length']
df


I am working on the file "04-Dictionaries/00.1-DictionaryMethod_AdditionalExercises_Solutions.ipynb".
The first task is:
Since the number of children literaturs is a lot to analyze, we'll just randomly select 5 books to do a sentiment analysis using the dictionary method.
Note: In case you're not familiar with seed. Seed is just a function that initializes a fixed state for random number generatoring. Basically if everyone uses the same number as an input to seed(), then everyone will get the same result when generating randomly.
Can you write Python code for it?

import numpy as np
np.random.seed(1)
df = df.sample(5)
df


I am working on the file "04-Dictionaries/01-ChiSquared.ipynb".
The first task is:
1. Chi-Squared for sample words
The chi-sqaured statistic is a simple calculation:

Let's look at the frequency for two words, "sister" and "child".
Can you write Python code for it?


I am working on the file "04-Dictionaries/01-ChiSquared.ipynb".
The first task is:
3. Chi-Squared for every word, using a for-loop
Now we can calculate this for each word in our corpus. To do this we have to introduce the for loop. We've seen this before in list comprehension, but we're splitting it out now into multiple lines. To think this intuitively, take this example:
For every child that knocks on my door on Halloween I will do the following:
1. Ask them what their costume is
2. Give them a piece of candy
3. Cackle wildly
The for loop in Python is intuitively the same. For every element in a list, we want to do something to that element.
In this case, we will loop through all columns in our dataframe and calculate the chi-squared statistic. We will then append both the column name (our word) and the chi-squared statistic to a list using .append().
Can you write Python code for it?

columns = list(novels_df)
chi_list = []

for c in columns:
    chi_list.append([c,((novels_df.loc[0,c] - novels_df[c].sum(axis=0)/2)**2 / novels_df[c].sum(axis=0)/2) + ((novels_df.loc[1,c] - novels_df[c].sum(axis=0)/2)**2 / novels_df[c].sum(axis=0)/2)])

chi_list[:10]


I am working on the file "04-Dictionaries/01-ChiSquared.ipynb".
The first task is:
Exercise:
Calculate the partisan score for each word in the corpus and print the most partisan words for each novel.
Can you write Python code for it?


I am working on the file "04-Dictionaries/01-ChiSquared.ipynb".
The first task is:
These are large values. Let's try a word that has a much closer frequency across both novels. The result is a much smaller chi-squared statistic.
Can you write Python code for it?

novels_df['writing']

expected_writes = novels_df['writing'].sum(axis=0)/2
chi_writes = ((novels_df.loc[0,'writing'] - expected_writes)**2 / expected_writes) + ((novels_df.loc[1,'writing'] - expected_writes)**2 / expected_writes)
chi_writes


I am working on the file "04-Dictionaries/01-ChiSquared.ipynb".
The first task is:
To calculate the chi_squares we subtract the expected frequency for each novel from the actual frequency for each novel, square this value, and divide by the expected frequency, and add the two numbers together.
Can you write Python code for it?

chi_sister = ((novels_df.loc[0,'sister'] - expected_sister)**2 / expected_sister) + ((novels_df.loc[1,'sister'] - expected_sister)**2 / expected_sister)
chi_sister

chi_child = ((novels_df.loc[0,'child'] - expected_child)**2 / expected_child) + ((novels_df.loc[1,'child'] - expected_child)**2 / expected_child)
chi_child


I am working on the file "04-Dictionaries/01-ChiSquared.ipynb".
The first task is:
We can now sort this list by the second element in each tuple (it's not technically a tuple, but no matter), and print the top 50 "partisan" words.
Can you write Python code for it?

chi_list.sort(key=lambda x: x[1], reverse=True)
chi_list[:50]


I am working on the file "04-Dictionaries/01-ChiSquared.ipynb".
The first task is:
What does a partisan score of 0 mean?
Can you write Python code for it?


I am working on the file "04-Dictionaries/01-ChiSquared_ExerciseSolutions.ipynb".
The first task is:
1. Chi-Squared for sample words
The chi-sqaured statistic is a simple calculation:

Let's look at the frequency for two words, "sister" and "child".
Can you write Python code for it?


I am working on the file "04-Dictionaries/01-ChiSquared_ExerciseSolutions.ipynb".
The first task is:
2. Partisan Score
Next, we can find the partisan score for our chosen words. We do this simply, by multiplying the word frequency in Pride and Prejudice by 1, and multiple the word frequency in Garland for Girls by -1, and adding these together. A partisan score above 0 will indicate it's used more often in Austen, a negative score will mean it is used more often in Alcott.
Can you write Python code for it?

sister_corr = novels_df.loc[0,'sister']*1 + novels_df.loc[1,'sister']*-1
sister_corr

child_corr = (novels_df.loc[0,'child']*1) + (novels_df.loc[1,'child']*-1)
child_corr

writing_corr = (novels_df.loc[0,'writing']*1) + (novels_df.loc[1,'writing']*-1)
writing_corr

writes_corr = (novels_df.loc[0,'writes']*1) + (novels_df.loc[1,'writes']*-1)
writes_corr


I am working on the file "04-Dictionaries/01-ChiSquared_ExerciseSolutions.ipynb".
The first task is:
3. Chi-Squared for every word, using a for-loop
Now we can calculate this for each word in our corpus. To do this we have to introduce the for loop. We've seen this before in list comprehension, but we're splitting it out now into multiple lines. To think this intuitively, take this example:
For every child that knocks on my door on Halloween I will do the following:
1. Ask them what their costume is
2. Give them a piece of candy
3. Cackle wildly
The for loop in Python is intuitively the same. For every element in a list, we want to do something to that element.
In this case, we will loop through all columns in our dataframe and calculate the chi-squared statistic. We will then append both the column name (our word) and the chi-squared statistic to a list using .append().
Can you write Python code for it?


I am working on the file "04-Dictionaries/01-ChiSquared_ExerciseSolutions.ipynb".
The first task is:
Exercise:
Calculate the partisan score for each word in the corpus and print the most partisan words for each novel.
Can you write Python code for it?

columns = list(novels_df)
pearson_list = []

for c in columns:
    pearson_list.append([c,((novels_df.loc[0,c]*1) + (novels_df.loc[1,c]*-1))])

#Words most associated with Jane Austen
pearson_list.sort(key=lambda x: x[1], reverse=True)
pearson_list[:50]

#Words most associated with Alcott
pearson_list.sort(key=lambda x: x[1], reverse=False)
pearson_list[:50]


I am working on the file "04-Dictionaries/01-ChiSquared_ExerciseSolutions.ipynb".
The first task is:
To calculate the chi-squared statistic for these two words we need to know the expected frequency, if these two words were used the same in both novels. To do this, we divide the total frequency across both novels by two.
Can you write Python code for it?

expected_sister = novels_df['sister'].sum(axis=0)/2
expected_sister

expected_child = novels_df['child'].sum(axis=0)/2
expected_child


I am working on the file "04-Dictionaries/01-ChiSquared_ExerciseSolutions.ipynb".
The first task is:
To calculate the chi_squares we subtract the expected frequency for each novel from the actual frequency for each novel, square this value, and divide by the expected frequency, and add the two numbers together.
Can you write Python code for it?


I am working on the file "04-Dictionaries/01-ChiSquared_ExerciseSolutions.ipynb".
The first task is:
We can now sort this list by the second element in each tuple (it's not technically a tuple, but no matter), and print the top 50 "partisan" words.
Can you write Python code for it?


I am working on the file "04-Dictionaries/01-ChiSquared_ExerciseSolutions.ipynb".
The first task is:
What does a partisan score of 0 mean?
Can you write Python code for it?


I am working on the file "04-Discriminating-Words/DTM_and_Discriminating_Words.ipynb".
The first task is:
2. Creating the DTM: scikit-learn
Ok, that's the summary of the metadata. Next, we turn to analyzing the text of the reviews. Remember, the text is stored in the 'body' column.
Our first step is to turn the text into a document term matrix (DTM). A DTM is a different way of representing text called vector representation. The goal is to turn the text into numbers. To do so, we transform each document into a vector, where each number in the vector represents the count of a particular word. To get a feel for this we'll jump right into an example.
We've learned how to count each word using Python's NLTK. We could, then, construct a DTM manually. Luckily, however, Python has a great package, scikit-learn with a built-in function to do this called CountVectorizer().
Let's first look at the documentation for CountVectorizer.
We'll implement this by creating a CountVectorizer() object.
Can you write Python code for it?

#import the function CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer

countvec = CountVectorizer()

#fit and transform our text into a DTM. Ask me about what this code does...
sklearn_dtm = countvec.fit_transform(df.body)
print(sklearn_dtm)


I am working on the file "04-Discriminating-Words/DTM_and_Discriminating_Words.ipynb".
The first task is:
How is this output different that the previous time we used the DataFrame.mean() function?
Can you write Python code for it?

##EX: Print the maximum score for each genre.

##Bonus EX: Find the artist with the highest average score. Find the artist with the lowest average score.


I am working on the file "04-Discriminating-Words/DTM_and_Discriminating_Words.ipynb".
The first task is:
It save a lot of memory to store the dtm in this format, but it is difficult to look at for a human. To illustrate the techniques in this lesson we will first convert this matrix back to a Pandas dataframe, a format we're more familiar with. For larger datasets, you will have to use the Compressed Sparse Format. Putting it into a DataFrame, however, will enable us to get more comfortable with Pandas!
Can you write Python code for it?

#we do the same as we did above, but covert it into a Pandas dataframe
#Don't worry about understanding every line of this code
dtm_df = pandas.DataFrame(countvec.fit_transform(df.body).toarray(), columns=countvec.get_feature_names(), index = df.index)

#view the dtm dataframe
dtm_df


I am working on the file "04-Discriminating-Words/DTM_and_Discriminating_Words.ipynb".
The first task is:
Ok! We have successfully identified content words, without removing stop words. What else do you notice about this list?
5. Identifying Distinctive Words
What can we do with this? These scores are best used when you want to identify distinctive words for individual documents, or groups of documents, compared to other groups or the corpus as a whole. To illustrate this, let's compare three genres and identify the most distinctive words by genre.
First we add the genre of the document into our dtm weighted by tf-idf scores.
Why can we do this? Remember dataframes are immutable. What does this mean, and why does this ensure the below code works?
Can you write Python code for it?

#creat dataset with document index and genre

#firs make a copy of our dtm_tfidf_df
dtm_tfidf_df_genre = dtm_tfidf_df

#add a GENRE column to it. Why am I making the name of the column GENRE and no genre?
dtm_tfidf_df_genre['GENRE'] = df['genre']
dtm_tfidf_df_genre


I am working on the file "04-Discriminating-Words/DTM_and_Discriminating_Words.ipynb".
The first task is:
This format is called Compressed Sparse Format. How do we know what each number indicates? We can access the words themselves through the CountVectorizer function get_feature_names.
Can you write Python code for it?

print(countvec.get_feature_names()[:10])

##EX: What word is indicated by the first row of the DTM printed above?
###Hint: Think back to the tutorial on lists, and how to slice a list.


I am working on the file "04-Discriminating-Words/DTM_and_Discriminating_Words.ipynb".
The first task is:
We now have a pandas object. We can perform most of the in-built Pandas functions on this object, but we'll see slightly different output than what we saw with a dataframe object.
Let's find the average score by genre by calling the DataFrame.mean() function on our grouped by object. We'll discuss the different options we can use.
Can you write Python code for it?

#calculate the mean score by genre, print out the results
df_genres['score'].mean().sort_values(ascending=False)


I am working on the file "04-Discriminating-Words/DTM_and_Discriminating_Words.ipynb".
The first task is:
What about the other columns? The columns that contain strings, not numbers? One way we can summarize these columns is by counting the unique string values in the column. We can do this using the DataFrame.value_counts() function.
Let's see what the different genres in our dataframe are:
Can you write Python code for it?

df['genre'].value_counts()

##EX: Print the most frequent reviewers and artists in the dataframe.


I am working on the file "04-Discriminating-Words/DTM_and_Discriminating_Words.ipynb".
The first task is:
You can see that this provides summary statistics for numerical columns. In our case, our only numerical column is score. We can access each of these summaries statistics separately, using Pandas built-in functions. For example, we can get the max score using the syntax DataFrame[Column Name].max()
Can you write Python code for it?

df['score'].max()

##EX: Print the mean and the standard deviation for the score column.


I am working on the file "04-Exemplo - Análise de sobreviventes do Titanic.ipynb".
The first task is:
A informação acima mostra que esse dataset contém informações sobre 891 passageiros: seus nome, gênero, idade, etc (para uma descrição completa do significado de cada coluna, confira este link).
Valores em branco
Antes de iniciar a análise em si, precisamos checar como está a "saúde" dos dados verificando quanta informação está presente de fato em cada coluna.
Can you write Python code for it?

# POrcentagem de valores em branco em cada coluna
(raw_data.isnull().sum() / len(raw_data)) * 100.0


I am working on the file "04-Exemplo - Análise de sobreviventes do Titanic.ipynb".
The first task is:
Com uma simples árvore de decisão, o resultado cima indica que seria possível prever corretamente a sobrevivência de cerca de 80% dos passageiros.
Um exercício interessante de se fazer após treinar um modelo é dar uma olhada nos casos em que ele erra:
Can you write Python code for it?

test_data.is_copy = False
test_data['Predicted'] = tree.predict(test_data[['Sex', 'Age', 'Fare']])
test_data[test_data.Predicted != test_data.Survived]


I am working on the file "04-Exemplo - Análise de sobreviventes do Titanic.ipynb".
The first task is:
Finalmente, cerca de 20% dos passageiros não possuem informação de idade. Não parece razoável exluir todos eles e tampouco descartar a coluna inteira, então uma solução possível é preenchermos os valores em branco dessa coluna com o valor mediano dela no dataset:
Can you write Python code for it?

raw_data.fillna({'Age': raw_data.Age.median()}, inplace=True)
(raw_data.isnull().sum() / len(raw_data)) * 100.0


I am working on the file "04-Exemplo - Análise de sobreviventes do Titanic.ipynb".
The first task is:
No geral, 38% dos passageiros sobreviveram.
Vamos agora segmentar a proporção de pessoas que sobreviveram ao longo de diferentes recortes (o código usado para gerar os gráficos abaixo foram retirados desse link).
Recorte de gênero
Can you write Python code for it?

survived_sex = raw_data[raw_data['Survived']==1]['Sex'].value_counts()
dead_sex = raw_data[raw_data['Survived']==0]['Sex'].value_counts()
df = pd.DataFrame([survived_sex,dead_sex])
df.index = ['Survivors','Non-survivors']
df.plot(kind='bar',stacked=True, figsize=(15,8));


I am working on the file "04-Exemplo - Análise de sobreviventes do Titanic.ipynb".
The first task is:
Pode-se ver que 77% dos passageiros não apresentam informação sobre em qual cabine eles estavam alojados. Essa informação pode ser útil para análise posterior mas, por enquanto, vamos descartar essa coluna:
Can you write Python code for it?

raw_data.drop('Cabin', axis='columns', inplace=True)
raw_data.info()


I am working on the file "04-Exemplo - Análise de sobreviventes do Titanic.ipynb".
The first task is:
Previsão de chances de sobrevivência usando árvores de decisão
Usaremos um simples modelo de Árvore de Decisão para prever se um passageiro teria sobrevivido ao Titanic usando seu gênero, idade e preço de passagem.
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier().fit(train_data[['Sex', 'Age', 'Fare']], train_data.Survived)
tree.score(test_data[['Sex', 'Age', 'Fare']], test_data.Survived)


I am working on the file "04-Exemplo - Análise de sobreviventes do Titanic.ipynb".
The first task is:
Recorte de idade
Can you write Python code for it?

figure = plt.figure(figsize=(15,8))
plt.hist([raw_data[raw_data['Survived']==1]['Age'], raw_data[raw_data['Survived']==0]['Age']],
         stacked=True, color=['g','r'],
         bins=30, label=['Survivors','Non-survivors'])
plt.xlabel('Idade')
plt.ylabel('No. passengers')
plt.legend();


I am working on the file "04-Funkcije.ipynb".
The first task is:
Funkcija u Pajtonu je blok organizovanog koda koji izvršava određenu akciju i može se koristiti na više mesta bez ponavljanja čitavog bloka, referenciranjem imena funkcije.
Pajton sadrži "ugrađene" funkcije, poput:

* print - funkcija za prikaz teksta na ekranu
* input - služi za čitanje korisničkog unosa s tastature
* int, float - služe za konverziju u celi i razlomljeni brojni tip, respektivno
Pored "ugrađenih" funkcija, koje su nam na raspolaganju, Pajton nam omogućava da definišemo i svoje funkcije. Napisaćemo funkciju koja prilikom svakog poziva ispisuje nekoliko redova teksta, i pozvati je nekoliko puta. Za definisanje funkcije koristi se ključna reč def, praćena imenom funkcije, zagradama (u kojima se mogu naći parametri funkcije) i znakom dve tačke (:), koji je praćen telom funkcije. Telo funkcije je kod koji će se izvršiti prilikom svakog njenog poziva. Sav kod koji čini telo funkcije mora se naći u novom redu nakon dve tačke, uvučen za jedan tabulator (TAB). Kada hoćemo da završimo telo funkcije i nastavimo da unosimo kod programa (u kome možemo pozvati, odnosno izvršiti svoju funkciju), vratićemo se nivo unazad (tekst neće više biti uvučen).
Can you write Python code for it?

# A simple function that just prints a few lines of text
def print_some_text():
    print("Hello!")
    print("This is my function.")
    print("It just prints out a few lines of text.")
    print("Good bye!")

# A test for our function
print("See my function in action!\n\n")
print_some_text()
print_some_text()
print_some_text()


I am working on the file "04-Funkcije.ipynb".
The first task is:
Funkcije u Pajtonu mogu imati argumente. Oni se u definiciji nalaze između zagrada, odvojeni zapetama ukoliko ih ima više od jednog. U telu funkcije se koriste kao promenljive, s tim što imaju vrednost koja im je prosleđena prilikom poziva funkcije.
Napisaćemo funkciju koja ima dva argumenta - ime i prezime. Zadatak funkcije će biti da pozdravi korisnika po imenu i prezimenu, koristeći formatiranje stringova. Ime i prezime ćemo u programu učitati s tastature. Pokrenite program više puta i unesite različite vrednosti imena i prezimena.
Can you write Python code for it?

# A function that greets the user, mentioning their first and last name
def greet_user(firstname, lastname):
    greeting = "Hello, %s %s! How are you?" % (firstname, lastname)
    print(greeting)

# Read user's first and last name
# No need to convert the input, since the input function returns a string
first = input("What is your first name?\n")
last = input("What is your last name?\n")

# Call our greeting function
greet_user(first, last)


I am working on the file "04-Funkcije.ipynb".
The first task is:
Kao što mogu imati argumente (ulazne vrednosti), funkcije mogu imate i povratne vrednosti (rezultat). Napisaćemo funkciju za obim kruga i funkciju za površinu kruga. Za povratne vrednosti, koristi se ključna reč return. Iza ključne reči return može se naći literal, promenljiva, izraz...
Can you write Python code for it?

# The pi constant
pi = 3.14

# Circle circumference
def circle_circumference(radius):
    circumference = 2 * radius * pi
    return circumference

# Circle area
def circle_area(radius):
    area = radius * radius * pi
    return area

# Get the radius from user and calculate the circumference and the area
r = float(input("radius = ")) # we need to convert the input (a string) to float
C = circle_circumference(r)
A = circle_area(r)

print("Circumference = %f" % C)
print("Area = %f" % A)


I am working on the file "04-Funkcije.ipynb".
The first task is:
Na početku ove lekcije, napisali smo program koji kao ulaz uzima poluprečnik kruga, a zatim računa obim i površinu tog kruga i kruga čiji je poluprečnik za 1 veći. Napišimo isti program, bez ponavljanja obrazaca za obim i površinu (bez duplikacije tih delova koda), koristeći funkcije.
Can you write Python code for it?

# The pi constant
pi = 3.14

# Circle circumference
def circle_circumference(radius):
    circumference = 2 * radius * pi
    return circumference

# Circle area
def circle_area(radius):
    area = radius * radius * pi
    return area

# Read the radius
radius = float(input("radius = "))
radius_large = radius + 1

# "Smaller" circle circumference and area
circumference = circle_circumference(radius)
area = circle_area(radius)

# "Larger" circle circumference and area
circumference_large = circle_circumference(radius_large)
area_large = circle_area(radius_large)

# Print the values
print("The smaller circle circumference is %f and area is %f." % (circumference, area))
print("The larger circle circumference is %f and area is %f." % (circumference_large, area_large))


I am working on the file "04-Funkcije.ipynb".
The first task is:
Ukoliko rezultat funkcije nije jedna, već više od jedne vrednosti, koristimo sličan koncept kao kod formatiranja stringa, kada u string umećemo više vrednosti. Koristi se poseban tip podataka u Pajtonu - uređena N-torka (<i>tuple</i>). Napisaćemo funkciju koja računa kvadrat i treći stepen broja.
Can you write Python code for it?

def square_and_cube(a):
    square = a * a
    cube = square * a
    return (square, cube)

# Test the function
(square, cube) = square_and_cube(10)
print("square: %d" % square)
print("cube: %d" % cube)


I am working on the file "04-Graph.ipynb".
The first task is:
2個のサイコロを100回振った結果を保存
Can you write Python code for it?


I am working on the file "04-Graph.ipynb".
The first task is:
2個の目の合計を計算
Can you write Python code for it?


I am working on the file "04-Graph.ipynb".
The first task is:
計算結果をグラフに表示
Can you write Python code for it?

fig = plt.figure(figsize=(14,4))
subplot = fig.add_subplot(1,3,1)
subplot.set_title('Sum of 2dices')
subplot.set_xlabel('Total')
subplot.set_ylabel('Count')
subplot.set_xlim(1,13)
subplot.hist(total, bins=11, range=(1.5, 12.5), label='Sum')

subplot = fig.add_subplot(1,3,2)
subplot.set_title('Doublets counts')
subplot.set_xlabel('Number')
subplot.set_ylabel('Count')
subplot.set_xlim(0.5, 6.5)
subplot.bar(range(1,7), doublets, align='center')

subplot = fig.add_subplot(1,3,3)
subplot.set_title('Pair counts')
subplot.set_xlabel('Dice1')
subplot.set_ylabel('Dice2')
subplot.imshow(counts,
    origin='lower', extent=(0.5,6.5,0.5,6.5),
    interpolation='nearest')

from numpy.random import normal

def generate_data01(n):
    data_x = []
    data_y = []
    for i in range(n):
        x = float(i) / float(n-1) # [0, 1]をn等分したi番目の値
        y = np.sin(2*np.pi*x) + normal(0, 0.3)
        data_x.append(x)
        data_y.append(y)
    return data_x, data_y

def generate_data02(n):
    data_x = np.linspace(0,1,n)
    data_y = np.sin(2*np.pi*data_x) + normal(0, 0.3, n)
    return data_x, data_y

fig = plt.figure()

data_x, data_y = generate_data01(10)
#data_x, data_y = generate_data02(10)

subplot = fig.add_subplot(1,1,1)
subplot.set_xlabel('Observation point')
subplot.set_ylabel('Value')
subplot.set_xlim(-0.05,1.05)

# 生成したデータを表示
subplot.scatter(data_x, data_y,
                marker='o', color='blue',
                label='Observed value')

# 三角関数の曲線を表示
linex = np.linspace(0,1,100)
liney = np.sin(2*np.pi*linex)
subplot.plot(linex, liney,
             linestyle='--', color='green',
             label='Theoretical curve')

# 凡例を表示
subplot.legend(loc=1)


I am working on the file "04-InitSeqtoSeq/dlnd_language_translation.ipynb".
The first task is:
Build the Graph
Build the graph using the neural network you implemented.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_path = 'checkpoints/dev'
(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()
max_source_sentence_length = max([len(sentence) for sentence in source_int_text])

train_graph = tf.Graph()
with train_graph.as_default():
    input_data, targets, lr, keep_prob = model_inputs()
    sequence_length = tf.placeholder_with_default(max_source_sentence_length, None, name='sequence_length')
    input_shape = tf.shape(input_data)

    train_logits, inference_logits = seq2seq_model(
        tf.reverse(input_data, [-1]), targets, keep_prob, batch_size, sequence_length, len(source_vocab_to_int), len(target_vocab_to_int),
        encoding_embedding_size, decoding_embedding_size, rnn_size, num_layers, target_vocab_to_int)

    tf.identity(inference_logits, 'logits')
    with tf.name_scope("optimization"):
        # Loss function
        cost = tf.contrib.seq2seq.sequence_loss(
            train_logits,
            targets,
            tf.ones([input_shape[0], sequence_length]))

        # Optimizer
        optimizer = tf.train.AdamOptimizer(lr)

        # Gradient Clipping
        gradients = optimizer.compute_gradients(cost)
        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]
        train_op = optimizer.apply_gradients(capped_gradients)


I am working on the file "04-InitSeqtoSeq/dlnd_language_translation.ipynb".
The first task is:
Build the Neural Network
Apply the functions you implemented above to:

Apply embedding to the input data for the encoder.
Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob).
Process target data using your process_decoding_input(target_data, target_vocab_to_int, batch_size) function.
Apply embedding to the target data for the decoder.
Decode the encoded input using your decoding_layer(dec_embed_input, dec_embeddings, encoder_state, vocab_size, sequence_length, rnn_size, num_layers, target_vocab_to_int, keep_prob).
Can you write Python code for it?

# ?decoding_layer

def seq2seq_model(input_data, target_data, keep_prob, batch_size, sequence_length, source_vocab_size, target_vocab_size,
                  enc_embedding_size, dec_embedding_size, rnn_size, num_layers, target_vocab_to_int):
    """
    Build the Sequence-to-Sequence part of the neural network
    :param input_data: Input placeholder
    :param target_data: Target placeholder
    :param keep_prob: Dropout keep probability placeholder
    :param batch_size: Batch Size
    :param sequence_length: Sequence Length
    :param source_vocab_size: Source vocabulary size
    :param target_vocab_size: Target vocabulary size
    :param enc_embedding_size: Decoder embedding size
    :param dec_embedding_size: Encoder embedding size
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: Tuple of (Training Logits, Inference Logits)
    """

    input_embedding = tf.contrib.layers.embed_sequence(input_data, source_vocab_size, enc_embedding_size)
    encoder_state = encoding_layer(keep_prob=keep_prob, num_layers=num_layers, rnn_inputs=input_embedding, rnn_size=rnn_size)
    processed_target_data = process_decoding_input(target_data, target_vocab_to_int, batch_size)
#     target_embedding = tf.contrib.layers.embed_sequence(target_data, target_vocab_size, dec_embedding_size)
    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, dec_embedding_size]))
    target_embedding = tf.nn.embedding_lookup(dec_embeddings, processed_target_data)
    layer = decoding_layer(target_embedding, dec_embeddings, encoder_state, target_vocab_size, sequence_length, rnn_size, \
                   num_layers, target_vocab_to_int, keep_prob)
    return layer

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_seq2seq_model(seq2seq_model)


I am working on the file "04-InitSeqtoSeq/dlnd_language_translation.ipynb".
The first task is:
Decoding - Inference
Create inference logits using tf.contrib.seq2seq.simple_decoder_fn_inference() and tf.contrib.seq2seq.dynamic_rnn_decoder().
Can you write Python code for it?

def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id,
                         maximum_length, vocab_size, decoding_scope, output_fn, keep_prob):
    """
    Create a decoding layer for inference
    :param encoder_state: Encoder state
    :param dec_cell: Decoder RNN Cell
    :param dec_embeddings: Decoder embeddings
    :param start_of_sequence_id: GO ID
    :param end_of_sequence_id: EOS Id
    :param maximum_length: The maximum allowed time steps to decode
    :param vocab_size: Size of vocabulary
    :param decoding_scope: TensorFlow Variable Scope for decoding
    :param output_fn: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: Inference Logits
    """
    dynamic_fn_inference = tf.contrib.seq2seq.simple_decoder_fn_inference(encoder_state=encoder_state,
                                                                          embeddings=dec_embeddings,
                                                                          start_of_sequence_id = start_of_sequence_id,
                                                                          end_of_sequence_id = end_of_sequence_id,
                                                                          maximum_length = maximum_length,
                                                                          output_fn = output_fn,
                                                                          num_decoder_symbols = vocab_size,
                                                                          dtype=tf.int32, name=None)
    inference_logits, final_state, final_context_state = tf.contrib.seq2seq.dynamic_rnn_decoder(decoder_fn=dynamic_fn_inference, cell=dec_cell, scope=decoding_scope)
    return inference_logits

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_infer(decoding_layer_infer)


I am working on the file "04-InitSeqtoSeq/dlnd_language_translation.ipynb".
The first task is:
Decoding - Training
Create training logits using tf.contrib.seq2seq.simple_decoder_fn_train() and tf.contrib.seq2seq.dynamic_rnn_decoder().  Apply the output_fn to the tf.contrib.seq2seq.dynamic_rnn_decoder() outputs.
Can you write Python code for it?

def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length, decoding_scope,
                         output_fn, keep_prob):
    """
    Create a decoding layer for training
    :param encoder_state: Encoder State
    :param dec_cell: Decoder RNN Cell
    :param dec_embed_input: Decoder embedded input
    :param sequence_length: Sequence Length
    :param decoding_scope: TenorFlow Variable Scope for decoding
    :param output_fn: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: Train Logits
    """
    dynamic_fn_train = tf.contrib.seq2seq.simple_decoder_fn_train(encoder_state)
    rnn_outputs, final_state, final_context_state = tf.contrib.seq2seq.dynamic_rnn_decoder(
        decoder_fn=dynamic_fn_train,
        cell=dec_cell,
        inputs=dec_embed_input,
        sequence_length=sequence_length,
        scope=decoding_scope,
    )

    train_logits =  output_fn(rnn_outputs)
    train_logits = tf.nn.dropout(train_logits, keep_prob)
    return train_logits

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_train(decoding_layer_train)


I am working on the file "04-InitSeqtoSeq/dlnd_language_translation.ipynb".
The first task is:
Encoding
Implement encoding_layer() to create a Encoder RNN layer using tf.nn.dynamic_rnn().
Can you write Python code for it?

def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob):
    """
    Create encoding layer
    :param rnn_inputs: Inputs for the RNN
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param keep_prob: Dropout keep probability
    :return: RNN state
    """
    rnn = tf.contrib.rnn.BasicLSTMCell(rnn_size)
    cell = tf.contrib.rnn.MultiRNNCell([rnn] * num_layers)
    layer = tf.contrib.rnn.DropoutWrapper(cell, output_keep_prob=keep_prob)
    encoding_layer, rnn_state = tf.nn.dynamic_rnn(layer, rnn_inputs, dtype=tf.float32)
    return rnn_state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_encoding_layer(encoding_layer)


I am working on the file "04-InitSeqtoSeq/dlnd_language_translation.ipynb".
The first task is:
Explore the Data
Play around with view_sentence_range to view different parts of the data.
Can you write Python code for it?

view_sentence_range = (0, 10)

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np

print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))

sentences = source_text.split('\n')
word_counts = [len(sentence.split()) for sentence in sentences]
print('Number of sentences: {}'.format(len(sentences)))
print('Average number of words in a sentence: {}'.format(np.average(word_counts)))

print()
print('English sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(source_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
print()
print('French sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(target_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))


I am working on the file "04-InitSeqtoSeq/dlnd_language_translation.ipynb".
The first task is:
Neural Network Training
Hyperparameters
Tune the following parameters:

Set epochs to the number of epochs.
Set batch_size to the batch size.
Set rnn_size to the size of the RNNs.
Set num_layers to the number of layers.
Set encoding_embedding_size to the size of the embedding for the encoder.
Set decoding_embedding_size to the size of the embedding for the decoder.
Set learning_rate to the learning rate.
Set keep_probability to the Dropout keep probability
Can you write Python code for it?

# Number of Epochs
epochs = 3
# Batch Size
batch_size = 256
# RNN Size
rnn_size = 512
# Number of Layers
num_layers = 2
# Embedding Size
encoding_embedding_size = 256
decoding_embedding_size = 256
# Learning Rate
learning_rate = 0.0011
# Dropout Keep Probability
keep_probability = 0.5
# Network parameters are from in www.alivelearn.net/wp-content/uploads/2017/04/dlnd_language_translation.html


I am working on the file "04-InitSeqtoSeq/dlnd_language_translation.ipynb".
The first task is:
Process Decoding Input
Implement process_decoding_input using TensorFlow to remove the last word id from each batch in target_data and concat the GO ID to the beginning of each batch.
Can you write Python code for it?

def process_decoding_input(target_data, target_vocab_to_int, batch_size):
    """
    Preprocess target data for dencoding
    :param target_data: Target Placehoder
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param batch_size: Batch Size
    :return: Preprocessed target data
    """

    '''This function is from http://www.alivelearn.net/wp-content/uploads/2017/04/dlnd_language_translation.html'''
    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])
    target_data = tf.concat([tf.fill([batch_size, 1], target_vocab_to_int['<GO>']), ending], 1)
    return target_data

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_process_decoding_input(process_decoding_input)


I am working on the file "04-InitSeqtoSeq/dlnd_language_translation.ipynb".
The first task is:
Sentence to Sequence
To feed a sentence into the model for translation, you first need to preprocess it.  Implement the function sentence_to_seq() to preprocess new sentences.

Convert the sentence to lowercase
Convert words into ids using vocab_to_int
Convert words not in the vocabulary, to the <UNK> word id.
Can you write Python code for it?

def sentence_to_seq(sentence, vocab_to_int):
    """
    Convert a sentence to a sequence of ids
    :param sentence: String
    :param vocab_to_int: Dictionary to go from the words to an id
    :return: List of word ids
    """
    sentence = sentence.lower()
    words = sentence.split(' ')
    ids = []
    for w in words:
        ids.append(vocab_to_int.get(w, vocab_to_int.get('<UNK>')))
    return ids

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_sentence_to_seq(sentence_to_seq)


I am working on the file "04-InitSeqtoSeq/dlnd_language_translation.ipynb".
The first task is:
Translate
This will translate translate_sentence from English to French.
Can you write Python code for it?

translate_sentence = 'he saw a old yellow truck .'

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)

loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_path + '.meta')
    loader.restore(sess, load_path)

    input_data = loaded_graph.get_tensor_by_name('input:0')
    logits = loaded_graph.get_tensor_by_name('logits:0')
    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')

    translate_logits = sess.run(logits, {input_data: [translate_sentence], keep_prob: 1.0})[0]

print('Input')
print('  Word Ids:      {}'.format([i for i in translate_sentence]))
print('  English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))

print('\nPrediction')
print('  Word Ids:      {}'.format([i for i in np.argmax(translate_logits, 1)]))
print('  French Words: {}'.format([target_int_to_vocab[i] for i in np.argmax(translate_logits, 1)]))


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Compare the projected and geometric solid angles. Note that directly under the reference point the geometric and projected solid angles are equal (because $\cos(0)=1$), but at the corners the cosine weighting function starts to weigh the projected solid angle down.
Can you write Python code for it?

p = ryplot.Plotter(1,1,1,figsize=(5,5))
p.mesh3D(1, x, y, 100 * (prjsolid - geoSolid) / prjsolid ,
         ptitle='Normalised difference between\nprojected and geometric solid angle',
         xlabel='x', ylabel='y', zlabel='% Difference',
         maxNX=3, maxNY=3, maxNZ=4, alpha=0.5, xInvert=True, yInvert=True);


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Converting spectral density values are similar to converting between spectral domains. In this example the pyradi.ryplanck.planck function
is used to calculate the spectral density of Planck radiance (Sec 3.1, Sec D.4.1) for a temperature of 1000 K as a wavelength spectral density. This value is then converted to a wavenumber spectral density. Finally, the Planck radiance is calculated in the wavenumber domain, which should give the same result as the converted spectral radiance.
Can you write Python code for it?

import pyradi.ryplanck as ryplanck

wl = np.linspace(1, 14, 101)#.reshape(-1,1) # wavelength
wn = ryutils.convertSpectralDomain(wl, type='ln') # wavenumber

radiancewl = ryplanck.planck(wl,1000.0, 'el') / np.pi # W/(m2.sr.um)
_,radiancewn1 = ryutils.convertSpectralDensity(wl,radiancewl, 'ln')
radiancewn2 = ryplanck.planck(wn,1000, 'en') / np.pi

p = ryplot.Plotter(2,1,3,figsize=(12,4))
p.plot(1,wl, radiancewl,'Planck radiance','Wavelength $\mu$m',
       'Radiance W/(m$^2$.sr.$\mu$m)',maxNX=4)
p.plot(2,wn, radiancewn1,'Planck radiance','Wavenumber cm$^{-1}$',
       'Radiance W/(m$^2$.sr.cm$^{-1}$)',maxNX=4)
p.plot(3,wn, radiancewn2,'Planck radiance','Wavenumber cm$^{-1}$',
       'Radiance W/(m$^2$.sr.cm$^{-1}$)',maxNX=4);


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Converting wideband values between radiant and photon rate units
The relationship between the photon rate radiometric units and radiant units are well defined for a single wavelength. It is best to calculate the spectral photon rate units from first principles and then integrate over the spectral band of interest. However on occasion the information may only be available in wideband radiant units, but the quantity must be expressed in photon rate units. For wideband data the question then arises which wavelength should be used in the conversion?  This section determines the appropriate wavelength to be used for the conversion for a given spectral band and source spectrum.
The appropriate conversion wavelength is determined at the wavelength where an accurate spectral calculation gives the same answer as an approximate wideband calculation.  The experimentation below indicates that the conversion wavelength is slightly longer than the center of the spectral band - photons at longer wavelengths have lower energy than at shorter wavelengths.
Responsivity is given by
\begin{equation}
R_e = \frac{\eta q \lambda}{hc},
\end{equation}
from which quantum efficiency can be determined as
\begin{equation}
\eta = \frac{R_e hc}{q \lambda}.
\end{equation}
Can you write Python code for it?

# to investigate the conversion for broadband data between photon rate units and radiant units
import scipy.constants as const

spec = np.loadtxt('data/InSb.txt')
wl = spec[:,0].reshape(-1)
respW = 2. * spec[:,2].reshape(-1)
respQ = respW * const.h * const.c / (wl * 1e-6 * const.e)

wlNan = np.where(respW==0,np.nan,wl)
wlcent = np.mean([np.nanmax(wlNan),  np.nanmin(wlNan)])
print('wl min = {}'.format(np.nanmin(wlNan)))
print('wl mid = {}'.format(wlcent))
print('wl max = {}'.format(np.nanmax(wlNan)))

p = ryplot.Plotter(1,2,2,figsize=(10,6));
p.plot(1, wl,respW, 'Radiant domain','Wavelength $\mu$m','Responsivity A/W',maxNX=5,xAxisFmt="%.2f");
p.plot(2, wl,respQ,'Quantum domain ','Wavelength $\mu$m','Quantum efficiency',maxNX=5,xAxisFmt="%.2f");

q = ryplot.Plotter(2,1,1,figsize=(8,4))
for temp in [250.,300.,350.]:
    Ve = respW * ryplanck.planck(wl,temp,'el') / np.pi
    Vq = respQ * ryplanck.planck(wl,temp,'ql') / np.pi
    Vei = np.trapz(Ve,-wl,axis=0)
    Vqi = np.trapz(Vq,-wl,axis=0)

    # convert from W to q/s
    qW = const.h * const.c /  (wl * 1e-6)
    ratio =  ((Vei/qW) / Vqi ) / (np.max(respW)/np.max(respQ))
    wlu = np.interp(1.,ratio[::-1],wl[::-1])
    q.plot(1, wl,ratio, 'Ratio (normalised responses) as function of conversion wavelength',
           'Wavelength $\mu$m', 'Ratio',
           label=['{} K, crossover at {:.3f} $\mu$m'.format(temp,wlu)],maxNX=5,xAxisFmt="%.2f");

    p.plot(3, wl,Ve, 'Radiant domain',
           'Wavelength $\mu$m', 'A/(m$^2$.sr.$\mu$m)',
           label=['{} K, {:.3e} A/(m$^2$.sr)'.format(temp,Vei)],maxNX=5,xAxisFmt="%.2f");
    p.plot(4, wl,Vq, 'Quantum domain ',
           'Wavelength $\mu$m', 'q/(s.m$^2$.sr.$\mu$m)',
           label=['{} K, {:.3e} q/(s.m$^2$.sr)'.format(temp,Vqi)],maxNX=5,xAxisFmt="%.2f");
#     p.plot(4, wl,Ve/const.e);


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Detector spectral response
Detector spectral responsivity values are also normally measured or taken from detector data sheets. In the absence of such information the shape can be calculated from a detailed bulk detector model, such as provided in pyradi.rydetector, but sometimes a simplified shape is required.  Such a simple photon detector responsivity can be calculated with the pyradi.ryutils.responsivity function. The function takes the form (Sec D.4.3):
$${\cal R}\lambda= k\left[
\left(\frac{\lambda}{\lambda_c} \right)^a -
\left(\frac{\lambda}{\lambda_c} \right)^n
\right]\;\;\;{\rm for}\;\;\; {\cal R}\lambda>0, \;\;\; 0 \;\;\;{\rm otherwise}.$$
This function takes a wavelength spectral variable $\lambda$, the peak wavelength value $\lambda_c$, the sharpness of cuton (lower wavelength range)  $a$, the sharpness of cutoff (longer wavelength range) $n$, and an amplitude scaling factor $k$. Best results are obtained for $0\leq a \leq 5$ and $5\leq n \leq 50$.
The use of the function is best demonstrated by graphical means.
Can you write Python code for it?

lwavepeak = 1.2
params = [(0.5, 5), (1, 10), (1, 20), (1, 30), (1, 1000), (2, 20)] #cut-on and cut-off
parameterTxt = ['a={0}, n={1}'.format(s[0], s[1]) for s in params ]
responsivities = ryutils.responsivity(wavelength,lwavepeak, params[0][0], params[0][1], 1.0)
for param in params[1:]:
    responsivities =  np.hstack((responsivities, ryutils.responsivity(wavelength,lwavepeak, param[0], param[1], 1.0)))

smpleplt = ryplot.Plotter(1, 1, 1, figsize=(10, 4))
smpleplt.plot(1, wavelength, responsivities, "Detector Responsivity for $\lambda_c$=1.2 $\mu$m, k=1", r'Wavelength $\mu$m',\
           r'Responsivity', \
           ['r', 'g', 'y','b', 'k', 'm'],label=parameterTxt,maxNX=5,xAxisFmt="%.2f");

# here we simply multiply the responsivity and spectral filter spectral curves.
# this is a silly example, but demonstrates the spectral integral.
filtreps = responsivities * filters
parameterTxt = [str(s)+' & '+str(f) for (s, f) in zip(params, filterExp) ]

smpleplt = ryplot.Plotter(1, 1, 1, figsize=(10, 4))
smpleplt.plot(1, wavelength, filtreps, "Filtered Detector Responsivity",
              r'Wavelength $\mu$m',\
           r'Responsivity', \
           ['r', 'g', 'y','b', 'k', 'm'],label=parameterTxt,legendAlpha=0.5,maxNX=5,xAxisFmt="%.2f");


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Effective value normalisation
The effective value of a spectral variable is given by (Sec 7.2.2)
$$
{F}{\rm eff} = \frac
{\int{0}^{\infty}{F} {G}
d\lambda}
{\int_{0}^{\infty}{G} \lambda},
$$
where ${F}$ is the variable in question, and ${G}$ is a weighting function. Note that the effective value
of $F$ depends on the  shapes of both $F$ and $G$; the effective value of $F$ thus calculated therefore applies only to the specific weighting function $G$ - it is incorrect for any other shapes.
To demonstrate the sensitivity of the effective value normalisation to the spectral variability of the input spectra, consider the effective detector responsivity calculated at different blackbody source temperatures from 500 K to 2000 K.  The InSb detector is modelled using the simple detector model in ryutils.responsivity.  The source radiance is the weighting function $G$, calculated by using ryplanck.planck.  The following graphs show the spectral detector responsivity ($F$ in the above equation) and the effective responsivity as calculated for the range of temperatures.  The calculation requires the calculation of a separate quotient and integral for each temperature, yet there are no loops!  This magic is achieved by constructing two-dimensional arrays with wavelengths along the row (0'th) axis and temperature along the column (1'st) axis.  The first of these arrays is returned by the ryplanck.planck function (because a vector of wavelengths and a vector of temperatures are passed to the function). The second array is obtained by using the numpy.meshgrid array (google to see how this function works).  The two arrays are used to calculate the numerator of the quotient.  The integrals are implemented using the numpy.trapz function, integrating along the wavelength axis=0 direction. This integration returns a vector of values corresponding to the temperature values along the columns axis=1.
Note the very wide variation in effective responsivity when calculated with different source temperatures. This stems from the fact...
Can you write Python code for it?

wavelength = np.linspace(0.1, 20, 305).reshape(-1, 1)
detector = ryutils.responsivity(wavelength, 6.1, 1, 15, 1.0)
temperatures = np.linspace(500, 2000, 41)
spectralBaseline = ryplanck.planck(wavelength,temperatures,'el')
_, detVar = np.meshgrid(temperatures, detector)
spectralWeighted = spectralBaseline * detVar
effResp = np.trapz(spectralWeighted, wavelength, axis=0 ) / np.trapz(spectralBaseline, wavelength, axis=0)

p = ryplot.Plotter(2,1,3,figsize=(18,6))
p.plot(1,wavelength,detector,'Spectral responsivity','Wavelength $\mu$m', 'Responsivity A/W',pltaxis=[0.5,7,0,0.8],maxNX=5,xAxisFmt="%.2f");
p.plot(2,temperatures,effResp,'Effective responsivity','Temperature K', 'Responsivity A/W',maxNX=5,xAxisFmt="%.0f");


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Experiment a little with the meshgrid and its results. What do these variables look like graphically? Calculate and plot, for each tile: (1) the distance from the reference point, (2) the solid angle of the angle to the reference point, (3) the geometric solid angle of each tile, and (4) the projected solid angle of each tile.  Along the way plot the values of x and y as well.
Can you write Python code for it?

distFromRef = np.sqrt(x ** 2 + y ** 2 + h * h)
cosAngle = h / distFromRef
geoSolid = w * d / (samples ** 2) * cosAngle / distFromRef ** 2
prjsolid = geoSolid * cosAngle

p = ryplot.Plotter(1,2, 3, figsize=(15,10))
p.mesh3D(1, x, y, x, ptitle='Distance from centre\nalong x-axis',
         xlabel='x', ylabel='y', zlabel='Distance',
         maxNX=3, maxNY=3, maxNZ=4, alpha=0.5, xInvert=True, yInvert=True);
p.mesh3D(2, x, y, y, ptitle='Distance from centre\nalong y-axis',
         xlabel='x', ylabel='y', zlabel='Distance',
         maxNX=3, maxNY=3, maxNZ=4, alpha=0.5, xInvert=True, yInvert=True);
p.mesh3D(3, x, y, distFromRef, ptitle='Distance from reference\npoint to tile',
         xlabel='x', ylabel='y', zlabel='Distance',
         maxNX=3, maxNY=3, maxNZ=4, alpha=0.5, xInvert=True, yInvert=True);
p.mesh3D(4, x, y, cosAngle, ptitle='Cosine of angle to tile',
         xlabel='x', ylabel='y', zlabel='Cosine',
         maxNX=3, maxNY=3, maxNZ=4, alpha=0.5, xInvert=True, yInvert=True);
p.mesh3D(5, x, y, geoSolid, ptitle='Geometric solid angle per tile',
         xlabel='x', ylabel='y', zlabel='Solid angle sr',
         maxNX=3, maxNY=3, maxNZ=4, alpha=0.5, xInvert=True, yInvert=True);
p.mesh3D(6, x, y, prjsolid, ptitle='Projected solid angle per tile',
         xlabel='x', ylabel='y', zlabel='Solid angle sr',
         maxNX=3, maxNY=3, maxNZ=4, alpha=0.5, xInvert=True, yInvert=True);


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
File loading and saving
Data are easily read in from files.  The format most easy to read/write is ASCII coding in a tabular format (also known as a flat file). ASCII tabular files can use commas (csv format) or whitespace (tabs, spaces) to delimit the columns from each other.  Python provides specialist CSV readers or just ASCII format readers.  The data processing package Pandas also has very powerful data reading and manipulation facilities, especially for data in complex structural formats. In the pyradi context we are more interested in opening files with numpy and this analysis will focus on reading and writing files for Numpy applications.
Numpy (in versions later than 1.6) has powerful functions to read and write data files to and from Numpy arrays.  The file formats include ASCII, binary format and compressed binary formats.
ASCII flat files are convenient data stores because these files can be read/written by a text editor and with a little effort with Excel (import data from txt, save as txt).  In this example an array is created, written to file and read back from the file.  When writing the text file, the string format is set to truncate the number of decimal values - on purpose, because we want to compare the array in memory with the array read in from file.  The numpy.loadtxt and numpy.savetxt functions does the work.  These functions take as a parameter the filename or file handle (if the file is opened elsewhere).  If the filename has the extension .gz, the file is handled as a gzip compressed file.
When reading and writing files, the with context manager is used. The motivation for using with is that if something goes wrong in the opening, reading or writing, the file is closed in an orderly manner.  Also, it is more concise and requires less typing, because the with context  manager for the file closes the file automatically when leaving the block.  Note that the with context manager provides a file handle: the gz extension is lost and the file is not compressed.
In this example the file...
Can you write Python code for it?

arr = np.random.randn(3,4)
print('original data:\n{}'.format(arr))

#write/read uncompressed file
filename = 'arrfile.txt'
with open(filename, 'wb' ) as filehandle:
    np.savetxt(filehandle, arr, fmt='%12.5e',
            header='# this is a header\n#Line two of header', comments='#')
with open(filename, 'rb' ) as filehandle:
    arr2 = np.loadtxt(filehandle, comments='#')
print('read from text file {} data:\n{}'.format(filename, arr2))

#write/read gzip compressed file - does not work with `with`
filename = 'arrfile.gz'
np.savetxt(filename, arr, fmt='%12.5e',
            header='# this is a header', comments='#')
arrz = np.loadtxt(filename, comments='#')
print('read from text file {} data:\n{}'.format(filename, arrz))
usecols=[0,3]
arrz = np.loadtxt(filename, usecols=usecols, comments='#')
print('read columns {} from text file {} data:\n{}'.format(usecols,
            filename, arrz))


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Finally plot the colour coordinates for each sample-source combination on the diagram, together with the monochromatic horseshoe.  The graph is plotted trice; once to get the big picture and then two zoomed in on the detail.
Can you write Python code for it?

if dlNames:
    ciexyplt = ryplot.Plotter(4, 1, 3, figsize=(17,7))
    for i,pltaxis in enumerate([[0, 0.9, 0, 0.9],[0.3, 0.65, 0.35, 0.5],[0.52, 0.6, 0.43, 0.46]]):
        ciexyplt.resetPlotCol()
        #plot monochromatic horseshoe
        ciexyplt.plot(i+1, xm, ym, plotCol=['k'],pltaxis=pltaxis)
        #plot chromaticity loci for samples
        styleSample=['r--', 'g-.', 'y', 'g', 'b-', 'k']
        for iSmpl in range(samples.shape[1]):
            ciexyplt.plot(i+1,xs[iSmpl],ys[iSmpl], [styleSample[iSmpl]],
                    label=[samplesTxt[iSmpl]], legendAlpha=0.5,
                    pltaxis=pltaxis)

        #plot source markers
        styleSource=['*', 'o', '^', 'v']
        for iSmpl in range(samples.shape[1]):
            for iSrc in range(sources.shape[1]):
                if iSmpl==0:
                    legend=[sourcesTxt[iSrc]]
                else:
                    legend=''
                ciexyplt.plot(i+1,xs[iSmpl,iSrc],ys[iSmpl,iSrc],
                              "CIE chromaticity diagram", r'x',r'y',\
                        ['k'],label=legend,legendAlpha=0.5,
                        pltaxis=pltaxis, markers=[styleSource[iSrc]],maxNX=5,xAxisFmt="%.2f");


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Generic spectral filter response
In most real-world system analyses, filter spectral responses will be measured and used in tabular formats.  However, in some instances it is convenient to create spectral filter shapes according to a parameter set. The pyradi library has a function pyradi.ryutils.sfilter to assist in calculating a variety of spectral filter shapes.  This function has the form (Sec D.4.2):
$$\tau_\lambda =\tau_s + \tau_p
\exp\left[ -\left(\frac{2(\lambda-\lambda_c)}{\Delta\lambda}\right)^s\right].
$$
This function takes a spectral variable $\lambda$, the centre wavelength $\lambda_c$, the spectral width $\Delta\lambda$, the sharpness of cutoff $s$  (if $s=2$, the
curve has a Gaussian shape, and if $s=\infty$, the curve has a square top-hat shape), transmittance in the pass band $\tau_p$ and stop band $\tau_s$ and the type of filter. The use of the function is best demonstrated by graphical means.
Can you write Python code for it?

wavelength = np.linspace(0.1, 1.3, 350).reshape(-1, 1)

#demonstrate the different filter types
width = 0.4
center = 0.9
filterExp=[2, 2,  4, 6,  8, 12, 500] # shaprness
filterPass=[0.4, 0.5,  0.6, 0.7,  0.8, 0.9, 0.99] # tau pass
filterSupp = np.asarray(filterPass) * 0.1 # tau stop
filterType=['bandpass', 'lowpass', 'highpass', 'bandpass', 'lowpass',
            'highpass', 'bandpass']
filterTxt = [r's={0}, $\tau_p$={2}, {1}'.format(s,y,z) for s,y,
             z in zip(filterExp, filterType, filterPass) ]
filters = ryutils.sfilter(wavelength,center, width, filterExp[0],
                          filterPass[0],  filterSupp[0], filterType[0])

for i,exponent in enumerate(filterExp[1:]):
    tau=ryutils.sfilter(wavelength,center, width, filterExp[i+1],filterPass[i+1],
                        filterSupp[i+1], filterType[i+1])
    filters =  np.hstack((filters,tau))

#plot sample filters
smpleplt = ryplot.Plotter(1, 1, 1, figsize=(10, 4));
smpleplt.plot(1, wavelength, filters,
    r"Optical filter for $\lambda_c$={0}, $\Delta\lambda$={1}".format(center,width),
    r'Wavelength $\mu$m', r'Transmittance', \
            ['r', 'g', 'y','g', 'b', 'm','c'],label=filterTxt,maxNX=5,xAxisFmt="%.2f");

#all passband, different shapes
width = 0.5
center = 0.7
filterExp=[2,  4, 6,  8, 12, 500] # sharpness

filterTxt = ['s={0}'.format(s) for s in filterExp ]
filters = ryutils.sfilter(wavelength,center, width, filterExp[0], 0.8,  0.1)
for exponent in filterExp[1:]:
    filters =  np.hstack((filters, ryutils.sfilter(wavelength,center, width,
                                              exponent, 0.8,  0.1)))
smpleplt = ryplot.Plotter(1, 1, 1, figsize=(10, 4));
smpleplt.plot(1, wavelength, filters,
    r"Optical filter for $\lambda_c$=0.7,$\Delta\lambda$=0.5,$\tau_{s}$=0.1, $\tau_{p}$=0.8",
    r'Wavelength $\mu$m', r'Transmittance', \
            ['r', 'g', 'y','b', 'k', 'm'],label=filterTxt,maxNX=5,xAxisFmt="%.2f");


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
If the spectral interval is not constant, i.e, wavelength domain converted to wavenumber domain, the  dx or pre-multiplication approach is no longer valid. The only possibility to calculate the integral is to give the value of x, with its non-constant intervals. Continuing from the previous example, convert the wavelength domain to the wavenumber domain, calculate the spectral density in wavenumber units W/(m$^2$.sr.cm$^{-1}$) and calculate the integral using the (non-constant-interval) wavenumber domain as x:
Can you write Python code for it?

wn = ryutils.convertSpectralDomain(wl, type='ln') # wavenumber
radiancewn = ryplanck.planck(wn, temperature, 'en') / np.pi
print('Integral(wn) = {} W/(m^2.sr)'.format( np.trapz(radiancewn, wn )))


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
It appears that in Numpy a rank-one array (N,) behaves like a row vector (1,N) in some instances. In pyradi it is more convenient and natural to consider spectral variables as column vectors (N,1), therefore this rehaping from (N,) to (N,1) takes place quite frequently.  Why consider spectral variables as (N,1) arrays? When saving spectral data to file, it seems more natural (and easier to view/edit) if the spectral range varies along the rows in the file: each spectral value is on a new line in the file. This approach also lends itself to saving more than one spectral variable in a file where the first column represents the spectral domain and all remaining columns represent spectral variables. A case in point is the Modtran tape7 file. So as a general rule: pyradi always expect spectral variables of one of the these shapes:
Can you write Python code for it?

wl1 = np.linspace(0.4, 0.8, 101) #rank (N,)
wl2 = wl1.reshape(-1,1) #convert to rank (N,1)


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
It is quite evident that for the two 'daylight' sources (the sun 5900 K and the fluorescent source) the sample colour coordinates are widely scattered over the coordinate space.  When the same samples are viewed under 2850 K source illumination (incandescent lamp) the colours tend to cluster towards the yellow - this matches the human experience that incandescent lamps produce yellow colour photographs.  Finally, when viewed under sodium illumination the sample colour coordinates converge on the colour of the lamp, meaning that the samples all appear yellow  in colour (589 nm) and only differ in shade.
The CIE 1931 chromaticity diagram in colour is shown below.
Can you write Python code for it?

HTML('<img src="images/2000px-CIE-1931_diagram_in_LAB_space.svg.png" width=600 height=600/>')


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Mixing spectral data from different sources
When working with mixed domains, e.g., Modtran transmittance data in wavenumber domain and sensor spectral response in wavelength domain, you have to resample from the one to the other, to at least get samples at the same spectral intercepts. So, work in either domain, but be sure that the sampled data all relate to the same fundamental spectral baseline. The pyradi library provides functions to resample data when reading in the data from file, with minimal coding overhead (see  ryfiles.loadColumnTextFile).
Performing spectral integrals
A repeating pattern in computational radiometry is the spectral integral: integrating the sum of a spectral variable between an upper and a lower limit. For example, this calculation is required to determine the total inband radiance over a wide spectral range.  We use the mathematical integral notation to indicate such integrals, but in practice, these are calculated using numerical summation.
$$
L = \int_{\lambda_0}^{\lambda_1} L_\lambda d\lambda = \sum_{i=0}^{N}L(i)\Delta\lambda(i)
$$
where the sample $i=0$ corresponds with the sample at $\lambda_0$ and the sample $i=N$ corresponds with the sample at $\lambda_N$.
The sum above is a general statement and does not make any assumption about the spectral domain sampling interval.
If the spectral domain sampling interval has fixed increments $\Delta\lambda(i)=\Delta\lambda$ then
$$
L = \int_{\lambda_0}^{\lambda_1} L_\lambda d\lambda = \Delta\lambda\sum_{i=0}^{N}L(i).
$$
The spectral interval (constant interval on wavelength or wavenumber) is not of major significance when performing  spectral multiplications or when plotting the data. It is however necessary to allow for the (potentially varying) spectral intervals when doing spectral integrals.

The picture above shows non-constant intervals on the $x$ axis. When using the trapezoidal rule the integral is calculated as the summation of areas where the area is given by the $\Delta x$sampling along the $x$ axis and the mean height...
Can you write Python code for it?

wl = np.linspace(1, 14, 1001)#.reshape(-1,1) # wavelength
temperature = 1000.0 # K
radiancewl = ryplanck.planck(wl,temperature, 'el') / np.pi
print('Integral(1) = {} W/(m^2.sr)'.format( (wl[1]-wl[0]) * np.trapz(radiancewl)) )
print('Integral(2) = {} W/(m^2.sr)'.format( np.trapz(radiancewl, dx=(wl[1]-wl[0]))) )
print('Integral(3) = {} W/(m^2.sr)'.format( np.trapz(radiancewl,wl)) )


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Next load the sample reflectance data.  The data file for this data
has the wavelength ($\mu$m) values along the first column, with many columns
following to represent the different samples.  Only a few of the samples (samplesSelect = [1,2,3,8,10,11]) are loaded here.
After reading in the numeric values, the headers in the first row is read in.
These headers are used in labels in the graphs.
Can you write Python code for it?

if dlNames:

    samplesSelect = [1,2,3,8,10,11]
    samples = ryfiles.loadColumnTextFile('samplesVis.txt', abscissaOut=wavelength,
                                         loadCol=samplesSelect,  comment='%')
    samplesTxt=ryfiles.loadHeaderTextFile('samplesVis.txt',
                loadCol=samplesSelect, comment='%')
    smpleplt = ryplot.Plotter(1, 1, 1, figsize=(8,4))
    smpleplt.plot(1, wavelength, samples, "Sample reflectance", r'Wavelength $\mu$m',
                r'Reflectance', ['r', 'g', 'y','g', 'b', 'm'],label=samplesTxt,legendAlpha=0.5,pltaxis=[0.38,0.72,0,1],maxNX=5,xAxisFmt="%.2f");


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Now calculate the CIE x,y coordinates for each combination of source and sample.  The x,y coordinates indicate the colour of the sample under the specific source illumination.  This cell uses the rychroma.chromaticityforSpectralL function to do the coordinate calculation from the spectra.  For equations and more details see (Sec 2.10).
We have a number of samples and a number of sources, essentially these two are orthogonal to each other.  The following calculation is therefore done in two loops and the results written to two-dimensional arrays for storage and subsequent plotting.  The two arrays are pre-allocated and filled with zeros before the calculation.
Can you write Python code for it?

if dlNames:
    import pyradi.rychroma as rychroma

    xs = np.zeros((samples.shape[1],sources.shape[1]))
    ys = np.zeros((samples.shape[1],sources.shape[1]))
    for iSmpl in range(samples.shape[1]):
        for iSrc in range(sources.shape[1]):
            [ xs[iSmpl,iSrc], ys[iSmpl,iSrc], Y]=\
                rychroma.chromaticityforSpectralL(wavelength,
                (samples[:,iSmpl]*sources[:,iSrc]).reshape(-1, 1),
                bar[:,0], bar[:,1], bar[:,2])


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Plotting of spectral variables
The title appears trivial, but there is a hidden subtlety when working with and plotting radiometric variables. In the section above  `spectral variables in pyradi'', the data was created in the wavelength domain (using thelinspace` function), but plotted in both wavelength and wavenumber domains. Note that the data was not resampled to wavenumber domain before plotting: the wavenumber plots are not at constant intervals.  The wavelength domain was merely recalculated to wavenumber domain. This is illustrated in the graph below.
Can you write Python code for it?

wlc = np.linspace(0.5, 8.5, 17).reshape(-1,1) # wavelength
wnc = ryutils.convertSpectralDomain(wlc, type='ln') # wavenumber
p = ryplot.Plotter(2,1,3,figsize=(18,2))
for i in range(wlc.shape[0]):
    p.plot(1,np.array([wlc[i],wlc[i]]), np.array([0,1.]), 'Constant wavelength intervals',
           'Wavelength $\mu$m', plotCol=['r'], markers=['o'], maxNY=1)
    p.plot(2,np.array([wnc[i],wnc[i]]), np.array([0,1.]), 'Non-constant wavenumber intervals',
           'Wavenumber cm$^{-1}$',plotCol=['r'], markers=['o'], maxNX=5, maxNY=1, pltaxis=[0, 10000, 0, 1]);


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Python and module versions, and dates
Can you write Python code for it?

try:
    import pyradi.ryutils as ryutils
    print(ryutils.VersionInformation('matplotlib,numpy,pyradi,scipy,pandas'))
except:
    print("pyradi.ryutils not found")


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
See the section 'Performing-spectral-integrals' below.
Interpolation of Spectral Variables
SciPy has a number of interpolation options in scipy.interpolate.
The one-dimensional interpolation scheme enables the definition of the lookup table as a function by using scipy.interpolate.interp1d.  The user can define the interpolation scheme to be used - the default is kind='linear', to do cubic interpolation use kind='cubic'.  In the example below observe the difference between the two schemes.
The example below is an ill-posed problem: normally one would have a much denser input data set.  The purpose with using this poor data set is to illustrate the dangers of interpolation. Do not use a potentially unstable interpolation scheme such as cubic interpolation on a poor data set such as the one shown below.
The input data set has only four samples.  The ordinate (y) data has a slight oscillation.  The abscissa (x) data are defined in the wavelength space. Equivalently, the abscissa (x) data are also defined in the wavenumber space, because $\tilde{\nu}=10^4/\lambda$. This is the only information available, there is no requirement on the values between the stated abscissa.
Important: by selecting the interpolation scheme (linear, cubic, etc.), you are implicitly making a modelling assumption.  Linear interpolation in the wavelength space means a non-linear interpolation in wavenumber space and vice versa (because wavelength and wavenumber are nonlinearly related). Stated differently: if your input data set was compiled to be interpolated linearly in wavelength space (blue lines), then linear interpolation in wavenumber space will give the wrong results (red lines).
Also, by selecting between the linear and cubic (or any other scheme) you are also making a modelling assumption.  Compare the difference between the results for the linear interpolation (red/blue) and the cubic interpolation (green/cyan) lines.
It is evident in the cubic interpolation graphs (green/cyan) that cubic/high-order interpolation schemes do...
Can you write Python code for it?

from scipy import interpolate
import pyradi.ryplot as ryplot
import numpy as np
%matplotlib inline

# create input data in wavelength and wavenumber domain
maxx = 24
wl = np.asarray([1, 2, 5, 10, maxx])
wn = 1e4 / wl # wl and wn values precisely align
fn = np.asarray([2,3,1, 2, -1])

# create the plotting samples - much tighter than input data
wli = np.linspace(1,maxx,400)
wni = 1e4 / wli

#create the lookup functions
fwll = interpolate.interp1d(wl,fn)
fwlc = interpolate.interp1d(wl,fn, kind='cubic')
fwnl = interpolate.interp1d(wn,fn)
fwnc = interpolate.interp1d(wn,fn, kind='cubic')

p = ryplot.Plotter(1,2,2,figsize=(12,8))
p.plot(1,wli,fwll(wli),label=['wl scale interpolation - linear'],plotCol=['b'])
p.plot(1,wli,fwnl(wni),'','Wavelength $\mu$m',label=['wn scale interpolation - linear'],plotCol=['r'], maxNX=5)
p.resetPlotCol()
p.plot(2,wli,fwll(wli),label=['wl scale interpolation - linear'],plotCol=['b'])
p.plot(2,wli,fwlc(wli),label=['wl scale interpolation - cubic'],plotCol=['g'])
p.plot(2,wli,fwnc(wni),'','Wavelength $\mu$m',label=['wn scale interpolation - cubic'],plotCol=['c'], maxNX=5)
p.resetPlotCol()
p.plot(3,wni,fwll(wli),label=['wl scale interpolation - linear'],plotCol=['b'])
p.plot(3,wni,fwnl(wni),'','Wavenumber cm$^{-1}$',label=['wn scale interpolation - linear'],plotCol=['r'], maxNX=5)
p.resetPlotCol()
p.plot(4,wni,fwll(wli),label=['wl scale interpolation - linear'],plotCol=['b'])
p.plot(4,wni,fwlc(wli),label=['wl scale interpolation - cubic'],plotCol=['g'])
p.plot(4,wni,fwnc(wni),'','Wavenumber cm$^{-1}$',label=['wn scale interpolation - cubic'],plotCol=['c'], maxNX=5);


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
Solid Angles and Spatial Integrals
Computational radiometry has a strong element of geometry: sources and receivers have size, shape and orientation and are related to each over in some geometrical relationship. The calculation of solid angle is an essential task in most radiometric analyses.
The calculation of a solid angle requires integration across a spatial extent as described in (Sec 2.5.4).  Array processing in Python condenses and simplifies most two-dimensional calculations significantly.
Can you write Python code for it?

from IPython.display import SVG
SVG(filename='images/solidangleflatplate.svg')


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
The atmospheric transmittance model is next Savitzky-Golay filtered. This filter is poular in the spectroscopy community, so it should work well for the atmospheric transmittance data used here.  It indeed does provide nicely filtered spectra, as seen below.  More work is required to accurately translate the SG filter parameters to spectral widths.
Can you write Python code for it?

from scipy.interpolate import  interp1d
import pyradi.ryfiles as ryfiles

bunsenfile = ryfiles.downloadFileUrl(url = 'https://raw.githubusercontent.com/NelisW/pyradi/master/pyradi/data/bunsenspec.txt')
specRad = np.loadtxt(bunsenfile, comments='%', delimiter=' ')

atmofile = ryfiles.downloadFileUrl(url = 'https://raw.githubusercontent.com/NelisW/pyradi/master/pyradi/data/atmotrans5m.txt')
tauAtmo = np.loadtxt(atmofile, comments='%', delimiter=' ' )
wavenum =  tauAtmo[:, 0]
tauA = tauAtmo[:, 1]

# convolve transmittance from 1cm-1 to 4 cm-1
tauAtmo4 = ryutils.savitzkyGolay1D(tauA, window_size=9, order=3, deriv=0, rate=1)

# convolve transmittance from 1cm-1 to 40 cm-1
tauAtmo40 = ryutils.savitzkyGolay1D(tauA, window_size=81, order=3, deriv=0, rate=1)

#interpolate bunsen spectrum to atmo sampling
#first construct the interpolating function, using bunsen
bunInterp1 = interp1d(specRad[:,0], specRad[:,1])
#then call the function on atmo intervals
bunsen = bunInterp1(wavenum)

atmoplot = tauA.copy()
atmoplot =  np.vstack((atmoplot, tauAtmo4, tauAtmo40))
convplot02 = ryplot.Plotter(1, 1, 2,figsize=(20,5))
convplot02.plot(1, wavenum, atmoplot.T, "Atmospheric Transmittance: Savitzky Golay Filter", r'Wavenumber cm$^{-1}$',
            r'Transmittance', ['r', 'g','b'],label=['1 cm-1', '4 cm-1', '40 cm-1' ],legendAlpha=0.5,
                pltaxis=[2900, 4000, 0, 1.1],maxNX=5,xAxisFmt="%.0f");
convplot02.plot(2, wavenum, atmoplot.T, "Atmospheric Transmittance: Savitzky Golay Filter", r'Wavenumber cm$^{-1}$',
            r'Transmittance', ['r', 'g','b'],label=['1 cm-1', '4 cm-1', '40 cm-1' ],legendAlpha=0.5,
            pltaxis=[3700, 3900, 0, 1.1],maxNX=5,xAxisFmt="%.0f");


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
The data and the plot are equally 'valid' or true in either domain, because the domain merely samples the data at different spectral intercepts.  So we can load and process data in either domain and plot in either domain, without losing any truth value. Plot in whatever domain illustrates your argument the best.
You should however take care when dealing with spectral densities: it is possible and mathematically allowable to plot a wavenumber density with units of W/(m$^2$.sr.cm$^{-1}$) against a wavelength domain with units of $\mu$m, but it does not make much practical sense to do so.
Consider the intervals in both cases, here calculated with numpy.diff. Note that the intervals are constant and positive for the wavelength scale, but negative and non-constant for the wavenumber scale as converted from the wavelength scale.  Recall from Section `Spectral-variables' above, that  $d \tilde{\nu} = -d\lambda \,10^4/\lambda^2=-d\lambda\, \tilde{\nu}^2 /10^4$. In other words the spectral intervals change in the opposite sense: the wavelength domain scale increases (positive intervals), but when converted to wavenumber domain the scale decreases (negative intervals).
Can you write Python code for it?

p = ryplot.Plotter(3,1,3,figsize=(18,2))
p.plot(1,(wlc[1:]+wlc[:-1])/2, np.diff(wlc,axis=0), 'Constant wavelength intervals',
       'Wavelength $\mu$m',maxNX=5,xAxisFmt="%.2f");
p.plot(2,(wnc[1:]+wnc[:-1])/2, np.diff(wnc,axis=0), 'Non-constant wavenumber intervals',
       'Wavenumber cm$^{-1}$',maxNX=5,xAxisFmt="%.0f");


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
The next example uses spectral Bunsen flame measurement data. The data is read in from the pyradi Google Code site, so you must be online to run this cell.  The data is read using the urllib2 library, given the URL, the urllib2 returns a file handle that is used by Numpy to read the data.  The spectral width for the Bunsen flame measurement was 4 cm$^{-1}$ and the signal was sampled at 2 cm$^{-1}$. The Bunsen data file format is: first column contains the wavenumber and the second column contains the raw signal (proportional to radiance). Modtran transmittance data were calculated for a 5 m path length and a spectral width of 1 cm$^{-1}$ and sampling of 1 cm$^{-1}$. The transmittance file format is: first column contains the wavenumber and the second column contains the transmittance. The atmospheric transmittance is convolved to resolutions of 4 cm$^{-1}$ and 40 cm$^{-1}$.  The spectral baselines for the Bunsen data and Modtran data are not the same, so the Bunsen data is re-sampled at the Modtran spectral values. All the data is then plotted for visual inspection.  The convolution smoothing effect is clearly visible in the graphs.
Can you write Python code for it?

from scipy.interpolate import  interp1d
import pyradi.ryfiles as ryfiles

bunsenfile = ryfiles.downloadFileUrl(url = 'https://raw.githubusercontent.com/NelisW/pyradi/master/pyradi/data/bunsenspec.txt')
specRad = np.loadtxt(bunsenfile, comments='%', delimiter=' ')

atmofile = ryfiles.downloadFileUrl(url = 'https://raw.githubusercontent.com/NelisW/pyradi/master/pyradi/data/atmotrans5m.txt')
tauAtmo = np.loadtxt(atmofile, comments='%', delimiter=' ' )
wavenum =  tauAtmo[:, 0]
tauA = tauAtmo[:, 1]

# convolve transmittance from 1cm-1 to 4 cm-1
tauAtmo4,  windowfn = ryutils.convolve(tauA, 1, 1, 4)
# convolve transmittance from 1cm-1 to 40 cm-1
tauAtmo40,  windowfn = ryutils.convolve(tauA, 1, 1, 40)

#interpolate bunsen spectrum to atmo sampling
#first construct the interpolating function, using bunsen
bunInterp1 = interp1d(specRad[:,0], specRad[:,1])
#then call the function on atmo intervals
bunsen = bunInterp1(wavenum)

atmoplot = tauA.copy()
atmoplot =  np.vstack((atmoplot, tauAtmo4, tauAtmo40))
convplot02 = ryplot.Plotter(1, 1, 2,figsize=(20,5))
convplot02.plot(1, wavenum, atmoplot.T, "Atmospheric Transmittance", r'Wavenumber cm$^{-1}$',
            r'Transmittance', ['r', 'g','b'],label=['1 cm-1', '4 cm-1', '40 cm-1' ],legendAlpha=0.5,
                maxNX=5,xAxisFmt="%.0f",pltaxis=[1750,4000,0,1]);
convplot02.plot(2, wavenum, atmoplot.T, "Atmospheric Transmittance", r'Wavenumber cm$^{-1}$',
            r'Transmittance', ['r', 'g','b'],label=['1 cm-1', '4 cm-1', '40 cm-1' ],legendAlpha=0.5,
            pltaxis=[3700, 3900, 0, 1],maxNX=5,xAxisFmt="%.0f");


I am working on the file "04-IntroductionToComputationalRadiometryWithPyradi.ipynb".
The first task is:
The other alternative is to flip the spectral domain around (to obtain positive increments) and then do the integral. But there is a catch here: the intervals are not constant, so you must flip (numpy.flipud) both the spectral density and spectral domain variables, to ensure matching elements.
Can you write Python code for it?

#flip only wavenumber but not the wavenumber spectral density
print('Integral(wn) = {} W/(m^2.sr) - wrong because of a mismatch between x and y ordering'.\
      format( np.trapz(radiancewn, np.flipud(wn) )))
#flip both wavenumber and wavenumber spectral density
print('Integral(wn) = {} W/(m^2.sr)'.format( np.trapz(np.flipud(radiancewn), np.flipud(wn) )))


I am working on the file "04-NotebookTour.ipynb".
The first task is:
However, if you pass the object to display, the actual image is shown:
Can you write Python code for it?


I am working on the file "04-NotebookTour.ipynb".
The first task is:
If you print the object, you see its textual representation:
Can you write Python code for it?


I am working on the file "04-NotebookTour.ipynb".
The first task is:
Images
The Image object has a JPEG/PNG representation that is rendered by the Notebook:
Can you write Python code for it?

from IPython.display import Image

i = Image("images/jupyter_logo.png", width="50%")


I am working on the file "04-NotebookTour.ipynb".
The first task is:
JavaScript
The Javascript object has a "representation" that runs JavaScript code in the context of the Notebook.
Can you write Python code for it?

from IPython.display import Javascript

display(Javascript("alert('hi');"))


I am working on the file "04-NotebookTour.ipynb".
The first task is:
LaTeX
This display architecture also understands objects that have a LaTeX representation. This is best illustrated by SymPy, which is a symbolic mathematics package for Python.
Can you write Python code for it?

from __future__ import division
from sympy import *
x, y, z = symbols("x y z")
init_printing(use_latex='mathjax')


I am working on the file "04-NotebookTour.ipynb".
The first task is:
Or, show the image by returning the object as the final result of an expression:
Can you write Python code for it?


I am working on the file "04-NotebookTour.ipynb".
The first task is:
When a symbolic expression is passed to display or returned from an expression, the LaTeX representation is computed and displayed in the Notebook:
Can you write Python code for it?


I am working on the file "04-Pandas-Data-Tables/APD-Crime-Data_class.ipynb".
The first task is:
Atlanta Police Department
The Atlanta Police Department provides Part 1 crime data at http://www.atlantapd.org/i-want-to/crime-data-downloads
A recent copy of the data file is stored in the cluster. Please, do not copy this data file into your home directory!
Can you write Python code for it?

### Load libraries
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

help(plt.legend)


I am working on the file "04-Pandas-Data-Tables/APD-Crime-Data_class.ipynb".
The first task is:
Convert into date-time type
Can you write Python code for it?

df['occur_ts'] = pd.to_datetime(df.occur_date+' '+df.occur_time)

#df[['offense_id', 'occur_date', 'occur_time', 'occur_ts', 'rpt_date']][1:10]

df['occur_ts'] = pd.to_datetime(df.occur_date+' '+df.occur_time)

df['occur_month'] = df['occur_ts'].map(lambda x: x.month)
df['occur_woy'] = df.occur_ts.dt.weekofyear

df.describe()

df.shape

df.columns

df.iloc[9]

resdf.index

resdf.loc[['AUTO_THEFT', 6]]

resdf = df.groupby(['UC2 Literal', 'occur_month']).offense_id.count()
resdf

resdf['BURGLARY-RESIDENCE'].as_matrix()

resdf['BURGLARY-RESIDENCE'].iloc(0)

%matplotlib inline
fig = plt.figure(figsize=(10,6)) # 10inx10in
#plt.plot(resdf['BURGLARY-RESIDENCE'].index, resdf['BURGLARY-RESIDENCE'])
plt.scatter(resdf['BURGLARY-RESIDENCE'].index, resdf['BURGLARY-RESIDENCE'], marker='x')
plt.scatter(resdf['BURGLARY-NONRES'].index, resdf['BURGLARY-NONRES'], marker='o')

plt.ylim(0, 500)
plt.title('BURGLARY-RESIDENCE')
plt.xticks(range(13), ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
fig.savefig('BurglaryResidence_over_month.svg')
x = 1

def getTheMonth(x):
    return x.month

df['occur_month'] = df['occur_ts'].map(getTheMonth)

df = pd.read_excel('/home/data/APD/COBRA083016_2015.xlsx', sheetname='Query')
df['occur_ts'] = pd.to_datetime(df.occur_date+' '+df.occur_time)
df['occur_month'] = df['occur_ts'].map(lambda x: x.month)
df['occur_woy'] = df.occur_ts.dt.weekofyear

%matplotlib inline
resdf = df.groupby(['UC2 Literal', 'occur_month']).offense_id.count()
fig = plt.figure(figsize=(10,6))
plt.scatter(resdf['BURGLARY-RESIDENCE'].index, resdf['BURGLARY-RESIDENCE'], marker='x')
plt.ylim(0, 500)
plt.title('BURGLARY-RESIDENCE')
plt.xticks(range(13), ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.savefig('quiz3-burglary-residence.png')

''

plt.savefig('quiz3-burglary-residence.png')


I am working on the file "04-Pandas-Data-Tables/APD-Crime-Data_class.ipynb".
The first task is:
Crime per year
Let's look at the
Can you write Python code for it?

crime_year = dff[dff.occur_year.between(2009, 2015)].groupby(by=['UC2 Literal', 'occur_year']).offense_id.count()

%matplotlib inline
fig = plt.figure(figsize=(40,30))
crime_types = crime_year.index.levels[0]
years = crime_year.index.levels[1]
for c in range(len(crime_types)):
    y_max = max(crime_year.loc[crime_types[c]])

    plt.subplot(4,3,c+1)
    plt.hlines(crime_year.loc[crime_types[c]].iloc[-1]*100/y_max, years[0], years[-1], linestyles="dashed", color="r")
    plt.bar(crime_year.loc[crime_types[c]].index, crime_year.loc[crime_types[c]]*100/y_max, label=crime_types[c], alpha=0.5)
    ##plt.legend()
    plt.ylim(0, 100)
    plt.xticks(years+0.4, [str(int(y)) for y in years], rotation=0, fontsize=24)
    plt.yticks([0,20,40,60,80,100], ['0%','20%','40%','60%','80%','100%'], fontsize=24)
    plt.title(crime_types[c], fontsize=30)
    None