I am working on the file "3_Positional_Astronomy/3_4_Direction_Cosine_Coordinates.ipynb".
The first task is:
Import section specific modules:
Can you write Python code for it?

from IPython.display import HTML
HTML('../style/code_toggle.html')


I am working on the file "3_Positional_Astronomy/3_4_Direction_Cosine_Coordinates.ipynb".
The first task is:
Recall that we can use Eq. 3.1 ⤵ <!--\label{pos:eq:convert_lmn_ra_dec}--> to convert between equatorial
    and direction cosine coordinates, in terms of the current example this translates into the python code below. Note that before we can do the conversion we first need to calculate $\Delta \alpha$.
Can you write Python code for it?

RA_delta_rad = RA_rad-RA_rad[0] #calculating delta alpha

l = np.cos(DEC_rad) * np.sin(RA_delta_rad)
m = (np.sin(DEC_rad) * np.cos(DEC_rad[0]) - np.cos(DEC_rad) * np.sin(DEC_rad[0]) * np.cos(RA_delta_rad))
print "l (degrees) = ", l*(180./np.pi)
print "m (degrees) = ", m*(180./np.pi)


I am working on the file "3_Positional_Astronomy/3_4_direction_cosine_coordinates.ipynb".
The first task is:
Import section specific modules:
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_4_direction_cosine_coordinates.ipynb".
The first task is:
Plotting the result.
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)
plt.xlim([-4., 4.])
plt.ylim([-4., 4.])
plt.xlabel("$l$ [degrees]")
plt.ylabel("$m$ [degrees]")
plt.plot(l[0], m[0], "bx")
plt.hold("on")
plt.plot(l[1:]*(180/np.pi), m[1:]*(180/np.pi), "ro")
counter = 1
for xy in zip(l[1:]*(180/np.pi)+0.25, m[1:]*(180/np.pi)+0.25):
    ax.annotate(Flux_sources_labels[counter], xy=xy, textcoords='offset points',horizontalalignment='right',
                verticalalignment='bottom')
    counter = counter + 1

plt.grid()


I am working on the file "3_Positional_Astronomy/3_4_direction_cosine_coordinates.ipynb".
The first task is:
Recall that we can use Eq. 3.1 ⤵ <!--\label{pos:eq:convert_lmn_ra_dec}--> to convert between equatorial
    and direction cosine coordinates, in terms of the current example this translates into the python code below. Note that before we can do the conversion we first need to calculate $\Delta \alpha$.
Can you write Python code for it?


I am working on the file "3_Preprocessing_of_Test_Data.ipynb".
The first task is:
Load the data into memory
For the testing of StarNet, it is necessary to obtain the spectra, error spectra, combined S/N, and labels, but we need to make eliminations to the test set to obtain the labels of highest validity to compare with, so we will first include the APOGEE_IDs, the S/N of the combined spectra, $T_{\mathrm{eff}}$, $\log(g)$, [Fe/H], $V_{scatter}$, STARFLAGs, and ASPCAPFLAGs to make certain eliminations. Once the stars for the test sets have been collected we will then gather the spectra and error spectra and save the two test sets to an h5 file.
Can you write Python code for it?

ap_id = F['IDs'][:,0]
combined_snr = F['stacked_snr'][:]
starflag = F['star_flag']
aspcapflag = F['aspcap_flag']
teff = F['TEFF'][:]
logg = F['LOGG'][:]
fe_h = F['FE_H'][:]
vscatter = F['VSCATTER']

print('Obtainined data for '+str(len(list(set(list(ap_id)))))+' stars.')


I am working on the file "3_Preprocessing_of_Test_Data.ipynb".
The first task is:
Separate data for High S/N test set
Can you write Python code for it?

indices_test = [i for i, item in enumerate(ap_id) if item not in train_ap_id]

test_ap_id = ap_id[indices_test]
test_teff = teff[indices_test]
test_logg = logg[indices_test]
test_fe_h = fe_h[indices_test]
test_combined_snr = combined_snr[indices_test]

indices_test_set = indices[indices_test] # These indices will be used to index through the spectra

print('Test set includes '+str(len(test_ap_id))+' combined spectra')


I am working on the file "3_Preprocessing_of_Test_Data.ipynb".
The first task is:
Separate out a dataset with good labels.
- STARFLAGs = 0
- ASPCAPFLAGs = 0
- 4000K < $T_{\mathrm{eff}}$ < 5500K
- -3.0 < [Fe/H]
- $\log(g)$ $\neq$ -9999. (value defined by ASPCAP when no ASPCAP labels are given)
- $V_{scatter}$ < 1.0 km/s
Can you write Python code for it?

teff_min = 4000.
teff_max = 5500.
vscatter_max = 1.
fe_h_min = -3.

indices, cols = np.where((aspcapflag[:]==0.)&(starflag[:]==0.)&(vscatter[:]<vscatter_max)&(fe_h[:]>fe_h_min)&(teff[:]>teff_min)&(teff[:]<teff_max)&(logg[:]!=-9999.).reshape(len(ap_id),1))

ap_id = ap_id[indices]
teff = teff[indices]
logg = logg[indices]
fe_h = fe_h[indices]
combined_snr = combined_snr[indices]

print(str(len(list(set(list(ap_id)))))+' stars remain.')


I am working on the file "3_SHARE_Data_in_the_Wide_World_py3.ipynb".
The first task is:
If we were interested to see an analysis of what sources these names came from, we can add an aggregation.
Can you write Python code for it?

name_search.aggs.bucket(
    'sources',  # Every aggregation needs a name
    'terms',  # There are many kinds of aggregations, terms is a pretty useful one though
    field='sources',  # We store the source of a document in its type, so this will aggregate by source
    size=0,  # These are just to make sure we get numbers for all the sources, to make it easier to combine graphs
    min_doc_count=1
)

# We are putting all the results into a new list called name_results
# name_search is our original list, and .execute() is a built-in function (one that the library provides, and we
    # don't have to write) that puts the results of the loop above into a new list
name_results = name_search.execute()

# We are aggregating these into a DataFrame from Pandas (which we called pd)
pd.DataFrame(name_results.aggregations.sources.to_dict()['buckets'])


I am working on the file "3_SHARE_Data_in_the_Wide_World_py3.ipynb".
The first task is:
Let's export this pandas dataframe to a csv file, and to an excel file.
The next cell will work when running locally!
Can you write Python code for it?

# Note: Uncomment the following lines if running locally:

description_dataframe.to_csv('SHARE_Counts_with_Descriptions.csv')
description_dataframe.to_excel('SHARE_Counts_with_Descriptions.xlsx')


I am working on the file "3_SHARE_Data_in_the_Wide_World_py3.ipynb".
The first task is:
Working with outside data
Let's say we had a list of names of researchers that were from a particular University. We're interested in seeing if their full names appear in any sources across the SHARE data set.
Can you write Python code for it?

# this is a simple list
names = ["Susan Jones", "Ravi Patel"]

#this is a
name_search = ShareSearch()

# We are searching the entire SHARE dataset for each item in the list we called name, i.e. Susan Jones and Ravi Patel
for name in names:
    name_search = name_search.query(
        {
            "bool": {
                "should": [
                    {
                        "match": {
                            "contributors.full_name": {
                                "query": name,
                                "operator": "and",
                                "type" : "phrase"
                            }
                        }
                    }
                ]
            }
        }
    )

# We are putting all the results into a new list called name_results
# name_search is our original list, and .execute() is a built-in function (one that the library provides, and we
    # don't have to write) that puts the results of the loop above into a new list
name_results = name_search.execute()

# Prints out the number of documents that have those
print('There are {} documents with contributors who have any of those names.'.format(name_search.count()))

# Just visual queues for us to make it more readable
print('Here are the first 10:')
print('---------')

# Loops over the list called "name_results" and prints out 10
for result in name_results:
    print(
        '{} -- with contributors {}'.format(
            result.title,
            [contributor.full_name for contributor in result.contributors]
        )
    )


I am working on the file "3_SciPy_Stack.ipynb".
The first task is:
Let's compute the element-wise sum of all these numbers: the first element of x plus the first element of y, and so on. We use a for loop in a list comprehension.
Can you write Python code for it?

z = [x[i] + y[i] for i in range(n)]
z[:3]


I am working on the file "3_SciPy_Stack.ipynb".
The first task is:
The arrays xa and ya contain the exact same numbers than our original lists x and y. Whereas those lists where instances of a built-in class list, our arrays are instances of a NumPy class ndarray. Those types are implemented very differently in Python and NumPy. We will see that, in this example, using arrays instead of lists leads to drastic performance improvements.
Now, to compute the element-wise sum of these arrays, we don't need to do a for loop anymore. In NumPy, adding two arrays means adding the elements of the arrays component by component.
Can you write Python code for it?


I am working on the file "3_SciPy_Stack.ipynb".
The first task is:
We generate two Python lists x and y, each one containing one million random numbers between 0 and 1.
Can you write Python code for it?

n = 1000000
x = [random.random() for _ in range(n)]
y = [random.random() for _ in range(n)]

x[:3], y[:3]


I am working on the file "3_SciPy_Stack.ipynb".
The first task is:
We use the %precision magic (defined in IPython) to show only 3 decimals in the Python output. This is just to alleviate the text.
Can you write Python code for it?


I am working on the file "3_Titanic/.ipynb_checkpoints/titanic_survival_exploration-checkpoint.ipynb".
The first task is:
After exploring the survival statistics visualization, fill in the missing code below so that the function will make your prediction.
Make sure to keep track of the various features and conditions you tried before arriving at your final prediction model.
Hint: You can start your implementation of this function using the prediction code you wrote earlier from predictions_2.
Can you write Python code for it?

def predictions_3(data):
    """ Model with multiple features. Makes a prediction with an accuracy of at least 80%. """

    predictions = []
    for _, passenger in data.iterrows():
        if(passenger['Sex'] == "female"):
            if(passenger['Pclass'>=2.5]):
                if(passenger['Fare'>=23]):
                    z = 0
                elif(passenger['Embarked'] == 'S' and passenger['Fare'] > 11):
                    z = 0
                elif(passenger['Embarked'] == 'S'):
                    z = 1
            z = 1
        elif(passenger['Age'] >= 6.5):
            z = 0
        elif(passenger['SibSp'] >= 2.5)
            z = 0
        else z = 1
        predictions.append(z)

    # Return our predictions
    return pd.Series(predictions)

# Make the predictions
predictions = predictions_3(data)


I am working on the file "3_Titanic/.ipynb_checkpoints/titanic_survival_exploration-checkpoint.ipynb".
The first task is:
Answer: 61.62%

Let's take a look at whether the feature Sex has any indication of survival rates among passengers using the survival_stats function. This function is defined in the titanic_visualizations.py Python script included with this project. The first two parameters passed to the function are the RMS Titanic data and passenger survival outcomes, respectively. The third parameter indicates which feature we want to plot survival statistics across.
Run the code cell below to plot the survival outcomes of passengers based on their sex.
Can you write Python code for it?


I am working on the file "3_Titanic/.ipynb_checkpoints/titanic_survival_exploration-checkpoint.ipynb".
The first task is:
Answer: 79.35%

Adding the feature Age as a condition in conjunction with Sex improves the accuracy by a small margin more than with simply using the feature Sex alone. Now it's your turn: Find a series of features and conditions to split the data on to obtain an outcome prediction accuracy of at least 80%. This may require multiple features and multiple levels of conditional statements to succeed. You can use the same feature multiple times with different conditions.
Pclass, Sex, Age, SibSp, and Parch are some suggested features to try.
Use the survival_stats function below to to examine various survival statistics.
Hint: To use mulitple filter conditions, put each condition in the list passed as the last argument. Example: ["Sex == 'male'", "Age < 18"]
Can you write Python code for it?

survival_stats(data, outcomes, 'Age', ["Sex == 'male'", "Age < 18"])
# Note: exploration was done in R, as it lends itself better to
# data analysis. After building a decision tree, we will implement the logic
# from it in Python


I am working on the file "3_Titanic/.ipynb_checkpoints/titanic_survival_exploration-checkpoint.ipynb".
The first task is:
Answer: Accuracy of 78.68%

Using just the Sex feature for each passenger, we are able to increase the accuracy of our predictions by a significant margin. Now, let's consider using an additional feature to see if we can further improve our predictions. For example, consider all of the male passengers aboard the RMS Titanic: Can we find a subset of those passengers that had a higher rate of survival? Let's start by looking at the Age of each male, by again using the survival_stats function. This time, we'll use a fourth parameter to filter out the data so that only passengers with the Sex 'male' will be included.
Run the code cell below to plot the survival outcomes of male passengers based on their age.
Can you write Python code for it?

survival_stats(data, outcomes, 'Age', ["Sex == 'male'"])


I am working on the file "3_Titanic/.ipynb_checkpoints/titanic_survival_exploration-checkpoint.ipynb".
The first task is:
Question 2
How accurate would a prediction be that all female passengers survived and the remaining passengers did not survive?
Hint: Run the code cell below to see the accuracy of this prediction.
Can you write Python code for it?


I am working on the file "3_Titanic/.ipynb_checkpoints/titanic_survival_exploration-checkpoint.ipynb".
The first task is:
Question 3
How accurate would a prediction be that all female passengers and all male passengers younger than 10 survived?
Hint: Run the code cell below to see the accuracy of this prediction.
Can you write Python code for it?


I am working on the file "3_Titanic/.ipynb_checkpoints/titanic_survival_exploration-checkpoint.ipynb".
The first task is:
Question 4
Describe the steps you took to implement the final prediction model so that it got an accuracy of at least 80%. What features did you look at? Were certain features more informative than others? Which conditions did you use to split the survival outcomes in the data? How accurate are your predictions?
Hint: Run the code cell below to see the accuracy of your predictions.
Can you write Python code for it?


I am working on the file "3_Titanic/.ipynb_checkpoints/titanic_survival_exploration-checkpoint.ipynb".
The first task is:
Tip: If you save an iPython Notebook, the output from running code blocks will also be saved. However, the state of your workspace will be reset once a new session is started. Make sure that you run all of the code blocks from your previous session to reestablish variables and functions before picking up where you last left off.

Making Predictions
If we were asked to make a prediction about any passenger aboard the RMS Titanic whom we knew nothing about, then the best prediction we could make would be that they did not survive. This is because we can assume that a majority of the passengers (more than 50%) did not survive the ship sinking.
The predictions_0 function below will always predict that a passenger did not survive.
Can you write Python code for it?

def predictions_0(data):
    """ Model with no features. Always predicts a passenger did not survive. """

    predictions = []
    for _, passenger in data.iterrows():

        # Predict the survival of 'passenger'
        predictions.append(0)

    # Return our predictions
    return pd.Series(predictions)

# Make the predictions
predictions = predictions_0(data)


I am working on the file "3_Titanic/titanic_survival_exploration.ipynb".
The first task is:
After exploring the survival statistics visualization, fill in the missing code below so that the function will make your prediction.
Make sure to keep track of the various features and conditions you tried before arriving at your final prediction model.
Hint: You can start your implementation of this function using the prediction code you wrote earlier from predictions_2.
Can you write Python code for it?

def predictions_3(data):
    """ Model with multiple features. Makes a prediction with an accuracy of at least 80%. """

    predictions = []
    for _, passenger in data.iterrows():
        if passenger['Sex'] == "male":
            if passenger['Age'] >= 6.5:
                z = 0
            if passenger['Age'] < 6.5:
                if passenger['SibSp'] >= 2.5:
                    z = 0
                else: z = 1
            else:
                z = 0
        if passenger['Sex'] == 'female':
            if passenger['Pclass'] >= 2.5:
                if passenger['Fare'] >= 23.35:
                    z = 0
                if passenger['Fare'] < 23.35:
                    if passenger['Embarked'] == "S":
                            z = 0
                    if passenger['Embarked'] in ["C", "Q"]:
                        z = 1
            if passenger['Pclass'] < 2.5:
                z = 1
        predictions.append(z)

        # Return our predictions
    return pd.Series(predictions)

# Make the predictions
predictions = predictions_3(data)

print accuracy_score(outcomes, predictions)


I am working on the file "3_Titanic/titanic_survival_exploration.ipynb".
The first task is:
Answer: 61.62%

Let's take a look at whether the feature Sex has any indication of survival rates among passengers using the survival_stats function. This function is defined in the titanic_visualizations.py Python script included with this project. The first two parameters passed to the function are the RMS Titanic data and passenger survival outcomes, respectively. The third parameter indicates which feature we want to plot survival statistics across.
Run the code cell below to plot the survival outcomes of passengers based on their sex.
Can you write Python code for it?


I am working on the file "3_Titanic/titanic_survival_exploration.ipynb".
The first task is:
Answer: Accuracy of 78.68%

Using just the Sex feature for each passenger, we are able to increase the accuracy of our predictions by a significant margin. Now, let's consider using an additional feature to see if we can further improve our predictions. For example, consider all of the male passengers aboard the RMS Titanic: Can we find a subset of those passengers that had a higher rate of survival? Let's start by looking at the Age of each male, by again using the survival_stats function. This time, we'll use a fourth parameter to filter out the data so that only passengers with the Sex 'male' will be included.
Run the code cell below to plot the survival outcomes of male passengers based on their age.
Can you write Python code for it?


I am working on the file "3_Titanic/titanic_survival_exploration.ipynb".
The first task is:
Examining the survival statistics, a large majority of males did not survive the ship sinking. However, a majority of females did survive the ship sinking. Let's build on our previous prediction: If a passenger was female, then we will predict that they survived. Otherwise, we will predict the passenger did not survive.
Fill in the missing code below so that the function will make this prediction.
Hint: You can access the values of each feature for a passenger like a dictionary. For example, passenger['Sex'] is the sex of the passenger.
Can you write Python code for it?

int(data[:1]['Sex'] == "female")

def predictions_1(data):
    """ Model with one feature:
            - Predict a passenger survived if they are female. """

    predictions = []
    for _, passenger in data.iterrows():
        # Simple way of returning 1 if female, 0 if male
        if(passenger['Sex']=="female"):
            z = 1
        else:
            z = 0
        predictions.append(z)

    # Return our predictions
    return pd.Series(predictions)

# Make the predictions
predictions = predictions_1(data)


I am working on the file "3_Titanic/titanic_survival_exploration.ipynb".
The first task is:
Examining the survival statistics, the majority of males younger than 10 survived the ship sinking, whereas most males age 10 or older did not survive the ship sinking. Let's continue to build on our previous prediction: If a passenger was female, then we will predict they survive. If a passenger was male and younger than 10, then we will also predict they survive. Otherwise, we will predict they do not survive.
Fill in the missing code below so that the function will make this prediction.
Hint: You can start your implementation of this function using the prediction code you wrote earlier from predictions_1.
Can you write Python code for it?

def predictions_2(data):
    """ Model with two features:
            - Predict a passenger survived if they are female.
            - Predict a passenger survived if they are male and younger than 10. """

    predictions = []
    for _, passenger in data.iterrows():
        if(passenger['Sex'] == "female"):
            z = 1
        elif(passenger['Sex'] == "male" and passenger["Age"] < 10):
            z = 1
        else:
            z = 0
        predictions.append(z)

    # Return our predictions
    return pd.Series(predictions)

# Make the predictions
predictions = predictions_2(data)


I am working on the file "3_Titanic/titanic_survival_exploration.ipynb".
The first task is:
From a sample of the RMS Titanic data, we can see the various features present for each passenger on the ship:
- Survived: Outcome of survival (0 = No; 1 = Yes)
- Pclass: Socio-economic class (1 = Upper class; 2 = Middle class; 3 = Lower class)
- Name: Name of passenger
- Sex: Sex of the passenger
- Age: Age of the passenger (Some entries contain NaN)
- SibSp: Number of siblings and spouses of the passenger aboard
- Parch: Number of parents and children of the passenger aboard
- Ticket: Ticket number of the passenger
- Fare: Fare paid by the passenger
- Cabin Cabin number of the passenger (Some entries contain NaN)
- Embarked: Port of embarkation of the passenger (C = Cherbourg; Q = Queenstown; S = Southampton)
Since we're interested in the outcome of survival for each passenger or crew member, we can remove the Survived feature from this dataset and store it as its own separate variable outcomes. We will use these outcomes as our prediction targets.
Run the code cell below to remove Survived as a feature of the dataset and store it in outcomes.
Can you write Python code for it?

# Store the 'Survived' feature in a new variable and remove it from the dataset
outcomes = full_data['Survived']
data = full_data.drop('Survived', axis = 1)

# Show the new dataset with 'Survived' removed
display(data.head())


I am working on the file "3_Titanic/titanic_survival_exploration.ipynb".
The first task is:
Question 1
Using the RMS Titanic data, how accurate would a prediction be that none of the passengers survived?
Hint: Run the code cell below to see the accuracy of this prediction.
Can you write Python code for it?


I am working on the file "3_Titanic/titanic_survival_exploration.ipynb".
The first task is:
Tip: If you save an iPython Notebook, the output from running code blocks will also be saved. However, the state of your workspace will be reset once a new session is started. Make sure that you run all of the code blocks from your previous session to reestablish variables and functions before picking up where you last left off.

Making Predictions
If we were asked to make a prediction about any passenger aboard the RMS Titanic whom we knew nothing about, then the best prediction we could make would be that they did not survive. This is because we can assume that a majority of the passengers (more than 50%) did not survive the ship sinking.
The predictions_0 function below will always predict that a passenger did not survive.
Can you write Python code for it?


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
1. Corpus acquisition.
In these notebooks we will explore some tools for text processing and analysis and two topic modeling algorithms available from Python toolboxes.
To do so, we will explore and analyze collections of Wikipedia articles from a given category, using wikitools, that makes easy the capture of content from wikimedia sites.
(As a side note, there are many other available text collections to test topic modelling algorithm. In particular, the NLTK library has many examples, that can explore them using the nltk.download() tool.
import nltk
nltk.download()

for instance, you can take the gutemberg dataset
Mycorpus = nltk.corpus.gutenberg
text_name = Mycorpus.fileids()[0]
raw = Mycorpus.raw(text_name)
Words = Mycorpus.words(text_name)

Also, tools like Gensim or Sci-kit learn include text databases to work with).
In order to use Wikipedia data, we will select a single category of articles:
Can you write Python code for it?

site = wiki.Wiki("https://en.wikipedia.org/w/api.php")
# Select a category with a reasonable number of articles (>100)
cat = "Economics"
# cat = "Pseudoscience"
print cat


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
2.4. Saving results
The dictionary D and the Bag of Words in corpus_bow are the key inputs to the topic model algorithms analyzed in the following notebook. Save them to be ready to use them during the next session.
Can you write Python code for it?

import pickle
data = {}
data['D'] = D
data['corpus_bow'] = corpus_bow
pickle.dump(data, open("wikiresults.p", "wb"))


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
At this point, it is good to make sure to understand what has happened. In corpus_clean we had a list of token lists. With it, we have constructed a Dictionary, D, which assign an integer identifier to each token in the corpus.
After that, we have transformed each article (in corpus_clean) in a list tuples (id, n).
Can you write Python code for it?

print "Original article (after cleaning): "
print corpus_clean[0][0:30]
print "Sparse vector representation (first 30 components):"
print corpus_bow[0][0:30]
print "The first component, {0} from document 0, states that token 0 ({1}) appears {2} times".format(
    corpus_bow[0][0], D[0], corpus_bow[0][0][1])


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
Before starting with the semantic analyisis, it is interesting to observe the token distribution for the given corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (I):
# Create a "flat" corpus with all tuples in a single list
corpus_bow_flat = [item for sublist in corpus_bow for item in sublist]

# Initialize a numpy array that we will use to cont tokens.
# token_count[n] should store the number of ocurrences of the n-th token, D[n]
token_count = np.zeros(n_tokens)

# Count the number of occurrences of each token.
for x in corpus_bow_flat:
    # Update the proper element in token_count
    # scode: <FILL IN>

# Sort by decreasing number of occurences
ids_sorted = np.argsort(- token_count)
tf_sorted = token_count[ids_sorted]


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
Exercise  (bigrams): nltk provides an utility to compute n-grams from a list of tokens, in nltk.util.ngrams. Join all tokens in corpus_clean in a single list and compute the bigrams. Plot the 20 most frequent bigrams in the corpus.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>
# Check the code below to see how ngrams works, and adapt it to solve the exercise.
# from nltk.util import ngrams
# sentence = 'this is a foo bar sentences and i want to ngramize it'
# sixgrams = ngrams(sentence.split(), 2)
# for grams in sixgrams:
#     print grams


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
Exercise (All in one): Note that, for pedagogical reasons, we have used a different for loop for each text processing step creating a new corpus_xxx variable after each step. For very large corpus, this could cause memory problems.
As a summary exercise, repeat the whole text processing, starting from corpus_text up to computing the bow, with the following modifications:

Use a single for loop, avoiding the creation of any intermediate corpus variables.
Use lemmatization instead of stemming.
Remove all tokens appearing in only one document and less than 2 times.
Save the result in a new variable corpus_bow1.
Can you write Python code for it?


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
Exercise: Count the number of tokens appearing only in a single article.
Can you write Python code for it?


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
Exercise: There are usually many tokens that appear with very low frequency in the corpus. Count the number of tokens appearing only once, and what is the proportion of them in the token list.
Can you write Python code for it?

# scode: cold_tokens = <FILL IN>

print "There are {0} cold tokens, which represent {1}% of the total number of tokens in the dictionary".format(
    len(cold_tokens), float(len(cold_tokens))/n_tokens*100)


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
In the following we plot the most frequent terms in the corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

# Example data
n_bins = 25
hot_tokens = [D[i] for i in ids_sorted[n_bins-1::-1]]
y_pos = np.arange(len(hot_tokens))
z = tf_sorted[n_bins-1::-1]/n_art

plt.barh(y_pos, z, align='center', alpha=0.4)
plt.yticks(y_pos, hot_tokens)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()

# SORTED TOKEN FREQUENCIES:

# Example data
plt.semilogy(tf_sorted)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
Now, we have stored the whole text collection in two lists:

corpus_titles, which contains the titles of the selected articles
corpus_text, with the text content of the selected wikipedia articles

You can browse the content of the wikipedia articles to get some intuition about the kind of documents that will be processed.
Can you write Python code for it?

# n = 5
# print corpus_titles[n]
# print corpus_text[n]


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
Task: Apply the .lemmatize() method, from the WordNetLemmatizer object created in the first line, to corpus_filtered.
Can you write Python code for it?

wnl = WordNetLemmatizer()

# Select stemmer.
corpus_lemmat = []

for n, token_list in enumerate(corpus_filtered):
    print "\rLemmatizing article {0} out of {1}".format(n + 1, n_art),

    # scode: lemmat_tokens = <FILL IN>

    # Add art to the stemmed corpus
    # scode: <FILL IN>

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_lemmat[0][0:30]


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
Task: Insert the appropriate call to word_tokenize in the code below, in order to get the tokens list corresponding to each Wikipedia article:
Can you write Python code for it?

corpus_tokens = []

for n, art in enumerate(corpus_text):
    print "\rTokenizing article {0} out of {1}".format(n + 1, n_art),
    # This is to make sure that all characters have the appropriate encoding.
    art = art.decode('utf-8')

    # Tokenize each text entry.
    # scode: tokens = <FILL IN>

    # Add the new token list as a new element to corpus_tokens (that will be a list of lists)
    # scode: <FILL IN>

print "\n The corpus has been tokenized. Let's check some portion of the first article:"
print corpus_tokens[0][0:30]

Test.assertEquals(len(corpus_tokens), n_art, "The number of articles has changed unexpectedly")
Test.assertTrue(len(corpus_tokens) >= 100,
                "Your corpus_tokens has less than 100 articles. Consider using a larger dataset")


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
You can try with any other categories. Take into account that the behavior of topic modelling algorithms may depend on the amount of documents available for the analysis. Select a category with at least 100 articles. You can browse the wikipedia category tree here, https://en.wikipedia.org/wiki/Category:Contents, for instance.
We start downloading the text collection.
Can you write Python code for it?

# Loading category data. This may take a while
print "Loading category data. This may take a while..."
cat_data = category.Category(site, cat)

corpus_titles = []
corpus_text = []

for n, page in enumerate(cat_data.getAllMembersGen()):
    print "\r Loading article {0}".format(n + 1),
    corpus_titles.append(page.title)
    corpus_text.append(page.getWikiText())

n_art = len(corpus_titles)
print "\nLoaded " + str(n_art) + " articles from category " + cat


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
and a bow representation of a corpus with
Can you write Python code for it?

print "{0} Wikipedia articles".format(len(corpus_bow))


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
ids_sorted is a list of all token ids, sorted by decreasing number of occurrences in the whole corpus. For instance, the most frequent term is
Can you write Python code for it?


I am working on the file "3_TopicModeling/TM1_NLP_student.ipynb".
The first task is:
which appears
Can you write Python code for it?

print "{0} times in the whole corpus".format(tf_sorted[0])


I am working on the file "3_TopicModeling/TM2_Topic_Modeling_student.ipynb".
The first task is:
3. Semantic Analysis
The dictionary D and the Bag of Words in corpus_bow are the key inputs to the topic model algorithms. The topic model algorithms in gensim assume that input documents are parameterized using the tf-idf model.
Can you write Python code for it?


I am working on the file "3_TopicModeling/TM2_Topic_Modeling_student.ipynb".
The first task is:
3.1. Latent Semantic Indexing (LSI)
Now we are ready to apply a topic modeling algorithm. Latent Semantic Indexing is provided by LsiModel.
Task: Generate a LSI model with 5 topics for corpus_tfidf and dictionary D. You can check de sintaxis for gensim.models.LsiModel.
Can you write Python code for it?

# Initialize an LSI transformation
n_topics = 5
# scode: lsi = <FILL IN>


I am working on the file "3_TopicModeling/TM2_Topic_Modeling_student.ipynb".
The first task is:
3.2.2. LDA using Sci-kit Learn
The input matrix to the sklearn implementation of LDA contains the token-counts for all documents in the corpus.
sklearn contains a powerfull CountVectorizer method that can be used to construct the input matrix from the corpus_bow.
First, we will define an auxiliary function to print the top tokens in the model, that has been taken from the sklearn documentation.
Can you write Python code for it?

# Adapted from an example in sklearn site
# http://scikit-learn.org/dev/auto_examples/applications/topics_extraction_with_nmf_lda.html

# You can try also with the dataset provided by sklearn in
# from sklearn.datasets import fetch_20newsgroups
# dataset = fetch_20newsgroups(shuffle=True, random_state=1,
#                              remove=('headers', 'footers', 'quotes'))

def print_top_words(model, feature_names, n_top_words):
    for topic_idx, topic in enumerate(model.components_):
        print("Topic #%d:" % topic_idx)
        print(" ".join([feature_names[i]
                        for i in topic.argsort()[:-n_top_words - 1:-1]]))
    print()

    " ".join(ListaTokens)


I am working on the file "3_TopicModeling/TM2_Topic_Modeling_student.ipynb".
The first task is:
From LSI, we can check both the topic-tokens matrix and the document-topics matrix.
Now we can check the topics generated by LSI. An intuitive visualization is provided by the show_topics method.
Can you write Python code for it?

lsi.show_topics(num_topics=-1, num_words=10, log=False, formatted=True)


I am working on the file "3_TopicModeling/TM2_Topic_Modeling_student.ipynb".
The first task is:
However, a more useful representation of topics is as a list of tuples (token, value). This is provided by the show_topic method.
Task: Represent the columns of the topic-token matrix as a series of bar diagrams (one per topic) with the top 25 tokens of each topic.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

n_bins = 25

# Example data
y_pos = range(n_bins-1, -1, -1)

pylab.rcParams['figure.figsize'] = 16, 8  # Set figure size

for i in range(n_topics):

    ### Plot top 25 tokens for topic i
    # Read i-thtopic
    # scode: <FILL IN>
    topic_i = lsi.show_topic(i, topn=n_bins)
    tokens = [t[0] for t in topic_i]
    weights = [t[1] for t in topic_i]

    # Plot
    # scode: <FILL IN>
    plt.subplot(1, n_topics, i+1)
    plt.barh(y_pos, weights, align='center', alpha=0.4)
    plt.yticks(y_pos, tokens)
    plt.xlabel('Top {0} topic weights'.format(n_bins))
    plt.title('Topic {0}'.format(i))

plt.show()


I am working on the file "3_TopicModeling/TM2_Topic_Modeling_student.ipynb".
The first task is:
LSI approximates any document as a linear combination of the topic vectors. We can compute the topic weights for any input corpus entered as input to the lsi model.
Can you write Python code for it?

# On real corpora, target dimensionality of
# 200–500 is recommended as a “golden standard”
# Create a double wrapper over the original
# corpus bow  tfidf  fold-in-lsi
corpus_lsi = lsi[corpus_tfidf]
print corpus_lsi[0]


I am working on the file "3_TopicModeling/TM2_Topic_Modeling_student.ipynb".
The first task is:
Or to apply a transformation to a whole corpus
Can you write Python code for it?


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
A loop becomes infinite loop if a condition never becomes FALSE. You must use caution when using while loops because of the possibility that this condition never resolves to a FALSE value. This results in a loop that never ends. Such a loop is called an infinite loop.
An infinite loop might be useful in client/server programming where the server needs to run continuously so that client programs can communicate with it as and when required.
for Loop
Executes a sequence of statements multiple times and abbreviates the code that manages the loop variable.
Can you write Python code for it?

fruits = ['banana', 'apple',  'mango']
for fruit in fruits:        # Second Example
    print('Current fruit :', fruit)


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
Finding the total number of items in the dictionary:
Can you write Python code for it?


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
Flow Control
In general, statements are executed sequentially: The first statement in a function is executed first, followed by the second, and so on. There may be a situation when you need to execute a block of code several number of times. In Python a block is delimitated by intendation, i.e. all lines starting at the same space are one block.
Programming languages provide various control structures that allow for more complicated execution paths.
A loop statement allows us to execute a statement or group of statements multiple times.
while Loop
Repeats a statement or group of statements while a given condition is TRUE. It tests the condition before executing the loop body.
Can you write Python code for it?

count = 0
while (count < 9):
    print('The count is: ' + str(count))
    count += 1

print('Good bye!')


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
If this should not make sense, you can print some documentation:
Can you write Python code for it?


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
In principle one could also iterate over an index going from 0 to the number of elements:
Can you write Python code for it?

for index in range(len(fruits)):
    print('Current fruit:', fruits[index])


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
Nested Loops
Python programming language allows to use one loop inside another loop.
A final note on loop nesting is that you can put any type of loop inside of any other type of loop. For example a for loop can be inside a while loop or vice versa.
Can you write Python code for it?

for x in range(1, 3):
    for y in range(1, 4):
        print(f'{x} * {y} = {x*y}')


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
Numbers
Can you write Python code for it?

abs(-1)

import math

math.floor(4.5)

math.exp(1)

math.log(1)

math.log10(10)

math.sqrt(9)

round(4.54,1)


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
Produces a printable string representation of a dictionary:
Can you write Python code for it?


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
Slicing
Can you write Python code for it?

print(string)
print(string[0])
print(string[2:5])
print(string[2:])
print(string[:5])
print(string * 2)
print(string + 'TEST')
print(string[-1])


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
String Operations
Can you write Python code for it?

print(string/2)

print(string - 'TEST')

print(string**2)


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
Strings
Can you write Python code for it?

string = 'Hello World!'
string2 = "This is also allowed, helps if you want 'this' in a string and vice versa"
len(string)


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
This is equivalent to the following loop:
Can you write Python code for it?

fruits_with_b = []
for fruit in fruits:
    if fruit.startswith('b'):
        fruits_with_b.append(fruit)
fruits_with_b


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
Tuples
Tuples are immutable and can be thought of as read-only lists.
Can you write Python code for it?

y_tuple = ('john', '2.', '1')
type(y_tuple)

y_list

y_list[0] = 'Erik'
y_list

y_tuple[0] = 'Erik'


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
When duplicate keys encountered during assignment, the last assignment wins
Can you write Python code for it?

dic = {'Name': 'Zara', 'Age': 7, 'Name': 'Manni'}
dic


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
continue
The continue statement rejects all the remaining statements in the current iteration of the loop and moves the control back to the top of the loop (like a "skip").
The continue statement can be used in both while and for loops.
Can you write Python code for it?

for letter in 'Python':
    if letter == 'h':
        continue
    print('Current Letter: ' + letter)


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
for loops can be elegantly integrated for creating lists
Can you write Python code for it?

fruits_with_b = [fruit for fruit in fruits if fruit.startswith('b')]
fruits_with_b


I am working on the file "3_Types_Functions_FlowControl.ipynb".
The first task is:
if
The if statement will evaluate the code only if a given condition is met (used with logical operator such as ==, <, >, =<, =>, not, is, in, etc.
Optionally we can introduce a elsestatement to execute an alternative code when the condition is not met.
Can you write Python code for it?

x = 'Mark'

if x in ['Mark', 'Jack', 'Mary']:
    print('present!')
else:
    print('absent!')

x = 'Tom'

if x in ['Mark', 'Jack', 'Mary']:
    print('present!')
else:
    print('absent!')


I am working on the file "3_data_visualization.ipynb".
The first task is:
1. Confronto Salvini, Renzi e M5S
Can you write Python code for it?

# Dimensione Temporale -> ANNO
df_posts_m5s['Post_Date'] = df_posts_m5s['Post_Date'].str[:4]
df_posts_salvini['Post_Date'] = df_posts_m5s['Post_Date'].str[:4]
df_posts_renzi['Post_Date'] = df_posts_m5s['Post_Date'].str[:4]

df_posts_m5s = df_posts_m5s.groupby('Post_Date',as_index=False).agg({'ID':'count', 'Likes': 'sum'})
df_posts_salvini = df_posts_salvini.groupby('Post_Date',as_index=False).agg({'ID':'count', 'Likes': 'sum'})
df_posts_renzi = df_posts_renzi.groupby('Post_Date',as_index=False).agg({'ID':'count', 'Likes': 'sum'})

df_posts_m5s.rename(columns={'ID': 'Posts_M5S', 'Likes': 'Likes_M5S'}, inplace=True)
df_posts_m5s = df_posts_m5s.set_index(['Post_Date'])
df_posts_m5s.head(2)

df_posts_renzi.rename(columns={'ID': 'Posts_Renzi', 'Likes': 'Likes_Renzi'}, inplace=True)
df_posts_renzi = df_posts_renzi.set_index(['Post_Date'])
df_posts_renzi.head(2)

df_posts_salvini.rename(columns={'ID': 'Posts_Salvini', 'Likes': 'Likes_Salvini'}, inplace=True)
df_posts_salvini = df_posts_salvini.set_index(['Post_Date'])
df_posts_salvini.head(2)

# Numero Posts
result_post = pd.concat([df_posts_renzi, df_posts_salvini, df_posts_m5s], axis=1)
del result_post['Likes_Renzi']
del result_post['Likes_Salvini']
del result_post['Likes_M5S']
result_post.rename(columns={'Posts_Renzi': 'Renzi', 'Posts_Salvini': 'Salvini', 'Posts_M5S': 'M5S'}, inplace=True)

result_post.plot(
        kind='bar'
)

result_post.to_csv(os.path.join(dir_out,r'Distr_Posts.csv'),header=True, index=True)

# Numero Likes
result_likes = pd.concat([df_posts_renzi, df_posts_salvini, df_posts_m5s], axis=1)
del result_likes['Posts_Renzi']
del result_likes['Posts_Salvini']
del result_likes['Posts_M5S']
result_likes.rename(columns={'Likes_Renzi': 'Renzi', 'Likes_Salvini': 'Salvini', 'Likes_M5S': 'M5S'}, inplace=True)

result_likes.plot(
        kind='bar'
)

result_likes.to_csv(os.path.join(dir_out,r'Distr_Likes.csv'),header=True, index=True)


I am working on the file "3_data_visualization.ipynb".
The first task is:
2. Dettaglio Salvini
Can you write Python code for it?

# Dataset Salvini
df_filename = r'df_posts_likes_salvini.pkl'
df_fullpath = os.path.join(dir_df, df_filename)
df_posts = pd.read_pickle(df_fullpath)

# Estraggo la Data da Str
df_posts['Post_Date'] = df_posts['Post_Date'].str[:10]

# Converto in Date
df_posts['Post_Date'] = pd.to_datetime(df_posts['Post_Date'])

# Ordino per Data
df_posts = df_posts.sort_values(by='Post_Date')

# Mi tengo DS totale per Data per analisi successive
df_posts_dett = df_posts
df_posts_dett = df_posts_dett.set_index(['Post_Date'])

# Raggruppo per Data
df_posts = df_posts.groupby('Post_Date',as_index=False).agg({'ID':'count', 'Likes': 'sum'})

# Elimino i le date per cui non ho post con likes (privacy ?)
df_posts = df_posts[np.isfinite(df_posts['Likes'])]

# Setto indice la Data
df_posts = df_posts.set_index(['Post_Date'])

# Lavoro con TimeSeries, raggruppo tutto per Anno/Mese (la data era per giorno)
df_posts = df_posts.groupby(pd.TimeGrouper("M")).sum()

# Elimino Numero di Posts
del df_posts['ID']

# Ok, i numeri tornano dopo le elaborazioni
df_posts['Likes'].sum()


I am working on the file "3_data_visualization.ipynb".
The first task is:
2. Focus su Anni 2014, 2015, 2016
Can you write Python code for it?

df_post_14 = df_posts['20140101':'20141231']
tp_14 = df_post_14.plot()
fig_posts_14 = tp_14.get_figure()
fig_posts_14.tight_layout()
fig_posts_14.savefig(os.path.join(dir_out,'posts_2014.png'), format='png', dpi=300)

# Dettaglio 2014
df_posts_dett['20140101':'20141231'].sort_values(by=['Likes'],ascending=False).head(5)
# Analizzo gli ID direttamente dall API Graph Tool di Facebook


I am working on the file "3_data_visualization.ipynb".
The first task is:
Principali post tra Ottobre e Novembre

(90.000 likes) Uno STUPRATORE tunisino di 28 anni, già in galera per violenza sessuale, è evaso dal carcere di Pordenone e ha violentato una ragazza di 28 anni. È stato arrestato. Fossi ministro, applicherei (come già sperimentato in numerosi Paesi europei) la CASTRAZIONE CHIMICA e poi lo rimanderei in Tunisia. Che dite?",
Can you write Python code for it?

df_post_15 = df_posts['20150101':'20151231']
tp_15 = df_post_15.plot()
fig_posts_15 = tp_15.get_figure()
fig_posts_15.tight_layout()
fig_posts_15.savefig(os.path.join(dir_out,'posts_2015.png'), format='png', dpi=300)

# Controllo 2015
df_posts_dett['20150101':'20151231'].sort_values(by=['Likes'],ascending=False).head(10)
# Analizzo gli ID direttamente dall API Graph Tool di Facebook


I am working on the file "3_matplotlib.ipynb".
The first task is:
A better way...
A much better way to "build" plots is to use a more object-oriented approach.  In this case, we create the objects and manipulate them which allows us to have more control over how we create plots.  Here we will create the annotated plots so that we have examples on how to do the same things (notice that they are very similar).  The basic premise of this approach is that you generate objects that can be manipulated and remain persistent.
Can you write Python code for it?

x = numpy.linspace(-5, 5, 100)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
growth_curve = axes.plot(x, x**2 + 2 * x + 3, 'r--')
axes.set_xlabel("time (s)")
axes.set_ylabel("Tribble Population")
axes.set_title("Tribble Growth")
axes.set_xlim([0, 4])
axes.set_ylim([0, 25])


I am working on the file "3_matplotlib.ipynb".
The first task is:
Here is the other example with a few extra tricks added in.
Can you write Python code for it?

x = numpy.linspace(-1, 1, 100)
y = numpy.linspace(-1, 1, 100)
X, Y = numpy.meshgrid(x, y)

fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1, aspect='equal')
tribble_density = axes.pcolor(X, Y, numpy.sin(X**2) + numpy.cos(Y**2), cmap=plt.get_cmap("Oranges"))
axes.set_xlabel("x (km)")
axes.set_ylabel("y (km)")
axes.set_title("Tribble Density ($N/km^2$)")
cbar = fig.colorbar(tribble_density, ax=axes)
cbar.set_label("$1/km^2$")

axes = fig.add_subplot(1, 2, 2, aspect='equal')
klingon_population_density = axes.pcolor(X, Y, X * Y + 1, cmap=plt.get_cmap("RdBu"))
axes.set_xlabel("x (km)")
axes.set_ylabel("y (km)")
axes.set_title("Klingon Population ($N$)")
cbar = fig.colorbar(klingon_population_density, ax=axes)
cbar.set_label("Number (thousands)")

plt.show()


I am working on the file "3_matplotlib.ipynb".
The first task is:
Here we have used a new way to create NumPy ndarrays so that we can easily evaluate 2-dimensional functions.  The function meshgrid takes two 1-dimensional arrays and turns them into 2, 2-dimensional arrays whose matching indices will provide an easy to use set of arrays that can be evaluate 2-dimensional functions.
Now as pretty as this rainbow plot is, it does not actually communicate as effectively as it could.  This default colormap (a colormap is how we decide which values are colored which way) called "jet" is fairly ubiquitous in scientific and engineering visualization.  Unfortunately the jet colormap tends to emphasize only certain parts of the range of values.  I highly suggest looking at this explanation of how to choose a colormap if you are interested.  The bottom line is, do not use jet if you can help it.
Anyway, let us make this plot also more useful.
Can you write Python code for it?

color_map = plt.get_cmap("Oranges")
plt.gcf().set_figwidth(plt.gcf().get_figwidth() * 2)
plt.subplot(1, 2, 1, aspect="equal")
plt.pcolor(X, Y, F, cmap=color_map)

plt.colorbar()
plt.xlabel("x (m)")
plt.ylabel("y (m)")
plt.title("Tribble Density ($N/m^2$)")

plt.subplot(1, 2, 2, aspect="equal")
plt.pcolor(X, Y, X*Y, cmap=plt.get_cmap("RdBu"))
plt.xlabel("x (m)")
plt.ylabel("y (m)")
plt.title("Klignon Population ($N$)")
plt.colorbar()

plt.autoscale(enable=True, tight=False)
plt.show()


I am working on the file "3_matplotlib.ipynb".
The first task is:
The othe major way to plot data is via a pseudo-color plot.  A pseudo-color plot takes 3-dimensional data (or 2-dimensional depending on how you look at it) and plots it using color to represent the values of the function.  Say we have a function $f(x, y)$, we can plot a pseudo-color by using the following commands.
Can you write Python code for it?

x = numpy.linspace(-1, 1, 100)
y = numpy.linspace(-1, 1, 100)
X, Y = numpy.meshgrid(x, y)
F = numpy.sin(X**2) + numpy.cos(Y**2)
plt.pcolor(X, Y, F)
plt.show()


I am working on the file "3_matplotlib.ipynb".
The first task is:
We can also label our plot (please do this), set the bounds and change the style of the plot:
Can you write Python code for it?

plt.plot(x, y, 'r--')
plt.xlabel("time (s)")
plt.ylabel("Tribble Population")
plt.title("Tribble Growth")
plt.xlim([0, 4])
plt.ylim([0, 25])


I am working on the file "3_model selection_evalutation.ipynb".
The first task is:
(2) max features
Can you write Python code for it?

recall_range = []
max_features_range = []

for i in np.arange(1, 15, 1):
    clf_RF = RandomForestClassifier(oob_score=True, n_estimators=18, max_features=i).fit(X_train, y_train)
    clf_RF_predicted = clf_RF.predict(X_test)
    recall = round(recall_score(clf_RF_predicted, y_test), 2)

    max_features_range.append(i)
    recall_range.append(recall)
    dictionary = dict(zip(max_features_range, recall_range))

plt.figure(figsize=(10, 3))
plt.plot(max_features_range, recall_range, color='#EA5959', label='max recall: %(n)0.2f \n%(s)s: %(v)2d' %
         {'n':max(dictionary.values()), 's':'max features', 'v':max(dictionary, key=lambda i: dictionary[i])})
plt.scatter([max(dictionary, key=lambda i: dictionary[i]), ], [max(dictionary.values()), ], 80, color='#EA5959')
axhline(max(dictionary.values()), color='#EA5959', linewidth=1, linestyle='--')
axvline(max(dictionary, key=lambda i: dictionary[i]), color='#EA5959', linewidth=1, linestyle='--')
plt.legend(loc='lower right', prop={'size':12})
plt.xlim(min(max_features_range), max(max_features_range))
plt.ylim(min(recall_range)*0.98, max(recall_range)*1.02)
plt.ylabel('Recall')
plt.xlabel('max features');


I am working on the file "3_model selection_evalutation.ipynb".
The first task is:
(3) min sample leaf
Can you write Python code for it?

recall_range = []
min_samples_leaf_range = []

for i in np.arange(1, 20, 1):
    clf_RF = RandomForestClassifier(oob_score=True, n_estimators=18, max_features=14, min_samples_leaf=i).fit(X_train, y_train)
    clf_RF_predicted = clf_RF.predict(X_test)
    recall = round(recall_score(clf_RF_predicted, y_test), 2)

    min_samples_leaf_range.append(i)
    recall_range.append(recall)
    dictionary = dict(zip(min_samples_leaf_range, recall_range))

plt.figure(figsize=(10, 3))
plt.plot(min_samples_leaf_range, recall_range, color='#EA5959', label='max recall: %(n)0.2f \n%(s)s: %(v)2d' %
         {'n':max(dictionary.values()), 's':'min samples leaf', 'v':max(dictionary, key=lambda i: dictionary[i])})
plt.scatter([max(dictionary, key=lambda i: dictionary[i]), ], [max(dictionary.values()), ], 80, color='#EA5959')
axhline(max(dictionary.values()), color='#EA5959', linewidth=1, linestyle='--')
axvline(max(dictionary, key=lambda i: dictionary[i]), color='#EA5959', linewidth=1, linestyle='--')
plt.legend(loc='lower right', prop={'size':12})
plt.xlim(min(min_samples_leaf_range), max(min_samples_leaf_range))
plt.ylim(min(recall_range)*0.98, max(recall_range)*1.02)
plt.ylabel('Recall')
plt.xlabel('min_samples_leaf_range');


I am working on the file "3_model selection_evalutation.ipynb".
The first task is:
2) Grid Search

compared to the above, performance is better (0.75/1.0)
Can you write Python code for it?

clf_RF = RandomForestClassifier(n_estimators=10, max_features=19, min_samples_leaf=27, criterion='entropy', oob_score=True).fit(X_train, y_train)
clf_RF_predicted = clf_RF.predict(X_test)

from sklearn.metrics import classification_report, confusion_matrix
target_names = ['not helpful', 'helpful']
print(classification_report(y_test, clf_RF_predicted, target_names=target_names))

plt.figure(figsize=(4, 4))
cm = confusion_matrix(y_test, clf_RF_predicted)
print(cm)
target_names = ['not helpful', 'helpful']
plt.grid(False)
plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
plt.title('Confusion matrix')
plt.colorbar()
tick_marks = np.arange(len(target_names))
plt.xticks(tick_marks, target_names, rotation=45)
plt.yticks(tick_marks, target_names)
plt.tight_layout()
plt.ylabel('True label')
plt.xlabel('Predicted label');


I am working on the file "3_model selection_evalutation.ipynb".
The first task is:
2) Grid Search

very efficient way to find optimal parameter values
assigning 'appropriate' value range seems essential in grid search. For instance, I set the value range for 'max_features' to 10 to 28, since I wanted the model to be robust, at the expense of performance to some extent.
Can you write Python code for it?

from sklearn.pipeline import Pipeline
pipeline_clf_train = Pipeline(
    steps=[
        ('clf_RF', RandomForestClassifier()),
    ]
);

from sklearn.grid_search import GridSearchCV
parameters = {
    'clf_RF__min_samples_leaf' : np.arange(1, 28, 1),
    'clf_RF__max_features' : np.arange(10, 28, 1),
    'clf_RF__criterion' :['gini', 'entropy'],
    'clf_RF__n_estimators' : [10],
    #'clf_RF__oob_score' : ['True]
}
gs_clf = GridSearchCV(pipeline_clf_train, parameters, n_jobs=-1, scoring='recall')
gs_clf = gs_clf.fit(X_train, y_train)

best_parameters, score, _ = max(gs_clf.grid_scores_, key=lambda x: x[1])
for param_name in sorted(parameters.keys()):
    print("%s: %r" % (param_name, best_parameters[param_name]))
print('------------------------------')
print('recall score :', score.round(2))


I am working on the file "3_model selection_evalutation.ipynb".
The first task is:
2) Grid Search
Can you write Python code for it?

clf_RF = RandomForestClassifier(n_estimators=10, max_features=19, min_samples_leaf=27, criterion='entropy', oob_score=True).fit(X_train, y_train)
clf_RF_predicted = clf_RF.predict(X_test)

fpr_rf, tpr_rf, thresholds_rf = roc_curve(y_test, clf_RF.predict_proba(X_test)[:, 0], pos_label=0)
fpr_base, tpr_base, thresholds_base = roc_curve(y_test,clf_Dummy.predict_proba(X_test)[:, 0], pos_label=1)

plt.figure(figsize=(5, 5))
plt.plot(fpr_rf, tpr_rf, color='#E45A84', linewidth=3, linestyle='-',
         label = 'random forest: %(performance)0.2f' % {'performance':auc(fpr_rf, tpr_rf)})
plt.plot(fpr_base, tpr_base, color='#FFACAC', linewidth=2, linestyle='--',
         label = 'baseline: %(performance)0.2f' % {'performance':auc(fpr_base, tpr_base)})
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate (Fall-Out)')
plt.ylabel('True Positive Rate (Recall)')
plt.title('ROC (Receiver operating characteristic)', fontdict={'fontsize': 12})
plt.legend(loc="lower right");


I am working on the file "3_model selection_evalutation.ipynb".
The first task is:
2. Take a look at confusion matrix (what is confusion matrix?)
1) Greedy method

focal point of confusion matrix in this project is 'helpful' + 'recall'
with parameters obtained from greed method, I got 0.68/1.0 meaning that out of 129 'helpful reviews' I predicted 88 reviews correctly.
Can you write Python code for it?

clf_RF = RandomForestClassifier(n_estimators=18, max_features=14, min_samples_leaf=9, oob_score=True).fit(X_train, y_train)
clf_RF_predicted = clf_RF.predict(X_test)

from sklearn.metrics import classification_report, confusion_matrix
target_names = ['not helpful', 'helpful']
print(classification_report(y_test, clf_RF_predicted, target_names=target_names))

plt.figure(figsize=(4,4))
cm = confusion_matrix(y_test, clf_RF_predicted)
print(cm)
target_names = ['not helpful', 'helpful']
plt.grid(False)
plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
plt.title('Confusion matrix')
plt.colorbar()
tick_marks = np.arange(len(target_names))
plt.xticks(tick_marks, target_names, rotation=45)
plt.yticks(tick_marks, target_names)
plt.tight_layout()
plt.ylabel('True label')
plt.xlabel('Predicted label');


I am working on the file "3_model selection_evalutation.ipynb".
The first task is:
RandomForest Parameter Tuning
=================================================================================================================
* Find optimal parameters through grid search and compare them with those obtained by greedy method
* Among numerous scoring types such as recall, accuracy, precision, f1 score, etc, this project will focus on recall-related scores.
* This is because, I believe, high 'true positive rate (tpr)' can be the clue to solve the problem I defined in this project
Can you write Python code for it?

# baseline confirmation, implying that model has to perform at least as good as it
from sklearn.dummy import DummyClassifier
clf_Dummy = DummyClassifier(strategy='most_frequent')
clf_Dummy = clf_Dummy.fit(X_train, y_train)
print('baseline score =>', round(clf_Dummy.score(X_test, y_test), 2))


I am working on the file "3_openai_gym/01_openai_gym_introduction.ipynb".
The first task is:
Descrete: 非負の整数 {0, 1, 2, ..., n-1}
Can you write Python code for it?


I am working on the file "3_openai_gym/01_openai_gym_introduction.ipynb".
The first task is:
Step 2: ランダムな行動選択（ランダム方策）
'CartPole-v0' という環境に対して、ランダムな行動を取ってみる（1000ステップ）。
Can you write Python code for it?

import time

# gym オープンソースライブラリの読み込み
import gym

# 環境を作る
env = gym.make('CartPole-v0')    # 'CartPole-v0' は環境ID
#env = gym.make('MountainCar-v0')    # 'MountainCar-v0'という別の環境
#env = gym.make('MsPacman-v0')    # 'MsPacman-v0'という別の環境

# ランダムな行動選択
env.action_space.sample()

# 行動空間（エージェントが選択可能な行動が定義されている空間）
env.action_space

# 環境の初期化（最初の観測が得られる）
env.reset()

for _ in range(100):
    time.sleep(0.1)    # 描画を遅くするために0.1秒スリープ
    env.render()    # 描画
    action = env.action_space.sample()    # ランダムな行動選択
    print(action), # 選択された行動をプリント
    print(env.step(action))    # 選択行動を実行

env.render(close=True) # 画面を閉じる


I am working on the file "3_openai_gym/01_openai_gym_introduction.ipynb".
The first task is:
スペースからサンプリングすることも、ある値がスペースに含まれているか調べることもできる。
Can you write Python code for it?

from gym import spaces

space = spaces.Discrete(8) #  {0, 1, 2, ..., 7}

# サンプリング
x = space.sample()
x

assert space.contains(x)

assert space.n == 8


I am working on the file "3_openai_gym/01_openai_gym_introduction.ipynb".
The first task is:
環境

gymの主な役割は、強化学習で使える多様な環境を提供すること。
OpenAI Gymが提供する環境　https://gym.openai.com/envs
Can you write Python code for it?

from gym import envs

# 使用可能な環境を列挙
envs.registry.all()


I am working on the file "3_openai_gym/01_openai_gym_introduction.ipynb".
The first task is:
観測

ランダムな行動より上手くやりたければ、それぞれの行動が環境にどんな影響を与えるのか知りたいところ。
行動が環境にどんな影響を与えたのかは、環境のstep関数が教えてくれる。
step関数は4つの値を返す
observation (object) 観測
reward  (float) 報酬
done (boolean) エピソードの終了フラグ（Trueになった時がreset()を呼ぶタイミング）
info (dict) サブ情報

done==Trueとなったときにエピソードを終了するのであれば、以下のようなコードになる。
Can you write Python code for it?

import numpy as np
np.set_printoptions(suppress=True)    # Scientific Notation (例 1.0e-0.5)を使わない

all_obs = []

import gym
env = gym.make('CartPole-v0')
for i_episode in range(5):    # 5エピソード回す
    observation = env.reset()    # 環境を初期化し、最初の観測を得る。
    all_obs.append(observation)    # 観測を記録
    for t in range(100):    # 各エピソードの最大ステップ数は100
        env.render()
        print(observation)
        action = env.action_space.sample()    # ランダム方策
        observation, reward, done, info = env.step(action)    # 選択行動の実行
        all_obs.append(observation)    # 観測を記録
        if done:
            print("Episode finished after {} timesteps\n".format(t+1))
            break

env.render(close=True)

%matplotlib inline
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(10, 4))
ax.plot(np.array(all_obs))
ax.legend(['x', 'x_dot', 'theta', 'theta_dot'])


I am working on the file "3_premiers_modeles.ipynb".
The first task is:
Ainsi

lorsque $\mu>\lambda$ la population croît exponentiellement
lorsque $\lambda<\mu$ la population tend exponentiellement vers 0.

On parlera de croissance (ou décroissance) malthusienne.
Lorsque  $\lambda<\mu$ la population décroît exponentiellement vite vers 0 mais à tout instant fini cette population est strictement positive, pourtant si $M=10^3$ et si $x(t)$ descend en dessous de $10^{-3}$ alors $x(t)$ représentera moins d'un individu. Ce point n'est pas cohérent avec l'hypothèse de population grande et donc limite l'intérêt de ce modèle pour les petites tailles de population.
Croissance logistique
\begin{align}
\ce{X -> 2X}\hskip1em \textrm{avec inhibition}
\end{align}
En 1838,
 Pierre François Verhulst (1804-1849)
 proposa un modèle de croissance  dont le taux de croissance diminue linéairement en fonction de la taille de la population rendant ainsi compte de la capacité maximale d'accueil du milieu.
$$
  \dot x(t) = r\times\left(1-\frac{x(t)}{K}\right)\,x(t)\,,\ x(0)=x_0
$$
admet l'unique solution:
$$
  x(t)
  =  K \,\frac{1}{1+\left(\frac {K}{x_{0}} - 1\right) \,e^{-r\,t}}
  =  \frac{1}{\frac{x_0}{K}\,\left(e^{rt} - 1\right)+1}\; x_0\,e^{r\,t}
$$
Can you write Python code for it?

t0, t1 = 0, 10

temps = np.linspace(t0,t1,300, endpoint=True)

population = lambda t: K*1/(1+ (K/x0-1) * np.exp(-r*t))

x0, K = 1, 5
legende = []
for r in [2, 1, 0.5]:
    plt.plot(temps, population(temps))
    legende = legende + [r'$r=$'+str(r),]
plt.ylim([0,K*1.2])
plt.legend(legende, loc='lower right',title=r'taux de croissance $r$')
plt.plot([t0, t1], [K, K], color="k", linestyle='--')
plt.text((t1-t0)/50, K, r"$K$ (capacité d'accueil)",
         verticalalignment='bottom', horizontalalignment='left')
plt.xlabel(r'temps $t$')
plt.ylabel(r'taille $x(t)$ de la population')
plt.show()

from ipywidgets import interact, fixed

def pltlogistique(x0,K,r):
    population2 = K*1/(1+ (K/x0-1) * np.exp(-r*temps))
    plt.plot(temps, population2)
    plt.ylim([0,6])
    plt.plot([t0, t1], [K, K], color="k", linestyle='--')
    plt.show()

interact(pltlogistique, x0=(0.01,6,0.1), K=(0.01,6,0.1), r=(0.1,20,0.1))
plt.show()


I am working on the file "3_premiers_modeles.ipynb".
The first task is:
j'intègre l'EDO
Can you write Python code for it?

a, b, c, d = 0.4, 0.002, 0.001, 0.7
def f(x, t):
    x1, x2 = x
    return [a * x1 - b * x1 * x2,
            c * x1 * x2 - d * x2]

x0 = [600, 400]
t = np.linspace(0, 50, 250)
x_t = odeint(f, x0, t)

%matplotlib inline

plt.plot(t, x_t[:,0], label=r"proies $x_1(t)$")
plt.plot(t, x_t[:,1], label=r"prédateurs $x_2(t)$")
plt.xlabel("temps")
plt.ylabel("nombre d'animaux")
plt.legend()
plt.show()


I am working on the file "3_regularization.ipynb".
The first task is:
First reload the data we generated in notmist.ipynb.
Can you write Python code for it?

pickle_file = 'notMNIST.pickle'

with open(pickle_file, 'rb') as f:
  save = pickle.load(f)
  train_dataset = save['train_dataset']
  train_labels = save['train_labels']
  valid_dataset = save['valid_dataset']
  valid_labels = save['valid_labels']
  test_dataset = save['test_dataset']
  test_labels = save['test_labels']
  del save  # hint to help gc free up memory
  print('Training set', train_dataset.shape, train_labels.shape)
  print('Validation set', valid_dataset.shape, valid_labels.shape)
  print('Test set', test_dataset.shape, test_labels.shape)


pickle_file = 'notMNIST.pickle'

with open(pickle_file, 'rb') as f:
  save = pickle.load(f)
  train_dataset = save['train_dataset']
  train_labels = save['train_labels']
  valid_dataset = save['valid_dataset']
  valid_labels = save['valid_labels']
  test_dataset = save['test_dataset']
  test_labels = save['test_labels']
  del save  # hint to help gc free up memory
  print 'Training set', train_dataset.shape, train_labels.shape
  print 'Validation set', valid_dataset.shape, valid_labels.shape
  print 'Test set', test_dataset.shape, test_labels.shape


I am working on the file "3_regularization.ipynb".
The first task is:
Reformat into a shape that's more adapted to the models we're going to train:
- data as a flat matrix,
- labels as float 1-hot encodings.
Can you write Python code for it?

image_size = 28
num_labels = 10

def reformat(dataset, labels):
  dataset = dataset.reshape((-1, image_size * image_size)).astype(np.float32)
  # Map 2 to [0.0, 1.0, 0.0 ...], 3 to [0.0, 0.0, 1.0 ...]
  labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)
  return dataset, labels
train_dataset, train_labels = reformat(train_dataset, train_labels)
valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)
test_dataset, test_labels = reformat(test_dataset, test_labels)
print('Training set', train_dataset.shape, train_labels.shape)
print('Validation set', valid_dataset.shape, valid_labels.shape)
print('Test set', test_dataset.shape, test_labels.shape)

def accuracy(predictions, labels):
  return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
          / predictions.shape[0])


image_size = 28
num_labels = 10

def reformat(dataset, labels):
  dataset = dataset.reshape((-1, image_size * image_size)).astype(np.float32)
  # Map 2 to [0.0, 1.0, 0.0 ...], 3 to [0.0, 0.0, 1.0 ...]
  labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)
  return dataset, labels
train_dataset, train_labels = reformat(train_dataset, train_labels)
valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)
test_dataset, test_labels = reformat(test_dataset, test_labels)
print 'Training set', train_dataset.shape, train_labels.shape
print 'Validation set', valid_dataset.shape, valid_labels.shape
print 'Test set', test_dataset.shape, test_labels.shape

def accuracy(predictions, labels):
  return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
          / predictions.shape[0])


I am working on the file "3_space_charge.ipynb".
The first task is:
Comparison with ASTRA
Beam tracking with ASTRA was performed by Igor Zagorodnov (DESY).
Can you write Python code for it?

sa, bx_sc, by_sc, bx_wo_sc, by_wo_sc = np.loadtxt("astra_sim.txt", usecols=(0, 1, 2, 3, 4), unpack=True)

s = [tw.s for tw in tws_track]
bx = [tw.beta_x for tw in tws_track]
by = [tw.beta_y for tw in tws_track]
ax = plot_API(lat, legend=False)
ax.plot(s, bx, "r", label="Ocelot, bx")
ax.plot(sa-3.2, bx_sc, "b-",label="ASTRA, bx")
ax.plot(s, by, "r", label="Ocelot, by")
ax.plot(sa-3.2, by_sc, "b-",label="ASTRA, by")
ax.legend()
plt.show()


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Batches
Implement get_batches to create batches of input and targets using int_text.  The batches should be a Numpy array with the shape (number of batches, 2, batch size, sequence length). Each batch contains two elements:
- The first element is a single batch of input with the shape [batch size, sequence length]
- The second element is a single batch of targets with the shape [batch size, sequence length]
If you can't fill the last batch with enough data, drop the last batch.
For exmple, get_batches([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], 2, 3) would return a Numpy array of the following:
```
[
  # First Batch
  [
    # Batch of Input
    [[ 1  2  3], [ 7  8  9]],
    # Batch of targets
    [[ 2  3  4], [ 8  9 10]]
  ],
# Second Batch
  [
    # Batch of Input
    [[ 4  5  6], [10 11 12]],
    # Batch of targets
    [[ 5  6  7], [11 12 13]]
  ]
]
```
Can you write Python code for it?

def get_batches(int_text, batch_size, seq_length):
    """
    Return batches of input and target
    :param int_text: Text with the words replaced by their ids
    :param batch_size: The size of batch
    :param seq_length: The length of sequence
    :return: Batches as a Numpy array
    """

    def parse_text(offset):
        starts = range(offset, len(int_text), seq_length)
        ends = range(offset+seq_length, len(int_text), seq_length)
        return [ int_text[start:end] for start, end in zip(starts,ends) ]

    n_batches = int(len(int_text)/(batch_size*seq_length))
    batches = np.empty((n_batches, 2, batch_size, seq_length))

    inputss = parse_text(0)
    targetss = parse_text(1)

    indices = [(inputs_i, batches_i) for inputs_i in range(batch_size) \
                                                         for batches_i in range(n_batches)]

    for (inputs_i, batch_i), inputs, targets in zip(indices, inputss, targetss):
        # Inputs
        batches[batch_i, 0, inputs_i] = inputs
        # Targets
        batches[batch_i, 1, inputs_i] = targets

    return batches

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_batches(get_batches)


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Build RNN
You created a RNN Cell in the get_init_cell() function.  Time to use the cell to create a RNN.
- Build the RNN using the tf.nn.dynamic_rnn()
 - Apply the name "final_state" to the final state using tf.identity()
Return the outputs and final_state state in the following tuple (Outputs, FinalState)
Can you write Python code for it?

def build_rnn(cell, inputs):
    """
    Create a RNN using a RNN Cell
    :param cell: RNN Cell
    :param inputs: Input text data
    :return: Tuple (Outputs, Final State)
    """
    outputs, state = tf.nn.dynamic_rnn(cell, inputs, dtype=tf.float32)

    final_state = tf.identity(state, name='final_state')

    return outputs, final_state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_build_rnn(build_rnn)


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Build the Neural Network
Apply the functions you implemented above to:
- Apply embedding to input_data using your get_embed(input_data, vocab_size, embed_dim) function.
- Build RNN using cell and your build_rnn(cell, inputs) function.
- Apply a fully connected layer with a linear activation and vocab_size as the number of outputs.
Return the logits and final state in the following tuple (Logits, FinalState)
Can you write Python code for it?

def build_nn(cell, rnn_size, input_data, vocab_size):
    """
    Build part of the neural network
    :param cell: RNN cell
    :param rnn_size: Size of rnns
    :param input_data: Input data
    :param vocab_size: Vocabulary size
    :return: Tuple (Logits, FinalState)
    """
    embed_dim = 5000
    embedded_input = get_embed(input_data, vocab_size, embed_dim)
    outputs, final_state = build_rnn(cell, embedded_input)

    logits = tf.contrib.layers.fully_connected(outputs, vocab_size, activation_fn=None)
    return logits, final_state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_build_nn(build_nn)


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Build the Neural Network
You'll build the components necessary to build a RNN by implementing the following functions below:
- get_inputs
- get_init_cell
- get_embed
- build_rnn
- build_nn
- get_batches
Check the Version of TensorFlow and Access to GPU
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf

# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.0'), 'Please use TensorFlow version 1.0 or newer'
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Check Point
This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
import numpy as np
import problem_unittests as tests

int_text, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Checkpoint
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()
seq_length, load_dir = helper.load_params()


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Choose Word
Implement the pick_word() function to select the next word using probabilities.
Can you write Python code for it?

def pick_word(probabilities, int_to_vocab):
    """
    Pick the next word in the generated text
    :param probabilities: Probabilites of the next word
    :param int_to_vocab: Dictionary of word ids as the keys and words as the values
    :return: String of the predicted word
    """
    def throw_the_dice(words):
        # filter out some words based on their probability
        return [word for prob, (_, word) in words if prob >= np.random.rand()]

    k = 10
    vocab = sorted(zip(probabilities, int_to_vocab.items()), reverse=True)
    words = vocab[:k]

    leftover_words = []
    while not leftover_words:
        # Keep trying until we have at least one word that was not filter out
        leftover_words = throw_the_dice(words)

    return leftover_words[0]

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_pick_word(pick_word)


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Generate TV Script
This will generate the TV script for you.  Set gen_length to the length of TV script you want to generate.
Can you write Python code for it?

gen_length = 200
# homer_simpson, moe_szyslak, or Barney_Gumble
prime_word = 'moe_szyslak'

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_dir + '.meta')
    loader.restore(sess, load_dir)

    # Get Tensors from loaded model
    input_text, initial_state, final_state, probs = get_tensors(loaded_graph)

    # Sentences generation setup
    gen_sentences = [prime_word + ':']
    prev_state = sess.run(initial_state, {input_text: np.array([[1]])})

    # Generate sentences
    for n in range(gen_length):
        # Dynamic Input
        dyn_input = [[vocab_to_int[word] for word in gen_sentences[-seq_length:]]]
        dyn_seq_length = len(dyn_input[0])

        # Get Prediction
        probabilities, prev_state = sess.run(
            [probs, final_state],
            {input_text: dyn_input, initial_state: prev_state})

        pred_word = pick_word(probabilities[dyn_seq_length-1], int_to_vocab)

        gen_sentences.append(pred_word)

    # Remove tokens
    tv_script = ' '.join(gen_sentences)
    for key, token in token_dict.items():
        ending = ' ' if key in ['\n', '(', '"'] else ''
        tv_script = tv_script.replace(' ' + token.lower(), key)
    tv_script = tv_script.replace('\n ', '\n')
    tv_script = tv_script.replace('( ', '(')

    print(tv_script)


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Implement Generate Functions
Get Tensors
Get tensors from loaded_graph using the function get_tensor_by_name().  Get the tensors using the following names:
- "input:0"
- "initial_state:0"
- "final_state:0"
- "probs:0"
Return the tensors in the following tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)
Can you write Python code for it?

def get_tensors(loaded_graph):
    """
    Get input, initial state, final state, and probabilities tensor from <loaded_graph>
    :param loaded_graph: TensorFlow graph loaded from file
    :return: Tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)
    """

    get_tensor = loaded_graph.get_tensor_by_name

    input_ = get_tensor('input:0')
    initial_state = get_tensor('initial_state:0')
    final_state = get_tensor('final_state:0')
    probs = get_tensor('probs:0')

    return  input_, initial_state, final_state, probs

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_tensors(get_tensors)


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Input
Implement the get_inputs() function to create TF Placeholders for the Neural Network.  It should create the following placeholders:
- Input text placeholder named "input" using the TF Placeholder name parameter.
- Targets placeholder
- Learning Rate placeholder
Return the placeholders in the following the tuple (Input, Targets, LearingRate)
Can you write Python code for it?

def get_inputs():
    """
    Create TF Placeholders for input, targets, and learning rate.
    :return: Tuple (input, targets, learning rate)
    """
    inputs = tf.placeholder(shape=(None, None), dtype=tf.int32, name='input')
    targets = tf.placeholder(shape=(None, None), dtype=tf.int32)
    learning_rate = tf.placeholder(dtype=tf.float32)
    return inputs, targets, learning_rate

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_inputs(get_inputs)


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Tokenize Punctuation
We'll be splitting the script into a word array using spaces as delimiters.  However, punctuations like periods and exclamation marks make it hard for the neural network to distinguish between the word "bye" and "bye!".
Implement the function token_lookup to return a dict that will be used to tokenize symbols like "!" into "||Exclamation_Mark||".  Create a dictionary for the following symbols where the symbol is the key and value is the token:
- Period ( . )
- Comma ( , )
- Quotation Mark ( " )
- Semicolon ( ; )
- Exclamation mark ( ! )
- Question mark ( ? )
- Left Parentheses ( ( )
- Right Parentheses ( ) )
- Dash ( -- )
- Return ( \n )
This dictionary will be used to token the symbols and add the delimiter (space) around it.  This separates the symbols as it's own word, making it easier for the neural network to predict on the next word. Make sure you don't use a token that could be confused as a word. Instead of using the token "dash", try using something like "||dash||".
Can you write Python code for it?

def token_lookup():
    """
    Generate a dict to turn punctuation into a token.
    :return: Tokenize dictionary where the key is the punctuation and the value is the token
    """
    return dict([
            (".","period"),
            (",","comma"),
            ("\"","quotation_mark"),
            (";","semicolon"),
            ("!","exclamation_mark"),
            ("?","question_mark"),
            ("(","left_parentheses"),
            (")","right_parentheses"),
            ("--","dash"),
            ("\n","return")])

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_tokenize(token_lookup)


I am working on the file "3_tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Word Embedding
Apply embedding to input_data using TensorFlow.  Return the embedded sequence.
Can you write Python code for it?

def get_embed(input_data, vocab_size, embed_dim):
    """
    Create embedding for <input_data>.
    :param input_data: TF placeholder for text input.
    :param vocab_size: Number of words in vocabulary.
    :param embed_dim: Number of embedding dimensions
    :return: Embedded input.
    """
    embedding = tf.Variable(tf.random_uniform((vocab_size, embed_dim), -1, 1, dtype=tf.float32))

    embed = tf.nn.embedding_lookup(embedding, input_data)

    return embed

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_embed(get_embed)


I am working on the file "3b-RegressionLineaire-Salaires.ipynb".
The first task is:
Application: estimation du salaire à Princeton
Pour appliquer la régression linaire, nous allons d'abord charger un ensemble de données qui contient des informations sur les salaires à l'Université de Princeton. Les données brutes sont organisées de la façon suivante :

sx = sexe, encodé avec 1 pour les femmes et 0 pour les hommes

rk = rang, encodé avec 1 pour assistant professor, 2 pour associate professor, et 3 pour full professor

yr = nombre d'années dans le rang courant

dg = plus haut diplôme, 1 pour le doctorat, 0 pour le master

yd = nombre d'années depuis que le dernier diplôme a été obtenu

sl = salaire annuel universitaire, en US dollar

Les données sont librement accessible sur http://data.princeton.edu/wws509/datasets/#salary
Can you write Python code for it?

!wget http://data.princeton.edu/wws509/datasets/salary.raw
raw_data = loadtxt('salary.raw')
print (raw_data[0:3, :].astype(int))


I am working on the file "3b-RegressionLineaire-Salaires.ipynb".
The first task is:
Comparons le résultat:
Can you write Python code for it?

plot_salaires(X_men[:, 1], Y_men, color='b', label='Homme')
plot_regression(theta_men, 'b-')
plot_salaires(X_women[:, 1], Y_women, color='r', label='Femme')
plot_regression(theta_women, 'r-')


I am working on the file "3b-RegressionLineaire-Salaires.ipynb".
The first task is:
Introduction et notations
On reprends les notations et les algorithmes du précédent notebook.
Can you write Python code for it?

def h(theta, x):
    y_estimated = 0.
    for theta_i, x_i in zip(theta, x):
        y_estimated += theta_i*x_i
    return y_estimated

def J(theta, X, Y):
    cost = 0.
    m = X.shape[0]
    for x_i, y_i in zip(X, Y):
        cost += (h(theta, x_i)  - y_i)**2
    return 1./(2.*m)*cost

def batch_update(theta, X, Y, alpha=0.01):
    new_theta = zeros_like(theta)
    m = X.shape[0]
    for j, theta_j in enumerate(theta):
        sum_derivative = 0.
        for i, (x_i, y_i) in enumerate(zip(X,Y)):
            sum_derivative += (y_i - h(theta,x_i))*x_i[j]
        new_theta[j] = theta_j + alpha*(sum_derivative)
    return new_theta

def stochastic_update(theta, X, Y, alpha=0.01):
    new_theta = theta.copy()
    m = X.shape[0]
    for i, (x_i, y_i) in enumerate(zip(X,Y)):
        for j, theta_j in enumerate(new_theta):
            new_theta[j] = new_theta[j] + alpha*(y_i - h(new_theta, x_i))*x_i[j]
    return new_theta

def normal_equation(X, Y):
    return pinv(X.T.dot(X)).dot(X.T).dot(Y)


I am working on the file "3b-RegressionLineaire-Salaires.ipynb".
The first task is:
On applique notre algorithme batch avec des iterations successives:
Can you write Python code for it?

m = data.shape[0]
n = data.shape[1]

X = ones(shape=(m, 2))
X[:,1] = data[:,0]
Y = data[:, 1]
theta = zeros(shape=(n, 1))

plot_salaires(X[:,1], Y)

alpha = 0.0002
iteration = 1000
J_batch = zeros(iteration)
for i in range(iteration):
    theta = batch_update(theta, X, Y, alpha)
    J_batch[i] =  J(theta, X, Y)
    if (mod(i, iteration/10) == 0):
        plot_regression(theta, 'b-', alpha=float(i)/iteration)
plot_regression(theta, 'k-')


I am working on the file "3b-RegressionLineaire-Salaires.ipynb".
The first task is:
Regardons maintenant la solution obtenue avec l'équation normale:
Can you write Python code for it?

plot_salaires(X[:,1], Y)
theta = normal_equation(X, Y)
plot_regression(theta, 'g-')


I am working on the file "3d/floating_bar/floating_bar.ipynb".
The first task is:
Define some functions to help monitor the calculation
Can you write Python code for it?

import numpy as np
import numpy
from pythreejs import *

from IPython.core.display import clear_output

def plot_current_results(in_place=True):
    import numpy as np
    #from mpl_toolkits.mplot3d import Axes3D
    #import matplotlib.pyplot as  plt
    """Makes a blocking call to retrieve remote data and displays the solution mesh
    as a contour plot.

    Parameters
    ----------
    in_place : bool
        By default it calls clear_output so that new plots replace old ones.  Set
        to False to allow keeping of all previous outputs.
    """
    global nn,x,y,u,vertices,triangles,domain,rigid_bar
    # We make a blocking call to load the remote data from the simulation into simple named
    # variables we can read from the engine namespaces
    #load_simulation_globals()
    view.apply_sync(load_simulation_globals)
    # And now we can use the view to read these variables from all the engines.  Then we
    # concatenate all of them into single arrays for local plotting
    #x = np.concatenate(view['x'])
    #y = np.concatenate(view['y'])
    #z = np.concatenate(view['z'])
    vertices = np.concatenate(view['vertices'])
    shifts = np.cumsum([0]+view['nn'][:-1])
    flat_triangles = np.concatenate([ tri + shift for tri,shift in zip(view['triangles'], shifts) ])
    #flat_triangles=triangles
    # We can now call the matplotlib plotting function we need
    #fig, ax = plt.subplots(subplot_kw=dict(aspect='equal'))
    #fig = plt.figure()
    #ax = fig.gca(projection='3d')
    #print x.shape,y.shape,flat_triangles.shape
    #print flat_triangles.flat[:].max(),flat_triangles.flat[:].min()
    #ax.plot_trisurf(X=x, Y=y, TRIANGLES=flat_triangles)#, Z=z)
    #ax.plot_trisurf(x,y,z)
    widget_surface.geometry = FaceGeometry(vertices=list(vertices.flatten()),
                                           face3=list(flat_triangles.flatten()))
    #from matplotlib import tri
    #mesh = tri.Triangulation(x,y,flat_triangles)
    #help(mesh)
    #ip = tri.LinearTriInterpolator(triangulation, u,...


I am working on the file "3d/floating_bar/floating_bar.ipynb".
The first task is:
Define the tank geometry
Importing the tank module reads a text file for the bathymetry transect and generates a polygonal domain. There are several options for how we can modify it:

add a file upload box to just upload a different bottom bathymetry
manipulate an interactive version of the resulting domain (an interactive version of the plot below)
manipulate vertices directly in domain.vertices and domain.segments
Can you write Python code for it?

#!tar czvf mydir.tar.gz  ls_p.py tank_so.py twp_navier_stokes_n.py redist_n.py   twp_navier_stokes_p.py redist_p.py vof_n.py vof_p.py ls_consrv_n.py ls_consrv_p.py ls_n.py tank.py
#!scp mydir.tar.gz spirit01.afrl.hpc.mil:

#%%px
#import subprocess
#if rank == 0:
  #  status_gz = subprocess.call("gunzip " +  "mydir.tar.gz", shell=True)
    #status_tar = subprocess.call("tar "+"xvf mydir.tar",shell=True)
#    os.system('tar xzvf mydir.tgz')

%%px
import floating_bar,floating_bar_so


I am working on the file "3d/floating_bar/floating_bar.ipynb".
The first task is:
IPython Parallel "Magics"
As a first example of executing code in parallel on our, we use the cell magic %%px to execute some basic MPI code on every engine in our cluster.
Can you write Python code for it?

%%px
from mpi4py import MPI
mpi = MPI.COMM_WORLD
bcast = mpi.bcast
barrier = mpi.barrier
rank = mpi.rank
print "MPI rank: %i/%i" % (mpi.rank,mpi.size)


I am working on the file "3d/floating_bar/floating_bar.ipynb".
The first task is:
Load Proteus
Proteus is a Python package consiting of multiple modules. We are going to define a problem and run a simulation interactively, so we pull in the iproteus (interactive proteus) module to set up a basic environment.
Notes:

Proteus runs with logging that records verying amounts of information by setting logLevel from 1 to 11
The log is stored  in a .log file, which can be downloaded
If Profiling.verbose is set  to True then logging will show up in certain output cells of the notebook, which is usually not what you want
Can you write Python code for it?

%%px
import sys
from proteus.iproteus import *

%%px
from proteus import default_n, default_s, default_so
Profiling.logLevel=5
Profiling.verbose=False


I am working on the file "3d/floating_bar/floating_bar.ipynb".
The first task is:
Physics and Numerics
Load the modules the define the equations to be solved and the numerical methods to use.
Can you write Python code for it?

%%px
from proteus import Comm
from petsc4py import PETSc

so = floating_bar_so
so.tnList = so.tnList
pList=[]
nList=[]
so.sList=[]
OptDB = PETSc.Options()
for (p,n) in so.pnList:
    so.sList.append(default_s)
    pList.append(__import__(p))
    nList.append(__import__(n))
    pList[-1].name = p
    nList[-1].multilevelLinearSolver = default_n.KSP_petsc4py
    nList[-1].levelLinearSolver = default_n.KSP_petsc4py
    OptDB.setValue(nList[-1].linear_solver_options_prefix+"ksp_type", "preonly")
    OptDB.setValue(nList[-1].linear_solver_options_prefix+"pc_type", "lu")
    OptDB.setValue(nList[-1].linear_solver_options_prefix+"pc_factor_mat_solver_package","superlu_dist")
opts.save_dof = True
opts.dataDir='.'
opts.probDir='.'
opts.logLevel=7
opts.verbose=True


I am working on the file "3d/floating_bar/floating_bar.ipynb".
The first task is:
Post-process the numerical solution
Can you write Python code for it?

%%px
from tables import  openFile
archive = openFile('tank_p%d.h5' % (rank,),'r')

def load_post_simulation_globals(it):
    """Put some variables we need in engine namespace.

    These can then be retrieved by clients for inspection, visualization, etc.
    """
    global phi
    print it
    phi=archive.getNode("/phi"+`it`)[:];

x = np.concatenate(view['x'])
y = np.concatenate(view['y'])
shifts = numpy.cumsum([0]+view['nn'][:-1])
flat_triangles = np.concatenate([ tri + shift for tri,shift in zip(view['triangles'], shifts) ])
triplot(x,y,flat_triangles)

!rm phi*png
import tank
for it,t in enumerate(view['tank_so.tnList'][0]):
    view.apply_sync(load_post_simulation_globals,it)
    phi = np.concatenate(view['phi'])
    pyplot.clf()
    plt.xlabel(r'z[m]')
    plt.ylabel(r'x[m]')
    colors = ['b','g','r','c','m','y','k','w']
    pylab.xlim(tank.domain.x[0]-0.1*tank.domain.L[0],tank.domain.x[0]+tank.domain.L[0]+0.1*tank.domain.L[0])
    pyplot.axis('equal')
    for si,s in enumerate(tank.segments):
        pyplot.plot([tank.domain.vertices[s[0]][0],
                     tank.domain.vertices[s[1]][0]],
                    [tank.domain.vertices[s[0]][1],
                     tank.domain.vertices[s[1]][1]],
                    color=colors[tank.domain.segmentFlags[si]-1],
                    linewidth=2,
                    marker='o')
    pyplot.tricontour(x,y,flat_triangles,phi,[0])
    pyplot.title('T=%2.2f' % (t,))
    pyplot.savefig('phi%4.4d.png' % (it,))

!avconv -y -i phi%4d.png -c libx264 -qscale 1 tankPhi.mp4

from IPython.core.display import HTML
data_uri_mp4 = open("tankPhi.mp4", "rb").read().encode("base64").replace("\n", "")
video_tag = """<video controls>
<source type ="video/mp4" src="data:video/mp4;base64,{mp4}"/>
Your browser does not support the video tag
</video>""".format(mp4=data_uri_mp4)
HTML(data=video_tag)

from IPython.display import FileLink,FileLinks
FileLink('tankPhi.mp4')


I am working on the file "3d/floating_bar/floating_bar.ipynb".
The first task is:
Run the tank
Can you write Python code for it?

%px simulation_thread.start()

view.apply_sync(load_simulation_globals)
vertices = np.concatenate(view['vertices'])
shifts = np.cumsum([0]+view['nn'][:-1])
flat_triangles = np.concatenate([ tri + shift for tri,shift in zip(view['triangles'], shifts) ])

from pythreejs import *
from IPython.display import display
domain = view['domain'][0]
vertices = vertices#verticesview['vertices'][0]
triangles=flat_triangles#view['triangles'][0]
#rigid_bar = view['rigid_bar'][0]
#L = view['ns'][0].pList[0].domain.L
#x_ll = view['ns'][0].pList[0].domain.x_ll
#b = view['ns'][0].nList[0].auxiliaryVariables[0].bar
#sx,sy,sz = bar.boxsize
sx,sy,sz = (0.5,0.5,0.5)
widget_bar = Mesh(geometry=BoxGeometry(width=sx,height=sy,depth=sz),
                    material=LambertMaterial(color='black'),
                    position=(0.5,0.5,0.5))#b.getPosition())
#widget.quaternion_from_rotation(b.getRotation())
center = (0.5*(domain.x[0]+domain.L[0]),
          0.5*(domain.x[1]+domain.L[1]),
          0.5*(domain.x[2]+domain.L[2]))
widget_surface = Mesh(geometry=FaceGeometry(vertices=list(vertices.flatten()),face3=list(flat_triangles.flatten())),
                      material=LambertMaterial(color='aqua'))
#widget.quaternion_from_rotation(b.getRotation())
center = (0.5*(domain.x[0]+domain.L[0]),
          0.5*(domain.x[1]+domain.L[1]),
          0.5*(domain.x[2]+domain.L[2]))

children=[widget_bar,widget_surface]
children.append(Mesh(geometry=BoxGeometry(width=domain.L[0],height=domain.L[1],depth=domain.L[2]),
                     material=LambertMaterial(color=0xffffff, #color=0xccccff,
                                              opacity=0.7,
                                              refractionRatio=0.985,
                                              reflectivity= 0.9,
                                              transparent=True),
                    ...


I am working on the file "3d_meshing.ipynb".
The first task is:
Example: Randomly Sampled Points on a Cylinder
Can you write Python code for it?

def cylinder_points_and_hull_given_sample_size(sample_size):
    points = []
    for i in range(sample_size/2):
        x = random.uniform(-1,1)
        z = random.uniform(0,1)
        s = (-1.0, 1.0)[random.uniform(0,1) < 0.5]
        y = s * (1 - x**2) ** (0.5)
        points.append(np.array([x,y,z]))
    for z in range(0,2):
        for i in range(n/4):
            x = random.uniform(-1,1)
            s = (-1.0, 1.0)[random.uniform(0,1) < 0.5]
            y = s * random.uniform(0,1) * (1 - x**2) ** (0.5)
            points.append(np.array([x,y,z]))
    points = np.array(points)
    triangles_vertices, hull_points = compute_mesh(points)
    return points, hull_points, triangles_vertices

random.seed(42)
n = 100
points, hull_vertices, triangles_vertices = cylinder_points_and_hull_given_sample_size(n)
points[:3]

triangles_vertices[:3]

graph_points_triangles([[points, triangles_vertices]])


I am working on the file "3d_meshing.ipynb".
The first task is:
Meshing and Volume Calculations
Can you write Python code for it?

import numpy as np
import random
from scipy.spatial import ConvexHull

def compute_mesh(points):
    hull = ConvexHull(points)
    indices = hull.simplices
    return indices, hull.vertices


I am working on the file "3d_ml/neural_rendering/Test.ipynb".
The first task is:
Data Processing
Can you write Python code for it?

data_dir = Path.home() / 'Documents/generated_data/neural_rendering'

from mesh_to_sdf import mesh_to_voxels, sample_sdf_near_surface

import trimesh
import skimage.measure


I am working on the file "3d_ml/neural_rendering/Test.ipynb".
The first task is:
Predict
Can you write Python code for it?

pred_voxel = deep_sdf.model.predict([all_xyz, np.tile(latent, (all_xyz.shape[0], 1))])
pred_voxel.shape

pred_voxel


I am working on the file "3d_ml/neural_rendering/Test.ipynb".
The first task is:
Sample SDF
Can you write Python code for it?

mesh = trimesh.load(data_dir / 'suzanne.obj')

number_of_points = 250000
points, sdf = sample_sdf_near_surface(mesh, number_of_points=number_of_points)
points.shape