I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The time required to fit the model is lower than in any other case presenting a higher accuracy. In this case, the accuracy is close to 84.12% while the classification report shows values close to one, showing that there is overfitting. Hence, from the classifiers evaluated until now this is the one that presents better results, from an accuracy and a computational effort perspective. This is the best candidate to move into production for the moment.
Tf-idf
A multinomial classifier is trained on the features obtained using tfidf and evaluated on the holdout. In this case, as in the previous Navy Bayes classification used, alpha always gets the value cloaer to zero, therefore there is no additive smoothing used in this classifier. From a compuational effort standpoint, as in the previous case, this is the one that requires less time to fit making it a strong candidate to move into production.
Can you write Python code for it?

# Initialize and fit the model.
naive_bayes_multinomial_tfidf = MultinomialNB()

#Tune hyperparameters
#Create range of values to fit parameters

alpha = [0.01,0.1,0.5,1]
parameters = {'alpha': alpha}

#Fit parameters using gridsearch
naive_bayes_multinomial_tuned_tfidf = GridSearchCV(naive_bayes_multinomial_tfidf,
                                                 n_jobs = -1,
                                                 param_grid=parameters,
                                                 cv=kf, verbose = 1)

#Fit the tunned classifier in the training space
naive_bayes_multinomial_tuned_tfidf.fit(X_train_tfidf, y_train_tfidf)

#Print the best parameters
print(('Best paramenters Naive-Bayes Multinomial BoW:\n {}\n').format(
naive_bayes_multinomial_tuned_tfidf.best_params_))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
he model is evaluated using cross validation and five folds. In this case as in the case of logistic regression the model presents overfitting as it can be seen from the classification report. Both precision and recall is one for this reason.
Can you write Python code for it?

#Evaluation of the model (testing)

target_names = ['0.0', '1.0', '2.0', '3.0']

print(('Classification Report Tfidf: \n {}').format(classification_report(y_test_tfidf, predtest_y_tfidf,
                          target_names=target_names)))

confusion_tfidf = confusion_matrix(y_test_tfidf, predtest_y_tfidf)

print(('Confusion Matrix Tf-idf: \n\n {}\n').format(confusion_tfidf))

print(('Bernoulli Classifier Tf-Idf set accuracy Tf-idf: {0:.2f} % \n').format(cross_val_score(naive_bayes_bernoulli_tuned_tfidf,
                                                                                X_test_tfidf,
                                                                                y_test_tfidf,
                                                                                cv=kf).mean()*100))


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
2. Cleanse and parse and tokenize text
Before generating the features, and to increase the explanatory power of them, text has been cleaned and parsed accordingly.  The books have gone through an initial set of cleansing actions before been parsed using Spacy, to reduce the computing effort required by the latter and then have been cleaned again before the feature generation.
The initial cleansing action has had three steps. The first step consisted on deleting all references to the Gutenberg Project from every book. This way, it has been avoided that words like “Gutenberg” and “Gutenberg Project” appear as features and distort the clustering of the authors.
As described below, cleaning actions have gone from removing all references to chapters, digits double whitespaces and references to numbers like dates and ordinal numbers. This has been followed by removing punctuation and common stop words that will only add noise to the features that are generated afterwards.
The remaining words, considered to have the most explanatory power regarding each of the titles from the authors, have been lemmatized and stemmed reducing up to 60% the computing resources needed. In the first case words from the same family are reduced to their lemmas and in the second case, additional prefixes and suffixes are removed. All cleaning operations have been carried out in a way that remaining sentences are stored in a list of lists.
Can you write Python code for it?

#Create a set of stopwords in english from nltk
stop = set(stopwords.words('english'))

# Create a set of punctuation marks to exclude them from the text
exclude = set(string.punctuation)

#List of words that are in the books and can be removed from the text as they add noise
comms = ['im','mr', 'miss', 'lady', 'sir', 'man', 'it', 'i', 'u', 'you', 'one']

#Call the lemmatizer and the stemmer
lemma = WordNetLemmatizer()
stemms = PorterStemmer()

#Define a cleaning function that incorporates the different steps in the pipeline to clean the texts

def clean(doc):
    #Delete '--' from texts, specially used by Austen
    doc = re.sub(r'--','',doc)
    doc = re.sub("[\[].*?[\]]", "", doc) #
    #Remove references to chapters
    doc = re.sub(r'Chapter \d+', '', doc)
    doc = re.sub(r'CHAPTER .*', '', doc)
    #Substitute numbers by spaces
    doc = re.sub('[0-9]+', '', doc)
    #Remove digits and expressions such as 11th
    doc = re.sub("\s\s+", " ", doc)
    doc = re.sub("^\d+\s|\s\d+\s|\s\d+$", " ", doc)
    #Remove quotes
    doc = doc.replace('“', '').replace('”', '').replace('"', '').replace('‘', '').replace('’', '')
    #Remove stop words
    stop_free = " ".join([i for i in doc.lower().split() if i not in stop])
    #Remove punctuation
    punc_free = ''.join(ch for ch in stop_free if ch not in exclude)
    #Remove common words
    commonalities_free = " ".join([i for i in punc_free.lower().split() if i not in comms])
    #Lemmatize and stem words to reduce the total amount of words
    lemmatized = " ".join(lemma.lemmatize(word) for word in commonalities_free.split())
    normalized = " ".join(stemms.stem(word) for word in lemmatized.split())
    return normalized

#Create a list of lists with all the documents
doc_clean = [clean(doc) for doc in doc_complete]


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
3. Multi Layer Perceptron
The model is set up and the parameters are tunned so that the model gives the highest accuracy considering the small dataset that we have for this exercise. In this case, and after several trials, the number of words is set up to 25000 being the optimum.
Can you write Python code for it?

#Start the model
model = Sequential()

#Set up the outter layer and reduce the number of words in voc to the optimum 25000
model.add(Dense(32, activation='relu', input_shape=(25000,)))

# Dropout layers remove features and fight overfitting
model.add(Dropout(0.1))
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.1))

# End with a number of units equal to the number of classes we have for our outcome
model.add(Dense(10, activation='softmax'))

#Show the model features
model.summary()

# Compile the model to put it all together.
model.compile(loss='categorical_crossentropy',optimizer=RMSprop(), metrics=['accuracy'])


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
5. LTSM with Dropout
To the previous model, dropout is included to reduce overfitting. In this case, two forward dropouts are included with 0.2 dropout rate and a layer of 100 and 10 neurons
Can you write Python code for it?

#Set up the model including dropouts to fight overfitting
model_LTSM_wdrop = Sequential()
model_LTSM_wdrop.add(Embedding(top_words, embedding_vector_length, input_length=max_review_length))
model_LTSM_wdrop.add(Dropout(0.2))
model_LTSM_wdrop.add(LSTM(100))
model_LTSM_wdrop.add(Dropout(0.2))
model_LTSM_wdrop.add(Dense(10, activation='sigmoid'))

#Model Features
model_LTSM_wdrop.summary()

#Compile the model
model_LTSM_wdrop.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
7. LTSM & Recurrent Neural Network
In this case the LTSM model is run with a recurrent neural network. This network has a recurrent dropout of 0.2 and three layers maintaining LTSM to 100 features
Can you write Python code for it?

# Building the Model
model = Sequential()
model.add(Embedding(top_words, embedding_vector_length, input_length=max_review_length))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(10, activation='sigmoid'))

#Model features
model.summary()

# Compile the model to put it all together.
model.compile(loss='binary_crossentropy', optimizer='sgd',
              metrics=['accuracy'])


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
Data is split in a training and testing set with 25% for testing
Can you write Python code for it?

#Split data in a train and test set
X_tr, X_te, y_tr, y_te = train_test_split(sentences['Sentences'], sentences['Target'],test_size=0.25, random_state=42)


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
Once the model is evaluated, an accuracy of 90% is achieved in three epochs. This overall accuracy seems to be the maximum accuracy that can be obtained with the dataset available.
Can you write Python code for it?

#Fit the model
model_LTSM.fit(X_train_CNN, y_train,
          batch_size=64,
          epochs=3,
          verbose=1,
          validation_data=(X_test_CNN, y_test))

#Evaluate the model
score = model_LTSM_wdrop.evaluate(X_test_CNN, y_test, verbose=0)

#Print scores
print('Test loss:', score[0])
print('Test accuracy:', score[1])


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
Set up the parameters for the LTSM models
To run the LTSM and be able to do the dimensionality reduction the number of top words is defined setting the rest  to zero, the max review legnth truncating the senteces of each author and the embedding vector. After several trials the optimum parameters are the ones shown bellow.
Can you write Python code for it?

# load the dataset but only keep the top n words, zero the rest
top_words = 5000

# truncate and pad input sequences
max_review_length = 500
X_train_CNN = sequence.pad_sequences(x_train, maxlen=max_review_length)
X_test_CNN = sequence.pad_sequences(x_test, maxlen=max_review_length)

# Vector length
embedding_vector_length = 32

#Print the shape of the train and test set
print('x_train shape:', X_train_CNN.shape)
print(X_train_CNN.shape[0], 'train samples')
print(X_test_CNN.shape[0], 'test samples')


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
The information is added to the copus and stored as raw books so that they can be cleansed
Can you write Python code for it?

#Read and add the text of each book to corpus_raw.
corpus_raw_austen = u""
for book_filename in book_filenames_austen:
    print("Reading '{0}'...".format(book_filename))
    with codecs.open(book_filename, "r", "utf-8") as book_file:
        corpus_raw_austen += book_file.read()
    print("Corpus is now {0} characters long".format(len(corpus_raw_austen)))
    print()

#Read and add the text of each book to corpus_raw.
corpus_raw_chesterton = u""
for book_filename in book_filenames_chesterton:
    print("Reading '{0}'...".format(book_filename))
    with codecs.open(book_filename, "r", "utf-8") as book_file:
        corpus_raw_chesterton += book_file.read()
    print("Corpus is now {0} characters long".format(len(corpus_raw_chesterton)))
    print()

#Read and add the text of each book to corpus_raw.
corpus_raw_conandoyle = u""
for book_filename in book_filenames_conandoyle:
    print("Reading '{0}'...".format(book_filename))
    with codecs.open(book_filename, "r", "utf-8") as book_file:
        corpus_raw_conandoyle += book_file.read()
    print("Corpus is now {0} characters long".format(len(corpus_raw_conandoyle)))
    print()

#Read and add the text of each book to corpus_raw.
corpus_raw_dickens = u""
for book_filename in book_filenames_dickens:
    print("Reading '{0}'...".format(book_filename))
    with codecs.open(book_filename, "r", "utf-8") as book_file:
        corpus_raw_dickens += book_file.read()
    print("Corpus is now {0} characters long".format(len(corpus_raw_dickens)))
    print()

#Read and add the text of each book to corpus_raw.
corpus_raw_elliot = u""
for book_filename in book_filenames_elliot:
    print("Reading '{0}'...".format(book_filename))
    with codecs.open(book_filename, "r", "utf-8") as book_file:
        corpus_raw_elliot += book_file.read()
    print("Corpus is now {0} characters long".format(len(corpus_raw_elliot)))
    print()

#Read and add the text of each book to corpus_raw.
corpus_raw_huxley = u""
for book_filename in book_filenames_huxley:
   ...


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
The maximum accuracy achieved continues to be 0.9 which is reasonable for the size of the dataset used.
Can you write Python code for it?

#Fit the model
model_LTSM_wdrop.fit(X_train_CNN, y_train,
          batch_size=64,
          epochs=3,
          verbose=1,
          validation_data=(X_test_CNN, y_test))

#Evaluate the model
score = model_LTSM_wdrop.evaluate(X_test_CNN, y_test, verbose=0)

#Print the scores
print('Test loss:', score[0])
print('Test accuracy:', score[1])


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
The model achieves thebest result possible 90%. Although low for a neural network is reasonable neough for the size of the dataset.
Can you write Python code for it?

#Fit the model
model.fit(X_train_CNN, y_train,
          batch_size=64,
          epochs=3,
          verbose=1,
          validation_data=(X_test_CNN, y_test))

#Evaluate the model
score = model.evaluate(X_test_CNN, y_test, verbose=0)

#Print the score
print('Test loss:', score[0])
print('Test accuracy:', score[1])


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
The model is evaluated on the train and test set achieving an overall accuracy of 86%. in this case, a two layers of 32 neurons each is considered with dropout to reduce overfitting and softmax is used as the optimizer.
Can you write Python code for it?

#Run the model
history = model.fit(x_train, y_train,
                    batch_size=128,
                    epochs=5,
                    verbose=1,
                    validation_data=(x_test, y_test))

#Evaluate the model
score = model.evaluate(x_test, y_test, verbose=0)

#Print results
print('Test loss:', score[0])
print('Test accuracy:', score[1])


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
Tokens are grouped into sentences and labelled in a dataframe. A Target column is added to the dataframe for further use when classifying texts using supervised techniques. Some additional cleansing of the sentences has been carried out. Sentences under 50 words have been removed from the dataset, duplicates and empty rows.Additionally the length of each sentence has been added to the daataframe.
The graph below reflects the total number of characters and the contribution of each of the corpus to the total corpora. As titles were picked randomly, the contribution of each corpora is different by each author being the biggest Elliot and the smallest Huxley with nearly 8000 and 1500 contribution each.
Can you write Python code for it?

# Group into sentences.
austen_sents = [[str(sent), "Austen"] for sent in austen_doc.sents]
chesterton_sents = [[str(sent), "Chesterton"] for sent in chesterton_doc.sents]
conandoyle_sents = [[str(sent), "Conandoyle"] for sent in conandoyle_doc.sents]
dickens_sents = [[str(sent), "Dickens"] for sent in dickens_doc.sents]
elliot_sents = [[str(sent), "Elliot"] for sent in elliot_doc.sents]
huxley_sents = [[str(sent), "Huxley"] for sent in huxley_doc.sents]
shakespeare_sents = [[str(sent), 'Shakespeare'] for sent in shakespeare_doc.sents]
shaw_sents = [[str(sent), "Shaw"] for sent in shaw_doc.sents]
wells_sents = [[str(sent), "Wells"] for sent in wells_doc.sents]
wilde_sents = [[str(sent), "Wilde"] for sent in wilde_doc.sents]

#Combine the sentences from the two novels into one data frame.
names = ['Sentences','Author']
sentences = pd.DataFrame(austen_sents + chesterton_sents + conandoyle_sents +
                         dickens_sents + elliot_sents + huxley_sents +
                         shakespeare_sents + shaw_sents + wells_sents +
                         wilde_sents, columns = names)

#Aadd numerical column to tag the authors for supervised classification
sentences.loc[sentences['Author'] == 'Austen', 'Target'] = 0
sentences.loc[sentences['Author'] == 'Chesterton', 'Target'] = 1
sentences.loc[sentences['Author'] == 'Conandoyle', 'Target'] = 2
sentences.loc[sentences['Author'] == 'Dickens', 'Target'] = 3
sentences.loc[sentences['Author'] == 'Elliot', 'Target'] = 4
sentences.loc[sentences['Author'] == 'Huxley', 'Target'] = 5
sentences.loc[sentences['Author'] == 'Shakespeare', 'Target'] = 6
sentences.loc[sentences['Author'] == 'Shaw', 'Target'] = 7
sentences.loc[sentences['Author'] == 'Wells', 'Target'] = 8
sentences.loc[sentences['Author'] == 'Wilde', 'Target'] = 9

#Add length of sentences to the Dataframe
sentences['len'] = sentences['Sentences'].map(len)

#Remove duplicate description columns to avoid having the initial and final information about Gutenberg project
sentences =...


I am working on the file "Challenge Neural Networks.ipynb".
The first task is:
Words are tokenized and an integer is assigned to them. Once they are converted into integers each book is represented by a matrix and the labels are represented by 10 binary columns
Can you write Python code for it?

# integer encode sequences of words
tokenizer = Tokenizer(num_words=25000)
tokenizer.fit_on_texts(X_tr)
x_train = tokenizer.texts_to_matrix(X_tr, mode = "tfidf")
x_test = tokenizer.texts_to_matrix(X_te, mode = "tfidf")

# Convert class vectors to binary class matrices
# So instead of one column with 10 values, create 10 binary columns
y_train = keras.utils.to_categorical(y_tr, 10)
y_test = keras.utils.to_categorical(y_te, 10)


I am working on the file "Challenge observations -- cleanup and classification.ipynb".
The first task is:
Using APPLY to classify according to keywords or strings
Can you write Python code for it?

df = df.replace(np.nan,' ', regex=True)#MUST REPLACE NAN AS NAN IS NOT ITERABLE
#analysis_terms = ['analysis', 'density', 'average','variance','pH']
def mymunge(row): # a VARIETY OF CRITERIA
    if any( ['along' in row['note'], 'transect' in row['type'], 'grade' in row['type'], 'line' in row['type'], 'map' in row['type'], 'plot' in row['type'], 'remonitor' in row['type'], 'plants' in row['type']] ):
        return 'transect'
    if any( ['vert' in row['type'], 'ang' in row['type'], 'step back' in row['note'], 'hoop' in row['note'], 'Hoop' in row['note'], 'bare' in row['label1']] ):
        return 'cover'
    if any( ['infiltracion' in row['type'], 'infiltration' in row['type']] ):
        return 'infiltration'
    if any( ['analysis' in row['type'], 'composited' in row['type'], 'misc' in row['type'], 'density' in row['type'],'variance' in row['type'], 'average' in row['type'], 'pH' in row['type']] ):
        return 'analysis'
    if any( ['photo' in row['type']] ):
        return 'photo'
    if any( ['change' in row['type']] ):
        return 'change'
    if any( ['brix' in row['type'], 'biomass' in row['type'], 'clip' in row['type']] ):
        return 'food_analysis'

df['obs_type'] = df.apply(mymunge, axis=1)
#SEE HOW WE DID
print(len(df))
print(df.obs_type.value_counts(normalize=False, sort=True, ascending=False, bins=None, dropna=False))

df = df.replace(np.nan,' ', regex=True)#MUST REPLACE NAN AS NAN IS NOT ITERABLE
#analysis_terms = ['analysis', 'density', 'average','variance','pH']
def mymunge(row): # a VARIETY OF CRITERIA
    if any( ['along' in row['note'], 'transect' in row['type'], 'grade' in row['type'], 'line' in row['type'], 'map' in row['type'], 'plot' in row['type'], 'remonitor' in row['type'], 'plants' in row['type']] ):
        return 'transect'
    if any( ['vert' in row['type'], 'ang' in row['type'], 'step back' in row['note'], 'hoop' in row['note'], 'Hoop' in row['note'], 'bare' in row['label1']] ):
        return 'cover'
    if any( ['infiltracion' in row['type'], 'infiltration'...


I am working on the file "Challenge observations -- cleanup and classification.ipynb".
The first task is:
http://www.gregreda.com/2013/10/26/working-with-pandas-dataframes/
Can you write Python code for it?

food_analysis = df[(df['obs_type'] == "food_analysis")]
food_analysis.to_csv('/Users/Peter/Documents/scc/challenge/obs_types/food_analysis.csv', index=False)


I am working on the file "Challenge observations -- cleanup and classification.ipynb".
The first task is:
load csv
This notebook records some data operations on Soil Carbon Challenge data to prepare it for insertion in atlasbiowork database. The munging and cleanup process will benefit from alternating between using Excel and notebook for inspection and cleanup.
Can you write Python code for it?

#df = pd.read_csv('/Users/Peter/Documents/scc/challenge/PSQL/coverNEW.csv', parse_dates=['date'], usecols=['id','type','date','filename'])
#df = df.replace(np.nan,' ', regex=True)#MUST REPLACE NAN AS NAN IS NOT ITERABLE
#df = pd.read_csv('/Users/Peter/Documents/scc/challenge/datamungedOct2016c.csv')
df = pd.read_csv('/Users/Peter/Documents/atlas/atlasbiowork.com/atlasbiowork/db/changeREADYD.csv')
#df.dtypes

print df.to_json(orient='records')

df.to_csv('/Users/Peter/Documents/atlas/atlasbiowork.com/atlasbiowork/db/changeREADYD.csv', index=False)

df.values[5]

print df.values[1]


I am working on the file "Challenge observations -- cleanup and classification.ipynb".
The first task is:
save your work periodically but w. new filename
Can you write Python code for it?

#df.to_csv('/Users/Peter/Documents/scc/challenge/datamungedOct2016d.csv', index=False)
df.to_csv('/Users/Peter/Documents/scc/challenge/obs_types/changeREADYB.csv', index=False)

#ADD SOME COLUMNS and DROP OTHERS
df = df[df['id']!='geometry']
df['obs_type'] = ""
df['newurl'] =""
df = df.drop('lon', axis=1)
df = df.drop('marker', axis=1)


I am working on the file "Challenge observations -- cleanup and classification.ipynb".
The first task is:
search and replace in dataframe
First, get the strings you'd like to standardize. Then, enter your lists and replacement term. You have to target specific columns.
Can you write Python code for it?

#GET STRINGS OF LABELS "list_of_terms"
a1 = df.label1.unique()
a2 = df.label2.unique()
a3 = df.label3.unique()
#a4 = df.label4.unique()
#a5 = df.label5.unique()
list_of_terms = np.concatenate([a1,a2,a3])
#sorted(list_of_terms)

#SEARCH AND REPLACE
#searchterm = 'lichen'
#replaceterm = 'squish'
#the_list = [item for item in list_of_terms if searchterm in str(item)]

#need to target specific columns

mycols=df[['label1', 'label2','label3','label4','label5']]
#mycols.replace('=','wox', regex=True)

#mycols.replace(to_replace=the_list, value=replaceterm, inplace=True)

#df = df.replace(np.nan,' ', regex=True)
df.label4[df.label4.str.contains('litter')]

searchterm = 'lichen'
replaceterm = 'moss/algae/lichen'
the_list = [item for item in list_of_terms if searchterm in str(item)]

#need to target specific columns

mycols=df[['label1', 'label2','label3','label4','label5']]
mycols.describe()

#mycols.replace(to_replace=the_list, value=replaceterm, inplace=True)

searchterm = 'lichen'
replaceterm = 'moss/algae/lichen'
the_list = [item for item in list_of_terms if searchterm in str(item)]

#need to target specific columns

df[['label1', 'label2','label3','label4','label5']].replace(to_replace=thelist, value=replaceterm, inplace=True)

transects.to_csv('/Users/Peter/Documents/scc/challenge/PSQL/transectsOct23.csv', index=False)

linephotos = df[(df.type.str.contains('line'))]
angphotos = df[(df.type.str.contains('ang')) | (df.note.str.contains('step back'))]
vertphotos = df[df.type.str.contains('vert')]
len(vertphotos)
#re.findall('\d+', s) #finds digits in s
def get_num(x):
    digits = ''.join(ele for ele in x if ele.isdigit())
    if digits:
        return int(digits)
    pass

#get_num('Hoop 1, 125\'')
#df.ix[459]
for y in range(len(df)):

#basic row selection from http://www.gregreda.com/2013/10/26/working-with-pandas-dataframes/
#which has GREAT info on joins
peter_plots = df[(df.lat > 0) & (df.observer.str.contains('Peter Donovan'))]
features = df[(df.type == '*plot summary')|(df.type ==...


I am working on the file "Challenge+Boston+marathon.ipynb".
The first task is:
Compare Spectral Clustering against kMeans using Similarity
As there is no ground truth, the criteria used to evaluate clusters produced using Spectral and kmeans is the silhouette coefficient. From the results obtained, it can be appreaciated that Spectral Clustering requires 6 clusters to have the silhouette score similar to the one obtained with 3 clusters with kmeans.
Can you write Python code for it?

#Compare from a silhouette_score perspective kmeans against Spectral Clustering
range_n_clusters = np.arange(10)+2

for n_clusters in range_n_clusters:
# The silhouette_score gives the average value for all the samples.
    # This gives a perspective into the density and separation of the formed
    # clusters

# Initialize the clusterer with n_clusters value and a random generator
    # seed of 10 for reproducibility.
    spec_clust = SpectralClustering(n_clusters=n_clusters)
    cluster_labels1 = spec_clust.fit_predict(X_tr_std)
    silhouette_avg1 = silhouette_score(X_tr_std, cluster_labels1)

    kmeans = KMeans(n_clusters=n_clusters, init='k-means++', n_init=10).fit(X_tr_std)
    cluster_labels2 = kmeans.fit_predict(X_tr_std)
    silhouette_avg2 = silhouette_score(X_tr_std, cluster_labels2)

    print("For n_clusters =", n_clusters,
          "av. sil_score for Spec. clust is :", silhouette_avg1,
         "av. sil_score for kmeans is :",silhouette_avg2 )


I am working on the file "Challenge+Boston+marathon.ipynb".
The first task is:
Mean Shift
Can you write Python code for it?

# Here we set the bandwidth. This function automatically derives a bandwidth
# number based on an inspection of the distances among points in the data.
bandwidth = estimate_bandwidth(X_tr_std, quantile=0.9)

# Declare and fit the model.
ms = MeanShift(bandwidth=bandwidth, bin_seeding=True).fit(X_tr_std)

# Extract cluster assignments for each data point.
labels = ms.labels_

# Coordinates of the cluster centers.
cluster_centers = ms.cluster_centers_

# Count our clusters.
n_clusters_ = len(np.unique(labels))

#Glue back to original data
X_tr['clusters'] = labels
X_tr['Gender'] = boston_marathon_scores.gender
X_tr['Overall'] = boston_marathon_scores.overall

#Add the column into our list
clmns.extend(['clusters','Gender','Overall'])

#Lets analyze the clusters
print("Number of estimated clusters: {}".format(n_clusters_))
pd.DataFrame(X_tr.groupby(['clusters']).mean())

clusters_summary = X_tr.groupby(['clusters']).describe()
clusters_summary_transposed = clusters_summary.transpose()
clusters_summary_transposed

# Reduce it to two components.
X_pca = PCA(2).fit_transform(X_tr_std)

# Calculate predicted values.
bandwidth = estimate_bandwidth(X_tr_std, quantile=0.9)
y_pred = MeanShift(bandwidth=bandwidth, bin_seeding=True).fit_predict(X_pca)

# Plot the solution.
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y_pred)
plt.show()

# Declare the model and fit it in one statement.
# Note that you can provide arguments to the model, but we didn't.
af = AffinityPropagation().fit(X_tr_std)
print('Done')

# Pull the number of clusters and cluster assignments for each data point.
cluster_centers_indices = af.cluster_centers_indices_
n_clusters_ = len(cluster_centers_indices)
labels = af.labels_

#Glue back to original data
X_tr['clusters'] = labels
X_tr['Gender'] = boston_marathon_scores.gender
X_tr['Overall'] = boston_marathon_scores.overall

#Add the column into our list
clmns.extend(['clusters','Gender','Overall'])

#Lets analyze the clusters
print("Number of estimated clusters:...


I am working on the file "Challenge+Boston+marathon.ipynb".
The first task is:
Spectral Clustering
Can you write Python code for it?

# We know we're looking for 6 clusters from the comparison with the kmeans.
n_clusters=6

# Declare and fit the model.
sc = SpectralClustering(n_clusters=n_clusters).fit(X_tr_std)

# Extract cluster assignments for each data point.
labels = sc.labels_

#Glue back to original data
X_tr['clusters'] = labels
X_tr['Gender'] = boston_marathon_scores.gender
X_tr['Overall'] = boston_marathon_scores.overall

#Add the column into our list
clmns.extend(['clusters','Gender','Overall'])

#Lets analyze the clusters

pd.DataFrame(X_tr.groupby(['clusters']).mean())

clusters_summary = X_tr.groupby(['clusters']).describe()
clusters_summary_transposed = clusters_summary.transpose()
clusters_summary_transposed

# Reduce it to two components.
X_pca = PCA(2).fit_transform(X_tr_std)

# Calculate predicted values.

y_pred = SpectralClustering(n_clusters=3).fit_predict(X_pca)

# Plot the solution.
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y_pred)
plt.show()


I am working on the file "Challenge+Boston+marathon.ipynb".
The first task is:
The elbow method shows that the optimal number of clusters to be used in the kmeans method is 3, considering the euclidean distance between cluster centers. From an analytical perspective, the inertia functions shows the same results: 3 clusters were the difference between the results obtained by the inertia function are smaller when shifting from 3 to 4 clusters.
Can you write Python code for it?

#Evaluate the best number of clusters
for i in range(1,10):
 km = KMeans(n_clusters=i, init='k-means++', n_init=10).fit(X_tr_std)
 print (i, km.inertia_)

#Cluster the data
kmeans = KMeans(n_clusters=3, init='k-means++', n_init=10).fit(X_tr_std)
labels = kmeans.labels_

#Glue back to original data
X_tr['clusters'] = labels
X_tr['Gender'] = boston_marathon_scores.gender
X_tr['Overall'] = boston_marathon_scores.overall

#Add the column into our list
clmns.extend(['clusters','Gender','Overall'])

#Lets analyze the clusters
pd.DataFrame(X_tr.groupby(['clusters']).mean())

clusters_summary = X_tr.groupby(['clusters']).describe()
clusters_summary_transposed = clusters_summary.transpose()
clusters_summary_transposed

# Reduce it to two components.
X_pca = PCA(2).fit_transform(X_tr_std)

# Calculate predicted values.
y_pred = KMeans(n_clusters=3, random_state=42).fit_predict(X_pca)

# Plot the solution.
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y_pred)
plt.show()

Graph_kmeans_official = pd.pivot_table(X_tr, 'official', ['clusters', 'gender'])
Graph_kmeans_pace = pd.pivot_table(X_tr, 'pace', ['clusters', 'gender'])
Graph_kmeans_age = pd.pivot_table(X_tr, 'age', ['clusters', 'gender'])
print(Graph_kmeans_official, Graph_kmeans_pace, Graph_kmeans_age)


I am working on the file "Challenge+Boston+marathon.ipynb".
The first task is:
the optimal number of kmeans will be determined using the elbow method. Once the kmeans number of clusters is set, the number of clusters using spectral clustering will be used so that it equals the silhouette score obtained in the first case.
K-Means
Can you write Python code for it?

#Use the elbow method to determine the number of clusters
# k-means determine k
distortions = []
K = range(1,10)
for k in K:
    kmeanModel = KMeans(n_clusters=k).fit(X_tr)
    kmeanModel.fit(X_tr)
    distortions.append(sum(np.min(cdist(X_tr, kmeanModel.cluster_centers_, 'euclidean'), axis=1)) / X_tr.shape[0])

# Plot the elbow
plt.plot(K, distortions, 'bx-')
plt.xlabel('k')
plt.ylabel('Distortion')
plt.title('The Elbow Method showing the optimal k')
plt.show()


I am working on the file "Challenge+Iterate+and+evaluate+your+classifier (2).ipynb".
The first task is:
Feature Selection & Engineering
Can you write Python code for it?

#Final iteration tried across different cuts. Accuracy >55%
keywords = [ 'nice', 'pleased', 'better',
        'like', 'easy', 'excellent',
        'love','impressed',
        'satisfied','pretty',
        'best','works great']

for key in keywords:
    # Note that we add spaces around the key so that we're getting the word,
    # not just pattern matching.
    amazon[str(key)] = amazon.Sentence.str.contains(
        ' '+str(key)+' ',
        case=False
    )

#Use the whole dataset
amazon['allcaps'] = amazon.Sentence.str.isupper()
amazon['Sentiment'] = (amazon['Sentiment'] == 1)
plt.figure(figsize=(20, 7))
sns.heatmap(amazon.corr())

# Define data & target for the model
data = amazon[keywords + ['allcaps']]
target = amazon['Sentiment']

# Initantiate our model and store it in a new variable.
bnb = BernoulliNB()

# Fit our model to the data.
bnb.fit(data, target)

# Classify, storing the result in a new variable.
y_pred = bnb.predict(data)

# Display our results.
print("Number of mislabeled points (w/o training) out of a total {} points : {}".format(
    data.shape[0],
    (target != y_pred).sum()
))

#Calculate Confusion Martrix
print(confusion_matrix(target, y_pred))

# from confusion matrix calculate accuracy
cm = confusion_matrix(target, y_pred)

#Accuracy
total=sum(sum(cm))
accuracy=(cm[0,0]+cm[1,1])/total
print ('Accuracy : ', accuracy)

#Specifity
Specifity = cm[0,0]/(cm[0,0]+cm[0,1])
print('Specifity : ', Specifity )

#Sensitivity
Sensitivity = cm[1,1]/(cm[1,0]+cm[1,1])
print('Sensitivity : ', Sensitivity)


I am working on the file "Challenge+Iterate+and+evaluate+your+classifier (2).ipynb".
The first task is:
Principal Components Analysis
Can you write Python code for it?

#Prepare data for PCA

X_std = StandardScaler().fit_transform(data)
mean_vec = np.mean(X_std, axis=0)
cov_mat = (X_std - mean_vec).T.dot((X_std - mean_vec)) / (X_std.shape[0]-1)

#Eigenvectores & Eigenvalues

cov_mat = np.cov(X_std.T)
eig_vals, eig_vecs = np.linalg.eig(cov_mat)

# Inspecting the eigenvalues and eigenvectors.
for i in range(len(eig_vals)):
    eigvecs = eig_vecs[:, i].reshape(1, len(keywords)+1).T
    print('Eigenvector {}: \n{}'.format(i + 1, eigvecs))
    print('Eigenvalue {}: {}'.format(i + 1, eig_vals[i]))
    print(40 * '-')

sklearn_pca = PCA(n_components=len(keywords))
Y_sklearn = sklearn_pca.fit_transform(X_std)

print(
    'The percentage of total variance in the dataset explained by each',
    'component from Sklearn PCA.\n',
    sklearn_pca.explained_variance_ratio_
)


I am working on the file "Challenge+Iterate+and+evaluate+your+classifier (2).ipynb".
The first task is:
Recursive Feature Elimination
Can you write Python code for it?

# create the RFE model and select attributes

rfe = RFE(bnb, len(keywords))
fit = rfe.fit(data,target)

# summarize the selection of the attributes

result_RFE = pd.DataFrame(list(zip(data.head(0), rfe.ranking_, rfe.support_)),columns=['Features','Ranking','Support'] )
result_RFE.sort_values('Ranking')

#Result as a list for "keywords"
result= result_RFE[result_RFE.Support]
my_list = result["Features"].tolist()
my_list


I am working on the file "Challenge+Iterate+and+evaluate+your+classifier (2).ipynb".
The first task is:
Splitting the data in a train and a test subset
Training the clasifier with 80% of the data
Can you write Python code for it?

for key in keywords:
    # Note that we add spaces around the key so that we're getting the word,
    # not just pattern matching.
    train[str(key)] = train.Sentence.str.contains(
        ' '+str(key)+' ',
        case=False
    )

#Using the train subset
train['allcaps'] = train.Sentence.str.isupper()
train['Sentiment'] = (train['Sentiment'] == 1)

#Plot correlation matrix
plt.figure(figsize=(20, 7))
sns.heatmap(train.corr())

# Define training data & target for the model
data1 = train[keywords + ['allcaps']]
target1 = train['Sentiment']

# Initantiate our model and store it in a new variable.
bnb2 = BernoulliNB()

# Fit our model to the data.
bnb2.fit(data1, target1)

# Classify, storing the result in a new variable.
y_pred1 = bnb2.predict(data1)

# Display our results.
print("Number of mislabeled points in TRAIN out of a total {} points : {}".format(
    data1.shape[0],
    (target1 != y_pred1).sum()
))

confusion_matrix(target1, y_pred1)

# Display our results.
print("Number of mislabeled points in TRAIN out of a total {} points : {}".format(
    data1.shape[0],
    (target1 != y_pred1).sum()
))

confusion_matrix(target1, y_pred1)

# from confusion matrix calculate accuracy
cm1 = confusion_matrix(target1, y_pred1)

total1=sum(sum(cm1))
accuracy1=(cm1[0,0]+cm1[1,1])/total1
print ('Accuracy : ', accuracy1)

Specifity1 = cm1[0,0]/(cm1[0,0]+cm1[0,1])
print('Specifity : ', Specifity1 )

Sensitivity1 = cm1[1,1]/(cm1[1,0]+cm1[1,1])
print('Sensitivity : ', Sensitivity1)


I am working on the file "Challenge+Make+Your+Network.ipynb".
The first task is:
Feature Selection
Can you write Python code for it?

#View all the predictors to make the feature selection
X1.columns

#Feature Selection using Random Forest
X3 = X1[['perimeter_worst', 'area_worst', 'concave points_mean', 'concavity_mean','radius_worst','perimeter_mean',
'concavity_worst', 'compactness_mean','concave points_worst','compactness_worst']]

#Feature Selection using RFE & PCA
X2 = X1[['radius_worst','concave points_worst','perimeter_worst']]

#Split the data into training and testing datasets. Split: 70/30; train/test

X_train, X_test, y_train, y_test = train_test_split(X2,Y, test_size=0.3, random_state=123)

#Initiating the cross validation generator, N splits = 5

kf = KFold(5)


I am working on the file "Challenge+Make+Your+Network.ipynb".
The first task is:
Gradient Boosting
Can you write Python code for it?

# Train model

GBC = GradientBoostingClassifier()

k1 = ['deviance','exponential']
k2 = np.arange(100)+1
k5 = ['friedman_mse','mse','mae']

parameters = {'loss': k1,
             'n_estimators': k2,
             'criterion': k5}

#Fit parameters

GBC1 = GridSearchCV(GBC, param_grid=parameters, cv=kf)

#Fit the tunned model
GBC1.fit(X_train, y_train)

#The best hyper parameters set
print("Best Hyper Parameters:", GBC1.best_params_)

#Fit on the test set
GBC1.fit(X_test, y_test)

# Predict on test set

predtestgb_y = GBC1.predict(X_test)

#Test Scores
target_names = ['0', '1']
print(classification_report(y_test, predtestgb_y, target_names=target_names))
cnf = confusion_matrix(y_test, predtestgb_y)
print(cnf)

table_test = pd.crosstab(y_test, predtestgb_y, margins=True)

test_tI_errors = table_test.loc[0.0,1.0]/table_test.loc['All','All']
test_tII_errors = table_test.loc[1.0,0.0]/table_test.loc['All','All']

accGBC1 = cross_val_score(GBC1,X_test,y_test,cv=kf).mean()
accGBC1pca = cross_val_score(GBC1,XPCA,Y,cv=kf).mean()

print((
    'Gradient Boosting accuracy:{}\n'
    'Gradient Boosting accuracy PCA:{}\n'
    'Percent Type I errors: {}\n'
    'Percent Type II errors: {}'
).format(accGBC1,accGBC1pca,test_tI_errors, test_tII_errors))


I am working on the file "Challenge+Make+Your+Network.ipynb".
The first task is:
Neural Network
Can you write Python code for it?

# Initialize and fit the model.
mlp = MLPClassifier(max_iter=1000, tol=0.01)

#Tune hyperparameters
#Create range of values to fit parameters

hidden_layer_sizes= [(50,),(100,),(1000,),(1000,10),(100,20)]

alpha  = 10.0 ** -np.arange(1, 3)

activation = ['identity', 'logistic', 'tanh', 'relu']

parameters = {'hidden_layer_sizes': hidden_layer_sizes,
             'alpha': alpha,
             'activation': activation}

#Fit parameters using gridsearch
mlp_tuned = GridSearchCV(mlp, param_grid=parameters, cv=5)

#Fit the tunned model
mlp_tuned.fit(X_train, y_train)

#Fit on the test set
mlp_tuned.fit(X_test, y_test)

# Predict on test set

predtest_y = mlp_tuned.predict(X_test)

#Test Scores
target_names = ['0', '1']
print(classification_report(y_test, predtest_y, target_names=target_names))
cnf = confusion_matrix(y_test, predtest_y)
print(cnf)

table_test = pd.crosstab(y_test, predtest_y, margins=True)

test_tI_errors = table_test.loc[0.0,1.0]/table_test.loc['All','All']
test_tII_errors = table_test.loc[1.0,0.0]/table_test.loc['All','All']

accmlp = cross_val_score(mlp_tuned,X_test,y_test,cv=kf).mean()
accmlppca = cross_val_score(mlp_tuned,XPCA,Y,cv=kf).mean()

print((
    'MLP accuracy:{}\n'
    'MLP accuracy PCA:{}\n'
    'Percent Type I errors: {}\n'
    'Percent Type II errors: {}'
).format(accmlp,accmlppca,test_tI_errors, test_tII_errors))

#Print the best parameters
print(mlp_tuned.best_params_)


I am working on the file "Challenge+Make+Your+Network.ipynb".
The first task is:
PCA Analysis
Can you write Python code for it?

# Build up the correlation mtrix
Z = X1
correlation_matrix = Z.corr()

#Eigenvectores & Eigenvalues

eig_vals, eig_vecs = np.linalg.eig(correlation_matrix)

sklearn_pca = PCA(n_components=len(Z.columns))
Y_sklearn = sklearn_pca.fit_transform(correlation_matrix)

#From the Scree plot.

plt.plot(eig_vals)
plt.show()

print(
    'The percentage of total variance in the dataset explained by each',
    'component from Sklearn PCA.\n',
    sklearn_pca.explained_variance_ratio_
)

#PCA features

# Create a scaler object
sc = StandardScaler()

# Fit the scaler to the features and transform
X_std = sc.fit_transform(X1)

# Create a PCA object from Scree plot the number of components is 3
pca = decomposition.PCA(n_components=3)

# Fit the PCA and transform the data
X_std_pca = pca.fit_transform(X_std)

# View the new feature data's shape
X_std_pca.shape

# Create a new dataframe with the new features

XPCA = pd.DataFrame(X_std_pca)
XPCA.head()

#Calculate Feature Importance using Random Forest
rf = RandomForestClassifier()
rf.fit(X1, Y)

#Define feature importance
feature_importance = rf.feature_importances_

# Make importances relative to max importance.
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
pos = np.arange(sorted_idx.shape[0]) + .5
plt.figure(figsize=(7, 30))
plt.subplot(1, 1, 1)
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, X1.columns[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Diagclass')
plt.show()

#Feature Selection. Scores for the most relevant features (should we start with the one that has more explanatory power)

# feature extraction
test = SelectKBest()
fit = test.fit(X1, Y)

#Identify features with highest score from a predictive perspective (for all programs)
names2 = X1.columns
Bestfeatures = pd.DataFrame(fit.scores_, index = names2)
Bestfeatures.columns = ['Best Features']
Bestfeatures.sort_values(by=['Best Features'], ascending=False)

# create the RFE model and select...


I am working on the file "Challenge+Preparing+a+dataset+for+modeling (1).ipynb".
The first task is:
From the correlation matrix it seems that displacement, horsepower and weight are strongly correlated. Acceleration is less correlated with the rest thus providing more information.
Can you write Python code for it?

df1 = df.drop(['carname'], axis=1)
df1.head()

# Plot all the variables with boxplots
dfb = df1.drop(['origin','modelyear'], axis=1)
df_long = dfb
df_long = pd.melt(df_long, id_vars=['cylinders'])

g = sns.FacetGrid(df_long, col="variable",size=10, aspect=.5)
g = g.map(sns.boxplot, "cylinders", "value")
g.fig.get_axes()[0].set_yscale('log')
sns.despine(left=True)
plt.show()


I am working on the file "Challenge+Preparing+a+dataset+for+modeling (1).ipynb".
The first task is:
Modelyear on average is equivalent regarding the population per year. There are differences regarding the cylinders values.
The group size differences are large enough to reflect differences on the population.
Create 10 new features
Can you write Python code for it?

#Feature 1: Standard number of cylinders vs high end number of cylinders

features = pd.get_dummies(dffinal1['cylinders'])
features['High_end'] = np.where((dffinal1['cylinders'].isin(['6.0', '8.0'])), 1, 0)
#print(pd.crosstab(features['High_end'], dffinal1['cylinders']))

#Feature 2: # Cars from the 70s and cars from the 80s.

features = pd.get_dummies(dffinal1['modelyear'])
features['decade'] = np.where((dffinal1['modelyear'].isin(range(70,80))), 1, 0)
#print(pd.crosstab(features['decade'], dffinal1['modelyear']))

# Feature 3: National cars vs imported cars

features = pd.get_dummies(dffinal1['origin'])
features['national'] = np.where((dffinal1['origin'].isin(['1'])), 1, 0)
#print(pd.crosstab(features['national'], dffinal1['origin']))

# Feature 4: Nacceleration: Normalized acceleration
# Making a four-panel plot.
fig = plt.figure()

fig.add_subplot(221)
plt.hist(dffinal1['acceleration'].dropna())
plt.title('Raw')

fig.add_subplot(222)
plt.hist(np.log(dffinal1['acceleration'].dropna()))
plt.title('Log')

fig.add_subplot(223)
plt.hist(np.sqrt(dffinal1['acceleration'].dropna()))
plt.title('Square root')

ax3=fig.add_subplot(224)
plt.hist(1/df['acceleration'].dropna())
plt.title('Inverse')
plt.show()

features['nacceleration'] = np.sqrt(dffinal1['acceleration'])

# Feature 5: CAR DHW. Composite of highly correlated variables

corrmat = dffinal1.corr()

# Set up the matplotlib figure.
f, ax = plt.subplots(figsize=(12, 9))

# Draw the heatmap using seaborn
sns.heatmap(corrmat, vmax=.8, square=True)
plt.show()

means = dffinal1[['displacement','horsepower','weight']].mean(axis=0)
stds = dffinal1[['displacement','horsepower','weight']].std(axis=0)
features['car_dhw'] = ((dffinal1[['displacement','horsepower','weight']] - means) / stds).mean(axis=1)

# Check how well the composite correlates with each of the individual variables.
plotdffinal1= dffinal1.loc[:, ['displacement','horsepower','weight']]
plotdffinal1['dhw'] = features['car_dhw']
corrmat2 = plotdffinal1.corr()

print(corrmat2)

# Feature 6:...


I am working on the file "Challenge+Preparing+a+dataset+for+modeling (1).ipynb".
The first task is:
The difference for all variables for each cylinders value is significant (except for acceleration when comparing 4 & 6)
Can you write Python code for it?

plt.figure(figsize=(20,5))
ax = sns.countplot(x="modelyear", hue='cylinders', data=dffinal1, palette="Set3")
plt.show()

# Table of counts
counttable = pd.crosstab(dffinal1['modelyear'], dffinal1['cylinders'])
print(counttable)

print(stats.chisquare(counttable, axis=None))


I am working on the file "Challenge+Preparing+a+dataset+for+modeling (1).ipynb".
The first task is:
The number of counts for cylinders = 3 and 5 is very small so they are discarded considering only 4, 6 & 8
Can you write Python code for it?

df1['cylinders'] = df1["cylinders"].astype(float)
df1 = df1.drop( df[(df.cylinders == 3.0)].index )
df1 = df1.drop( df[(df.cylinders == 5.0)].index )

df1['cylinders'] = df1['cylinders'].astype(str)
dffinal1 = df1[['cylinders','modelyear','origin','mpg','displacement','horsepower','weight','acceleration']]
dffinal1.head()

dffinal1['cylinders'].unique()

for col in dffinal1.loc[:,'mpg':'acceleration'].columns:
    print(col)
    print(stats.ttest_ind(
        dffinal1[dffinal1['cylinders'] == '4.0'][col],
        dffinal1[dffinal1['cylinders'] == '6.0'][col]
    ))

for col in dffinal1.loc[:,'mpg':'acceleration'].columns:
    print(col)
    print(stats.ttest_ind(
        dffinal1[dffinal1['cylinders'] == '4.0'][col],
        dffinal1[dffinal1['cylinders'] == '8.0'][col]
    ))

for col in dffinal1.loc[:,'mpg':'acceleration'].columns:
    print(col)
    print(stats.ttest_ind(
        dffinal1[dffinal1['cylinders'] == '6.0'][col],
        dffinal1[dffinal1['cylinders'] == '8.0'][col]
    ))


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+%28Feature+Selection%29.ipynb".
The first task is:
Feature selection using filtering methods
Feature Extraction with Univariate Statistical Tests (Chi-squared for classification)
Can you write Python code for it?

# Feature Extraction with Univariate Statistical Tests (Chi-squared for classification)

# Arrange data and transform the values into positive values
df_positive = df_scaled.abs().astype(str)
array = df_positive.values
X = array[:,0:10]
Y = array[:,7]
# feature extraction
test = SelectKBest(score_func=chi2, k=5)
fit = test.fit(X, Y)
# summarize scores
numpy.set_printoptions(precision=3)
print(fit.scores_)
features = fit.transform(X)
#summarize selected features
print(features[0:5,:])
df_positive.head(0)


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+%28Feature+Selection%29.ipynb".
The first task is:
From the correlation matrix it seems that displacement, horsepower and weight are strongly correlated. Acceleration is less correlated with the rest thus providing more information.
Can you write Python code for it?

#Arrange data for a bloxplot
df1 = df.drop(['carname'], axis=1)
df1.head()

# Plot all the variables with boxplots
dfb = df1.drop(['origin','modelyear'], axis=1)
df_long = dfb
df_long = pd.melt(df_long, id_vars=['cylinders'])

g = sns.FacetGrid(df_long, col="variable",size=10, aspect=.5)
g = g.map(sns.boxplot, "cylinders", "value")
g.fig.get_axes()[0].set_yscale('log')
sns.despine(left=True)
plt.show()


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+%28Feature+Selection%29.ipynb".
The first task is:
Getting a grasp of the data set. Which variables are continuous and which ones are categorical
Can you write Python code for it?

df['horsepower'].unique()

df['cylinders'].unique()

df['origin'].unique()

df['modelyear'].unique()

df['mpg'].unique()


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+%28Feature+Selection%29.ipynb".
The first task is:
The Research Questions is: Which are the features of the cars that best explain the miles per gallon.
1. Using a dataset of your choice, select an outcome variable and then pick four or five other variables (one to two categorical, three to four continuous) to act as the basis for features
Outcome variable: mpg
Categorical variables: cylynders, origin and year
Continuous: displacement, horsepower, weight, acceleration, displacement
Understand the relationship and behavior of the variables
Can you write Python code for it?

#Plotting the relationships between variables
sns.set_style("white")

#Drop the variables that will not be used
dfcont = df.drop(['carname','cylinders','modelyear','origin'], axis=1)
# Scatterplot matrix.
g = sns.PairGrid(dfcont, diag_sharey=False)
g.map_upper(plt.scatter, alpha=.5)
# Fit line summarizing the linear relationship of the two variables.
g.map_lower(sns.regplot, scatter_kws=dict(alpha=0))
# Give information about the univariate distributions of the variables.
g.map_diag(sns.kdeplot, lw=3)
plt.show()


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+%28Feature+Selection%29.ipynb".
The first task is:
Understand the correlation between variables
Can you write Python code for it?

# Make the correlation matrix.
corrmat = dfcont.corr()
print(corrmat)

# Set up the matplotlib figure.
f, ax = plt.subplots(figsize=(12, 9))

# Draw the heatmap using seaborn.
sns.heatmap(corrmat, vmax=.8, square=True)
plt.show()


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+PCA.ipynb".
The first task is:
Features for a car from the 70s or 80s that will give better mpg
Using a dataset of your choice, select an outcome variable and then pick four or five other variables (one to two categorical, three to four continuous) to act as the basis for features
Outcome variable: mpg
Categorical variables: cylynders, origin and year
Continuous: displacement, horsepower, weight, acceleration, displacement
Can you write Python code for it?

#Plotting the relationships between variables
sns.set_style("white")

dfcont = df.drop(['carname','cylinders','modelyear','origin'], axis=1)
# Declare that you want to make a scatterplot matrix.
g = sns.PairGrid(dfcont, diag_sharey=False)
# Scatterplot.
g.map_upper(plt.scatter, alpha=.5)
# Fit line summarizing the linear relationship of the two variables.
g.map_lower(sns.regplot, scatter_kws=dict(alpha=0))
# Give information about the univariate distributions of the variables.
g.map_diag(sns.kdeplot, lw=3)
plt.show()

#Some warnings will show up below because the plot does not include a legend.

# Make the correlation matrix.
corrmat = dfcont.corr()
print(corrmat)

# Set up the matplotlib figure.
f, ax = plt.subplots(figsize=(12, 9))

# Draw the heatmap using seaborn.
sns.heatmap(corrmat, vmax=.8, square=True)
plt.show()


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+PCA.ipynb".
The first task is:
For cylinders = 6 & 8: mpg, displacement & horsepower present outliers
For cylinders = 4, acceleration present outliers
Can you write Python code for it?

# Descriptive statistics by group.
df1.groupby('cylinders').describe().transpose()


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+PCA.ipynb".
The first task is:
From the correlation matrix it seems that displacement, horsepower and weight are strongly correlated. Acceleration is less correlated with the rest thus providing more information.
Can you write Python code for it?


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+PCA.ipynb".
The first task is:
Modelyear on average is equivalent regarding the population per year. There are differences regarding the cylinders values.
The group size differences are large enough to reflect differences on the population.
Create 10 new features
Can you write Python code for it?


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+PCA.ipynb".
The first task is:
The difference for all variables for each cylinders value is significant (except for acceleration when comparing 4 & 6)
Can you write Python code for it?


I am working on the file "Challenge+Preparing+a+dataset+for+modeling+PCA.ipynb".
The first task is:
The number of counts for cylinders = 3 and 5 is very small so they are discarded considering only 4, 6 & 8
Can you write Python code for it?


I am working on the file "Challenge+Project+Euler.ipynb".
The first task is:
Problem 1
If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23. Find the sum of all the multiples of 3 or 5 below 1000.
Can you write Python code for it?

#Set up the sum of multiples and reset to zero
multiples_sum = 0

#Create the function that divides all the numbers from 1-1000 by 3 or 5 and add them
for i in range(1, 1000):
    if (i % 3 == 0 or i % 5 == 0):
        multiples_sum = multiples_sum + i

#Print results
print (multiples_sum)


I am working on the file "Challenge+Project+Euler.ipynb".
The first task is:
Problem 10
The sum of the primes below 10 is 2 + 3 + 5 + 7 = 17. Find the sum of all the primes below two million.
Can you write Python code for it?

#Define the number that is prime
def prime(num):
    if num > 2 and num % 2 == 0:
        return False
    else:
        for i in range(3, int(math.sqrt(num)) + 1, 2):
            if num % i == 0:
                return False
    return True

#Define a function that is adding up to the limit
def adding_up(limit):
    sum = 0
    for i in range(2, limit):
        if prime(i):
            sum += i
    return sum

print(adding_up(2000000))


I am working on the file "Challenge+Project+Euler.ipynb".
The first task is:
Problem 4
A palindromic number reads the same both ways. The largest palindrome made from the product of two 2-digit numbers is 9009 = 91 × 99. Find the largest palindrome made from the product of two 3-digit numbers.
Can you write Python code for it?

#Start the number
n = 0

#check all the multiplications made of numbers of three digits and stop when read in both directions is equal
for a in range(999, 100, -1):
    for b in range(a, 100, -1):
        x = a * b
        if x > n:
            s = str(a * b)
            if s == s[::-1]:
                n = a * b

#Print results
print(n)


I am working on the file "Challenge+Project+Euler.ipynb".
The first task is:
Problem 9
A Pythagorean triplet is a set of three natural numbers, a < b < c, for which,
a2 + b2 = c2
For example, 32 + 42 = 9 + 16 = 25 = 52.
There exists exactly one Pythagorean triplet for which a + b + c = 1000.
Find the product abc.
Can you write Python code for it?

#check all the numbers from 1 to 1000 that fulfill the a+b+c = 1000 condition
#if the lowest number fulfills the condition check the product of the squares
for a in range(1, 1000):
     for b in range(a, 1000):
         c = 1000 - a - b
         if c > 0:
             if c*c == a*a + b*b:
                 print (a*b*c)
                 break


I am working on the file "Challenge+Validating+a+linear+regression.ipynb".
The first task is:
Clean and prepare the new dataset
Can you write Python code for it?

#Transform FBI Raw Data
#Rename columns with row 3 from the original data set
testfbi2014 = testfbi2014.rename(columns=testfbi2014.iloc[3])

#Delete first three rows don´t contain data for the regression model
testfbi2014 = testfbi2014.drop(testfbi2014.index[0:4])

#Delete columns containing "Rape"
testfbi2014 = testfbi2014.drop(['City','Arson3','Rape\n(revised\ndefinition)1','Rape\n(legacy\ndefinition)2'], axis = 1)

#Change names in Columns
testfbi2014 = testfbi2014.rename(columns={'Violent\ncrime': 'Violent Crime', 'Murder and\nnonnegligent\nmanslaughter': 'Murder', 'Robbery': 'Robbery', 'Aggravated\nassault': 'Assault', 'Property\ncrime': 'PropertyCrime', 'Burglary': 'Burglary', 'Larceny-\ntheft': 'Larceny & Theft', 'Motor\nvehicle\ntheft': 'MotorVehicleTheft'})

#Clean NaN values from dataset and reset index
testfbi2014 = testfbi2014.dropna().reset_index(drop=True)

#Convert objects to floats
testfbi2014.astype('float64').info()

#Scale and preprocess the dataset
names = testfbi2014.columns
fbi2014_scaled = pd.DataFrame(preprocessing.scale(testfbi2014), columns = names)


I am working on the file "Challenge+Validating+a+linear+regression.ipynb".
The first task is:
Cross Validation & Predictive Power of the "Challenge: make your own regression model" model
Can you write Python code for it?

X1 = fbi2014_scaled.drop(['Violent Crime','Murder','Larceny & Theft','PropertyCrime','MotorVehicleTheft','Assault'],axis=1)
Y1 = fbi2014_scaled['PropertyCrime'].values.ravel()

#Initiating the cross validation generator, N splits = 10

kf = KFold(20)

#Cross validate the model on the folds
loaded_model.fit(X1,Y1)
scores = cross_val_score(loaded_model, X1, Y1, cv=kf)
print('Cross-validated scores:', scores)
print('Cross-validation average:', scores.mean())

#Predictive accuracy

predictions = cross_val_predict(loaded_model, X1, Y1, cv=kf)
accuracy = metrics.r2_score(Y1, predictions)
print ('Cross-Predicted Accuracy:', accuracy)

# Instantiate and fit our model.
regr1 = linear_model.LinearRegression()
regr1.fit(X1, Y1)

# Inspect the results.
print('\nCoefficients: \n', regr1.coef_)
print('\nIntercept: \n', regr1.intercept_)
print('\nVariables in the model: \n',list(X1.columns))

#Cross validate the new model on the folds

scores = cross_val_score(regr1, X1, Y1, cv=kf)
print('Cross-validated scores:', scores)
print('Cross-validation average:', scores.mean())

#Cross validation, scores
predictions = cross_val_predict(regr1, X1, Y1, cv=kf)
accuracy = metrics.r2_score(Y1, predictions)
print ('Cross-Predicted Accuracy:', accuracy)

# Fit a linear model using Partial Least Squares Regression.
# Reduce feature space to 2 dimensions.
pls1 = PLSRegression(n_components=2)

# Reduce X to R(X) and regress on y.
pls1.fit(X1, Y1)

# Save predicted values.
PLS_predictions = pls1.predict(X1)
print('R-squared PLSR:', pls1.score(X1, Y1))
print('R-squared LR:', scores.mean())

# Compare the predictions of the two models
plt.scatter(predictions,PLS_predictions)
plt.xlabel('Predicted by original 3 features')
plt.ylabel('Predicted by 2 features')
plt.title('Comparing LR and PLSR predictions')
plt.show()


I am working on the file "Challenge+Validating+a+linear+regression.ipynb".
The first task is:
Load new dataset
Can you write Python code for it?

#Load data form excel spreadsheet into pandas
xls_file = pd.ExcelFile('D:\\Users\\Borja.gonzalez\\Desktop\\Thinkful-DataScience-Borja\\Test_fbidata2014.xlsx')

# View the excel file's sheet names
#xls_file.sheet_names

# Load the xls file's 14tbl08ny as a dataframe
testfbi2014 = xls_file.parse('14tbl08ny')


I am working on the file "Challenge+What+test+to+use+.ipynb".
The first task is:
In the Tukey HSD test the null hypothesis is that all means being compared are from the same population.
This test confirms the result obtained by the one way ANOVA test. In this case the rejection of the null hypothesis is true being the means significantly different and therefore having populations that are belive that others are more unfair if they live alone than when they live with their partners.
5. Pick three or four of the countries in the sample and compare how often people met socially in 2014. Are there differences, and if so, which countries stand out?
Can you write Python code for it?

#Get info of the dataset once reduced to the trust variable
#Slice the dataframe to the people trust variable
meet = raw_data[['cntry','year','sclmeet']]

#Info
meet.info()

#Slice by the year 2014
meet_2014 = meet[meet.year == 7]

#Drop year
meet_2014bis = meet_2014.drop('year',axis = 1)

#Get information of the new dataframe
meet_2014bis.info()

#Clean the values in the dataframe that are null (country)
meet_2014bis_cntr_cl = meet_2014bis[meet_2014bis.cntry.notnull()]

#Clean the values in the dataframe that are null (social meet)
meet_cle = meet_2014bis_cntr_cl[meet_2014bis_cntr_cl.sclmeet.notnull()]

#Reset index
meet_clea = meet_cle.reset_index()

#Drop old index
meet_clean = meet_clea.drop('index',axis = 1)

#Check Values
meet_clean.info()

#SLice the dataframe for three countries
meet_final = meet_clean.loc[meet_clean['cntry'].isin(['ES','SE','CH'])]

#Plot happy variables for visual inspection per country and year
meet_final.groupby('cntry').hist(sharey=True,sharex=True)

#Downsample majority class (due to computational restrictions we downsample the majority instead of upsampling the minority)

# Separate majority and minority classes
meet_final_majority_1 = meet_final[meet_final.cntry == 'ES']
meet_final_majority_2 = meet_final[meet_final.cntry == 'SE']
meet_final_minority = meet_final[meet_final.cntry == 'CH']

# Downsample the majority
meet_final_majority_1_downsampled = resample(meet_final_majority_1, replace=False, n_samples=772, random_state=123)
meet_final_majority_2_downsampled = resample(meet_final_majority_2, replace=False, n_samples=772, random_state=123)

# Combine minority class with downsampled majority class
meet_complete = pd.concat([meet_final_majority_1_downsampled, meet_final_majority_2_downsampled, meet_final_minority])

# Display new class counts
meet_complete.cntry.value_counts()

#Build a new dataframe with the total tv hours per gender
meet_reordered = pd.DataFrame()

#Create lists splitting between countries
ES = list(meet_complete[meet_complete.cntry ==...


I am working on the file "Challenge+What+test+to+use+.ipynb".
The first task is:
In this case we have run all the comparisons between countries. We are assuming that all of them follow a normal distribution.
From all the comparisons we are only focusing on those that are within countries between both years.
In the Tukey HSD test the null hypothesis is that all means being compared are from the same population.
Only in the cases of Spain and Norway the rejection of the null hypothesis is true being the means significantly different and therefore having having populations that are more trusting the case of Spain and less in the case of Norway. The rest of the countries under analysis remain the same.
2. Did people become happier from 2012 to 2014? Compute results for each country in the sample
Can you write Python code for it?

#Get info of the dataset once reduced to the trust variable

#Slice the dataframe to the people happiness variable
pplhappy = raw_data[['cntry','cntry_year','happy']]

#Get info about the data
pplhappy.info()

#Clean the values in the dataframe that are null
pplhappy_clean = pplhappy[pplhappy.happy.notnull()]

remove_n = 7
drop_indices = np.random.choice(pplhappy_clean.index, remove_n, replace=False)
pplhappy_clean = pplhappy_clean.drop(drop_indices)
#Check the remaining values
pplhappy_clean.info()

#Check the sizes of data by country
pplhappy_clean.groupby('cntry_year').size()

#Plot happy variables for visual inspection per country and year
pplhappy_clean.groupby('cntry_year').hist(sharey=True,sharex=True)

#Build the lists for each country each year
H_CH_6 = pplhappy_clean[pplhappy_clean.cntry_year == 'CH_6']['happy'].tolist()
H_CH_7 = pplhappy_clean[pplhappy_clean.cntry_year == 'CH_7']['happy'].tolist()
H_CZ_6 = pplhappy_clean[pplhappy_clean.cntry_year == 'CZ_6']['happy'].tolist()
H_CZ_7 = pplhappy_clean[pplhappy_clean.cntry_year == 'CZ_7']['happy'].tolist()
H_DE_6 = pplhappy_clean[pplhappy_clean.cntry_year == 'DE_6']['happy'].tolist()
H_DE_7 = pplhappy_clean[pplhappy_clean.cntry_year == 'DE_7']['happy'].tolist()
H_ES_6 = pplhappy_clean[pplhappy_clean.cntry_year == 'ES_6']['happy'].tolist()
H_ES_7 = pplhappy_clean[pplhappy_clean.cntry_year == 'ES_7']['happy'].tolist()
H_NO_6 = pplhappy_clean[pplhappy_clean.cntry_year == 'NO_6']['happy'].tolist()
H_NO_7 = pplhappy_clean[pplhappy_clean.cntry_year == 'NO_7']['happy'].tolist()
H_SE_6 = pplhappy_clean[pplhappy_clean.cntry_year == 'SE_6']['happy'].tolist()
H_SE_7 = pplhappy_clean[pplhappy_clean.cntry_year == 'SE_7']['happy'].tolist()

#Build the arrays for comparison
happy = np.asarray(H_CH_6 + H_CH_7 +
                   H_CZ_6 + H_CZ_7 +
                   H_DE_6 + H_DE_7 +
                   H_ES_6 + H_ES_7 +
                   H_NO_6 + H_NO_7 +
                   H_SE_6 + H_SE_7)

group_happy = np.array(['H_CH_6', 'H_CH_7',
                    ...


I am working on the file "Challenge+What+test+to+use+.ipynb".
The first task is:
It appears that the groups do not differ significantly. The test says there is a 43.8% chance of getting our data if there were no systematic difference between in the total tv watching hours per gender in the population. Based on our data, tv watching hours doesn't seem to differ systematically across gender.
4. Who was more likely to believe people were fair in 2012, people living with a partner or people living alone?
Can you write Python code for it?

#Get info of the dataset once reduced to the trust variable
#Slice the dataframe to the people trust variable
pplfairness = raw_data[['partner','year','pplfair']]

#Info
pplfairness.info()

#Slice by the year 2012
pplfairness_2012 = pplfairness[pplfairness.year == 6]

#Drop year
pplfairness_2012bis = pplfairness_2012.drop('year',axis = 1)

#Get information of the new dataframe
pplfairness_2012bis.info()

#Clean the values in the dataframe that are null (partner)
pplfairness_2012bis_partner_cl = pplfairness_2012bis[pplfairness_2012bis.partner.notnull()]

#Clean the values in the dataframe that are null (total hours)
pplfairness_cle = pplfairness_2012bis_partner_cl[pplfairness_2012bis_partner_cl.pplfair.notnull()]

#Reset index
pplfairness_clea = pplfairness_cle.reset_index()

#Drop old index
pplfairness_clean = pplfairness_clea.drop('index',axis = 1)

#Check Values
pplfairness_clean.info()

#Downsample majority class (due to computational restrictions we downsample the majority instead of upsampling the minority)

# Separate majority and minority classes
pplfairness_clean_majority = pplfairness_clean[pplfairness_clean.partner == 1.0]
pplfairness_clean_minority = pplfairness_clean[pplfairness_clean.partner == 2.0]

# Downsample the majority
pplfairness_clean_majority_downsampled = resample(pplfairness_clean_majority, replace=False, n_samples=1608, random_state=123)

# Combine minority class with downsampled majority class
pplfairness_complete = pd.concat([pplfairness_clean_majority_downsampled, pplfairness_clean_minority])

# Display new class counts
pplfairness_complete.partner.value_counts()

#Plot happy variables for visual inspection per country and year
pplfairness_complete.groupby('partner').hist(sharey=True,sharex=True)


I am working on the file "Challenge+What+test+to+use+.ipynb".
The first task is:
It appears that the two groups of people (in 2012) living with a partner or alone are likely to differ in the population (p < .005).  Visual inspection suggests that the people living alone believe that people are not fair compared to the ones living with their partner. To test whether the people living alone is the odd one out, we compare each group to each other.
The results of this test are interesting! Contrary to what we might have expected from a visual inspection of the histograms, the means of all three groups seem to be significantly different from one another. Wood is highest, then Steel, then Plastic.
Can you write Python code for it?

#Tukey HSD test

#Build the arrays for comparison
fair = np.asarray(wpartner + alone)

group_fair = np.array(['wpartner', 'alone'])

group_fair = np.repeat(group_fair, len(fair)/2)

#Run the pairwise t-test using tukey hsd
tukey = pairwise_tukeyhsd(endog=fair,    # Data
                          groups=group_fair,   # Groups
                          alpha=0.05)     # Significance level

#Summarize the data
tukey.summary()


I am working on the file "Challenge+What+test+to+use+.ipynb".
The first task is:
The distribution of total tv watching hours is non parametric as it can be appreciated when plotting the hours by gender for 2012.
Can you write Python code for it?

#Build a new dataframe with the total tv hours per gender
tv_total_reordered = pd.DataFrame()

#Create lists splitting between men and women the total tv watching hours
men = list(tv_total_complete[tv_total_complete.gndr == 1.0]['tvtot'])
women = list(tv_total_complete[tv_total_complete.gndr == 2.0]['tvtot'])

#Build columns and add to the dataframe
tv_total_reordered['men'] = men
tv_total_reordered['women'] = women

# Kruskal-Wallace Test

# Join all ratings together into a list, then ranking them.
ranks = stats.rankdata(
    tv_total_reordered['men'].tolist() +
    tv_total_reordered['women'].tolist())

# Add the new ranked variables to the data frame.
tv_total_reordered['Ranks_men'] = ranks[0:2140]
tv_total_reordered['Ranks_women'] = ranks[2140:4280]

# Average rank per group.
groups = ['Ranks_men', 'Ranks_women']
print(tv_total_reordered[groups].apply(np.mean))

# Overall average rank.
print(np.mean(ranks))

# Print the test statistic followed by the probability of getting this result
# if the groups were not different.
stats.kruskal(
    tv_total_reordered['men'],
    tv_total_reordered['women'])


I am working on the file "Challenge01.ipynb".
The first task is:
Try pandas.merge
Why merge might be the better apporach?
Can you write Python code for it?

# How to use merge to make a combined dataframe?


I am working on the file "Challenge01.ipynb".
The first task is:
Using pivot_table to summarize data

How many snow days and non-snow days are there for each month?
Can you generate the following result, say, with the merged data?
Dose the result make sense to you
If not, why it dosen't and how to fix it?
Can you write Python code for it?

# Assuming your now have the combined dataframe from above, named df
# How to make the following summary table using pivot_table?


I am working on the file "Challenge01_key.ipynb".
The first task is:
Try pandas.concat
Can you write Python code for it?

pd.concat((df_prcp,
           pd.DataFrame(data=df_temp.values,
                        index=df_prcp.index,
                        columns=df_temp.columns)),
          axis=1).head()


I am working on the file "Challenge01_key.ipynb".
The first task is:
Try pandas.merge
Why merge might be the better apporach?
Can you write Python code for it?

pd.merge(left=df_prcp,
         right=df_temp,
         left_on=df_prcp.index.dayofyear,
         right_index=True,
         how='left').head()


I am working on the file "Challenges/MNIST with Multi-Layer Perceptron.ipynb".
The first task is:
Cost and Optimization Functions
We'll use Tensorflow's built-in functions for this part (check out the documentation for a lot more options and discussion on this):
Can you write Python code for it?

# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=x))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)


I am working on the file "Challenges/MNIST with Multi-Layer Perceptron.ipynb".
The first task is:
In order to get a numerical value for our predictions we will need to use tf.cast to cast the Tensor of booleans back into a Tensor of Floating point values in order to take the mean of it.
Can you write Python code for it?

correct_predictions = tf.cast(correct_predictions, "float")

print(correct_predictions[0])


I am working on the file "Challenges/MNIST with Multi-Layer Perceptron.ipynb".
The first task is:
Initialization of Variables
Now initialize all those tf.Variable objects we created earlier. This will be the first thing we run when training our model:
Can you write Python code for it?

# Initializing the variables
init = tf.initialize_all_variables()


I am working on the file "Challenges/MNIST with Multi-Layer Perceptron.ipynb".
The first task is:
MultiLayer Model
It is time to create our model, let's review what we want to create here.
First we receive the input data array and then to send it to the first hidden layer. Then the data will begin to have a weight attached to it between layers (remember this is initially a random value) and then sent to a node to undergo an activation function (along with a Bias as mentioned in the lecture). Then it will continue on to the next hidden layer, and so on until the final output layer. In our case, we will just use two hidden layers, the more you use the longer the model will take to run (but it has more of an opportunity to possibly be more accurate on the training data).
Once the transformed "data" has reached the output layer we need to evaluate it. Here we will use a loss function (also called a cost function) to evaluate how far off we are from the desired result. In this case, how many of the classes we got correct.
Then we will apply an optimization function to minimize the cost (lower the error). This is done by adjusting weight values accordingly across the network. In out example, we will use the Adam Optimizer, which keep in mind, relative to other mathematical concepts, is an extremely recent development.
We can adjust how quickly to apply this optimization by changing our earlier learning rate parameter. The lower the rate the higher the possibility for accurate training results, but that comes at the cost of having to wait (physical time wise) for the results. Of course, after a certain point there is no benefit to lower the learning rate.
Now we will create our model, we'll start with 2 hidden layers, which use the [RELU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks) activation function, which is a very simple rectifier function which essentially either returns x or zero. For our final output layer we will use a linear activation with matrix multiplication:
Can you write Python code for it?

def multilayer_perceptron(x, weights, biases):
    '''
    x : Place Holder for Data Input
    weights: Dictionary of weights
    biases: Dicitionary of biases
    '''

    # First Hidden layer with RELU activation
    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
    layer_1 = tf.nn.relu(layer_1)

    # Second Hidden layer with RELU activation
    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
    layer_2 = tf.nn.relu(layer_2)

    # Last Output layer with linear activation
    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
    return out_layer


I am working on the file "Challenges/MNIST with Multi-Layer Perceptron.ipynb".
The first task is:
Now we use the tf.reduce_mean function in order to grab the mean of the elements across the tensor.
Can you write Python code for it?

accuracy = tf.reduce_mean(correct_predictions)

type(accuracy)


I am working on the file "Challenges/MNIST with Multi-Layer Perceptron.ipynb".
The first task is:
Parameters
We'll need to define 4 parameters, it is really (really) hard to know what good parameter values are on a data set for which you have no experience with, however since MNIST is pretty famous, we have some reasonable values for our data below. The parameters here are:

Learning Rate - How quickly to adjust the cost function.
Training Epochs - How many training cycles to go through
Batch Size - Size of the 'batches' of training data
Can you write Python code for it?

# Parameters
learning_rate = 0.001
training_epochs = 150
batch_size = 100


I am working on the file "Challenges/MNIST with Multi-Layer Perceptron.ipynb".
The first task is:
Running the Session
Now it is time to run our session! Pay attention to how we have two loops, the outer loop which runs the epochs, and the inner loop which runs the batches for each epoch of training. Let's breakdown each step!
Can you write Python code for it?

# Launch the session
sess = tf.InteractiveSession()

# Intialize all the variables
sess.run(init)

# Training Epochs
# Essentially the max amount of loops possible before we stop
# May stop earlier if cost/loss limit was set
for epoch in range(training_epochs):

    # Start with cost = 0.0
    avg_cost = 0.0

    # Convert total number of batches to integer
    total_batch = int(n_samples/batch_size)

    # Loop over all batches
    for i in range(total_batch):

        # Grab the next batch of training data and labels
        batch_x, batch_y = mnist.train.next_batch(batch_size)

        # Feed dictionary for optimization and loss value
        # Returns a tuple, but we only need 'c' the cost
        # So we set an underscore as a "throwaway"
        _, c = sess.run([optimizer, cost], feed_dict={x: batch_x, y: batch_y})

        # Compute average loss
        avg_cost += c / total_batch

    print("Epoch: {} cost={:.4f}".format(epoch+1,avg_cost))

print("Model has completed {} Epochs of Training".format(training_epochs))


I am working on the file "Challenges/MNIST with Multi-Layer Perceptron.ipynb".
The first task is:
TensorFlow Graph Input
Can you write Python code for it?

x = tf.placeholder("float", [None, n_input])
y = tf.placeholder("float", [None, n_classes])


I am working on the file "ChamaleonI_Tab.ipynb".
The first task is:
Calculate the Disk Mass with the function above
Can you write Python code for it?

d=160        #pc distance of Chamaleon in pc
r=1          #dust to gass mass ratio
k=2.3*(338./230)**(0.4)        #cm^2/g at 887um
lam=0.887    #wavelenght in mm

Td=20        #Temperature of the disk in K

Md=Mdisk(Fmm*1e-3,d,r,k,Td,lam)/3e-6         #Fmm has to be in *Jy*; Md in Earth masses!
logMd=np.log10(Md)

e_Md=eMd(ecal_Fmm,Fmm,Md)  #error on Md

print name[0], logMd[0]

T=Table([name,logMs,up_logMs,down_logMs,upp, Md,e_Md, Fmm, e_Fmm,ecal_Fmm],
        names=('Name','logM*','up_logM*','down_logM*','l_Md','Md','e_Md', 'Fmm','e_Fmm','e+cal_Fmm'))

T.write('ChaI_Tab.fit', format='fits', overwrite='True')


I am working on the file "Chap3-01-最尤推定.ipynb".
The first task is:
${\bf w}$ の最尤推定
重みの最尤推定は以下の式でもとまる
$$
{\bf w}_{ML} = (\Phi^T\Phi)^{-1} \Phi^T{\bf t}
$$
ただし、${\bf \Phi} \in R^{N \times M}$ は計画行列で、その $i$ 行 $j$ 列要素 は　$\phi_j(x_i)$
Can you write Python code for it?

# 計画行列を求める
# s は全て 1 で固定。

def makeDesignMatrix(x, basis, mu, s):
    ret = np.zeros((len(x), len(mu)))
    for i in range(len(x)):
        for j in range(len(mu)):
            ret[i][ j] = basis(x[i], j, mu, s)
    return ret

mu = np.linspace(min(x_opt), max(x_opt), M)
s = 1

designMatrix = {}
plt.figure(figsize=(16, 3))
for ikey, key in enumerate(basis):
    designMatrix[key] = makeDesignMatrix(x_opt, basis[key], mu, s)
    plt.subplot(1, 4, ikey+1)
    plt.title(key + "(M=%d)" % M)
    plt.imshow(designMatrix[key], aspect="auto", interpolation="nearest")

# 基底関数ごとに最尤推定を行い、結果をプロット
for M in [1, 3, 5, 7, 10]:
    mu = np.linspace(min(x_opt), max(x_opt), M)
    s = 1.0

    plt.figure(figsize=(16, 3))
    for ikey, key in enumerate(basis):
        phi = makeDesignMatrix(x_opt, basis[key], mu, s) + np.random.uniform(0, 0.001, (N, M))
        phit = phi.transpose()
        wml = np.dot(np.dot(np.linalg.inv(np.dot(phit, phi)), phit), y_obs)
        bml = 1.0 / (1.0 / N + sum([(y_obs[i] - np.dot(wml.transpose(), phi[i, :]))**2 for i in range(N)]))

        plt.subplot(1, 4, ikey+1)
        plt.title(key + "(M=%d)" % M)
        plt.plot(x_opt, [np.dot(wml.transpose(), phi[i, :]) for i in range(N)], "k", label="Estimated")
        plt.plot(x_opt, y_opt, "r", label="Oracle")
        plt.plot(x_opt, y_obs, "r.",  markersize=2)

        plt.ylim([-3, 10])
        plt.xlim([min(x_opt), max(x_opt)])
        plt.legend()


I am working on the file "Chap3-01-最尤推定.ipynb".
The first task is:
正則化項
誤差関数　$E_D({\bf w})$ に正則化項 $E_W({\bf w})$ と正則化係数 $\lambda$ を追加する
$$
\underset{\bf w}{\operatorname{argmin}} E_D({\bf w}) + \lambda E_W({\bf x})\
E_D({\bf w}) = \frac{1}{2} \sum_{i=1}^{N} \left(t_i - {\bf w}^T\phi({\bf x}_i)\right)^2 \
E_W({\bf w}) = \frac{1}{2} {\bf w}^T{\bf w}
$$
最尤推定は
$$
{\bf w}_{ML} = (\lambda{\bf I} + \Phi^T\Phi)^{-1} \Phi^T{\bf t}
$$
Can you write Python code for it?

# 基底関数ごとに, M と　lambda を変えながら w の最尤推定を行い、結果をそれぞれプロット
for M in [1, 5, 10, 50]:
    mu = np.linspace(min(x_opt), max(x_opt), M)
    s = 1.0

    plt.figure(figsize=(16, 3))
    for ikey, key in enumerate(basis):
        phi = makeDesignMatrix(x_opt, basis[key], mu, s) + np.random.uniform(0, 0.001, (N, M))
        phit = phi.transpose()

        plt.subplot(1, 4, ikey+1)
        plt.title(key + "(M=%d)" % M)
        for lmd in np.linspace(0, 5, 5):
            wml = np.dot(np.dot(np.linalg.inv(lmd * np.eye(M) + np.dot(phit, phi)), phit), y_obs)
            plt.plot(x_opt, [np.dot(wml.transpose(), phi[i, :]) for i in range(N)], "k", label="Estimated" if lmd == 0 else "")

        plt.plot(x_opt, y_opt, "r", label="Oracle")
        plt.plot(x_opt, y_obs, "r.",  markersize=2)

        plt.ylim([-3, 10])
        plt.xlim([min(x_opt), max(x_opt)])
        plt.legend()

mu = np.linspace(min(x_opt), max(x_opt), M)
phi = makeDesignMatrix(x_opt, basis["Gauss"], mu, s)
phit = phi.transpose()
wml = np.dot(np.dot(np.linalg.inv(lmd * np.eye(M) + np.dot(phit, phi)), phit), y_obs)

plt.figure(figsize=(16, 3))
plt.subplot(1, 4, 1)
plt.imshow(phi, aspect="auto", interpolation="nearest")
plt.title("Design Matrix")
plt.ylabel("x")
plt.xlabel("weight")

plt.subplot(1, 4, 2)
out = np.zeros((N, M))
plt.plot(wml)
plt.title("Weight Vector")
#plt.imshow(out,  aspect="auto", interpolation="nearest")

plt.subplot(1, 4, 3)
dw = (wml.transpose() * phi).transpose()
plt.imshow(dw.transpose(), aspect="auto", interpolation="nearest")
plt.title("Design x Weight")

plt.subplot(1, 4, 4)
plt.plot(x_opt, dw.sum(0))
plt.plot(x_opt, y_opt, "r:", linewidth=3)
plt.title("sum(Design x Weight)")


I am working on the file "Chap3-02-ベイズ推定.ipynb".
The first task is:
${\bf w}$ のベイズ推定
${\bf w}$ の事前分布
観測は加法性ガウスノイズを持つので、共役事前分布はガウス分布
$$
p({\bf w}) = N ({\bf w} | {\bf m_0, S_0})
$$
$w$ に以下の仮定を置く
$$
p({\bf w}) = N ({\bf w} | {0}, \alpha^{-1}{\bf I})
$$
${\bf w}$ の事後分布
事後分布
$$
p({\bf w} | {\bf t}) = N({\bf w} | {\bf m}_N, {\bf S}_N)\
{\bf m}_N = \beta {\bf S}_N{\bf \Phi}^T{\bf t}\
{\bf S}_N^{-1}=\alpha{\bf I}+ \beta {\bf \Phi}^T{\bf \Phi}
$$
予測分布の分散 (エラーバー)
$$
\sigma^2(x) = \frac{1}{\beta} + \phi({\bf x})^T {\bf S}_N \phi({\bf x})
$$
$\alpha$, $\beta$ の推定

適当な $\alpha$, $\beta$　を初期値に設定
次の $\gamma$ と、上式で定義される ${\bf m}_N$ を計算
$$
\gamma = \sum_i^{N} \frac{\lambda_i}{\alpha + \lambda_i}
$$
$\alpha$, $\beta$ を更新
$$
\alpha = \frac{\gamma}{{\bf m}N^T{\bf m}_N}
$$
$$
\frac{1}{\beta} = \frac{1}{N-\gamma}\sum{n=1}^{N}\left(t_n-{\bf m}_N^T \phi({\bf x}\right)^2
$$
2 に戻る
Can you write Python code for it?

# 計画行列を求める
def makeDesignMatrix(x, basis, mu, s):
    ret = np.zeros((len(x), len(mu)))
    for i in range(len(x)):
        for j in range(len(mu)):
            ret[i][ j] = basis(x[i], j, mu, s)
    return ret, ret.transpose()

M, key = 15, "Sigmoid"

alphas = []
betas = []

plt.figure(figsize=(16,8))
for index, size in enumerate(np.linspace(3, len(x_opt)-1, 10)):
    # extract data
    size = int(size)
    indices = sorted(np.random.choice(range(len(x_opt)), size, False))
    x_opt_tmp = x_opt[indices]
    y_opt_tmp = y_opt[indices]
    y_obs_tmp = y_obs[indices]

    # generate Phi
    mu = np.linspace(min(x_opt_tmp), max(x_opt_tmp), M)
    s = 1.0
    phi, phit = makeDesignMatrix(x_opt_tmp, basis[key], mu, s)
    phit = phi.transpose()

    # eigenvalues of Phi
    eigs = np.linalg.eigvalsh(beta * phit.dot(phi))

    # estimate values including alpha, beta
    alpha, beta = 0.0001, 0.1
    for i in range(10):
        sN = np.linalg.inv((alpha) * np.eye(M) + beta * phit.dot(phi))
        mN = beta * sN.dot(phit).dot(y_obs_tmp)
        mNt = mN.transpose()
        gamma = sum([eig / (alpha + eig) for eig in eigs])

        beta = 1 / (1/(size - gamma) * sum([(y_obs_tmp[i] - mNt.dot(phi[i, :]))**2 for i in range(size)]))
        alpha = gamma / (mNt.dot(mN))
        beta = np.clip(float(beta), 0, 10)
        alpha = np.clip(float(alpha), 0, 10)
    alphas.append(alpha)
    betas.append(beta)

    # Plot estimations and sigmas from estimated values
    # alpha, beta, mN, SN
    phi, phit = makeDesignMatrix(x_opt, basis[key], mu, s)
    y_out = np.array([np.dot(mN.transpose(), phi[i, :]) for i in range(N)])
    sigma = np.sqrt(np.array([1.0 / beta + phi[i, :].transpose().dot(sN).dot(phi[i, :]) for i in range(N)]))

    # Plot estimations
    plt.subplot(2, 5, index+1)
    plt.plot(x_opt, y_opt, "k:")
    plt.plot(x_opt_tmp, y_opt_tmp, "ko")
    plt.plot(x_opt, y_out, "r-")
    plt.fill_between(x_opt, y_out[:, 0] - sigma, y_out[:, 0] + sigma, facecolor="k", alpha=0.4)
    plt.ylim([0,...


I am working on the file "Chap3-02-ベイズ推定.ipynb".
The first task is:
例題:  $\sin$ 関数を使った以下の関係式を、$M$ 次元で回帰
観測は加法性ノイズ $\epsilon$ が含まれる
$$
y(x) = 2  \sin(x) + 3 + \epsilon \
\epsilon \sim N(0, \beta^{-1})
$$
* $x \in [-5, 5]$
* $M$: 線形回帰の次元数
* $N$: サンプル数
* $\beta$: 加法性ノイズの精度
Can you write Python code for it?

N = 30   # サンプル数
M = 5     # モデル次元
b  = 2.   # ノイズの精度

x_opt = np.linspace(-5, 5, N)
y_opt = 2 * np.sin(x_opt) + 3
y_obs = (y_opt + np.random.normal(0, 1.0/b, len(x_opt)))[:, np.newaxis]

plt.plot(x_opt, y_obs, "k.", label="Observation")
plt.plot(x_opt, y_opt, "r-", linewidth=3, label="Truth")
plt.xlim(-5, 5)
plt.legend()


I am working on the file "Chap3-02-ベイズ推定.ipynb".
The first task is:
基底関数

そのまま
    $$\phi_j(x) = x$$
多項式
    $$\phi_j(x) = x^j$$
ガウス
    $$\phi_j(x) = \exp\left(-\frac{(x-\mu_j)^2}{2s^2}\right)$$
ロジスティックシグモイド
    $$\phi_j(x) = \frac{1}{1 + \exp\left((x - \mu_j)/s\right)}$$

(図 3.1 (p137))
Can you write Python code for it?

def basisNone(x, dmy1=None, dmy2=None, dmy3=None):
    return x

def basisPoly(x, j, dmy1=None, dmy2=None):
    return x ** j

def basisGauss(x, j, mu, s):
    return np.exp(- (x-mu[j]) ** 2 / (2 * s ** 2))

def basisSigmoid(x, j, mu, s):
    return 1.0 / (1 + np.exp(-(x - mu[j])/s))

basis = {
    "Linear": basisNone,
    "Polynomial": basisPoly,
    "Gauss": basisGauss,
    "Sigmoid": basisSigmoid,
}

x = np.linspace(-1, 1, 100)
mu = np.linspace(-1, 1, 10)

plt.figure(figsize=(16, 3))
for ikey, key in enumerate(basis):
    plt.subplot(1, 4, ikey + 1)
    plt.title(key + " Kernel")
    for j in range(len(mu)):
        plt.plot(x, [basis[key](tmpx, j, mu, 0.1) for tmpx in x])

def y(x, w, basisName, mu=None, s=None):
    ret = w[0]
    for index in range(1, len(w)):
        ret += w[index] * basis[basisName](x, index, mu, s)
    return ret


I am working on the file "Chapman/Ch1-Example_1-10.ipynb".
The first task is:
Calculate the currents flowing in the motor:
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Example_1-10.ipynb".
The first task is:
Define all the parameters:
Can you write Python code for it?

VB = 120.0    # Battery voltage (V)
r = 0.3       # Resistance (ohms)
l = 1.0       # Bar length (m)
B = 0.6       # Flux density (T)


I am working on the file "Chapman/Ch1-Example_1-10.ipynb".
The first task is:
Plot the velocity of the bar versus force:
Can you write Python code for it?

plot(F, v_bar);
rc('text', usetex=True)   # enable LaTeX commands for plot
title(r'\textbf{Plot of velocity versus applied force}')
xlabel(r'\textbf{Force (N)}')
ylabel(r'\textbf{Velocity (m/s)}')
axis([0, 50, 0, 200])
grid()


I am working on the file "Chapman/Ch1-Example_1-10.ipynb".
The first task is:
Select the forces to apply to the bar:
Can you write Python code for it?

F = arange(0,51,10)    # Force (N)
F   # Lets print the variable to check.
# Can you exaplain why "arange(0,50,10)" gives not the array below?


I am working on the file "Chapman/Ch1-Problem_1-05.ipynb".
The first task is:
SOLUTION
There are three regions in this core. The top and bottom form one region, the left side forms a
second region, and the right side forms a third region. If we assume that the mean path length of the flux
is in the center of each leg of the core, and if we ignore spreading at the corners of the core, then the path
lengths are:
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-05.ipynb".
The first task is:
The flux density $B = \frac{\phi}{A}$ at the right side of the core is:
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-05.ipynb".
The first task is:
The flux density $B = \frac{\phi}{A}$ on the top of the core is:
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-05.ipynb".
The first task is:
The total reluctance is thus $\mathcal{R}_\text{TOT} = \mathcal{R}_1 + \mathcal{R}_2 + \mathcal{R}_3$:
Can you write Python code for it?

Rtot=  R1 + R2 + R3
print('Rtot = {:.1f} kAt/Wb'.format(Rtot/1000) )


I am working on the file "Chapman/Ch1-Problem_1-05.ipynb".
The first task is:
and the magnetomotive force required to produce a flux of 0.005 Wb is $\mathcal{F} = \phi \mathcal{R}_\text{TOT}$:
Can you write Python code for it?

F = phi * Rtot
print('F = {:.1f} At'.format(F) )


I am working on the file "Chapman/Ch1-Problem_1-06.ipynb".
The first task is:
And the reluctances are hence:
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-06.ipynb".
The first task is:
Description
A ferromagnetic core is shown in Figure P1-3:
<img src="figs/FigC_P1-3.jpg" width="70%">
The relative permeability of the core is:
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-06.ipynb".
The first task is:
The dimentions are as shown in the diagram, and the depth of the core is 5 cm.
The air gaps on the left and right sides of the core are 0.050 and 0.070 cm, respectively.
Because of fringing effects, the effective area of the air gaps is 5 percent larger than their physical size.
If there are 300 turns in the coil wrapped around the center leg of the core and if the current in the coil is 1.0 A,

What is the flux in each of the left, center, and right legs of the core?
What is the flux density in each air gap?

SOLUTION
This core can be divided up into five regions. Let:

$\mathcal{R}_1$ be the reluctance of the left-hand portion of the core,
$\mathcal{R}_2$ be the reluctance of the left-hand air gap,
$\mathcal{R}_3$ be the reluctance of the right-hand portion of the core,
$\mathcal{R}_4$ be the reluctance of the right-hand air gap,
$\mathcal{R}_5$ be the reluctance of the center leg of the core.

If we assume that the mean path length of the flux is in the center of each leg of the core, and if we ignore spreading at the corners of the core, then the path lengths are:
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-06.ipynb".
The first task is:
The flux density $B = \frac{\phi}{A}$ in the right air gap is:
Can you write Python code for it?

B4 = phi_right / A4
print('B4 = {:.3f} T'.format(B4))


I am working on the file "Chapman/Ch1-Problem_1-06.ipynb".
The first task is:
The fluxes in the left and right legs can be found by the "flux divider rule", which is analogous to the current divider rule.
Thus the flux in the left leg $\phi_\text{left} = \frac{(\mathcal{R}3 + \mathcal{R}_4)}{\mathcal{R}_1 + \mathcal{R}_2 + \mathcal{R}_3 + \mathcal{R}_4}\phi\text{TOT}$ is:
Can you write Python code for it?

phi_left = (R3 + R4) / (R1 + R2 + R3 + R4) * phi_cent
print('phi_left = {:.3f} mWb'.format(phi_left*1000))


I am working on the file "Chapman/Ch1-Problem_1-06.ipynb".
The first task is:
and the magnetomotive force is $\mathcal{F} = \mathcal{N} \mathcal{I}$:
Can you write Python code for it?

N = 300   # t   given in description
I = 1.0   # A   given in description
F = N * I


I am working on the file "Chapman/Ch1-Problem_1-07.ipynb".
The first task is:
Description
A two-legged core is shown in Figure P1-4 below:
<img src="figs/FigC_P1-4.jpg" width="70%">
The winding on the left leg of the core ($N_1$) has 600 turns,
and the winding on the right ($N_2$) has 200 turns. The coils are wound in the directions shown in the figure.

If the dimensions are as shown, then what flux would be produced by currents $i_1 = 0.5\,A$ and $i_2 = 1.0\,A$?

Assume $\mu_r = 1200$ and constant.
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-07.ipynb".
The first task is:
SOLUTION
The two coils on this core are wound so that their magnetomotive forces are additive, so the
total magnetomotive force on this core is
$$\mathcal{F}_\text{TOT} = N_1 i_1 + N_2 I_2$$
Can you write Python code for it?

F_tot = N1 * i1 + N2 * i2
print('F_tot =  {:.1f} At'.format(F_tot))


I am working on the file "Chapman/Ch1-Problem_1-07.ipynb".
The first task is:
The total reluctance in the core is $\mathcal{R}_\text{TOT} = \frac{l}{\mu_0 \mu_r A}$:
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-08.ipynb".
The first task is:
Description
A core with three legs is shown in Figure P1-5 below:
<img src="figs/FigC_P1-5.jpg" width="70%">
Its depth is 5 cm, and there are 100 turns on the leftmost
leg. The relative permeability of the core can be assumed to be 2000 and constant.

What flux exists in each of the three legs of the core?
What is the flux density in each of the legs?

Assume a 5% increase in the effective area of the air gap due to fringing effects.
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-08.ipynb".
The first task is:
The flux in the right leg $\phi_\text{right} = \frac{\mathcal{R}2 + \mathcal{R}_3}{\mathcal{R}_2 + \mathcal{R}_3 + \mathcal{R}_4}\phi\text{TOT}$ is:
Can you write Python code for it?

phi_right = (R2 + R3) / (R2 + R3 + R4) * phi_left
print('phi_right = {:.3f} mWb'.format(phi_right*1000))


I am working on the file "Chapman/Ch1-Problem_1-08.ipynb".
The first task is:
The fluxes in the center and right legs can be found by the "flux divider rule", which is analogous to the current divider rule.
Thus the flux in the left leg $\phi_\text{center} = \frac{ \mathcal{R}4}{\mathcal{R}_2 + \mathcal{R}_3 + \mathcal{R}_4}\phi\text{TOT}$ is:
Can you write Python code for it?

phi_center = R4 / (R2 + R3 + R4) * phi_left
print('phi_center = {:.3f} mWb'.format(phi_center*1000))


I am working on the file "Chapman/Ch1-Problem_1-08.ipynb".
The first task is:
The reluctances of the regions in core are: $\mathcal{R}_\text{CORE} = \frac{l}{\mu_0 \mu_r A}$,
The reluctances of the regions in the air gaps are: $\mathcal{R}_\text{GAP} = \frac{l}{\mu_0 A }$.

The areas can be calculated as:
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-08.ipynb".
The first task is:
The total flux in the core $\phi_\text{TOT}$ is equal to the flux in the left leg $\phi_\text{left} = \frac{\mathcal{F}}{\mathcal{R}_\text{TOT}}$ , which is:
Can you write Python code for it?

phi_left = F / Rtot
print('phi_left = {:.3f} mWb'.format(phi_left*1000))


I am working on the file "Chapman/Ch1-Problem_1-08.ipynb".
The first task is:
Then the total reluctance of the core is $\mathcal{R}_\text{TOT} = \mathcal{R}_1 + \frac{(\mathcal{R}_2 + \mathcal{R}_3)\mathcal{R}_4}{\mathcal{R}_2 + \mathcal{R}_3 + \mathcal{R}_4}$.
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-12.ipynb".
The first task is:
Description
The core shown in Figure P1-4 below:

and is made of a steel whose magnetization curve is shown in Figure P1-9 below:

Repeat Problem 1-7, but this time do not assume a constant value of $\mu_r$.

How much flux is produced in the core by the currents specified?
What is the relative permeability of this core under these conditions?
Was the assumption in Problem 1-7 that the relative permeability was equal to 1200 a good assumption for these conditions?
Is it a good assumption in general?
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-12.ipynb".
The first task is:
From the magnetization curve:
<img src="figs/FigC_P1-9sol.png" width="60%">
Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-12.ipynb".
The first task is:
The relative permeability of the core can be found from the reluctance as follows:
$$\mathcal{R}\text{TOT} = \frac{\mathcal{F}\text{TOT}}{\mathcal{\phi}_\text{TOT}} = \frac{l_c}{\mu_0 \mu_r A}$$
Solving for $\mu_r$ yields:
$$\mu_r = \frac{\mathcal{\phi}\text{TOT}l_c}{\mathcal{F}\text{TOT}\mu_0 A}$$
Can you write Python code for it?

mu_r = phi_tot * lc / (F_tot * c.mu_0 * A)
print('μ = {:.1f}'.format(mu_r))


I am working on the file "Chapman/Ch1-Problem_1-12.ipynb".
The first task is:
Therefore, the magnetizing intensity $H$ is:
$$ H = \frac{\mathcal{F_\text{TOT}}}{l_c}$$
Can you write Python code for it?

lc = 4 * (0.075 + 0.5 + 0.075)   # [m] core length on all 4 sides.
H = F_tot / lc
print('H = {:.1f} At/m'.format(H))


I am working on the file "Chapman/Ch1-Problem_1-12.ipynb".
The first task is:
and the total flux in the core is:
$$\phi_\text{TOT} = BA$$
Can you write Python code for it?

A = 0.15**2  # [m²]
phi_tot = B*A
print('ϕ = {:.3f} mWb'.format(phi_tot*1000))


I am working on the file "Chapman/Ch1-Problem_1-16.ipynb".
The first task is:
Description
The core shown in Figure P1-2:

has the flux $\phi$ shown in Figure P1-12:

Can you write Python code for it?


I am working on the file "Chapman/Ch1-Problem_1-16.ipynb".
The first task is:
Lets pretty-print the result:
Can you write Python code for it?

t = 0  # time [s], whose initial value is set here
# print the table head
print('''
|--------------+----------|
|     Time     |  e_ind   |
|--------------+----------|''')
# We use a simple for-loop to print a row per result:
for i in range(4):
    print('| {:.0f} < t < {:.0f} ms | {:>5.2f} kV |'.format(
              *( t*1000,       # start of time inteval [ms]
                (t+dt[i])*1000, # end time of the inteval [ms]
                e_ind[i]/1000 # value of e_ind [V]
                )))
    t = t + dt[i]
print('|=========================|')