I am working on the file "04-VotingProfile/PersonAnalysis/SplitDeputy.ipynb".
The first task is:
Summary of the different votes possible
|1 | 2 | 3      | 4    | 5               | 6 |                7                       |
|------|-------|------------|----------|---------------------|-------|--------------------------------------------|
| Yes  | No    | Abstention | No entry | Did not participate |Excused| The president of the session does not vote |
Counting the percentage of abstention by person, in order to see whether there is a person that would abstain significantly more often than the rest. It does not appear to be the case.
Can you write Python code for it?

#count_abst = lambda x:
voting_test = voting_df[['Name','Decision']].groupby('Name').apply(lambda x: np.sum(x.Decision==3)/len(x.Decision))
voting_test.sort_values(ascending=False).head()


I am working on the file "04-VotingProfile/PersonAnalysis/Split_Vote_Voting.ipynb".
The first task is:
2. Map name of the law to an ID
Allows to load a light file, which will serve as link between the initial visualisation and the law specific.
Aside: Removing the not relevant laws from the Vote file
We see here a disparity, as in our Voting file, the smallest IdVote is 6392, which removes a third of the laws we actually have. This is due to the scraping not retrieving everything, but this is not a bad thing, as it removes the oldest entries, which are not the most relevant ones.
Can you write Python code for it?

voting_df.IdVote.sort_values().unique()

vote_df = load_vote_df()
vote_df.head()


I am working on the file "04-VotingProfile/PersonAnalysis/Split_Vote_Voting.ipynb".
The first task is:
Indeed, we see here that here, the ID, which corresponds to the IdVote on the Voting file, starts from 1, but the subject is about the oldest votes. We will now generate an "epurated" version of the Vote dataframe, with only the votes which have a counterpart in the Voting file.
Moreover, we know that there are some empty BillTitle entries, so we want to put the BusinessTitle as BillTitle to be able to handle the subject generally.
Can you write Python code for it?

vote_df = vote_df.loc[vote_df.ID>=6392]

# Setting the entries with a null BillTitle to have BusinessTitle as their entry
vote_df.loc[vote_df.BillTitle.isnull(),'BillTitle'] =  vote_df.loc[vote_df.BillTitle.isnull(),'BusinessTitle']

directory = '../../datas/treated_data/Vote/'
if not os.path.exists(directory):
    os.makedirs(directory)

vote_df.to_csv(directory+'legiid_47-50.csv')


I am working on the file "04-VotingProfile/PersonAnalysis/Split_Vote_Voting.ipynb".
The first task is:
We must first have a table in which we can link the BillTitle to the corresponding file, which will contain all the informations regarding a certain BillTitle.
Can you write Python code for it?

bills_dict = split_df_dict(vote_df, 'ID_Bill')

directory = '../../datas/analysis/bill_link/'
if not os.path.exists(directory):
    os.makedirs(directory)

for ID_Bill, df in bills_dict.items():
    df.to_csv(directory+'bill_'+str(ID_Bill)+'.csv')


I am working on the file "04-VotingProfile/TopicAnalysis.ipynb".
The first task is:
Evolution of topics in the votes
We take a look at the data in order to observe which topics are most talked about in the votes, and see whether or not it changes over the years.
Can you write Python code for it?

voting_df_copy = voting_df
voting_df_copy.head(3)

voting_df_copy['VoteEnd'] = [x[:4] for x in voting_df_copy['VoteEnd']]
voting_df_copy.head(3)
dates = voting_df_copy['VoteEnd'].drop_duplicates()
dates = np.sort(dates)
dates

#voting_df_copy.index = voting_df_copy['VoteEnd']
voting_df_copy = voting_df_copy.set_index(['VoteEnd', 'Name'])

voting_df_copy.head(3)

voting_df_copy.columns

#voting_df_copy.loc['2009-09'][' armée']
#topicArmee = np.mean(voting_df_copy.loc['2009-09'][' armée'])

#print(topicArmee)
#indices = voting_df_copy.index[:].values
#indices

topics = ['armée', 'asile / immigration', 'assurances', 'budget', 'dunno', 'entreprise/ finance', 'environnement', 'famille / enfants', 'imposition', 'politique internationale', 'retraite  ']
topicEvolution = pd.DataFrame(index = dates, columns = topics)
#topicArmeeEvolution = []

for topic in topics:
    for date in dates:
        topicEvolution.loc[date][topic] = np.mean(voting_df_copy.loc[date][' '+topic])
        #topicArmeeEvolution.append(np.mean(voting_df_copy.loc[date][' '+topic]))
    #topicEvolution[topic].plot
    plt.plot(dates,topicEvolution[topic])

plt.legend(topics)
topicEvolution

topicEvolution.to_json("topicEvolution.json")


I am working on the file "04-VotingProfile/TopicAnalysis.ipynb".
The first task is:
Extract vote decision for each subject
We exctract here for each voted subject the final result of the votation, by comparing the number of Yes to the number of No.
Can you write Python code for it?

voting_df = pd.read_csv(path+'voting_with_topics_sentiment.csv')
voting_df_Decision = voting_df
#voting_df_TopicAcceptation = voting_df_TopicAcceptation.set_index(['IdVote', 'Name'])
voting_df_Decision.head(3)

texts = voting_df_Decision['text'].unique()
print(len(texts))
texts[0]

voting_df_Decision = voting_df_Decision.set_index(['text', 'LastName'])
voting_df_Decision.head()

decisions_dict = {}
decisions_dict[texts[0]] = np.sum(voting_df_Decision.loc[texts[0]].Decision == 1) > np.sum(voting_df_Decision.loc[texts[0]].Decision == 2)
for t in texts:
    decisions_dict[t] = np.sum(voting_df_Decision.loc[t].Decision == 1) > np.sum(voting_df_Decision.loc[t].Decision == 2)

len(decisions_dict)

decisions_df = pd.DataFrame.from_dict(decisions_dict, 'index')
decisions_df.columns = ['Decision']
decisions_df.head()


I am working on the file "04-VotingProfile/TopicAnalysis.ipynb".
The first task is:
Ratio of acceptation for each topic
We now want to see if there is a significant correlation between a vote being accepted and the topic of the vote.
Can you write Python code for it?

path = '../datas/nlp_results/'
voting_df = pd.read_csv(path+'voting_with_topics_unique_sentiment.csv')
voting_df_TopicAcceptation = voting_df
#voting_df_TopicAcceptation = voting_df_TopicAcceptation.set_index(['IdVote', 'Name'])
voting_df_TopicAcceptation.head(3)

voting_df_TopicAcceptation['Accepted'] = decisions_df.loc[voting_df_TopicAcceptation['text']].Decision.values
voting_df_TopicAcceptation.head(3)

topics = [' armée', ' asile / immigration', ' assurances', ' budget', ' dunno', ' entreprise/ finance', ' environnement', ' famille / enfants', ' imposition', ' politique internationale', ' retraite  ']

voting_df_TopicAcceptation['Topic'] = voting_df_TopicAcceptation[topics].idxmax(axis=1)

voting_df_TopicAcceptation.head(3)

decisions_df2 =voting_df[['text','Accepted','Topic']]
decisions_df2 = decisions_df2.set_index('text')
decisions_df2.to_csv('topic_accepted.csv')

meanAcceptation = np.mean(voting_df_TopicAcceptation.loc[:, 'Accepted'])
print('Precentage of acceptation of a votation : ', meanAcceptation)

voting_df_TopicAcceptation = voting_df_TopicAcceptation[['Topic', 'Accepted']]
voting_df_MeanAcceptation = voting_df_TopicAcceptation.groupby(by='Topic', axis=0).mean()
voting_df_MeanAcceptation


I am working on the file "04-VotingProfile/TopicAnalysis.ipynb".
The first task is:
We observe qualitatively that votations about army, assurances and environnement are less accepted than the votations about other topics in general. Conversly, votations about asile / immigration, enterprise / finance and imposition are more likely to be accepted. This analysis is performed based on a quick look to the data. The aim of the next part is to determine whether or not these observed variatiations are statistically significant.
Can you write Python code for it?

#voting_df_MeanAcceptation['Accepted'].plot.hist(by = 'Topic')
#voting_df_MeanAcceptation.hist()
#plt.show()

voting_df_TopicAcceptation = voting_df_TopicAcceptation.set_index('Topic')
voting_df_TopicAcceptation.head(3)


I am working on the file "04-basic-plotting.ipynb".
The first task is:
The command %matplotlib inline is not a Python command, but an IPython command. When using the console, or the notebook, it makes the plots appear inline. You do not want to use this in a plain Python code.
Can you write Python code for it?

from math import sin, pi

x = []
y = []
for i in range(201):
    x.append(0.01*i)
    y.append(sin(pi*x[-1])**2)

pyplot.plot(x, y)
pyplot.show()


I am working on the file "04-basic-plotting.ipynb".
The first task is:
We have defined two sequences - in this case lists, but tuples would also work. One contains the $x$-axis coordinates, the other the data points to appear on the $y$-axis. A basic plot is produced using the plot command of pyplot. However, this plot will not automatically appear on the screen, as after plotting the data you may wish to add additional information. Nothing will actually happen until you either save the figure to a file (using pyplot.savefig(<filename>)) or explicitly ask for it to be displayed (with the show command). When the plot is displayed the program will typically pause until you dismiss the plot.
If using the notebook you can include the command %matplotlib inline or %matplotlib notebook before plotting to make the plots appear automatically inside the notebook. If code is included in a program which is run inside spyder through an IPython console, the figures may appear in the console automatically. Either way, it is good practice to always include the show command to explicitly display the plot.
This plotting interface is straightforward, but the results are not particularly nice. The following commands illustrate some of the ways of improving the plot:
Can you write Python code for it?

from math import sin, pi

x = []
y = []
for i in range(201):
    x_point = 0.01*i
    x.append(x_point)
    y.append(sin(pi*x_point)**2)

pyplot.plot(x, y, marker='+', markersize=8, linestyle=':',
            linewidth=3, color='b', label=r'$\sin^2(\pi x)$')
pyplot.legend(loc='lower right')
pyplot.xlabel(r'$x$')
pyplot.ylabel(r'$y$')
pyplot.title('A basic plot')
pyplot.show()


I am working on the file "04-basic-plotting.ipynb".
The first task is:
We have defined two sequences - in this case lists, but tuples would also work. One contains the $x$-axis coordinates, the other the data points to appear on the $y$-axis. A basic plot is produced using the plot command of pyplot. However, this plot will not automatically appear on the screen, as after plotting the data you may wish to add additional information. Nothing will actually happen until you either save the figure to a file (using pyplot.savefig(<filename>)) or explicitly ask for it to be displayed (with the show command). When the plot is displayed the program will typically pause until you dismiss the plot.
This plotting interface is straightforward, but the results are not particularly nice. The following commands illustrate some of the ways of improving the plot:
Can you write Python code for it?

from math import sin, pi

x = []
y = []
for i in range(201):
    x.append(0.01*i)
    y.append(sin(pi*x[-1])**2)

pyplot.plot(x, y, marker='+', markersize=8, linestyle=':',
            linewidth=3, color='b', label=r'$\sin^2(\pi x)$')
pyplot.legend(loc='lower right')
pyplot.xlabel(r'$x$')
pyplot.ylabel(r'$y$')
pyplot.title('A basic plot')
pyplot.show()


I am working on the file "04-basic-plotting.ipynb".
The first task is:
Whilst most of the commands are self-explanatory, a note should be made of the strings line r'$x$'. These strings are in LaTeX format, which is the standard typesetting method for professional-level mathematics. The $ symbols surround mathematics. The r before the definition of the string is Python notation, not LaTeX. It says that the following string will be "raw": that backslash characters should be left alone. Then, special LaTeX commands have a backslash in front of them: here we use \pi and \sin. Most basic symbols can be easily guess (eg \theta or \int), but there are useful lists of symbols, and a reverse search site available. We can also use ^ to denote superscripts (used here), _ to denote subscripts, and use {} to group terms.
By combining these basic commands with other plotting types (semilogx and loglog, for example), most simple plots can be produced quickly.
Here are some more examples:
Can you write Python code for it?

from math import sin, pi, exp, log

x = []
y1 = []
y2 = []
for i in range(201):
    x.append(1.0+0.01*i)
    y1.append(exp(sin(pi*x[-1])))
    y2.append(log(pi+x[-1]*sin(x[-1])))

pyplot.loglog(x, y1, linestyle='--', linewidth=4,
              color='k', label=r'$y_1=e^{\sin(\pi x)}$')
pyplot.loglog(x, y2, linestyle='-.', linewidth=4,
              color='r', label=r'$y_2=\log(\pi+x\sin(x))$')
pyplot.legend(loc='lower right')
pyplot.xlabel(r'$x$')
pyplot.ylabel(r'$y$')
pyplot.title('A basic logarithmic plot')
pyplot.show()

from math import sin, pi, exp, log

x = []
y1 = []
y2 = []
for i in range(201):
    x.append(1.0+0.01*i)
    y1.append(exp(sin(pi*x[-1])))
    y2.append(log(pi+x[-1]*sin(x[-1])))

pyplot.semilogy(x, y1, linestyle='None', marker='o',
              color='g', label=r'$y_1=e^{\sin(\pi x)}$')
pyplot.semilogy(x, y2, linestyle='None', marker='^',
              color='r', label=r'$y_2=\log(\pi+x\sin(x))$')
pyplot.legend(loc='lower right')
pyplot.xlabel(r'$x$')
pyplot.ylabel(r'$y$')
pyplot.title('A different logarithmic plot')
pyplot.show()


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
As you can see, the In column is always the same and is the phone number of User Number One.
There are in total 3648 calls recorded.
Plot all the call locations
We use the Tower's Latitude and Longitude to see where the call have been received.
The X-Coordinate should be the Longitude, and the Y-coordinate should be the tower Latitude.
This is a mobile phone, so it can happen in different locations.
Can you write Python code for it?

user1.plot.scatter(x='TowerLon', y='TowerLat', c='gray', alpha=0.1, title='Call Locations');


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
Before unitizing / standardizing / normalizing the data in preparation for K-Means, it's a good idea to get a quick peek at it. You can do this using the .describe() method, or even by using the built-in pandas .plot.hist():
Can you write Python code for it?


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
Clustering
We run the K-Means algorithm with a K=1. There really should only be one single area of concentration.
If you notice multiple areas that are "hot" (multiple areas the user spends a lot of time at that are FAR apart from one another), then increase K=2, with the goal being that one of the centroids will sweep up the annoying outliers; and the other will zero in on the user's approximate home location.
Or rather the location of the cell tower closest to their home.....
We nloy feed in Lat and Lon coordinates to the KMeans algorithm, since none of the other data is suitable for your purposes. Since both Lat and Lon are (approximately) on the same scale, no feature scaling is required. finally we print out the centroid locations and add them onto the previous scatter plot using a distinguishable marker and color.
Can you write Python code for it?

sliceK = user1[['TowerLon', 'TowerLat']]

model = KMeans(n_clusters=1, n_init=10, init='random')
model.fit(sliceK)

centroids = model.cluster_centers_

centroids

  #
  # Print and plot the centroids...

fig = plt.figure()
ax = fig.add_subplot(111)
  # calls location
ax.scatter(user1.TowerLon,user1.TowerLat, c='g', marker='o', alpha=0.2)
  # centroid
ax.scatter(centroids[:,0], centroids[:,1], marker='x', c='red', alpha=0.5,
             linewidths=3, s=169);

ax.set_title('Weekend Calls with Centroid');


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
Data normalisation
What are you interested in?
Depending on what you're interested in, you might take a different approach to normalizing/standardizing the data.
You should note that all columns left in the dataset are of the same unit.
You might ask yourself, do I even need to normalize / standardize the data?
The answer depends on what you're trying to accomplish. For instance, although all the units are the same (generic money unit), the price per item in your store isn't. There may be some cheap items and some expensive one. If your goal is to find out what items people buy tend to buy together but you didn't unitize properly before running kMeans, the contribution of the lesser priced item would be dwarfed by the more expensive item.
Suffice to say, at the end of the day, you're going to have to know what question you want answered and what data you have available in order to select the best method for your purpose. Luckily, SKLearn's interfaces are easy to switch out so in the mean time, you can experiment with all of them and see how they alter your results.
5-sec summary:

NORMALIZATION: Let's say your user spend a LOT. Normalization divides each item by the average overall amount of spending. Stated differently, your new feature is = the contribution of overall spending going into that particular item: money spent on feature / money overall spent by sample.

MINMAX: What % in the overall range of money spent by all users on THIS particular feature is the current sample's feature at? When you're dealing with all the same units, this will produce a near face-value amount. Be careful though: if you have even a single outlier, it can cause all your data to get squashed up in lower percentages.
Imagine your buyers usually spend 100 dollars on wholesale milk, but today only spent 20 dollars. This is the relationship you're trying to capture with MinMax.
NOTE: MinMax doesn't standardize (std. dev.); it only normalizes / unitizes your feature, in the mathematical sense.
MinMax can be used as an...
Can you write Python code for it?

T = wsc # No Change, no normalisation
#
# : Un-comment just ***ONE*** of lines at a time and see how alters the results
# Pay attention to the direction of the arrows, as well as their LENGTHS
#T = preprocessing.StandardScaler().fit_transform(wsc)
#T = preprocessing.MinMaxScaler().fit_transform(wsc)
#T = preprocessing.MaxAbsScaler().fit_transform(wsc)
#T = preprocessing.Normalizer().fit_transform(wsc)


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
Do PCA after to visualize the results.
Project the centroids as well as the samples into the new 2D feature space for visualization purposes.
Can you write Python code for it?

display_pca = doPCA(T)
T = display_pca.transform(T)
CC = display_pca.transform(centroids)

import matplotlib.pyplot as plt

#
# Parameters to play around with
PLOT_TYPE_TEXT = False    # If you'd like to see indices
PLOT_VECTORS = True       # If you'd like to see your original features in P.C.-Space

c = ['red', 'green', 'blue', 'orange', 'yellow', 'brown'] # up to six centroids

# Visualize all the samples. Give them the color of their cluster label
fig = plt.figure()
ax = fig.add_subplot(111)

if PLOT_TYPE_TEXT:
  # Plot the index of the sample, so you can further investigate it in your dset
  for i in range(len(T)): ax.text(T[i,0], T[i,1], wsc.index[i], color=c[labels[i]], alpha=0.75, zorder=600000)
  ax.set_xlim(min(T[:,0])*1.2, max(T[:,0])*1.2)
  ax.set_ylim(min(T[:,1])*1.2, max(T[:,1])*1.2)
else:
  # Plot a regular scatter plot
  sample_colors = [ c[labels[i]] for i in range(len(T)) ]
  ax.scatter(T[:, 0], T[:, 1], c=sample_colors, marker='o', alpha=0.2)

# Plot the Centroids as X's, and label them
ax.scatter(CC[:, 0], CC[:, 1], marker='x', s=169, linewidths=3, zorder=1000, c = c[:3])

for i in range(len(centroids)): ax.text(CC[i, 0], CC[i, 1], str(i), zorder=500010, fontsize=18, color=c[i])

# Display feature vectors for investigation:
if PLOT_VECTORS:
    #drawVectors(T, display_pca.components_, wsc.columns, plt)

#def drawVectors(transformed_features, components_, columns, plt):
    num_columns = len(wsc.columns)

  # This function will project your *original* feature (columns)
  # onto your principal component feature-space, so that you can
  # visualize how "important" each one was in the
  # multi-dimensional scaling

  # Scale the principal components by the max value in
  # the transformed set belonging to that component
    xvector = display_pca.components_[0] * max(T[:,0])
    yvector = display_pca.components_[1] * max(T[:,1])

  ## Visualize projections

  # Sort each column by its length. These are your *original*
  # columns, not the principal components.
    import math
   ...


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
It may or may not have invalid values (NaN) in it.
We make sure to catch and remove them, by setting them to '0'.
This is valid for this dataset, since if the value is missing, you can assume no $ was spent on it.
Can you write Python code for it?

if wsc.isnull().values.any() == True:
  print("substituted all NaN with Zero")
  wsc.fillna(0, inplace=True)
else:
  print("No NaN found!")

wsc.head()


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
K-Means
And we use K-Means to try and find seven (an arbitrary number) cluster centers in this dataset:
Can you write Python code for it?

model = KMeans(n_clusters=7, n_init=10, init='random')
model.fit(sliceK)


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
Print out the centroids.
They're currently in feature-space, which is good. We print them out before we transform them into PCA space for viewing.
The K-means algorithm aims to choose centroids that minimise the inertia, or within-cluster sum-of-squares criterion.
Inertia is the sum of squared distances of samples to their closest cluster center.
Can you write Python code for it?

def clusterInfo(model):
  print ("Cluster Analysis Inertia: ", model.inertia_)
  print ('------------------------------------------')
  for i in range(len(model.cluster_centers_)):
    print ("\n  Cluster ", i)
    print ("    Centroid ", centroids[i])
    print ("    #Samples ", (labels==i).sum())

clusterInfo(model)

def doPCA(data, dimensions=2):
  from sklearn.decomposition import PCA
  model = PCA(n_components=dimensions, svd_solver='randomized', random_state=7)
  model.fit(data)
  return model


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
Sometimes people perform PCA before doing KMeans, so that KMeans only operates on the most meaningful features.
In our case, there are so few features that doing PCA ahead of time isn't really necessary and you can do KMeans in the feature space.
But keep in mind you have the option to transform your data to bring down its dimensionality. If you take that route, then your Clusters will already be in PCA-transformed feature space and you won't have to project them again for visualization.
Apply K-means
Can you write Python code for it?

from sklearn.cluster import KMeans

model = KMeans(n_clusters= 3, n_init=10, init='random')
model.fit(T)

labels = model.labels_
centroids = model.cluster_centers_


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
The Date feature is a string object so we coerce it into real date and confirm by re-printing the dtypes.
NOTE: This is a slow process...
Can you write Python code for it?

cc.Date = pd.to_datetime(cc.Date, errors='coerce')

cc.dtypes


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
The locations map above is too "busy" to really wrap head around. This is where domain expertise comes into play. The intuition tells us that people are likely to behave differently on weekends:
On Weekends:
   1. People probably don't go into work
   2. They probably sleep in late on Saturday
   3. They probably run a bunch of random errands, since they couldn't during the week
   4. They should be home, at least during the very late hours, e.g. 1-4 AM
On Weekdays:
   1. People probably are at work during normal working hours
   2. They probably are at home in the early morning and during the late night
   3. They probably spend time commuting between work and home everyday
Let's examine only the records that came in on weekends (sat/sun).
Can you write Python code for it?

#
# Add more filters to the user1 slice we created. Add bitwise logic for the weekend
user1 = user1[(user1.DOW == "Sat") | (user1.DOW == "Sun")]


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
They have been greatly reduced.
Now we visualise the dataframe with a scatter plot as a sanity check.
At this point, we still don't yet know exactly where the user is located just based off the cell phone tower position data; but considering the below are for calls that arrived in the twilight hours of weekends, it's likely that wherever they are bunched up is probably near where is the caller's residence:
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(user1.TowerLon,user1.TowerLat, c='g', marker='o', alpha=0.2)
ax.set_title('Weekend Calls (<6am or >10p)');


I am working on the file "04-clustering/clustering.ipynb".
The first task is:
We get rid of the 'Channel' and 'Region' columns, since we'll be investigating as if this were a single location wholesaler, rather than a national / international one.
Leaving these fields in here would cause KMeans to examine and give weight to them.
The other features are the product categories.
Can you write Python code for it?

wsc.drop(['Channel', 'Region'], axis=1, inplace = True)

print('Wholesale customers has {} samples with {} features each'.format(*wsc.shape))

wsc.describe()


I am working on the file "04-concurrency/mutial_exclusion.ipynb".
The first task is:
Notice that both tasks are CPU-bound. This means that using threading has not any wall time advantage compared to an iterative implementation of both taks.
Using processes
Can you write Python code for it?

# Two processes that have a critical section executed sequentially

import multiprocessing
import time
import ctypes

def task_1(lock, counter):
    for i in range(10000):
        with lock:
            counter.value += 1

def task_2(lock, counter):
    for i in range(10001):
        with lock:
            counter.value -= 1

lock = multiprocessing.Lock()

manager = multiprocessing.Manager()
counter = manager.Value(ctypes.c_int, 10)

process_1 = multiprocessing.Process(target=task_1, args=(lock, counter))
process_2 = multiprocessing.Process(target=task_2, args=(lock, counter))

now = time.perf_counter()
process_1.start()
process_2.start()
print("Both tasks started")
process_1.join()
process_2.join()
print("Both tasks finished")
elapsed = time.perf_counter() - now
print(f"elapsed {elapsed:0.2f} seconds")
print('counter =', counter.value)


I am working on the file "04-concurrency/mutial_exclusion.ipynb".
The first task is:
The same example, using mutual exclusion (using a lock):
Can you write Python code for it?

# Two threads that have a critical section executed sequentially.

import threading
import time

lock = threading.Lock()
counter = 10

def task_1():
    global counter
    for i in range(10**6):
        with lock:
            counter += 1

def task_2():
    global counter
    for i in range(10**6+1):
        with lock:
            counter -= 1

thread_1 = threading.Thread(target=task_1)
thread_2 = threading.Thread(target=task_2)

now = time.perf_counter()  # Real time (not only user time)
thread_1.start()
thread_2.start()
print("Both threads started")
thread_1.join()
thread_2.join()
print("Both threads finished")
elapsed = time.perf_counter() - now
print(f"elapsed {elapsed:0.2f} seconds")
print('counter =', counter)


I am working on the file "04-concurrency/mutial_exclusion.ipynb".
The first task is:
Unlike threading, multiprocessing is suitable for reducing the running times in the case of CPU-bound problems.
Using coroutines
Like threads, coroutines should be only used when the coroutines must wait (typically for an I/O transaction). Otherwhise, use multiprocessing.
Can you write Python code for it?

import asyncio

counter = 10

async def task_1():
    global counter
    for i in range(10):
        print("o", end='', flush=True)
        counter += 1
        await task_2()

async def task_2():
    global counter
    print("O", end='', flush=True)
    counter -= 1

await task_1()

print('\ncounter =', counter)

import asyncio
import time

counter = 10

async def task_1():
    global counter
    for i in range(10**6):
        counter += 1
        await task_2()

async def task_2():
    global counter
    counter -= 1

now = time.perf_counter()
await task_1()
elapsed = time.perf_counter() - now
print(f"\nelapsed {elapsed:0.2f} seconds")
print('counter =', counter)


I am working on the file "04-concurrency/sharing_a_variable.ipynb".
The first task is:
3. "Low-level" coroutines
Notice that each iteration of the coroutines must be "fired" explicitally.
Can you write Python code for it?

import time

a = ''

def task_1():
    global a
    for i in range(20):
        print('o', end='', flush=True)
        a += 'o'
        yield
        time.sleep(0.5)

def task_2():
    global a
    for i in range(20):
        print('O', end='', flush=True)
        a += 'O'
        yield
        time.sleep(0.25)

t1 = task_1()
t2 = task_2()

now = time.perf_counter()  # Real time (not only user time)
for i in range(20):
    t1.__next__()  # Call task_1()
    t2.__next__()  # Call task_2()
elapsed = time.perf_counter() - now
print(f"\nelapsed {elapsed:0.2f} seconds.")
print(20*0.5 + 20*0.25)
print("a =", a)


I am working on the file "04-concurrency/sharing_a_variable.ipynb".
The first task is:
3. Using 'asyncio'
In this case, the coroutines yields their control to the "event loop" (also called, the "coordinator") with a await command, which search for the next ready coroutine to be run in a round-robin list. Notice that for this reason, the wall time of this version is smaller than in the previous case.
Can you write Python code for it?

import asyncio
import time

a = ''

async def task_1():
    global a
    for i in range(20):
        print('o', end='', flush=True)
        a += 'o'
        await asyncio.sleep(0.5)

async def task_2():
    global a
    for i in range(20):
        print('O', end='', flush=True)
        a += 'O'
        await asyncio.sleep(0.25)

now = time.perf_counter()
await asyncio.gather(task_1(), task_2())
elapsed = time.perf_counter() - now
print(f"\nelapsed {elapsed:0.2f} seconds.")
print("a =", a)


I am working on the file "04-concurrency/sharing_a_variable.ipynb".
The first task is:
But ... why a has not been modified? Why the processed do not share a?
By definition, processes must fork (make a copy of itself, that is, the code and the used memory) before start running. In the previous example, the Python interpreter forks twice and the two childs are run in parallel while the parent process waits for their completition. Neither, the child processes nor the parent process share their global state (where a is defined).
How to share data between processes?
There are several options. One of them is to use a shared memory Manager():
Can you write Python code for it?

import multiprocessing
import time
import ctypes

def task_1(a):
    for i in range(10):
        print('o', end='', flush=True)
        a.value += 'o'
        time.sleep(1) # Blocking -> yield to other process

def task_2(a):
    for i in range(20):
        print('O', end='', flush=True)
        a.value += 'O'
        time.sleep(0.6) # Blocking -> yield to other process

manager = multiprocessing.Manager()
# See https://docs.python.org/3/library/ctypes.html#module-ctypes
a = manager.Value(ctypes.c_char_p, "")

process_1 = multiprocessing.Process(target=task_1, args=(a,))
process_2 = multiprocessing.Process(target=task_2, args=(a,))

process_1.start()
process_2.start()
print("(Both processes have started)")

process_1.join()
process_2.join()
print("\nBoth processes have finished")

print(a.value)


I am working on the file "04-concurrency/sharing_a_variable.ipynb".
The first task is:
Notice that both threads share the same process memory space.
2. Processes
Can you write Python code for it?

# This code does not work!

import multiprocessing
import time

a = '' # Does not work because each process has its own 'a'

def task_1():
    global a
    for i in range(10):
        print('o', end='', flush=True)
        a += 'o'
        print(a)
        time.sleep(1) # Blocking -> yield to other process

def task_2():
    global a
    for i in range(20):
        print('O', end='', flush=True)
        a += 'O'
        print(a)
        time.sleep(0.6) # Blocking -> yield to other process

process_1 = multiprocessing.Process(target=task_1)
process_2 = multiprocessing.Process(target=task_2)

process_1.start()
process_2.start()
print("(Both processes have started)")

process_1.join()
process_2.join()
print("\nBoth processes have finished")

print(a)


I am working on the file "04-depreciacion.ipynb".
The first task is:
Comparación
Can you write Python code for it?

pd.DataFrame({'impuesto_1':impuesto_1, 'impuesto_2':impuesto_2, 'impuesto_3':impuesto_3}).round(2)

pd.DataFrame({'cf_1':cf_1, 'cf_2':cf_2, 'cf_3':cf_3}).round(2)


I am working on the file "04-depreciacion.ipynb".
The first task is:
Depreciación en línea recta
La depreciación permite la deducción de los costos de las inversiones (pérdida de valor de ellas) con el fin de reducir el impuesto de renta.
La depreciación en todos los años es igual.

$P$ -- valor de compra en libros.

$N$ -- número de años en que ocurre la depreciación (vida útil del activo).

$D_n$ -- depreciación en el año $n$.

$F$ -- valor de salvamento al final de la vida útil.

$$D_n = \frac{P-F}{N}$$
Ejemplo.-- Un activo cuesta  $ 200. Si la depreciación ocurre durante 5 años, ¿cuál es su depreciación anual?.
Can you write Python code for it?

costs = cf.cashflow(const_value=0,  # valor 0 por defecto
                    periods=6,      # compra + vida útil
                    start=2000,
                    freq='A')
costs[0] = 200
costs

life  = cf.cashflow(const_value=0, # vida util = 0
                    periods=6,     # compra + vida útil
                    start=2000,
                    freq='A')
life[0]=5 # vida útil del activo
life

cf.depreciation_sl(costs=costs,    # costos de las inversiones
                   life=life)      # vida útil de cada inversión


I am working on the file "04-depreciacion.ipynb".
The first task is:
Depreciación por suma de los dígitos de los años (o depreciación acelerada)
En este método de depreciación se hace mayor al principio y más pequeña al final. Se basa en la progresión desde $N$ hasta 1.
$$S = \sum_{i=1}^N i = \frac{1}{2} N(N + 1)$$
$$D_n = (P - F) * \frac{(N - n + 1)}{S} = 2 * (P - F) * \frac{N-n+1}{N(N+1)}$$
Ejemplo.-- Un activo cuesta  $ 200. Si la depreciación ocurre durante 5 años, ¿cuál es su depreciación anual?.
Can you write Python code for it?

costs = cf.cashflow(const_value=[200]+[0]*5,
                    start=2000,
                    freq='A')

life  = cf.cashflow(const_value=[5]+[0]*5,
                    start='2000',
                    freq='A')

cf.depreciation_soyd(costs=costs, life=life)


I am working on the file "04-depreciacion.ipynb".
The first task is:
Note que en el resultado anterior el ingreso después de impuestos es mayor que en el caso 1.
Caso 3: depreciación acelerada.
Can you write Python code for it?

depreciacion_3 = cf.depreciation_soyd(costs=costs, life=life)['Depr']
depreciacion_3

impuesto_3, cf_3 = compute_cf(ingreso_operativo, tax_rate, depreciacion_3)
impuesto_3

cf_3


I am working on the file "04-e2e-pipeline.ipynb".
The first task is:
2. Set Pipeline Configurations
Can you write Python code for it?

os.environ['PROJECT_ID'] = PROJECT_ID
os.environ['REGION'] = REGION
os.environ['BUCKET'] = BUCKET
os.environ['WORKSPACE'] = WORKSPACE

os.environ['TRAINING_PIPELINE_NAME'] = TRAINING_PIPELINE_NAME
os.environ['MODEL_NAME'] = MODEL_NAME
os.environ['MODEL_VERSION'] = MODEL_VERSION
os.environ['MODEL_DISPLAY_NAME'] = MODEL_DISPLAY_NAME

os.environ['MEMORY_LIMIT'] = '680'
os.environ['CPU_LIMIT'] = '96'
os.environ['GPU_LIMIT'] = '8'
os.environ['GPU_TYPE'] = 'NVIDIA_TESLA_A100'

os.environ['MACHINE_TYPE'] = 'a2-highgpu-1g'
os.environ['ACCELERATOR_TYPE'] = 'NVIDIA_TESLA_A100'
os.environ['ACCELERATOR_NUM'] = '1'
os.environ['NUM_WORKERS'] = '12'

os.environ['NUM_SLOTS'] = '26'
os.environ['MAX_NNZ'] = '2'
os.environ['EMBEDDING_VECTOR_SIZE'] = '11'
os.environ['MAX_BATCH_SIZE'] = '64'
os.environ['MODEL_REPOSITORY_PATH'] = '/model'

os.environ['NVT_IMAGE_URI'] = NVT_IMAGE_URI
os.environ['HUGECTR_IMAGE_URI'] = HUGECTR_IMAGE_URI
os.environ['TRITON_IMAGE_URI'] = TRITON_IMAGE_URI


I am working on the file "04-e2e-pipeline.ipynb".
The first task is:
3. Build Pipeline Container Images
The following three commands build the NVTabular preprocessing, HugeCTR training, and Triton serving container images using Cloud Build, and store the container images in Container Registry.
Build NVTabular preprocessing container image
Can you write Python code for it?

FILE_LOCATION = './src'
! gcloud builds submit --config src/cloudbuild.yaml --substitutions _DOCKERNAME=$NVT_DOCKERNAME,_IMAGE_URI=$NVT_IMAGE_URI,_FILE_LOCATION=$FILE_LOCATION --timeout=2h --machine-type=e2-highcpu-8


I am working on the file "04-e2e-pipeline.ipynb".
The first task is:
4. Configure pipeline parameters
Change the following variables according to your definitions.
Can you write Python code for it?

# List of path(s) to criteo file(s) or folder(s) in GCS.
# Training files
TRAIN_PATHS = ['gs://renatoleite-criteo-full/'] # Training CSV file to be preprocessed.
# Validation files
VALID_PATHS = ['gs://renatoleite-criteo-full/day_0'] # Validation CSV file to be preprocessed.

# Data preprocessing parameters
num_output_files_train = 24 # Number of output files after converting CSV to Parquet
num_output_files_valid = 1 # Number of output files after converting CSV to Parquet

# Training parameters
NUM_EPOCHS = 0
MAX_ITERATIONS = 25000
EVAL_INTERVAL = 1000
EVAL_BATCHES = 500
EVAL_BATCHES_FINAL = 2500
DISPLAY_INTERVAL = 200
SNAPSHOT_INTERVAL = 0
PER_GPU_BATCHSIZE = 2048
LR = 0.001
DROPOUT_RATE = 0.5

parameter_values = {
    'train_paths': TRAIN_PATHS,
    'valid_paths': VALID_PATHS,
    'shuffle': json.dumps(None), # select PER_PARTITION, PER_WORKER, FULL, or None.
    'num_output_files_train': num_output_files_train,
    'num_output_files_valid': num_output_files_valid,
    'per_gpu_batch_size': PER_GPU_BATCHSIZE,
    'max_iter': MAX_ITERATIONS,
    'max_eval_batches': EVAL_BATCHES ,
    'eval_batches': EVAL_BATCHES_FINAL ,
    'dropout_rate': DROPOUT_RATE,
    'lr': LR ,
    'num_epochs': NUM_EPOCHS,
    'eval_interval': EVAL_INTERVAL,
    'snapshot': SNAPSHOT_INTERVAL,
    'display_interval': DISPLAY_INTERVAL
}


I am working on the file "04-e2e-pipeline.ipynb".
The first task is:
5. Compile KFP pipeline
Can you write Python code for it?

from src.pipelines import training_pipelines

compiled_pipeline_path = 'merlin_training_pipeline.json'
compiler.Compiler().compile(
       pipeline_func=training_pipelines.training_pipeline,
       package_path=compiled_pipeline_path
)


I am working on the file "04-e2e-pipeline.ipynb".
The first task is:
Build HugeCTR training container image
Can you write Python code for it?

FILE_LOCATION = './src'
! gcloud builds submit --config src/cloudbuild.yaml --substitutions _DOCKERNAME=$HUGECTR_DOCKERNAME,_IMAGE_URI=$HUGECTR_IMAGE_URI,_FILE_LOCATION=$FILE_LOCATION --timeout=2h --machine-type=e2-highcpu-8


I am working on the file "04-e2e-pipeline.ipynb".
The first task is:
The following cell lists the configuration values in config.py
Can you write Python code for it?

from src.pipelines import config
import importlib
importlib.reload(config)

for key, value in config.__dict__.items():
    if key.isupper(): print(f'{key}: {value}')


I am working on the file "04-evaluation-metrics.ipynb".
The first task is:
1. Classification Metrics
scikit-learn documentation
Accuracy
Accuracy is the fraction of the correct predictions:
$$ accuracy(y, \hat{y}) = \frac{1}{m} \sum_{i=1}^m 1(y^{(i)} = \hat{y}^{(i)}) $$
where $ 1(x) $ is the indicator function and $ m $ is the number of samples.
Can you write Python code for it?

import numpy as np
from sklearn.metrics import accuracy_score
y_pred = [0, 2, 1, 3]
y_true = [0, 1, 2, 3]
accuracy_score(y_true, y_pred)


I am working on the file "04-evaluation-metrics.ipynb".
The first task is:
2. Regression Metrics
scikit-learn documentation
Mean Absolute Error
$$ MAE(y, \hat{y}) = \frac{1}{m} \sum_{i=1}^m \mid y^{(i)} - \hat{y}^{(i)} \mid $$
Can you write Python code for it?

from sklearn.metrics import mean_absolute_error
y_true = [3, -0.5, 2, 7]
y_pred = [2.5, 0.0, 2, 8]
mean_absolute_error(y_true, y_pred)


I am working on the file "04-evaluation-metrics.ipynb".
The first task is:
Confusion Matrix
For multiclass classification:
<img src="images/metrics/confusion-matrix.png" width="500" />
and for binary classification:
Can you write Python code for it?

import numpy as np
from sklearn.metrics import confusion_matrix
y_true = [0, 0, 0, 1, 1, 1, 1, 1]
y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
np.array([[tn, fp],
          [fn, tp]])


I am working on the file "04-evaluation-metrics.ipynb".
The first task is:
Cross Entropy
$$ l(y, \hat{y}) = - \frac{1}{m} \sum_{i=1}^m y^{(i)} log(\hat{y}^{(i)}) + (1 - y^{(i)})log(1 - \hat{y}^{(i)}) $$
Can you write Python code for it?

from sklearn.metrics import log_loss
y_true = [0, 0, 1, 1]
y_pred = [[.9, .1], [.8, .2], [.3, .7], [.01, .99]]
log_loss(y_true, y_pred)


I am working on the file "04-evaluation-metrics.ipynb".
The first task is:
Mean Squared Error
$$ MSE(y, \hat{y}) = \frac{1}{m} \sum_{i=1}^m (y^{(i)} - \hat{y}^{(i)})^2 $$
Can you write Python code for it?

from sklearn.metrics import mean_squared_error
y_true = [3, -0.5, 2, 7]
y_pred = [2.5, 0.0, 2, 8]
mean_squared_error(y_true, y_pred)


I am working on the file "04-model-and-evaluation.ipynb".
The first task is:
Build a model
Can you write Python code for it?

#create linear regression object
#lm = linear_model.LinearRegression()

#create random forest object
#rf = ensemble.RandomForestClassifier(n_estimators=10) - did not work due to memory errors during fitting
rf = ensemble.RandomForestRegressor(n_estimators=100)

#train the model using the training data
rf.fit(X_train, y_train)


I am working on the file "04-model-and-evaluation.ipynb".
The first task is:
Check and convert all data types to numerical
Can you write Python code for it?

# Check data types
df_train.dtypes

#these are all numerical but are not continuous values and therefore don't have relative significant to one another, except for week
#however, creating dummy variables for all these is too memory intensive. as such, might have to explore using a random forest model
#in addition to the linear regression model


I am working on the file "04-model-and-evaluation.ipynb".
The first task is:
Check for missing values and drop or impute
Can you write Python code for it?

# Check counts for missing values in each column
print "train dataset missing values:"
print df_train2.isnull().sum()
print
print "test dataset missing values:"
print df_test2.isnull().sum()


I am working on the file "04-model-and-evaluation.ipynb".
The first task is:
Part 1. Identify the Problem
Problem: Given various sales/client/product data, we want to predict demand for each product at each store on a weekly basis. Per the train dataset, the average demand for a product at a store per week is 7.2 units. However, this does not factor in cases in which store managers under-predict demand for a product which we can see when returns=0 for that week. There are 74,180,464 records in the train data, of which 71,636,003 records have returns=0 or approx 96%. This generally means that managers probably often under predict product demand (unless that are exact on the money, which seems unlikely).
Goals: The goal is to predict demand for each product at each store on a weekly basis while avoiding under-predicting demand.
Hypothesis: As stated previously, the average product demand at a store per week is 7.2 units per the train data. However, given the likelihood of managers underpredicint product demand, I hypothesize a good model should return a number higher than 7.2 units to more accurately predict demand.

Part 2. Acquire the Data
Kaggle has provided five files for this dataset:
train.csv: Use for building a model (contains target variable "Demanda_uni_equil")
test.csv: Use for submission file (fill in for target variable "Demanda_uni_equil")
cliente_tabla.csv: Contains client names (can be joined with train/test on Cliente_ID)
producto_tabla.csv: Contains product names (can be join with train/test on Producto_ID)
town_state.csv: Contains town and state (can be join with train/test on Agencia_ID)
Notes: I will further split train.csv to generate my own cross validation set. However, I will use all of train.csv to train my final model since Kaggle has already supplied a test dataset. Additionally, I am only using a random 10% of the train data given to me for EDA and model development. Using the entire train dataset proved to be too time consuming for the quick iternations needed for initial modeling building and EDA efforts. I plan to use 100% of the train dataset once I...
Can you write Python code for it?

# Load train data
# Given size of training data, I chose to use only 10% for speed reasons
# QUESTION - how can i randomize with python? i used sql to create the random sample below.
df_train = pd.read_csv("../train_random10percent.csv")

# Check head
df_train.head()

# Load test data
df_test = pd.read_csv("../test.csv")

# Check head. I noticed that I will have to drop certain columns so that test and train sets have the same features.
df_test.head()

#given that i cannot use a significant amount of variables in train data, i created additoinal features using the mean
#i grouped on product id since i will ultimately be predicting demand for each product

df_train_mean = df_train.groupby('Producto_ID').mean().add_suffix('_mean').reset_index()
df_train_mean.head()

#from above, adding 2 additional features, the average sales units and the average demand
df_train2 = df_train.merge(df_train_mean[['Producto_ID','Venta_uni_hoy_mean', 'Demanda_uni_equil_mean']],how='inner',on='Producto_ID')
df_train2.sample(5)

# Adding features to the test set in order to match train set
df_test2 = df_test.merge(df_train_mean[['Producto_ID','Venta_uni_hoy_mean', 'Demanda_uni_equil_mean']],how='left',on='Producto_ID')
df_test2.head()


I am working on the file "04-model-and-evaluation.ipynb".
The first task is:
Part 3. Parse, Mine, and Refine the data
Perform exploratory data analysis and verify the quality of the data.
Check columns and counts to drop any non-generic or near-empty columns
Can you write Python code for it?

# Check columns
print "train dataset columns:"
print df_train2.columns.values
print
print "test dataset columns:"
print df_test2.columns.values

# Check counts
print "train dataset counts:"
print df_train2.count()
print
print "test dataset counts:"
print df_test2.count()


I am working on the file "04-model-and-evaluation.ipynb".
The first task is:
Part 4. Build a Model
Create a cross validation split, select and build a model, evaluate the model, and refine the model
Create cross validation sets
Can you write Python code for it?

#create cross validation sets

#set target variable name
target = 'Demanda_uni_equil'

#set X and y
X = df_train2.drop([target], axis=1)
y = df_train2[target]

# create separate training and test sets with 60/40 train/test split
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size= .4)

#QUESTION - do i have to cross validate when using kaggle data? prob not.


I am working on the file "04-model-and-evaluation.ipynb".
The first task is:
Part 5: Present the Results
Generate summary of findings and kaggle submission file.
NOTE: For the purposes of generating summary narratives and kaggle submission, we can train the model on the entire training data provided in train.csv.
Load Kaggle training data and use entire data to train tuned model
Can you write Python code for it?

# Set target variable name
target = 'Demanda_uni_equil'

# Set X_train and y_train
X_train = df_train2.drop([target], axis=1)
y_train = df_train2[target]

# Build tuned model
#create linear regression object
#lm = linear_model.LinearRegression()

#create random forest object
rf = ensemble.RandomForestRegressor(n_estimators=50)
#n_estimators is too memory instensive for 30gb of ram, so trying 50

#train the model using the training data
#lm.fit(X_train,y_train)
rf.fit(X_train,y_train)

# Score tuned model
print "Score: %0.3f" % rf.score(X_train, y_train)
#score is .906 when n_estimators=10, .93 when n_estimators = 50


I am working on the file "04-model-and-evaluation.ipynb".
The first task is:
Perform exploratory data analysis
Can you write Python code for it?

# Get summary statistics for data
df_train2.describe()

#Show box plot of demand by week
sns.factorplot(
    x='Semana',
    y='Demanda_uni_equil',
    data=df_train2,
    kind='box')


I am working on the file "04-model-and-evaluation.ipynb".
The first task is:
Wrangle the data to address any issues from above checks
Can you write Python code for it?

# Drop columns not included in test dataset
df_train2 = df_train2.drop(['Venta_uni_hoy', 'Venta_hoy', 'Dev_uni_proxima', 'Dev_proxima'], axis=1)

# Check data
df_train2.head()

# Drop blank values in test set and replace with mean

# Replace missing values for venta_uni_hoy_mean using mean
df_test2.loc[(df_test2['Venta_uni_hoy_mean'].isnull()), 'Venta_uni_hoy_mean'] = df_test2['Venta_uni_hoy_mean'].dropna().mean()

# Replace missing values for demand using mean
df_test2.loc[(df_test2['Demanda_uni_equil_mean'].isnull()), 'Demanda_uni_equil_mean'] = df_test2['Demanda_uni_equil_mean'].dropna().mean()

print "test dataset missing values:"
print df_test2.isnull().sum()


I am working on the file "04-model-performance/exercise.ipynb".
The first task is:
A common task when building a new model is to select only those variables that are "best"
for the problem. This selection procedure can take many different shapes, here we will
compute the correlation of each feature with the target, select the 20 features that
have the highest correlation and use those in our gradient boosted tree ensemble.
We will then use cross validation to evaluate the performance.
Can you write Python code for it?

def make_data(N=1000, n_vars=10,
              n_classes=2):
    X = np.random.normal(size=(N,n_vars))
    y = np.random.choice(n_classes, N)

    return X, y

X,y = make_data(N=2000, n_vars=50000)

select = SelectKBest(f_regression, k=20)
X_sel = select.fit_transform(X, y)

clf = GradientBoostingClassifier()
scores = cross_val_score(clf, X_sel, y, cv=5, n_jobs=8)

print("Scores on each subset:")
print(scores)
avg = (100*np.mean(scores), 100*np.std(scores)/np.sqrt(scores.shape[0]))
print("Average score and uncertainty: (%.2f +- %.3f)%%" % avg)


I am working on the file "04-model-performance/lecture.ipynb".
The first task is:
As we explore different values of C are we going to find a setting which does the equivalent of "memorising" the dataset? Is the score obtained on the testing dataset
a fair estimate of the classifier's performance on new, unseen data?
The answer is no.
Three way split
We need a third set, a validation set. To get the naming right:

our whole dataset is X
we split X into training and testing
we further split the testing dataset into a testing and validation set

Or we directly split X into training, validation, and testing. The testing dataset should be locked away into a vault. You can only use it once you have frozen all the parameters and
choices involved in your model.
One drawback of splitting things into three sets is that we reduce the number of points
available for training, testing and validation. A method to maximise the amount of data we can use from our test and validation set is called cross-validation.
We still create a testing dataset that gets locked in a vault. However we split our
training dataset using cross-validation (CV). The idea is the following:
Split the training data into k blocks. Select one of the k blocks as testing data,
train your model on the other k-1 blocks. Change k, and repeat. This allows you to
obtain k independent estimates of your models performance.
<img src="cv.png" />
In scikit-learn you can obtain estimates using CV as follows:
Can you write Python code for it?

from sklearn.model_selection import cross_val_score
scores = cross_val_score(clf, X_train, y_train, cv=5)
print(scores)

# as you have several estimates you can also compute a measure of the spread
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std()))


I am working on the file "04-model-performance/lecture.ipynb".
The first task is:
Each model we have met so far has several parameters that need tuning. So called hyper-parameters. This lecture will discuss methods for systematically searching for the best
hyper-parameters and the nassessing how well the model will perform on unseen data.
Overfitting
A model that simply memorises all training points will obtain a perfect score on the
dataset used for training. In general it will perform poorly on new, unseen data. What
we really want is not a model that memorises but one that learns to generalise. This means
we should use a dataset that was not seens as part of the model fitting to evaluate the
performance of the model.
In the past lectures we split our data into a training and testing set, then tried different
values of a hyper-parameter and picked the one that performs best on the test dataset:
Can you write Python code for it?

from sklearn.model_selection import train_test_split
from sklearn import datasets
from sklearn import svm

iris = datasets.load_iris()

X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target,
                                                    test_size=0.3, random_state=1)

clf = svm.SVC(kernel='linear', C=1).fit(X_train, y_train)
clf.score(X_test, y_test)


I am working on the file "04-model-performance/lecture.ipynb".
The first task is:
Tune up
So far we only ever tuned one parameter at a time for a model. Most models have several
parameters that influence each other. To tune them all in one go we can perform a grid
search. We specify a grid of parameters to evaluate and for each point we perform CV:
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV

parameter_grid = [{'kernel': ['rbf'], 'gamma': [1e-1, 1e-2, 1e-3, 1e-4],
                   'C': [1, 10, 20, 30, 40, 50, 100, 1000]},
                  {'kernel': ['linear'], 'C': [1, 10, 20, 30, 40, 50, 100, 1000]}]

clf = GridSearchCV(svm.SVC(C=1), parameter_grid, cv=5)
clf.fit(X_train, y_train)

# best parameters and score. Should we report this as our performance?
print(clf.best_score_, clf.best_params_)

# test scores for the whole grid
means = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']
for mean, std, params in zip(means, stds, clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r" % (mean, std, params))

# score on the dataset we had in our vault
clf.score(X_test, y_test)


I am working on the file "04-model-performance/lecture.ipynb".
The first task is:
Why is random better?
For illustration purposes let's use a function which has two parameters X1 and X2. However the output only depends on one of them: X1.
Can you write Python code for it?

def f(x):
    return (0.5-x[0])**2

X1 = np.linspace(-5, 5, 5)
X2 = np.linspace(-5, 5, 5)

param_grid = []
for x1 in X1:
    for x2 in X2:
        param_grid.append((x1, x2))
param_grid = np.array(param_grid)

plt.scatter(param_grid[:, 0], param_grid[:, 1], c=[f(x) for x in param_grid], s=260)
plt.colorbar()
plt.xlabel("X1")
plt.ylabel("X2");

vals = []
for x1 in X1:
    for x2 in X2:
        vals.append((x1, x2, f((x1, x2))))
vals = np.array(vals)

plt.plot(vals[:, 0], vals[:, 2], 'o')
plt.xlabel("X1")
plt.ylabel("f(X)");

vals = []
for n in range(25):
    x1 = np.random.uniform(-5, 5)
    x2 = np.random.uniform(-5, 5)
    vals.append((x1, x2, f((x1, x2))))
vals = np.array(vals)

plt.plot(vals[:, 0], vals[:, 2], 'o')
plt.xlabel("X1")
plt.ylabel("f(X)");

plt.scatter(vals[:, 0], vals[:, 1], c=vals[:, 2], s=260)
plt.colorbar()
plt.xlabel("X1")
plt.ylabel("X2");


I am working on the file "04-more-supervised.ipynb".
The first task is:
Predictions and Evaluations
Now let's predict using the trained model.
Can you write Python code for it?

predictions = model.predict(X_test)

from sklearn.metrics import classification_report,confusion_matrix

print(confusion_matrix(y_test,predictions))

print(classification_report(y_test,predictions))


I am working on the file "04-more-supervised.ipynb".
The first task is:
Train Test Split
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(df_in, np.ravel(df_target), test_size=0.30, random_state=101)


I am working on the file "04-more-supervised.ipynb".
The first task is:
Train the Support Vector Classifier
Can you write Python code for it?

from sklearn.svm import SVC

model = SVC(kernel='rbf')

model.fit(X_train,y_train)


I am working on the file "04-more-supervised.ipynb".
The first task is:
Woah! Notice that we are classifying everything into a single class! This means our model needs to have it parameters adjusted (it may also help to normalize the data).
We can search for parameters using a GridSearch!
Gridsearch
Finding the right parameters (like what C or gamma values to use) is a tricky task! But luckily, we can be a little lazy and just try a bunch of combinations and see what works best! This idea of creating a 'grid' of parameters and just trying out all the possible combinations is called a Gridsearch, this method is common enough that Scikit-learn has this functionality built in with GridSearchCV! The CV stands for cross-validation which is the
GridSearchCV takes a dictionary that describes the parameters that should be tried and a model to train. The grid of parameters is defined as a dictionary, where the keys are the parameters and the values are the settings to be tested.
Can you write Python code for it?

param_grid = {'C': [0.1,1, 10, 100, 1000], 'gamma': [1,0.1,0.01,0.001,0.0001], 'kernel': ['rbf']}

from sklearn.model_selection import GridSearchCV

grid = GridSearchCV(SVC(),param_grid,refit=True,verbose=3)

grid.fit(X_train,y_train)

grid.best_params_

grid.best_estimator_

grid_predictions = grid.predict(X_test)

print(confusion_matrix(y_test,grid_predictions))

print(classification_report(y_test,grid_predictions))


I am working on the file "04-notebook.ipynb".
The first task is:
Aufgabe 3     Beliebig lange Phrasen
Erweitern Sie die Grammatik nun derart, dass Nominalphrasen auch Adjektive enthalten dürfen – und zwar beliebig viele.
Beispiele:
Can you write Python code for it?

test_sentences.extend([
    "die kluge schöne Frau kennt das Buch",
    "der schöne kluge Mann gibt der Frau das dicke Buch",
    "das dicke schöne kluge Buch schläft"
])

grammar3 = """
    S -> NP VP
    NP -> DET N
    DET -> "der" | "die" | "das"
    N -> "Mann" | "Frau" | "Buch"
    VP -> V NP NP
    V -> "gibt" | "schenkt"
"""

test_grammar(grammar3, test_sentences)


I am working on the file "04-notebook.ipynb".
The first task is:
Aufgabe 5     Präpositionalphrasen
Erweitern Sie die Grammatik nun derart, dass sowohl Nominalphrasen als auch Verbalphrasen durch Präpositionalphrasen modifiziert werden können. Erkennen Sie die Ambiguität von Übungsblatt 1 wieder?
Folgende Sätze sollten akzeptiert werden:
Can you write Python code for it?

test_sentences.extend([
    "der Mann schläft neben dem Buch",
    "die Frau kennt das dicke Buch über Chomsky",
    "die Frau schenkt dem Mann das Buch auf dem Tisch"
])


I am working on the file "04-notebook.ipynb".
The first task is:
Die folgende Funktion kann Ihnen helfen, eine Reihe von Sätzen zu analysieren.
Je nach Ihrem Geschmack können Sie Zeile 14 ggf. anpassen.
Can you write Python code for it?

import nltk
from IPython.display import display
import sys

def test_grammar(grammar, sentences):
    cfg = nltk.CFG.fromstring(grammar)
    rd_parser = nltk.RecursiveDescentParser(cfg)

    for i, sent in enumerate(sentences, 1):
        print("Satz {}: {}".format(i, sent))
        results = rd_parser.parse(sent.split())
        analyzed = False
        for tree in results:
            tree.pretty_print(unicodelines=True)  # tree.draw() oder display(tree)
            analyzed = True
        if not analyzed:
            print("Keine Analyse möglich", file=sys.stderr)


I am working on the file "04-notebook.ipynb".
The first task is:
Stellen Sie außerdem sicher, dass folgende Sätze KEINE Analyse liefern!
Can you write Python code for it?

negative_examples = [
    "Mann gibt Frau Buch",
    "Mann schläft"
]

grammar4 = """
    S -> NP VP
    NP -> DET N
    DET -> "der" | "die" | "das"
    N -> "Mann" | "Frau" | "Buch"
    VP -> V NP NP
    V -> "gibt" | "schenkt"
"""

test_grammar(grammar4, test_sentences)


I am working on the file "04-packing-and-allocation/bin_packing.ipynb".
The first task is:
1. First lets make some fake data
Can you write Python code for it?

items=['item_%d'%i for i in range(50)]

item_sizes = dict( (i,np.random.randint(1,20)) for i in items)


I am working on the file "04-packing-and-allocation/bin_packing.ipynb".
The first task is:
And the result:
Can you write Python code for it?

print(value(prob.objective))

for b in bins:
    if value(y[b]) !=0:
        print(b,':',', '.join([ i for i in items if value(x[i,b]) !=0 ]))


I am working on the file "04-packing-and-allocation/bin_packing.ipynb".
The first task is:
Solve it!
Can you write Python code for it?


I am working on the file "04-packing-and-allocation/bin_packing.ipynb".
The first task is:
The Model
Lets model the each possible bins as having at most N spots to be filled by item 0,...,N.
$$x_{i,b} = \begin{cases}
    1, & \text{if item i is in bin b } \
    0, & \text{otherwise}
\end{cases}
$$
We need to make sure each item is placed in exactly one bin. Ie for any given item, summing $x_{i,b}$ along the bins should equal 1.
$$\sum_{b} x_{i,b} = 1 \ \forall i$$
We also need to make sure that if a bin is used, it is not used beyond its capacity.
$$\sum_{i} x_{i,b} \leq \text{bin_capacity}*y_{b} \ \forall b$$
Finally, we are trying to minimize the number of needed bins, so our objective is:
$$\text{Minimize} \ \sum_{b} y_{b}$$
Can you write Python code for it?

bin_size = 40

#average item size
avg_size = np.mean([ item_sizes[k] for k in item_sizes])
plt.barh([0,1],[avg_size,bin_size],height=.99)
plt.gca().text(.5,1.5,'Bin Size',verticalalignment='center', fontsize=12)
plt.gca().text(.5,0.5,'Avg item Size', verticalalignment='center', fontsize=12)
plt.ylim(0,2)
plt.gca().axis('off');
plt.title('Bin size vs Object Size')

bins = ['bin_%d'%i for i in range(len(items))]

x = LpVariable.dicts('x',[(i,b) for i in items for b in bins],0,1,LpBinary)

y = LpVariable.dicts('bin',bins,0,10, LpBinary)

#create the problme
prob=LpProblem("bin_packing",LpMinimize)

#the objective
cost = lpSum([ y[b] for b in bins])
prob+=cost

#every item is placed in exactly one bin
for i in items:
    prob+= lpSum([x[i,b] for b in bins]) == 1

#if a bin is used, it has a capacity constraint
for b in bins:
    prob+=lpSum([ item_sizes[i]*x[i,b] for i in items]) <= bin_size*y[b]


I am working on the file "04-packing-and-allocation/knapsack_problem.ipynb".
The first task is:
And the result:
Can you write Python code for it?

for i in items:
    print(i, value(x[i]))

print(value(prob.objective))

print(sum([ item_weights[i]*value(x[i]) for i in items]))


I am working on the file "04-pandas.ipynb".
The first task is:
Com o pandas vocês podem facilmente "re-amostar" uma série ou tabela
para uma nova frequência temporal.
Can you write Python code for it?

import numpy as np

fig, ax = plt.subplots(figsize=(9, 5))

ax = df.resample('M', how=np.median).plot(ax=ax)


I am working on the file "04-pandas.ipynb".
The first task is:
Group-by months/years (climatology)
Can you write Python code for it?

key = lambda x: x.month

grouped = df.groupby(key)

monthly = grouped.mean()
monthly.shape

fig, ax = plt.subplots(figsize=(9, 5))

ax = monthly.plot(ax=ax)

key = lambda x: x.year

grouped = df.groupby(key)

yearly = grouped.mean()
yearly.shape

fig, ax = plt.subplots(figsize=(9, 5))

ax = yearly.plot(ax=ax)

ax = df.resample('A', how='mean').plot()

for col in df.columns:
    yearly[col] = ((yearly[col] - yearly[col].mean()) /
                   yearly[col].std(ddof=0))

fig, ax = plt.subplots(figsize=(9, 5))

ax = yearly.plot(ax=ax)

yearly.corr()

!head ./data/15t30717.3f1

from datetime import datetime
from pandas import read_table

cols = ['j', 'u', 'v', 'temp', 'sal', 'y', 'mn', 'd', 'h', 'mi']

df = read_table('./data/15t30717.3f1' , delim_whitespace=True, names=cols)
dates = [datetime(*x) for x in
         zip(df['y'], df['mn'], df['d'], df['h'], df['mi'])]
df.index = dates
df.drop(['y', 'mn', 'd', 'h', 'mi', 'j'], axis=1, inplace=True)
df.head()

ax = df['v'].plot()

from oceans import lanc

freq = 1./40  # Hours
window_size = 96+1+96
pad = np.zeros(window_size) * np.NaN

wt = lanc(window_size, freq)
res = np.convolve(wt, df['v'], mode='same')

df['low'] = res
df['high'] = df['v'] - df['low']

ax = df[['low', 'high']].plot(figsize=(9, 3))


I am working on the file "04-pandas.ipynb".
The first task is:
Outra facilidade é a limpeza da tabela, aqui jogamos fora todos os inválidos
(Not-a-Number) apenas quando todos os valores do eixo 1 (colunas) foram
NaN.
Can you write Python code for it?

df.dropna(how='all', axis=1, inplace=True)
df.head()

desc = df.describe()
desc


I am working on the file "04-pipeline-deployment.ipynb".
The first task is:
2. Run the training pipeline using Vertex Pipelines
Set the pipeline configurations for the Vertex AI run
Can you write Python code for it?

os.environ["DATASET_DISPLAY_NAME"] = DATASET_DISPLAY_NAME
os.environ["MODEL_DISPLAY_NAME"] = MODEL_DISPLAY_NAME
os.environ["PIPELINE_NAME"] = PIPELINE_NAME
os.environ["PROJECT"] = PROJECT
os.environ["REGION"] = REGION
os.environ["GCS_LOCATION"] = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}"
os.environ["TRAIN_LIMIT"] = "85000"
os.environ["TEST_LIMIT"] = "15000"
os.environ["BEAM_RUNNER"] = "DataflowRunner"
os.environ["TRAINING_RUNNER"] = "vertex"
os.environ["TFX_IMAGE_URI"] = f"gcr.io/{PROJECT}/{DATASET_DISPLAY_NAME}:{VERSION}"
os.environ["ENABLE_CACHE"] = "1"

from src.tfx_pipelines import config
import importlib
importlib.reload(config)

for key, value in config.__dict__.items():
    if key.isupper(): print(f'{key}: {value}')


I am working on the file "04-pipeline-deployment.ipynb".
The first task is:
3. Execute the pipeline deployment CI/CD steps in Cloud Build
The CI/CD routine is defined in the pipeline-deployment.yaml file, and consists of the following steps:
1. Clone the repository to the build environment.
2. Run unit tests.
3. Run a local e2e test of the pipeline.
4. Build the ML container image for pipeline steps.
5. Compile the pipeline.
6. Upload the pipeline to Cloud Storage.
Build CI/CD container Image for Cloud Build
This is the runtime environment where the steps of testing and deploying the pipeline will be executed.
Can you write Python code for it?

!echo $CICD_IMAGE_URI

!gcloud builds submit --tag $CICD_IMAGE_URI build/. --timeout=15m --machine-type=e2-highcpu-8


I am working on the file "04-pipeline-deployment.ipynb".
The first task is:
Build the ML container image
This is the TFX runtime environment for the training pipeline steps.
Can you write Python code for it?

!echo $TFX_IMAGE_URI

!gcloud builds submit --tag $TFX_IMAGE_URI . --timeout=15m --machine-type=e2-highcpu-8


I am working on the file "04-pipeline-deployment.ipynb".
The first task is:
Compile pipeline
Can you write Python code for it?

from src.tfx_pipelines import runner

pipeline_definition_file = f'{config.PIPELINE_NAME}.json'
pipeline_definition = runner.compile_training_pipeline(pipeline_definition_file)

PIPELINES_STORE = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}/compiled_pipelines/"
!gsutil cp {pipeline_definition_file} {PIPELINES_STORE}


I am working on the file "04-pipeline-deployment.ipynb".
The first task is:
Extracting pipeline runs metadata
Can you write Python code for it?

from google.cloud import aiplatform as vertex_ai

pipeline_df = vertex_ai.get_pipeline_df(PIPELINE_NAME)
pipeline_df = pipeline_df[pipeline_df.pipeline_name == PIPELINE_NAME]
pipeline_df.T


I am working on the file "04-pipeline-deployment.ipynb".
The first task is:
Run CI/CD from pipeline deployment using Cloud Build
Can you write Python code for it?

REPO_URL = "https://github.com/GoogleCloudPlatform/mlops-with-vertex-ai.git" # Change to your github repo.
BRANCH = "main"

GCS_LOCATION = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}/"
TEST_GCS_LOCATION = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}/e2e_tests"
CI_TRAIN_LIMIT = 1000
CI_TEST_LIMIT = 100
CI_UPLOAD_MODEL = 0
CI_ACCURACY_THRESHOLD = 0.1
BEAM_RUNNER = "DataflowRunner"
TRAINING_RUNNER = "vertex"
VERSION = 'tfx-1.2'
PIPELINE_NAME = f'{MODEL_DISPLAY_NAME}-train-pipeline'
PIPELINES_STORE = os.path.join(GCS_LOCATION, "compiled_pipelines")

TFX_IMAGE_URI = f"gcr.io/{PROJECT}/{DATASET_DISPLAY_NAME}:{VERSION}"

SUBSTITUTIONS=f"""\
_REPO_URL='{REPO_URL}',\
_BRANCH={BRANCH},\
_CICD_IMAGE_URI={CICD_IMAGE_URI},\
_PROJECT={PROJECT},\
_REGION={REGION},\
_GCS_LOCATION={GCS_LOCATION},\
_TEST_GCS_LOCATION={TEST_GCS_LOCATION},\
_BQ_LOCATION={BQ_LOCATION},\
_BQ_DATASET_NAME={BQ_DATASET_NAME},\
_BQ_TABLE_NAME={BQ_TABLE_NAME},\
_DATASET_DISPLAY_NAME={DATASET_DISPLAY_NAME},\
_MODEL_DISPLAY_NAME={MODEL_DISPLAY_NAME},\
_CI_TRAIN_LIMIT={CI_TRAIN_LIMIT},\
_CI_TEST_LIMIT={CI_TEST_LIMIT},\
_CI_UPLOAD_MODEL={CI_UPLOAD_MODEL},\
_CI_ACCURACY_THRESHOLD={CI_ACCURACY_THRESHOLD},\
_BEAM_RUNNER={BEAM_RUNNER},\
_TRAINING_RUNNER={TRAINING_RUNNER},\
_TFX_IMAGE_URI={TFX_IMAGE_URI},\
_PIPELINE_NAME={PIPELINE_NAME},\
_PIPELINES_STORE={PIPELINES_STORE}\
"""

!echo $SUBSTITUTIONS

!gcloud builds submit --no-source --timeout=60m --config build/pipeline-deployment.yaml --substitutions {SUBSTITUTIONS} --machine-type=e2-highcpu-8


I am working on the file "04-pipeline-deployment.ipynb".
The first task is:
Run unit tests
Can you write Python code for it?

!py.test src/tests/datasource_utils_tests.py -s

!py.test src/tests/model_tests.py -s


I am working on the file "04-pipeline-deployment.ipynb".
The first task is:
Set configurations
Can you write Python code for it?

BQ_LOCATION = 'US'
BQ_DATASET_NAME = 'playground_us' # Change to your BQ dataset name.
BQ_TABLE_NAME = 'chicago_taxitrips_prep'

VERSION = 'v01'
DATASET_DISPLAY_NAME = 'chicago-taxi-tips'
MODEL_DISPLAY_NAME = f'{DATASET_DISPLAY_NAME}-classifier-{VERSION}'
PIPELINE_NAME = f'{MODEL_DISPLAY_NAME}-train-pipeline'

CICD_IMAGE_NAME = 'cicd:latest'
CICD_IMAGE_URI = f"gcr.io/{PROJECT}/{CICD_IMAGE_NAME}"

!rm -r src/raw_schema/.ipynb_checkpoints/


I am working on the file "04-pipeline-deployment.ipynb".
The first task is:
Setup Google Cloud project
Can you write Python code for it?

PROJECT = '[your-project-id]' # Change to your project id.
REGION = 'us-central1' # Change to your region.
BUCKET =  '[your-bucket-name]' # Change to your bucket name.
SERVICE_ACCOUNT = "[your-service-account]"

if PROJECT == "" or PROJECT is None or PROJECT == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT = shell_output[0]

if SERVICE_ACCOUNT == "" or SERVICE_ACCOUNT is None or SERVICE_ACCOUNT == "[your-service-account]":
    # Get your GCP project id from gcloud
    shell_output = !gcloud config list --format 'value(core.account)' 2>/dev/null
    SERVICE_ACCOUNT = shell_output[0]

if BUCKET == "" or BUCKET is None or BUCKET == "[your-bucket-name]":
    # Get your bucket name to GCP project id
    BUCKET = PROJECT
    # Try to create the bucket if it doesn't exists
    ! gsutil mb -l $REGION gs://$BUCKET
    print("")

print("Project ID:", PROJECT)
print("Region:", REGION)
print("Bucket name:", BUCKET)
print("Service Account:", SERVICE_ACCOUNT)


I am working on the file "04-pipeline-deployment.ipynb".
The first task is:
Submit run to Vertex Pipelines
Can you write Python code for it?

from kfp.v2.google.client import AIPlatformClient

pipeline_client = AIPlatformClient(
    project_id=PROJECT, region=REGION)

job = pipeline_client.create_run_from_job_spec(
    job_spec_path=pipeline_definition_file,
    parameter_values={
        'learning_rate': 0.003,
        'batch_size': 512,
        'hidden_units': '128,128',
        'num_epochs': 30,
    }
)


I am working on the file "04-python-prat/data_science/Python and Data Science _sem_spoilers.ipynb".
The first task is:
Carregando um arquivo csv em um DataFrame do Pandas

pd.DataFrame.from_csv(file_name)
Se, ao usar este comando, você se deparar com um UnicodeDecodingError, adicione o parâmetro encoding='utf-8'

cast.csv
release_dates.csv
titles
df.head(n):
* Visualizar as primeiras n linhas.
* Default: n = 5.
df.tail(n):
* Visualizar as últimas n linhas.
* Default: n = 5.
Quantos registros há no conjunto?
len(df):
* Tamanho do df
Quais são os possíveis valores para a coluna type?
df[col]:
* Visualizar uma coluna do df
ou
df.col:
* Se o nome da coluna não tiver, espaços, caracteres especiais ou for uma variável
Obs: Ao selecionar uma coluna e manipulá-la fora de um DataFrame, a mesma é tratada como uma Série.
df[col].unique():
* Mostrar os possíveis valores de uma coluna
Quantos atores e quantas atrizes há no conjunto?
df[col].value_counts():
* Contagem de quantos registros há para cada valor possível da coluna col (somente se col for categórica)
Operações com colunas
Operações Aritméticas
Comparações
Filtrar

Por valor específico de uma coluna

Por colunas

Por valor nulo ou não nulo

Por vetor de booleanos

Preencher valores nulos

Por DataFrame
Por coluna
Quantos atores atuaram em cada ano?
Qual foi a diferença entre o número de atores e atrizes que atuaram em cada década?
Datas
Quanto % dos filmes foram lançados na sexta-feira?
Merge
Qual o nome e ano do filme mais antigo?
Quantos filmes são de 1960?
Quantos filmes são de cada ano dos anos 70?
Quantos filmes foram lançados desde o ano que você nasceu até hoje?
Quais são os nomes dos filmes de 1906?
Quais são os 15 nomes de filmes mais comuns?
Em quantos filmes Judi Dench atuou?
Liste os filmes nos quais Judi Dench atuou como o ator número 1, ordenado por ano.
Liste os atores da versão de 1972 de Sleuth pela ordem do rank n.
Quais atores mais atuaram em 1985?
SciKit Learn (http://scikit-learn.org)

Biblioteca Python para mineração e análise de dados

Como instalar

Anaconda...
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import confusion_matrix
from sklearn.cross_validation import train_test_split
import pickle
import time
time1=time.strftime('%Y-%m-%d_%H-%M-%S')


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Carregando um arquivo csv em um DataFrame do Pandas

pd.DataFrame.from_csv(file_name)
Se, ao usar este comando, você se deparar com um UnicodeDecodingError, adicione o parâmetro encoding='utf-8'

cast.csv
Can you write Python code for it?

%%time
cast = pd.DataFrame.from_csv('data/cast.csv', index_col=None, encoding='utf-8')


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Comparações
Can you write Python code for it?


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Datas
Can you write Python code for it?

release_dates.head()

release_dates.date.dt.year.head()

release_dates.date.dt.dayofyear.head() # segunda=0, domingo=6


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Em quantos filmes Judi Dench atuou?
Can you write Python code for it?


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Filtrar

Por valor específico de uma coluna
Can you write Python code for it?


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Liste os filmes nos quais Judi Dench atuou como o ator número 1, ordenado por ano.
Can you write Python code for it?

c = cast
c = c[c.name == 'Judi Dench']
c = c[c.n == 1]
c.sort_values('year')

c =cast
c = c[c.name == 'Judi Dench']
c


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Operações Aritméticas
Can you write Python code for it?


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Por valor nulo ou não nulo
Can you write Python code for it?


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Por vetor de booleanos
Can you write Python code for it?

h[[True, False, True, False, False]]

h.year > 2000

h[h.year > 2000]

h[(h.year > 2000) & (h.year < 2016)]    # & para 'and', | para 'or'


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Quais são os nomes dos filmes até 1906?
Can you write Python code for it?

titles[titles.year <= 1906][['title']]

titles.year.min()

titles.set_index('year').sort_index().loc[1894:1906]


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Quais são os possíveis valores para a coluna type?
df[col]:
* Visualizar uma coluna do df
ou
df.col:
* Se o nome da coluna não tiver, espaços, caracteres especiais ou for uma variável
Obs: Ao selecionar uma coluna e manipulá-la fora de um DataFrame, a mesma é tratada como uma Série.
Can you write Python code for it?

cast['type']

cast.type.head()

c = 'type'
cast[c].head()    #cast.c.head() não vai funcionar!


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Qual foi a diferença entre o número de atores e atrizes que atuaram em cada década?
Can you write Python code for it?

g = cast.groupby([cast.year // 10 * 10, 'type']).size()
g

u = g.unstack()
u

a = u['actor'] - u['actress']
a

a.plot()


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Quanto % dos filmes foram lançados na sexta-feira?
Can you write Python code for it?


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Quantos atores atuaram em cada ano?
Can you write Python code for it?

cast.year.value_counts()#.head(10)

cast.year.value_counts().plot()

cast.year.value_counts().sort_index()#.head()

cast.year.value_counts().sort_index().plot()

bins = pd.np.arange(1880, 2040, 2)
cast.year.hist(bins=bins)


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Quantos atores e quantas atrizes há no conjunto?
df[col].value_counts():
* Contagem de quantos registros há para cada valor possível da coluna col (somente se col for categórica)
Can you write Python code for it?


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Quantos filmes foram lançados desde o ano que você nasceu até hoje?
Can you write Python code for it?

birth = 1990
len(titles[(titles.year >= birth) & (titles.year <= 2016)])


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Quantos filmes são de 1960?
Can you write Python code for it?


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Quantos registros há no conjunto?
len(df):
* Tamanho do df
Can you write Python code for it?


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
Treinar modelo de Árvore de Decisão
Can you write Python code for it?

dt = DecisionTreeClassifier()

target_data = iris['species']
features = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width']
feature_data = iris[features]

dt = dt.fit(features_train, target_train)


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
df.head(n):
* Visualizar as primeiras n linhas.
* Default: n = 5.
Can you write Python code for it?


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
iris.csv
Can you write Python code for it?

iris = pd.DataFrame.from_csv('iris.csv', index_col=None, encoding='utf-8')

iris.columns

target_data = iris['species']
features = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width']
feature_data = iris[features]

features_train, features_test, target_train, target_test = train_test_split(feature_data, target_data, test_size=0.33, random_state=42)


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
release_dates.csv
Can you write Python code for it?

%%time
release_dates = pd.read_csv('data/release_dates.csv', index_col=None,
                            parse_dates=['date'], infer_datetime_format=True)


I am working on the file "04-python-prat/data_science/Python and Data Science.ipynb".
The first task is:
titles
Can you write Python code for it?

cast.columns

titles = cast[['title', 'year']].drop_duplicates().reset_index(drop=True)
titles.head()


I am working on the file "04-testing.ipynb".
The first task is:
Check obvious fractions:
Can you write Python code for it?


I am working on the file "04-testing.ipynb".
The first task is:
Each of these tests has their uses and may show different potential problems. What counts as "correct" depends on how you want your code to handle certain situations.
If are function didn't do what we wanted on one of these tests then we'd have to alter it and test again. This can be error prone, so it's better to write functions. We want these functions to complain loudly if something is wrong, but be quiet if all is well. To do this we can use the assert statement:
Can you write Python code for it?

def test_integer_division():
    assert(divide(4, 2) == 2)

test_integer_division()


I am working on the file "04-testing.ipynb".
The first task is:
Note the difference in the syntax between the two ways of testing, which are annoying.
py.test
We now have a set of tests - a testsuite, as it is sometimes called - encoded in functions, with meaningful names, which give useful error messages if the test fails. Every time the code is changed, we want to re-run all the tests to ensure that our change has not broken the code. This can be tedious. A better way would be to run a single command that runs all tests. pytest is that command.
The easiest way to use it is to put all tests in the same file as the function being tested. So, create a file test_divide.py containing all the functions above:
```python
from numpy.testing import assert_equal, assert_allclose
import pytest
def divide(x, y):
    """
    Divide two numbers
Parameters
----------

x : float
    Numerator
y : float
    Denominator

Returns
-------

x / y : float
"""
return x / y

def test_integer_division():
    assert_equal(divide(4, 2), 2,
                 err_msg="Dividing 4 by 2: answer should be 2")
def test_real_division1():
    assert_allclose(divide(1, 3), 0.33333333333333,
                 err_msg="Dividing 1 by 3: answer should be 1/3")
def test_power_division():
    a = 1.234
    assert_allclose(divide(a7, a), a6,
                 err_msg="Dividing a^7 by a: answer should be a^6")
def test_large_division():
    assert_equal(divide(1, 1e1000), 0,
                 err_msg="Dividing by a large enough number returns zero")
def test_zero_division():
    with pytest.raises(ZeroDivisionError,
                       message="Dividing by zero should give a ZeroDivisionError"):
        divide(1, 0)
The following command will run when the file is executed:
pytest.main()
```
Then execute the file by running it in the Spyder editor. This will define all the functions and the last line will run the tests. You should see output like
```
===================================== test session starts ======================================
platform darwin -- Python 3.6.1, pytest-3.1.1, py-1.4.33,...
Can you write Python code for it?

def divide(x, y):
    """
    Divide two numbers

    Parameters
    ----------

    x : float
        Numerator
    y : float
        Denominator

    Returns
    -------

    x / y : float
    """
    return x // y


I am working on the file "04-testing.ipynb".
The first task is:
This did not allow us to check that the division by zero was handled correctly. For that we'll need pytest:
Can you write Python code for it?

import pytest

def test_zero_division():
    with pytest.raises(ZeroDivisionError,
                       message="Dividing by zero should give a ZeroDivisionError"):
        divide(1, 0)

test_zero_division()


I am working on the file "04-testing.ipynb".
The first task is:
What happens if you divide by a really large number?
Can you write Python code for it?


I am working on the file "04-testing.ipynb".
The first task is:
What happens if you divide by zero? (What should happen?)
Can you write Python code for it?


I am working on the file "04-your-turn-build-predict-iterate.ipynb".
The first task is:
Load Data

Load Training Data
Split into Train and Validation
Can you write Python code for it?


I am working on the file "04-your-turn-build-predict-iterate.ipynb".
The first task is:
Model Tuning

Tune the Model if it's overfitting
Can you write Python code for it?


I am working on the file "04-your-turn-build-predict-iterate.ipynb".
The first task is:
Vectorize

Vectorize using Scikit-Learn

Hints: Checkout CountVectorizer, TFIDFVectorizer and others in sklearn.feature_extraction.text
Can you write Python code for it?


I am working on the file "04. 데이터 분석의 소개/01. 데이터 분석의 소개.ipynb".
The first task is:
데이터 분석의 소개
데이터 분석이란 용어는 상당히 광범위한 용어이므로 여기에서는 통계적 분석과 머신 러닝이라는 두가지 세부 영역에 국한하여 데이터 분석을 설명하도록 한다.
데이터 분석이란
데이터 분석이란 어떤 데이터가 주어졌을 때

데이터 간의 관계를 파악하거나
파악된 관계를 사용하여 원하는 데이터를 만들어 내는 과정

으로 볼 수 있다.
데이터 분석의 유형

예측(Prediction)
클러스터링(Clustering)
모사(Approximation)

데이터 분석의 유형은 다양하다. 그 중 널리 사용되는 전형적인 방법으로는 예측(prediction), 클러스터링(clustering), 모사(approximation) 등이 있다.
예측은 어떤 특정한 유형의 입력 데이터가 주어지면 데이터 분석의 결과로 다른 유형의 데이터가 출력될 수 있는 경우이다. 예를 들어 다음과 같은 작업은 예측이라고 할 수 있다.

부동산의 위치, 주거환경, 건축연도 등이 주어지면 해당 부동산의 가치를 추정한다.
꽃잎의 길이와 너비 등 식물의 외형적 특징이 주어지면 해당하는 식물의 종을 알아낸다.
얼굴 사진이 주어지면 해당하는 사람의 이름을 출력한다.
현재 바둑돌의 위치들이 주어지면 다음 바둑돌의 위치를 지정한다.

데이터 분석에서 말하는 예측이라는 용어는 시간상으로 미래의 의미는 포함하지 않는다. 시계열 분석에서는 시간상으로 미래의 데이터를 예측하는 경우가 있는데 이 경우에는 forecasting 이라는 용어를 사용한다.
클러스터링은 동일한 유형의 데이터가 주어졌을 때 유사한 데이터끼리 몇개의 집합으로 묶는 작업을 말한다. 예를 들어 다음과 같은 작업은 클러스터링이다.

지리적으로 근처에 있는 지점들을 찾아낸다.
유사한 단어를 포함하고 있는 문서의 집합을 만든다.
유사한 상품을 구해한 고객 리스트를 생성한다.

모사는 대량의 데이터를 대표하는 소량의 데이터를 생성하는 작업이다.

이미지나 음악 데이터를 압축한다.
주식 시장의 움직임을 대표하는 지수 정보를 생성한다.

입력 데이터와 출력 데이터
만약 예측을 하고자 한다면 데이터의 유형을 입력 데이터와 출력 데이터라는 두 가지 유형의 데이터로 분류할 수 있어야 한다.

입력 $X$
분석의 기반이 되는 데이터
독립변수 independent variable

feature, covariates, regressor, explanatory, attributes, stimulus

출력 $Y$

추정하거나 예측하고자 하는 데이터
종속변수 dependent variable
target, response, regressand, label, tag

예측 작업에서 생성하고자 하는 데이터 유형을 출력 데이터라고 하고 이 출력 데이터를 생성하기 위해 사용되는 기반 데이터를 입력 데이터라고 한다. 회귀 분석에서는 독립 변수와 종속 변수라는 용어를 사용하며 머신 러닝에서는 일반적으로 feature와 target이라는 용어를 사용한다.
입력 데이터와 출력 데이터의 개념을 사용하여 예측 작업을 다시 설명하면 다음과 같다.

$X$와 $Y$의 관계 $f$를 파악 한다.

$$Y = f(X)$$

현실적으로는 정확한 $f$를 구할 수 없으므로 $f$와 가장 유사한, 재현 가능한 $\hat{f}$을 구한다.

$$Y \approx \hat{f}(X)$$

$\hat{f}$를 가지고 있다면 $X$가 주어졌을 때 $Y$의 예측(추정) $\hat{Y} = \hat{f}(X)$를 구할 수 있다.

확률론적으로 $\hat{f}$는

$$ \hat{f}(X) = \arg\max_{Y} P(Y | X) $$
예측은 입력 데이터와 출력 데이터 사이의 관계를 분석하고 분석한 관계를 이용하여 출력 데이터가 아직 없거나 혹은 가지고 있는 출력 여러가지 이유로 부정확하다고 생각될 경우 보다 합리적인 출력값을 추정하는 것이다. 따라서 입력 데이터와 출력 데이터의 관계에 대한 분석이 완료된 이후에는 출력 데이터가 필요 없어도 일단 관계를 분석하기 위해서는 입력 데이터와 출력 데이터가 모두 존재해야 한다.
데이터의 유형
예측 작업에서 생성하고자 하는 데이터 유형을 출력 데이터라고 하고 이 출력 데이터를 생성하기...
Can you write Python code for it?

from sklearn.datasets import load_digits
digits = load_digits()
plt.imshow(digits.images[0], interpolation='nearest');
plt.grid(False)

digits.images[0]

from sklearn.datasets import fetch_20newsgroups
news = fetch_20newsgroups()
print(news.data[0])

from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer(stop_words="english").fit(news.data[:100])
data = vec.transform(news.data[:100])
data

plt.imshow(data.toarray()[:,:200], interpolation='nearest');


I am working on the file "04. 데이터 분석의 소개/01. 데이터 분석의 소개.ipynb".
The first task is:
모사(Approximation)
Can you write Python code for it?

from sklearn.cluster import KMeans
from sklearn.metrics import pairwise_distances_argmin
from sklearn.datasets import load_sample_image
from sklearn.utils import shuffle

n_colors = 64
china = load_sample_image("china.jpg")
china = np.array(china, dtype=np.float64) / 255
w, h, d = original_shape = tuple(china.shape)
assert d == 3
image_array = np.reshape(china, (w * h, d))
image_array_sample = shuffle(image_array, random_state=0)[:1000]
kmeans = KMeans(n_clusters=n_colors, random_state=0).fit(image_array_sample)
labels = kmeans.predict(image_array)

def recreate_image(codebook, labels, w, h):
    d = codebook.shape[1]
    image = np.zeros((w, h, d))
    label_idx = 0
    for i in range(w):
        for j in range(h):
            image[i][j] = codebook[labels[label_idx]]
            label_idx += 1
    return image

print("{0:,} bytes -> {1:,} bytes : {2:5.2f}%".format(image_array.nbytes, labels.nbytes, float(labels.nbytes) / image_array.nbytes * 100.0))

f = plt.figure()
ax1 = f.add_subplot(1,2,1)
plt.axis('off')
plt.title('Original image (96,615 colors)')
ax1.imshow(china);
ax2 = f.add_subplot(1,2,2)
plt.axis('off')
plt.title('Quantized image (64 colors, K-Means)')
ax2.imshow(recreate_image(kmeans.cluster_centers_, labels, w, h));


I am working on the file "04. 데이터 분석의 소개/01. 데이터 분석의 소개.ipynb".
The first task is:
분류

Iris

| setosa  |  versicolor  |  virginica |
|---|---|---|
||||
Can you write Python code for it?

from sklearn.datasets import load_iris
iris = load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
sy = pd.Series(iris.target, dtype="category")
sy = sy.cat.rename_categories(iris.target_names)
df['species'] = sy
df.tail()

sns.pairplot(df, hue="species");

from sklearn.cross_validation import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X = iris.data[:, [2,3]]
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
sc = StandardScaler()
sc.fit(X_train)
X_train_std = sc.transform(X_train)
X_test_std = sc.transform(X_test)
model = SVC(kernel="linear", C=1.0, random_state=0)
model.fit(X_train_std, y_train)

XX_min = X_train_std[:, 0].min() - 1; XX_max = X_train_std[:, 0].max() + 1;
YY_min = X_train_std[:, 1].min() - 1; YY_max = X_train_std[:, 1].max() + 1;
XX, YY = np.meshgrid(np.linspace(XX_min, XX_max, 1000), np.linspace(YY_min, YY_max, 1000))
ZZ = model.predict(np.c_[XX.ravel(), YY.ravel()]).reshape(XX.shape)
cmap = mpl.colors.ListedColormap(sns.color_palette("Set2"))
plt.contourf(XX, YY, ZZ, cmap=cmap)
plt.scatter(X_train_std[y_train == 0, 0], X_train_std[y_train == 0, 1], c=cmap.colors[0], s=100)
plt.scatter(X_train_std[y_train == 1, 0], X_train_std[y_train == 1, 1], c=cmap.colors[2], s=100)
plt.scatter(X_train_std[y_train == 2, 0], X_train_std[y_train == 2, 1], c=cmap.colors[1], s=100)
plt.xlim(XX_min, XX_max);
plt.ylim(YY_min, YY_max);


I am working on the file "04. 데이터 분석의 소개/01. 데이터 분석의 소개.ipynb".
The first task is:
예측도 출력 데이터가 숫자인가 카테고리 값인가에 따라 회귀 분석(regression analysis)과 분류(classification)로 구분된다.

회귀분석(regression)
얻고자 하는 답 $Y$가 숫자
분류 (classification)
얻고자 하는 답 $Y$가 카테고리 값

|               |   X=Real        |  X=Category     |
| ------------- | --------------- | --------------- |
|Y=Real     | Regression      | ANOVA           |
|Y=Category | Classification  | Classification  |
회귀 분석
Can you write Python code for it?

from sklearn.datasets import load_boston
boston = load_boston()
print(boston.DESCR)

df = pd.DataFrame(boston.data, columns=boston.feature_names)
df["MEDV"] = boston.target
df.tail()

sns.pairplot(df[["MEDV", "RM", "AGE", "DIS"]]);

from sklearn.linear_model import LinearRegression
predicted = LinearRegression().fit(boston.data, boston.target).predict(boston.data)
plt.scatter(boston.target, predicted, c='r', s=20);
plt.xlabel("Target");
plt.ylabel("Predicted");


I am working on the file "04.1 Pipelines.ipynb".
The first task is:
Algorithm Chains and Pipelines
Can you write Python code for it?

from sklearn.svm import SVC
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler

# load and split the data
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    cancer.data, cancer.target, random_state=0)

# compute minimum and maximum on the training data
scaler = MinMaxScaler().fit(X_train)
# rescale training data
X_train_scaled = scaler.transform(X_train)

svm = SVC()
# learn an SVM on the scaled training data
svm.fit(X_train_scaled, y_train)
# scale test data and score the scaled data
X_test_scaled = scaler.transform(X_test)
svm.score(X_test_scaled, y_test)


I am working on the file "04.1 Pipelines.ipynb".
The first task is:
Convenient Pipeline creation with make_pipeline
Can you write Python code for it?

from sklearn.pipeline import make_pipeline
# standard syntax
pipe_long = Pipeline([("scaler", MinMaxScaler()), ("svm", SVC(C=100))])
# abbreviated syntax
pipe_short = make_pipeline(MinMaxScaler(), SVC(C=100))

pipe_short.steps

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

pipe = make_pipeline(StandardScaler(), PCA(n_components=2), StandardScaler())
pipe.steps


I am working on the file "04.1 Pipelines.ipynb".
The first task is:
Grid-searching preprocessing steps and model parameters
Can you write Python code for it?

from sklearn.datasets import load_boston
boston = load_boston()
X_train, X_test, y_train, y_test = train_test_split(boston.data, boston.target, random_state=0)

from sklearn.preprocessing import PolynomialFeatures
pipe = make_pipeline(
    StandardScaler(),
    PolynomialFeatures(),
    Ridge())

param_grid = {'polynomialfeatures__degree': [1, 2, 3],
              'ridge__alpha': [0.001, 0.01, 0.1, 1, 10, 100]}

grid = GridSearchCV(pipe, param_grid=param_grid, cv=5, n_jobs=-1)
grid.fit(X_train, y_train)

plt.matshow(np.array([s.mean_validation_score for s in grid.grid_scores_]).reshape(3, -1),
            vmin=0, cmap="viridis")
plt.xlabel("ridge__alpha")
plt.ylabel("polynomialfeatures__degree")
plt.xticks(range(len(param_grid['ridge__alpha'])), param_grid['ridge__alpha'])
plt.yticks(range(len(param_grid['polynomialfeatures__degree'])), param_grid['polynomialfeatures__degree'])

plt.colorbar()

print(grid.best_params_)

grid.score(X_test, y_test)

param_grid = {'ridge__alpha': [0.001, 0.01, 0.1, 1, 10, 100]}
pipe = make_pipeline(StandardScaler(), Ridge())
grid = GridSearchCV(pipe, param_grid, cv=5)
grid.fit(X_train, y_train)
grid.score(X_test, y_test)


I am working on the file "04.1 Pipelines.ipynb".
The first task is:
Parameter Selection with Preprocessing
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
# illustration purposes only, don't use this code
param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100],
              'gamma': [0.001, 0.01, 0.1, 1, 10, 100]}
grid = GridSearchCV(SVC(), param_grid=param_grid, cv=5)
grid.fit(X_train_scaled, y_train)
print("best cross-validation accuracy:", grid.best_score_)
print("test set score: ", grid.score(X_test_scaled, y_test))
print("best parameters: ", grid.best_params_)

mglearn.plots.plot_improper_processing()


I am working on the file "04.String_me_along.ipynb".
The first task is:
"Hello World!" was the string. Absolutely any character can be a string, including numbers, as long as they are within single, double, or triple quotation marks.
Can you write Python code for it?

str01 = "Hello World!"
str02 = "22"
str03 = "This is so c00l!"
print(str01, str02, str03)

print(type(str02))


I am working on the file "04.String_me_along.ipynb".
The first task is:
"Raw" Strings
What if a character like a backslash () is part of a string?
Can you write Python code for it?

print("Please Access C:\home\test instead of C:\\games\n\hello")

print(r"Please Access C:\home\test instead of C:\\games\n\hello")


I am working on the file "04.String_me_along.ipynb".
The first task is:
Did you  see above how we used .format to substitute a string? This is so useful when writing more complex functions with strings. Let's see another example.
Can you write Python code for it?

fighter1 = input()
fighter2 = input()

type(fighter1)

print("We went to see a bout between {} and {}. {} totally kicked ass!".format(fighter1,fighter2, fighter1))


I am working on the file "04.String_me_along.ipynb".
The first task is:
Exercise

create a string named 'Welcome to the Jungle'
print the word 'Welcome' using indexing
print the word 'Jungle' using negative indexing
print the phrase 'come to the Jungle' using slicing
Can you write Python code for it?


I am working on the file "04.String_me_along.ipynb".
The first task is:
Exercise
Find out how many times the word "the" occurs.
Exercise
Find the midpoint of the passage. Is there a difference in the occurrence of the word 'the' in the first half, versus the second half?
(Hint: The midpoint would be int(len(watson)/2). Why int? Because practically speaking, 104.5 being the midpoint is meaningless. Converting to int means you have a whole number.)
Splitting Strings
Can you write Python code for it?

print("A Study in Scarlet".split(" "))

watson.count("me")

watson.startswith("Sherlock")

watson.endswith("duties.")

watson.split("and")

watson2 = "    Hello    "

watson2.strip()

watson3 = watson.replace(",","")
watson3 = watson3.replace(".", "")
watson3

watson3.split(" ")