I am working on the file "2.Topics.ipynb".
The first task is:
We decide to use a stemmer and not a lemmatizer (from nltk). The reason is that we want to group together words with the same meaning. For example if one publication contains algorithm and another one contains Algorithmic in this case it would help to map those 2 words to the same. Let's see the output of a stemmer and lemmatizer. Even if our model should be able to capture the similitude among those 2 words, it will help reduce the vocabulary and speed up the training
Can you write Python code for it?

lemmatizer = WordNetLemmatizer()
stemmer = EnglishStemmer()

print("Stemmer",stemmer.stem("Algorithm"), stemmer.stem("Algorithmic"))
print("Lemmatizer",lemmatizer.lemmatize("algorithm"), lemmatizer.lemmatize("Algorithmic"))


I am working on the file "2/0 - Regularization.ipynb".
The first task is:
First, see how normal regression fits
Can you write Python code for it?

# TODO: run standard linear regression, print coefficients and mean squared error


I am working on the file "2/0 - Regularization.ipynb".
The first task is:
Now, model a Ridge regression on the same dataset and compare the results by looking at the mean squared error
Can you write Python code for it?

# TODO: run ridge regression, print coefficients and mean squared error


I am working on the file "2/1 - SVM.ipynb".
The first task is:
Now, we can fit different SVM models using different kernels.
Can you write Python code for it?

C = 1.0  # SVM regularization parameter
svc = svm.SVC(kernel='linear', C=C).fit(X, y)
rbf_svc = svm.SVC(kernel='rbf', gamma=0.7, C=C).fit(X, y)
poly_svc = svm.SVC(kernel='poly', degree=3, C=C).fit(X, y)
lin_svc = svm.LinearSVC(C=C).fit(X, y)


I am working on the file "2/2.ipynb".
The first task is:
2. Generate two signals
Can you write Python code for it?

T = 1000
x1 = generate_signal(T, 1 / 30**2)
x2 = generate_signal(T, 1 / 100**2)

x1 = (x1 - np.mean(x1)) / np.std(x1)
x2 = (x2 - np.mean(x2)) / np.std(x2)
np.mean(x1), np.mean(x2), np.std(x1), np.std(x2)

plt.plot(x1, label=r'$\tau$ = 30')
plt.plot(x2, label=r'$\tau$ = 100')
plt.xlabel('Step')
plt.ylabel('Signal Value')
plt.legend()


I am working on the file "2/2.ipynb".
The first task is:
2. Receptive field input
Can you write Python code for it?

X = np.zeros((100, 9900))

for i in range(9900):
    X[:, i] = signal[i:i+100]

plt.plot(X[:, :3])

plt.plot(X[0])
plt.xlabel('Time')
plt.ylabel('Activation of first visual cell')


I am working on the file "2/2.ipynb".
The first task is:
3. SFA
Can you write Python code for it?

C = np.cov(X)
C

X_dot = np.diff(X)
C_dot = np.dot(X_dot, X_dot.T) / (T - 1)
C_dot

eigenvalues, eigenvectors = eig(C_dot, C)


I am working on the file "2/2.ipynb".
The first task is:
4. Sorting
Can you write Python code for it?

indices_for_sorting = np.argsort(eigenvalues)
sorted_eigenvectors = eigenvectors.T[indices_for_sorting].T

for i in range(5):
    plt.plot(eigenvectors[:, i], 'o', label='Eigenvector {}'.format(i))
plt.xlabel('Component of Eigenvector')
plt.ylabel('Value')
plt.legend()


I am working on the file "2/2.ipynb".
The first task is:
Exercise 1
1. Function that generates random signal
Can you write Python code for it?

import helper

def generate_signal(num_steps, epsilon):
    return helper.gaussian_spectrum_1D(np.random.rand(num_steps), epsilon)

plt.plot(generate_signal(1000, 1.))
plt.plot(generate_signal(1000, 0.0001))


I am working on the file "2/2.ipynb".
The first task is:
The signal with higher $\tau$ is smoother, i.e. it contains less high-frequency components.
3. Linear SFA
Can you write Python code for it?

X = np.vstack((x1, x2))
X.shape

C = np.cov(X)
C

X_dot = np.diff(X)
C_dot = np.dot(X_dot, X_dot.T) / (T - 1)
C_dot

from scipy.linalg import eig

eigenvalues, eigenvectors = eig(C_dot, C)
for i in range(len(eigenvalues)):
    eigenvalue = eigenvalues[i]
    normalized_eigenvector = eigenvectors[:, i] / np.linalg.norm(eigenvectors[:, i])
    print('Eigenvalue: {} - Normalized Eigenvector: {}'.format(eigenvalue, normalized_eigenvector))


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
*------------------------
Problem 3: (20 points total)
Correlation:  Men versus women
Let look more closely at the relationship between the pieces of data we have.

(2 points) Make a scatter plot of men versus women and their mean rating for every movie.
(2 points) Make a scatter plot of men versus women and their mean rating for movies rated more than 200 times.
(6 points) Compute the correlation coefficent between the ratings of men and women.
What do you observe?
Are the ratings similiar or not? Support your answer with data!

(10 points) Conjecture under what circumstances the rating given by one gender can be used to predict the rating given by the other gender.
For example, are men and women more similar when they are younger or older?
Be sure to come up with your own conjectures and support them with data!
Can you write Python code for it?

%matplotlib inline

import matplotlib
import matplotlib.pyplot as plt

# among total6040 users, how many female? how may male?
users.groupby('gender').size()

#among total 100209 rating records, how many was made by female? how many was made by male?
data.groupby('gender').size()


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
As before, we consider the percent ratings of 5 (per title) given by each age group.
Can you write Python code for it?

comedy_fives_data = comedy_data[comedy_data['rating']==5]

comedy_gender_fives = comedy_fives_data.pivot_table('rating', index='title', columns='gender', aggfunc='count')
comedy_gender_totals = comedy_data.pivot_table('rating', index='title', columns='gender', aggfunc='count')

comedy_gender_percents = comedy_gender_fives / comedy_gender_totals
comedy_gender_percents = comedy_gender_percents.ix[active_titles]

comedy_gender_percents.M.corr(comedy_gender_percents.F)

# Train a linear model to examine predictability

# Can't have any NaN values for linear regression.
comedy_gender_percents = comedy_gender_percents.dropna()

# Select out our predictor columns and our response columns
X = comedy_gender_percents.ix[:,['M']]
y = comedy_gender_percents.ix[:,['F']]

# Split the data into training data and testing data
X_train,X_test,y_train,y_test = cross_validation.train_test_split(X,
                                                                  y,
                                                                  test_size=0.8)
# Run the solver
reg = linear_model.LinearRegression(fit_intercept=True)
reg.fit(X_train,y_train)

# Plot the data and the model
plotX = np.linspace(0,1,100)
plotY = reg.predict(np.matrix(plotX).T)
plt.plot(X_train,y_train,'o', color='#FA5744')
plt.plot(X_test,y_test,'o', color='#00b27f')
plt.plot(plotX,plotY,'-', color='#5898f1')
plt.title('Percent 5 Rating of Comedies')
plt.ylabel('Female Average')
plt.xlabel('Male Average')

# Compute the slope and intercept of the linear model
print reg.intercept_
# Beta_1
print reg.coef_

# Compute testing and training error.
print 'training error'
print metrics.mean_squared_error(y_train,reg.predict(X_train))
print 'testing error'
print metrics.mean_squared_error(y_test,reg.predict(X_test))


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
Compute some Summary Statistics for the data
Can you write Python code for it?

#check statistics of data
data[['rating','age']].describe()


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
Conclusion: False.  Both correlation coefficients actually went down by about 0.1: this change in the largest significant digit should be meaningful.  That is, during late night/early morning hours, even the average behavior between the two genders is less correlated, and thus they are not behaving similarly
Conjecture 2.) Genders agree on what is "funny"!
Can you write Python code for it?

# Determine which movies have "Comedy" listed within its genres.
genres = data['genres']
all_genres = [string.split('|') for string in genres]
comedy_truth =  [ 'Comedy' in genres for genres in all_genres]
comedy_series = pd.DataFrame(comedy_truth, index=data.index)
data['comedy'] = comedy_series

comedy_data = data.ix[data['comedy'] == True]

# Determine comedies with at least 100 ratings
comedy_ratings_by_title = comedy_data.groupby('title').size()
comedy_active_titles = comedy_ratings_by_title.index[comedy_ratings_by_title > 100]

# Extract these titles
comedy_mean_ratings = comedy_data.pivot_table('rating',index = 'title',columns = 'gender',aggfunc = 'mean')
active_comedy_mean_ratings = comedy_mean_ratings.ix[comedy_active_titles]
active_comedy_mean_ratings[:10]

# Compute correlation between average men's and women's ratings.
active_comedy_mean_ratings.M.corr(active_comedy_mean_ratings.F)

from sklearn import cross_validation, linear_model, feature_selection, metrics

# Train a linear model to examine predictability

# Can't have any NaN values for linear regression.
active_comedy_mean_ratings = active_comedy_mean_ratings.dropna()

# Select out our predictor columns and our response columns
X = active_comedy_mean_ratings.ix[:,['M']]
y = active_comedy_mean_ratings.ix[:,['F']]

# Split the data into training data and testing data
X_train,X_test,y_train,y_test = cross_validation.train_test_split(X,
                                                                  y,
                                                                  test_size=0.8)
# Run the solver
reg = linear_model.LinearRegression(fit_intercept=True)
reg.fit(X_train,y_train)

# Plot the data and the model
plotX = np.linspace(0,5,100)
plotY = reg.predict(np.matrix(plotX).T)
plt.plot(X_train,y_train,'o', color='#FA5744')
plt.plot(X_test,y_test,'o', color='#00b27f')
plt.plot(plotX,plotY,'-', color='#5898f1')
plt.title('Average Rating of Comedies')
plt.ylabel('Female Average')
plt.xlabel('Male Average')

# Compute the slope and intercept of...


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
Conclusion: False.  If the conjecture were true, we would expect to see noticeably higher average ratings at very large and very small hours.  This is clearly not the case.
Problem 2 (20 points total): Expand our investigation to histograms
An obvious issue with any inferences drawn from Problem 1 is that we did not consider how many times a movie was rated.
* (2 points) Plot a histogram of the ratings of all movies.
* (2 points) Plot a histogram of the number of ratings each movie recieved.
* (2 points) Plot a histogram of the average rating for each movie.
* (6 points) Plot a histogram of the average rating for movies which are rated more than 100 times.
    * What do you observe about the tails of the histogram where you use all the movies versus the one where you only use movies rated more than 100 times?
    * Which highly rated movies would you trust are actually good?  Those rated more than 100 times or those rated less than 100 times?
* (8 points) Make some conjectures about the distribution of ratings? Support your answers with data!
    *  For example, what age range do you think has more extreme ratings?  Do you think children are more or less likely to rate a movie 1 or 5?
    *  Be sure to come up with your own conjectures and support them with data!
Plot a histogram of the ratings of all movies.
Can you write Python code for it?

#Plot a histogram of the ratings of all movies.
Rating_all=data.pivot_table('title',index='rating',aggfunc='count')
Rating_all.plot(kind='bar', color='#FA5744')
plt.title('Histogram of all ratings')
plt.ylabel('Total number')


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
Conclusion: False.  In fact, an older person is more likely than a younger person to give a rating of 5, and less likely to give a rating of 1.  More details presented in the report.
Conjecture 2.) Tired people are more easy to please
Can you write Python code for it?

import time
timestamps = data['timestamp']

# Time stamps are reported in seconds since epoch.  Convert these values to local time, and extract the hour.
hour = [time.localtime(stamp).tm_hour for stamp in timestamps.values]
hour_series = pd.DataFrame(hour, index=data.index)

# Append the hour each rating was reported to the data set.
data['hour'] = hour_series

# Use a pivot table to determine the average overall rating by hour.
avg_by_hour = data.pivot_table('rating',index = 'hour', aggfunc = np.mean)
avg_by_hour

wee_hours_data = data[np.logical_or(data['hour']>= 22,data['hour']<=5)]
wee_hours_5 = wee_hours_data[wee_hours_data['rating']==5]
wee_hours_1 = wee_hours_data[wee_hours_data['rating']==1]

total_5 = data[data['rating']==5]
total_1 = data[data['rating']==1]

wee_hours_5_percent = float(len(wee_hours_5))/len(wee_hours_data)
wee_hours_1_percent = float(len(wee_hours_1))/len(wee_hours_data)

total_5_percent = float(len(total_5))/len(data)
total_1_percent = float(len(total_1))/len(data)

compdat = {'Percent Ratings 5':[wee_hours_5_percent , total_5_percent],
                          'Percent Raings 1':[wee_hours_1_percent, total_1_percent]}
comp = pd.DataFrame(compdat, columns=['Percent Ratings 5','Percent Raings 1'], index=['Wee Hours','Total'])
comp


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
Conjecture 1.) People rate more similarly when they are tired.
Can you write Python code for it?

import time

# Convert time stamps to local time and extract the hour.
timestamps = data['timestamp']
hour = [time.localtime(stamp).tm_hour for stamp in timestamps.values]
hour_series = pd.DataFrame(hour, index=data.index)
data['hour'] = hour_series

# Isolate data for ratings submitted between 10PM and 5AM local time
wee_hours_data = data[np.logical_or(data['hour']>= 22,data['hour']<=5)]

# Determine the average ratings per title by gender during these late-night hours.
wee_hours_mean_ratings = wee_hours_data.pivot_table('rating', index = 'title', columns = 'gender', aggfunc = np.mean)

wee_hours_mean_ratings[:5]

#Calculate the correlation coefficient.
print 'Correlation coefficient between averege male and female ratings between 10PM and 5AM: {0}'.format(
    wee_hours_mean_ratings.M.corr(wee_hours_mean_ratings.F))

# We already know that men and women tend to disagree on movies with lower total ratings. Segment from the late-night data those with
# high total numbers of ratings.
wee_hours_over_200_mean_ratings = wee_hours_mean_ratings.ix[active_titles]
wee_hours_over_200_mean_ratings[:5]

#Compute Correlation Coefficient
'Correlation coefficient between averege male and female ratings between 10PM and 5AM, among movies with at least 200 total ratings: {0}'.format(wee_hours_over_200_mean_ratings.M.corr(wee_hours_over_200_mean_ratings.F))


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
Conjecture 2: The distribution of all ratings of older movies is less normally-distributed than that of newer movies.  Newer movies are likely being watched by audiences looking to be entertained.  Those watching older movies (i.e. movies made before 1960) likely have some nostalgia tied up in these older films.
Can you write Python code for it?

# Extract the year from the title

def extract(string, start='(', stop=')'):
        while string.index(stop) - (string.index(start)+1)!= 4:
            string = string[:string.index(start)] + string[string.index(stop)+1:]
        return string[string.index(start)+1:string.index(stop)]

titles = data['title']
year = [int(extract(title)) for title in titles]
year_series = pd.DataFrame(year, index=data.index)
data['year'] = year_series

data[:5]

year_array = list(set(data['year'].values))
average_year = int(float(sum(year_array))/len(year_array))
average_year

old_data = data[data['year']<= average_year]
old_rating_all=old_data.pivot_table('title',index='rating',aggfunc='count')
old_rating_all.plot(kind='bar', color='#FA5744')
plt.title('Histogram of all of early movies')
plt.ylabel('Total number')

newer_data = data[data['year']>= average_year]
new_rating_all=newer_data.pivot_table('title',index='rating',aggfunc='count')
new_rating_all.plot(kind='bar', color='#00b27f')
plt.title('Histogram of all of newer movies')
plt.ylabel('Total number')


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
Conjecture 3.) Men and Women rate similarly on highly-watched movies made most recently.
Can you write Python code for it?

# Extract those movies made in the last 10 years of those available.
newest_data = data[data['year']>= 1990]

# Use a pivot table to compute mean ratings per title by gender
newest_mean_ratings = newest_data.pivot_table('rating',index = 'title',columns = 'gender',aggfunc = 'mean')

over_200_newest_mean_ratings = newest_mean_ratings.ix[active_titles]

# Scatter this data.
plt.scatter(over_200_newest_mean_ratings['M'], over_200_newest_mean_ratings['F'])
plt.title('Average Ratings by Movie, after 1990')
plt.ylabel('Average female rating')
plt.xlabel('Average male rating')

print 'correlation coefficient between percent average males and females per title: {0}'.format(
    over_200_newest_mean_ratings.M.corr(over_200_newest_mean_ratings.F))

# Compute the percent 5 ratings by males and females for movies made in each year of this 10 year window.

new_fives_data = newest_data[newest_data['rating']==5]

year_gender_fives = new_fives_data.pivot_table('rating', index='title', columns='gender', aggfunc='count')
year_gender_totals = newest_data.pivot_table('rating', index='title', columns='gender', aggfunc='count')

year_gender_percents = year_gender_fives / year_gender_totals
year_gender_percents = year_gender_percents.ix[active_titles]

# Scatter this data.
plt.scatter(year_gender_percents['M'], year_gender_percents['F'])
plt.title('Percent ratings 5 by Movie, after 1990')
plt.ylabel('Female percent 5')
plt.xlabel('Male percent 5')

print 'Correlation coefficient between percent ratings of 5 between males and females by year of movie release: {0}'.format(
    year_gender_percents.M.corr(year_gender_percents.F))


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
From the chart, we can see that even though scientist, retired, and clerical/admin have the highest average rating for comedies, they also have low numbers of ratings, 7771, 4340, and 11870 respectively. In contrast, college/grad student, other, and executive/managerial have significantly more ratings (at least triple) 48672, 46500, and 35784 respectively. Although scientist has the highest average rating, it might be better to recommend comedies to students because while they have a lower average rating, they have almost seven times more ratings than scientists.
Can you write Python code for it?

job_total_ratings = data.pivot_table('title', index='occupation', aggfunc='count')
job_total_ratings.rename(index={0:'other', 1:'academic/educator',2: 'artist',3: 'clerical/admin',4: 'college/grad student',
                                  5: 'customer service',6: 'doctor/health care',7:'executive/managerial',8:'farmer',
                                  9: 'homemaker',10: 'K-12 student',11: 'lawyer',12 :'programmer',13: 'retired',
                                  14:'sales/marketing',15:'scientist',16: 'self-employed',17: 'technician/engineer',
                                  18: 'tradesman/craftsman',19 :'unemployed',20: 'writer'}, inplace=True)

job_percent_comedy_ratings = job_total_comedy_ratings / job_total_ratings
job_percent_comedy_ratings.plot(kind='bar', color='#E50E14')
plt.title('Percent Comedies vs. Occupation')
plt.xlabel('Occupation')
plt.ylabel('Percent of all movies watched')


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
How many movies have an average rating over 4.5 among men? How about women?
Can you write Python code for it?

# Use a pivot table to compute mean ratings per title, stratified by gender.
mean_ratings = data.pivot_table('rating',index = 'title',columns = 'gender',aggfunc = 'mean')

#Determine those title ranked high among females.
top_female_titles = mean_ratings.index[mean_ratings['F'] >= 4.5]

# Extract those titles
top_female_movies = mean_ratings.ix[top_female_titles]
print 'Total movies with an average ranking of (at least) 4.5 among women:'
print len(top_female_movies)
print
print 'Examples (average rankings):'
print top_female_movies[:5]

mean_ratings = data.pivot_table('rating',index = 'title',columns = 'gender',aggfunc = 'mean')

# Determine which titles had high average ratings among men
top_male_titles = mean_ratings.index[mean_ratings['M'] >= 4.5]

# Extract those titles
top_male_movies = mean_ratings.ix[top_male_titles]
print 'Total movies with an average ranking of (at least) 4.5 among men:'
print len(top_male_movies)
print
print 'Examples (average rankings):'
print top_male_movies[:5]


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
How many movies have an median rating over 4.5 among men over age 30? How about women over age 30?
Can you write Python code for it?

# Restrict data to those with raters aged over 30
data_over30 = data.ix[data['age']>30]

# Use a pivot table to compute the median ratings by title on this restricted data
median_ratings = data_over30.pivot_table('rating',index = 'title',columns = ['gender'],aggfunc = 'median')

# Determine which movies had a high median among men and extract those titles
top_male_median_titles = median_ratings.index[median_ratings['M'] >= 4.5]
top_male_median_movies = median_ratings.ix[top_male_median_titles]

print 'Total movies with an median ranking of (at least) 4.5 among men over 30:'
print len(top_male_median_movies)
print
print 'Examples, median scores among people over 30:'
print top_male_median_movies[:5]

# Determine which movies had a high median among men and extract those titles
top_female_median_titles = median_ratings.index[median_ratings['F'] >= 4.5]
top_female_median_movies = median_ratings.ix[top_female_median_titles]

print 'Total movies with an median ranking of (at least) 4.5 among women over 30:'
print len(top_female_median_movies)
print
print 'Examples, median scores among people over 30:'
print top_female_median_movies[:5]


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
Make a scatter plot of men versus women and their mean rating for every movie.
Can you write Python code for it?

# Use a pivot table to compute mean ratings per title by gender
mean_ratings = data.pivot_table('rating',index = 'title',columns = 'gender',aggfunc = 'mean')

# Scatter this data.
plt.scatter(mean_ratings['M'], mean_ratings['F'])
plt.title('Average Ratings by Movie')
plt.ylabel('Average female rating')
plt.xlabel('Average male rating')


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
Make some conjectures about how easy various groups are to please? Support your answers with data!
Conjecture 1.) The older a person gets, the more difficult they are to please.
Can you write Python code for it?

# Compute average rating by age group
age_avg_ratings = data.pivot_table('rating', index = 'age',aggfunc = np.mean)
age_avg_ratings

# Compute weighted average by weighting each rating by the total number of ratings that individual submits
avg_by_user = data.pivot_table('rating',index = ['age','user_id'], aggfunc = [ len , np.mean])
avg_by_user[:10]

avg_ratings = np.mean(avg_by_user['len'])

avg_by_user['weight'] = avg_by_user['len']/avg_ratings
avg_by_user['weighted_mean'] = avg_by_user['mean']*avg_by_user['weight']

age_avg_weighted_ratings = avg_by_user.pivot_table('weighted_mean', index = avg_by_user.index.droplevel(1), aggfunc = np.mean)
age_avg_weighted_ratings

# Compute average age per rating
avg_age_ratings = data.pivot_table('age', index = 'rating',aggfunc = np.mean)
avg_age_ratings

age_counts = data.pivot_table('title', index='age', columns='rating', aggfunc='count')
age_counts.rename(index={1: 'Under 18', 18: '18-24', 25: '25-34', 35: '35-44',
                                        45: '45-49', 50: '50-55', 56: '56+'}, inplace=True)
print 'Frequency of Age Groups Ratings'
print age_counts

#normalize
age_counts_norm = age_counts.div(age_counts.sum(1).astype(float), axis=0)
age_counts_norm

# plot percentage of each rate from each age group
age_counts_norm.plot(ylim=[0,0.4],kind='bar', color=['yellow','#E50E14','#ec971f','#00b27f','#5898f1'],title = "Percent of Ratings By Age").legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
Make some conjectures about the distribution of ratings? Support your answers with data!
Conjecture 1: Movies with fewer total ratings have a distribution of all ratings which is closer to a uniform distribution due to a larger percent of low-end extreme ratings.
Can you write Python code for it?

# Select the movies with less than half the average number of total ratings.
rating_by_title = data.groupby('title').size()
inactive_titles = rating_by_title.index[rating_by_title <= average_total_ratings/2]

inactive = [title in inactive_titles.values for title in data['title']]
inactive_series = pd.DataFrame(inactive, index = data.index)
data['Inactive'] = inactive_series

inactive_data = data[data['Inactive']]

inactive_rating_all=inactive_data.pivot_table('title',index='rating',aggfunc='count')
inactive_rating_all.plot(kind='bar', color='blue')
plt.title('Histogram of ratings of movies \n with less than half the average number of ratings')
plt.ylabel('Total number')

# Select the movies with less more than twice the average number of total ratings.
rating_by_title = data.groupby('title').size()
wayactive_titles = rating_by_title.index[rating_by_title >= average_total_ratings*2]

wayactive = [title in wayactive_titles.values for title in data['title']]
wayactive_series = pd.DataFrame(wayactive, index = data.index)
data['wayactive'] = wayactive_series

wayactive_data = data[data['wayactive']]

wayactive_rating_all=wayactive_data.pivot_table('title',index='rating',aggfunc='count')
wayactive_rating_all.plot(kind='bar', color='blue')
plt.title('Histogram of ratings of movies with \n more than twice the average number of ratings')
plt.ylabel('Total number')


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
This data seems to be somewhat misleading.  Based on the high correlation values, it seems that the ratings between men and women are similar, especially among movies watched more than 200 times.  However, this is the correlation between the MEAN rating per title between men and women.  What this is saying is that ON AVERAGE,  men and women rate movies similarly.  This doesn't indicate that the ratings themselves are actually similar!  For example, there could be a movie in which both men and women have an average rating of 3, but women rate it as either a 1 or a 5 and all men rate it as 3.  We need to explore the data more to understand if the ratings between men and women are actually similar.
For example, rather than consider mean rating, let's consider the percentage of ratings that are a 5.  That is, for each title we compute the total number of 5 ratings given by men and divide by the total number of ratings given by men.  We then determine the correlation in this data.
Can you write Python code for it?

fives_data = data[data['rating']==5]

five_ratings = fives_data.pivot_table('rating', index = 'title', columns = 'gender', aggfunc = 'count')
total_ratings = data.pivot_table('rating',index ='title',columns = 'gender',aggfunc = 'count')

fives_percent = pd.DataFrame(index = five_ratings.index)
fives_percent['M'] = five_ratings['M']/total_ratings['M']
fives_percent['F'] = five_ratings['F']/total_ratings['F']

print 'correlation coefficient between percent ratings of 5 by male and female per title: {0}'.format(fives_percent.M.corr(fives_percent.F))

over_200_fives_percent = fives_percent.ix[active_titles]

print 'correlation coefficient between percent ratings of 5 by male and female among titles with more than 200 ratings: {0}'.format(over_200_fives_percent.M.corr(over_200_fives_percent.F))

plt.scatter(over_200_fives_percent['M'], over_200_fives_percent['F'])
plt.title('Percent ratings 5 by Movie, \n Among movies rated more than 200 times')
plt.ylabel('Percent 5, female')
plt.xlabel('Percent 5, male')


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
We see that, for movies made in the 90's, the correlation coefficient for the percent of ratings given as 5 between males and females is
0.9023! And the correlation coefficient for the percent of ratings given as one or two (based on the year the movie was released in the 90's) is 0.905
Can you write Python code for it?

new_low_data = newest_data[newest_data['rating']<=2]

year_low = new_low_data.pivot_table('rating', index='title', columns='gender', aggfunc='count')
year_totals = newest_data.pivot_table('rating', index='title', columns='gender', aggfunc='count')

year_low_percents = year_low / year_totals
year_low_percents = year_low_percents.ix[active_titles]

plt.scatter(year_low_percents['M'], year_low_percents['F'])
plt.title('Percent ratings 1 or 2 by Movie, after 1990')
plt.ylabel('Female percent 1 or 2')
plt.xlabel('Male percent 1 or 2')

print 'Correlation between percent ratings of 1 or 2 between males and females on movies with the same release year: {0}'.format(
   year_low_percents.M.corr(year_low_percents.F))

# Train a linear model to predict average ratings between genders.

from sklearn import cross_validation, linear_model, feature_selection, metrics

# Can't have any NaN values for linear regression.
over_200_newest_mean_ratings = over_200_newest_mean_ratings.dropna()

# Select out our predictor columns and our response columns
X = over_200_newest_mean_ratings.ix[:,['M']]
y = over_200_newest_mean_ratings.ix[:,['F']]

# Split the data into training data and testing data
X_train,X_test,y_train,y_test = cross_validation.train_test_split(X,
                                                                  y,
                                                                  test_size=0.8)

# Run the solver
reg = linear_model.LinearRegression(fit_intercept=True)
reg.fit(X_train,y_train)

# Compute the slope and intercept of the linear model
print reg.intercept_
# Beta_1
print reg.coef_

# Plot the data and the model
plotX = np.linspace(0,5,100)
plotY = reg.predict(np.matrix(plotX).T)
plt.plot(X_train,y_train,'ro')
plt.plot(X_test,y_test,'go')
plt.plot(plotX,plotY,'b-')
plt.title('Average Ratings by Movie, after 1990')
plt.ylabel('Average female rating')
plt.xlabel('Average male rating')

# Compute testing and training error.
print 'training error'
print...


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
What age group watches the most adventure movies?
Can you write Python code for it?

# histogram of age group vs adventure genre
genres = data['genres']
all_genres = [string.split('|') for string in genres]
adventure_truth =  [ 'adventure' in genres for genres in all_genres]
adventure_series = pd.DataFrame(drama_truth, index=data.index)
data['adventure'] = adventure_series

adventure_data = data.ix[data['adventure'] == True]

adventure_ratings_per_age = adventure_data.pivot_table('title', index='age', columns='gender', aggfunc='count')
adventure_ratings_per_age.rename(index={1: 'Under 18', 18: '18-24', 25: '25-34', 35: '35-44',
                                        45: '45-49', 50: '50-55', 56: '56+'}, inplace=True)
adventure_ratings_per_age.plot(kind='bar', color=['#E50E14','#5898f1'])
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.title('Adventure Movies Rated per Age Group')
plt.xlabel('Age Group')
plt.ylabel('Count')

total_per_age = data.pivot_table('title', index='age', columns='gender', aggfunc='count')
total_per_age.rename(index={1: 'Under 18', 18: '18-24', 25: '25-34', 35: '35-44',
                                        45: '45-49', 50: '50-55', 56: '56+'}, inplace=True)
adventure_percent = adventure_ratings_per_age / total_per_age
adventure_percent.plot(kind='bar', color=['#E50E14','#5898f1'])
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.title('Percent Adventure Movies per Age Group')
plt.xlabel('Age Group')
plt.ylabel('Percent')

print adventure_ratings_per_age
print total_per_age
print adventure_percent


I am working on the file "2/CaseStudy2.ipynb".
The first task is:
What are the ten most popular movies?
    * Choose what you consider to be a reasonable defintion of "popular".
    * Be perpared to defend this choice.
We propose the following definition of a "Popular" movie:
* Above-average total number of ratings
* Above-average rating among women (i.e. the movie's average rating among women is above the average of ALL ratings given by women)
* Above-average rating among men ((i.e. the movie's average rating among men is above the average of ALL ratings given by men)
Among these "popular" movies we determine the top 10 MOST popular by using highest average rating overall.
Can you write Python code for it?

# Determine the overall total ratings and mean ratings per title
popularity_test = data.pivot_table('rating',index = 'title', aggfunc = [len, np.mean])

# Determine the mean ratings per title by gender
gender_popularity_test = data.pivot_table('rating',index = 'title', columns = 'gender', aggfunc = np.mean)

popularity_test[:5]

gender_popularity_test[:5]

# Calculate total number of ratings for each title
ratings_by_title = data.groupby('title').size()

# Determine the average number of total ratings per title
average_total_ratings = sum(ratings_by_title)/len(ratings_by_title)

# Determine which titles had above average total ratings and isolate those titles.
high_total_titles = popularity_test.index[popularity_test['len'] >= average_total_ratings]
high_total = popularity_test.ix[high_total_titles]

high_total[:5]

# Determine the average of ALL ratings given by men and by women.
gender_average_ratings = data.pivot_table('rating', index = 'gender',aggfunc = np.mean)
gender_average_ratings

# Determine the titles with above average female ratings and isolate those titles among the movies with above average total ratings.
high_female_titles = gender_popularity_test.index[gender_popularity_test['F'] >= gender_average_ratings['F']]
high_total_female = high_total.ix[high_female_titles]

# Among the above isolated titles, determine those with above average male ratings and isolate those titles.
high_male_titles = gender_popularity_test.index[gender_popularity_test['M'] >= gender_average_ratings['M']]
high_total_female_male = high_total_female.ix[high_male_titles]

# Determine the popular movies, given the definition above.
from numpy import nan as NA
popular_movies = high_total_female_male.dropna(how = 'all')
popular_movies[:5]

# Given the popluar movies, determine the 10 most popular.
most_popular_movies = popular_movies.sort_values(by='mean',ascending = False)
most_popular_movies[:10]


I am working on the file "2/Decision Trees and Random Forests.ipynb".
The first task is:
As you can probably tell, the problem with this strategy is that while it may work in cases where you are an expert, it doesn't help you figure out what to do when you aren't an expert. And there will be data science problems that you will need to solve where you no know nothing about the problem, and there aren't any experts.
Still, the above strategy has some positive factors:
+ It's fast to evaluate (just like linear models, unlike SVMs)
+ It's nonlinear (which means it's sometimes more flexible than linear models)
+ It's very easy to read
Don't discount the latter criteria - when your model is getting 98% accuracy and you want to get to 99% accuracy (to pass your class, to do your job, or to win a million dollars), you're going to want to improve results. What better way to improve your accuracy than to look at the errors that you made and figure out how to fix them? It's very hard to understand why you are making errors in many models, especially SVMs and neural nets, and this model is much easier.
How do we construct decision trees?
Ok, how can we construct these decision trees in cases when we aren't experts and know the correct splits beforehand? It turns out that there are a lot of different ways, most notably (today) CART (Classification and Regression Trees) and C5.0 (some people refer to this as ID3's successor). Let's look at a simplified version of the CART algorithm:
Suppose we want to solve the classification problem of deciding whether animals are good or bad (regression is similar) and we have a labeled data set of the form
$$S = {(\vec{x}, y)i}{i = 1}^n$$
Where $\vec{x}$ is written with an arrow to emphasize that it is a vector of features, $x_i[0]$, $x_i[1]$, ...etc.
We're going to construct the tree in a top-down approach - that is, we're going to start at the top, create a node, and then walk downwards, creating more nodes as we go. Suppose that we want to branch on feature $k$, and there are two possible values for feature $k$ (which is a label, rather than a continuous variable), cat...
Can you write Python code for it?


I am working on the file "2/Decision Trees and Random Forests.ipynb".
The first task is:
Let's take a look at an example:
Let's load up a sample dataset and walk through using a decision tree on it:
Can you write Python code for it?

from sklearn.datasets import load_iris
from sklearn import tree
import numpy as np

# load data
iris = load_iris()

# just to see what's going on
mask = np.random.randint(len(iris['data']), size=10)
print("Input:")
print(iris['data'][mask, :])
print("Output:")
print(iris['target'][mask])


I am working on the file "2/Decision Trees and Random Forests.ipynb".
The first task is:
So we're predicting class labels given 4 features. Let's do this
Can you write Python code for it?

clf = tree.DecisionTreeClassifier()
clf = clf.fit(iris.data, iris.target)


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Build the network
For the neural network, you'll build each layer into a function.  Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function.  This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.
If you're finding it hard to dedicate enough time for this course a week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use TensorFlow Layers or TensorFlow Layers (contrib) to build each layer, except "Convolutional & Max Pooling" layer.  TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.
If you would like to get the most of this course, try to solve all the problems without TF Layers.  Let's begin!
Input
The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions
* Implement neural_net_image_input
 * Return a TF Placeholder
 * Set the shape using image_shape with batch size set to None.
 * Name the TensorFlow placeholder "x" using the TensorFlow name parameter in the TF Placeholder.
* Implement neural_net_label_input
 * Return a TF Placeholder
 * Set the shape using n_classes with batch size set to None.
 * Name the TensorFlow placeholder "y" using the TensorFlow name parameter in the TF Placeholder.
* Implement neural_net_keep_prob_input
 * Return a TF Placeholder for dropout keep probability.
 * Name the TensorFlow placeholder "keep_prob" using the TensorFlow name parameter in the TF Placeholder.
These names will be used at the end of the project to load your saved model.
Note: None for shapes in TensorFlow allow for a dynamic size.
Can you write Python code for it?

import tensorflow as tf

def neural_net_image_input(image_shape):
    """
    Return a Tensor for a bach of image input
    : image_shape: Shape of the images
    : return: Tensor for image input.
    """
    # TODO: Implement Function
    #print(image_shape)
    x = tf.placeholder("float", shape=[None, image_shape[0], image_shape[1], image_shape[2]], name='x')
    #print(x.name)
    return x

def neural_net_label_input(n_classes):
    """
    Return a Tensor for a batch of label input
    : n_classes: Number of classes
    : return: Tensor for label input.
    """
    # TODO: Implement Function
    #print(n_classes)
    x = tf.placeholder("float", shape=[None, n_classes],name='y')
    return x

def neural_net_keep_prob_input():
    """
    Return a Tensor for keep probability
    : return: Tensor for keep probability.
    """
    # TODO: Implement Function
    x = tf.placeholder("float",name='keep_prob')
    return x

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Check Point
This is your first checkpoint.  If you ever decide to come back to this notebook or have to restart the notebook, you can start from here.  The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Checkpoint
The model has been saved to disk.
Test Model
Test your model against the test dataset.  This will be your final accuracy. You should have an accuracy greater than 50%. If you don't, keep tweaking the model architecture and parameters.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

# Set batch size if not already set
try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    """
    Test the saved model against the test dataset
    """

    test_features, test_labels = pickle.load(open('preprocess_training.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        # Load model
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        # Get Tensors from loaded model
        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')

        # Get accuracy in batches for memory limitations
        test_batch_acc_total = 0
        test_batch_count = 0

        for train_feature_batch, train_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: train_feature_batch, loaded_y: train_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        # Print Random Samples
        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y:...


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Convolution and Max Pooling Layer
Convolution layers have a lot of success with images. For this code cell, you should implement the function conv2d_maxpool to apply convolution then max pooling:
* Create the weight and bias using conv_ksize, conv_num_outputs and the shape of x_tensor.
* Apply a convolution to x_tensor using weight and conv_strides.
 * We recommend you use same padding, but you're welcome to use any padding.
* Add bias
* Add a nonlinear activation to the convolution.
* Apply Max Pooling using pool_ksize and pool_strides.
 * We recommend you use same padding, but you're welcome to use any padding.
Note: You can't use TensorFlow Layers or TensorFlow Layers (contrib) for this layer.  You're free to use any TensorFlow package for all the other layers.
Can you write Python code for it?

def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    # TODO: Implement Function
    #print(x_tensor.shape)
    x, x_h, x_w, x_depth = x_tensor.shape
    #print(conv_num_outputs)
    #print(conv_ksize)
    conv_h,conv_w = conv_ksize
    #print(conv_strides)
    s_h,s_w = conv_strides
    #print(pool_ksize)
    pool_h, pool_w = pool_ksize
    #print(pool_strides)
    pool_h_s,pool_w_s = pool_strides

    #weights

    fw = tf.Variable(tf.truncated_normal(shape=(int(conv_h), int(conv_w), int(x_depth), int(conv_num_outputs)),stddev=0.01, mean = 0))
    #bias
    fb = tf.Variable(tf.zeros(conv_num_outputs))
    padding = 'VALID'
    conv = tf.nn.conv2d(x_tensor, fw, strides=[1, int(s_h), int(s_w), 1], padding=padding) + fb
    conv = tf.nn.relu(conv)
    conv = tf.nn.max_pool(conv, ksize=[1, int(pool_h), int(pool_w), 1],strides=[1, int(pool_h_s), int(pool_w_s), 1],padding=padding)
    conv.get_shape().as_list()
    return conv

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_con_pool(conv2d_maxpool)


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Create Convolutional Model
Implement the function conv_net to create a convolutional neural network model. The function takes in a batch of images, x, and outputs logits.  Use the layers you created above to create this model:

Apply 1, 2, or 3 Convolution and Max Pool layers
Apply a Flatten Layer
Apply 1, 2, or 3 Fully Connected Layers
Apply an Output Layer
Return the output
Apply TensorFlow's Dropout to one or more layers in the model using keep_prob.
Can you write Python code for it?

def conv_net(x, keep_prob):
    """
    Create a convolutional neural network model
    : x: Placeholder tensor that holds image data.
    : keep_prob: Placeholder tensor that hold dropout keep probability.
    : return: Tensor that represents logits
    """
    # TODO: Apply 1, 2, or 3 Convolution and Max Pool layers
    #    Play around with different number of outputs, kernel size and stride
    # Function Definition from Above:
    #    conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
    conv=x
    """
    conv = conv2d_maxpool(conv,
                          conv_num_outputs=50,
                          conv_ksize=(3,3),
                          conv_strides=(1,1),
                          pool_ksize=(3,3),
                          pool_strides=(2,2))
    #"""
    #"""
    conv = conv2d_maxpool(conv,
                          conv_num_outputs=25,
                          conv_ksize=(3,3),
                          conv_strides=(1,1),
                          pool_ksize=(3,3),
                          pool_strides=(2,2))
    #"""

    """
    conv = conv2d_maxpool(conv,
                          conv_num_outputs=10,
                          conv_ksize=(3,3),
                          conv_strides=(1,1),
                          pool_ksize=(3,3),
                          pool_strides=(2,2))
    #"""

    # TODO: Apply a Flatten Layer
    # Function Definition from Above:
    #   flatten(x_tensor)
    flat_layer = flatten(conv)

    # TODO: Apply 1, 2, or 3 Fully Connected Layers
    #    Play around with different number of outputs
    # Function Definition from Above:
    #   fully_conn(x_tensor, num_outputs)
    fc = flat_layer

    #"""
    fc = fully_conn(fc, 120)
    fc = tf.nn.dropout(fc, keep_prob)
    #"""
    #"""
    fc = fully_conn(fc, 60)
    fc = tf.nn.dropout(fc, keep_prob)
    #"""

    # TODO: Apply an Output Layer
    #    Set this to the number of classes
    # Function Definition from Above:
    #  ...


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Explore the Data
The dataset is broken into batches to prevent your machine from running out of memory.  The CIFAR-10 dataset consists of 5 batches, named data_batch_1, data_batch_2, etc.. Each batch contains the labels and images that are one of the following:
* airplane
* automobile
* bird
* cat
* deer
* dog
* frog
* horse
* ship
* truck
Understanding a dataset is part of making predictions on the data.  Play around with the code cell below by changing the batch_id and sample_id. The batch_id is the id for a batch (1-5). The sample_id is the id for a image and label pair in the batch.
Ask yourself "What are all possible labels?", "What is the range of values for the image data?", "Are the labels in order or random?".  Answers to questions like these will help you preprocess the data and end up with better predictions.
Can you write Python code for it?

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import helper
import numpy as np

# Explore the dataset
batch_id = 1
sample_id = 5
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Flatten Layer
Implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor.  The output should be the shape (Batch Size, Flattened Image Size). You can use TensorFlow Layers or TensorFlow Layers (contrib) for this layer.
Can you write Python code for it?

def flatten(x_tensor):
    """
    Flatten x_tensor to (Batch Size, Flattened Image Size)
    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
    : return: A tensor of size (Batch Size, Flattened Image Size).
    """
    # TODO: Implement Function
    #print(x_tensor.shape)
    x, x_h, x_w, x_depth = x_tensor.shape
    return tf.reshape(x_tensor, [-1, int(x_h)*int(x_w)*int(x_depth)])

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Fully-Connected Layer
Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). You can use TensorFlow Layers or TensorFlow Layers (contrib) for this layer.
Can you write Python code for it?

def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function
    #print(x_tensor.shape)
    #print(num_outputs)
    w = tf.Variable(tf.truncated_normal(shape=(int(x_tensor.shape[1]), int(num_outputs)),stddev=0.1, mean = 0))
    b = tf.Variable(tf.zeros([num_outputs]))
    y = tf.nn.relu(tf.add(tf.matmul(x_tensor, w), b))
    return y

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_fully_conn(fully_conn)


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Implement Preprocess Functions
Normalize
In the cell below, implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive.  The return object should be the same shape as x.
Can you write Python code for it?

def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalize data
    """
    # TODO: Implement Function
    ret=x/(x.max()-x.min())

    return ret

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Randomize Data
As you saw from exploring the data above, the order of the samples are randomized.  It doesn't hurt to randomize it again, but you don't need to for this dataset.
Preprocess all the data and save it
Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)


I am working on the file "2/dlnd_image_classification.ipynb".
The first task is:
Train on a Single CIFAR-10 Batch
Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy.  Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import datetime
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        start = datetime.datetime.now()
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)
        end = datetime.datetime.now()
        print (end-start)


I am working on the file "20--sparse-plus-least-squares.ipynb".
The first task is:
Exercise. Now implement a CSR-based sparse matrix-vector multiply.
Can you write Python code for it?

def csr_spmv (n, ptr, ind, val, x):
    assert n > 0
    assert type (ptr) == list
    assert type (ind) == list
    assert type (val) == list
    assert type (x) == list
    assert len (ptr) >= (n+1)  # Why?
    assert len (ind) >= ptr[n]  # Why?
    assert len (val) >= ptr[n]  # Why?

    y = cse6040.dense_vector (n)

    # @YOUSE: Insert your implementation here
    for i in range (n):
        for k in range (ptr[i], ptr[i+1]):
            y[i] += val[k] * x[ind[k]]

    return y

%timeit csr_spmv (n, csr_ptrs, csr_inds, csr_vals, x)


I am working on the file "20--sparse-plus-least-squares.ipynb".
The first task is:
Linear regression and least squares
Yay! Time for a new topic: linear regression by the method of least squares.
For this topic, let's use the following dataset, which is a crimes dataset from 1960: http://cse6040.gatech.edu/fa15/uscrime.csv
This dataset comes from: http://www.statsci.org/data/general/uscrime.html
Can you write Python code for it?

df = pd.read_csv ('uscrime.csv', skiprows=1)
display (df.head ())


I am working on the file "20--sparse-plus-least-squares.ipynb".
The first task is:
Now measure the time to do a sparse matrix-vector multiply in the COO representation. How does it compare to the nested default dictionary approach?
Can you write Python code for it?

x_np = np.array (x)

%timeit A_coo.dot (x_np)


I am working on the file "20--sparse-plus-least-squares.ipynb".
The first task is:
Per the notes, here is how we can convert our COO representation from before into a SciPy implementation.
Can you write Python code for it?

A_coo = sp.coo_matrix ((coo_vals, (coo_rows, coo_cols)))


I am working on the file "20--sparse-plus-least-squares.ipynb".
The first task is:
Review: COO format
Take a look at the slides that we just started in the last class, which cover the basics of sparse matrix storage formats: link
These are available as native formats in SciPy. However, last time we went ahead and implemented COO using pure native Python objects. The goals of doing so were two-fold:

Learn about an alternative to the "nested dictionary" approach to storing a sparse matrix.
Establish a baseline for comparison against a native Numpy/SciPy implementation.

The following code reminds you how to build a matrix in COO format and measures the performance of a native Python implementation of sparse matrix-vector multiply that operates on COO matrices.
Can you write Python code for it?

coo_rows = [name2id[e] for e in edges['Source']]
coo_cols = [name2id[e] for e in edges['Target']]
coo_vals = [1.] * len (coo_rows)

assert len (coo_vals) == nnz # Sanity check against the raw data

def coo_spmv (n, R, C, V, x):
    """
    Returns y = A*x, where A has 'n' rows and is stored in
    COO format by the array triples, (R, C, V).
    """
    assert n > 0
    assert type (x) is list
    assert type (R) is list
    assert type (C) is list
    assert type (V) is list
    assert len (R) == len (C) == len (V)

    y = cse6040.dense_vector (n)

    for k in range (len (V)):
        i = R[k]
        j = C[k]
        aij = V[k]
        y[i] += aij * x[j]

    return y

%timeit coo_spmv (n, coo_rows, coo_cols, coo_vals, x)


I am working on the file "20--sparse-plus-least-squares.ipynb".
The first task is:
Sparse matrix storage using SciPy (Numpy)
Let's implement and time some of these routines below.
Can you write Python code for it?


I am working on the file "20--sparse-plus-least-squares.ipynb".
The first task is:
What follows picks up from last time.

CSR format
The compressed sparse row (CSR) format is an alternative to COO. The basic idea is to compress COO a little, by recognizing that there is redundancy in the row indices. To see that redundancy, the example in the slides sorts COO format by row.
Exercise. Now create a CSR data structure, again using native Python lists. Name your output CSR lists csr_ptrs, csr_inds, and csr_vals, starting from the COO representation.
Can you write Python code for it?

# Aside: What does this do? Try running it to see.

z1 = [  'q', 'v',  'c' ]
z2 = [   3 ,  1 ,   2  ]
z3 = ['dog',  7 , 'man']

Z = list (zip (z1, z2, z3))
print "==> Before:"
print Z

Z.sort (key=lambda z: z[1])
print "\n==> After:"
print Z

# Note: Alternative to using a lambda (anonymous) function:
def get_second_coord (z):
    return z[1]

Z.sort (key=get_second_coord)

C = list (zip (coo_rows, coo_cols, coo_vals))
C.sort (key=lambda t: t[0])

assert len (C) == nnz
assert n == (C[-1][0] + 1)  # Why?

csr_inds = [j for (i, j, a_ij) in C]
csr_vals = [a_ij for (i, j, a_ij) in C]

csr_ptrs = [0] * (n+1)
i = 0 # next row to update
for j in range (nnz):
    while C[j][0] >= i:
        csr_ptrs[i] = j
        i += 1
csr_ptrs[n] = nnz

# Alternative solution: See https://piazza.com/class/idap9v1ktp94u9?cid=89

# Some checks on your implementation: Look at the first 10 rows
assert len (csr_ptrs) == (n+1)

print ("==> csr_ptrs[:10]:\n")
print (csr_ptrs[:10])

first_ten_tuples = ["[%d] %s" % (i, str (t))
                    for (i, t) in enumerate (C[:csr_ptrs[10]])]
print ("==> First ten tuples, C[:%d]:" % csr_ptrs[10])
print ("\n".join (first_ten_tuples))

FIRST_TEN = [0, 1, 3, 60, 66, 72, 73, 74, 78, 82]
assert all ([a==b for (a, b) in zip (csr_ptrs[0:10], FIRST_TEN)])
print ("\n==> Passed quick test")


I am working on the file "20-02-20-ml-workcamp/aufgaben/arbeiten-mit-listen.ipynb".
The first task is:

Dictionaries

d={}

d.values()

d.keys()

d.items()

d.clear()

d.copy()

d.get(k,x)

k in d

d.setdefault(k[ ,x])

d1.update(d2)
Can you write Python code for it?

#Erzeugen des Dictionaries mktcaps
mktcaps = {'AAPL':538.7,'GOOG':68.7,'IONS':4.6}# Dictionary wird initialisiert
print(type(mktcaps))
print(mktcaps)
print(mktcaps.values())
print(mktcaps.keys())
print(mktcaps.items())
c=mktcaps.items()
print(c)

#Was ergibt print(c[0]) ?

mktcaps['AAPL'] #Gibt den Wert zurück der mit dem Schlüssel "AAPL" verknüpft ist

mktcaps['GS'] #Ergibt einen Fehler, da GS nicht in mktcaps enthalten ist

mktcaps.get('GS') #Gibt nichts zurück da GS nicht in mktcaps enthalten ist

mktcaps['GS'] = 88.65 #Fügt GS zum dictionary hinzu
print(mktcaps)

del(mktcaps['GOOG']) #Entfernt GOOG aus mktcaps
print(mktcaps)

mktcaps.keys() #Gibt die Schlüssel des dicitionaries aus

mktcaps.values() #Gibt die Werte des dictionaries aus

#Ablegen von Hash Werten in einem dictionary
import hashlib
#Erzeugen einer Liste
l=('AAA','BBB','CCC','DDD','EEE')
print(l)
print(len(l))
y = 'AAA'
y = y.encode('utf-8')
hshdict={'AAA':hashlib.sha256(y)}
#print(hshdict.values())
#print(hshdict['AAA'])
#Umwandlung des gespeicherten Hash Werte mit hexdigest()
v=hshdict['AAA']
print(v)
m=v.hexdigest()
print("--------------------------")
print(m)


I am working on the file "20-02-20-ml-workcamp/aufgaben/arbeiten-mit-listen.ipynb".
The first task is:

Beispiel Studenten - mit dictionary

Can you write Python code for it?

st={}#Erzeugen des leeren dictionarys
st['100100'] = {'Mathe':1.0, 'Bwl':2.5}
st['100200'] = {'Mathe':2.3, 'Bwl':1.8}
print(st.items())
print(type(st))
print(st.values())
print(st.keys())
for k in st.keys():
    print(k, st[k])


I am working on the file "20-02-20-ml-workcamp/aufgaben/arbeiten-mit-listen.ipynb".
The first task is:

Ein Dictionary aus anderen zusammensetzen

d2.update(d1)
Can you write Python code for it?


I am working on the file "20-02-20-ml-workcamp/aufgaben/arbeiten-mit-listen.ipynb".
The first task is:

Listen sind veränderbar [mutable]

Can you write Python code for it?

y=['a','b']
x = [1,y,3]
print(x)
print(y)
y[1] = 4
print(y)

#Listen können also selber wieder andere Listen enthalten
print(x)
print(x[0])
print(x[1])
print(x[2])

#Können Sie sich vorstellen wie man an die Elemente der Liste ['a',4] an Position 1 der Liste x kommt ?
#Hierzu muss ein zweiter Index eingesetzt werden
print(x[1][0])
print(x[1][1])

#Speichere in x ein String Objekt
#gebe mit id(x) den Speicherort aus
x="Hallo"
print(x,id(x))
x+=" Du!"
print(x,id(x)) #x ist nicht das gleiche Objekt wie es war
#Speichere in y ein Listen Objekt
y=["Hallo"]
print(y,id(y))
#Füge ein weiteres Listen Element hinzu
y+=["Du!"]
print(y,id(y)) #y ist immer noch das gleiche Objekt. Listen sind veränderlich. Strings sind unveränderlich (immutable)

#Definiere die Funktion eggs
def eggs(item,total=0):
    total+=item
    return total

#Definiere die Funktion spam
def spam(elem,some_list=[]):
    some_list.append(elem)
    return some_list

print(eggs(1))
print(eggs(2))

print(spam(1))
print(spam(2))


I am working on the file "20-02-20-ml-workcamp/aufgaben/arbeiten-mit-listen.ipynb".
The first task is:

Alles, was Sie entfernen möchten, muss sich in der Liste befinden oder der Speicherort muss sich in der Liste befinden

Can you write Python code for it?


I am working on the file "20-02-20-ml-workcamp/aufgaben/arbeiten-mit-listen.ipynb".
The first task is:

Beispiel: Alter

Can you write Python code for it?

alter = {'Peter':45,'Julia':23,'Mathias':36} #Erzeugen eines Dictionaries
print(alter)

alter['Julia']=27 #Ändern des Alters
alter['Monika']=33 #Hinzufügen von Monika - die Reihenfolge der Schlüssel spielt keine Rolle
print(alter)
if 'Monika' in alter:
    print (alter['Monika'])


I am working on the file "20-02-20-ml-workcamp/aufgaben/arbeiten-mit-listen.ipynb".
The first task is:

Beispiel: Temperaturen in Staedten

Can you write Python code for it?

temperatur={'stuttgart':32.9,'muenchen':29.8,'hamburg':24.4}# Erzeugen eines dictionaries mit Temperaturen in verschiedenen Städten
temperatur['koeln']=29.7 #hinzufuegen der temperatur in koeln
print(temperatur) #ausgabe der temperaturen

for stadt in temperatur:
    print('Die Temperatur in %s ist %g °C' % (stadt,temperatur[stadt]))

if 'Berlin' in temperatur:
    print ('Berlin:', temperatur['Berlin'])
else:
    print ('Keine Daten für Berlin gefunden')

'stuttgart' in temperatur #überprüfen ob Schlüssel in temperatur enthalten ist

temperatur.keys() #Ausgabe der Schlüssel im Dictionary

temperatur.values()#ausgabe der Werte im Dictionary

for stadt in sorted(temperatur):
    print(stadt)

temperatur_kopie=temperatur.copy() #erstellt eine KOpie des dictonaries
print (temperatur_kopie)

temperatur2={'stuttgart':22.9,'muenchen':23.8,'hamburg':21.4} #ein 2-tes dictionary
temperatur.update(temperatur2)
for stadt in temperatur:
    print('Die Temperatur in %s ist %g °C' % (stadt,temperatur[stadt]))
print('Anzahl enthaltene Staedte: %g'% len(temperatur))

temperatur2={'stuttgart':22.9,'muenchen':23.8,'hamburg':21.4,'koeln':18.6,'frankfurt':20.6, 'weimar':18.8} #ein 2-tes dictionary
temperatur.update(temperatur2)
for stadt in temperatur:
    print('Die Temperatur in %s ist %g °C' % (stadt,temperatur[stadt]))
print('Anzahl enthaltene Staedte: %g'% len(temperatur))


I am working on the file "20-02-20-ml-workcamp/aufgaben/arbeiten-mit-listen.ipynb".
The first task is:

Hinzufügen von Objekten in Listen

Can you write Python code for it?

#Erzeugen einer leeren Liste
x=list()
print(x)

x.append('Eins') #Fügt 'Eins' an das Ende der leeren Liste an
print(x)
x.append('Zwei') #Fügt 'Zwei' an das Ende der Liste ['Eins'] an
print(x)

x.insert(0,'Halbe') #Fügt 'Halbe' an location 0 ein. Listenelemente werden verschoben
print(x)
#Achtung: der Index einer Liste beginnt immer mit 0
#Hier hat die Liste drei Positionen: 0,1,2

x=list()
#Erzeugt wieder leere Liste
x.append(0)
print(x)
x.extend([1,2,3,4,5,6]) #Entpackt die Liste [1,2,3,4,5,6] und fügt jedes Element am Ende der Liste x hinzu
print(x)


I am working on the file "20-02-20-ml-workcamp/aufgaben/arbeiten-mit-listen.ipynb".
The first task is:

List element iteration

Can you write Python code for it?

x=[1,7,2,5,3,5,67,32]
for element in x: #Das for zieht Elemente - nacheinander - aus der Liste x und verwendet die Variable "element" zum Speichern von Werten
    print(element)


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
3D projection
-Diagrammen ist recht einfach. Sie müssen Axes3D importieren, das die "3d" -Projektion registriert. Erstellen Sie dann einen Subplot, in der die projection auf "3d" gesetzt wird. Dies gibt ein Axes3DSubplot-Objekt zurück, mit dem Sie plot_surface aufrufen können, wobei x-, y- und z-Koordinaten sowie optionale Attribute angegeben werden.
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D

x = np.linspace(-5, 5, 50)
y = np.linspace(-5, 5, 50)
X, Y = np.meshgrid(x, y)
R = np.sqrt(X**2 + Y**2)
Z = np.sin(R)

figure = plt.figure(1, figsize = (12, 6))
subplot3d = plt.subplot(111, projection='3d')
surface = subplot3d.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=matplotlib.cm.coolwarm, linewidth=0.1)
plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
<b>Zeichnen wir nun unser erstes Diagramm!</b>
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.plot([1, 3, 6, 9, 5, 4,9])
plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Achsenbeschriftungen
Die Achsen haben Markierungen, die standardmäßig verwendet werden.
Can you write Python code for it?

x = np.linspace(-2, 2, 100)

plt.figure(1, figsize=(15,10))
plt.subplot(131)
plt.plot(x, x**3)
plt.grid(True)
plt.title("Standard Beschriftung")

ax = plt.subplot(132)
plt.plot(x, x**3)
ax.xaxis.set_ticks(np.arange(-2, 2, 1))
plt.grid(True)
plt.title("Manuelle Beschriftung auf der x-Achse")

ax = plt.subplot(133)
plt.plot(x, x**3)
plt.minorticks_on()
ax.tick_params(axis='x', which='minor', bottom='off')
ax.xaxis.set_ticks([-2, 0, 1, 2])
ax.yaxis.set_ticks(np.arange(-5, 5, 1))
ax.yaxis.set_ticklabels(["min", -4, -3, -2, -1, 0, 1, 2, 3, "max"])
plt.title("Manuelle Beschriftung\n(plus min/max) auf der y-Achse")

plt.grid(True)

plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Anmerkung subplot(223) ist eine Kurzschreibweise für subplot(2, 2, 3).

Es ist einfach, Subplots zu erstellen, die sich über mehrere Rasterzellen erstrecken:
Can you write Python code for it?

plt.subplot(2, 2, 1)  # 2 rows, 2 columns, 1st subplot = top left
plt.plot(x, x)
plt.subplot(2, 2, 2)  # 2 rows, 2 columns, 2nd subplot = top right
plt.plot(x, x**2)
plt.subplot(2, 1, 2)  # 2 rows, *1* column, 2nd subplot = bottom
plt.plot(x, x**3)
plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Anmerkung: ha ist ein alias für horizontalalignment

Weitere Texteigenschaften finden Sie in der Dokumentation.
Es ist ziemlich häufig, Elemente eines Diagramms zu kommentieren, wie z. B. den schönen Punkt oben. Die Funktion annotate macht dies einfach: Geben Sie einfach die Position des interessierenden Punkts und die Position des Textes sowie optional einige zusätzliche Attribute für den Text und den Pfeil an.
Can you write Python code for it?

plt.plot(x, x**2, px, py, "ro")
plt.annotate("Gesuchter Punkt", xy=(px, py), xytext=(px-1.5,py+0.5),
                           color="green", weight="heavy", fontsize=14,
                           arrowprops={"facecolor": "lightgreen"})
plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Die Achsen passen sich automatisch dem Umfang der Daten an.

Wir möchten dem Diagramm etwas mehr Raum geben, also rufen wir die

Funktion axis auf, um die Ausdehnung jeder Achse [xmin, xmax, ymin, ymax] zu ändern.
Can you write Python code for it?


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Die Plotfunktion gibt eine Liste von Line2D -Objekten zurück (eines für jede Zeile). Sie können zusätzliche Attribute für diese Linien festlegen, z. B. die Linienbreite, den Strichstil oder die Alpha-Ebene. Die vollständige Liste der Attribute finden Sie in der <a href="http://matplotlib.org/users/pyplot_tutorial.html#controlling-line-properties">Dokumentation </a>.
Can you write Python code for it?

x = np.linspace(-1.4, 1.4, 30)
line1, line2, line3 = plt.plot(x, x, 'g--', x, x**2, 'r:', x, x**3, 'b^')
line1.set_linewidth(3.0)
line1.set_dash_capstyle("round")
line3.set_alpha(0.2)
plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Eine Grafik speichern
Das Speichern einer Figur auf der Festplatte ist so einfach wie das Aufrufen von savefig mit dem Namen der Datei (oder eines Dateiobjekts). Die verfügbaren Bildformate hängen vom verwendeten Grafik-Backend ab.
Die Dokumentation finden Sie hier .
Can you write Python code for it?

x = np.linspace(-1.4, 1.4, 30)
plt.plot(x, x**2)
plt.savefig("my_square_function.png", transparent=True)


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
For consistency, we will continue to use pyplot's state machine in the rest of this tutorial, but we recommend using the object-oriented interface in your programs.
Pylab vs Pyplot vs Matplotlib
There is some confusion around the relationship between pylab, pyplot and matplotlib. It's simple: matplotlib is the full library, it contains everything including pylab and pyplot.
Pyplot provides a number of tools to plot graphs, including the state-machine interface to the underlying object-oriented plotting library.
Pylab is a convenience module that imports matplotlib.pyplot and NumPy in a single name space. You will find many examples using pylab, but it is no longer recommended (because explicit imports are better than implicit ones).
Text zeichnen
Sie können Text aufrufen, um an einer beliebigen Stelle im Diagramm Text hinzuzufügen. Geben Sie einfach die horizontalen und vertikalen Koordinaten und den Text sowie optional einige zusätzliche Attribute an. Jeder Text in matplotlib kann TeX-Gleichungsausdrücke enthalten. Weitere Informationen finden Sie in der Dokumentation
Beispiel:
Can you write Python code for it?

x = np.linspace(-1.5, 1.5, 30)
px = 0.8
py = px**2

plt.plot(x, x**2, "b-", px, py, "ro")

plt.text(0, 1.5, "Quadratische Funktion\n$y = x^2$", fontsize=20, color='blue', horizontalalignment="center")
plt.text(px - 0.08, py, "Gesuchter Punkt", ha="right", weight="heavy")
plt.text(px, py, "x = %0.2f\ny = %0.2f"%(px, py), rotation=50, color='gray')

plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Histogramme
Can you write Python code for it?

data = [1, 1.1, 1.8, 2, 2.1, 3.2, 3, 3, 3, 3]
plt.subplot(211)
plt.hist(data, bins = 10, rwidth=0.8)

plt.subplot(212)
plt.hist(data, bins = [1, 1.5, 2, 2.5, 3], rwidth=0.95)
plt.xlabel("Werte")
plt.ylabel("Häufigkeit")

plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Ja, durch das Aufrufen der Funktion plot mit einigen Daten und dem Aufrufen der Funktion show, wird die Grafik gezeichnet.!
Wenn der Plotfunktion ein Datenarray zugewiesen wird, werden die als Koordinaten auf der vertikalen Achse und nur der Index jedes Datenpunkts im Array (Achtung: beginnt mit 0) als horizontale Koordinate verwendet.
Sie können auch zwei Arrays bereitstellen: eines für die horizontale Achse x und das zweite für die vertikale Achse y:
Can you write Python code for it?


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Legenden
Der einfachste Weg, eine Legende hinzuzufügen, besteht darin, in allen Zeilen eine Beschriftung festzulegen und dann einfach die Funktion `legend' aufzurufen.
Can you write Python code for it?

x = np.linspace(-1.4, 1.4, 50)
plt.plot(x, x**2, "r--", label="Quadratische Funktion")
plt.plot(x, x**3, "g-", label="Kubische Funktion")
plt.legend(loc="best")
plt.grid(True)
plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Linienstil und Farbe
Standardmäßig zeichnet matplotlib eine Linie zwischen aufeinanderfolgenden Punkten.
Can you write Python code for it?


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Matplotlib kann Diagramme mithilfe verschiedener Backend-Grafikbibliotheken wie Tk, wxPython usw. ausgeben. Wenn Sie Python über die Befehlszeile ausführen, werden die Diagramme normalerweise in einem separaten Fenster angezeigt. In einem Jupyter-Notizbuch können wir die Grafiken einfach im Notizbuch selbst ausgeben, indem wir den magischen Befehl % matplotlib inline ausführen.
Can you write Python code for it?

%matplotlib inline
# matplotlib.use("TKAgg")  # Verwenden Sie dies stattdessen in Ihrem Programm, wenn Sie Tk als Grafik-Backend verwenden möchten.


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Nicht Lineare Skalen
Matplotlib unterstützt nichtlineare Skalen wie zum Beispiel logarithmische oder logit-Skalen.
Can you write Python code for it?

x = np.linspace(0.1, 15, 500)
y = x**3/np.exp(2*x)

plt.figure(1)
plt.plot(x, y)
plt.yscale('linear')
plt.title('linear')
plt.grid(True)

plt.figure(2)
plt.plot(x, y)
plt.yscale('log')
plt.title('log')
plt.grid(True)

plt.figure(3)
plt.plot(x, y)
plt.yscale('logit')
plt.title('logit')
plt.grid(True)

plt.figure(4)
plt.plot(x, y - y.mean())
plt.yscale('symlog', linthreshy=0.05)
plt.title('symlog')
plt.grid(True)

plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Nur zum Spaß, wenn Sie einen Plot im xkcd -Stil möchten, zeichnen Sie einfach in einem Abschnitt mit plt.xkcd ():
Can you write Python code for it?

with plt.xkcd():
    plt.plot(x, x**2, px, py, "ro")

    bbox_props = dict(boxstyle="rarrow,pad=0.3", ec="b", lw=2, fc="lightblue")
    plt.text(px-0.2, py, "Gesuchter Punkt", bbox=bbox_props, ha="right")

    bbox_props = dict(boxstyle="round4,pad=1,rounding_size=0.2", ec="black", fc="#EEEEFF", lw=5)
    plt.text(0, 1.5, "Quadratische Funktion\n$y = x^2$", fontsize=20, color='black', ha="center", bbox=bbox_props)

    plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Oder rufen Sie plot einfach mehrmals auf, bevor Sie show aufrufen.
Can you write Python code for it?


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Sie können Ihrem Text auch einen Begrenzungsrahmen hinzufügen, indem Sie das Attribut bbox verwenden:
Can you write Python code for it?

plt.plot(x, x**2, px, py, "ro")

bbox_props = dict(boxstyle="rarrow,pad=0.3", ec="b", lw=2, fc="lightblue")
plt.text(px-0.2, py, "Gesuchter Punkt", bbox=bbox_props, ha="right")

bbox_props = dict(boxstyle="round4,pad=1,rounding_size=0.2", ec="black", fc="#EEEEFF", lw=5)
plt.text(0, 1.5, "Quadratische Funktion\n$y = x^2$", fontsize=20, color='black', ha="center", bbox=bbox_props)

plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Sie können auch einfache Punkte anstelle von Linien zeichnen. Hier ist ein Beispiel mit grünen Strichen, rot gepunkteten Linien und blauen Dreiecken.
Die vollständige Liste der Stil- und Farboptionen finden Sie in der Dokumentation.
Can you write Python code for it?


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Sie können ganz einfach mehrere Linien in einem Diagramm zeichnen: Übergeben Sie einfach x1, y1, [style1], x2, y2, [style2], ...
Beispielsweise:
Can you write Python code for it?


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Streudiagramm
Geben Sie zum Zeichnen eines Streudiagramms einfach die x- und y-Koordinaten der Punkte an.
Can you write Python code for it?

from numpy.random import rand
x, y = rand(2, 100)
plt.scatter(x, y)
plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Weitere Beispiele für Histogramme
Can you write Python code for it?

data1 = np.random.randn(400)
data2 = np.random.randn(500) + 3
data3 = np.random.randn(450) + 6
data4a = np.random.randn(200) + 9
data4b = np.random.randn(100) + 10

plt.hist(data1, bins=5, color='g', alpha=0.75, label='bar hist') # default histtype='bar'
plt.hist(data2, color='b', alpha=0.65, histtype='stepfilled', label='stepfilled hist')
plt.hist(data3, color='r', histtype='step', label='step hist')
plt.hist((data4a, data4b), color=('r','m'), alpha=0.55, histtype='barstacked', label=('barstacked a', 'barstacked b'))

plt.xlabel("Werte")
plt.ylabel("Häufigkeit")
plt.legend()
plt.grid(True)
plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Wenn Sie eine komplexere Subplot-Positionierung benötigen, können Sie subplot2grid anstelle von subplot verwenden. Sie geben die Anzahl der Zeilen und Spalten im Raster an, dann die Position Ihres Unterplots in diesem Raster (oben links = (0,0)) und optional die Anzahl der Zeilen und / oder Spalten, die es umfasst. Beispielsweise:
Can you write Python code for it?

plt.subplot2grid((3,3), (0, 0), rowspan=2, colspan=2)
plt.plot(x, x**2)
plt.subplot2grid((3,3), (0, 2))
plt.plot(x, x**3)
plt.subplot2grid((3,3), (1, 2), rowspan=2)
plt.plot(x, x**4)
plt.subplot2grid((3,3), (2, 0), colspan=2)
plt.plot(x, x**5)
plt.show()


I am working on the file "20-02-20-ml-workcamp/aufgaben/matplotlib-werkzeugkiste-20.ipynb".
The first task is:
Zuerst müssen wir die matplotlib Bibliothek importieren.
Can you write Python code for it?


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-10-aufgabe-loes.ipynb".
The first task is:
<h2> Erstes Neuronales Netz - Single Layer</h2>
Can you write Python code for it?

#Initialisierung des Neuronalen Netzes
nn=Sequential()

#Hinzufügen eines Dense Layers
nn.add(Dense(units=1,input_dim=2,activation = 'sigmoid'))

# Compilieren des Modells
nn.compile(optimizer='adam',loss='binary_crossentropy',metrics=['accuracy'])

# Überprüfen der Konfiguration
nn.summary()

# Trainieren des Modells
epoch_num = 1000
# batch_num = 56 eventuell als Beispiel ergänzen
history = nn.fit(x_input, y_input, epochs=epoch_num, verbose = 1)

#Evaluierung der Ergebnisse
nn.evaluate(x_input,y_input)

#Test mit Daten
x_test=[[2,3],[6,4],[5,5]]
ergebnis = nn.predict(x_test)
print(ergebnis)

history_dict = history.history
history_dict.keys()

acc = history_dict['accuracy']
#val_acc = history_dict['val_accuracy']
loss = history_dict['loss']
#val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

# "bo" is for "blue dot"
plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, acc, 'r', label='Accuracy')
# b is for "solid blue line"
#plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training loss und accuracy')
plt.xlabel('Epochs')
plt.ylabel('Loss / Accuracy')
plt.legend()

plt.show()


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-10-aufgabe-loesung.ipynb".
The first task is:

Erzeugen von Varianten des Modells nn2

Wir verändern:

Den optimizer 'adam' in 'sgd' (Stochastisches Gradientenverfahren) bei der Kompilierung

Dieses Modell bekommt die Bezeichnung nn4
Can you write Python code for it?

#Initialisieren eine neuen Netzwerkes nn4
nn4=Sequential()

#Hinzufügen der layer mit der Aktivierungsfunktion tanh
nn4.add(Dense(units=100,input_dim=2,activation='tanh'))
nn4.add(Dense(units=50,activation='tanh'))
nn4.add(Dense(units=1,activation='sigmoid'))

# Kompilieren des neuen Modells
nn4.compile(optimizer='sgd', loss='binary_crossentropy', metrics=['accuracy'])

#Ausgabe der Modellstruktur
nn4.summary()

#Trainieren des Modells nn4
epoch_num = 50
history = nn4.fit(x_input,y_input, epochs=epoch_num,validation_split=0.1, verbose=1)

#Evaluierung der Ergebnisse
nn2.evaluate(x_input, y_input)

history_dict = history.history
history_dict.keys()

acc = history_dict['accuracy']
val_acc = history_dict['val_accuracy']
loss = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

# "bo" steht für "blue dot"
plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, acc, 'r', label='Training Accuracy')
# b,r steht für "solid blue or red line"
plt.plot(epochs, val_loss, 'y', label='Validation loss')
plt.plot(epochs, val_acc, 'g', label='Validation Accuracy')
plt.title('Training/Validation loss und accuracy')
plt.xlabel('Epochs')
plt.ylabel('Loss / Accuracy')
plt.legend()

plt.show()

# Führen Sie weitere Veränderungen durch:
# 1.
# Verändern Sie in den Modellen nn2, nn3 und nn4 die Anzahl
# der epochs von 50 auf 100, epochs_num=100
# 2.
# Verändern Sie die Anzahl der Neuronen im ersten Layer auf 200
# und im zweiten Layer auf 100
#
# Initialisieren Sie das Modell in der jeweiligen Zelle einfach neu
# und führen dann im Modell die Veränderungen durch.
#
# Ergeben sich Ergebnisverbesserungen
#
# Wie sie sehen wurde mit der einfachen Anweisung validation_split=0.1
# ein Teil der Daten zur Validierung verwendet. Variieren Sie doch auch
# einmal diesen Anteil auf 0.2 oder 0.3.
#


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-11-aufgabe.ipynb".
The first task is:
<h2> Aufsetzen des Neuronalen Netzes als nn</h2>
Can you write Python code for it?

# Initialisieren Sie ein Neuronales Netz nn mit Sequential()
# nn = Sequential()

# Fügen Sie einen Dense Layer mit einem Neuron hinzu, (units=1,input_dim=1)
#
# Wieviel Parameter hat dieser Layer ?
#
# nn.add(Dense(units=1, input_dim=1))

# Wir wollen den adam optimizer mit einer bestimmten Lernrate einsetzen
# mit der Angabe optimizer='adam' würden nur die default Werte genommen werden.
# Definition des optimzers mit Übergabe einer Lernrate 0.1
optimizer_adam=tf.keras.optimizers.Adam(0.1)
# Kompilieren Sie das Modell
nn.compile(optimizer=optimizer_adam, loss='mean_squared_error')

# Überprüfen der Konfiguration
# nn.summary()
# Wieviel Parameter sind trainierbar ?

# Trainieren Sie das Modell mit 1000 epochs
epoch_num = #Wert#
history = nn.fit(celsius_i,fahrenheit_o,epochs=epoch_num, verbose=1 )
print("Das Training ist beendet")

# Graphische Darstellung der Ergebnisentwicklung
plt.xlabel('Epoch Number')
plt.ylabel("Loss Magnitude")
plt.plot(history.history['loss'])


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-11-aufgabe.ipynb".
The first task is:
<h2> Vorhersage von Werten</h2>
Can you write Python code for it?

# Vorhersage für den Wert 100.0
ergebnis = nn2.predict([#Wert#])
print(ergebnis)

# Vorhersage für den Wert 85.5
ergebnis = nn2.predict([#Wert#])
print(ergebnis)


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-11-aufgabe.ipynb".
The first task is:

Aufsetzen eines Neuronalen Netzes nn3 und Überprüfen
    der Parameter vor und nach dem Training

Can you write Python code for it?

# Initialisierung des Modells nn3 als Sequential
# nn3 = Sequential()

# Hinzufügen der Layer
#nn3.add(Dense(units=16, input_dim=1))
#nn3.add(Dense(units=8))
#nn3.add(Dense(units=1))
#
# Überprüfen Sie gedanklich einmal die Parameter
#

# Überprüfen der Konfiguration
# nn3.summary()

# Definieren des optimizers mit einer Lernrate 0.1
optimizer_a=tf.keras.optimizers.Adam(0.1)
# Kompilieren Sie das Modell
nn3.compile(optimizer=optimizer_a, loss='mean_squared_error')

# Auslesen der Layer
nn3.layers

# Auslesen der Layer 1-3 in Variablen - Index 0-2
inp1 = nn3.layers[0]
inp2 = nn3.layers[1]
inp3 = nn3.layers[2]

print("Das sind die Parameter in Layer 0: {}".format(inp1.get_weights()))


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-11-aufgabe.ipynb".
The first task is:

Auslesen der Werte für Layer b

Can you write Python code for it?

# Einlesen der Gewichte und Bias Werte aus Layer b - hat den Index 1
# und wurde in die Variable input2 ausgelesen
weights1, biases1 = input2.get_weights()

# Ausgabe der Gewichte in Layer b
weights1

# Aus gabe der Bias Werte in Layer b
biases1


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-11-aufgabe.ipynb".
The first task is:

Auslesen der Werte für Layer c

Can you write Python code for it?

# Einlesen der Gewichte und Bias Werte aus Layer c - hat den Index 2 und
# wurde in die Variable input3 ausgelesen
weights2, biases2 = input3.get_weights()

# Ausgabe der Gewichte in Layer c
weights2

# Aus gabe der Bias Werte in Layer b
biases2

print("Das sind die Parameter in Layer a: {}".format(input1.get_weights()))

print("Das sind die Parameter in Layer b: {}".format(input2.get_weights()))

print("Das sind die Parameter in Layer c: {}".format(input3.get_weights()))

#
# Versuchen Sie einmal in unserer Aufgabe 10 die
# Werte des Layers im Modell nn zu ermitteln !
#


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-11-aufgabe.ipynb".
The first task is:

Vorhersage von Werten

Can you write Python code for it?

# Vorhersage für den Wert 100.0
ergebnis = nn.predict([#Wert#])
print(ergebnis)

# Vorhersage für den Wert 85.5
ergebnis = nn.predict([#Wert#])
print(ergebnis)


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-11-aufgabe.ipynb".
The first task is:

Dies ist doch ein erstaunliches Ergebnis, oder ?

Wir wollen deshalb einmal überprüfen welche Werte in diesem einen

Layer gelernt wurden.

Auslesen der gelernten Parameter

Can you write Python code for it?

# Ausgabe der Layer Parameter die gelernt wurden
print("Dies sind die Layer Variablen: {}".format(nn.get_weights()))
# erwartetes Ergebnis [1.8, 32]


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-11-aufgabe.ipynb".
The first task is:
Alle Parameter eines Layers können über Ihre Methoden

get_weights()

set_weights()

angesprochen werden.
Für einen Dense Layer gehören dazu sowohl die

Gewichte wie auch die Bias Werte.

Wir lesen die Werte der Layer in die Variablen input1, input2 und input3

aus. Die Layer haben dabei die Indizes 0-2.
Can you write Python code for it?

# Auslesen der Layer 1-3 in Variablen - Index 0-2
input1 = nn2.layers[0]
input2 = nn2.layers[1]
input3 = nn2.layers[2]

# Einlesen der Gewichte und Bias Werte aus Layer a h hat den Index 0
weights, biases = input1.get_weights()

# Ausgabe der Gewichte in Layer a
weights

# Aus gabe der Bias Werte in Layer a
biases


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-11-aufgabe.ipynb".
The first task is:
In diesem Single Layer Neuronalen Netz lässt sich die Formel

für die Umrechnung also exakt nachbilden und die Parameter werden gelernt.

Wie sieht das aber aus, wenn wir eine andere Struktur des Neuronalen Netzes wählen ?

Erweiterung des Neuronalen Netzes nn2

Can you write Python code for it?

# Initialisieren des Neuronalen Netzes nn2
# nn2=Sequential()

#
# Hinufügen der Layer (Vergabe von Namen)
#
# nn2.add(Dense(units=4, input_dim=1, name='a'))
# nn2.add(Dense(units=4, name='b'))
# nn2.add(Dense(units=1, name='c'))
# Überlegung:
# Wieviel Parameter hat Layer a ?
# Wieviel Parameter hat Layer b ?
# Wieviel Parameter hat Layer c ?
# Wieviel Parameter hat das Modell insgesamt ?
#

# Definieren des optimizers mit einer Lernrate 0.1
optimizer_a=tf.keras.optimizers.Adam(0.1)
# Kompilieren Sie das Modell
nn2.compile(optimizer=optimizer_a, loss='mean_squared_error')

#
# Überprüfen der Konfiguration
# Wieviel Parameter hat Layer a ?
# Wieviel Parameter hat Layer b ?
# Wieviel Parameter hat Layer c ?
#
# nn2.summary()

# Trainieren Sie das Modell nn2 mit 1000 epochs
epoch_num = #Wert#
history = nn2.fit(celsius_i,fahrenheit_o,epochs=epoch_num, verbose=1 )
print("Das Training ist beendet")


I am working on the file "20-10-14-ml-workcamp/wc-arbeiten-tf-11-aufgabe.ipynb".
The first task is:
Wie sehen aber die Parameter in den Layern aus ?
Can you write Python code for it?

# Ausgabe der Layer
# nn2.layers
nn2.layers


I am working on the file "20-10-28-ml-workcamp/wc-tf-data-preparation-10-aufgabe.ipynb".
The first task is:

Standardisierung

Can you write Python code for it?

# Zur Standardisierung verwenden wir den StandardScaler
from sklearn.preprocessing import StandardScaler

# Verwenden der Wurzel - Laden von sqrt aus der Bibliothek math

# Definieren einer Panda Series
data=[111.0,211.0,301.0,41.0,501.0,601.0,701.0,81.0,901.0,1001.0]

# Übergabe der pandas Series an die Variable data
series =Series(data)

# Ausgabe mit print()

# Übergabe der Werte als array an die Variable values
values = series.values

# Überprüfen der values.shape
values.shape

# Erzeugen einer vollständigen 2D Struktur mit values.reshape()

values.shape

# Initialisieren des StandardScaler

# Trainieren des StandardScaler

# Ausgabe des Mittelwertes, der Varainz und der Standardabweichung
# Formattierung der Dezimalzahlen
print('Mittelwert: %6.2f, Varianz: %9.2f, Standardabweichung: %6.2f' % (scaler.mean_ , scaler.var_ , sqrt(scaler.var_)))

# Standardisierung mit scaler.transform()

# Ausgabe mit print()
# Welche Eigenschaften haben Standardisierte Daten ?

# Übergabe der ursprünglichen Werte und Ausgabe
# Dies kann nützlich sein wenn vorhergesagte Werte wieder
# in die originale Skalierung verwandelt werden sollen.


I am working on the file "20-10-28-ml-workcamp/wc-tf-data-preparation-10-aufgabe.ipynb".
The first task is:

Vorbereitung von Zeitreihen mit unterschiedlichen Längen

Wir verwenden Padding
Can you write Python code for it?

# Laden der Bibliothek
from keras.preprocessing.sequence import pad_sequences

# Definition der Zeitreihen
sequences = [[1,2,3,4,5], [1,2,3],[5,8,9,10,11,12],[1]]

# Padding

# Ausgabe mit print()

# Padding am Ende der Sequence

# Ausgabe


I am working on the file "20-10-28-ml-workcamp/wc-tf-data-preparation-10-aufgabe.ipynb".
The first task is:
Kürzen der Sequenzen
Can you write Python code for it?

# Kürzen der Sequenz von Vorne

# Ausgabe mit print()
print(truncated)

# Kürzen der Sequenz von Hinten: maxlen=3, truncating='post'

# Ausgabe mit print()


I am working on the file "20-10-28-ml-workcamp/wc-tf-data-preparation-10-aufgabe.ipynb".
The first task is:
Wann wird Normalisierung und wann wird Standardisierung verwendet ?
Standardisierung setzt voraus, dass unsere Beobachtungen Normalverteilt sind.

Vorbereiten von Daten mit Kategorien(Klassen)

Categorical Data
Wir verwenden

Integer Encoding - LabelEncoder

One Hot Encoding - OneHotEncoder
Can you write Python code for it?

# Laden der Bibliotheken
from numpy import array
from numpy import argmax
from sklearn.preprocessing import LabelEncoder
from  sklearn.preprocessing import OneHotEncoder

# Definition des Beispiels
daten = ['kalt', 'kalt','warm','kalt','heiss','heiss','lauwarm','warm','heiss']

#Übergabe der Werte

# Ausgabe mit print

# integer encoding
# Initialisierung

# fit_transform()

# Ausgabe mit print()

# OneHot Encoding mit sparse = False, categories='auto'

# values=values.reshape(len(values),1)

# Ausgabe mit print()
print(onehot_encoded)


I am working on the file "20. 나이브 베이즈/01. 나이브 베이즈 분류 모형.ipynb".
The first task is:
CountVectorize 사용
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report

clf_1 = Pipeline([
            ('vect', CountVectorizer()),
            ('clf', MultinomialNB()),
        ])

%%time
clf_1.fit(X_train, y_train)

pprint(list(clf_1.named_steps["vect"].vocabulary_)[:10])

%%time
print(classification_report(y_test, clf_1.predict(X_test)))


I am working on the file "20. 나이브 베이즈/01. 나이브 베이즈 분류 모형.ipynb".
The first task is:
감성 분석 Sentiment Analysis

서울대 박은정님의 네이버 영화 감상평에 대한 감성 분석 예제
https://github.com/e9t/nsmc
https://www.lucypark.kr/slides/2015-pyconkr/

데이터 전처리
Can you write Python code for it?

import codecs
def read_data(filename):
    with codecs.open(filename, encoding='utf-8', mode='r') as f:
        data = [line.split('\t') for line in f.read().splitlines()]
        data = data[1:]   # header 제외
    return data
train_data = read_data('/home/dockeruser/data/nsmc/ratings_train.txt')
test_data = read_data('/home/dockeruser/data/nsmc/ratings_test.txt')

X = zip(*train_data)[1]
y = zip(*train_data)[2]
y = np.array(y, dtype=int)

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=10000, test_size=10000)
len(X_train), len(X_test)

from konlpy.utils import pprint
pprint((X[0], y[0]))

from konlpy.tag import Twitter
pos_tagger = Twitter()

%%time
def tokenize(doc):
    return ['/'.join(t) for t in pos_tagger.pos(doc, norm=True, stem=True)]

train_docs = [(tokenize(row[1]), row[2]) for row in train_data[:10000]]
tokens = [t for d in train_docs for t in d[0]]
import nltk
text = nltk.Text(tokens, name='NMSC')
mpl.rcParams["font.family"] = "NanumGothic"
plt.figure(figsize=(12,10))
text.plot(50)
plt.show()


I am working on the file "20. 나이브 베이즈/01. 나이브 베이즈 분류 모형.ipynb".
The first task is:
베르누이 분포 나이브 베이즈 모형
베르누이 나이브 베이즈 모형에서는 타겟 변수뿐 아니라 독립 변수도 0 또는 1의 값을 가져야 한다. 예를 들어 전자우편과 같은 문서 내에 특정한 단어가 포함되어 있는지의 여부는 베르누이 확률 변수로 모형화할 수 있으므로 스팸 필터링에 사용할 수 있다.
Can you write Python code for it?

np.random.seed(0)
X = np.random.randint(2, size=(10, 4))
y = np.array([0,0,0,0,1,1,1,1,1,1])
print(X)
print(y)

from sklearn.naive_bayes import BernoulliNB
clf_bern = BernoulliNB().fit(X, y)

clf_bern.classes_

clf_bern.class_count_

np.exp(clf_bern.class_log_prior_)

fc = clf_bern.feature_count_
fc

fc / np.repeat(clf_bern.class_count_[:, np.newaxis], 4, axis=1)

theta = np.exp(clf_bern.feature_log_prob_)
theta

x_new = np.array([1, 1, 0, 0])

clf_bern.predict_proba([x_new])

p = ((theta**x_new)*(1-theta)**(1-x_new)).prod(axis=1)*np.exp(clf_bern.class_log_prior_)
p / p.sum()

x_new = np.array([0, 0, 1, 1])

clf_bern.predict_proba([x_new])

p = ((theta**x_new)*(1-theta)**(1-x_new)).prod(axis=1)*np.exp(clf_bern.class_log_prior_)
p / p.sum()


I am working on the file "20. 나이브 베이즈/01. 나이브 베이즈 분류 모형.ipynb".
The first task is:
예 1: 뉴스 그룹
Can you write Python code for it?

from sklearn.datasets import fetch_20newsgroups
from sklearn.cross_validation import train_test_split
news = fetch_20newsgroups(subset="all")
X_train, X_test, y_train, y_test = train_test_split(news.data, news.target, test_size=0.1, random_state=1)

from sklearn.feature_extraction.text import TfidfVectorizer, HashingVectorizer, CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

clf_1 = Pipeline([
            ('vect', CountVectorizer()),
            ('clf', MultinomialNB()),
        ])
clf_2 = Pipeline([
            ('vect', TfidfVectorizer()),
            ('clf', MultinomialNB()),
        ])
clf_3 = Pipeline([
            ('vect', TfidfVectorizer(token_pattern=r"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b")),
            ('clf', MultinomialNB()),
        ])
clf_4 = Pipeline([
            ('vect', TfidfVectorizer(stop_words="english",
                                     token_pattern=r"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b")),
            ('clf', MultinomialNB()),
        ])
clf_5 = Pipeline([
            ('vect', TfidfVectorizer(stop_words="english",
                                     token_pattern=ur"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b")),
            ('clf', MultinomialNB(alpha=0.01)),
        ])

from sklearn.cross_validation import cross_val_score, KFold
from scipy.stats import sem

for i, clf in enumerate([clf_1, clf_2, clf_3, clf_4, clf_5]):
    scores = cross_val_score(clf, X_test, y_test, cv=5)
    print(("Model {0:d}: Mean score: {1:.3f} (+/-{2:.3f})").format(i, np.mean(scores), sem(scores)))


I am working on the file "20. 나이브 베이즈/01. 나이브 베이즈 분류 모형.ipynb".
The first task is:
최적화
Can you write Python code for it?

clf_4 = Pipeline([
            ('vect', TfidfVectorizer(tokenizer=tokenize_pos, ngram_range=(1,2))),
            ('clf', MultinomialNB()),
        ])

%%time
clf_4.fit(X_train, y_train)

%%time
print(classification_report(y_test, clf_4.predict(X_test), digits=4))


I am working on the file "2001 SAT Scores Analysis/Analysis of 2001 Iowa SAT Scores.ipynb".
The first task is:
# Step 3: Describe the data
12. Print the min and max of each column
Can you write Python code for it?

#Convert to a pandas dataframe to perform functions.
SAT_scores = pd.DataFrame(sat_data)
SAT_scores['Math'] = SAT_scores.Math.astype(int)
SAT_scores['Verbal'] = SAT_scores.Verbal.astype(int)
SAT_scores['Rate'] = SAT_scores.Rate.astype(int)

print 'The minimum Verbal score is' + ' ' + str(min(SAT_scores.Verbal))
print 'The maximum Verbal score is' + ' ' + str(max(SAT_scores.Verbal))
print 'The minimum Math score is' + ' ' + str(min(SAT_scores.Math))
print 'The maximum Math score is' + ' ' + str(max(SAT_scores.Math))
print 'The minimum Rate is' + ' ' + str(min(SAT_scores.Rate))
print 'The maximum Rate is' + ' ' + str(max(SAT_scores.Rate))