I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
Now, complete the next cell to be able to evaluate the function compute_Pearson_correlation( ).
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# 1. From trainingRDD, create an RDD where each element is (userID, (ItemID, rating)), i.e,
# the userID is the key and the pair (ItemID, rating) is the value.
RDD_users_as_key = #FILL IN

# 2. Group the elements of RDD_users_as_key by key (see groupByKey() method)
# Each element of this new RDD is (userID, spark-iterable), where the spark iterable has
# a list with all the rated items elements (ItemID, rating)
RDD_users_ratings = #FILL IN

# 3. Extract the spark-iterable element with all the ratings of users 1 and 2
id_u1 = 1
ratings_u1 =  #FILL IN
id_u2 = 2
ratings_u2 =  #FILL IN

# 4. Compute its similarity
n_items_th = 4
compute_Pearson_correlation(ratings_u1, ratings_u2,  n_items_th)

###########################################################
# TEST CELL
###########################################################

Test.assertEquals(np.round(similarity,2), 0.80, 'incorrect result: similarity value is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
Now, let's evaluate the performance of the mean based baseline.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# Compute the MAE error for each mean based baseline
MAE_mean = # FILL IN
# Compute the RMSE error for each mean based baseline
RMSE_mean = # FILL IN

print 'Mean model ... MAE: %2.2f , RMSE: %2.2f ' % (MAE_mean, RMSE_mean)

###########################################################
# TEST CELL
###########################################################

Test.assertEquals(np.round(MAE_mean,2), 0.83, 'incorrect result: MAE value of mean recommeder is incorrect')
Test.assertEquals(np.round(RMSE_mean,2), 1.04, 'incorrect result: RMSE value of mean recommeder is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
Once we can compute similarities between two users, let's compute, for each user, its distance with all the remaining users. The output of this cell will be an RDD of similarities where each element is (UserID, spark-iterable), where spark-iterable is a iterable list with pairs of (UserID, similarity).
Note that it is enough if this list only saves the users with a similarity larger than zero or larger than a given threshold.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# 1. From trainingRDD, create an new RDD with elements (userID, spark-iterable), where
# spark iterable has a list [(ItemID, rating), ...] with all the items rated by UserID
# (see previous section)
RDD_users_ratings  = # FILL IN

# 2. Create all the combinations of pairs with the users (see cartesian method of RDD elements)
# Note that cartesian returns an RDD with elements ((id_1, iterable_ratings 1), (id_2, iterable_ratings 2))
pairs_users = # FILL IN

# 3. Compute correlation values with the function compute_Pearson_correlation()
n_items_th = 4
correlation_values = # FILL IN

# 4. Select correlation values larger than the similarity threshold (filter method)
sim_th = 0.2
correlation_values_sel = # FILL IN

# 5. Let's reorganize each element of the RDD to get user 1 as key and the tuple
# (user2, similarity) as value
all_correlations_with_userid = # FILL IN

# 6. Group the elements of all_correlations_with_userid by key (groupByKey() method)
# Each element of this new RDD is (userID, spark-iterable), where the spark iterable has
# a list with all the similar users (UserID, similarity)
RDD_sim_users = # FILL IN
RDD_sim_users.cache()

###########################################################
# TEST CELL
###########################################################

id_user = 1
sim_user1 = RDD_sim_users.filter(lambda x : x[0]==id_user).first()
sim_check = sc.parallelize(list(sim_user1[1]))

Test.assertEquals(np.round(sim_check.filter(lambda x: x[0] == 22).first()[1],2), 0.34, 'incorrect result: similarity value is incorrect')
Test.assertEquals(np.round(sim_check.filter(lambda x: x[0] == 120).first()[1],2), 0.37, 'incorrect result: similarity value is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
Performance evaluation
Finally, let's evaluate the model performance over the test data using the get_MAE( ) and get_RMSE( ) functions.
Can you write Python code for it?


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
Step 2. Making predictions
Once you know how a user is similar to other users, you would like to know which items should be recommended for this user.
For this purpose, we have to assign a rating to each item by averaging the ratings that the similar users have given to that item according to this expression:
$$ pred(user_a, item_i) = \bar{r_a} + \frac{\sum_{b \in N} sim(user_a, user_b) * (r_{b,i}- \bar{r_b})}{\sum_{b \in N} sim(user_a, user_b)}$$
where N is the number of neighbors of user a ($sim >sim_th$) which have rated item i.
Next cell contains the necessary code to compute the above expression given the average rating of a user. Review this function, paying special attention to the input parameters.
Can you write Python code for it?

def compute_predictions(med_user, list_sim, list_ratings):

    """ Estimate the rating that a user u would assign over a item i
    Args:
        med_user: average rating of the user u
        list_sim: list of tuples (id_user, similarity) with the users who are
        similar to the user u and its similarity value
        list_rantings: list of tuples (id_user, rating) with the ratings that the remaining
        users have already assigned to the item i. Note that the rating values are normalized
        (the average rating of the corresponding user has been previously subtracted so that
        this function implements the above expression)
    Returns:
        pred_value: estimated rating for the user u to the item i
    """
    if (list_sim is not None) & (list_ratings is not None):
        dict1 = dict(list_sim)
        dict2 = dict(list_ratings)
        list_intersect = [(k, dict1[k], dict2[k]) for k in sorted(dict1) if k in dict2]
        # We have build a list with: (user_id_similar, sim_value, rating_user_sim)
        if len(list_intersect)>0:
            aux = [(sim*rat, sim) for (id_user, sim, rat) in list_intersect]
            numerator, denominator = zip(*aux)
            pred_value = med_user + sum(numerator)/sum(denominator)
        else:
            pred_value = med_user
    else:
         pred_value = med_user
    return pred_value


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
To obtain the predicted outputs for the test data and evaluate the performance of the user based recommender, we need to compute all the needed input arguments. Follow the steps of the next sections to obtain them.
1. Computing the average rating of each user
Please, review section baseline recommender (Subsection 2).
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# 1. From trainingRDD create a new RDD with the fields (user, rating), and convert it to
# (user, list_ratings). Hint: GroupByKey()
RDD_users_ratings = # FILL IN
# Convert this RDD (user, list_ratings) -> (user, mean_user). Use getAverages() function
RDD_users_mean  = # FILL IN

###########################################################
# TEST CELL
###########################################################

id_user = 1
mean_user1 = RDD_users_mean.filter(lambda x : x[0]==id_user).first()

Test.assertEquals(np.round(mean_user1[1],2), 3.6, 'incorrect result: mean rating value is incorrect')


I am working on the file "5_Test_Model.ipynb".
The first task is:
Define a function that predicts on a test set by using batches
This allows the user to use and hdf5 matrix as the input rather than loading the entire dataset in to a numpy array
Can you write Python code for it?

def batch_predictions(model, spectra, batch_size, denormalize):

    predictions = np.zeros((len(spectra),3))
    num_batches = int(len(spectra)/batch_size)
    for i in range(num_batches):
        inputs = spectra[i*batch_size:(i+1)*batch_size]
        # Mask any nan values
        indices_nan = np.where(np.isnan(inputs))
        inputs[indices_nan]=0.
        predictions[i*batch_size:(i+1)*batch_size] = denormalize(model.predict(inputs))
    num_remainder = int(len(spectra)%batch_size)
    if num_remainder>0:
        inputs = spectra[-num_remainder:]
        indices_nan = np.where(np.isnan(inputs))
        inputs[indices_nan]=0.
        predictions[-num_remainder:] = denormalize(model.predict(inputs))
    return predictions


I am working on the file "5_Test_Model.ipynb".
The first task is:
Define functions to obtain test data
Can you write Python code for it?

def get_data(filename):
    f = h5py.File(datadir + filename, 'r')
    spectra_array = f['spectrum']
    ap_ids = f['Ap_ID'][:]
    labels_array = np.column_stack((f['TEFF'][:],f['LOGG'][:],f['FE_H'][:]))
    snr_array = f['combined_snr'][:]
    return  (ap_ids, snr_array, spectra_array, labels_array)


I am working on the file "5_Test_Model.ipynb".
The first task is:
Load test data and model
Can you write Python code for it?

test_ap_ids, test_snr, test_spectra, test_labels = get_data('test_data.h5')
print('Test set contains '  + str(len(test_spectra))+' stars')
model = keras.models.load_model(datadir + 'starnet_cnn.h5')


I am working on the file "5_Test_Model.ipynb".
The first task is:
Obtain data for normalizing labels and define function to denormalize labels**
Can you write Python code for it?

mean_and_std = np.load(datadir + 'mean_and_std.npy')
mean_labels = mean_and_std[0]
std_labels = mean_and_std[1]
num_labels = mean_and_std.shape[1]

def denormalize(lb_norm):
    return ((lb_norm*std_labels) + mean_labels)


I am working on the file "5_Test_Model.ipynb".
The first task is:
Show residuals on test set label predictions
Can you write Python code for it?

# Some plotting variables for asthetics
%matplotlib inline
# Label names
label_names = ['$T_{\mathrm{eff}}$', '$\log(g)$', '$[Fe/H]$']

# Pipeline names
x_lab = 'ASPCAP'
y_lab = 'StarNet'

plt.rcParams['axes.facecolor']='white'
sns.set_style("ticks")
plt.rcParams['axes.grid']=True
plt.rcParams['grid.color']='gray'
plt.rcParams['grid.alpha']='0.4'

plt.rc('text', usetex=True)
plt.rc('font', family='serif')

all_targets = test_labels
all_pred = test_predictions
z = test_snr
z[z>250]=250

resid = all_pred - all_targets

# Overplot high S/N
order = (z).reshape(z.shape[0],).argsort()
all_targets = all_targets[order]
resid = resid[order]
z = z[order,0]

bias = np.median(resid, axis=0)
scatter = np.std(resid, axis=0)

indices_a = np.where(z>=150)
indices_b = np.where(z<100)
resid_a = resid[indices_a,:]
resid_b = resid[indices_b,:]

cmap = sns.cubehelix_palette(8, start=2.8, rot=.1, dark=0, light=.95, as_cmap=True)

lims = [[(3800,5800),(-0.50,4.50),(-2.5,0.6)],[(-1000,1000),(-2.0,2.0),(-1.,1.)]]
ditribution_lims = [(-200,200),(-0.4,0.4),(-0.2,0.2)]

fig = plt.figure(figsize=(38, 30))
gs = gridspec.GridSpec(3, 2,  width_ratios=[4., 1])

for i in range(num_labels):

    ax0 = plt.subplot(gs[i,0])
    points = ax0.scatter(all_targets[:,i], resid[:,i], c=z, s=100, cmap=cmap)

    ax0.set_xlabel(x_lab + ' ' + label_names[i], fontsize=70)

    ax0.set_ylabel(r'$\Delta$ %s ' % (label_names[i]) +
                       '\n' + r'(%s - %s)' % (y_lab, x_lab), fontsize=70)

    ax0.tick_params(labelsize=50, width=1, length=10)
    ax0.set_xlim(lims[0][i])
    ax0.set_ylim(lims[1][i])
    ax0.plot([lims[0][i][0],lims[0][i][1]], [0,0], 'k--', lw=2)

    xmin, xmax = ditribution_lims[i]

    y_a = resid_a[0,:,i][(resid_a[0,:,i]>=xmin)&(resid_a[0,:,i]<=xmax)]
    y_b = resid_b[0,:,i][(resid_b[0,:,i]>=xmin)&(resid_b[0,:,i]<=xmax)]

    ax1 = plt.subplot(gs[i,1])
    a = sns.distplot(y_a, vertical=True,hist=False, rug=False, ax=ax1,kde_kws={"color": cmap(200), "lw": 10})
    b =...


I am working on the file "5_VisualizationSchoolCleanupEDA.ipynb".
The first task is:
EDA
Can you write Python code for it?

# Load schools_v2.csv (clean school data)
school_df = pd.read_csv("tempdata/schools_v2.csv")
school_df.rename(columns={"Unnamed: 0" : "rowid"    }, inplace=True)
school_df.head(10)

print len(school_df)
df_summary = school_df.describe()
df_summary

# Function takes DataFrame as input and returns
# 1. DF with column_name, unique values in a column, length of unique values, perc of NaN in column values
# 2. Invalid Columns (with more than 85% NaNs)]
# 3. Indicator Columns
def df_desc(in_DF):
    col_dicts = []
    INVALID_COLS = []
    INDICATOR_COLS = []
    for col in in_DF.columns:
        col_dict={}
        uniq_vals=in_DF[col].unique()
        l_uniq = len(uniq_vals)
        l_NaN = len(in_DF[pd.isnull(school_df[col])])
        l_DF = len(in_DF)
        NaN_perc = l_NaN/99074.0
        col_dict['col_name']=col
        col_dict['col_uniqval_cnt']=l_uniq
        col_dict['col_NaN_perc'] = NaN_perc
        col_dict['uniq_vals'] = uniq_vals
        col_dicts.append(col_dict)
        if NaN_perc < .85:
            if l_uniq <= 10: # Adding columns with few unique values to Indicators columns list
                INDICATOR_COLS.append(col)
        else:    # Adding columns with high NaN % to invalid columns list
            INVALID_COLS.append(col)
    df_col_uv = pd.DataFrame(col_dicts).sort(['col_uniqval_cnt','col_NaN_perc'], ascending=[True, False]) #DF to store column unique value counts
    return df_col_uv, INVALID_COLS,INDICATOR_COLS

INVALID_COLUMNS=[]
INDICATOR_COLUMNS = []
df_col_uv, INVALID_COLUMNS, INDICATOR_COLUMNS = df_desc(school_df)
df_col_uv.to_csv("tempdata/schools_cols_uniq_val.csv")

for index, row in df_col_uv.iterrows():
    if row['col_name'] not in INVALID_COLUMNS:
        print '-------------------------------------------------------------------------------------------------'
        print row['col_name'], row['col_NaN_perc'], row['col_uniqval_cnt']#, row['uniq_vals']
        print school_df[row['col_name']].value_counts(dropna=False)

# Identified list of columns that are amenable for...


I am working on the file "5_VisualizationSchoolCleanupEDA.ipynb".
The first task is:
School Demographics : Column Notations
| Notation       | Description                              |
| :------------- | ---------------------------------------- |
| pk             | School Year : Prekindergarten            |
| k              | School Year : Kindergarten               |
| g (g1,g2… g12) | School Year : Grade (Grade1, Grade2… Grade12) |
| ug             | School Year : Ungraded                   |
| an             | Race : American Indian/Asian Native      |
| a              | Race : Asian/Pacific Islander            |
| h              | Race : Hispanic                          |
| b              | Race : Black                             |
| w              | Race : White                             |
| hp             | Race : Hawaiian Native/Pacific Islander  |
| mr             | Race : Two or more races                 |
| m              | Sex : Male                               |
| f              | Sex : Female                             |
Can you write Python code for it?

#sdf_v1.to_csv("tempdata/schools_v1.csv")
#sdf_v1 = pd.read_csv("tempdata/schools_v1.csv")

# Trimming cells that contain :
# ="abc" to abc
#  "abc" to abc
for i, col in enumerate(sdf_v1.columns):
    if sdf_v1[col].dtype == object:
        sdf_v1[col] = sdf_v1[col].str.replace('="', '')
        sdf_v1[col] = sdf_v1[col].str.replace('"', '')

# Replacing Missing Data / NA / Bad Quality data with NaN
# Ref : http://pandas.pydata.org/pandas-docs/version/0.15.2/missing_data.html

sdf_v1 = sdf_v1.replace('\xe2\x80\x93', np.nan) # Replace "-" (Missing Data) with NaN
sdf_v1 = sdf_v1.replace('\xe2\x80\xa0', np.nan) # Replace "†" (Not Applicable) with NaN
sdf_v1 = sdf_v1.replace('\xe2\x80\xa1', np.nan) # Replace "‡" (Bad Quality) with NaN

# Converting some columns to boolean data types by replacing "1-Yes", "2-No" to True & False resp.
tfmap =  {'1-Yes': True, '2-No': False}
vtfmap = {'A virtual school': True, 'Not a virtual school': False} # virtual_school
ltfmap = {'Yes under Community Eligibility Option (CEO)': True,
          'Yes participating without using any Provision or the CEO': True,
          'Yes under Provision 3': True,
          'Yes under Provision 2': True,
          'Yes under Provision 1': True,
          'No': False }

sdf_v1 = sdf_v1.replace(tfmap)
sdf_v1 = sdf_v1.replace(vtfmap)
sdf_v1 = sdf_v1.replace(ltfmap)

sdf_v1.to_csv("tempdata/schools_v2.csv")
sdf_v2 = pd.read_csv("tempdata/schools_v2.csv")

for i, col in enumerate(sdf_v2.columns):
    print i," : ", col, " : ", sdf_v2[col].dtype

sdf_v2.shape


I am working on the file "5__Merging_Concatenating.ipynb".
The first task is:
Also notice how the resultant dataframe is sorted by the index, i.e. 'IPL Team' alphabetically.
Can you write Python code for it?

# Creating a new column - 'Win Percentage'

Total['Win Percentage'] = Total['Matches Won']/Total['Matches Played']
Total

# Sorting to determine the teams with most number of wins. If the number of wins of two teams are the same, sort by the win percentage.

Total.sort_values(by = (['Matches Won', 'Win Percentage']), ascending = False)


I am working on the file "5__Merging_Concatenating.ipynb".
The first task is:
Concatenating Dataframes Having the Same Rows
You may also have dataframes having the same rows but different columns (and having no common columns). In this case, you may want to concat them side-by-side. For e.g.:
Can you write Python code for it?

df1 = pd.DataFrame({'Name': ['Aman', 'Joy', 'Rashmi', 'Saif'],
                    'Age': ['34', '31', '22', '33'],
                    'Gender': ['M', 'M', 'F', 'M']}
                  )
df1

df2 = pd.DataFrame({'School': ['RK Public', 'JSP', 'Carmel Convent', 'St. Paul'],
                    'Graduation Marks': ['84', '89', '76', '91']}
                  )
df2

# To join the two dataframes, use axis = 1 to indicate joining along the columns axis
# The join is possible because the corresponding rows have the same indices
pd.concat([df1, df2], axis = 1)


I am working on the file "5__Merging_Concatenating.ipynb".
The first task is:
Merging Dataframes
Say you want to select all orders and observe the Sales of the customer segment Corporate. Since customer segment details are present in the dataframe customer_df, we will first need to merge it with market_df.
Can you write Python code for it?

# Merging the dataframes
# Note that Cust_id is the common column/key, which is provided to the 'on' argument
# how = 'inner' makes sure that only the customer ids present in both dfs are included in the result
df_1 = pd.merge(market_df, customer_df, how='inner', on='Cust_id')
df_1.head()

# Now, you can subset the orders made by customers from 'Corporate' segment
df_1.loc[df_1['Customer_Segment'] == 'CORPORATE', :]

# Example 2: Select all orders from product category = office supplies and from the corporate segment
# We now need to merge the product_df

df_2 = pd.merge(df_1, product_df, how='inner', on='Prod_id')
df_2.head()

# Select all orders from product category = office supplies and from the corporate segment
df_2.loc[(df_2['Product_Category']=='OFFICE SUPPLIES') & (df_2['Customer_Segment']=='CORPORATE'),:]


I am working on the file "5__Merging_Concatenating.ipynb".
The first task is:
Note that you can also use the pd.concat() method to merge dataframes using common keys, though here we will not discuss that. For simplicity, we have used the pd.merge() method for database-style merging and pd.concat() for appending dataframes having no common columns.
Performing Arithmetic Operations on two or more dataframes
We can also perform simple arithmetic operations on two or more dataframes. Below are the stats for IPL 2018 and 2017.
Can you write Python code for it?

# Teamwise stats for IPL 2018
IPL_2018 = pd.DataFrame({'IPL Team': ['CSK', 'SRH', 'KKR', 'RR', 'MI', 'RCB', 'KXIP', 'DD'],
                         'Matches Played': [16, 17, 16, 15, 14, 14, 14, 14],
                         'Matches Won': [11, 10, 9, 7, 6, 6, 6, 5]}
                       )

# Set the 'IPL Team' column as the index to perform arithmetic operations on the other rows using the team as reference
IPL_2018.set_index('IPL Team', inplace = True)
IPL_2018

# Similarly, we have the stats for IPL 2017
IPL_2017 = pd.DataFrame({'IPL Team': ['MI', 'RPS', 'KKR', 'SRH', 'KXIP', 'DD', 'GL', 'RCB'],
                         'Matches Played': [17, 16, 16, 15, 14, 14, 14, 14],
                         'Matches Won': [12, 10, 9, 8, 7, 6, 4, 3]}
                       )
IPL_2017.set_index('IPL Team', inplace = True)
IPL_2017

# Simply add the two DFs using the add opearator

Total = IPL_2018 + IPL_2017
Total


I am working on the file "5__Merging_Concatenating.ipynb".
The first task is:
Notice that there are a lot of NaN values. This is because some teams which played in IPL 2017 were not present in IPL 2018. In addition, there were also new teams present in IPL 2018. We can handle these NaN values by using df.add() instead of the simple add operator. Let's see how.
Can you write Python code for it?

# The fill_value argument inside the df.add() function replaces all the NaN values in the two dataframes w.r.t. each other with zero.
Total = IPL_2018.add(IPL_2017, fill_value = 0)
Total


I am working on the file "5__Merging_Concatenating.ipynb".
The first task is:
Similary, you can merge the other dimension tables - shipping_df and orders_df to create a master_df and perform indexing using any column in the master dataframe.
Can you write Python code for it?

# Merging shipping_df
df_3 = pd.merge(df_2, shipping_df, how='inner', on='Ship_id')
df_3.shape

# Merging the orders table to create a master df
master_df = pd.merge(df_3, orders_df, how='inner', on='Ord_id')
master_df.shape
master_df.head()


I am working on the file "5__Merging_Concatenating.ipynb".
The first task is:
Similary, you can perform left, right and outer merges (joins) by using the argument how = 'left' / 'right' / 'outer'.
Concatenating Dataframes
Concatenation is much more straightforward than merging. It is used when you have dataframes having the same columns and want to append them (pile one on top of the other), or having the same rows and want to append them side-by-side.
Concatenating Dataframes Having the Same columns
Say you have two dataframes having the same columns, like so:
Can you write Python code for it?

# dataframes having the same columns
df1 = pd.DataFrame({'Name': ['Aman', 'Joy', 'Rashmi', 'Saif'],
                    'Age': ['34', '31', '22', '33'],
                    'Gender': ['M', 'M', 'F', 'M']}
                  )

df2 = pd.DataFrame({'Name': ['Akhil', 'Asha', 'Preeti'],
                    'Age': ['31', '22', '23'],
                    'Gender': ['M', 'F', 'F']}
                  )
df1

df2

# To concatenate them, one on top of the other, you can use pd.concat
# The first argument is a sequence (list) of dataframes
# axis = 0 indicates that we want to concat along the row axis
pd.concat([df1, df2], axis = 0)

# A useful and intuitive alternative to concat along the rows is the append() function
# It concatenates along the rows
df1.append(df2)


I am working on the file "5_data_model_for_text.ipynb".
The first task is:
Back to the PyCon proposals dataset
Can you write Python code for it?


I am working on the file "5_data_model_for_text.ipynb".
The first task is:
Exercise: Find the top 10 most frequently occuring words in the "description" column of the dataset.
Can you write Python code for it?


I am working on the file "5_data_model_for_text.ipynb".
The first task is:
Pick a corpus consisting only of the title of the talk
Can you write Python code for it?

corpus = df['title']
vect = CountVectorizer(stop_words='english')
X = vect.fit_transform(corpus).toarray()
pd.DataFrame(X, columns=vect.get_feature_names())


I am working on the file "5_data_model_for_text.ipynb".
The first task is:
Quantifying text: The Term-Document Matrix

Three types of metrics:

Presence or absense of a word

The number of times a word appears

The probability of a word occuring
Can you write Python code for it?

corpus = [
    "it was the best of times",
    "it was the worst of times",
    "it was the age of wisdom",
    "it was the age of foolishness"
]

vect = CountVectorizer(binary=True)
X = vect.fit_transform(corpus).toarray()
pd.DataFrame(X, columns=vect.get_feature_names())

vect = CountVectorizer()
X = vect.fit_transform(corpus).toarray()
pd.DataFrame(X, columns=vect.get_feature_names())

vect = TfidfVectorizer()
X = vect.fit_transform(corpus).toarray()
pd.DataFrame(X, columns=vect.get_feature_names())


I am working on the file "5_data_model_for_text.ipynb".
The first task is:
What just happened?
Can you write Python code for it?

corpus = df['title']
vect = CountVectorizer(stop_words='english', max_features=10)
X = vect.fit_transform(corpus).toarray()
pd.DataFrame(X, columns=vect.get_feature_names())


I am working on the file "5_face_generation/dlnd_face_generation.ipynb".
The first task is:
CelebA
Run your GANs on CelebA.  It will take around 20 minutes on the average GPU to run one epoch.  You can run the whole epoch or stop when it starts to generate realistic faces.
Can you write Python code for it?

batch_size = 128
z_dim = 100
learning_rate = 0.0004
beta1 = 0.

tf.reset_default_graph()

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
epochs = 1

celeba_dataset = helper.Dataset('celeba', glob(os.path.join(data_dir, 'img_align_celeba/*.jpg')))
with tf.Graph().as_default():
    train(epochs, batch_size, z_dim, learning_rate, beta1, celeba_dataset.get_batches,
          celeba_dataset.shape, celeba_dataset.image_mode)


I am working on the file "5_face_generation/dlnd_face_generation.ipynb".
The first task is:
CelebA
The CelebFaces Attributes Dataset (CelebA) dataset contains over 200,000 celebrity images with annotations.  Since you're going to be generating faces, you won't need the annotations.  You can view the first number of examples by changing show_n_images.
Can you write Python code for it?

show_n_images = 25

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
mnist_images = helper.get_batch(glob(os.path.join(data_dir, 'img_align_celeba/*.jpg'))[:show_n_images], 28, 28, 'RGB')
pyplot.imshow(helper.images_square_grid(mnist_images, 'RGB'))


I am working on the file "5_face_generation/dlnd_face_generation.ipynb".
The first task is:
Discriminator
Implement discriminator to create a discriminator neural network that discriminates on images.  This function should be able to reuse the variabes in the neural network.  Use tf.variable_scope with a scope name of "discriminator" to allow the variables to be reused.  The function should return a tuple of (tensor output of the generator, tensor logits of the generator).
Can you write Python code for it?

# Helper functions for _discriminator_ and _generator_

from operator import mul
from functools import reduce

def leaky_relu(x, alpha = 0.2):
    return tf.maximum(x,alpha*x)

def batch_norm(x, training):
    return tf.layers.batch_normalization(x, training=True)

def flatten(x):
    image_shape = x.get_shape()[1:4]
    image_shape = map(int, image_shape)
    image_size = reduce(mul, image_shape, 1)

    return tf.reshape(x, (-1, image_size))

def deconv2d(x, n_filters, padding, kernel, stride, is_train=False, activation=leaky_relu, normalize=True):
    x = tf.layers.conv2d_transpose(x, n_filters, kernel, stride, padding, use_bias=(not normalize))

    if normalize == True:
        x = batch_norm(x, is_train)

    return activation(x)

def conv2d(x, n_filters, activation=leaky_relu, normalize=True):
    x = tf.layers.conv2d(x, n_filters, (5,5), (2,2), 'same', use_bias=(not normalize))

    if normalize == True:
        # Discriminator is only used during training
        x = batch_norm(x, training=True)

    return activation(x)

def dense(x, width, height, n_units, is_train=False, activation=leaky_relu):
    x = tf.layers.dense(x, width*height*n_units, use_bias=False)
    x = tf.reshape(x, (-1, width, height, n_units))
    x = batch_norm(x, is_train)

    return activation(x)

def discriminator(images, reuse=False):
    """
    Create the discriminator network
    :param image: Tensor of input image(s)
    :param reuse: Boolean if the weights should be reused
    :return: Tuple of (tensor output of the discriminator, tensor logits of the discriminator)
    """
    with tf.variable_scope('discriminator', reuse=reuse):

        # Convolutional input layer without batch normalization
        out = conv2d(images, 32, normalize=False)

        # Convolution hidden layers with leaky relu activation, no pooling
        out = conv2d(out, 64)
        out = conv2d(out, 128)

        # Fully connected output layer with sigmoid activation
        out = flatten(out)
 ...


I am working on the file "5_face_generation/dlnd_face_generation.ipynb".
The first task is:
Explore the Data
MNIST
As you're aware, the MNIST dataset contains images of handwritten digits. You can view the first number of examples by changing show_n_images.
Can you write Python code for it?

show_n_images = 25

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
import os
from glob import glob
from matplotlib import pyplot

mnist_images = helper.get_batch(glob(os.path.join(data_dir, 'mnist/*.jpg'))[:show_n_images], 28, 28, 'L')
pyplot.imshow(helper.images_square_grid(mnist_images, 'L'), cmap='gray')


I am working on the file "5_face_generation/dlnd_face_generation.ipynb".
The first task is:
Input
Implement the model_inputs function to create TF Placeholders for the Neural Network. It should create the following placeholders:
- Real input images placeholder with rank 4 using image_width, image_height, and image_channels.
- Z input placeholder with rank 2 using z_dim.
- Learning rate placeholder with rank 0.
Return the placeholders in the following the tuple (tensor of real input images, tensor of z data)
Can you write Python code for it?

import problem_unittests as tests

def model_inputs(image_width, image_height, image_channels, z_dim):
    """
    Create the model inputs
    :param image_width: The input image width
    :param image_height: The input image height
    :param image_channels: The number of image channels
    :param z_dim: The dimension of Z
    :return: Tuple of (tensor of real input images, tensor of z data, learning rate)
    """

    input_ = tf.placeholder(shape=(None, image_width, image_height, image_channels), dtype=tf.float32)
    z = tf.placeholder(shape=(None, z_dim), dtype=tf.float32)
    learning_rate = tf.placeholder(tf.float32)

    return input_, z, learning_rate

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_inputs(model_inputs)


I am working on the file "5_face_generation/dlnd_face_generation.ipynb".
The first task is:
MNIST
Test your GANs architecture on MNIST.  After 2 epochs, the GANs should be able to generate images that look like handwritten digits.  Make sure the loss of the generator is lower than the loss of the discriminator or close to 0.
Can you write Python code for it?

batch_size = 128
z_dim = 100
learning_rate = 0.001
beta1 = 0.

tf.reset_default_graph()

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
epochs = 2

mnist_dataset = helper.Dataset('mnist', glob(os.path.join(data_dir, 'mnist/*.jpg')))
with tf.Graph().as_default():
    train(epochs, batch_size, z_dim, learning_rate, beta1, mnist_dataset.get_batches,
          mnist_dataset.shape, mnist_dataset.image_mode)


I am working on the file "5_face_generation/dlnd_face_generation.ipynb".
The first task is:
Preprocess the Data
Since the project's main focus is on building the GANs, we'll preprocess the data for you.  The values of the MNIST and CelebA dataset will be in the range of -0.5 to 0.5 of 28x28 dimensional images.  The CelebA images will be cropped to remove parts of the image that don't include a face, then resized down to 28x28.
The MNIST images are black and white images with a single [color channel](https://en.wikipedia.org/wiki/Channel_(digital_image%29) while the CelebA images have [3 color channels (RGB color channel)](https://en.wikipedia.org/wiki/Channel_(digital_image%29#RGB_Images).
Build the Neural Network
You'll build the components necessary to build a GANs by implementing the following functions below:
- model_inputs
- discriminator
- generator
- model_loss
- model_opt
- train
Check the Version of TensorFlow and Access to GPU
This will check to make sure you have the correct version of TensorFlow and access to a GPU
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf

# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.0'), 'Please use TensorFlow version 1.0 or newer.  You are using {}'.format(tf.__version__)
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))


I am working on the file "5_face_generation/dlnd_face_generation.ipynb".
The first task is:
Train
Implement train to build and train the GANs.  Use the following functions you implemented:
- model_inputs(image_width, image_height, image_channels, z_dim)
- model_loss(input_real, input_z, out_channel_dim)
- model_opt(d_loss, g_loss, learning_rate, beta1)
Use the show_generator_output to show generator output while you train. Running show_generator_output for every batch will drastically increase training time and increase the size of the notebook.  It's recommended to print the generator output every 100 batches.
Can you write Python code for it?

def train(epoch_count, batch_size, z_dim, learning_rate, beta1, get_batches, data_shape, data_image_mode):
    """
    Train the GAN
    :param epoch_count: Number of epochs
    :param batch_size: Batch Size
    :param z_dim: Z dimension
    :param learning_rate: Learning Rate
    :param beta1: The exponential decay rate for the 1st moment in the optimizer
    :param get_batches: Function to get batches
    :param data_shape: Shape of the data
    :param data_image_mode: The image mode to use for images ("RGB" or "L")
    """

    n_images, width, height, n_channels = data_shape
    n_batches = n_images // batch_size
    images, z, lr = model_inputs(width, height, n_channels, z_dim)
    d_loss, g_loss = model_loss(images, z, n_channels)
    d_opt, g_opt = model_opt(d_loss, g_loss, lr, beta1)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        epochs = range(epoch_count)
        for epoch_i in epochs:
            print("Epoch: ", epoch_i)
            g_losses, d_losses = [], []
            for batch_i, batch_images in enumerate(get_batches(batch_size), 1):

                # Generating input for the generator
                batch_z = np.random.uniform(-1, 1, size=(batch_size, z_dim))
                # Scaling for tanh domain, [-1,1]
                batch_real = 2*batch_images

                # Train
                g_feed = {z: batch_z, lr: learning_rate}
                d_feed = {z: batch_z, images: batch_real, lr: learning_rate}
                sess.run( g_opt, g_feed )
                sess.run( d_opt, d_feed )
                print(".", end="",flush=True)

                # Check Losses
                if batch_i % 100 == 0:
                    generator = sess.run(g_loss, g_feed)
                    discriminator = sess.run(d_loss, d_feed)

                    print("\t{0}/{1} Losses:\tD={2:6.4f}\t\tG={3:6.4f}".format(
                        batch_i, n_batches, discriminator, generator))
            ...


I am working on the file "5_feature_prediction.ipynb".
The first task is:
TRAINING TIME!
Can you write Python code for it?

#Train the model
model.fit(
    X_train,
    y_train,
    batch_size=512,
    nb_epoch=200,
    validation_split=0.05)


I am working on the file "5_feature_prediction.ipynb".
The first task is:
This is what our data looks like
We have 6 features: The opening price, the daily high, low, the adjusted closing price, the volume and finally the actual closing price
We will first preprocess our data and then build a LSTM network in order to model the fluctuations of the S&P 500 as a time series
Here's a look at what the data looks like:
Can you write Python code for it?


I am working on the file "5_lda_blank.ipynb".
The first task is:
Changing the hyperparameter gamma
Just as we were able to see the effect of alpha by plotting topic weights for a document, we expect to be able to visualize the impact of changing gamma by plotting word weights for each topic. In this case, however, there are far too many words in our vocabulary to do this effectively. Instead, we'll plot the total weight of the top 100 words and bottom 1000 words for each topic. Below, we plot the (sorted) total weights of the top 100 words and bottom 1000 from each topic in the high, original, and low gamma models.
Now we will consider the following two models:
 - tpm_low_gamma, a model trained with gamma = 0.02 and default alpha
 - tpm_high_gamma, a model trained with gamma = 0.5 and default alpha
Can you write Python code for it?

del tpm_low_alpha
del tpm_high_alpha
tpm_low_gamma = gl.load_model('lda_low_gamma')
tpm_high_gamma = gl.load_model('lda_high_gamma')

a_top = np.sort([sum(tpm_low_gamma.get_topics(topic_ids=[i], num_words=100)['score']) for i in range(10)])[::-1]
b_top = np.sort([sum(topic_model.get_topics(topic_ids=[i], num_words=100)['score']) for i in range(10)])[::-1]
c_top = np.sort([sum(tpm_high_gamma.get_topics(topic_ids=[i], num_words=100)['score']) for i in range(10)])[::-1]

a_bot = np.sort([sum(tpm_low_gamma.get_topics(topic_ids=[i], num_words=547462)[-1000:]['score']) for i in range(10)])[::-1]
b_bot = np.sort([sum(topic_model.get_topics(topic_ids=[i], num_words=547462)[-1000:]['score']) for i in range(10)])[::-1]
c_bot = np.sort([sum(tpm_high_gamma.get_topics(topic_ids=[i], num_words=547462)[-1000:]['score']) for i in range(10)])[::-1]

ind = np.arange(len(a))
width = 0.3

param_bar_plot(a_top, b_top, c_top, ind, width, ylim=0.6, param='gamma',
               xlab='Topics (sorted by weight of top 100 words)',
               ylab='Total Probability of Top 100 Words')

param_bar_plot(a_bot, b_bot, c_bot, ind, width, ylim=0.0002, param='gamma',
               xlab='Topics (sorted by weight of bottom 1000 words)',
               ylab='Total Probability of Bottom 1000 Words')


I am working on the file "5_lda_blank.ipynb".
The first task is:
Comparing LDA to nearest neighbors for document retrieval
So far we have found that our topic model has learned some coherent topics, we have explored these topics as probability distributions over a vocabulary, and we have seen how individual documents in our Wikipedia data set are assigned to these topics in a way that corresponds with our expectations.
In this section, we will use the predicted topic distribution as a representation of each document, similar to how we have previously represented documents by word count or TF-IDF. This gives us a way of computing distances between documents, so that we can run a nearest neighbors search for a given document based on its membership in the topics that we learned from LDA. We can contrast the results with those obtained by running nearest neighbors under the usual TF-IDF representation, an approach that we explored in a previous assignment.
We'll start by creating the LDA topic distribution representation for each document:
Can you write Python code for it?

wiki['lda'] = topic_model.predict(wiki_docs, output_type='probability')


I am working on the file "5_lda_blank.ipynb".
The first task is:
For each of our two different document representations, we can use GraphLab Create to compute a brute-force nearest neighbors model:
Can you write Python code for it?

model_tf_idf = gl.nearest_neighbors.create(wiki, label='name', features=['tf_idf'],
                                           method='brute_force', distance='cosine')
model_lda_rep = gl.nearest_neighbors.create(wiki, label='name', features=['lda'],
                                            method='brute_force', distance='cosine')


I am working on the file "5_lda_blank.ipynb".
The first task is:
From these two plots we can see that the low gamma model results in higher weight placed on the top words and lower weight placed on the bottom words for each topic, while the high gamma model places relatively less weight on the top words and more weight on the bottom words. Thus increasing gamma results in topics that have a smoother distribution of weight across all the words in the vocabulary.
Quiz Question: For each topic of the low gamma model, compute the number of words required to make a list with total probability 0.5. What is the average number of words required across all topics? (HINT: use the get_topics() function from GraphLab Create with the cdf_cutoff argument).
Can you write Python code for it?

tpm_low_gamma.get_topics(cdf_cutoff = 0.5).print_rows(num_rows = 50)


I am working on the file "5_lda_blank.ipynb".
The first task is:
Here we see that, for our topic model, the top 10 words only account for a small fraction (in this case, between 5% and 13%) of their topic's total probability mass. So while we can use the top words to identify broad themes for each topic, we should keep in mind that in reality these topics are more complex than a simple 10-word summary.
Finally, we observe that some 'junk' words appear highly rated in some topics despite our efforts to remove unhelpful words before fitting the model; for example, the word 'born' appears as a top 10 word in three different topics, but it doesn't help us describe these topics at all.
Topic distributions for some example documents
As we noted in the introduction to this assignment, LDA allows for mixed membership, which means that each document can partially belong to several different topics. For each document, topic membership is expressed as a vector of weights that sum to one; the magnitude of each weight indicates the degree to which the document represents that particular topic.
We'll explore this in our fitted model by looking at the topic distributions for a few example Wikipedia articles from our data set. We should find that these articles have the highest weights on the topics whose themes are most relevant to the subject of the article - for example, we'd expect an article on a politician to place relatively high weight on topics related to government, while an article about an athlete should place higher weight on topics related to sports or competition.
Topic distributions for documents can be obtained using GraphLab Create's predict() function. GraphLab Create uses a collapsed Gibbs sampler similar to the one described in the video lectures, where only the word assignments variables are sampled.  To get a document-specific topic proportion vector post-facto, predict() draws this vector from the conditional distribution given the sampled word assignments in the document.  Notice that, since these are draws from a distribution over topics that the model has...
Can you write Python code for it?

obama = gl.SArray([wiki_docs[int(np.where(wiki['name']=='Barack Obama')[0])]])
pred1 = topic_model.predict(obama, output_type='probability')
pred2 = topic_model.predict(obama, output_type='probability')
print(gl.SFrame({'topics':themes, 'predictions (first draw)':pred1[0], 'predictions (second draw)':pred2[0]}))


I am working on the file "5_lda_blank.ipynb".
The first task is:
Identifying topic themes by top words
We'll start by trying to identify the topics learned by our model with some major themes. As a preliminary check on the results of applying this method, it is reasonable to hope that the model has been able to learn topics that correspond to recognizable categories. In order to do this, we must first recall what exactly a 'topic' is in the context of LDA.
In the video lectures on LDA we learned that a topic is a probability distribution over words in the vocabulary; that is, each topic assigns a particular probability to every one of the unique words that appears in our data. Different topics will assign different probabilities to the same word: for instance, a topic that ends up describing science and technology articles might place more probability on the word 'university' than a topic that describes sports or politics. Looking at the highest probability words in each topic will thus give us a sense of its major themes. Ideally we would find that each topic is identifiable with some clear theme and that all the topics are relatively distinct.
We can use the GraphLab Create function get_topics() to view the top words (along with their associated probabilities) from each topic.
Quiz Question: Identify the top 3 most probable words for the first topic.
Can you write Python code for it?

topic_model

topic_model['topics']

topic_model.get_topics([0],3)


I am working on the file "5_lda_blank.ipynb".
The first task is:
In the above plot, each line corresponds to one of our ten topics. Notice how for each topic, the weights drop off sharply as we move down the ranked list of most important words. This shows that the top 10-20 words in each topic are assigned a much greater weight than the remaining words - and remember from the summary of our topic model that our vocabulary has 547462 words in total!
Next we plot the total weight assigned by each topic to its top 10 words:
Can you write Python code for it?

top_probs = [sum(topic_model.get_topics(topic_ids=[i], num_words=10)['score']) for i in range(10)]

ind = np.arange(10)
width = 0.5

fig, ax = plt.subplots()

ax.bar(ind-(width/2),top_probs,width)
ax.set_xticks(ind)

plt.xlabel('Topic')
plt.ylabel('Probability')
plt.title('Total Probability of Top 10 Words in each Topic')
plt.xlim(-0.5,9.5)
plt.ylim(0,0.15)
plt.show()


I am working on the file "5_lda_blank.ipynb".
The first task is:
It is certainly useful to have pre-implemented methods available for LDA, but as with our previous methods for clustering and retrieval, implementing and fitting the model gets us only halfway towards our objective. We now need to analyze the fitted model to understand what it has done with our data and whether it will be useful as a document classification system. This can be a challenging task in itself, particularly when the model that we use is complex. We will begin by outlining a sequence of objectives that will help us understand our model in detail. In particular, we will

get the top words in each topic and use these to identify topic themes
predict topic distributions for some example documents
compare the quality of LDA "nearest neighbors" to the NN output from the first assignment
understand the role of model hyperparameters alpha and gamma

Load a fitted topic model
The method used to fit the LDA model is a randomized algorithm, which means that it involves steps that are random; in this case, the randomness comes from Gibbs sampling, as discussed in the LDA video lectures. Because of these random steps, the algorithm will be expected to yield slighty different output for different runs on the same data - note that this is different from previously seen algorithms such as k-means or EM, which will always produce the same results given the same input and initialization.
It is important to understand that variation in the results is a fundamental feature of randomized methods. However, in the context of this assignment this variation makes it difficult to evaluate the correctness of your analysis, so we will load and analyze a pre-trained model.
We recommend that you spend some time exploring your own fitted topic model and compare our analysis of the pre-trained model to the same analysis applied to the model you trained above.
Can you write Python code for it?


I am working on the file "5_lda_blank.ipynb".
The first task is:
Let's compare these nearest neighbor models by finding the nearest neighbors under each representation on an example document. For this example we'll use Paul Krugman, an American economist:
Can you write Python code for it?

model_tf_idf.query(wiki[wiki['name'] == 'Paul Krugman'], label='name', k=10)

model_lda_rep.query(wiki[wiki['name'] == 'Paul Krugman'], label='name', k=10)


I am working on the file "5_lda_blank.ipynb".
The first task is:
Let's look at the top 10 words for each topic to see if we can identify any themes:
Can you write Python code for it?

[x['words'] for x in topic_model.get_topics(output_type='topic_words', num_words=10)]


I am working on the file "5_lda_blank.ipynb".
The first task is:
Measuring the importance of top words
We can learn more about topics by exploring how they place probability mass (which we can think of as a weight) on each of their top words.
We'll do this with two visualizations of the weights for the top words in each topic:
 - the weights of the top 100 words, sorted by the size
 - the total weight of the top 10 words
Here's a plot for the top 100 words by weight in each topic:
Can you write Python code for it?

for i in range(10):
    plt.plot(range(100), topic_model.get_topics(topic_ids=[i], num_words=100)['score'])
plt.xlabel('Word rank')
plt.ylabel('Probability')
plt.title('Probabilities of Top 100 Words in each Topic')


I am working on the file "5_lda_blank.ipynb".
The first task is:
Next we add the TF-IDF document representations:
Can you write Python code for it?

wiki['word_count'] = gl.text_analytics.count_words(wiki['text'])
wiki['tf_idf'] = gl.text_analytics.tf_idf(wiki['word_count'])


I am working on the file "5_lda_blank.ipynb".
The first task is:
Notice that that there is no overlap between the two sets of top 10 nearest neighbors. This doesn't necessarily mean that one representation is better or worse than the other, but rather that they are picking out different features of the documents.
With TF-IDF, documents are distinguished by the frequency of uncommon words. Since similarity is defined based on the specific words used in the document, documents that are "close" under TF-IDF tend to be similar in terms of specific details. This is what we see in the example: the top 10 nearest neighbors are all economists from the US, UK, or Canada.
Our LDA representation, on the other hand, defines similarity between documents in terms of their topic distributions. This means that documents can be "close" if they share similar themes, even though they may not share many of the same keywords. For the article on Paul Krugman, we expect the most important topics to be 'American college and politics' and 'science and research'. As a result, we see that the top 10 nearest neighbors are academics from a wide variety of fields, including literature, anthropology, and religious studies.
Quiz Question: Using the TF-IDF representation, compute the 5000 nearest neighbors for American baseball player Alex Rodriguez. For what value of k is Mariano Rivera the k-th nearest neighbor to Alex Rodriguez? (Hint: Once you have a list of the nearest neighbors, you can use mylist.index(value) to find the index of the first instance of value in mylist.)
Quiz Question: Using the LDA representation, compute the 5000 nearest neighbors for American baseball player Alex Rodriguez. For what value of k is Mariano Rivera the k-th nearest neighbor to Alex Rodriguez? (Hint: Once you have a list of the nearest neighbors, you can use mylist.index(value) to find the index of the first instance of value in mylist.)
Can you write Python code for it?

alex_list = model_tf_idf.query(wiki[wiki['name'] == 'Alex Rodriguez'], label='name', k=10000)

alex_list =  alex_list['reference_label']

list(alex_list).index('Mariano Rivera')

alex_lda_list = model_lda_rep.query(wiki[wiki['name'] == 'Alex Rodriguez'], label='name', k=10000)

index =  list(alex_lda_list['reference_label']).index('Mariano Rivera')

print index


I am working on the file "5_lda_blank.ipynb".
The first task is:
Quiz Question: How many topics are assigned a weight greater than 0.3 or less than 0.05 for the article on Paul Krugman in the high alpha model? Use the average results from 100 topic predictions.
Can you write Python code for it?

print average_predictions(tpm_high_alpha, paul, 100 )


I am working on the file "5_lda_blank.ipynb".
The first task is:
Quiz Question: What are the top 3 topics corresponding to the article about English football (soccer) player Steven Gerrard? Use the average results from 100 topic predictions.
Can you write Python code for it?

steven = gl.SArray([wiki_docs[int(np.where(wiki['name']=='Steven Gerrard')[0])]])

print average_predictions(topic_model, steven, 100)


I am working on the file "5_lda_blank.ipynb".
The first task is:
Quiz Question: What was the value of gamma used to fit our original topic model? Remember that GraphLab Create uses "beta" instead of "gamma" to refer to the hyperparameter that influences topic distributions over words.
Can you write Python code for it?


I am working on the file "5_lda_blank.ipynb".
The first task is:
Understanding the role of LDA model hyperparameters
Finally, we'll take a look at the effect of the LDA model hyperparameters alpha and gamma on the characteristics of our fitted model. Recall that alpha is a parameter of the prior distribution over topic weights in each document, while gamma is a parameter of the prior distribution over word weights in each topic.
In the video lectures, we saw that alpha and gamma can be thought of as smoothing parameters when we compute how much each document "likes" a topic (in the case of alpha) or how much each topic "likes" a word (in the case of gamma). In both cases, these parameters serve to reduce the differences across topics or words in terms of these calculated preferences; alpha makes the document preferences "smoother" over topics, and gamma makes the topic preferences "smoother" over words.
Our goal in this section will be to understand how changing these parameter values affects the characteristics of the resulting topic model.
Quiz Question: What was the value of alpha used to fit our original topic model?
Can you write Python code for it?


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Additionally, suppose $g'(x)$ is defined for $x \in [a,b]$ and $\exists K < 1$ s.t. $|g'(x)| \leq K < 1 ~~~ \forall ~~~ x \in (a,b)$, then $g$ has a unique fixed point $P \in [a,b]$
Can you write Python code for it?

x = numpy.linspace(0.4, 0.8, 100)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, -numpy.exp(-x), 'r')
axes.set_xlabel("x")
axes.set_ylabel("f(x)")
plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Algorithm

Initialize $x_k$
Begin loop and calculate what $x_{k+1}$
Check stopping criteria
Can you write Python code for it?

P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)
f_prime = lambda r, A=A, m=m, P=P, n=n: \
            -P*m*n*(1.0 + r/m)**(m*n)/(r*(1.0 + r/m)) \
                + P*m*((1.0 + r/m)**(m*n) - 1.0)/r**2

# Algorithm parameters
MAX_STEPS = 100
TOLERANCE = 1e-4

# Initial guess
x_k = 0.06

# Setup figure to plot convergence
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

for n in xrange(1, MAX_STEPS + 1):
    axes.text(x_k, f(x_k), str(n))
    x_k = x_k - f(x_k) / f_prime(x_k)
    if numpy.abs(f(x_k)) < TOLERANCE:
        break

if n == MAX_STEPS:
    print "Reached maximum number of steps!"
else:
    print "Success!"
    print "  x* = %s" % x_k
    print "  f(x*) = %s" % f(x_k)
    print "  number of steps = %s" % n

axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
axes.set_title("Newton-Raphson Steps")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Algorithm
Given $f(x)$, given bracket $[a,b]$, a TOLERANCE, and a MAX_STEPS

Initialize $x_1 = a$, $x_2 = b$, $f_1 = f(x_1)$, and $f_2 = f(x_2)$
Loop until either MAX_STEPS is reached or TOLERANCE is achieved
Calculate new update $x_{k+1}$
Check for convergence and break if reached
Update parameters $x_1$, $x_2$, $f_1 = f(x_1)$ and $f_2(x_2)$
Celebrate
Can you write Python code for it?

P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)
f_prime = lambda r, A=A, m=m, P=P, n=n: \
            -P*m*n*(1.0 + r/m)**(m*n)/(r*(1.0 + r/m)) \
                + P*m*((1.0 + r/m)**(m*n) - 1.0)/r**2

# Algorithm parameters
MAX_STEPS = 100
TOLERANCE = 1e-4

# Initial guess
x_k = 0.07
x_km = 0.06

# Setup figure to plot convergence
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

for n in xrange(1, MAX_STEPS + 1):
    axes.plot(x_k, f(x_k), 'o')
    x_kp = x_k - f(x_k) * (x_k - x_km) / (f(x_k) - f(x_km))
    x_km = x_k
    x_k = x_kp
    if numpy.abs(f(x_k)) < TOLERANCE:
        break

if n == MAX_STEPS:
    print "Reached maximum number of steps!"
else:
    print "Success!"
    print "  x* = %s" % x_k
    print "  f(x*) = %s" % f(x_k)
    print "  number of steps = %s" % n

axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
axes.set_title("Secant Method")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Better ways for root-finding/optimization
If $x^$ is a fixed point of $g(x)$ then $x^$ is also a root of $f(x^) = g(x^) - x^$ s.t. $f(x^) = 0$.
$$f(r) = r - \frac{m P}{A} \left [ \left (1 + \frac{r}{m} \right)^{m n} - 1 \right ] =0 $$
or
$$f(r) = A - \frac{m P}{r} \left [ \left (1 + \frac{r}{m} \right)^{m n} - 1 \right ] =0 $$
Classical Methods

Bisection (linear convergence)
Newton's Method (quadratic convergence)
Secant Method (super-linear)

Combined Methods

RootSafe (Newton + Bisection)
Brent's Method (Secant + Bisection)

Bracketing and Bisection
A bracket is an interval $[a,b]$ s.t. $\text{sign}(f(a)) \neq \text{sign}(f(b))$.
Theorem:  If $f(x) \in C[a,b]$ and $\text{sign}(f(a)) \neq \text{sign}(f(b))$ then there exists a number $c \in (a,b)$ s.t. $f(c) = 0$.  (proof uses intermediate value theorem)
Can you write Python code for it?

P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.1, 100)
f = lambda r, A, m, P, n: A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r, A, m, P, n), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')
axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

a = 0.075
b = 0.095
axes.plot(a, f(a, A, m, P, n), 'ko')
axes.plot([a, a], [0.0, f(a, A, m, P, n)], 'k--')
axes.plot(b, f(b, A, m, P, n), 'ko')
axes.plot([b, b], [f(b, A, m, P, n), 0.0], 'k--')

plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Bisection Algorithm
Given a bracket $[a,b]$ and a function $f(x)$ -
1. Initialize with bracket
2. Iterate
   1. Cut bracket in half and check to see where the zero is
   2. Set bracket to new bracket based on what direction we went
Can you write Python code for it?

P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)

# Initialize bracket
a = 0.07
b = 0.10

# Setup figure to plot convergence
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r, A, m, P, n), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')
axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
# axes.set_xlim([0.085, 0.091])
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
axes.plot(a, f(a, A, m, P, n), 'ko')
axes.plot([a, a], [0.0, f(a, A, m, P, n)], 'k--')
axes.plot(b, f(b, A, m, P, n), 'ko')
axes.plot([b, b], [f(b, A, m, P, n), 0.0], 'k--')

# Algorithm parameters
TOLERANCE = 1e-4
MAX_STEPS = 100

# Initialize loop
f_a = f(a)
f_b = f(b)
delta_x = b - a

# Loop until we reach the TOLERANCE or we take MAX_STEPS
for step in xrange(MAX_STEPS):
    c = a + delta_x / 2.0
    f_c = f(c)
    if numpy.sign(f_a) != numpy.sign(f_c):
        b = c
        f_b = f_c
    else:
        a = c
        f_a = f_c
    delta_x = b - a

    # Plot iteration
    axes.text(c, f(c), str(step))

    # Check tolerance - Could also check the size of delta_x
    if numpy.abs(f_c) < TOLERANCE:
        break

if step == MAX_STEPS:
    print "Reached maximum number of steps!"
else:
    print "Success!"
    print "  x* = %s" % c
    print "  f(x*) = %s" % f(c)
    print "  number of steps = %s" % step


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Comments

Secant method as shown is equivalent to linear interpolation
Can use higher order interpolation for higher order secant methods
Convergence is not quite quadratic
Not gauranteed to converge
Do not preserve brackets
Almost as good as Newton's method if your initial guess is good.

Hybrid Methods
Combine attributes of methods with others to make one great algorithm to rule them all (not really)
Goals

Robustness:  Given a bracket $[a,b]$, maintain bracket
Efficiency:  Use superlinear convergent methods when possible

Options

Methods requiring $f'(x)$
NewtSafe (RootSafe, Numerical Recipes)
Newton's Method within a bracket, Bisection otherwise
Methods not requiring $f'(x)$
Brent's Algorithm (zbrent, Numerical Recipes)
Combination of bisection, secant and inverse quadratic interpolation

scipy.optimize package

Optimization (finding extrema)
I want to find the extrema of a function $f(x)$ on a given interval $[a,b]$.
A few approaches:
 - Bracketing Algorithms:  Golden-Section Search (linear)
 - Interpolation Alogithms: Repeated parabolic interpolation
 - Hybrid Algorithms
Bracketing Algorithm (Golden Section Search)
Given $f(x) \in C[a,b]$ that is convex over an interval $x \in [a,b]$ reduce the interval size until it brackets the minimum.
Note that we no longer have the $x=0$ help we had before so bracketing and doing bisection is a bit more tricky in this case.  In particular choosing your initial bracket is important!
Bracketing Algorithm (Golden Section Search)
Given $f(x) \in C[a,b]$ that is convex over an interval $x \in [a,b]$ reduce the interval size until it brackets the minimum.
Note that we no longer have the $x=0$ help we had before so bracketing and doing bisection is a bit more tricky in this case.  In particular choosing your initial bracket is important!
Golden Section Search - Picking Intervals
We also may want to choose the search points $c$ and $d$ so that the distance between $a$ and $d$, say $\Delta_{ad}$, and $b$ and $c$, say $\Delta_{bc}$, is carefully choosen.  For Golden...
Can you write Python code for it?

# New Test Function!
def f(t):
    """Simple function for minimization demos"""
    return -3.0 * numpy.exp(-(t - 0.3)**2 / (0.1)**2) \
           +      numpy.exp(-(t - 0.6)**2 / (0.2)**2) \
           +      numpy.exp(-(t - 1.0)**2 / (0.2)**2) \
           +      numpy.sin(t)                        \
           -      2.0

t = numpy.linspace(0, 2, 200)
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, f(t))
axes.set_xlabel("t (days)")
axes.set_ylabel("People (N)")
axes.set_title("Decrease in Population due to SPAM Poisoning")

plt.show()

phi = (numpy.sqrt(5.0) - 1.0) / 2.0

TOLERANCE = 1e-4
MAX_STEPS = 100

a = 0.2
b = 0.5
c = b - phi * (b - a)
d = a + phi * (b - a)

t = numpy.linspace(0, 2, 200)
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, f(t))
axes.set_xlabel("t (days)")
axes.set_ylabel("People (N)")
axes.set_title("Decrease in Population due to SPAM Poisoning")

success = False
for n in xrange(1, MAX_STEPS + 1):
    axes.plot(a, f(a),'ko')
    axes.plot(b, f(b),'ko')
    fc = f(c)
    fd = f(d)

    if fc < fd:
        b = d
        d = c
        c = b - phi * (b - a)
    else:
        a = c
        c = d
        d = a + phi * (b - a)

    if numpy.abs(b - a) < TOLERANCE:
        success = True
        break

if success:
    print "Success!"
    print "  t* = %s" % str((b + a) / 2.0)
    print "  f(t*) = %s" % f((b + a) / 2.0)
    print "  number of steps = %s" % n
else:
    print "Reached maximum number of steps!"

plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Comments

Secant method as shown is equivalent to linear interpolation
Can use higher order interpolation for higher order secant methods
Convergence is not quite quadratic
Not gauranteed to converge
Do not preserve brackets
Almost as good as Newton's method if your initial guess is good.

Hybrid Methods
Combine attributes of methods with others to make one great algorithm to rule them all (not really)
Goals

Robustness:  Given a bracket $[a,b]$, maintain bracket
Efficiency:  Use superlinear convergent methods when possible

Options

Methods requiring $f'(x)$
NewtSafe (RootSafe, Numerical Recipes)
Newton's Method within a bracket, Bisection otherwise
Methods not requiring $f'(x)$
Brent's Algorithm (zbrent, Numerical Recipes)
Combination of bisection, secant and inverse quadratic interpolation

scipy.optimize package

Optimization (finding extrema)
I want to find the extrema of a function $f(x)$ on a given interval $[a,b]$.
A few approaches:
 - Bracketing Algorithms:  Golden-Section Search (linear)
 - Interpolation Alogithms: Repeated parabolic interpolation
 - Hybrid Algorithms
Bracketing Algorithm (Golden Section Search)
Given $f(x) \in C[a,b]$ that is convex over an interval $x \in [a,b]$ reduce the interval size until it brackets the minimum.
Note that we no longer have the $x=0$ help we had before so bracketing and doing bisection is a bit more tricky in this case.  In particular choosing your initial bracket is important!
Golden Section Search - The Idea
Assume $f(x)$ is unimodal (has a single minimum in the bracket) and given a bracket $[a,b]$, pick a point inside of the bracket that is not in the middle say $c$.  Since $f(x)$ is unimodal we know $f(c) < f(a)$ and $f(c) < f(b)$.  Now we attempt to choose a new point, say $d$, which has two possibilities:
1. $f(d) < f(c)$ in which case we know the minimum lies between $c$ and $b$.
2. $f(d) > f(c)$ in which case we know the minimum lies between $a$ and $c$

Golden Section Search - Picking Intervals
We also may want to choose the search points $c$ and...
Can you write Python code for it?

def f(t):
    """Simple function for minimization demos"""
    return -3.0 * numpy.exp(-(t - 0.3)**2 / (0.1)**2) \
           +      numpy.exp(-(t - 0.6)**2 / (0.2)**2) \
           +      numpy.exp(-(t - 1.0)**2 / (0.2)**2) \
           +      numpy.sin(t)                        \
           -      2.0

t = numpy.linspace(0, 2, 200)
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, f(t))
axes.set_xlabel("t (days)")
axes.set_ylabel("People (N)")
axes.set_title("Decrease in Population due to SPAM Poisoning")

plt.show()

phi = (numpy.sqrt(5.0) - 1.0) / 2.0

TOLERANCE = 1e-4
MAX_STEPS = 100

a = 0.3
b = 0.5
c = b - phi * (b - a)
d = a + phi * (b - a)

success = False
for n in xrange(1, MAX_STEPS):
    fc = f(c)
    fd = f(d)
    if fc < fd:
        b = d
        d = c
        c = b - phi * (b - a)
    else:
        a = c
        c = d
        d = a + phi * (b - a)
    if numpy.abs(c - d) < TOLERANCE:
        success = True
        break

if success:
    print "Success!"
    print "  t* = %s" % str((b + a) / 2.0)
    print "  f(t*) = %s" % f((b + a) / 2.0)
    print "  number of steps = %s" % n
else:
    print "Reached maximum number of steps!"


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Convergence of Bisection
$$|e_{k+1}| = C |e_k|^n$$
$$e_k \approx \Delta x_k$$
$$e_{k+1} \approx \frac{1}{2} \Delta x_k$$
$$|e_{k+1}| = \frac{1}{2} |e_k|$$
$\Rightarrow$ Linear convergence
Newton's Method (Newton-Raphson)

Given a bracket, bisection is guaranteed to converge linearly to a root
However bisection uses almost no information about $f(x)$ beyond its sign at a point

Basic Idea: Given $f(x)$ and $f'(x)$ use a linear approximation to $f(x)$ "locally" and use x-intercept of the resulting line to predict where $x^*$ might be.
Given current location $x_k$, we have $f(x_k)$ and $f'(x_k)$ and form a line through the point $(x_k, f(x_k))$:
Form equation for the line:
$$y = f'(x_k) x + b$$
Solve for the y-intercept value $b$
$$f(x_k) = f'(x_k) x_k + b$$
$$b = f(x_k) - f'(x_k) x_k$$
and simplify.
$$y = f'(x_k) x + f(x_k) - f'(x_k) x_k$$
$$y = f'(x_k) (x - x_k) + f(x_k)$$
Now find the intersection of our line and the x-axis (i.e. when $y = 0$) and use the resulting value of $x$ to set $x_{k+1}$
$$0 = f'(x_k) (x_{k+1}-x_k) + f(x_k)$$
$$x_{k+1} = x_k-\frac{f(x_k)}{f'(x_k)}$$
Can you write Python code for it?

P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)
f_prime = lambda r, A=A, m=m, P=P, n=n: \
            -P*m*n*(1.0 + r/m)**(m*n)/(r*(1.0 + r/m)) \
                + P*m*((1.0 + r/m)**(m*n) - 1.0)/r**2

# Algorithm parameters
MAX_STEPS = 100
TOLERANCE = 1e-4

# Initial guess
x_k = 0.06

# Setup figure to plot convergence
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

# Plot x_k point
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x_k, f(x_k), 'ko')
axes.text(x_k, -5e4, "$x_k$", fontsize=16)
axes.text(x_k, f(x_k) + 2e4, "$f(x_k)$", fontsize=16)
axes.plot(r, f_prime(x_k) * (r - x_k) + f(x_k), 'k')

# Plot x_{k+1} point
x_k = x_k - f(x_k) / f_prime(x_k)
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x_k, f(x_k), 'ko')
axes.text(x_k, 1e4, "$x_k$", fontsize=16)
axes.text(0.089, f(x_k) - 2e4, "$f(x_k)$", fontsize=16)

axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
axes.set_title("Newton-Raphson Steps")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Example 2:
Let $f(x) = x - e^{-x}$, solve $f(x) = 0$
Equivalent to $x = e^{-x}$ or $x = g(x)$ where $g(x) = e^{-x}$
Note that this problem is equivalent to $x = -\ln x$.
Can you write Python code for it?

x = numpy.linspace(0.2, 1.0, 100)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, numpy.exp(-x), 'r')
axes.plot(x, x, 'b')
axes.set_xlabel("x")
axes.set_ylabel("f(x)")

x = 0.4
for steps in xrange(7):
    print "x = ", x
    print "Residual = ", numpy.exp(-x) - x
    x = numpy.exp(-x)
    print
    axes.plot(x, numpy.exp(-x),'o',)

plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Example 3:
Let $f(x) = \ln x + x$ and solve $f(x) = 0$ or $x = -\ln x$.
Can you write Python code for it?

x = numpy.linspace(0.1, 1.0, 100)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, -numpy.log(x), 'r')
axes.plot(x, x, 'b')
axes.set_xlabel("x")
axes.set_ylabel("f(x)")
axes.set_ylim([0.0, 1.5])

x = 0.5
for steps in xrange(3):
    print "x = ", x
    print "Residual = ", numpy.log(x) + x
    x = -numpy.log(x)
    print
    axes.plot(x, -numpy.log(x),'o',)

plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Example:
$$f(x) = x - e^{-x}$$
$$f'(x) = 1 + e^{-x}$$
$$x_{k+1} = x_k - \frac{f(x_k)}{f'(x_k)} = x_k - \frac{x_k - e^{-x_k}}{1 + e^{-x_k}}$$
Asymptotic Convergence of Newton's Method
For a simple root (non-multiplicative) - Let $g(x) = x - \frac{f(x)}{f'(x)}$, then
$$x_{k+1} = g(x_k)$$
Definitions of errors and iteration:
$$x_{k+1} = x^ + e_{k+1} ~~~~~ x_k = x^ + e_k$$
General Taylor expansion:
$$x^ + e_{k+1} = g(x^ + e_k) = g(x^) + g'(x^) e_k + \frac{g''(x^*) e_k^2}{2!} + \ldots$$
Note that as before $x^$ and $g(x^)$ cancel:
$$e_{k+1} = g'(x^) e_k + \frac{g''(x^) e_k^2}{2!} + \ldots$$
What about $g'(x^*)$ though:
$$g'(x) = 1 - \frac{f'(x)}{f'(x)} + \frac{f(x)}{f''(x)}$$
which simplifies when evaluated at $x = x^*$ to
$$g'(x^) = \frac{f(x^)}{f''(x^*)} = 0$$
The expansion then simplifies to
$$e_{k+1} = \frac{g''(x^*) e_k^2}{2!} + \ldots$$
leading to the conclusion that
$$|e_{k+1}| = \left | \frac{g''(x^*)}{2!} \right | |e_k|^2$$
Newton's method is therefore quadratically convergent where the the constant is controlled by the second derivative.
For a multiple root (e.g. $f(x) = (x-1)^2$) the case is not particularly rosey unfortunately.
Example:
$f(x) = \sin (2 \pi x)$
$$x_{k+1} = x_k - \frac{\sin (2 \pi x)}{2 \pi \cos (2 \pi x)}= x_k - \frac{1}{2 \pi} \tan (2 \pi x)$$
Can you write Python code for it?

x = numpy.linspace(0, 2, 1000)
f = lambda x: numpy.sin(2.0 * numpy.pi * x)
f_prime = lambda x: 2.0 * numpy.pi * numpy.cos(2.0 * numpy.pi * x)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, f(x),'b')
axes.plot(x, f_prime(x), 'r')
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_title("Comparison of $f(x)$ and $f'(x)$")
axes.set_ylim((-2,2))
axes.set_xlim((0,2))
axes.plot(x, numpy.zeros(x.shape), 'k--')

x_k = 0.3
axes.plot([x_k, x_k], [0.0, f(x_k)], 'ko')
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x, f_prime(x_k) * (x - x_k) + f(x_k), 'k')

x_k = x_k - f(x_k) / f_prime(x_k)
axes.plot([x_k, x_k], [0.0, f(x_k)], 'ko')
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')

plt.show()

x = numpy.linspace(0, 2, 1000)
f = lambda x: numpy.sin(2.0 * numpy.pi * x)
x_kp = lambda x: x - 1.0 / (2.0 * numpy.pi) * numpy.tan(2.0 * numpy.pi * x)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, f(x),'b')
axes.plot(x, x_kp(x), 'r')
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_title("Comparison of $f(x)$ and $f'(x)$")
axes.set_ylim((-2,2))
axes.set_xlim((0,2))
axes.plot(x, numpy.zeros(x.shape), 'k--')

plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Fixed Point Iteration
How do we go about solving this?
Could try to solve at least partially for $r$:
$$ A = \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ] ~~~~ \Rightarrow ~~~~~$$
$$ r = \frac{P \cdot m}{A} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ] ~~~~ \Rightarrow ~~~~~$$
$$ r = g(r)$$
or
$$ g(r) - r = 0$$
Can you write Python code for it?

def g(P, m, r, n, A):
    """Reformulated minimization problem

    Based on following formula:

    g(r) = \frac{P \cdot m}{A} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ]

    :Input:
     - *P* (float) - Payment amount per compounding period
     - *m* (int) - number of compounding periods per year
     - *r* (float) - annual interest rate
     - *n* (float) - number of years to retirement
     - *A* (float) - total value after $n$ years

     :Returns:
     (float) - value of g(r)

    """

    return P * m / A * ( (1.0 + r / float(m))**(float(m) * n)
                                 - 1.0)

P = 1500.0
m = 12
n = 20.0

r = numpy.linspace(0.00, 0.1, 100)
goal = 1e6

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, g(P, m, r, n, goal))
axes.plot(r, r, 'r--')
axes.set_xlabel("r (interest rate)")
axes.set_ylabel("$g(r)$")
axes.set_title("When can I retire?")
axes.set_ylim([0, 0.12])
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Guess at $r_0$ and check to see what direction we need to go...

$r_0 = 0.0800$, $g(r_0) - r_0 = -0.009317550125425428$
$r_1 = 0.0850$, $g(r_1) - r_1 = -0.00505763375972$
$r_2 = 0.0875$, $g(r_2) - r_2 = -0.00257275331014$

A bit tedious, we can also make this algorithmic:
Can you write Python code for it?

r = 0.09
for steps in xrange(10):
    print "r = ", r
    print "Residual = ", g(P, m, r, n, goal) - r
    r = g(P, m, r, n, goal)
    print


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Other Issues
Need to supply both $f(x)$ and $f'(x)$, could be expensive
Example:  FTV equation $f(r) = A - \frac{m P}{r} \left[ \left(1 + \frac{r}{m} \right )^{m n} - 1\right]$
Can use symbolic differentiation (sympy)
Secant Methods
Is there a method with the convergence of Newton's method but without the extra derivatives?  Maybe something that calculates the derivative rather than expects it?
Given $x_k$ and $x_{k-1}$ represent the derivative as
$$f'(x) \approx \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}}$$
Combining this with the basic approach of Newton leads to
$$x_{k+1} = x_k - \frac{f(x_k) (x_k - x_{k-1}) }{f(x_k) - f(x_{k-1})}$$
This leads to superlinear convergence (the exponent on the convergence is $\approx 1.7$)
Alternative interpretation, fit a line through two points and see where they intersect the x-axis.
$$(x_k, f(x_k)) ~~~~~ (x_{k-1}, f(x_{k-1})$$
$$y = \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x - x_k) + b$$
$$b = f(x_{k-1}) - \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x_{k-1} - x_k)$$
$$ y = \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x - x_k) + f(x_k)$$
Now solve for $x_{k+1}$ which is where the line intersects the x-axies ($y=0$)
$$0 = \frac{f(x_k) - f(x_{k-1})}{x_k - x_{k-1}} (x_{k+1} - x_k) + f(x_k)$$
$$x_{k+1} = x_k - \frac{f(x_k)  (x_k - x_{k-1})}{f(x_k) - f(x_{k-1})}$$
Can you write Python code for it?

P = 1500.0
m = 12
n = 20.0
A = 1e6
r = numpy.linspace(0.05, 0.11, 100)
f = lambda r, A=A, m=m, P=P, n=n: \
        A - m * P / r * ((1.0 + r / m)**(m * n) - 1.0)

# Initial guess
x_k = 0.07
x_km = 0.06

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, f(r), 'b')
axes.plot(r, numpy.zeros(r.shape),'r--')

axes.plot(x_k, 0.0, 'ko')
axes.plot(x_k, f(x_k), 'ko')
axes.plot([x_k, x_k], [0.0, f(x_k)], 'k--')
axes.plot(x_km, 0.0, 'ko')
axes.plot(x_km, f(x_km), 'ko')
axes.plot([x_km, x_km], [0.0, f(x_km)], 'k--')

axes.plot(r, (f(x_k) - f(x_km)) / (x_k - x_km) * (r - x_k) + f(x_k), 'k')
x_kp = x_k - (f(x_k) * (x_k - x_km) / (f(x_k) - f(x_km)))
axes.plot(x_kp, 0.0, 'ro')
axes.plot([x_kp, x_kp], [0.0, f(x_kp)], 'r--')
axes.plot(x_kp, f(x_kp), 'ro')

axes.set_xlabel("r (%)")
axes.set_ylabel("f(r)")
axes.set_title("Secant Method")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))

plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Root Finding and Optimization
GOAL: Find where $f(x) = 0$.
Example:  Future Time Annuity
When can I retire?
$$ A = \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n} - 1 \right ] $$
$P$ is payment amount per compounding period
$m$ number of compounding periods per year
$r$ annual interest rate
$n$ number of years to retirement
$A$ total value after $n$ years
If I want to retire in 20 years what does $r$ need to be?
Set $P = \frac{\$18,000}{12} = \$1500, ~~~~ m=12, ~~~~ n=20$.
Can you write Python code for it?

def total_value(P, m, r, n):
    """Total value of portfolio given parameters

    Based on following formula:

    A = \frac{P}{(r / m)} \left[ \left(1 + \frac{r}{m} \right)^{m \cdot n}
                - 1 \right ]

    :Input:
     - *P* (float) - Payment amount per compounding period
     - *m* (int) - number of compounding periods per year
     - *r* (float) - annual interest rate
     - *n* (float) - number of years to retirement

     :Returns:
     (float) - total value of portfolio

    """

    return P / (r / float(m)) * ( (1.0 + r / float(m))**(float(m) * n)
                                 - 1.0)

P = 1500.0
m = 12
n = 20.0

r = numpy.linspace(0.05, 0.1, 100)
goal = 1e6

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(r, total_value(P, m, r, n))
axes.plot(r, numpy.ones(r.shape) * goal, 'r--')
axes.set_xlabel("r (interest rate)")
axes.set_ylabel("A (total value)")
axes.set_title("When can I retire?")
axes.ticklabel_format(axis='y', style='sci', scilimits=(-1,1))
plt.show()


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Scipy Optimization
Scipy contains a lot of ways for optimization!
Can you write Python code for it?

import scipy.optimize as optimize
optimize.golden(f, brack=(0.2, 0.25, 0.5))


I am working on the file "5_root_finding_optimization.ipynb".
The first task is:
Theorem 2: Asymptotic convergence behavior of fixed point iterations
$$x_{k+1} = g(x_k)$$
Assume that $\exists ~ x^$ s.t. $x^ = g(x^*)$
$$x_k = x^ + e_k ~~~~~~~~~~~~~~ x_{k+1} = x^ + e_{k+1}$$
$$x^ + e_{k+1} = g(x^ + e_k)$$
Using a Taylor expansion we know
$$g(x^ + e_k) = g(x^) + g'(x^) e_k + \frac{g''(x^) e_k^2}{2}$$
$$x^ + e_{k+1} = g(x^) + g'(x^) e_k + \frac{g''(x^) e_k^2}{2}$$
Note that because $x^ = g(x^)$ these terms cancel leaving
$$e_{k+1} = g'(x^) e_k + \frac{g''(x^) e_k^2}{2}$$
So if $|g'(x^*)| \leq K < 1$ we can conclude that
$$|e_{k+1}| = K |e_k|$$
which shows convergence (although somewhat arbitrarily fast).
Convergence of iterative schemes
Given any iterative scheme where
$$|e_{k+1}| = C |e_k|^n$$
If $C < 1$ and
 - $n=1$ then the scheme is linearly convergence
 - $n=2$ then the scheme exhibits quadratic convergence
 - $n > 1$ the scheme can also be called superlinearly convergent
If $C > 1$ then the scheme is divergent
Examples Revisited
$g(x) = e^{-x}$ with $x^* \approx 0.56$
$$|g'(x^)| = |-e^{-x^}| \approx 0.56$$
$g(x) = - \ln x$ with $x^* \approx 0.56$
$$|g'(x^)| = \frac{1}{|x^|} \approx 1.79$$
$g(r) = \frac{m P}{A} ((1 + \frac{r}{m})^{mn} - 1)$ with $r^* \approx 0.09$
$$|g'(r^*)| = \frac{P m n}{A} \left(1 + \frac{r}{m} \right)^{m n - 1} \approx 2.15$$
Can you write Python code for it?

import sympy
m, P, A, r, n = sympy.symbols('m, P, A, r, n')
(m * P / A * ((1 + r / m)**(m * n) - 1)).diff(r)


I am working on the file "5_word2vec.ipynb".
The first task is:
All the values are abstract, there is practical meaning of the them. Moreover, the final embeddings are normalized as you can see here:
Can you write Python code for it?

print(np.sum(np.square(final_embeddings[0])))

num_points = 400

tsne = TSNE(perplexity=30, n_components=2, init='pca', n_iter=5000)
two_d_embeddings = tsne.fit_transform(final_embeddings[1:num_points+1, :])

def plot(embeddings, labels):
  assert embeddings.shape[0] >= len(labels), 'More labels than embeddings'
  pylab.figure(figsize=(15,15))  # in inches
  for i, label in enumerate(labels):
    x, y = embeddings[i,:]
    pylab.scatter(x, y)
    pylab.annotate(label, xy=(x, y), xytext=(5, 2), textcoords='offset points',
                   ha='right', va='bottom')
  pylab.show()

words = [reverse_dictionary[i] for i in range(1, num_points+1)]
plot(two_d_embeddings, words)


I am working on the file "5_word2vec.ipynb".
The first task is:
Build the dictionary and replace rare words with UNK token.
Can you write Python code for it?

vocabulary_size = 50000

def build_dataset(words):
  count = [['UNK', -1]]
  count.extend(collections.Counter(words).most_common(vocabulary_size - 1))
  dictionary = dict()
  for word, _ in count:
    dictionary[word] = len(dictionary)
  data = list()
  unk_count = 0
  for word in words:
    if word in dictionary:
      index = dictionary[word]
    else:
      index = 0  # dictionary['UNK']
      unk_count = unk_count + 1
    data.append(index)
  count[0][1] = unk_count
  reverse_dictionary = dict(zip(dictionary.values(), dictionary.keys()))
  return data, count, dictionary, reverse_dictionary

data, count, dictionary, reverse_dictionary = build_dataset(words)
print('Most common words (+UNK)', count[:5])
print('Sample data', data[:10])
del words  # Hint to reduce memory.


vocabulary_size = 50000

def build_dataset(words):
  count = [['UNK', -1]]
  count.extend(collections.Counter(words).most_common(vocabulary_size - 1))
  dictionary = dict()
  for word, _ in count:
    dictionary[word] = len(dictionary)
  data = list()
  unk_count = 0
  for word in words:
    if word in dictionary:
      index = dictionary[word]
    else:
      index = 0  # dictionary['UNK']
      unk_count = unk_count + 1
    data.append(index)
  count[0][1] = unk_count
  reverse_dictionary = dict(zip(dictionary.values(), dictionary.keys()))
  return data, count, dictionary, reverse_dictionary

data, count, dictionary, reverse_dictionary = build_dataset(words)
print ('Most common words (+UNK)', count[:5])
print ('Sample data', data[:10])
del words  # Hint to reduce memory.


I am working on the file "5_word2vec.ipynb".
The first task is:
Download the data from the source website if necessary.
Can you write Python code for it?

url = 'http://mattmahoney.net/dc/'

def maybe_download(filename, expected_bytes):
  """Download a file if not present, and make sure it's the right size."""
  if not os.path.exists(filename):
    filename, _ = urlretrieve(url + filename, filename)
  statinfo = os.stat(filename)
  if statinfo.st_size == expected_bytes:
    print('Found and verified %s' % filename)
  else:
    print(statinfo.st_size)
    raise Exception(
      'Failed to verify ' + filename + '. Can you get to it with a browser?')
  return filename

filename = maybe_download('text8.zip', 31344016)


#url = 'http://mattmahoney.net/dc/'
import urllib.request
url = urllib.request.urlretrieve("http://mattmahoney.net/dc/")
def maybe_download(filename, expected_bytes):
  """Download a file if not present, and make sure it's the right size."""
  if not os.path.exists(filename):
    filename, _ = urllib.request.urlretrieve(url + filename, filename)
  statinfo = os.stat(filename)
  if statinfo.st_size == expected_bytes:
    print ('Found and verified', filename)
  else:
    print (statinfo.st_size)
    raise Exception('Failed to verify ' + filename + '. Can you get to it with a browser?')
  return filename

#filename = maybe_download("text8.zip",31344016)


I am working on the file "5_word2vec.ipynb".
The first task is:
Function to generate a training batch for the skip-gram model.
Can you write Python code for it?

data_index = 0

def generate_batch(batch_size, num_skips, skip_window):
  global data_index
  assert batch_size % num_skips == 0
  assert num_skips <= 2 * skip_window
  batch = np.ndarray(shape=(batch_size), dtype=np.int32)
  labels = np.ndarray(shape=(batch_size, 1), dtype=np.int32)
  span = 2 * skip_window + 1 # [ skip_window target skip_window ]
  buffer = collections.deque(maxlen=span)
  for _ in range(span):
    buffer.append(data[data_index])
    data_index = (data_index + 1) % len(data)
  for i in range(batch_size // num_skips):
    target = skip_window  # target label at the center of the buffer
    targets_to_avoid = [ skip_window ]
    for j in range(num_skips):
      while target in targets_to_avoid:
        target = random.randint(0, span - 1)
      targets_to_avoid.append(target)
      batch[i * num_skips + j] = buffer[skip_window]
      labels[i * num_skips + j, 0] = buffer[target]
    buffer.append(data[data_index])
    data_index = (data_index + 1) % len(data)
  return batch, labels

print('data:', [reverse_dictionary[di] for di in data[:8]])

for num_skips, skip_window in [(2, 1), (4, 2)]:
    data_index = 0
    batch, labels = generate_batch(batch_size=8, num_skips=num_skips, skip_window=skip_window)
    print('\nwith num_skips = %d and skip_window = %d:' % (num_skips, skip_window))
    print('    batch:', [reverse_dictionary[bi] for bi in batch])
    print('    labels:', [reverse_dictionary[li] for li in labels.reshape(8)])


data_index = 0

def generate_batch(batch_size, num_skips, skip_window):
  global data_index
  assert batch_size % num_skips == 0
  assert num_skips <= 2 * skip_window
  batch = np.ndarray(shape=(batch_size), dtype=np.int32)
  labels = np.ndarray(shape=(batch_size, 1), dtype=np.int32)
  span = 2 * skip_window + 1 # [ skip_window target skip_window ]
  buffer = collections.deque(maxlen=span)
  for _ in range(span):
    buffer.append(data[data_index])
    data_index = (data_index + 1) % len(data)
  for i in range(int(batch_size / num_skips)):
    target = skip_window  # target label at the center of the buffer
    targets_to_avoid = [ skip_window ]
    for j in range(num_skips):
      while target in targets_to_avoid:
        target = random.randint(0, span - 1)
      targets_to_avoid.append(target)
      batch[i * num_skips + j] = buffer[skip_window]
      labels[i * num_skips + j, 0] = buffer[target]
    buffer.append(data[data_index])
    data_index = (data_index + 1) % len(data)
  return batch, labels

batch, labels = generate_batch(batch_size=8, num_skips=2, skip_window=1)
for i in range(8):
    print (batch[i], '->', labels[i, 0])
    print (reverse_dictionary[batch[i]], '->', reverse_dictionary[labels[i, 0]])


I am working on the file "5_word2vec.ipynb".
The first task is:
Let's display the internal variables to better understand their structure:
Can you write Python code for it?

print(data[:10])
print(count[:10])
print(dictionary.items()[:10])
print(reverse_dictionary.items()[:10])


I am working on the file "5_word2vec.ipynb".
The first task is:
Note the instruction change on the loss function, with reduce_sum to sum the word vectors in the context:
Can you write Python code for it?

batch_size = 128
embedding_size = 128 # Dimension of the embedding vector.
###skip_window = 1 # How many words to consider left and right.
###num_skips = 2 # How many times to reuse an input to generate a label.
bag_window = 2 # How many words to consider left and right.
# We pick a random validation set to sample nearest neighbors. here we limit the
# validation samples to the words that have a low numeric ID, which by
# construction are also the most frequent.
valid_size = 16 # Random set of words to evaluate similarity on.
valid_window = 100 # Only pick dev samples in the head of the distribution.
valid_examples = np.array(random.sample(range(valid_window), valid_size))
num_sampled = 64 # Number of negative examples to sample.

graph = tf.Graph()

with graph.as_default(), tf.device('/cpu:0'):

  # Input data.
  train_dataset = tf.placeholder(tf.int32, shape=[batch_size, bag_window * 2])
  train_labels = tf.placeholder(tf.int32, shape=[batch_size, 1])
  valid_dataset = tf.constant(valid_examples, dtype=tf.int32)

  # Variables.
  embeddings = tf.Variable(
    tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))
  softmax_weights = tf.Variable(
    tf.truncated_normal([vocabulary_size, embedding_size],
                         stddev=1.0 / math.sqrt(embedding_size)))
  softmax_biases = tf.Variable(tf.zeros([vocabulary_size]))

  # Model.
  # Look up embeddings for inputs.
  embeds = tf.nn.embedding_lookup(embeddings, train_dataset)
  # Compute the softmax loss, using a sample of the negative labels each time.
  loss = tf.reduce_mean(
    tf.nn.sampled_softmax_loss(softmax_weights, softmax_biases, tf.reduce_sum(embeds, 1),
                               train_labels, num_sampled, vocabulary_size))

  # Optimizer.
  optimizer = tf.train.AdagradOptimizer(1.0).minimize(loss)

  # Compute the similarity between minibatch examples and all embeddings.
  # We use the cosine distance:
  norm = tf.sqrt(tf.reduce_sum(tf.square(embeddings), 1, keep_dims=True))
  normalized_embeddings = embeddings /...


I am working on the file "5_word2vec.ipynb".
The first task is:
Note: the labels is a sliding random value of the word surrounding the words of the batch.
It is not obvious with the output above, but all the data are based on index, and not the word directly.
Can you write Python code for it?


I am working on the file "5_word2vec.ipynb".
The first task is:
Problem
An alternative to Word2Vec is called CBOW (Continuous Bag of Words). In the CBOW model, instead of predicting a context word from a word vector, you predict a word from the sum of all the word vectors in its context. Implement and evaluate a CBOW model trained on the text8 dataset.
Can you write Python code for it?

# For CBOW we switch batch output and context label:

for num_skips, skip_window in [(2, 1), (4, 2)]:
    data_index = 0
    # NEW: switch batch and labels around!
    # batch, labels = generate_batch(batch_size=8, num_skips=num_skips, skip_window=skip_window)
    labels, batch = generate_batch(batch_size=8, num_skips=num_skips, skip_window=skip_window)
    print('\nwith num_skips = %d and skip_window = %d:' % (num_skips, skip_window))
    # NEW: reshaping needs to happen with batch now as that is a (batch_size,1) vector
    #print('    batch:', [reverse_dictionary[bi] for bi in batch])
    #print('    labels:', [reverse_dictionary[li] for li in labels.reshape(8)])
    print('    batch:', [reverse_dictionary[bi] for bi in batch.reshape(8)])
    print('    labels:', [reverse_dictionary[li] for li in labels])

batch_size = 128
num_skips = 2
skip_window = 1

num_steps = 100001

with tf.Session(graph=graph) as session:
  tf.initialize_all_variables().run()
  print('Initialized')
  average_loss = 0
  for step in range(num_steps):
    batch_data, batch_labels = generate_batch(batch_size, num_skips, skip_window)
    # NEW: Need to explicitly reshape dataset and labels following above switch:
    #feed_dict = {train_dataset : batch_data, train_labels : batch_labels}
    feed_dict = {train_dataset : batch_data.reshape(batch_size), train_labels : batch_labels.reshape(batch_size, 1)}
    _, l = session.run([optimizer, loss], feed_dict=feed_dict)
    average_loss += l
    if step % 2000 == 0:
      if step > 0:
        average_loss = average_loss / 2000
      # The average loss is an estimate of the loss over the last 2000 batches.
      print('Average loss at step %d: %f' % (step, average_loss))
      average_loss = 0
    # note that this is expensive (~20% slowdown if computed every 500 steps)
    if step % 10000 == 0:
      sim = similarity.eval()
      for i in xrange(valid_size):
        valid_word = reverse_dictionary[valid_examples[i]]
        top_k = 8 # number of nearest neighbors
        nearest = (-sim[i,...


I am working on the file "5_word2vec.ipynb".
The first task is:
Read the data into a string.
Can you write Python code for it?

filename=("text8.zip")
def read_data(filename):
  f = zipfile.ZipFile(filename)
  for name in f.namelist():
    return f.read(name).split()
  f.close()

words = read_data(filename)
print ('Data size', len(words))


I am working on the file "5_word2vec.ipynb".
The first task is:
Read the data into a string.
It's a 100MB of text file with ~17 million separate words.
Can you write Python code for it?

def read_data(filename):
  f = zipfile.ZipFile(filename)
  for name in f.namelist():
    return tf.compat.as_str(f.read(name)).split()
  f.close()

words = read_data(filename)
print('Data size %d' % len(words))


I am working on the file "5b_PyART_visualization.ipynb".
The first task is:
Py-ART Colormaps
Retrieve the names of colormaps and the colormap list dictionary.
The colormaps are registered with matplotlib and can be accessed by inserting 'pyart_' in front of any name.
Can you write Python code for it?

cm_names = pyart.graph.cm._cmapnames
cms = pyart.graph.cm.cmap_d

nrows = len(cm_names)
gradient = np.linspace(0, 1, 256)
gradient = np.vstack((gradient, gradient))

# Create a figure and axes instance
fig, axes = plt.subplots(nrows=nrows, figsize=(5,10))
fig.subplots_adjust(top=0.95, bottom=0.01, left=0.2, right=0.99)
axes[0].set_title('Py-Art Colormaps', fontsize=14)

# Loop through the possibilities
for nn, pymap in enumerate(cm_names):
    axes[nn].imshow(gradient, aspect='auto', cmap=cms[pymap])
    pos = list(axes[nn].get_position().bounds)
    x_text = pos[0] - 0.01
    y_text = pos[1] + pos[3]/2.
    fig.text(x_text, y_text, pymap, va='center', ha='right', fontsize=8)

# Turn off *all* ticks & spines, not just the ones with colormaps.
for ax in axes:
    ax.set_axis_off()


I am working on the file "5b_PyART_visualization.ipynb".
The first task is:
Py-ART RHI
Not only can we construct an RHI from a PPI volume, but RHI scans may be plotted as well.
Can you write Python code for it?

rhif = "data/noxp_rhi_140610232635.RAWHJFH"
rhir = pyart.io.read(rhif)
rhid = pyart.graph.RadarDisplay(rhir)

rhid.plot_rhi('reflectivity', 0, vmin=-5.0, vmax=70,)


I am working on the file "5b_PyART_visualization.ipynb".
The first task is:
Py-ART RadarMapDisplay or RadarMapDisplayCartopy
This converts the x-y coordinates to latitude and longitude overplotting on a map
Let us see what version we have. The first is works on Py-ART which uses a standard definition. For other packages that may not the second method should work
Can you write Python code for it?

pyart_ver = pyart.__version__

import pkg_resources
pyart_ver2 = pkg_resources.get_distribution("arm_pyart").version

if int(pyart_ver.split('.')[1]) == 8:
    print("8")
    nexmap = pyart.graph.RadarMapDisplayCartopy(nexr)
else:
    print("7")
    nexmap = pyart.graph.RadarMapDisplay(nexr)

limits = [-87., -82., 37., 42.]
fig, ax = plt.subplots(1, 1, figsize=(7, 7))

nexmap.plot_ppi_map('reflectivity', sweep=1, vmin=0., vmax=55., ax=ax,
                    min_lon=limits[0], max_lon=limits[1], min_lat=limits[2], max_lat=limits[3])


I am working on the file "5b_PyART_visualization.ipynb".
The first task is:
The RadarDisplay
This is the most commonly used class designed for surface-based scanning radar
Can you write Python code for it?

### Plot a NEXRAD file

nexf = "data/KILN20140429_231254_V06"
nexr = pyart.io.read(nexf)
nexd = pyart.graph.RadarDisplay(nexr)

nexr.fields.keys()

fig, ax = plt.subplots(nrows=2, ncols=2, figsize=(16, 12))
nexd.plot('reflectivity', sweep=1, cmap='pyart_NWSRef', vmin=0., vmax=55., mask_outside=False, ax=ax[0, 0])
nexd.plot_range_rings([50, 100], ax=ax[0, 0])
nexd.set_limits((-150., 150.), (-150., 150.), ax=ax[0, 0])

nexd.plot('velocity', sweep=1, cmap='pyart_NWSVel', vmin=-30, vmax=30., mask_outside=False, ax=ax[0, 1])
nexd.plot_range_rings([50, 100], ax=ax[0, 1])
nexd.set_limits((-150., 150.), (-150., 150.), ax=ax[0, 1])

nexd.plot('cross_correlation_ratio', sweep=0, cmap='pyart_BrBu12', vmin=0.85, vmax=1., mask_outside=False, ax=ax[1, 0])
nexd.plot_range_rings([50, 100], ax=ax[0, 1])
nexd.set_limits((-150., 150.), (-150., 150.), ax=ax[0, 1])

nexd.plot('differential_reflectivity', sweep=0, cmap='pyart_BuDOr12', vmin=-2, vmax=2., mask_outside=False, ax=ax[1, 1])
nexd.plot_range_rings([50, 100], ax=ax[1, 1])
nexd.set_limits((-150., 150.), (-150., 150.), ax=ax[1, 1])